The perils of Java companies

Joel Spolsky writes on the perils of Java schools.Read it. It is an excellent article.

Now dwell on the fact that many of the same conclusions could apply to you if you have never developed a piece of software in a language other than Java or C#. Have you ever created a 5000 line + program in a language other than Java/C#? Do you know (as in wrote some decent programs in) any paradigms other than OO? Do you really know OO? (If you've learned only Java you probably don't). Do you work in a Java(C#) company?

With a new year dawning, it is a good time to make a resolution to improve your programming skills. Alternatively you could just "climb the ladder" and slump into "project management" or "iso/cmm".

Year End Retrospective

Over the last year or so, I have worked alone. This was a nice change from being part of a "globally distributed team", attending stand up (and other) meetings, endless 'pairing' sessions, and other "agile" practices.

As the year draws to an end, I think it maybe worthwhile to explore and clarify what I learned and how I changed.

First, I am now utterly disgusted with 'off shored enterprise' coding. I was never really comfortable with the bland projects that wash ashore in this domain. Too often the projects were utterly boring, in spite of working with some of the best companies and people. The last year has been fantastic in terms of the challenge and complexity of programming and I hope I never have to work on a leasing/banking/insurance/blah database-to-web-and-back, C#-or-java, offshored-to-save-money-by-hiring-tonnes-of-cheap-coding_bodies project in my life again! So help me God!

Second, working on some extremely tough problems have taught me to tone down my enthusiasm for (a) "OO" and (b) "agile". Don't get me wrong, there is much that is valuable in both but these days I see their limits with painful clarity. Now, I see Objects as one (vs the only) way to tackle a problem and while I still follow some of the agile practices (like unit testing and Continous Integration), I am very skeptical of the value of "pairing" etc, outside the simplicities of the "enterprise app" domain.

Third, I am now very focussed on three major themes - (a)learning to create "tough" programs(kernel hacking is tough, creating a normal insurance app in j2ee is not), (b) applying "deep theory" to practical ends (applying Bayesian Classification to spam filtering vs design pattens/mock objects/what have you in a standard business app) and (c) working towards being *really* good at programming, in global terms (and not just be another cheap Indian offshore programmer, or even worse , manager).

More concretely, in the coming year I plan to deepen my knowledge of AI and Compilers and also learn to hack hardware,besides acquiring more mathematical understanding.

2006 looks glorious! Happy New Year all.

Update: Thanks to Teju's comment, I've thought about this a little more.Here is a quick checklist of questions

1. Is your project interesting? Do you feel exhilarated or bored when working on your project?
2. Are the people on your project there because of their expertise or are they there to increase billing? If you left the project would it be hard or easy to replace you? How unique are your skills?
3. Do you learn new things every day or do you just do the same things over and over again? (this is characteristic of most offshored "enterprise" work, imo)
4. If you had all the money you wanted, would you still choose to work on this project? Or would you do something else?

Update: Vivek Singh makes an excellent point. He says that complex enterprise systems that scale to m(b)illions of users are hard to create (well). I agree (while still believing that most apps that are outsourced to Bangalore have nowhere this level of complexity and are boring jsp/orm/db, "repeat these 3 steps a 100 times" type projects).

What Mr Fowler Really Said

Martin Fowler's post on "Humane Interfaces" has set off a maelstrom in the 'blogosphere'. (He has referred to some of the more interesting posts in his updates to his entry).

As is typical in the world of software, the original points were soon lost in a cacophony of arbitrarily drawn battle lines (java vs ruby, open vs closed class systems, Java's List interfaces vs LinkedList class etc).

This is what I understood from Martin's post. (Heavily paraphrased. Don't blame Martin for any butchery I commit on his prose)

...There are two approaches to designing interfaces. One I (Martin) label the "Humane" approach. Another is the "minimal" approach. Here is an example..{ } .. The distinctions between these approaches are worth thinking about. I (Martin) lean toward the humane approach...

As readers of this blog know, I am all in favor of subjecting opinions to logical scrutiny. But I fail to see how this can be any clearer. It seems fairly obvious that (a) the example used is just that - one example and (b) either approach can be used in any language. Where did "Java vs Ruby" come from? What am I missing?

[rant] I am getting fairly irritated at the lack of clear thinking prevalent in the software world. Aaargh! [end rant]

Why a Lexus is Not An Acceptable Ferrari

Eric Kidd explains on his blog why he thinks 'Ruby is an Acceptable Lisp'.

His thesis is debunked very effectively in the comments section of his blog entry. What I find fascinating is the thinking behind this position.

Let us look at how Eric tries to assert his position that Ruby is as dense (or occasionally denser) than Lisp. He first creates an example of a lisp function , then writes a ruby equivalent which is marginally shorter, while brushing off the capabilties of lisp (reader macros in this case) because "it is not commonly done"!

First , this thinking is fallacious. The fallacy is called "Biased Generalization". From the Nizkor Project,

Description of Biased Sample

This fallacy is committed when a person draws a conclusion about a population based on a sample that is biased or prejudiced in some manner. It has the following form:

1. Sample S, which is biased, is taken from population P.
2. Conclusion C is drawn about Population P based on S.

And this is the error. Eric takes a lisp function that can be expressed concisely in Ruby and simultaneously dismisses Lisp's built in capability to match and exceed that brevity. He ignores lisp functions that can not be expressed as well (see the sentence "Yes, yield is a special-purpose hack, and yes, it only works for functions which take a single lambda but.." -- recreating "yield" in Lisp is trivial anyway-- ) making his conclusion suspect.

Blindly equating conciseness to "power" is also suspect( I used to think this way as well, not too long ago) as is the (unsupported) argument that Ruby is 'more powerful' than Python.

The deeper problem is that, when you say "X is an acceptable Y", what you are saying is "Y is superior and we all know that and we would like to use Y, but because of reasons a, b and c we make do with X".

As someone who uses Ruby and Lisp every day (the days of hacking "enterprise" code with Java and C# are gone! Praise the Lord!), I prefer to think of them as distinct tools in my tool kit. A screw driver can be used as a hammer in an emergency but that does not make it an "acceptable" hammer.

Now I'll stop dissecting Eric's arguments (he has a right to say whatever he wants to on his blog. My irritation stems from seeing some ruby zealots (here in Bangalore) make ridiculous claims about 'the power of Ruby' while having no clue about its limits or alternatives) and focus on my notions of what makes Lisp 'more powerful' than Ruby (and vice versa!).

I consider a language superior to another based on what is possible in one that is not easily possible in another.

From this perspective the power of Lisp stems from the "code is data" concept. Thanks to the s-expression notation (the regularity of the syntax is what powers macros), code is malleable to a degree not possible in Ruby. Yes, you can jump through hoops to get some of the same effect in Ruby (e.g the ParseTree module) but nothing Ruby can do can match the full power of Lisp to manipulate code and the environment just like any other piece of data.

Thus if you are working on domains or projects that need this kind of raw power, use Lisp. Also, contary to what most "enterprise" or "agile" people might tell you , "Object Oriented " is just one paradigm amongst many. In Lisp, one can seamlessly switch from one paradigm to another and can mix and match pardigms to precisely map the problem at hand. While one can approximate some of this in Ruby with enough effort, the language leans unashamedly towards OO with a pinch of functional programming.

The more interesting question, of course is how Ruby is more powerful than lisp. The answer has 3 parts - libraries, community and ease of installation and use. Yes, from a strict 'linguistic' view point things like ease of installation are not language features, but from a user's point of view a language that is theoretically powerful but in practice needs days of configuration and compiling to get a substandard version of what Ruby offers out of the box is unusable. (compare Ruby's Rails framework with the patchwork of CL-SQL, Uncommon Web, etc frameworks). I have little sympathy for "but it is better in theory" arguments. The Lisp community, as compared to Ruby's, is extremely arrogant and unhelpful and hypersensitive to any criticism of their language. Harping on "ANSI standards" doesn't help when an easy to use, open source (no I am not willing to spend big $$ to buy a closed source implementation) implementation with comprehensive, bug free libraries for networking, database access and other "low brow" tasks is non existent.

Thus I think Eric's conclusion ("In 2005, I’d think long and hard before choosing Lisp over Ruby.") matches my experience, though for different reasons. You could substitute "Python" for "Ruby" and it would still be valid.

I don't think "ease of hiring" is an issue unless you are writing crapola "enterprise" code, in body shopping (oops make that "consulting") mode where "plug compatible programming units" is the reigning concept, "agile" rhetoric notwithstanding. If the programmer you hire can't get upto speed in either Lisp or Ruby in a couple of weeks you are wasting your money anyway. But that is the topic of another entry.

Thus the sensible thing to do would be to learn Lisp and Ruby (and Python, and Forth, and smalltalk , and C )well.They are fascinating languages and will teach you different things. Alternatively you could study one and use that exclusively. The utility of a language (and thus, its "power") depends on what you the programmer can do with it, not on some abstract notion of "power".

In other words, learn to use Ruby to maximum effect. Learn to use Lisp to maximum effect. Learn to use --your favorite language-- to maximum effect. This will give you an awareness of what each is good for and where each falls down. Whether Ruby is an effective (whatever that means) Lisp or vice versa is irrelevant. A Lexus may suit your needs better than a Ferrari but it is not an "effective Ferrari" (though it may be better than that overhyped Ford (I did not say "Java/C#" :-P ).

Availability Update

I am not available for consulting assignments till (at least) the end of March 2006. Before writing to me to enquire about availability, do read this first.

I have just started a very interesting project which attempts to integrate machine learning into a massively successful product suite.The size of the data sets and the business requirements make this fairly challenging.

This, when added to my existing commitments (including some very interesting Open Source work), leaves me no time to take on more work till end of Q1 2006.

How Agile Dies

Bill Caputo writes

Perhaps the Agile of today -- represented by things like the Microsoft TDD article, the attempts by many consulting firms to make Agile a methodology (instead of set of principles like it used to be), and the increasing adoption of "Agile" by big IT -- deserves this criticism, but the Agile movement I remember held their observation as a premise.

*That* agile movement seems to already be dying out, but its echoes might still carry the message (if not recognize the messengers)

When you see people offering to "enable" half wit teams for outrageous amounts of money, and "agile" becomes another merketdroid buzzword on the same level as 'ISO 9000 certified' or 'CMM Level 5', you know you'd better call for a priest to administer the last rites.

Many companies in Bangalore are eagerly "developing an agile practice", which (but of course!) goes hand in hand with ISO 9000/CMM Level X /Whatever other BS 'methodologies' are already in place.

One of my friends, who works in 'Big IT', recently responded to "So what do you do?" with "I work in Organized Crime. I am part of an international syndicate that cons businessmen out of insane amounts of money and delivers little value in return".

Heh!