One Man Hacking

Resurrection Part 2

2008-11-27T21:06:00.006+05:30

My new blog is here.

The Heart Must Pause

2008-03-18T16:46:00.009+05:30

I've been blogging for almost 5 years now. For the last few months, I've been feeling that this blog is more and more out of synch with who I am, what I care about and what I really want to write about.

I started writing on this blog because, when I asked someone how to get good enough to write a book, I got the reply, "First learn how to write a blog entry, then write articles, then write a series of linked articles and then you are ready for the book".

Good advice. What started as pure technical practice, along the lines of practising scales in music, turned into a medium for exploring and clarifying my thoughts. On the way, I gathered an audience and generated a "blog persona" (very unlike the real me but it was a lot of fun),met interesting women and got job offers. All good. The problem with developing such a "persona" or "voice" is that when your perspective shifts radically in a short time frame, it is hard to maintain continuity.

And so, and this is the last post on this particular blog and using this particular "voice". I do plan to write again, in the future, but whether such writing takes the form of a blog, and if so, whether I'll have an "open to all" blog, remains to be seen. If I do write, it will probably show up on my (rather bare now, but that will change, in time) website.

Anyone who has spent time reading this blog and approved or (sometimes violently) disapproved, Thank You for your time. T'was a lot of fun.

"One Man Hacking" is now dead. R.I.P

PS: for those who care about such things, this post's title comes from a poem

So We'll Go No More a-Roving by Lord Byron

SO we'll go no more a-roving

So late into the night,

Though the heart still be as loving,

And the moon still be as bright.

For the sword outwears its sheath,

And the soul outwears the breast,

And the heart must pause to breathe,

And love itself have rest.

Though the night was made for loving,

And the day returns too soon,

Yet we'll go no more a-roving

By the light of the moon.

The Implict Assembly Line

2008-03-15T09:49:00.008+05:30

George Lakoff in his book "Metaphors We Live By", notes that we model our conceptual system on metaphors but are often unaware of how they shape our thoughts, speech and acts. In other words, we use one concept to explain, understand, articulate, and work with another concept.

For example, one metaphor that is very pervasive is "Argument is War". We use war as a metaphor in thinking about argument. Witness, "He won that argument" "His criticisms were on target", "I destroyed his argument"

The point George Lakoff makes is that argument really doesn't have much to do with war, except that we make it so by choosing war as an analogy. For example, argument could be (but often isn't) a "joint search for truth" or a "game" rather than a form of abstract combat with winners and losers. Changing the underlying metaphor may change how you see something.

Once upon a time (but somewhat passe these days)the underlying metaphor for software development was "constructing a building (or bridge)" from which we still have "architects" and "engineers" , UML "modeling", and so on.

These days there seems to be a shift of metaphor. The underlying metaphor for much discussion on software development seems to be "Software Development (of the services variety) is an assembly line process".

The so called "Lean Development" (or "lean agile" or whatever the fad du jour is) talks about "pieces of work" being "flowing" through a development "pipeline" and endeavors to "reduce cycle time" through "kanban" using the "Toyota Production System".

Using this metaphor, software developers become equivalent to assembly line workers which managers or methodology experts "optimize". You can work on a "good" assembly lines like Toyota, where you have a degree of freedom to tinker with some of the details or a "bad" assembly line (uhh Wipro maybe? ) where you endlessly perform the same meaningless motions forever.

There are "kanban" or "lean" or "agile" experts out there who will "advise" you on how the assembly line should work, but will never work on an assembly line themselves! The assembly line worker doesn't have any choice on what he will work on, how long he will work on a particular assembly line, practically never comes into contact with a real customer (at best there is a "customer representative" or a "usability expert"), is relatively easily replacable. "Requirements" from an analyst "flow" to a developer and then "flows" to QA dept and so on ad infinitum.

The funny thing is that most developers subscribe to this metaphor.

I choose to think of programming skill as just that. A skill. Just because you have a skill at metalwork(say) you don't have to work on a boring job on a third world assembly line. Developers who build startups step beyond the "assembly line" mentality. They choose what to work on, how to work on it, what technology to use, what customers they target interact directly with their customers and so on. Others choose to work on opensource kernels or compilers. Others are dragged kicking and screaming into Project manager-dom.

Of course, you don't even have to make your money with your programming skills. You can be an investment banker (or astronaut or actor or musician) and do your brazing and welding on the weekends.

In my (totally personal) view, knowing how to program well is like being literate in a largely illiterate world. One day everyone (or almost everyone) will know how to write. But in the meantime you don't have to work as a scribe just because you know how to write. You could be a general or merchant or prince or cleric or warrior for hire or Dragon Slayer (umm ok note to self - less D & D) and your writing skills would give you a significant edge over your illiterate peers.

Just because you are a good programmer (you are, aren't you ? ;-)) you don't have to work for that 200,000 employee outsourcing company 12 hours a day on systems you don't care about with "industry standard",read "middle of the road" technology.

Unless you choose to.

DevCamp - A report

2008-02-11T15:22:00.002+05:30

I am still "offline" but due to popular demand, here is a very brief report on Devcamp.

Bangalore has long missed a conference for serious developers. In Bangalore you often find methodology based conferences (like the Agile confeences) or various company sponsored conferences which are thinly disguised propaganda pitches (Sun/Oracle/Microsoft Tech days for e.g). There weren't too many conferences where people who code (and like to code) could get together and exchange notes with likeminded souls.

DevCamp plugs that gap very neatly. The organizers had explicitly filtered out the usual "hands on training on blub framework X or Snake Oil Methodology Y" type events which proliferate locally.The blurb for the event read "Please assume a high level of exposure and knowledge on the part of your audience and tailor your sessions to suit. Avoid 'Hello World' and how-to sessions which can be trivially found on the net. First hand war stories, in-depth analysis of topics and hands on demos are best". Consequently, most sessions were very compelling, with people demoing their pet code bases and showing off cool hacks. People projecting code (vs slides) and actually coding on stage, made this event a refreshing break from the usual slideware based mumbo jumbo. The unconference format was a particularly good fit, because you didn't have to distort your session to fit some arbitrary conceptual boundary.

I couldn't attend as many sessions as I wanted. But I did catch Bejoy's's LinQ presentation (decent, though a bit shallow), Karthik's Erlang Testing tool (when this is finished it will bury JMeter) and Vivek Singh's session on White (brilliant, brilliant work!). The sessions I would have liked to attend but missed out on were the one on MOOSE and Siddharth's presentation on running linux on a Nintendo DS.

Thoughtworks organized the conference with clockwork precision and everything flowed smoothly. It was great to be back at TW and meet my ex colleagues - which brought home to me (again) how much I miss working with bright opinionated people- aargh I have to fix this. . I am (very) glad I don't write enterprise software any more, particularly the outsourced variety, but I do miss working with bright people.

So what could be improved? Not much really. One of the projectors didn't work with Ubuntu so I had to switch to Windows (bleh) for my presentation. My talk on monads seemed to go down well. Explaining esoterica like monads in a 30 minute session was an interesting challenge in communication, but it turned out ok. I had requests for a repeat session but I was too exhausted. - Some other day. The AC and the network conked out occasionally but these were very minor issues. There were a few too many (imo) "passive" attendees who wanted to listen more than speak, which is against the Xcamp ethos but I am hopeful that will change. Another thing future organizers need to watch out for is disguised product/recruitment pitches. There were at least one "you can code against our APIs and help us make more money" talk with zero technical content. Again not an issue for this conference , but something to watch out for in the future. All in all, a very refreshing change form the usual frivolous frothiness of the BarCamps. Not a single "blogger" or "seo marketing person" or "movie club member" in sight. Just developers and code. Bliss!

Devcamp is something that Bangalore really needs. I look forward to the next one.

[Ann] Blog Lockdown till March 15th 2008

2008-02-03T21:55:00.001+05:30

I am "away".

Regds,

Ravi

P.S.: I am attending Devcamp. Sorry for the confusion.

About GRE scores

2008-01-27T15:07:00.000+05:30

Ever since I published my GRE scores on this blog, many people write to me asking me how to prepare etc. After writing the nth email with the same content, I thought I'll write the "official" answer here once and for all.

This is the "official" answer to "What advice can you give me on how to prepare for the GRE"?.

The answer is "Nothing"!

I skimmed the Barron's guide vocabulary section and the (16 iirc) sections (in the same guide) for the quantitative section to refresh my memory one day before the exam. I was overloaded with work then and had no time to prepare.

My exam was scheduled for 8 o clock in the morning and I was awake till about 05.00 working on a program. Since I got only about 2 hours of sleep, I was in this weird half asleep/awake state, which had the effect that I was totally relaxed and did not have the bandwidth to track the time remaining etc. I just answered each question as it came up. I got the very last question in the quantitative section wrong because for the first and only time I checked the clock and found I had like 3 seconds to answer and so I panicked and randomly clicked an answer(which turned out to be wrong). Oh well.

Beyond the above I have nothing to say on the GRE. Don't bother asking.

Engineering - Some Working Definitions

2008-01-25T21:19:00.000+05:30

This is the third in a series of four blog posts. Read parts one and two, to get some context

Dr Douglas Lauffenberger's (from MIT's Biological Engineering Dept) talk (warning - mp3) provided enough ideas for a (sufficient for my purposes) working definition of "engineering".

(paraphrase begins)

Dr Lauffenberger begins by breaking down engineering into two aspects - science ( the study of things that exist ) and technology ( making things that don't exist).

So,

engineering = science (analysis - studying things that exist, break down into components, and methods of combining them) + technology (synthesis, building things by putting together the components identified by analysis ).

Engineering further adds a "design principles" (how things get put together) focus to both analysis and synthesis.

All engineers study mathematics. (This is a given).

An engineering discipline has a base of science for its components and methods of combination. A branch of engineering picks a branch of science to base itself on. So, for example, Mechanical Engineering has a base of Physics and Materials Engineering has a base of Chemistry ( + Physics and the omnipresent Mathematics).

Another way of thinking about engineering is

"measure (properties of systems of interest), (use mathematics to ) model, manipulate (components and methods of combination, guided by the model) and make (things that don't exist)". --> (1)

Yet another way to think about engineering,

Engineering = mathematics + science + application area --> (2)

There can be various combinations of (and subcomponents to) each of these three components.

The "science" component in that equation needs to be manipulatable, quantifiable, modellable etc.

(paraphrase ends)

Later in the speech, Dr L goes into why Biology only recently became modelable etc and so before that, how the various branches of BioEngineering used Physics, Chemistry etc as the underlying science, rather than biology. Biology was often the "application area", but not the underlying science. Thus at MIT biology would be a minor and other engineering disciplines like mechanical or electrical engineering (including comp sci like robotics and algorithms) would be applied to a biological domain like pharmaceuticals or prostheses.

Dr L goes on to explain how this changed and why and how Biology is, these days, a science you can base an engineering discipline on (and you really ought to listen to the full speech), but for the purposes of this post (1) ind (2) are what I am interested in. ie,

engineering = mathematics + science + application area and

(doing) engineering = measure (properties of systems of interest), ( use mathematics to ) model, manipulate (components and methods of combination, guided by the model) and make (things that don't exist).

I suggest that programming fits into the "model" part of things, complementing mathematics. This is just an insight, not rigorously tested etc, but there are a couple of straws in the wind that make me think I am right.

First, a scientist I work with explicitly identified the combination of programming and mathematical skills as a "force multiplier" that enables someone who has mastered both to zoom past someone who is strong only in one, explicitly mentioning a programmer (a genius at programming, way better than I am, who couldn't make as much progress as I could because he couldn't wrap his head around the "maths as a modelling tool" idea) and another person, a scientist this time, who gets stuck periodically because he couldn't write production quality code.

There are analogues in enterprise programming where someone who has mastered a domain *and* programming can provide an order of magnitude more business value (which is the main metric in enterprise programming) than someone who knows only banking or J2EE.

Richard Hamming says in his speech "You and Your Research" (if you haven't read this you really ought to do it right away!)

"... ``How will computers change science?'' For example, I came up with the observation at that time that nine out of ten experiments were done in the lab and one in ten on the computer. I made a remark to the vice presidents one time, that it would be reversed, i.e. nine out of ten experiments would be done on the computer and one in ten in the lab. They knew I was a crazy mathematician and had no sense of reality. I knew they were wrong and they've been proved wrong while I have been proved right. They built laboratories when they didn't need them. I saw that computers were transforming science because I spent a lot of time asking ``What will be the impact of computers on science and how can I change it?'' I asked myself, ``How is it going to change Bell Labs?'' I remarked one time, in the same address, that more than one-half of the people at Bell Labs will be interacting closely with computing machines before I leave. Well, you all have terminals now. I thought hard about where was my field going, where were the opportunities, and what were the important things to do. Let me go there so there is a chance I can do important things. ..."

If that were true in 1986, when Hamming made his speech, how much more true is it likely to be now?

The mistake most scientists make is to consider programming a "blue collar" activity, not worth focusing on (this might be true for the top 1% or so whose native genius will carry them through, but most scientists I know can use all the tools they can get). I hypothesize that in the 21st century a scientist (or engineer) who can't code is handicapped - not so badly as a poor public speaker or a poor writer would be (and both are very vital skills for a research career), but handicapped nonetheless. Maybe there is a case for making (say) SICP + python + algorithm analysis + usage of basic version control tools a part of the science curriculum.

The mistake most programmers (who loathe their cubicle farms and the brain dead enterprise codebases they maintain) make is to think that research and engineering are somehow beyond their ability to tackle. One crucial contributing factor to this perception is that most people learn mathematics as a bunch of formulae to memorize for an exam than as a powerful modeling tool that penetrates and simplifies complex systems. It doesn't help that, in India at least, the engineering and science educational system is fundamentally broken and emphasizes rote learning and obedience to authority over curiosity and intellectual rigor.

I write my blog to help me clarify my thinking. I couldn't care less if no one reads it. (Having said that, I have "met" some brilliant people through the blog). This and the two preceding blog entries came about because I have been struggling with nailing down a research statement - something beyond the current "I am interested in Robotics and Compilers". One of my mentors asked me to do this. I think this is good advice. More on this in the next (and last in this series) post.

Dinner with HBS grads

2008-01-24T23:00:00.000+05:30

I just finished dinner with a couple of Harvard Business School graduates, generally tossing around ideas about startups and so forth (Thanks Bharath, for making the introductions).

It is a great pleasure to talk to smart people, even when they have different focii from your own (in this case business models vs hard core tech). One of my friends keep pestering me to try for an Ivy League MBA because "you 'd fit in so well there!". I have no plans to do such an MBA but now I can articulate what I miss in Bangalore/India.

One of the strong attractions in moving to California (say) is that I could meet plenty of super smart people without trying very hard. It is possible in Bangalore/India but the effort needed is an order of magnitude higher.

So what's wrong if you aren't an engineer?

2008-01-24T09:27:00.000+05:30

Nothing at all! To quote Reg Brathwaite (again! But the man has a way of using words that is very eloquent. I can't resist).

[This is the second in a series of four blog entries. If you haven't already you might want to read part one first.]

What's wrong with being a clerk? Nothing. It's only a problem if deep in your heart you despise clerks and you spend your life in denial about the career you have chosen. I wouldn't wish that on anyone, so I asked my readers to think about that carefully.

Likewise, we can argue about what activities from programming can or cannot be considered Engineering. But really, even if you don't do any Engineering, what's wrong with that?

Heh!.

My last blog post seems to have ignited a mini firestorm. Observing how people react to an idea is sometimes more fun than the original idea itself. Reg gets it. In one of his replies to a comment (do read his blog entry), he says

"the statement “~p implies ~q” says nothing about whether p implies q." .(typo corrected. Thanks Arne!)

This is a bit more subtle than it seems. Try substituting p = "You use mathematics" and q = "You are an engineer") and try working out "~p => ~q" and " p => q" ( => is implication and ~ is not). You may be surprised !

I am continually astounded at how many people respond to arguments or claims without doing a logical analysis of what's being said. (Note: I am using "argument" and "claim" (and other words like "theory") in the logical/scientific sense, NOT in the " I had this 3 hours argument with my wife. She asked me to wash the car and I refused. At the end I was shouting and she was in tears. My theory is that women are a different species" sense).

When I was a debater, in my younger days, one of the lessons I learned early is to understand that you don't counter an argument from your emotional or "gut" level, by calling your opponent names, or attributing motives to him (unless you are trying to be a politician, when these tactics do pay off).The way to counter an argument is to dissect its logical structure, and show it is invalid (in specific contexts, if required). Rhetoric by itself can be powerful (and many politicians know this), but when layered on top of a logically sound argument is devastating. There are very precise ways of doing this, going back many centuries, at least as far as Aristotle and Plato.

In my last blog post, I made the claim that most software developers are not engineers. Here, I'll make another claim, even more provocative. Most software developers don't understand logic either. You think I am wrong? Quick, (assuming you are a software developer) what is the difference between the "if .. then" construct in programming languages like java and the logical "if ..then" (aka implication, often denoted by =>) ? If you, a software developer, answered correctly without having to think about it, rest assured, you are in a minority.

Using logical implication you can say(assuming we are talking about this Earth and this time stream) "If Napoleon Bonaparte was born in Europe, the Sun rises in the East" and have it evaluate to true. But of course. What's so surprising? "If Napoleon Bonaparte was born in India, the sun rises in the West" or "If Napoleon Bonaparte was born in India, the sun rises in the East" also evaluate to True! :-D. ("If Napoleon Bonaparte was born in Europe, the sun rises in the West" evaluates to False.) What does the birthplace of Napoleon Bonaparte have to do with where the sun rises? ;-)

Confused? heh! Don't worry it is a most people get totally zonked when they see this example for th first time. The key is to realize that implication is not causation. (neither is correlation but that is another topic. See this debunking of a claim that "research supports the effectiveness of TDD" to see an example of correlation vs causation - The trick I pulled is of course that many variants of the English "If.. then" are different from the logical "if ..then". [1]). Many people learn the truth table of implication without really internalizing what it means.

When I teach programmers first order logic, this is a constant stumbling block. The solution is simple. I ask them to think of the logical "if X then Y" (where x and Y are booleans or boolean valued expressions) construct as equivalent to a programmatic "If X then Y else True". The "else True " is key. In other words, (thinking programatically) does X have a value of true? if so return (the truth value of) Y else return true. Apply this to the "Napoleon" arguments and you'll get the correct (logical) answer for all possible combinations.

Why is such a confusing notion important? Proofs are logical structures using the primitives of FOL. A large part of mathematics(and science) is proofs. From science comes engineering. and if you are not using mathematics you are not an engineer (ducks for cover ;-)).

There are 5 connectives (not, and, or, if (or implication) and iff (or double implication) ) and two quantifications (Universal and existential) in First Order Logic, which need to be mastered before one can go on to things like proofs and logical structure. That's the bad news. The good news is that working through a book on logic (and there are PLENTY of those) will teach you how to use logic.

Reg goes on to say (in the same comment stream)

I remind everyone that "exists x such that x ~member of E does not imply that for all x, x ~member of E."

In other words, the fact that not using math means you aren't an Engineer does not imply that using math makes you an Engineer, for whatever definition of math we agree on.

Exactly so!

The point of the last blog post wasn't "you suck you enterprise developer subhuman moron", but "Don't delude yourself". No more, No less. Do what you love. And have fun!

PS: Once you know logic, using to to construct (or deconstruct) an argument is trivial. But for those who want to make sound arguments without necessarily studying "raw" logic (I hope you are not a sw dev ;-)), take a look at "The Craft of Argument" by Joseph Williams and Gregory Colomb.)

[1] Wikipedia

Part Three of this series is here.

Ratcatchers and Engineers

2008-01-23T09:16:00.000+05:30

Ever wondered if what you are doing is (software) engineering? Here is a heuristic.

If you don't use mathematics in your day to day work, you aren't (an engineer). All engineers (say those who build bridges, or space craft, or cars) make heavy use of mathematics and/or hard sciences like Physics on a regular basis.

Now, not being an engineer is ok. Being a carpenter or a plumber is a perfectly honorable choice as is being a musician or actor or teacher. If you enjoy being a carpenter/plumber/automobile mechanic, more power to you. You should do what makes you happy and puts bread on the table. That said, a craftsman is not an engineer. The guy in the garage who fixes your engine is not an automobile engineer who could design the next generation car. Not close.

This insight was triggered by Raganwald's "No Disrespect" blog post. I quote

...Let me tell you the cold, hard, truth. You aren’t going to like this, but I ask you to believe me when I say that I am telling you this for your own good:

There is a culture of pretending business programming is more than it is. Some of you calling for more Java in University may take false hope that I am on your side. You may think that the people arguing for Scheme, Haskell, and OCaml are elitists. Wrong. They do not have a problem. You are the one with a problem because you don’t want to tell all your friends you have a job as a clerk.....

We all know what the typical software "engineer" job ad looks like. A job ad for a real engineer would look like this. (Noe the absence of "10 years in java/dotNet/Ror" type crap. Note that he explicitly asks for a Phd (and tell you under what circumstances he will waive it)

The distinguishing trait of an engineer (and Werner's job description explicitly ask for this) is that he builds and works with mathematical models to design a real world effect or system. They also use other tools(simulations, prototypes, experiments etc) but (mathematical) modeling is key. A scientist,as distinct from an engineer, uses roughly the same tools to advance the state of knowledge without necessarily affecting the real world. The borders are fuzzy. You have scientist-engineers and engineer-scientists, as well as people who focus on "pure" science or engineering.

"Modeling" is a deep topic. Read the book I've referred to at the end of this blog for examples of how this works. Suffice to say If I am not building (say) algorithmic models to help me decide how to build my software, or to generalize, if I am not using "theory" on a day to day basis, I am not *engineering* anything. Modeling has a very precise meaning in Engineering and Science. (No, UML diagrams, or "story cards" are not engineering artifacts no matter what the methodology vendors say ;-) ).

Enterprise software is the least amenable to the modeling/engineering approach. There are exceptions but most "enterprise" developers are the equivalent of clerks, as Raganwald so eloquently points out. There is nothing wrong with being a clerk as long as you know you are one and are not deceiving yourself. Most enterprise software projects, in keeping with their clerical nature, are life draining. But hey if you like it, go for it.

Another friend of mine, who is a very talented programmer (he recently moved away from enterprise app development) , when asked why he changed the focus of his career, told me,(paraphrased) "Humanity has only a very limited amount of talented people. It is a crime against humanity to employ that talent to bug fix enterprise applications or futz around with Ruby On Rails deployment issues. I want to do something meaningful". (As you can see I have interesting friends :-) ).

To conclude, the title "Software Engineer" is (most of the time) a particularly deceptive one. To be accurate it should something like "Software Maintenance Worker" or "Software Handyman", but I guess it is easier to hire someone if his job title is "Rodent Officer" instead of "Ratcatcher". .

PS:- Please, before anyone feels offended and sends me hate mail or snarky comments, please read (something like) "The Idea Factory - Learning to Think at MIT". (This was the book that jolted me out of my complacency and set my feet firmly on the research/engineering path). Then actually think about it :-). Either be happy as a rat catcher or do something about it.

[This is the first in a series of four blog posts. Read part two and part three]

Resolutions for 2008

2008-01-16T23:06:00.000+05:30

Well, the more public (publishable?) ones.

Upgrade tech skills.
Publish two (or more) research papers in peer reviewed journals. This is a tough one but I have mentors who'll guide me through the process, so it should come down to sustaining intense work for a few months. (Added: - This isn't tough because it is particularly hard to do, like running a mile in under four minutes. Quite a few people, and not all of them geniuses by any stretch of the imagination, get published worldwide every year. It is tough because it requires a different style of thinking and habits which are somewhat alien to people not in the academic world)
Solve the "work with good people" dilemma. Consulting is lucrative but lonely and consumes enormous amounts of time. Working for someone else leads to working on insignificant problems, churning through jsp pages or whatever, at least in Bangalore, Bangalore being the place where companies send work no one in the West wants to do.
I've been working by myself for the last three years and would really love to work with a small team.
Given that I'll be in Bangalore for most of 2008, this is a tough nut to crack. Whenever I mention this to someone, they say "Oooh why don't you join Google?". One of my friends has just joined Google (not in Bangalore!!) where he'll work with some of the best Unix developers in the world on an ultra hush hush, technically challenging project. (He is a brilliant programmer and quite deserves the dream job). But, given that Google Bangalore doesn't have anyone of that calibre (or any projects of that calibre), joining the big G (or the Big Y, or the big M) in India isn't an option. I don't want to work on Orkut ! :-).
And there are precious few companies in Bangalore who work on tough problems. As I said, a tough nut to crack, but we'll see.
Learn a classical language, to the point where I can read a classic work of literature in the orginal (I choose Sanskrit for 2008 - I know the basics. Maybe Hebrew or classical Greek for 2009?).
Memorize a complete Shakespeare play (I choose Julius Caesar). If you are wondering why I'd do something like this, read the book "Flow"

Dev Camp Bangalore

2008-01-16T21:49:00.000+05:30

Some folks at my former employer, Thoughtworks, are planning a DevCamp in Bangalore on February 9th, 2008.

This is an excellent idea, well overdue. Barcamps in Bangalore are overrun by people trying to finance the next "social networking" (gag!!) or "web 2.0"(double gag) "startup" by finding some dumb non-technical vc who doesn't know what's going on, photography and movie clubs and so on. And what technical sessions do exist are of the "Introduction to X" variety where X element of { Ruby On Rails, Erlang, Fad-du-jour} mostly cut and pasted from the web.

The Dev Camp web page has these instructions prominently displayed (emphasis mine).

"However, please assume a high level of exposure and knowledge on the part of your audience and tailor your sessions to suit. Avoid 'Hello World' and how-to sessions which can be trivially found on the net. First hand war stories, in-depth analysis of topics and live demos are best."

This should keep the fakes away (touch wood). Also there will be a few participants from Thoughtworks, and if the past is any guide the Thoughtworkers sessions will be well worth listening to.

One of the tenets of a "camp" is that there are no passive participants. Given that this is meant for developers, I hope to attend some high quality sessions. And if I attend, in the spirit of Xcamp,

I'll be presenting too. I will speak on one of

Monads in Depth - what they are, the underlying mathematics, what they are good for and how to use them in your favorite language
Reinforcement Learning - Algorithms and Applications in Robotics
Proof Techniques - a tutorial for Developers
Vajra - a high performance Lisp for Robotics Programming

Of course the topic list is highly fluid and I just might end up speaking on something else, in keeping with the spirit of XCamp, but if you have a preference, send me email!

if any readers of this blog attend, stop by and say hello!

Sliding Into Scala

2008-01-01T10:16:00.000+05:30

I've been dissatisfied with java for a while now, but to give the devil his due, it does hit a sweet spot. When you are looking for a combination of cross platform, fast, statically typed, easily deployable language with tonnes of libraries, there isn't much else available. But on the other hand the language itself is mind numbingly verbose and since most of my coding these days is in a combination of scheme and C, when I do switch back to java, it is as if I am suddenly running through quicksand, eclipse notwithstanding.

Generics (the implementation not the idea) was the first misstep in Java's evolution. It makes it easy to write code that is almost impossible to read. For the AIMA code for example, when I used pure Java 5, generics heavy code, students complained that they couldn't make out what the code was doing and so for code that other people have to maintain or extend, I end up writing Java in a style I call "1.4 +" - Java 1.4 + enums + new for loop + generics *for collections* (only). The actual type system of Java 5 is a fairly simple (even simplistic) one (once you've worked through some books like TAPL), but the syntax is atrocious. The present controversy on "closures" (closure != anonymous function and I am tired of programming illiterates abusing terms with clearly defined meanings but that is a rant for another day) convinced me that Java is on the wrong track and is well on its way to obsolescence. "Java is the new Cobol" indeed.

What COBOL never had was an open source cross platform VM that other languages could target and a few hundred thousand libraries. Java != JVM, in other words. Couldn't I just use another language on the JVM ? Well yes, but - I don't particularly like Ruby. Jython hasn't caught up with the latest version of Python. Both these languages have dynamic type systems and are slower than compiled java. So there isn't too much actual choice given the parameters I listed above (fast, statically typed ... ).

Enter Scala. I looked at Scala a year ago, tried some sample scripts, stumbled across a bug and gave up. But things have changed since then. I've been experimenting with Scala over the last few days and I am tremendously impressed. Besides writing small scripts to explore various language features, I've been rummaging through the code for the compiler and type checker (written in Scala of course). Martin Odersky has something that is all too rare in the software world today - a strong sense of design (I am looking at you, Ruby On Rails).Scala has too many brilliant features to go into any great detail here, (there are plenty of blogs that do go into detail see this or this for e.g) but in essence code compresses to almost nothing, the type system is brilliant, and pattern matching is something I've wanted in a jvm language for a long time. One can even do monadic programming in Scala! I am VERY impressed.

Now don't get me wrong, there are a few rough edges. For example one of the things it doesn't have (as of this moment) is a good reflection API. I tried to write a xunit clone to get my feet wet and got stuck on identifying all the methods that start with "test", for example. (It *can* be done by leveraging the underlying Java but not from native Scala). The build system is a mess (try building from source and watch the build fail for lack of heap space - this is unforgivable in 2008) and there is less of a focus on testing/regression than I'd like. And oh, if you are using the interpreter make sure you use jline ("sbaz jline" should do it- I don't know why this isn't part of the standard distribution).

But these are minor quibbles - Scala is a brilliant accomplishment. I think Sun should just declare Scala to be Java 8 and be done with it.

Managers consume what?

2007-12-28T11:28:00.000+05:30

from this site,

....managers are voracious consumers of theory. In other words, they value ways to think about their world, and mental tools that will let them make decisions and predictions with a level of confidence higher than they get from experience and intuition alone.

Ummmm. Ok. :-P

Teaching Mathematics

2007-12-16T19:55:00.000+05:30

I've just taken up an assignment to mentor one of my friends in maths/proof/logic.

My "student" has no background in maths or logic (if you ignore the negative knowledge gained by sitting through a bachelors degree in Engineering in India's f***ed up educational system), but is a good programmer.

The expected post condition is that my "student" will be able to read and work out mathematical proofs, design an experiment, analyze an existing research paper, work out an algorithm for a given task, understand basic calculus and linear algebra and in general gain the ability to conduct independent research in Machine Vision. I (and some others, like my friend Rajesh, for e,g.) have worked out some of these "how to"s over the last few years, mostly by brute force search of the landscape, discovering pitfalls by falling into them (and making a careful map, which it is now time to pass on).

"How to work out a proof" for example is rarely taught (well) in a classroom setting and I have had to synthesize knowledge from many sources to come up with a workable technique. I can save my "student" hundreds of hours of work.

Teaching is how one crystallizes one's knowledge. I've always enjoyed teaching but what I've done earlier was in a many to one to many format, involving speaking to a group of people in using a blackboard to derive a proof or a projector to display code.

The one to one format will be a new experience. I spent a few hours today working out a "syllabus" and it looks very good, even if I do say so myself :-)

Update : I plan to teach first order logic and proofs first, probably using books by Velleman, Polya, and Doets and Van Ejck.

You can't catch up

2007-12-13T22:14:00.000+05:30

One of my friends recently told me (I have interesting friends :-)) "Every time I meet you, you have moved further ahead in [area of focus], so how do I catch up with you?"

The answer == "You can't, unless (a) I stand still and wait for you to catch up - Why should I? (b) You figure out how to move a few hundred percent faster than me (Good luck with that)"

Imo, the right thing to do with life is not waste cycles "catching up" with other people. I'd rather do something that is uniquely mine. Competing with oneself is hard enough without wasting bandwidth on what other people are doing or how they are "drawing ahead" or whatever.

Robotics and the future of warfare

2007-12-13T14:31:00.000+05:30

I was speaking to one of my friends doing research in robotics in a major American university when it struck me that most of the researchers in robotics and machine learning are gentle pacifists who wouldn't want to hurt a fly, while one of the greatest uses of robots is going to be in real battlefields.

And I am not talking about robots engaging in non lethal activities like scouting or mine clearing, though that's how robots will initially appear on the battlefield. I am talking of robots (on land ,sea and air) armed with guns and other weaponry and increasingly deciding for themselves when to fire and what to fire at.

I believe that in wars of the future two major features will be asymmetrical war (what we call "insurgency" -blurring the boundaries between "civilians" and "soldiers") and increased participation by increasingly sophisticated autonomous robots/remote control weapons. (see this column for some thinking along these lines). I'll ignore the first aspect and focus on the second, in the interests of keeping the size of this post manageable.

Here is how robotics will develop in war. First, robots will engage in non lethal activities like mine clearing or I.E.D detection. (This is happening today) Then you'll see them accompany human combat units as augmenters and enablers on real battle fields. (This is beginning to happen) As robotics gets more and more sophisticated, they will take up potentially lethal but non combat operations like patrolling camp perimeters or no fly areas, and open fire only when "provoked" (This is beginning to happen too). The final stage will be when robotic weapons are an integral part of the battlefield, just like "normal", human controlled machines are today and make autonomous or near autonomous combat decisions. These robots will look nothing like the vaguely humanoid robots of Star Wars - they will have shapes suited to their roles - think tiny dragon fly / cockroach like systems to computerized tanks and helicopters.

And this type of robotic battle system will not be confined to the major superpowers. As of today, what gives robots an edge is not really dependent on hardware, but on sophisticated software, something that does not depend on superpower status. Someone in Korea or India or China, or even a semi military corporation can gain a temporary edge in robotics software comparatively easily (though the Red Queen Effect will kick in soon enough to close the gaps) sufficient to tip a battle or a war one way or another.

The funny thing is that quite a bit of the required technology is available, sometimes in embryonic form, today. "The future is here, just not widely distributed" - as William Gibson said. I know that the Indian defense forces have made substantial progress in battle field robotics systems (I can't be more specific - the details are classified so don't ask! :-D ), and what India is on the verge of doing, others are, too.

None of this is to claim that future combat will occur between machines, all clinical and detached with no blood being spilled. On the contrary.

Historically, every innovation in technology has been hailed as bringing in an era of bloodless warfare. When the airplane was invented, for example, many columns were written on how they made war obsolete because, obviously, with planes roaming the sky no army could make unobserved movements and so war was now futile! Dresden and the Blitz and Hiroshima were just around the corner.

My prediction is that war will become more lethal, with increasingly sophisticated autonomous weapons systems and ever more vicious asymmetrical warfare twisting the business of making "the other fellow die for his country" (or ideology or religion or ...) into newer and ever more horrific forms. And that is something that all of us working on robotics or machine learning should think about.

Update: First Armed Robots deployed in Iraq . While these are not really autonomous systems (they are remote controlled) and substantial technical challenges remain on the path to truly autonomous systems, people conducting research in robotics know how fast these barriers are falling. Those who are not researchers in robotics/ai may mull over the progress in autonomous navigation of vehicles from 2004 to 2007 as exemplified by the DARPA Grand Challenge. Now extend that rate of progress to areas other than navigation.

Books for 2008

2007-11-26T16:05:00.000+05:30

These are the books I'll be working through in 2008.

Meet "Spineless"

2007-11-16T23:39:00.000+05:30

Spineless is a small Django app my friend Manoj whipped up to help me keep track of my books. People generally borrow books from me and I have a hard time remembering who took what when. Even after giving away almost all my 300 or so java/j2ee/dotNet/"agile"/"enterprise" books, I still have a few hundred books lying around in Bangalore and more in Trivandrum. I have entered the first 50 or so books into the app (actually I type some keywords and the app looks up the books in Amazon and grabs the details and plonks them into a database). There has been some demand for something like this, so all who asked, enjoy! I'll keep adding books whenever I find the time (which is a commodity in very short supply right now :-( ).

One of my friends asked me to enter my book list into Library Thing, but I have a visceral dislike of "social networks". Maybe too strong a mental association with Ruby on Rails, I don't know exactly why.

And before you ask, the name doesn't really refer to books' spines or anything to do with books really. It has to do with a private joke about something someone we know said recently. Manoj liked it so much that he named the app after it. Mention "optimization" to Abey or me and we'll roll around laughing. Mention "Spineless" to me or Manoj and you'll get the same reaction. Commemorating stupid things said by intelligent people (or vice versa) is a bit of a tradition in my circle of friends. What's life without people to make you laugh ?

Fish Eye Plans

2007-11-15T13:29:00.000+05:30

One of the emergent effects of my time management meta focus is that I have I call a "fish eye" plan.

I know in great detail what I will be working on in the immediate future (say the next few days). Next week, I know, but in less detail.I know what I'll be working on next month, but at a higher level of abstraction. Next quarter, even more abstract, but I *do* know what I'll be doing. So if you ask me what I'll be doing in (say) March 2008, I'll have an answer. Not many people can answer that question. This is a very refreshing change.

This is not to say that plans won't change. They will, but you know what they say about failing to plan.

What pleases me the most is that I didn't set out to do this. I just followed Dr Randy Pausch's "system" of time management (with many tweaks and some significant additions from elsewhere) and it all just fell out.

Life is good.

Question: Is anyone in Bangalore (anywhere in India will do at a pinch) doing significant programming in Haskell? Mail me please. Thanks in advance.

I won't go seal hunting with you

2007-11-14T12:39:00.000+05:30

I told someone a few minutes ago "I think you are very capable and like you a lot but I don't trust you one inch". (Yeah I do say things like that in real life :-D).

After the (very unproductive) conversation was over, I was trying to recollect where that phrase/concept came from. It was certainly not an original thought/phrase and I was sure I read it somewhere. Turns out it was from Edward de Bono who said (paraphrased) "some languages have one word which says 'I like you very much, but I would not go seal-hunting with you'." so one can say (in that language) "I [word] you" where [word] has the meaning of the phrase above.

I forget the book he used the phrase in but iirc, the context was that sometimes if we lack the precise word for something, we need to repeat a long phrase.

This holds in technical conversation too. We don't say "change the structure of the code without changing the behaviour", we say "refactor". In the absence of the word "refactor" we would probably use the longer phrase, till someone got tired of it and invented the word.

Back to seal hunting. In this case, the context was one of doing a startup together and I suggested the phrase as an explanation why a few people don't "openly talk" to him, while the other people do talk to me "openly". Note that this doesn't mean that person X *is* untrustworthy. All it means is that I (and a few other people) *think* that in a crunch, X wouldn't stand by us and would do what's best for him vs what's best for us.

Is not trusting someone a problem? In a normal context, not at all. We all work with people whose competence we admire but we wouldn't want to trust them to keep our objectives and safety in mind when the stakes are high. We just add a double dose of caution, keep our guards up and are careful with words around that person, read the fine print very carefully and so on.

I am not very sure that would work in the context of a startup though. In a startup (from what I've read, not that I've ever founded a startup, though I've been employed in a couple) the pressure is so intense that one doesn't have the time to do all this armor donning and being careful. Founding startups is not like seal hunting -it is like hunting Great White sharks or the aliens from the movie "Predator".The founders absolutely *must* trust each other to do the right thing when the bullets (or harpoons or laser beams) are raining down.

Another thing that I noticed from the conversation was the use of phrases which don't really mean anything but are invitations to games (in an Eric Bernian, Transactional Analysis sense - the person proferring the invitation doesn't often know what he is doing, from a group dynamics perspective).

For e.g the conversation started with "I don't want to jeopardize our friendship but .. " and I was thinking "What friendship? we aren't friends - we were never friends- We are professional acquaintances who speak about technology occasionally and have met at a couple of parties".

To me a friend is someone who you have let into the "inner courtyards" of your life. You can have many positive, enriching relationships which are of a lesser intensity than a friendship, but the word does have a defined meaning and it didn't make sense in this context.

That opening phrase puzzled me for a while, but then I realized that he didn't really mean it either. It is just a phrase the meaning of which depends very much on the context of its usage, in this particular instance meaning something like "I am very uneasy about the fact that people don't relate to me like they relate to you. I am going to vent my feelings a bit but I don't know how you'll react to that, so I'll just use this phrase and hope you believe " maintain friendship" is the reason I am screaming".

Real friends don't need to use those kind of weasel phrases and would say something like "I don't know wtf you are doing, but I am concerned" or something to that effect. I try to avoid these meaningless phrases as much as possible, not because it is a particularly noble thing to do, but because I don't want to clutter a conversation with superfluities. "Say what you mean, mean what you say" is a surprisingly effective, not to mention efficient means of communication.

If you keep an open mind you learn something from the most trivial incident. I learned a few things today. Not too bad for a 20 minute conversation. Not bad at all.

A list of topics

2007-11-13T19:46:00.000+05:30

My focus on Time Management has affected how I write on this blog. Earlier, when I felt strongly about something or when I was feeling bored, I'd just dash something off. This wreaks havoc with schedules and "getting things done",and I have often stopped writing for a few weeks, months whatever. But this is dissatisfying in its own way.

Now, when I think of something that I feel like writing about, I just add it to a list (on a wiki, if I am at a computer, in a small notebook if I am not) and carry on with what I am doing. When a "write a blog post" size chunk of free time comes around, 15-20 minutes, typically) and I am working on the comp, I just pick an item off the list and write about it.

The style of writing is different than it used to me because I am writing against a timer. More direct, less textured, which is probably the appropriate style for a blog post. The longer wait between conception and recording also allows the good ideas to percolate and the bad ones to show their unworthiness.

For anyone who is curious, my present topic list (this is a continuously mutating entity) looks like this.

San Fransisco vs Bangalore - the right place for software startups?
The Idea Bottleneck and how it really isn't [DONE]
Stealing a leaf from Paul Graham.
SICP Stumbling Blocks
A Billion Dollars And Then?
Topic List on a Wiki [DONE]
Practical SICP - How SICP helps in real life projects
Derivative Trading as Programming Pedagogy
Growing into a (technical) book
Memorizing Shakespeare (and Shaw)

This is just a record of things I might write about. But the time management meta focus seems to be paying off already. I am writing more frequently than usual, and not feeling stressed at all.

Someone stole my cell phone

2007-11-12T22:11:00.000+05:30

so if you are reading this, and are on my phone list, please send me your phone numbers to repopulate the address book on my new cell phone.

Thanks in advance.

Update - I got a new sim card but decided not to get a new phone till I complete my research paper.Meanwhile, please use email.

The Idea Bottleneck That Isn't

2007-11-12T18:41:00.000+05:30

One of my friends was talking about doing a startup in Bangalore. This is a conversation that comes up regularly. I tried to get a startup off the ground twice, but failed because the people who I wanted to work with didn't live in Bangalore and I didn't want to move.

Those attempts isn't what this post is about. Because this friend was talking about a startup, and wanted some people to work with him , I spent some time talking to other friends who might be interested (I am well connected to interesting, technically capable people who are hungry for success and money).

One question I keep hearing is "What about ideas? Do you have an idea?". I have never understood this question. Generating decent ideas has never been a problem for me. I could generate 20 workable ideas in 20 minutes. And repeat that exercise a few times before I need to pause.

The first time I encountered this "idea bottleneck" concept I didn't pay it much heed. But then someone I respect said I was not paying enough attention to "finding an idea" if we were to attempt that hypothetical startup. So I reached into my mental pile of ideas and gave him an idea to work with, while we work through other issues. He seems happy about it and sent me a link showing that a company with a similar idea (actually a trivial subset) had been acquired for 50 million $ cash.

Another friend heard of this and said "Dude, how can you give up your idea like that? What if he rips you off? Be careful"? First, the friend I told the idea to is a decent chap. Second, I don't really care if he (or anyone else for that matter) "rips me off".

My operating concept is that having ideas per se isn't all that important. It is the actual execution that matters and so I am not afraid of people "stealing" my ideas. It isn't as easy to do that as many people think. The idea I suggested to my friend, for example, sounds simple but there are all sorts of interlocking little problems that all have to be resolved perfectly for it to be successful. I can imagine someone spending a few months (and a few hundred thousand dollars) in an attempt to get it to work and not getting anywhere, because he isn't aware of what to tweak to get a particular effect or how to fix a subtle "error". In fact if anyone can execute the idea better, faster or more comprehensively than I can he is welcome to it.

Paul Graham says it better than I can (in news.ycombinator).

.... if we told everyone about everyone else's ideas, it wouldn't be as bad as you might think. Secrecy is not as important as beginning founders think, because (a) ideas are less valuable than they think, and (b) the most common form of death for startups is suicide, not being killed by competitors."

The last sentence says it all.

Update: One hour after this was written, the proposed startup project committed suicide :-D.

R.I.P :-D

Experiments with Time Management

2007-11-09T11:50:00.000+05:30

After viewing Randy Pausch's lecture on Time Management, I've been experimenting with various time management practices. (yeah I know how management-ey and buzzword-ey that sounds, sorry). It helps that I am going through the busiest month in my entire life, so I am more motivated than usual to make this work.

The most successful habit I've adopted is to put a money value on my time. As Randy says in the first few minutes of his talk, people are generally willing to give you their time, but not their money. If you ask people to spend a quarter of an hour to help you arrange the furniture for a talk, they will happily do so, but they won't hand over 10 $ for no obvious reason.

Once you start putting a dollar value on your time, though, strange things happen. For example, I decided not to attend Barcamp. (I'd registered, I cancelled it). Say an hour of my time is worth say a measly 25 $ (being a cheap outsourced-to Indian programmer). Travelling to and from Barcamp, and attending 3 or 4 sessions would take, at a minimum 6 hours. 6 * 25 = 150 $.

Would I spend 150 $ to attend Barcamp? Not on your life! To be honest, there are other disincentives also - I really don't want to attend a meet which is so "implementation lite" - where there is so large a focus on things like search engine optimization (!) and blogging.

People actually think blogging is a big enough deal to form "collectives" about blogging? - there is even a "collective" for "radical ideas" -- yeah right, that makes sense .. NOT. The mind boggles. I consider blogging the intellectual equivalent of writing a post-it note. I cannot imagine self-identifying as a "blogger" and making a big deal out of it. God have Mercy.

Back to Time Management. Since I don't (in general) attend phone calls from unknown numbers/people when I am working, I don't have interruption issues. And once I concentrate on something, it is hard even for people (even if they are sitting right next to me) to jerk me out of the zone - I don't hear anything when I focus on work so they'd have to nudge me, and risk getting a laptop thrown at their heads) so I didn't benefit so much from that part of his talk. But the money value of time is an excellent perception to have. Would I spend 50 $ to have a good conversation with a friend/someone doing interesting things? definitely.

He is also right about keeping a time journal. The result of such an experiment is the most horrifying document you will ever see in your life. The good news is that with such an abysmal base, you can make substantial improvements in a very short order.

To round things off, a couple of snippets from a Phil Greenspun article.

Ask a wage slave what he'd like to accomplish. Chances are the response will be something like "I'd start every day at the gym and work out for two hours until I was as buff as Brad Pitt. Then I'd practice the piano for three hours. I'd become fluent in Mandarin so that I could be prepared to understand the largest transformation of our time. I'd really learn how to handle a polo pony. I'd learn to fly a helicopter. I'd finish the screenplay that I've been writing and direct a production of it in HDTV."

Why hasn't he accomplished all of those things? "Because I'm chained to this desk 50 hours per week at this horrible [insurance|programming|government|administrative|whatever] job.

So he has no doubt that he would get all these things done if he didn't have to work? "Absolutely none. If I didn't have the job, I would be out there living the dream."

Suppose that the guy cashes in his investments and does retire. What do we find? He is waking up at 9:30 am, surfing the Web, sorting out the cable TV bill, watching DVDs, talking about going to the gym, eating Doritos, and maybe accomplishing one of his stated goals.

Retirement forces you to stop thinking that it is your job that holds you back. For most people the depressing truth is that they aren't that organized, disciplined, or motivated.

This is so true it is not funny. And one doesn't have to retire to encounter this phenomenon.I know many people who drop out of the 9-5 rat race for various reasons ( startups, research , search for the meaning of life .. whatever), and then find that months and years pass with nothing substantial being achieved.

Another interesting suggestion from the same article.

publish a public Web diary of what you do every day, thus discouraging you from wasting time because you'll be ashamed to admit that all you accomplished yesterday was a 15-minute oil change and a trip to Target

That is a very interesting idea, but a little forbidding, exposing one's lack of productivity to the world. But I should probably grit my teeth and try it for a month or so. If you want to see what this would look like for someone who is really really good at what he does (vs humbler mortals like you and me) see John Carmack's dot plan files (warning - Flash, sorry I couldn't find an html document of these and I am running out of my "write a blog entry" time slot). There's even a theory that these (Carmacks' plan files) were the origins of the present day blogs, no matter what Dave Winer says.

While I don't quite have the guts to "dot plan" my puny daily efforts, I could probably list down what I plan to do each month on the first day of the month and a report on the last day of the month detailing what actually got done.

So, as an experiment, here is what I need to do this month (November 2007) (client work excluded - privacy concerns blah).

Research - Solve the last 2 issues, code, write up research results into a paper and send out for review. Hard deadline of 24th Nov
Write a Reinforcement Learning library for the friendly folks at at the DRDO. This is a a lot of work
Implement the algorithms in chapters 9-12 of AIMA. (First Order Logic and planning, for anyone working through the book). Release the next version of the AIMA Java code.

Even with the newly adopted time management practices, that is a tonne of work which brings me to -

Blogging Time Over. Back to Work.

Hackers Month of Learning Haskell aka X Y Month

2007-11-06T14:22:00.000+05:30

The local newspapers are all going crazy about National Novel Writing Month. Apparently NaNoWriMo is some kind of cultural phenomenon in which random people commit to writing 50,000 words in a month, hopefully finishing a (substantial part of a ) novel.

I am not sure any worthwhile novels actually get written this way. If you won't do it anway, you probably won't do it just because other people are also doing it but the idea has its possibilities. The "national" doesn't make sense in a networked world, so we'll replace it with the variable X.

Instead of novel writing we could do something more fun so we'll call it Y.

Let X = { .bunch of adjectives...} Let Y = { ..bunch of verbs.. }.

so we could have "Personal Month Of Derivative Trading" or whatever.

From the nanorimo website,

Valuing enthusiasm and perseverance over painstaking craft, NaNoWriMo is a novel-writing program for everyone who has thought fleetingly about writing a novel but has been scared away by the time and effort involved.

I can certainly think of analogues in programming. many people want to learn Haskell, (say) but never get around to doing it because at any given point of time, there are more important things to do. maybe someone should declare December to be "Hackers Learn Haskell" month and ask anyone participating to write, say, 500 functions in Haskell in a month. Or work through SICP completely. Or whatever.

from the nonwrimo site, Because of the limited writing window, the ONLY thing that matters in NaNoWriMo is output. It's all about quantity, not quality. The kamikaze approach forces you to lower your expectations, take risks, and write on the fly.

Make no mistake: You will be writing a lot of crap. And that's a good thing. By forcing yourself to write so intensely, you are giving yourself permission to make mistakes. To forgo the endless tweaking and editing and just create. To build without tearing down..

Programmers write a lot of crap code anyway so at least that part won't be new.

The Algorithm Quotient of a Software Project

2007-11-04T21:11:00.000+05:30

Still thinking of the DARPA Car Races, it strikes me that a project's dependence on algorithmic innovation for a successful outcome is in direct proportion to its "interestingness" froma developers point of view.

Think about it. All the cars that autonomously traverse 160 miles of desert or travel through an urban environment while obeying traffic rules have essentially the same hardware. The difference in performance is about how algorithmically sophisticated the *software* of each car is.

On the other end of the spectrum most "enterprise" projects are algorithmically trivial (I said "most" so if anyone's ego out there is tied to his knowledge of JSP or whatever, spare me the righteous indignation) and the most complex data structure used is often a hashtable.

I don't think this is always a valid heuristic, but it might be valuable to rank projects on the scale of the algorithmic complexity involved in each (which in turn probably depends on how close the project comes to exploring the unknown, as the DARPA race efforts do), and then by how compelling they are and see if there is a correlation.

In my experience, while enterprise projects have their own set of problems to be solved, they are rarely algorithmic in nature (vs economic or scheduling issues for e.g).

Another supporting factor might be to see if interviews for the interesting dev jobs focus heavily on testing algorithmic knowledge. At first glance this seems to hold as well. If you are interviewing for Google Research, you probably need to be great at algorithms. If you are interviewing for Wipro, not so much - a knowledge of the contemporary buzzwords (agile, xp, tdd, ruby dsl blah) should get you through.

Watching the DARPA Car Race

2007-11-04T02:40:00.000+05:30

It is almost 3 in the morning here in Bangalore as I type this. I've been watching the webcast of the DARPA Car Race. This years challenge was for the autonomous ( == no human control) robot cars to accomplish three sub-missions in an urban environment, following all traffic laws (obeying traffic signs, recognizing and conceding right of way etc). This task is a magnitude tougher than finishing a 130 mile autonomous drive in the desert, with zero human control, which was the goal of the DARPA 2005 car race.

Stanford's entry, Junior, just pulled in to the finishing area. While it isn't yet clear who won (because this is not a first past the post race), I'd put my money on Junior. (His (its? :-) ) predecessor Stanley won the 2005 Grand Challenge. This is history being made before our eyes.

The sight of the cars driving themselves, with the steering visible through the windows turning with no human hand at the controls is .. exhilarating... exalting... too cool for words. This is **REAL** programming in action. The people who made this happen are just incredible. Congratulations to all!

How lucky does one have to be to live in these times?

Now to drag myself off to bed (and dream of autonomous vehicles).

Is that office work?

2007-11-03T14:14:00.000+05:30

So, it is late at night and two of my friends are talking to each other. One is a programmer, a very good one, and the other is a manager, I have no idea how good a manager he is.

The programmer is multi tasking, carrying on a conversation in one mental thread and working on some code in another. As the conversation ebbs and flows, his attention also focuses more or less on the code as appropriate, sometimes muttering to himself, sometimes whooping out loud, sometimes ignoring the code and focussing on the conversation.

The manager gets more and more curious as to what exactly the programmer is doing and finally blurts out "Is that office work ? something you have to deliver tomorrow?" . The programmer looks confused and says " No, no it is just something that I am fiddling with". Now the manager looks confused.

And in that moment, *I* understood something. Many people think of programming as work. It sounds very normal and rational when you write it out like that, but most good programmers I know don't think of programming as something you "do at work". Programming is a means of expressing oneself, which also *happens to be* something people pay for. And this is lost on some people who can't imagine "working on" something outside the office..

Just to be clear this has nothing to do with managers vs developers. I can imagine a manager poring over company balance sheets or whatever to find a good investment opportunity and some random programmer who doesn't write a line of code outside office hours asking the same question and getting the same answer.

The difference is something more fundamental. People either have something they work on in their free time or they don't. Now that I think of it, a staggering percentage of the people I know would just switch on the TV, or pick up a book, or idly surf the web or pick up the phone and call someone when they have a couple of hours of free time. The person who picks up a guitar, or writes a program, or analyzes a balance sheet or writes an essay is much rarer than the first variety. Many people don't work *on* anything outside their office hours (and then wonder why "life is boring".

I am sure many people say "yeah so what?" but as someone who has many irons in the fire and is always "working on" something or the other, it was a big revelation to me that most people **don't** have more things to do in life than time to do them in.

One lives. One learns.

As usual, once I finish writing a blog entry I find someone expressing the idea better than I can.

Excerpts from Dan Russell's essay on marathon running (you should read the whole thing).

I don't know how many people have told me that I'm crazed. But the reaction I get when I tell someone I'm running in the Boston marathon is either a "that's wonderful -- how exciting!" or a "you must be out of your mind..."

Usually, it's the latter. And if the person I'm talking with is not a runner or bicyclist or someone who enjoys the pleasures of a workout, then the conversation usually turns toward the health hazards of rigorous exercise. I've been told to watch for vitamin depletion, unexpected calcifications, loss of sensation in the private bits, shin splints, and even unexplained loss of sleep. It's touching, really, especially the ones that are worried about vague causal factors like "getting the body too riled up..."

....So it's hard to explain sometimes why you'd even want to run so far. ..Don't mistake me, running the marathon will be tremendously exciting; there's nothing quite like the sensation of running into the chutes at the finish line, all of the cheers, hearing your name and time called out over the PA -- it's a little like being born and first love and finally getting permission to stop all at once. It's relief, desire and fulfillment borne over the past several months all condensed sweetly into that final few minutes of running.

But as someone one said about practicing the viola, "the epiphanies come in practice." Don't mistake me, running the marathon will be tremendously exciting; ......So I've been running six days a week for the past year just to make it to Boston. Or have I? Seems to me that running is about, well, not to put too fine a Zenish point on it, running. Like anything else, once you've put enough hours of practice into a skill, the doing of the skill transcends the normal, pedestrian everyday experience. Once you get past the first thousand hours or so, the act of running becomes larger than the little aches and pains of physical motion. When I run, I'm moving over terrain; I glide through topography; my body becomes the vehicle that transports me through space. This is an odd, almost out-of-body feeling that starts to happen when you run long miles for training. More than a few times I've run and had the feeling that it would be fun to explore down this road; so I turn, and run down this road. Natural enough, but when you're running 18 or 20 miles, these side roads and random ventures can be two or three miles long each. That kind of distance really begins to change the nature of the experience. You cross towns on a whim and run over sizeable chunks of real estate just to see what's on the other side. On more than one occasion I've run north around a mountain or decided to shift to another watershed during a run. It's the change in scale that's so quietly dramatic. You stop worrying about distance and begin thinking about time -- "if I go this way, will it take another hour?"...

Exactly! And that is what my programmer friend was doing. Exploring. What happens if I try this? And then that? The pleasure from the act of exploring is something that my manager friend, who has nothing comparable in his experience to validate the process against, doesn't quite understand. (Note: This doesn't make him a bad person). So he falls back to "Is that office work"? or in other words, "where are you running to? Can't you get a taxi ?"

To "learn AI"

2007-11-03T12:23:00.000+05:30

first learn basic mathematics.

I keep getting asked this question and I keep saying the same thing - to three people in the last week, for e.g, two of whom were working through (or planning to work through) AIMA - so I thought I'd put this down here (and point anyone who asks the same question to this entry in the future).

Learning AI (or any deep comp.sci for that matter) is not like learning J2EE or ruby "dsl"s or whatever the fad du jour in the enterprise software world is - read a few chapters of the latest bestselling "pragmatic" book, write some crappy web site and hey presto you are the expert.

"Real" comp sci doesn't quite work like that.

To understand a standard 3 layer feed forward neural network, for example, you need to have a solid grip on

vector spaces
basis vectors and change of bases
eigen vectors and eigen values
Basic matrix operations like inversion
multi dimensional performance surfaces
Quadratic functions and finding their global maxima within Newton's method and Conjugate gradients
performance measures and steepest descent
partial differentiation of vector values
numerical stability of algorithms

Without that background you will be able to use conceive of "class NeuralNetwork" that "has an" instance of "class Node" etc, but you will *not*,(repeat NOT) be able to do anything useful with real world data (writing a XOR classifier doesn't count!).

And, a feed forward neural network is only one type of pattern recognizer (or function approximator). There are many more, each with its own trade offs , and you have to know the math to make the trade offs.

That being said, the best book to start with is AIMA as long as you (1) learn the required math parallely (2) do all the exercises at the end of every chapter.(I can't emphasize this enough). Reading through != "working through".

If one is willing to work hard, there are very few fields as fascinating as the various branches of AI.

Once you start down the rabbit hole, however it may be very hard to continue writing all that heavy lifting enterprise software without hearing the giant sucking sound of your life going down the drain as you write yet another jsp page for the latest leasing system. ;-)

Be warned! :-D.

Anyway the really hard part of "learning AI" is not getting the books or working through them systematically (which is hard enough), but that is for another blog entry.

Post Script - Some book recommendations for beginners.

Linear Algebra by Gilbert Strang
Calculus by Gilbert Strang
Artifcial Intelligence A Modern Approach By Russell and Norvig
Machine Learning By Tom Mitchell
Neural Networks for Pattern Recognition by Chris Bishop
Paradigms of Artificial Intelligence Programming by Peter Norvig (ok, this one is not quite an cutting edge AI book, but imo this is the best book on programming ever written and if you call yourself a programmer you should have it on your book shelf)

Intoxicating Conversation

2007-10-28T22:11:00.000+05:30

I just got back from our end of the month "Losers' Club" meetings (more on these here). In the absence of the one person in our group who maintains an interest in enterprise software, we focussed on the more "blue sky" aspects of software/comp sci.

Topics that came up for discussion - Haptic sensors for robots, micro robots, challenges in building a simulation of the human cardio vascular system, metrics for each component of a research effort, Randy Pausch and his lecture on Time Management, the Bhagavad Gita, Reinforcement Learning as a methodology to learn control algorithms, Computational Fluid Dynamics and Aerodynamic modelling, building SkyNet, using histograms to model discrete probabilistic distributions, Algorithms for probability Density Tree construction, how to keep tight focus on research projects that spin off more areas for exploration with every step taken aka how to balance exploitation and exploration in the context of Rogue Research, proof techniques as weapons in a personal arsenal, abstract problem spaces, Lush as a lisp for day to day work, overcoming the fear of Calculus, Django (yay!) vs Rails (bleh!), the non importance of learning java script, Prototype vs jquery, Goal Driven vs Capability Driven learning styles, continous phenomena in genotype decoding, a proposal for a facebook style app for researchers, Aikido, conferences in Goa, and of course status reports, "iteration reviews" and plans for the next month.

Aaah, Life is Good!

A Few Good Men

2007-10-19T16:44:00.000+05:30

One of the oddities of India's "silicon valley" is that there is very little hapening that is technically interesting.

For example, take interest groups. User groups for various topics of technical interest certainly do exist - for some value of "exist". They are unfortunately not very effective. Some common patterns I've seen are

FizzBust - There is a lot of enthusiasm for setting up a group, drawing up an agenda and holding an initial meeting. After that first (or occasionally second) meeting, nothing much happens and the group dies away.

"Eternal Beginners Syndrome" - Every single user group meeting has talks of the variety "Introduction to X" where X is the topic of interest. This is certainly fine *if* people ever move on to intermediate or advanced topics. This rarely happens. What's worse, every one of these "introductions" are (a)focussed on power point presentations and (b) every slide in those horrendous presentations are cut n pasted from the same old books. Encouraging powerpoint for non visual data is the infallible identifying sign of a group that has passed its peak and is now declining in utility and relevance.

A closely related pattern is "Propaganda over Exploration" - the group takes on the attributes of a cult with some members claiming to have "superior wisdom", the dissemination of which becomes the primary aim of the group. Thus rather than have a a collegial group exploring and learning and where appropriate criticizing the focus of the group, one has "masters" and "followers". This quickly drives away any good folks.

"Philosophy Over Code" - This is where people wax lyrical over the virtues of GPL vs the BSD license but haven't written a single piece of open source code in their lives and don't intend to.

"Uncritical Acceptance" - Once in a X user group I suggested that one possible topic for an upcoming "X Conference" would be "Where is the use of X inappropriate". I was quickly shushed with "That is a valid question but asking it in public would "hurt the movement" ". My reaction was "Duh? "movement"? what movement?" but I didn't press further.

The way to run an effective (vs "popular") user group would seem to actively invert all these phenomena. Have a small bunch of experts meet regularly to discuss things of interest. Don't actively try to expand the group or conduct "outreach". Avoid all formal organization and "outreach programs". If someone wanders in after hearing some word of mouth description of what goes on that's fine, but don't actively aim at recruiting or expanding the group. Have everyone bring code (or proofs or papers or whatever the metric of actual progress in the field is) and actively discourage empty philosophizing.

I've been experimenting with some of the above ideas for the last few months and they see to work well. One kind of "community" I'd like to see is of amateur researchers in hard comp sci - amateur in the sense of "pursuing research strictly out of interest and not to get paid" , not "amateur" as in "lacking skill". Assembling a group like this has its own difficulties however. With a few honorable exceptions most members of (Indian) "research groups" of technical companies could be stood up against the nearest wall and shot with no appreciable impact on the output of those groups. Likewise for most researchers in (Indian) government service or the (Indian)educational institutions. But "most" != "all" and that is what keeps me from abandoning the idea.

Snapshots from my Robot Simulator

2007-10-06T17:06:00.000+05:30

I've built a tiny robot simulator for my ongoing research. Like the (absolutely brilliant) Stage Simulator, my simulator enables the simulation of robots in a 2D world. Of course my simulator is much simpler than Stage, primarily because it doesn't try to do as much as Stage does. On the other hand it has a couple of important differences that are important to my research. For e.g. it has a built in sensor and motion error model and (being implemented) a sophisticated Sonar Model.

Anyway, here are some early snapshots of a simple environment.

A simple world for a robot to navigate - The world is 10m X 20m with a resolution of 2 cm. The corridor is 2.5 m wide.

A robot (the little blue dot)stands in the middle of a corridor.

The world overlaid with a quadtree that overlays and recursively decomposes the 2D world and enables efficient ray projection.

The world illuminated with a laser. The illuminated points are rendered in yellow. The scan has a resolution of 1 degree,a "pan" of 360 degrees and a range of 3 m. I was a bit concerned that this would be slow but it takes no appreciable time, even when the angular resolution is 0.1 degree. All hail the quadtree!

Of course the robot sees only the illuminated points, not the complete length of the walls etc.

This is what the robot "sees". There are other sensors simulated (e.g. an odometer) but they are not rendered here. You'll notice that the laser sensor senses the environment "perfectly", i.e the range detected is the actual range to an obstacle that intercepts the laser beam. No real sensor works like that. To simulate this imperfection in a simulator, a probabilistic error model is added to filter the reading so the robot sees an approximation to the real range.

Here we've added a bivariate gaussian error (with a standard deviation of 10 cm) to the laser sensor. If you squint, you can see that the readings now only approximate the wall surface. To see this more clearly, we'll remove the unseen parts of the corridor walls and the overlaid quadtree to get a picture of what the robot sees, with the simple error model in place.

This is what the robot "sees" through the error filter. Note the "scatter" of the readings. Now the robot doesn't know for certain where the walls (or whatever objects obstruct the laser - different parts of the world react differently to different sensors - some surfaces absorb light for e.g returning no readings to the laser sensor) are.

The robot simulator is not the central thrust of my research effort, but it helps to have an environment in which one can plug in various sensor models, motion models and error models and see what the robot would see. I thoroughly enjoyed coding this up.

The next step is to replicate a realistic sonar (vs laser) sensor. There are a few interesting issues to be resolved to get this to work properly. Sonar works differently from lasers - cones of ultrasound have different propagation and reflection characteristics from rays of light. Stage simulates sonar with the same "ray trace" approach as is used for lasers. For my research I need a more accurate model (and hence the need for a custom simulator). Early next week, I will be visiting the Robotics Lab of the DRDO (The Defense Research and Development Organization - For non Indians reading this, they are the local equivalent of DARPA) to talk to some scientists working on similar problems. I have a basic sonar model working already, the key challenge of which was to layer different continuous probability distributions and sample data from them in reasonably efficient fashion, but to describe that would make an already long post longer, so I'll stop now.

The Symbolism of the Bended Knee

2007-09-29T15:12:00.000+05:30

After a friend informed me that her fiance had proposed marriage on bended knee, I decided to look up the symbolism.

here, I found

"..... There is no clear historical origin of the idea of proposing with a bent knee, but the gesture bears striking resemblance to many other ceremonial situations, including:

Kneeling during prayers and other religious ceremonies, including wedding vows for some faiths.

Kneeling while proposing, then, has the same solemn and spiritual connotation.

Knights kneeling while being awarded honors from kings and queens.

Offering or accepting a marriage proposal is just as much an honor as any medal or award.

Bowing in supplication before a victorious enemy, typically as a gesture of surrender.

Committing to a relationship is surrendering oneself to being part of a couple.

Kneeling to genuflect when entering a church or temple.

Again, proposing on bended knee is a sign of respect and spiritualism.

..."

On applying some (no doubt unromantic) analysis, the first two "interpretations" seem distasteful. In both cases there is the distinct notion of kneeling to someone you have acknowledged as a superior power.People don't kneel to equals. When was the last time you knelt to your peers (or your manager for that matter)?.

If the fourth reason were valid, both parties should kneel. (Going off on a tangent I wonder.. if the roles were reversed, would feminists see this as a "patriarchal custom to oppress women"? They do seem to take umbrage at changing surnames etc).

That leaves the third reason - "gesture of surrender to a victorious enemy". Hmmmm.....Interesting.

:-)

Seriously though, does anyone know what the kneeling is supposed to symbolize? This is a Western custom (though gaining popularity among the urbanized youth in India too or so I hear) so its origins probably lie somewhere in medieval Europe. I couldn't find any concrete information.

Saluting Randy Pausch

2007-09-20T14:02:00.000+05:30

First read this (via news.ycombinator).

Then see the streaming video. It is terribly slow and broken up on a typical Indian internet connection but just see it anyway. You'll thank me afterwards.

If you are a parent, a spouse, a teacher (of any subject), a student, a programmer, an artist, a designer, a public speaker, just see it.

Dr Pausch, I have never met you but you just changed my life. Thank You.

Update: The video is on YouTube in ten pieces. This is much better than CMU's streaming video, because in these you can load the full video before viewing, go back and forth etc.

The Lame and the Blind - Proofs And Code in research software

2007-09-18T18:35:00.000+05:30

There is this old story about a blind man teaming up with a legless man. The blind man carries the legless man on his shoulders and provides the locomotion while the sighted cripple tells him where to head.

Similar divisions of labor exist in enterprise software (think "developer" and "analyst") but this two fold division of labour is exacerbated when one focuses on research software. Here the scientists are the "sighted folks" who know where to go but can't move and the developers are (ideally, as we'll see later) the folks who know how to move the project forward by writing code, but don't have the requisite mathematical/scientific skills to navigate and choose one direction over another.

Earlier today, I was talking to a scientist trying to move a multi million dollar machine learning project forward, who was feeling hamstrung because it was impossible to find a few people who could follow advanced calculus and also write good C code.

Unlike in the folktale, a real life arrangement consisting of a blind man and a cripple would be very inefficient, with the occasional fall, not to mention the constant stubbing of toes and odd outburst of swearing. But equally, nothing in real life mandates this kind of "pairing". Nothing prevents a scientist from learning how to code (except a feeling that coding is a relatively "blue collar" activity) and nothing prevents a developer from learning the requisite maths and science (except that most developers, especially the enterprise/agile/dsl types either don't have the caliber to learn hard math or think erroneously that these skills are beyond them or just don't want to work that hard).

The few folks who do master both proofs and code, can and do make a tremendous difference. The very best people in every research area that incorporates software and maths are very capable scientists/mathematicians and very capable programmers. Peter Norvig is an example. As is Donald Knuth. Or Manolis Kellis. Or Yann leCun.

If you are running a research software project you'd want to staff it with high level multi-class (think dungeons and dragons) scientist-mathematician-developers. Unlike in D and D specializing in one area does not need to involve trading off the other.

After all the lame man/blind man arrangement is not the best possible one. The best (and normal) situation is having two (or n) people who can both walk and see.

Erlang is the new Ruby

2007-09-04T09:48:00.000+05:30

There is a particular type of programmer who tries to keep up with the "coolness" crowd by jumping on to the latest bandwagons and hanging on for dear life (till it is time to jump again). The word "fanboy" doesn't quite describe these folks but it gets some of the nuances right. The key characteristic is that they talk a lot about what essentially they don't really understand.

For a while , the fad was ruby. "Ooooh look we can use select instead of a loop" "Oooh everything is an object". Then it was creating half baked "DSL"s, especially in the context of Rails.

Now Ruby seems to be losing some of its sheen, and the lemmings are migrating to Erlang. Now, Erlang is a very powerful language that is worth studying and it has been around for the last twenty years or so, but the next time I meet someone "evangelizing" Erlang, without the faintest idea of how functional programming, pattern matching or concurrency really works, I am going to do something nasty.

Don't get me wrong. Erlang is well worth studying. But first write a 10,000 line+ non trivial program in it. Create an open source library or two. Submit a few patches. Then try to convert the heathen. But I guess if you were to do that you wouldn't be a fanboy anymore.

Ahhh. Choices.

Indian "Journalists" strike again

2007-08-07T13:51:00.000+05:30

SmartTechie (what a horrible name for a magazine !! ) has an article purportedly written by Roy Singham, CEO of Thoughtworks. Having worked at Thoughtworks and having met and interacted with Roy, I am very skeptical about the attributed authorship. The article has many examples of "Indian English" and is very tortured writing to boot. Roy is a very articulate speaker and writer and speaks perfect "American". So if you read the article and go "WTF?!!" do keep in mind that the writer is probably some Indian hack pretending to be Roy

Here is a sample " To over take other automobile companies and to reach at the helm, Toyota had devised a comprehensive action plan which included the Toyota Production System. " .

"reach at the helm"? WTF? You expect me to believe Roy writes or speaks like that?

I can only conclude that some half baked journalist essentially made up the article and attributed it to Roy.

Thoughtworks is an excellent company, with many world class writers in its ranks and this kind of ghostwriting goes against everything Thoughtworks stands for. And if the editors at the magazine must have someone pretending to be Roy, they magazine ought to get someone who understands how to write English well.

It amazes me that people whose profession is all about writing well don't know the rudiments of the language in which they write.

On Having Friends Who'll Change the World

2007-08-04T16:01:00.000+05:30

A few months ago I and a few friends started meeting at the end of every month for an "iteration review" of how much progress we've made (or not) on our "life work" that month. The structured part of the meeting consists of each participant reporting what went right, what went wrong and what was learned during the preceding month and what the plans for the coming month are. Anyone can ask clarifying questions. This is followed by unstructured discussion over dinner (and drinks, for those who indulge in that particular vice). Apart from the obvious benefits of meeting up with friends regularly, the act of vocalizing what was learned has tremendous value.

The last meeting was focussed, for instance, on the difficulties of transitioning from the habits instilled from working in India's "Silicon Valley" to those needed for doing world class work, particularly in research/scientific software, the price one needs to pay and the metrics of progress. While that was a fascinating discussion, what struck me then was how lucky I am to know people who are unwilling to accept the status quo and exert themselves to become the best they can be.

This extends beyond the attendees of our end of month meetings. I just got back from lunch with another friend I haven't met in a while. I am amazed at how much she has learned in the intervening period and how many different areas of improvement she has targeted for the future. Indian society has the nasty habit of grinding down women who want to excel in any field, but then she has extraordinary potential, so that will be an interesting career to watch. If she ever starts a company, I will be the first to invest in it.

On reflection, I realize that I subconsciously rank people by (a) their potential to excel and (b)the ratio of actual achievement to potential achievement. I pay attention proportional to how high I think they score on these parameters and so I end up ignoring people who, in my subjective opinion, have low scores on both parameters.

I am not very certain that this is the "right" way to judge people or decide who to befriend (I guess I am an unconscious "elitist") but it results in my knowing people who strive to excel. And given the caliber of some of my friends, I effectively end up knowing people who'll change the world.

Misquoted in the Media

2007-07-08T20:13:00.000+05:30

A reporter from the Economic Times asked me for some information about our wargames club, conducted a telephonic "interview" with me and a couple of friends, and then wrote an article in the paper.

So far so good. What is not good is that he has quoted me as saying things I never said , and in horrendous English, to boot.

Here is what I am supposed to have said.

(Apologies for the terrible English and mangled tenses. *I* didn't say *any* of this. The journalist just says I did. The language and ideas are his, not mine. I checked with some journalist friends of mine and apparently this practice of expressing journalists' ideas as "quotes" from the people they interview is common practice in India! And I thought only the Indian Software Industry was a con game!)

Anyhow, onto the quotes (from the Economic Times dated 8th July 2007) - my comments below each quote

"We are an informal group of Wargamers based in Bangalore. We are interested in topics like strategy in wars, the history of warfare, historical re-creation of battles, battles of all times and eras. We do not use miniatures in our games but make do with suitable counters instead,” says Ravi Mohan, group co-ordinator."

[ I never said this. This paragraph is lifted straight from our club website. See the first paragraph].

The report continues to "quote" me.

In the last one year, the group strength has increased substantially. “

[huh? This is outright fiction. The club membership has held steady over many years and there were no additions in the last year. Needless to say, I never said this either! ]

As kids, we all use to play games such as this. So there is a natural interest. And, as awareness spreads, we'll see more people joining the fray,”

[Aargh! This is tortured, atrocious and ungrammatical English , and cliched to boot. Does anyone talk like this? "we'll see more people joining the fray" ?!!

The least a professional journalist could do (even when inventing quotes) is to use correct grammar.]

There are many unintentionally funny errors in the article- E.g: the use of "Canon" for "Cannon"- but it is too much effort to list them all here.

The next time someone wants to interview me for the mainstream media (and may that horrible day never come!) I'll just write the damn article myself. Or even better, run like hell!

Zenter - Working Habits for a succesful startup

2007-06-22T08:43:00.000+05:30

I am still "offline" but this was so striking I wanted to record my awe. from here, (emphases mine)

What was a typical day or week like at YC?

We basically lived by the following schedule 7 days a week for 3 months. 10am - Noon: roll out of bed and stumble into the living room to sit at the computer, check email, blogs, news for the day and start coding Noon - 12:30: eat our $2 Lean Cuisine lunch - My personal favorite is Swedish Meatballs 12:30 - 7pm: Heads down coding, music played all day long and we alternated days to pick the play list which worked well 7 - 8: Kick Robby's ass at Tennis on the apartment courts. I think I may have won 3 sets over the entire 3 months. But they were always like 6-4 and it got us out for some exercise. 8 - 9: Make and eat gourmet dinner - mostly tombstone pizza, but occasionally Red Baron when they were on sale. Care packages from family were also extremely helpful to make sure we didn't eat too poorly. 9pm - 4am: Code like the wind - maybe do a design session or two if we are trying work through something particularly hard. 4am - 10am: sleep

I am impressed!

Archive Visibility Fixed

2007-06-21T18:12:00.000+05:30

The archives for this blog were not visible. Now they are.

Thanks to the reader who sent this in.

Blog On Ice For a While

2007-06-15T03:10:00.000+05:30

the nutshell version:

I am *extremely* busy and won't be paying too much attention to this blog for the next couple of months. If you need to contact me, write an email or call. Thanks

Longer version:

I just spent an hour or so responding to a comment and an hour a day is something I just can't afford these days. I have a lot of travel coming up. My research has finally "clicked" and while there is a tonne of work to be done, there is no longer any ambiguity about what to do or how to do it. But I really need to buckle down and work. It's all very exhilarating but time is at a premium.

I have a roaming cell phone number and check my email twice a day. So do write or call if anyone needs anything. Friends can almost always ping me on messenger or gtalk.

I will be back in late August/Early September.Thank you for your understanding.

Rogue Researcher's Route Map - Addendum

2007-06-14T00:20:00.000+05:30

I wrote about what I learned when attempting to do Comp Sci research on my own (A Rogue researcher's route map).

Something I've learned since then might help people read research papers- Every (good) research paper asks (and hopefully answers) a question. Try to discover the question being asked and the answer and nail them both down by trying to express them in a couple of sentences each.

This really helps you understand the significance of a paper for your research. And stops you from panicking when you discover papers which seem to cover ground similar to what you are doing).

Paul Graham , talking about startup ideas, says,

Treating a startup idea as a question changes what you're looking for. If an idea is a blueprint, it has to be right. But if it's a question, it can be wrong, so long as it's wrong in a way that leads to more ideas.

Seems to work for research as well, especially for solo hackers attempting research outside institutional frameworks.

The Stratal Effect

2007-06-13T22:37:00.000+05:30

Dr de Bono defines a "stratal" thus.

"A stratal is a number of parallel statements or observations that are put together as a whole. The statements need not be connected in any way whatsoever. There is no attempt to be comprehensive. There is no attempt to cover all aspects or to be descriptive. There is no attempt to be analytical. Just as a random word is used because you want to use it, different statements are put together as a "stratal" simply because you want to put them together in this way. A stratal is a number of unconnected statements put together solely to form a stratal. "

The key is that the statements are loosely connected or not connected at all. De Bono suggests that a stratal be used for generating ideas or to convey a particular effect.Some of the best poems can be seen as stratals.

But one can also use stratals for different purposes. One of the most fascinating uses (at least for me) is to lay out two (or more) random statements or ideas in parallel, especially in conversation and then see which of the ideas gets picked up. Let us say you are talking to someone who feels has a problem with relationships and you think there might be an empathy skills deficit. How do you check? One way to do it is to lay out two statements in parallel, one which checks for empathy, one which checks for it's opposite(say self centred ness). and see which one gets picked up. Say something like "I feel very frustrated when I talk to you so perhaps you can change that behaviour in the future" There are two ideas here, one starting with "I" and one starting with "you". A person having problems with empathy will probably pick up the sentence beginning with "you" and answer something like "Oh I know I need to think about changing my behaviour". A more empathetic person will place the focus of consciousness outside himself and on the other person say something like "Oh you feel frustrated? Why do you say that?"

There are all kinds of variants of this technique in combination with all sorts of other psychological/counselling tools, but the essence is simple. Layout parallel thoughts.

I picked this up a variant of this technique from John Hundreiser (aka "H", once head of recruiting at Thoughtworks ) who frequently uses a similar technique in his interviews. He would ask you a chain of three or four questions and see which one you'd pick up . He was the head of HR and his interview would come after many rounds of interviews. he'd ask something like "How did the previous interviewers from TW treat you ? Did the explore technical questions in depth? Are you feeling comfortable?" There are three distinct questions here and many people would answer one of these questions, at most two and forget the third. It is the very rare (and very self aware) person who says something like "To answer your questions in order, first ...second , .. third .. "

Of course the big problem with these tools of applied psychology is that they are all very context dependent and to use them properly the user needs to be (relatively) free of complexes and biases himself and also have the ability to evaluate the results properly. But that is the subject for another blog entry.

"Concentration is the Enemy"

2007-06-13T11:43:00.000+05:30

A fragment from a (chat) conversation with my friend Abey.Context == Talking about programming,

me: Hey did you know about this " I think maybe concentration is the enemy. Seriously. If you're working on something that is so complex that you actually need to concentrate, there's too much chance that it's too hard. - Ron Jeffries"

Abey: yeah that makes sense[/sarcasm]

me: I guess one could always pair to avoid the "danger" of concentration. Hard to focus if someone is yammering in your ear all the time.

Abey: Yup! Pairing always helps. Instead of one person fretting and strumming the guitar and singing (takes too much "concentration"), you should have one doing the fretting and one doing the strumming

me: Let's "turn the dial up to 10". How about two pairs, one pair on the frets and another on the plectrum. Lots of redundancy and no need to concentrate!

Abey: yeah Extreme Guitar .. perhaps we could write a book?

On an unwillingness to pay the price

2007-06-04T08:48:00.000+05:30

This is something I've started noticing recently. A lot of people claim they want something and then balk at paying the price. The pattern is

I want X. I soooooo wish I could have/be X
Sure why not, Do A, B and C and there is no reason you can't have/be X.
Urmm yeah but that's too tough.. I don't want to do that.

Here is an example.

A friend says "Dude I wish I could stop working on enterprise software and work on cool projects like you do". ME :"Sure you can , just work through these books, learn the required math and you can work on these things too. I have more work than I can handle and would be glad to give you some." Friend:" Maths!!! I hate maths!" or "yeah dude but that's gonna take me two years!! That's too long"

A variant on this, particularly applicable to relationships (of any kind) is an unwillingness to reciprocate. If you expect people to care about you and devote significant amounts of time to you, common sense says you have to do the same for them. I've seen many (too many) people who are very focussed on "taking" and not so much on "giving". Which is fine but then I hear complaints of "I am un appreciated/my husband left me/my girlfriend acts remote/ I don't have friends I can trust".

I don't get it. Is it too old fashioned to believe that one has to actively and diligently work for the really good things in life, whether the aim is a good job or a good relationship or something else?

FAQ (from my mail-box) #3 Can we hire you ?

2007-06-04T07:45:00.000+05:30

Probably Not.

Some context. Every single day, I get 3-5 emails from people trying to offer me a job. The obvious form letters go into the trash can immediately, but I'm old fashioned enough to write a response to any mail that looks as if it has taken a human being some time to compose (Examples of both type sof mail at the end of this blog entry). Of late however, I find myself writing the same thing one too many times, hence this blog entry.

So, you are company X . Will I work for you ?

If your business model depends on body shopping (where body shopping is defined as "we don't care what projects we work on or for whom and the business model is x number of developers farmed out to random clients for y $/hour"), no, I'll never work for you. Been there, done that, got the T shirt. There is nothing you can offer that I want. I don't want to be an "architect" who does "high level design". I don't want "extensive travel" or "green card sponsorship". I don't want to be a Project Manager type and "lead enterprise changing teams for Fortune 500 clients". I do not want to be an "Agile Coach" (yuck!).

Please don't write to me. You are wasting your time and mine.

If you are a product company with a strong technical focus or even a tiny startup with an interesting idea, while I don't promise to work for you, I'll give your email serious consideration. I don't promise to accept your offer, but you'll get a reply.

Here is an example of a recruiting mail that goes straight into a the trash bin.

Dear Candidate,

I saw your resume in out database and would like to talk to you about some positions with our Direct Client in Naples , FL. If you match to any of the below mentioned positions then please mail me your resume with the job title in the subject line.

Location: Naples, FL, USA

Duration: 2 years +

Start Date: ASAP

Requirement : Java Architect

* 8+ years of software development experience

* Strong OO concepts

* Experience with RDBMS (Oracle and SQL Server)

* experience in leading a development team and working with Fortune 500 clients.

* Good problem solving skills

Please email me your resume if you are interested on [email] or call me at [phone number] if you have any queries.

Regards, [Name]

It boggles my mind to imagine that people expect responses to emails that begin with "Dear candidate"(!!!).

Here is an example of an email that gets answered in the polite negative.

Hi Ravi,

I hope you remember me. We worked together in [company] where I was working as a recruiter. Well I've moved on to [Outsourcing company] and we have a desperate need for senior people. Compensation is not a problem and even if I do say so myself, we have an interesting culture and great projects. if you are looking for a change, do touch base with me.

Regards,

[Name]

This is the kind of email to which I hope to forestall having to reply by writing this blog entry. Let me repeat, if your business model is built on selling developer time by the person hour, I am not interested.

Lastly here is a mail I got from a recruiter in a certain company head quartered in Mountain View. It is still very much a slightly customized form letter and if I wanted to work for the big G, I'd probably ask someone I know who works for them to reccommend me, but this letter got a reply.

Hi Ravi,

I would like to introduce myself. My name is [Recruiter Name] and I am a technical recruiter with Google. I came across your profile online and I thought you would be a good fit for one of our Software Engineering roles on our test tool development teams. If the proposition sounds interesting and you would like to learn more, please contact me directly via e-mail or phone. I would love to share with you the exciting approach Google takes to software testing and the amazing advances we are making in test tool development.

Thank you so much for your time and consideration. I look forward to hearing from you and talking to you about a career with Google. Respectfully,

-- [Recruiter Name] Google Staffing [Recruiter Phone Number]

1600 Amphitheater Parkway Mountain View, CA 94043 [Recruiter Phone Number]

Product Company? Check. Technical focus. Check. So I spent about 10 minutes thinking about it and then wrote a "Thanks but no thanks" email. I don't want to write testing tools, (though I know a few people who are) and I am not sure working for Google now is a good idea. Their stock prices are at unheard of levels and Bangalore (where I am located) is probably very far away from the real technical action in Google, if the "Bangalore centres" of other companies I know (and are competing with Google) are anything to go by. I could be wrong, but I've never come across anyone who worked for Google Bangalore. If I knew someone I admired who worked in Goog Bangalore, then yeah sure I'd consider it. I also hear on the grapevine that the old "Google culture" is fading a bit. I'd rather work for a startup with 5 smart people than for Google.

Which brings me to an interesting point. Beyond a certain level of expertise, I doubt if recruiters can be very effective at all in hiring good developers. Most developers are pulled into new jobs by their network of fellow developers / people they worked with before. So here is a hint for recruiters trying to hire good developers. Ask the best developers you have to contact the best developers they know. Beyond a point, I suspect that's the only way.

FAQ (from my mail-box) #3 I am just starting out in the sw industry/I am still in shool. Can you give me some advice?

2007-05-28T17:29:00.000+05:30

Sure. Giving advice is easy. Following it is hard :-)

If you like coding (and technology in general), seriously consider if you want to study further. A B.Tech from most Indian Universities is utterly useless. A good MS or PhD will get you the good technical jobs. If possible get some deep grounding in science or maths. Or if you are the non tech type consider an MBA.

Even if you can't study further, start specializing in a "tough" sub domain of software. If you are working in Java, learn C and assembly. If you are wrestling with java and xml on the job , acquire some expertise in algorithms or networking stacks. which brings me to

Decide on whether you want to be an "enterprise" programmer early. Pros = fairly easy work, easy to get that first(and subsequent) job, plenty of travel. Cons = ultimately unsatisfying if you are a seriously technical person (you may need to move up to be a manager if you want good work or money, but being a manager is not a technically intensive job), it is too easy to get into an endless grind of developing the next website for an arbitrary series of industries.

If possible, join a startup/small company/technically sharp company (like Google) vs one of the behemoths (like Wipro or Infosys). [Useful heuristic - if the company is an ISO/CMM./other process heavy company, run for your life!]. You'll learn more in a shorter period of time. You can always join Wipro or TCS after a couple of years when you are ready to "settle down" (and be mediocre).

Be prepared to work like crazy for the first couple of years.

Be prepared for intensely political environments (the more so if you join a big company). I am not saying be political, just be aware of politics. Politics is a part of every human endeavor which incorporates more than one person. Either scrupulously stay above it or learn to play the game well. Don't get caught in a "middle path" where you play sufficient politics to get hurt but not enough to use it get more money or power.

Be aware that a job is not a continuation of college. It is surprising how many people make this mistake. (Some companies even encourage this in a ludicrous attempt to be "cool". If the company HR asks you to play Dumb Charades on a company outing, or if the company workforce is divided into "houses" which compete against each other in "skits" or "Dance competitions", you are probably in one of these companies) Seriously consider leaving. There is a subtle distinction between a job that's fun and a boring job with pseudo hip "activities" layered around it.

Above all if you don't enjoy coding, for God's sake move on. Do something interesting with your life. There are enough doofuses in the Great Outsourcing Destination without you adding to it.

FAQ (from my mail-box) #2 Should I apply to Thoughtworks and/or can you help me get in?

2007-05-24T16:15:00.000+05:30

This is a question that comes up a lot.

My answer to questions of the form "Hi , I am ... my experience is ..... Do you think I should apply to Thoughtworks?" ( i can't imagine writing a letter like this to someone I've never met, but many people seem to think it is ok, so .. ) is

I have no clue! It is up to you to decide what your career should look like. In general, TW is a good company to work for if (a) you want to work on "agile" (b) you like to travel (more so if you are applying to the TW offices outside India) (c) you like working on enterprise systems (d) you are very good at whatever you do (coding, analysis, dba whatever).

TW is very un hierarchical (less so these days than before but still largely true) and if you are the type who measures your status by what your title says or how many people report to you then you should NOT apply. Oh yeah, you'll find the interviews either very tough or very easy. (Folks who are really good at what they do find the interviews easy. Those who are not, find them correspondingly tough or "unfair"). There are way less idiots in TW (India) compared to almost any company in Bangalore I know of (Google excluded), but there ARE a couple and if you face them in an interview that's your bad luck. Just move on.

In a nutshell, I will **not** reccomend people I don't know (It is amazing how many people ask me to do this). Just for context, I have reccomended exactly one person to TW (needless to say he is very very good) in the last 5 years (hi Sheroy!).

I left TW almost 3 years ago and have no inside information (and if I did I wouldn't share it with people I don't know). If you *have* worked with me in any capacity, please feel free to ping me and I can talk to you about whether TW would be a good fit for you (you'll have to buy me dinner/beer).

PS: the Thoughtworks India recruiting page is here.

I would like to point to the recruiting pages for the other offices , but some moron of a web developer has changed the Thoughtworks Home Page so it redirects to a "Select Country" page (wtf!). So I can't be bothered to find all of them and point to them, but in general, go the appropriate country's hiring page and apply online.

FAQ (from my mail-box) #1 What are development costs in Bangalore?

2007-05-23T15:18:00.000+05:30

[context] I get all sorts of questions in my email every day. I try to answer all my mail but I am getting tired of answering the same questions again and again. So I am going to post these common answers here and not answer by mail anymore [/context].

The minimum cost of a developer(aka a "coding body") in Bangalore is about 20-25 $/hour. This is the rough rate charged by Wipro/Tcs/other "outsourced enterprise" bottom feeder companies, for a "standard" development project. Of course most of these projects are seriously overstaffed, but that's a different issue. The hard part here is getting a good team who can deliver good code vs process documents or whatever, but at 20-25 $/hr you will get a pool of candidates.

If you need *really good* developers (the situation for folks who are trying to optimize time to market or trying to do something unusual, say a startup (vs crank out umpteen jsp pages or make minor bugfixes to a legacy app or whatever), the cost (in Bangalore) is 40 $ +. How much that "+" represents depends on a lot of factors. I've seen rates of upto 200$/hr for truly exceptional coders/projects, but that's (very very) rare. 40 $ is a good baseline.

It is VERY VERY hard to find good developers in Bangalore, just as is true elsewhere in the world. So the ability/willingness to pay the above rates does NOT guarantee you good developers. 95% of the software "industry" in Bangalore is staffed by people who shouldn't be coding. Think carefully before you spend your money in "outsourcing".

Bangalore is one of the most expensive cities (if not the most expensive city) in India. If you can find good developers in less expensive parts of India, like say Trivandrum or Bhubaneshwar you can expect to pay about 25$/hr for a really good developer and about 15$/hr for a "coding body" type. The density of good developers is much lower in such small cities, but if you can find good people, good for you.

Ok that's one question I don't have to answer by mail anymore.

Back In Bangalore - A Ramble

2007-05-20T18:43:00.000+05:30

After some travel (and more to come soon) I am back in Bangalore. On Saturday, I went to the Thoughtworks "MasterClass" conference. Overall this year's MC was an improvement over its predecessors, because it focussed on people who actually code on a daily basis and instead of high flown abstractions about "agile" etc , the talks were very focussed and drew on their expriences. Less "consultant speak" and more "programmer speak". Nice.

Pramod Sadalage demonstrated database refactoring. Vivek Prahlad,, author of Frankenstein, gave a competent lecture about Evolutionary Testing. Srihari Srinivasan described how his team had created a mini language ( I won't use the over abused "DSL" -- to quote Alan parsons from a comment he made on reddit "It seems anyone with 2 minutes ruby experience calls a method with a inline map, and no brackets, and calls it a "DSL" " :-) ) and then build some IDE like screens around the language. The interesting thingwas that instead of the usual ruby DSL fanboy froth, this was a very sober presentation about an "external DSL" with a proper grammar, parser etc. There was a bit of "reinventing compilers" flavour to the talk but I enjoyed it nonetheless.

Overall fairly interesting and the organization of the conference was top notch.

I thought the presentations were somewhat basic , but I was not obviously part of the intended audience. To illustrate, Vivek asked the crowd how many people had *heard of* Selenium and about 6 people (of maybe 250 or so) raised their hands! 6! Considering they are all "working professionals" that's .... scary.

I also used the conference to poll some of the best programmers in TW about what they thought of "Behaviour Driven Design". I was never very impressed with either the people or the ideas in "BDD", but Dan North seems (from his writings) like a thoughtful fellow, quite an aberration fornm the usual BDD crowd. All the people I asked, dismissed BDD with varying degrees of amusement and derision (think "that'a bull****" to "It doesn't make sense. people are just making noise").

Meanwhile here is something I thought of but can't summon the energy to do. Download JUnit, change the name of "TestCase" to "Context", the part that looks for "testXXX" methods to look for "shouldXXX" methods and "asserts" to various forms of"expect" , or.shouldDotBeDotPseudoDotEnglish methods. (With JUnit 4 one would need to hack the annotations , but the principle is the same). hey presto (almost) "BDD" on the cheap.

I've always thought "BDD" was about consultants trying to create new material out of old to write new books and get a few consulting assignments. It is good to know that most of the capable developers in TWI are equally unimpressed.

The audience questions were dumber than usual. I am not talking of questions that arise because the questioner is ignorant of the subject. Those are all right. No one knows everything. These were questions that came from not listening to what the speaker was saying. In addition, they were thinly disguised statements trying to prove that their half baked "opinions" had some technical merit. (roll eyes).

Vivek in particular handled these ultra stupid questions extremely well. His strategy (which I've noted for future use) seems to be along the order of "acknowledge the question, ignore its latent stupidity, say something useful that benefits the crowd and conclude with a "That was a good question" . I would simply have said something like "duh... is that a question or your untested opinion?" or something equally nasty.

I wish there were some kind of gathering place in Bangalore for people who actually write code, because they LIKE to write code (vs because they want to get paid). I have a suspicion not many of these idiots would attend. And the conversations would be of a higher quality.

Speaking of which, the best thing about the conference was that it enabled me to touch base with an old college mate of mine, Peter Thomas. (In college, Peter was one of the good guys while I was more .... ummm.. radical). Peter is pretty unique in that he is a manager (with an MBA from one of the top institutes in India) who gave up being a suit and is focussed on writing code. Very impressive. He is the author of JTrac and his blog makes compulsive reading. As the conference wound down, Peter I and Siddhartha hit Koshy's for some conversation over beer. We talked of code, music, wargames, etc till Siddhartha had to leave to catch his flight to Chennai.

I also managed to get a sneak peek at Thoughtworks's agile project management app, Mingle. the folks manning the booth mostly weren't developers (or at least they didn't sound like developers) so they couldn't answer some technical questions, but overall I think it's a very good attempt. If they make it free for Open Source projects (as I heard one of the TWers claim) I'll give it a whirl.

All is not sweetness and light. Now that I'm in town,I have one of those "meaning of this relationship" talks coming up with the girlfriend (a force of nature is probably a better description). Urrgh! I'll never understand women! I do NOT intend to marry anybody. do NOT! NOT! Thanks!

(end ramble)

IIT Grads ! (necessarily) == Good Programmers Redux

2007-04-27T09:45:00.000+05:30

When I wondered some time ago why the best programmers didn't come from the population of IIT grads, many people claimed I had all sorts of motives (ranging from jealousy to fear) for having such "heretical" thoughts.

Now some other folks seem to offer a degree of vindication.

From the latest Outlook magazine (You can read the first page of the article here - I am not sure how long the url will be valid). For those too lazy to click on the link, an excerpt

What has triggered the discussion within the fraternity is the concern expressed by two distinguished IIT alumni about the general decline in standards of excellence at the institutes in recent years. Their remarks have questioned the calibre of students who make it into the IITs by subjecting themselves to the killing rigours of coaching factories in places like Kota and Hyderabad. The alumni seemed to conclude that these products of coaching factories—who now form, according to Wikipedia, 95 per cent of students at IITs—had a blinkered approach to education, did not recognise new ideas and had lost the spirit of inquiry and innovation. In short, elements that had built Brand IIT over the decades had now gone missing

The first salvo was fired by Tata Steel MD and IIT Madras alumnus B. Muthuraman. Speaking at a Ruby Union meeting of the pioneer batch organized at his alma mater in January 2007, he said that tisco was "not likely to recruit" IIT graduates any longer. He based his opinion on a recent interaction he had had with a set of final year students, who he said did not even know the authors of books they were supposed to have studied. He made many other related statements suggesting that IITs thrive only on their "past reputation" and that he actually preferred other college students who were amenable to company training.

Now before the frothing-at-the-mouth defenders descend on this blog, I am not saying that IIT grads are poor programmers. What I AM saying is that in India there is no correlation between programming ability and the school of the programmer. Any company that hires only IITians AND (boolean AND - do read this sentence carefully before sending off that hate mail) hopes to recruit impressive programming talent by doing so is asking for disaster.

An IIT is *not* Stanford or MIT or CMU. Not even close. If you doubt this, first take a look at a typical PhD paper (in say Robotics) from any IIT you choose vs say Stanford. Then tell me I'm wrong. Thanks in advance.

Learning From Sudoku Solvers

2007-04-24T17:06:00.013+05:30

Ron Jeffries attempts to create a sudoku solver - here, here, here, here and here. (You really ought to read these articles. They are ummm...{cough} ...err.... enlightening.)

Peter Norvig creates a Sudoku Solver.

Compare. Learn.

Update: Discussion on reddit

somewhat peripherally related but similiar (this about the bowling game) discussion

The devgrind post

Update 2:

(Oct 2009) Peter Norvig spoke about this post in Peter Seibel's book, "Coders at Work". My reaction..

Update 3:

Peter Siebel author of "Coders at Work" weighs in. Some gems there.

"One thing I noticed, reading through Jeffries’s blog posts, was that he got fixated on the problem of how to represent a Sudoku board. He immediately started writing tests of the low-level details of a few functions for manipulating a data structure representing the 9×9 Sudoku board and a few functions for getting at the rows, columns, and boxes of the board. (“Boxes” are what Sudoku players call the 3×3 squares subsquares of the 9×9 board.)

Then he basically wandered around for the rest of his five blog postings fiddling with the representation, making it more “object oriented” and then fixing up the tests to work with the new representation and so on until eventually, it seems, he just got bored and gave up, having made only one minor stab at the problem of actually solving puzzles.

I suspect, having done a small amount of TDD myself, that this is actually a pattern that arises when a programmer tries to apply TDD to a problem they just don’t know how to solve. If I was a high-priced consultant/trainer like Jeffries, I’d probably give this pattern a pithy name like “Going in Circles Means You Don’t Know What You’re Doing”.

.....

(Norvig has) 7 definitions in 12 lines of code and he’s done with data representation. I’m not sure how much code Jeffries ended up with. In his fourth installment he had about 81 lines devoted to providing slightly less functionality than Norvig provided in the code we just looked at. In the fifth (and mercifully final) installment, he started adding classes and subclasses and moving things around but never presented all the code again. Safe to say it ended up quite a lot more than 12 lines; if he’s lucky it stayed under 120. "

Siebel's post is worth reading in its entirety.

Restored Comments

2007-04-19T14:40:00.000+05:30

A quick perusal of blogspot's comment moderation page showed me that some comments were pending. Generally I get a notification in my gmail account, but for some reason these didn't make it.

Sorry.

All the comments have been restored. They are scattered across many posts. The best was Ryan Cooper's comment on "If Toyota outsourced .. ".

Ryan, I agree with most of what you say. I still think the Poppendieck's books are worthless. But reasonable people can disagree. :-)

A Rogue Researcher's Route Map

2007-04-19T10:32:00.000+05:30

One of my resolutions for this year is to make the transition from coder to researcher. While the journey is still very much ongoing and the end point is very far away, I believe I have a sufficient number of bruises from falling into pits (and the callusses from digging myself out of them) to make a few notes that may help illuminate the first few miles for anyone attempting a similair journey.

One of the problems of "doing research" is that it is mostly undertaken in the context of somone undergoing a PhD or other academic program. There are really no guides for the non-academic researcher who wants to do research for the fun of it or as an adjunct to his day to day work. Most of the online 'guides for the perplexed' assume you have been admitted into a stellar PhD program. This advice is useless to someone who is not in academia.

What follows comes from my experience in trying to scale the walls of research. I believe that a formal academic program is the best way to acquire research skills. If you can manage an MIT PhD, go for it. But I also believe one can contribute to the sum of human knowledge without that background. I believe I can more or less do anything I want to. I am kinda stubborn that way. So take all my advice with a mountain of salt.

What follows is written for the non academic researcher, working part time. Those in a full time academic program have much better resources and don't need my ramblings.

With all those caveats in mind, here goes,

Be sure you want to be a researcher. And why.
The answer to "why" is intensely personal. You have to find your own answers.
I want to be a researcher because I want my working life to be intellectually rich. Enterprise programming is too boring. Back when I was building the umpteenth banking/retailing/insurance/leasing app, I found I could coast too easily and work with maybe 10% of my ability and still keep up with some of the best programmers in that space (Most of them were working at 10% capacity too. This led to endless "losers club" sessions. Some of them sold out and became project managers. And got more frustrated. Others just got frustrated).
The nature of the work (at least the variety that washes up on the shores of the Great Outsourcing Destination), is an endless repetition of a few mundane tasks. The complexities of enterprise software are in my opinion arbitrary. Too many snake oil methodologies, not enough deep thinking. Lucrative, yes. Stimulating, no. In a nutshell, I was bored stiff.
I actually like programming. I enjoy reading technical books. I read technical (and these days mathematical) articles or books on a plane jouney or just before sleeping. I chose programming as a career because of the intellectual challenge inherent in it - that a good chunk of it ended up writing horribly boring enterprise software is a tragedy we won't delve into. At least in the future, I would like my working life to consist of wrestling with significant problems, not assembling api's like a jigsaw puzzle to create systems that I don't care about. (hey if you like enterprise software , good for you. I don't. Please don't fill my mailbox with whiney comments about how Banking apps are really uber cool).
As Douglas Comer says in his article (after a lot of warnings about not getting into a pHd Program for a the wrong reasons),
"If you have the capability and interest, a research career can bring rewards unequaled in any other profession. You will meet and work with some of the brightest people on the planet. You will reach for ideas beyond your grasp, and in so doing extend your intellectual capabilities. You will solve problems that have not been solved before. You will explore concepts that have not been explored. You will uncover principles that change the way people use computers."
I want all of that. I'll probably go through a PhD sometime in the future but think a lot of what Dr Comer outlines is well within the grasp of a dedicated individual with good programming skills and access to the internet (well, perhaps not the "work with the brightest people on the planet" part, unless you consider collaboration over the internet "working with"). I can live with that.
Do the groundwork.
This is avery hard step for someone who is working in the industry vs in academia. There is almost always a substrate of very technical (and often highly mathematical) material you have to digest before you can comprehend (leave alone work in) research areas. My interests are (broadly) in Artificial Intelligence (more specifically Machine Learning) and Compilers and Programming Language Theory. The former calls for a good grounding in Calculus, Linear Algebra, Information Theory and Algorithmics. The latter needs a good grasp of things like the (typed) lambda calculus and Automata Theory.
Theoretically if you've gone through an engineering or science Bachelors degree (BTech or Bsc, particularly in Comp Sci), you are supposed to know a good chunk of that. Anyone who has done a Bachelors in India (excluding the IITs and IIScs) knows that most people study this stuff 2 days before the exams (which are too often memory oriented) and forget everything two hours after the exams.
For those who doubt this, I suggest the following experiment. First get a moderately tough problem in say Calculus or Probability. Find someone in your office who has graduated from an Indian university in Engineering or Science with a good GPA . Ask them to solve the problem. Enjoy the reaction. Contemplate the fact that an "engineer" (or "scientist") is ignorant about Calculus.
Indian technical education very often consists of memorizing formulae without getting an intuition for the concepts behind the formulae. Un(?)fortunately ,to work in research one has to understand the underlying math or science.
Working in Enterprise Software makes this situation worse. In spite of the Industry's endless thirst for Engineering and Science graduates the cold hard fact is that you very seldom use any science or engineering when you work on the typical Insurance or Banking web app. You end up endlessly churning APIs (ORMs, Rails, hibernate blah) and methodology artifacts (e.g uml diagrams or storycards) and (more often than not) fitting chunks of horrendous code together with spittle and prayer. Work on such systems long enough and you forget what little you retained from those years of education.
It took me amost two and a half years (after leaving that "enterprise" job) to acquire the background skillset. It may take you less time. But it will take some time. And if this time is spent in adition to your 9 to 5 job, be aware that this part of the work is a very very lonely path.
You've achieved the necessary skill when you can read a paper from a contemporary journal or conference in your field, like you'd read a news paper article or a requirements document. Another milestone is when you lose your fear for mathematical notation and proofs :-)
Find interesting problems to work on
This is the toughest step for a non academic. You literally have no starting point, no end point and no route connecting the two . As my friend Tejaswi (an MTech from IIT btw)confirmed in a private conversation, most Masters degree theses are suggested by the "guide" professors who oversee your work. Even for PhD students, who are expected to do a lot of unsupervised work and thinking, the input of these guides is a vital component of problem selection. Rightly so. You can save humungous amounts of time and effort if someone knowledgable points you in the right direction.
But what do you do if you are not an academic but still want to do research? You have no "guides". Talented researchers have tonnes of work already and are too busy to respond to whiney emails from wannabe researchers, so how is a poor ex-enterprise coder to find significant problems?
Here are some things that worked for me.
1. (Once you've got a grip on the fundamentals) Work through the core textbooks in your chosen field with a particular focus on working through (and coding up) the end of chapter exercises. Every field has a set of "core books" written by the some of the best researchers in that field. Find a suitable undergrad text and get to work. (For AI, Artifical Intelligence , a Modern Approach is an excellent choice. For compiler theory something like TAPL or EOPL would be a good starting point).If you are diligent, you'll eventually go beyond these books and start working through graduate texts. Once you've got a couple of grad level texts under your belt, you are ready to find problems.
2. Read contemporary papers.
  This is the KEY step to finding problems. Most (good) papers have a "work to be done" section which is a rich source of problems to attack. Most papers and conference proceedings are onlinethese days.All hail the Internet!
3. Discuss your research with folks who are doing similair things. In a city like Bangalore, this is fairly tough, because most of the software "hackers" are doing cheap labor, not research, but maybe your uncle is a professor in one of the IITs or your cousin works in the ISRO (indian Space Research Organization, for those who don't know). Talk to these people. In my experience, they are glad to work with someone who is genuinely interested. If all else fails (shamelessly) tap your friends who are working on their Masters or Phd or post-doc degrees.
4. Start a research notebook to record your thoughts and activities.
  I have found this practice VERY useful. As would a blog ( a private one is better imo) to record your daily progress(or lack of it). The habit of taking small steps daily (or biweekly or whatever) is very important.
5. The essence of "finding a problem" is to ask a question that (a) hasn't been asked before and/or (b) has no answer yet. research is all about finding answers to these questions. Put like that, it doesn't sound *so* hard does it? (yeah well it DOES sound hard .. :-P )
6. Be self directed.
  This is a KEY personality trait. In mosts corporate settings you always have someone telling you what to do, (by) when to do it and sometimes how to do it. This is similair to a Masters course. A PhD student has to be much more self directed , but he still has deadlines and guides and colleagues. Someone working alone, in his spare time has none of this. Unless you have iron discipline and can work steadily over long periods and tolerate boatloads of ambiguity, you might as well not start. Life is too short.
Once you find a significant problem you need to solve it. And then publish the results. I don't have any advice to give on these steps because I am still working on my first research problem.
However I am making steady progress. Some of the best people in the field have expressed interest in seeing how the results turn out. My research notebook is filling up nicely. I write code every day. So hopefully in about 3-6 months I'll have something to report. Watch this space.

Once you have some results you have to somehow publish them. I don't know how the academic publishing process works. I don't care. I am not trying for tenure. I just want to solve some problems and "advance the state of knowledge" . I guess I'll just publish my results on the web. I am not saying publishing and peer review are not important. I just don't know how a lone researcher can achieve this. No clue. Zip. Zilch. Nada.

So is it all doom and gloom for non academic folks who try to attack tough research problems? Not really. One weapon you have and most academics don't is your skill in programming. Most academic code sucks BIG TIME. Most of the code behind that brilliant thesis is an unmaintainable, horrible tangled mess (most academic papers don't have links to the underlying code for just this very reason). If you can use your computer as a power tool and write well designed, maintainable, tested, documented, industrial strength code, you will have an edge(a very very tiny one but an edge nonetheless) over folks who've never worked in the sofwtare industry. And if you have workd in a leadership osition in a commercial organization (think PM, Architect, Project Lead), your communication skills (both oral and written) are likely top notch. This should help tremendously.

And that's all I have. Hopefully some of that advice will help you to avoid some of the common pitfalls that beset the rogue researcher.

Asok's Own Country

2007-04-15T13:49:00.000+05:30

Recently, a friend (we'll call him M) invited me to his company's annual day party. First the CEO made a speech about how they (the folks who work at the company) were "creators of history and not just passengers". People clapped. Then the party proper started.

Well, sorta kinda. The Mistress of Ceremonies declared that "the bar is now open". One small problem though. You couldn't just walk up to the bar and get a beer. First you had to locate a "coupon distributor", get a "beer coupon" and then walk over to the bar and hand over the coupon and the bartender would hand you a beer. If you wanted another beer, you had to rejoin the line. (I am NOT kidding. This really happened. I was there, as were 200 or so others, so I have plenty of witnesses).

M, (by now thoroughly embarassed) joined the long line, and after inching forward for about 20 minutes, arrived at the head of a line and asked for "two coupons please". The person dispensing the precious scraps of paper. The Person Behind the Counter -PBC from now-, took a long and incredulous look and then said frostily "Only one coupon per head. sir".

M: "so, if a party of four want
 beers,all four have to line
 up and get their beers"
PBC: "That's right"
M: "ok... if they want another
 beer, they need to join
 the line again."
PBC: "That's right sir!"
M: "All four of them"
PBC (beaming) : "That's right, sir!
 Here is your beer coupon"
M(handing it back): "Do me a favor friend,
 Have a beer on me" 
(walks away).

Creators of History? :-)

Scribes and Programmers

2007-04-08T08:41:00.000+05:30

A long time ago, there weren't too many people who knew how to write. So if John Ancient wanted something written he would visit the local scribe and pay a fee. I am not sure if the scribes ever formed a guild but I wouldn't be too surprised.

These days, if Joe Modern wants a program written, he needs to hire a programmer (or a bunch of programmers) who'll translate his needs into the appropriate incantations.

Imagine a world where a majority of people know programming - perhaps not enough to write an operating system kernel but enough to write a basic Rails application - say a ToDoList application. (I did say "imagine"). How would programmers evolve?

I think they will become largely "post technical" - a term used, sometimes pejoratively, to describe developers who've moved away from code-centric jobs. The term is mostly applied to folks who go on to become managers, an idea which is looked down upon by some developers.

In a world where most folks could program, post technicalism would be the norm. A large chunk of present day programming jobs would simply disappear. You can't get a job these days if you "know all 26 letters of the English alphabet and can form grammatically correct sentences in English", which is the level at which a large majority of "professional" developers "program". Knowledge of writing is required but not sufficient for a paying job today. A doctor needs to know how to write but he also needs to know much much more. Musicians, biologists, teachers - all could do with a basic knowledge of programming. Perhaps basic programming skill should be taught just like reading and writing are. Then the idea of writing a quick script to generate a graph from a database wouldn't be some kind of exotic skill , but a basic competence expected from educated people.

Would this disappearance (or submergence below the awareness level) of programming skills be a bad thing? Not really. Technological progress has rendered many jobs obsolete over the centuries and humanity has coped very well.

Also, there would still be a need for people having extremely high levels of programming ability, just as there are professions that need extremely high levels of writing/communication ability - think copywriters or diplomats. The folks who focus on algorithms and networks and compilers and other 'deep programming' would probably survive just fine.

Of course the writing analogy can't be stretched too far. Documents are created by one person or a few people at most, over fairly short periods of time. Most significant programming projects involve large groups of people working on the same codebase for significant amounts of time, sometimes decades. The main challenges (as measured in terms of completion and impact of the project) are about people issues - how to hire them, how to retain them, how to make them work together.

And this organizational difficulty of managing large groups of people and getting them to build something coherent is the space in which, at present, many snake oil vendors (think "lean " or agile") thrive. The presence of snake oil is often an indicator of a fundamental and intractable problem. So even if a large number of people knew how to program, the organizational aspects will need hands on resolution. Thus the "project manager" type person would be alive and well. The VB/j2ee/dotNet folks are the ones who'd disappear.

But is the "manager dude" the only evolutionary path available to code jocks? I think not. For example, the world of science and engineering (modulo software "engineering") would still be open. Yes, there would need to be a lot of study - mathematics and statistics and the sciences and particular engineering domains - you won't be able to grab a book called "Java X.Y in 24 hours" and get a well paying job. People wouldn't be able to make money doing the equivalent of writing other people's letters for them. But I suspect there will be a lot of fascinating problems to solve, even in our imaginary world.

Even in a world where everyone codes, the "path of the suit" would still remain the Dark Side. You would still need to think long and hard and try to balance gain and loss before setting out on that path.

Paul Graham On Enterprise Software

2007-03-28T08:42:00.000+05:30

From his latest essay

"If you don't think you're smart enough to start a startup doing something technically difficult, just write enterprise software. Enterprise software companies aren't technology companies, they're sales companies, and sales depends mostly on effort."

I like the "just" in "just write enterprise software".

If Toyota outsourced its assembly line

2007-02-27T15:37:00.000+05:30

to "consultants" who charged x$/hour/person_deployed, would we have a Toyota Production System today?

Think about it. Imagine someone like Wipro or Infosys providing the manpower (cheap offshore manpower if you will) for Toyota's assembly line. What would the probability be of Toyota coming up with a world beating manufacturing/design approach?

Many "Lean Software" advocates draw very tenuous relationships between a very unique achievement in the manufacturing world, try to spin their own practices (ahem - did I hear someone say " agile" ?) and say (in effect) "By the power of Toyota, Taichi Ohno and Kanban, you should do [consultant's favorite methodology practice item]".

I recently read a book on implementing Lean Software, by one of the "luminaries" of the methodology sales world , in which every chapter had the following pattern.

A one paragraph example of how Cool Company [Toyota/Boeing/Google] etc did something cool [developed the 747, say].
One aspect of this successful effort (failed efforts by the same companies, often using substantially similair methodologies, are ignored) - say, cross functional teams- vaguely resonates with "agile" or "lean software" principles.
One or more teeny weeny "experience reports" from the author which sound like stories from the Brothers Grimm, full of archetypes and soundbite morals-to-take-home, of the calibre of "Don't talk to wolves on the way to Gramma's house"
a "summary" that states point blank that software development could do much better if the unenlightened masses would adopt these ideas

Most of these "agile/lean" consultants haven't written any significant software, nor managed any world changing software efforts. In direct contrast, the TPS "senseis" are often people who have worked for decades on extremely successful, innovative, *world class* products that they can point to ("see that Prius/Corolla/other prominent model, I was its Chief Designer").

Software seems to be the only area in which people with no history of creating significant software can go around teaching people who do write software how to do it.

If you try to pin these folks down on any specific practice or project they fall back on ever more woolly explanations. The most creative one I heard was that "Lean Software is about the principles of effective development not about practices (which admittedly are quite different for sw development vs manufacturing )".

That's ..errr..comforting? No one can question motherhood and apple pie statements, especially when they are explicitly divorced from concrete achievements or practices.

To get intelligent people to listen, a lean (or anyother buzzword) salesman should (a) deliver outstanding, world beating software, (b) do so consistently for a long period of time(how does 20 years sound?) and then talk to others about how he or she pulled off all these astounding projects. Most people who are outstandingly good at development or management often have better things to do than be "methodology consultants". Can you imagine Linus Torvalds becoming a "How to create succesful open source projects " consultant? Or Sergei Brin "consulting" on "How to create a succesful product company"?

People who can, do. It is only after "doing" for a very long time and consistently succeeding that they can pass on that knowledge if they choose to do so.

Having said all that, if anyone wants to make a lot of money (in Bangalore/India), the best thing to do these days is to get a "Certified Scrum Master" certification. A *lot* of dumb companies here are looking for such "masters" and are willing to pay good money. If you need the money, go for it I say. Be sure to get in (and out) before the fad fizzles.

All you have to do is pay up, sit through the seminar(s) and hey presto, you are now qualified to spread the Word. I get enquiries *every week* on whether I can reccomend someone for (or do some) "Agile Coaching". A "Certified Scrum Master" can make a killing in Bangalore today. As I said, you need to be fast. Lots of people have sensed the money and are lining up for the ScrumMaster certification.

But if you are on the other end of that pile of money, beware of consultants bearing buzzwords.

Is it possible to learn from Toyota's processes and apply them in software? Perhaps. But such an effort would be less about listening to conmen and more about doing lots of original thinking, reading (the original sources .. not "lean software" type blurbs) and a tonne of "doing" and reflection in *your* context.

And oh, don't believe anyone who charges you by the person hour or is trying to get you to outsource software development is sincere about "lean software". They aren't.

Tech Skills - An Investment Metaphor

2007-01-27T11:57:00.000+05:30

A snippet of a (yahoo messenger) conversation with a friend who thinks differently.

"... friend: i'm actually quite happy being a follower with technology... i'd rather build cool apps than hack frameworks. Its similar to my approach to investing... get on board a stock just as it makes new highs :-) and ride the wave. ..."

Viewing technical skill acquisition (including the choice of language/frameworks/tools) through a metaphor of investment of scarce resources (that would probably benefit from the use of a deliberate strategy to maximise returns) is illuminating.

The metaphor fits loosely, with plenty of holes in it. The "investment" made by a developer in a particular technology (and thus by implication in the underlying paradigm, community, toolset etc) is primarily the time he spends on mastering the technology, and secondarily the mental effort required to do so (Kernel hacking is much harder then Ruby on Rails). Mental effort is not strictly equal to time consumed. If you have 24 hours of free time you can use it to work through a RAILS tutorial or understand continuation passing interpreters. The latter requires significantly more mental effort/unit time.

Spending time, however, is very different from spending money. Money does not diminish when you don't actively spend it(ignoring the time value of money), but the time available to you diminishes at a constant rate till it is all gone whether you consciously spend it or not. With money, you can make a marginally suboptimal investment, watch it for awhile, then re extract most of the value and reinvest it in another instrument without too much effort, losing just the "time value" of the money. You can't decide to learn RoR for a year, then extract most of the time invested, and reinvest it in say Kernel Hacking. Time, once "invested" is gone forever.

Like most non professional investors in stocks and shares, developers mostly invest their time into particular technologies by default rather by deliberate strategy and many people find themselves having "6 years experience in C++ " or whatever, without knowing quite how they got there. So it is useful to look at the "correct" ways of investing in shares and try to extract any lessons to apply to tech skill acquisition.

My friend's technology strategy is "similar to .. get on board a stock just as it makes new highs .. and ride the wave." . In the world on Finance, such a strategy is called "technical investing".

When the objective of the analysis is to determine what stock to buy and at what price, there are two dominant methodologies.- value investing and technical investing.

from investopedia

Fundamental analysis is a method of evaluating securities by attempting to measure the intrinsic value of a stock.
Technical analysis is the evaluation of securities by means of studying statistics generated by market activity, such as past prices and volume. Technical analysts do not attempt to measure a security's intrinsic value but instead use stock charts to identify patterns and trends that may suggest what a stock will do in the future.

Technical investors try to guess how other people (collectively called "the market") think. The key skill is timing - the ability to jump into the market when it is about to "take off".

Value investing is about cold blooded analysis of the financials - the key skill is finding the " intrinsic value" so you can buy when intrinsic value >> price in the market . The (short term) market "trends" are of little consequence to the value investor - the only 'trend' they believe in is reversion to mean (where mean == intrinsic value).

Let us take a concrete (but extremely simplified) example. Suppose a company has a million dollars in annual profit, with a million shares in the market.

The value investor thinks "If I buy a single share in this company, I 'earn' an income of a millionth part of a million dollars(the profit/year), i.e one dollar/year. If I want a 10% annual ROI, I can buy this share at any price <= 10$"

Thus the "intrinsic value" of a share in the company depends on how much "ownership" of the company (and thus a share of its revenue and profits) that share represents.

The technical investor looks at the same stock through a different conceptual lens. He doesn't care what the present price or intrinsic value of the share is. It could be 10$, 100 $ or 1000$. All he cares about is whether the price of the stock is going to surge or slump. If he thinks it is going to move from 100 $ to 120 $ in the next couple of weeks, he'll buy the stock and if he thinks it is going to move in the reverse dierction (say from 100$ to 80 $), he'll sell it. (As with the value investing example, this is an extremely simplified example. Real life strategies are more complex).

To judge whether a stock price is about to move up or down, the technical investor uses historical data, often in the form of graphs/charts of price movements over time. He looks for patterns in the charts which predict an upswing or fall in price and uses these patterns to judge when to buy and sell.

Evaluating financial instruments is easier than evaluating technology choices because for the former ,what you invest and the returns you get are measured in the same unit (money). As mentioned earlier, developers invest time and mental effort in mastering particular technologies. Measuring "returns" is a bit more problematic, and could be "employability" or "chances of getting rich through code " or "interesting work" etc and often multidimensional, often being a compound function of these factors. Thus investing in a particular technology would mean trading time(the cost)for money earned/month or the "Interest Quotient" of the job etc (the returns).

My friend uses a technical investment strategy (waiting till a wave builds and riding a wave - and presumably jumping off an abating wave ) to acquiring technical skills. An alternative approach is to use a more value oriented strategy - ignore the 'trend' of a technology and evaluate its "intrinsic value" - the advantage (or returns) it gives on a particular (set of) projects, as contrasted with other options.

Thus I can decide to learn Ruby on Rails (say) either because it is a 'hot' technology (the technical investment approach) or because cold analysis indicates that for developing a certain type of web app, it delivers a speed multiple at the cost of slower speed, and less tool support and an estimated 3 or 4 weeks of learning time (the value approach).

As in stock investment, some people use a mix of strategies. There are approaches to investing other than "technical" and "fundamental". E.g the Capital Asset Pricing Model which focusses on measures of volatility in price.

While the metaphor is imperfect, I find it illuminating. Of course, the fallback approach is to learn whatever fascinates me, irrespective of intrinsic value or trends or volatility or whatever- an approach that has provided rich "dividends", in my experience anyway.

PS:-Much Thanks to Yogi and Manoj for excellent feedback on the initial draft.

Comics as Literature

2007-01-20T16:29:00.000+05:30

"Comics" bring to mind Mickey Mouse, Archie, Superman and other "read and throw" reading. But there is no reason why the presence of pictures need to limit the range or quality of the story. Stretching one's mind, it is even possible to imagine truly excellent stories conceived and distributed in comic format.

"V for Vendetta" was originally released as a comic. The film's the director made it a "chick flick" with V falling in love with Evie (gag!! Anyone who had read the comic had their heads explode at this "romantic" development in the film). Constantine's lush and rich multi-realm occult backstory was dumbed down into a monochromatic Christian good vs evil fairytale in the movie. Both stories are the worser for conversion into film.

If you'd like to experiment with comics for adults, try Neil Gaman's Sandman series or Alan Moore's Watchmen. The latter was the only comic ever to win a Hugo award or be featured in Time magazine's list of 100 best English novels. It is a really good book. Even within the standard comic book framework, read "The Killing Joke" or "Arkham Asylum - Living Hell" for examples of what a creative team can do within the constraints of established backstory. The former explores the dynamic of the Joker/Batman animus (The sequel of Batman Begins is rumored to be based on this comic , at least as far the Joker's characterization goes) and the latter explores what happens when a rogue investment banker attempts a "not guilty by reason of insanity" plea to avoid jail and lands in Arkham Asylum. Batman is seen in about 3 panels in the book. The rest of the story is set almost completely inside the walls of Arkham and in spite of a slightly uneven "summon the Devil" subthread is a good read.

PS : the one series you want to avoid is Alan moore's Promethea. This is a story full of feminist nonsense splattered over a half baked understanding of the Jewish Tree of Life, and even some (equally halfbaked) Indian mysticism!

Shining Darkly

2006-12-07T07:05:00.001+05:30

My friend Amit Rathore wrote an excellent article on transitioning from a developer to a manager.

Richard Hamming's famous speech has excellent advice on the (similar) manager vs scientist decision that many researchers struggle with.

" ...Question: Would you compare research and management?

Hamming: If you want to be a great researcher, you won't make it being president of the company. If you want to be president of the company, that's another thing. I'm not against being president of the company. I just don't want to be. I think Ian Ross does a good job as President of Bell Labs. I'm not against it; but you have to be clear on what you want. Furthermore, when you're young, you may have picked wanting to be a great scientist, but as you live longer, you may change your mind.

For instance, I went to my boss, Bode, one day and said, ``Why did you ever become department head? Why didn't you just be a good scientist?'' He said, ``Hamming, I had a vision of what mathematics should be in Bell Laboratories. And I saw if that vision was going to be realized, I had to make it happen; I had to be department head.''

When your vision of what you want to do is what you can do single-handedly, then you should pursue it. The day your vision, what you think needs to be done, is bigger than what you can do single-handedly, then you have to move toward management. And the bigger the vision is, the farther in management you have to go. If you have a vision of what the whole laboratory should be, or the whole Bell System, you have to get there to make it happen. You can't make it happen from the bottom very easily. It depends upon what goals and what desires you have. And as they change in life, you have to be prepared to change.

I chose to avoid management because I preferred to do what I could do single-handedly. But that's the choice that I made, and it is biased. Each person is entitled to their choice. Keep an open mind. But when you do choose a path, for heaven's sake be aware of what you have done and the choice you have made.

Don't try to do both sides.

Of course Hamming assumes that people become managers because "your vision, what you think needs to be done, is bigger than what you can do single-handedly". In the great Outsourced Software Wasteland, most people choose to become managers because it is a better shield for mediocrity than being a developer and (mostly) you can make significantly more money. Nothing wrong with that. You pays your ~~money~~ life and you takes your choice.

Even worse is the developer who really likes being a developer and is or can be very good at it, but chooses to become a mediocre, unhappy manager because "that is the only way to make more money". These people deserve pity, but no sympathy.

Thinking as a learnable skill

2006-11-14T21:25:00.000+05:30

If I remember correctly, it was in one of Edward de Bono's books that I first encountered the idea that thinking might actually be a skill, and thus learnable (and improvable by practice).

Over the years, this belief has been reinforced and I was forced to pull it all together yesterday when a friend asked me "How can I improve the quality of my thinking"?

This is my answer, fwiw.

First learn critical thinking. Critical Thinking teaches you to evaluate an argument and separate chaff from grain. Suppose someone were to state, for e.g "Agile is cool and we'd like to adopt it in the next project", how do you react? To learn critical thinking one could go through a course in logic, learn all the different logical fallacies and then try to apply them to real life conversations and claims. But Critical Thinking - An Introduction offers a gentler path. Just make sure to skip most of Chapter 1. Just read Section 11.4, 1.2 and 1.3 an move on to Chapter 2. I've never seen a better book that teaches one how to cut through the layers of BS and get to the core of an argument. The Craft Of Argument is a good book as well, though oriented more towards writing good papers and articles.

Second, learn Lateral Thinking. Dr. de Bono,the inventor of the method, has written many many useless books and a couple of brilliant ones. The latter contain some powerful ideas.

Dr de Bono's central contribution is the notion of a "thinking tool", analogous to tools like a hammer or a screwdriver - each tool having a context of use and a particular effect. The books to get are Lateral Thinking For Management(non management folks can ignore the "for management". It just means that he uses management situations to illustrate various thinking tools. Nothing prevents you from using them in your own domains) and Serious Creativity. Both are chock full of systematic practical lessons about how to think in different situations.

Third, I'd suggest a brilliant thinking tool - Thinking Gray - from A Contrarian's Guide To Leadership by Steven Sample. The book deals with a lot more than thinking, and in addition is very hard to find (Thank You Learning Journey, for getting me a copy) and so I'll briefly explain "Thinking Gray".

In essence, Thinking Gray (TG from now) asks you not to make a decision on something, or choose a side in an argument till you absolutely have to. Sounds simple doesn't it ? However, the human mind has tendency to converge to a decision and classify data into binary categories (good/bad, black/white, friend/foe) and take sides in an argument. For e.g everyone has an opinion about (say) the American invasion of Iraq. Some may think it a "good" thing and others may decide that the war is a "bad" idea. The TG practitioner, while listening to all the data, refuses to take a binary stand till it becomes absolutely imperative that he does so.

This refusal to decide lets you avoid three dangerous tendencies - (1) the tendency to filter incoming data to support a pre decided conclusion (2) the tendency to vacillate between two conclusions, depending on who you last spoke to (and how persuasive he was) and (3) most dangerous, the tendency to slant your beliefs towards what you think people around you believe.

Steven Sample precisely distinguishes TG from skepticism. The skeptic has two mental "buckets" - one labelled "I believe" and another labelled "I don't believe". The skeptic's policy is that nothing goes into the "I believe" bucket without proof or a logical, convincing argument. Everything, by default goes into the "I don't believe" bucket and stays there till logic or proof moves it into the "belief" b. The TG practitioner on the other hand, has no "buckets".

One thing the TG practitioner has to be careful of is that other people may get confused about your "true" beliefs and intentions and may think that you agree with what they say because you seem to understand what they say and ask clarifying questions and so on.

Thinking Gray can be enhanced by projecting the multiple conclusions forward into the future (thus providing a rich tapestry of possibilities) and backward into the past(allowing multiple interpretations) . Combined with Paul Graham's "running upstairs" or "staying upwind" concept, TG allows you to navigate an uncertain future in a stress free fashion and maximize productivity in the present. More on this later.

To generate ideas, in other words, to be creative, de Bono's thinking tools work very well. I recently came across an approach called TRIZ. The basic idea seems to be worth investigating, but during 2003/2004, TRIZ ( and the associated concept ARIZ ) became the fad of the day just as "agile" and "lean" seem to be today's. Thus there is a lot of nonsense written about them as various authors bought out books and a lot of consultants were selling "ARIZ enablement" and hiring themselves out as "ARIZ coaches". As a result, there is a lot of worthless literature out there and some effort has to be extended to dig out the core of ARIZ and TRIZ. As soon as I get some free time(a precious commodity these days), I will look into this.

To conclude, all these methods treat thinking as a learnable skill and the one problem with this model is that to get good at thinking, as with finding the route to Carnegie Hall, practice is the only way. And practicing thinking is an odd concept for a lot of people.

Karma Capitalism?

2006-11-12T12:00:00.000+05:30

Sometimes, I toy with the idea of getting an Ivy League MBA. And then I come across something like this (via Evolving Excellence).

...For the members of the Young Presidents’ Association, meeting in New Jersey, this was no ordinary leadership seminar. They were being imbued with the values of the Hindu philosophy of Vedanta, by its most venerable proponent, Swami Parthasarathy. ...

On the syllabus at Harvard, Kellogg, Wharton and Ross business and management schools is the Bhagavad Gita, one of Hinduism’s most sacred texts. ....

He began studying the Bhagavad Gita, and has spent the past 50 years building a multimillion pound empire through explaining its practical benefits to wealthy corporations and executives.

He has recently returned to India from America where — in addition to the Young Presidents’ Organisation — he lectured students at Wharton Business School and executives at Lehman Brothers in Manhattan. His tours are booked well beyond next year, and will include Australia, New Zealand, Singapore and Malaysia.

While traditional business teaching has used the language of war and conquest, Parthasarathy uses the Bhagavad Gita to urge his students to turn inwards, to develop what he calls the intellect, by which he means their own personal understanding of themselves and the world, and to develop their “concentration, consistency and co- operation”.

The technical folk do have their religious fads, but compared to such arrant nonsense, they are the very epitome of logic and reason.

If the Ivy League schools and Lehmann Brothers and other corporations believe this nonsense enough to enrich "guru" (gag!!) Parthasarthy by many millions of pounds, why should I get an MBA? Hmmm on the other hand, that is precisely the reason why I should get one?

If anyone reading this blog has an Ivy League MBA and/or works on the Street, could you please explain how such rackets work? I can't make any sense out of this. Any help appreciated.

Cracking the GRE

2006-10-27T14:40:00.000+05:30

Update: Analytical Score updated.

I wrote the GRE a few days ago.

Scores -

quant- 790/800
verbal 800/800 (heh!)
Analytical Writing - ~~awaited (I am not too worried about this)~~.- 6.0/6.0

I am going through the annual "Should I apply for this year or next year, MS or PhD, Applied mAths or CompSci? " dance.

Choices .. choices ..

Listening to Steve Yegge (and skirmishing with Obie)

2006-10-04T19:14:00.000+05:30

Steve Yegge's blog post titled "Good Agile, Bad Agile" seems to have ignited one of those periodic feeding frenzies in the blogosphere. The comparison he makes between Agile and Scientology seems to have touched a nerve in many "agilists". The comments on Steve's post reveal a good deal of defensiveness, google envy and the occasional reasoned response.

Summarizing Steve's "anti agile" points (from a very long post), he says

The Chrysler Project, which was the exemplar for agile projects failed badly.
Mandatory 100% pairing sucks bigtime.
Agile isn't scientific. (at best, any proof for "improved productivity" is anecdotal).
There are a lot of flaky "agilists" trying to be "coaches" and sell books and seminars.
These "seminar agilists" create an artificially polarized world where any process that isn't Agile(TM) is "waterfall" or "cowboy".
Agile is very date driven, ignoring the human variations in productivity.
If anyone puts forward an alternative process , the "defenders" of agile use one of two responses to oppose it

"all the good stuff he described is really Agile"
"all the bad stuff he described is the fault of the team's execution of the process"

Now all of this is true and valid, thoughtful criticism. And reasonable people can disagree and debate rationally. So why do people react so violently?

I believe this massive reaction came about because Steve called Google's development process "Good Agile" and what we know as "standard Agile", "Bad Agile".

More than the "truth" of these assertions, what is highly amusing is the outraged reactions of people who make a living selling "Agile solutions", either as individuals or as consulting companies. Like Islamist radicals opposing the "Islam is a violent religion" meme by burning embassies, issuing death threats and destroying public property, anyone who calls Steve names and say things like "but we have waaay dumber developers than Google so we have to make do with Agile" really undermine their own cause.

I was very much on the sidelines (I love watching a clueless mob tear itself apart as much as anyone else) till a blog entry by (ThoughtWorker) Obie Fernandez claimed that the "real benefit" of Pairing was

"... pair programming is one of the only effective ways that a lot of us have ever witnessed keeping average developers from pissing away 95% of their productivity engaging in non-work such as reading and writing blogs, instant messaging, personal email, shopping online and otherwise wasting time on bullshit. When you're pairing, you simply HAVE to work all day...."

and "As far as I'm concerned, all the other benefits you get from pairing like continuous review and knowledge-sharing are just gravy."

WTF?!!! And when I commented that "If the key benefit of pairing is that it keeps otherwise lazy developers' nose to their keyboards, the company needs to look more carefully at the type of people it hires than at its development methodology." Obie said I was "sniping" and that the comment upset him.

Whoops! I didn't mean to "snipe" or otherwise upset anyone, least of all Obie. I was just saying that "you'll waste a lot of time unless you are pairing" is an incredibly feeble argument for the practice.

In my own experience, the benefits of Pairing (which does have benefits and drawbacks, like any other development practice) come from what Obie says is "gravy" - continous review and knowledge sharing.

In his latest blog entry Obie makes an even more exotic claim - apparently Attention Deficit Disorder (ADD)is rampant in our workplaces and most of us suffer from it without knowing we do. And "pairing" apparently serves as an antidote. Whoa!!

"It's not about laziness -- it's about the predominance of ATTENTION DEFICIT DISORDER, whether diagnosed or not, acknowledged or not, ADD is affecting almost all of us, probably as a result of the tons of distractions that we are subjected to on a constant basis. Working on something with a partner works miracles for postponing non-essential distractions such as IMs and checking emails and blogs. Don't suffer from ADD at all? Well, good for you. Your type of people will be in the minority soon if you aren't already."

yes, Obie, you are right - partially. I don't suffer from ADD. :-) .That's the part you got right. Thanks for the concern! I can focus for however long I want on my work without being compelled to "check email or blogs".

"...Ohhh the temptation ... all those mails and blog entries ... They are CALLING to me .... can ... not ... resist ... I still have to another 10 minutes to finish my "daily Solid Eight Hours"... All these distractions .. Help meeeeee!! ...[voice fades]..."

Yeah right!

In my opinion, anyone who is seduced by these apparent "distractions" to the point where you can't get work done unless a "pairing partner" keeps you on your toes is in the wrong profession.

And oh, according to wikipedia , "Attention-deficit/hyperactivity disorder (ADHD) (sometimes referred to as ADD) is thought to be a neurological disorder, always present from childhood, which manifests itself with symptoms such as hyperactivity, forgetfulness, poor impulse control, and distractibility.[2] In neurological pathology, ADHD is currently considered to be a chronic syndrome for which no medical cure is available. Both children and adults may present with ADHD, which is believed to affect between 3-5% of the population."

So, anyway, I (as in "your type of people" :-) )am not in "the minority" after all. Most people in the world do NOT suffer from ADD. Thank God!

more.

"ADHD is today generally regarded to be a non-curable neurological disorder for which, however, a wide range of effective treatments are available. Methods of treatment usually involve some combination of medication, psychotherapy, and other techniques".

Hmm we should probably add "pairing" to that list? :-) "The Agile Psychotherapist"... sounds like a killer book title.

Yeah Ok . Enough "sniping" at Obie.

Ok folks, listen up. Agile/XP is no panacea. It is just what worked for Kent Beck in certain contexts. The fact that there are a whole bunch of "agile consultants" both individuals and companies, out to part you from your money in exchange for the arcana of "True Agile" does not absolve you for thinking for yourselves.

lookup "Scrum" for an extraordinarily cynical MultiLevel Marketing scam. "Certified ScrumMaster" indeed! This is where the comparisons with Scientology gain traction.

Repeat.There is no silver bullet. The nearest to one is "hire good people and let them work the best way they can" . What they come up with will often look very different from "True Agile". If that is "Googlism" then so be it. Nuff Said.

And Oh, Steve, keep writing!

Spreadsheets and Self Sabotage A Case Study

2006-09-17T14:14:00.000+05:30

This is a true story. Names have been changed to protect the innocent (and the guilty).

Once upon a time,long long ago, in a faraway land, there was this company, a typical enterprisey company we'll call BankCo (in reality the company doesn't work in banking, the domain is similar but different, and I am just obfuscating things as explained above). BankCo was running all its "enterprise" operations off a set of excel sheets and at some point decided to redo it "properly". The folks at BankCo were essentially businessmen, not programmers so they decided to get some consultants at the cutting edge of technology. Enter AgileCo. AgileCo decided to rewrite the business logic as a 3 tired OO app using all the latest blub technologies.

The result? the BHS project (in AgileCo most projects have three letter acronyms) has frustrated developers, grumbling clients and absolutely yucky code.

Now there were a lot of things the team did wrong. Fundamentally the project was doomed the moment the team was "volunteered" for such an aggressive schedule (committed to the client by the sales manager, "agile planning" notwithstanding) but the thinking might have been that "hey we just have a bunch of spreadsheets, how hard can it be to convert them into a 3 tiered Blub app" . The (non intuitive) correct answer is "potentially very very hard".

Spreadsheets are very subtle beasts. They just look very simple. There are these rows and columns of cells and even non programmers can quickly create useful "programs" so it should be fairly easy to translate a set of spreadsheets into OO programs right?

Wrong.

For one, a spreadsheet is essentially a functional program.[warning - pdf]

[Excerpt]

"It may seem odd to describe a spreadsheet as a programming language. Indeed, one of the great merits of spreadsheets is that users need not think of themselves as doing “programming”, let alone functional programming — rather, they simply “write formulae” or “build a model”. However, one can imagine printing the cells of a spreadsheet in textual form, like this:

A1 = 3 A2 = A1-32 A3 = A2 * 5/9

and then it plainly is a (functional) program. Thought of as a programming language, though, a spreadsheet is a very strange one. In particular, it is completely ﬂat: there are no functions apart from the built-in ones(*). Instead, the entire program is a single ﬂat collection of equations of the form “variable = formula”. Amazingly, end users nevertheless use spreadsheets to build extremely elaborate models, with many thousands of cells and formulae."

Simon goes on to state how he and his colleagues worked on removing the "procedural abstraction barrier" in the functional language embodied by a typical spreadsheet. (There are folks working on integrating more powerful functional languages and other paradigms into spread sheets. The key point is that a spreadsheet is essentially functional. Besides, a spreadsheet propogates changes in values and formulae automatically, unlike imperative and oo languages (e.g if the cell A1 is set to 5 instead of 3 in the above sample, A2 automatically becomes -27).

Besides value propagation , the 2 dimensional spatial relationships between spreadsheet cells makes it unlike most programming interfaces in existence today.

" One of the most successful end-user programming techniques ever invented, the spreadsheet, uses what is essentially a functional model of programming. Each cell in a spreadsheet contains a functional expression that specifies the value for the cell, based on values in other cells. There are no imperative instructions, at least in the basic, original spreadsheet model. In a sense each spreadsheet cell pulls in the outside values it needs to compute its own value, as opposed to imperative systems where a central agent pushes values into cells. In some sense each cell may be thought of as an agent that monitors its depended-upon cells and updates itself when it needs to. As Nardi (1993)(Nardi 1993) points out, the control constructs of imperative languages are one of the most difficult things for users to grasp. Spreadsheets eliminate this barrier to end-user programming by dispensing with the need for control constructs, replacing them with functional constructs. " [ Link

Thus, when you try to convert the functionality of a complex set of spreadsheets to a blub-dot-net program, you are, in essence translating from a functional language with value propagation to a very different paradigm (OO in this case) and also adding things like persistence and multi user interaction which a normal spreadsheet does not have to deal with. This project needs competent developers who have enough resources (including time) to do a proper job,and are expert in both functional and object oriented programming with a keen awareness of the chasm between the two paradigms and a knowledge of how to bridge it. You wouldn't have much use for a German to English translator who didn't know any German, would you?

Besides this technical misstep there are a host of others that make AgileCo's BHS project an interesting case study.

The first is the premature time commitment to clients, bypassing the essence of AgileCo's favorite planning methodology. This is the killer mistake. The second is an arbitrary team structure with unclear roles and shaky team buy in. The third factor worth discussing is how the dominant methodology at AgileCo discourages deep thought by developers and encourages them to grab a "User Story" and hit the keyboard as soon as possible. This eliminates the chances of success for finding a technical approach that attacks the core of the problem. The fourth is how a totally unnecessary onshore/offshore team split adds dysfunctional management, unnecessary paperwork and communication overhead. The fifth and most crucial insight is about how dumb management can totally screw up a project's chances of success beyond repair.

Postscript:- The team is now on deathmarch mode and works on Saturdays and some Sundays. Of course the managers don't work on weekends. The brighter developers are looking for other jobs.

(*) Most spreadsheets have some kind of extension mechanism. In Excel, for e.g. you can use VB script to write your own extensions. But this means (a) and end user has to learn a programming language and (b) subtle errors can creep in when you mix imperative and functional code. Debugging spreadsheets is a nightmare at the best of times.

Rules For Failure #1 Create a group

2006-09-09T16:04:00.000+05:30

I've been notcing that people who never get anything done seem to follow a few (anti-)patterns when they start off their projects. The most common one says "To get something done, first create a committee".

The key insight is that people may not think they are creating a committee; they may think that they are "discussing" their ideas with friends. But what happens is that there is a lot of discussion and no action. If there is some action, it tapers off soon enough.

This can be seen in various attempts to "learn design patterns" or "evangelize agile" or whatever. The dynamics of forming a group give an initial emotional high and a short lived illusion of progress. In the end, the effort dies down and it is time for the next fad.

With all the emphasis on "teamwork", not enough people can summon up the gumption to start working on something by themselves and make some progress before talking about it, or trying to get other people involved.

Consciously reversing this pattern seems to help. To master a programming language or framework , for e.g, just shut up and code. When the coding is finished release the code and do something else. Doing that long enough, you'll find you attract like minded people.

So these days, when people say things like "I am very interested in learning mathematics" my counter question is "so how many problems/proofs did you work through today"? Nothing else is important. Replace "working through problems/proofs" with the appropriate context dependent measures of progress.

Don't yap. Work.

W[ear] Cloak Of Invisibility

2006-08-03T09:16:00.000+05:30

The compiler project has kicked off and I have zero free time. I am spending all my time coding and reading.So .... I am totally unavailable. I am off the net and the phones are off. There is no way you can reach me. I am travelling extensively as well (long story). Mail will (eventually) be answered.

End of bulletin.

Rajesh Babu goes (quietly) ballistic

2006-07-30T06:25:00.000+05:30

Quietly Ballistic? !!! Yes! Read On!

A self proclaimed "rails evangelist" went to a conference where someone was talking about seaside and proceeded to make snarky comments from the back row about how seaside has a lot of stuff built into it. Anyone see the irony of a rails evangelist making fun of seaside for having a lot of features? I did.I thought of writing a blog entry about this but then decided that the resulting bad blood was probably not worth it.

Rajesh , being a much nicer person than I am reacted to the situation much more effectively (and creatively). He created a "photo cartoon". (Go on, take a look).

Duck Typing and "cool" ducks talking about static type systems being ineffective!! ROTFL!

Lesson Learned: Humor can be more effective than huge blog posts and lots of logic. And way more concise.

Hey Rajesh, what tool did you use to create that cartoon? I guess some kind of apple osx magic? I guess it should be easy to whip up some kind of OS tool to do the same.

Another of Rajesh's 'cartoons', on the "DSL is cool ! Oh My GOD!" phenomenon.

Indo Pak War - A Slightly Different Perspective

2006-07-29T11:21:00.000+05:30

Read this and this. Whether one agrees with the author's (admittedly zany) ideas, these articles are a good counterweight to the hyperbole laden "patriotic" articles often found in the Indian Press.

But Martin, Enterprise Software IS Boring

2006-07-29T08:41:00.000+05:30

Martin Fowler writes about Customer Affinity, a factor he believes distinguishes a good enterprise developer from a bad one. So far,so good.

He says "I've often heard it said that enterprise software is boring, just shuffling data around, that people of talent will do "real" software that requires fancy algorithms, hardware hacks, or plenty of math."

This is almost exactly what I say.(Just remove the quotes around "real" ;-) ).

Martin goes on to disagree with this idea (which is fine) but then he says "I feel that this usually happens due to a lack of customer affinity." And this is where I disagree.

People who work on things like compilers and hardware hacks and "tough" algorithmic problems can have customers, just as the enterprise folks do. I know I have. The compiler I am writing now has a customer. The massively parallel neural network classifier framework I wrote a couple of years ago had a customer. The telecom fraud detection classifier cluster I worked on last year (along with some other talented programmers) is being used by a company which is very business (and customer) oriented. So the idea that a "customer" (and thus customer affinity) is restricted only to those folks who write database backed web apps is simply not true.

Math/algorithms/hardware hacks and the presence or absence of a "customer" (and thus, customer affinity) are orthogonal (you know, two axes at 90 degrees to each other) issues.

With the customer issue out of the way, let us examine the core issue - the notion that talented developers would prefer to do stuff like compilers and algorithms instead of a business app (by which I mean something like automating the loan disbursal processes of a bank).

The best way to get a sense of the truth is to examine the (desired) flow of people in both directions. I know dozens of people who are very very good at writing business software who yearn wistfully for a job doing "plumbing" like compilers and tcp/ip stacks, but I've never yet seen someone who is very very good at writing a compiler or operating system (and can make money doing so) desperately trying to get back to the world of banking software. A programmer might code enterprise apps for money, but at night, at home, he'll still hack on a compiler.

It's kind of a "Berlin Wall" effect. In the days of the cold war, to cut through the propoganda of whether Communism was better than Capitalism , all you had to do was to observe in which direction people were trying to breach the Berlin Wall. Hundreds of people risked their lives to cross from the East to the West but practically no one went the other way. Of course this could just mean that the folks didn't appreciate the virtues of people's power and the harangues of the comissars but I somehow doubt it.

To see whether living in India is better than living In Bangladesh (or whether living in the United States is better than living in India) look at who is trying to cross the barriers and in which direction.

Economic Theory suggests another way of finding an answer. Suppose you were offered say 10,000$ a month for say 2 years, to develop the best business app you can imagine and say, 8000 $ a month, for the same two years to work on the best systems/research project you can imagine. Which one would you choose? Now invert the payment for each type of work. Change the amounts till you can't decide one way or another and your preference for either job is equal. Finding this point of indifference (actually it is an "indifference curve") will teach you about your preference for "enterprise over systems". I believe that while individuals will choose many possible points as their "points of indifference" on that curve, a majority of the most talented debvelopers would prefer a lower pay for a good systems/research project in preference to an enterprise project, no matter how interesting. And to most developers, an enterprise project becomes interesting to the degree it needs "tough programming" skills (massively scalable databases say.. there is a good reason Amazon and Google emphasize algorithms , maths and other "plumbing" skills in their interviews and an equally good reason why a Wipro or a TCS doesn't bother testing for these skills).

The widespread (though not universal) preference of the very best software developers for "plumbing" over "enterprise" holds even inside Thoughtworks (where Martin works and where I once worked). I know dozens of Thoughtworkers who'd prefer to work in "core tech". While I was working for TW, the CEO,Roy Singham, periodically raised the idea of venturing into embedded and other non enterprise software and a large majority of developers (which invariably included most of the best devs) wanted this to happen. Sadly, this never actually came to pass.

A good number of thoughtworkers do business software to put bread on the table, or while working towards being good enough to write "plumbing", but in their heart, they yearn to hack a kernel or program a robot. (Another group of people dream about starting their own web appp companies). For most of these folks tomorrow never quite arrives, but some of the most "businessy" developers in Thoughtworks dream of writing game engines one day or learning deep math vodoo or earning an MS or PhD. And good people have left TW for all these reasons. And if this is the situation at arguably the best enterprise app development company (at least in the "consultants" subspace) one can imagine the situation in "lesser" companies.

The programmer who has the ability to develop business software and "plumbing" software and chooses to do business app development is so rare as to be almost non existent. Of course, most business app dev types are in no condition to even contemplate writing "hard stuff" but that is a topic for another day.

Paul Graham (admittedly a biased source) says (emphasis mine),

"It's pretty easy to say what kinds of problems are not interesting: those where instead of solving a few big, clear, problems, you have to solve a lot of nasty little ones....Another is when you have to customize something for an individual client's complex and ill-defined needs. To hackers these kinds of projects are the death of a thousand cuts.

The distinguishing feature of nasty little problems is that you don't learn anything from them. Writing a compiler is interesting because it teaches you what a compiler is. But writing an interface to a buggy piece of software doesn't teach you anything, because the bugs are random. [3] So it's not just fastidiousness that makes good hackers avoid nasty little problems. It's more a question of self-preservation. Working on nasty little problems makes you stupid. Good hackers avoid it for the same reason models avoid cheeseburgers."

Strangely enough, I heard Martin talk about the same concept (from a more positive view point, of course) when he last spoke in Bangalore. He explained how much of the complexity of business software is essentially arbitrary - how an arcane union regulation about paying overtime for one man going bear hunting on Thanksgiving can wreck the beauty of a software architecture. Martin said that this is what is fascinating about business software. But he and Paul essentially agree about business software being about arbitrariness. It is a strange sense of aesthetics that finds beauty in arbitrariness, but who am I to say it isn't valid?

Modulo these ideas, I agree with a lot of what Martin says in his blog post. The most significant sentence is

"The real intellectual challenge of business software is figuring out where what the real contribution of software can be to a business. You need both good technical and business knowledge to find that."

This is so true it needs repeating. And I have said this before.To write good banking sofwtare, for e.g. you need a deep knowledge of banking AND (say) the j2ee stack. Unfortunately the "projects and consultants" part of business software development doesn't quite work this way. I will go out on a limb here and say that software consulting companies *in general* (with the rare honorable exception blah blah ) have "business analysts" who are not quite good enough to be managers in the enterprises they consult for and "developers" who are not quite good enough to be "hackers" (in the Paul Graham sense).

Of course factoring in "outsourcing" makes the picture worse. By the time a typical business app project comes to India, most, if not all of the vital decisions have been made and the project moves offshore only to take advantage of low cost programmers, no matter what the company propoganda says about "worldwide talent", so at least in India this kind of "figuring out" is almost non existent and is replaced by an endless grind churning out jsp pages or database tables or whatever. It is hard to figure out the contribution of your software to people and businesses located half a world away, no matter how many "distributed agilists" or "offshore business analysts" you throw into the equation.

This is true for non businessy software also,though to a much lesser degree. The kind of work outsourced to India is still the non essential "grind" type (though often way less boring than their "enterprise" equivalents). The core of Oracle's database software for .e.g. is not designed or implemented in India. Neither is the core of Yahoo's multiple software offerings - the maintenance is, development is not. And contary to what many Indians say, this is NOT about the "greedy white man's" exploiting the cheap brown skins. You need to do hard things and prove yourself before you are taken seriously. Otherwise people WILL see you as cheap drone workers. That's just the way of the world).

And I say this as someone who is deeply interested in business. T'is not that I loved business less but I loved programming more :-).I'd rather read code than abusiness magazine but I prefer a business magazine to say, fashion news. I actually LIKE reading about new business models. I read every issue of The Harvard Business Review (thanks Mack) and have dozens of businessy books on my book shelf along with all the technical books. (I gave away all (300 + books) of my J2EE/dotNet/agile/enterpise dev type book collection , but that is a different story). I am working through books on finance and investing and logistics. I have friends who did their MBA in Corporate Strategy from Dartmouth or are studying in Harvard (or plan to do so). I have friends who work for McKinsey, and Bain and Co, and Booz and Allen, and Lehman Brothers. If I were 10 years younger (and thus had a few extra years to live), I'd probably do an MBA myself. And of course I have spent 10 years (too long, oh, too long) doing "enterprise" software. In my experience (which could turn out to be atypical),the "plumbing" software is way tougher and waaaaay more intellectually meaningful than enterprise software, even of the hardest kind. I might enjoy an MBA, but I will NEVER work in the outsourced business app development industry again even if I have to starve on the streets of Bangalore. I'll kill myself first. So, yeah I am prejudiced. :-). Take everything I say with appropriate dosages of salt.

And just so I am clear, I do NOT think Martin is deliberately obfuscating issues. I think Martin is the rare individual who actually chooses enterprise software over systems software and is lucky enough to consistently find himself working with the best minds and in a position to figure out the business impact of the code he writes. This is very different from the position of the average "coding body", especially when he is selected more for being cheap than because he knows anything useful ("worldwide talent", remember :-) ).

Btw, Martin's use of the word "plumbing" is not used in a derogatory sense (though I've seen some pseudo "hackers" take it that way, especially since the word makes a not-so-occasional appearnce in many of his speeches) but more in the sense of "infrastructure". If you are such a "hacker" offended by the use of the term, substitute "infrastructure" for "plumbing" in his article. That takes the blue collar imagery and sting out of that particular word, fwiw.

I think Martin has very many valuable things to say. I just think he is slightly off base in this blog post and so I respectfully disagree with some of his conclusions.

But then again, it could all be me being crazy. Maybe enterprise software is really as fascinating as systems software, and writing a banking or leasing or insurance app really as interesting as writing a compiler or operating system.(and Unicorns exist , and pigs can fly).

Hmmmm. Maybe. Meanwhile I'm halfway under the wall and have to come up on the other side before dawn. Back to digging.

[Politics] Tip Of The Hat To Israel

2006-07-26T20:56:00.000+05:30

I like to think I am "centrist" when it comes to politics and that the world has shades of grey and is not all black and white.

However, in the present Israel/Lebanon-Hezbollah conflict, I sympathize with Israel.

The key characteristic of a nation is that it has (or should have) a monopoly of violence within its borders. As far as I can make out, the Israeli withdrawal from Lebanon was conditional on the *state* of Lebanon reigning in the Hezbollah.

If the Lebanese turn over a chunk of their territory to murdering fanatics trying to impose an archaic religious code on the rest of the world, they can't cry foul when people who lose patience with Allah's foot soldiers invade that territory. Either they control the territory, in which case they should be in control of their borders, not an Islamic militia. Alternatively they don't control it, in which case they have no business screaming about Israeli agression or invasion.

The Israelis, could in an ideal world, have shown a more nuanced approach, but I guess their patience has its limits.

this blog entry says it best.

Missing Comments Fixed

2006-07-22T08:03:00.000+05:30

For some strange reason, some comments did not make it through to my mail account and were piling up in the blogger bin. Some of them were excellent (Mujib , S and some "anonymii" ) and have been approved while a few (all anonymous :-D) along the lines of "I hope you burn in hell, Pagan Devil Worshipper" "U R suXorz" etc have been deleted.

The latter are particularly interesting. I can't imagine someone having a life so empty that they first read a blog they don't like and then laboriously compose a "hate comment" and post it with the full knowledge that it will be deleted with a single mouseclick. Oh well, it takes all kinds to make a world, I guess! I get a laugh out of these outbursts of venom, so please keep sending them.

Apologies to those whose comments were in limbo. As to the person who wanted the comments to "popup" rather than be listed in the permalink page, sorry about that, but *I* hate popups. Still if enough readers want popups, I'll re consider the decision.

That ends the administrivia announcement. Have a nice day.

Status Update aka Thank You for the job offers

2006-07-21T18:10:00.000+05:30

In my last post, I wrote

"Aaargh! someone give me a job in the USA (or even better, an admission into a good PhD program) please! I give up on this country!"

A few readers (4 in all) were kind enough to write in with job offers in the USA! Nice! Even better, no one offered me an "enterprise" job. Two were in AI (related) companies and two in compilers.One of those jobs is veeeeery tempting.

Must-resist-temptation :-)

Thank you all. I've written to all of you individually.

That reminds me.Here is a long overdue status update.

I will be working on the "compiler project" (this needs a good name) till about the middle of November.

Then it looks like I will be working with KD building some stuff for his startup. KD and I think very differently so expect some fireworks. He will probably throw me out of the company the very first day :-)

My decision to leave India in 2007 is unchanged. I plan to try for a PhD rather than a job. That way I don't have to play the "H1 game" and sell myself into wage slavery immediately:-). Having said that, one of the job offers had a "higher education allowance" perk.

Hmmmmmm.

Update: The "blogspot ban" is partially removed. The "dangerous" sites are still blocked. Most of them are terrible and not worth reading (Of course I read them all. The best way to get people to do something is to forbid them to do so!). The only (somewhat) interesting nugget was Nathuram Godse (Gandhi's assassin)'s speech from the dock. If the Intelligence Bureau thinks these sites are going to "incite violence" they need their heads examined.

Indians can't see this blog entry

2006-07-17T17:10:00.000+05:30

because some f##$$$%%^ ing bureaucrat in the Indian Government has mandated that ISPs block access to blogspot.

If I wanted to live under a dictatorial government that condones this crap. I'd go and live in Iran. Or North Korea.

The bureaucrats responsible (and the weaselly ISP bosses who go along) should be lined up against the nearest wall and shot! I don't pay taxes to have my freedoms taken away by you stupid bastards.

Blogger.com is still accessible so I can post stuff using the managements screens [roll eyes] but I can't read any blogs on blogpsot through my browser.

What's next? Book Burning?

Aaargh! someone give me a job in the USA (or even better, an admission into a good PhD program) please! I give up on this country!

I am fed up with the arbitrariness of life in India. For 10 years I refused to go or work abroad because I wanted to live and work in India. But I'll be damned if I support a systems that actively pits nations people against themselves (with caste based reservation - read discrimination - poised to entrench itself in the private industry) and does crazy things in the name of "national security" (like block blogspot). On one side we have the ruling Indian National Congress plus their traitorous Communist Allies and on the other we have religious fanatics led by the Bharatiya Janata Party, who are in the opposition.

I'd rather live as a janitor in a country where citizenship means something than in slowly decaying "world's largest democracy". Any "patriotism" or "national security" that is fragile enough to be destroyed by some s.o.b nutcase putting up an "anti Indian " blog on blogpsot (I offer a bet at ten to one odds that this will be the reason trotted out. Alternatively someone in the government is trying to shake down google) is not worth preserving anyway.

I used to be proud to be Indian. Now I am ashamed.

If I can't get into a decent PhD program this year I'm going to be one of the "cheap brown ill paid programmer doing crap work and desperate for a green card" types in the USA. Or get on a boat and f#%^ing ROW to the West. Anything to get out, so help me God.

Update: The Indian Government has blocked blogspot because "some terrorists are using blogs to communicate". I'm speechless.

Update: How to by pass the ban

Algorithmics In The Dark Compiler Alley

2006-07-15T13:00:00.000+05:30

My upcoming "write a compiler" project is about writing a industrial strength compiler. The good news is that it is wonderful to be in a position where you get paid to do something like this (vs say, writing yet another J2EE/dotNet/framework du jour [something that starts with "R"? :-P] "enterprise app"). The bad news is that I have to deliver a 2 person year project in about 6 person months ( 1 person - Yours Truly - and about 6 months starting in August).

One way to "shrink" such a project is to use powerful languages. A compiler, while quite complex, has very few library dependencies. What this means is that you can write a compiler in any language you want and not sacrifice any advantages while doing so. The converse of this is that one can abandon brain dead languages like say, Java, and use something truly powerful, like say, Haskell.

I've spent the last few weeks writing 'compiler like' bits and pieces in various languages, and have short listed 3 - Lisp, Scheme, and Haskell. Besides their power, these languages have another feature in common - none of them have an open source, batteries included IDE. (And "no IDE" implies "use emacs". And the sheer customizability of emacs is exhilarating but more on that in another post).

An interesting (and unexpected) development is that evaluating alternative designs calls for a degree of algorithmic sophistication. For example, in the code generation phase of the compiler the decision to use an RTL (gcc's intermediate language)clone vs (say) a BURS (bottom up rewriting system) based graph traversal needs the ability to (mathematically) analyse the competing code gen algorithms and the trade offs of each. The key to getting a grip on BURS, for e.g., is to understand Dynamic Programming. More interestingly, there is often a need to augment an existing algorithm or data structure to accomodate application specific constraints. The paradigm embodied by the language under consideration determines the "shape" of the runtime. The demands of the runtime and the level of optimization required determines the appropriateness of a particular code gen algorithm, which in turn influences the transformations required on the abstract syntax tree, and so on.

There aren't too many books which teach you to design algorithms and data structures (vs memorizing a few standard ones). Of course if you are an "enterprise" programmer, you don't need any of this and can get by with simple lists and hashtables provided by your language's libraries. But if you (intend to) work in more challenging (and, if I may say so, interesting) domains, learn to analyze and design algorithms (and data structures). It might even help you get an interesting and satisfying job. Companies like Google and Amazon have very algorithmics heavy interviews. As Cormen et al say in Introduction To Algorithms (the best book ever written on Algorithms and Data Structures),

"Having a solid base of algorithmic knowledge and technique is one characteristic that separates the truly skilled programmers from the novices. With modern computing technology, you can acccomplish some tasks without knowing much about algorithms, but with a good background in algorithms, you can do much, much more)".

Another interesting, though less comprehensive and more lightweight textbook is The Algorithm Design Manual by Steven Skiena. It is more of a complement to Cormen and Rivest than a substitute and has a distinctive writing style. Every chapter has an accompanying "war story", drawn from the author's extensive consulting experience, where he reports how he applied the technique taught in the chapter to solve a real world problem or optimize (sometimes in a very dramatic fashion) an inefficient solution. Imagine - algorithmics consulting - I guess they won't be outsourcing those jobs anytime soon :-D.

The one "problem" with the Cormen and Rivest book is that it calls for a certain level of mathematical sophistication on the part of the student. This is unavoidable in a book that avoids a "cookbook of algorithms" approach and tries to teach how to design new algorithms and adapt existing ones to teh problem space. So, before you buy this book ( and it IS a book that should be on every programmer's bookshelf) be sure to learn basic discrete mathematics, probability, linear algebra and set theory. Working through Concrete Mathematics really helps. The effort will repay itself a thousand fold.

But be warned. Once you taste the power of algorithms you'll never be able to go back to your "build (or even worse, maintain) yet another Visual Blub dotNet banking app in Bangalore" day job. On the other hand, you may find that programming is challenging, meaningful, and above all, fun. Red Pill or Blue? :-)

My blog disappeared!

2006-07-09T15:37:00.000+05:30

A few hours ago my blog began to turn up with an empty page! No content visible(but the posta re still visible form the admin interface). Hopefully the folks at blogger will fix this soon.

This is atest post to see if creating anew post will recover the blog! (or at least update the atom/rss feeds)

It has reappeared! But have lost all the minor tweaks like my blogroll. Aaaaaargh!!!! Oh well I am too busy now to do any template hacking. Fwiw, the mailing list reveals that many people are having this problem. Google/Blogpsot seems to lose your template once every so many updates (Java threads running amok methinks. Or some el crappo database like mysql screwingup?

The best defence is to maintain a copy of your template somewhere (and your posts too if you value them enough. I don't :-) ).

Farewell to Windows

2006-06-27T13:50:00.000+05:30

I just switched the Thinkpad over to Ubuntu Linux. I am generally not prone to waxing lyrical about Linux distros, but Ubuntu is really awesome installs flawlessly on the Thinkpad in about 20 minutes flat (and that includes the windows partition resize step).

I had forgotten what it was to mind meld with your computer in the last few years when I used windows + cygwin almost exclusively on my laptop.

I am home again :-)

SW Methodology - Shaolin Style

2006-06-22T10:49:00.000+05:30

One advantage of blogging is that it enables you to articulate your vague inchoate thoughts more coherently and more importantly, it enables you to develop your thinking. After ranting about the inappropriate use of TDD, I've been able to develop my insights further. Expressing ideas helps to form them as Paul Graham says.

The most fruitful way to think of any pre defined software methodology seems to be that is an encoding of certain practices intended to teach you certain lessons.

In Karate (or any martial art) there are sequences of fixed movements called kata. In music, there are fixed forms called "etudes" (in western music. A rough analogue would be the varna in (South)Indian classical music). Conceptualising a methodology as an "encoding" of what worked for certain people in certain situations is an alternative to "methodology as religion"

The practitioner of a kata (or etude) seeks to perfect each component of the larger structure (a punch in a kata, a particular phrase in an etude) and since absolute perfection is elusive, the practice may go on for years or a life time.

If one thinks of the 12 practices of Extreme Programming (say) as forming a "fixed form" that teaches certain principles of software development (vs as some kind of sanctified system of "rules to follow or you'll burn in hellfire") then it becomes easy to both strive for perfection of each component (say unit tests) while practising, and modifying or discarding the same component during performance as need demands.

Combat is different from kata practice in a dojo. When you are in a dark alley facing someone who tries to gut you with a knife, the important things is to win that fight, not conform to the structure of the kata. When you are practising the kata, the idea is not be creative and do "free flow" but to focus on mastering the form exactly as expressed in the kata. The immediate goal of kata is "perfection of the form" (and the lessons learned therefrom) not "effectiveness in attaining the ultimate purpose".

Guitar Etudes have odd fingerings and chord sequences which will never make it into a lead solo. But when you practice the etude you do what the notation says , not what might best express your musical intent or satisfy an audience. While practising, a gutar maestro will be able to perform an etude perfectly as written in the notation, awkward fingerings and all, but when he is on stage, improvising a solo, he really doesn't think of "exactness of fingering". He focusses on musical effect and allows his fingers to go where they will.

Once you accept that a set of rules telling you how to develop software is just a "fixed form", it is easy to discern the two errors that software methodology practioners (not to mention the "evangelists" who have found the "One True Path") consistently fall into.

The first error is to apply the criteria of practising the form to the performance. iow, to insist that you should fight only using the movements of the kata you learned. "Ohhh you are not writing a test, you are not Agile... Code without tests is 'legacy code' Shame on you ".

The second, and more subtle, error is to apply the criteria of performance to the practice of the form, ie you treat the kata as a combat situation. "Aargh writing tests for javascript is impossible so I won't write any tests because obviously this unit test stuff is impractical" or " Writing tests means I write more code so I will be slower. So I will never write any tests".

Of late there have been some attempts to use the underlying principles of "kata" to improve software development skills. This is the "practise the form perfectly in a non combat situation" approach.

What has been missing is the more important second half -- "combat is not kata". In a hypothetical combat situation, in that dark alley with someone lunging at you with a knife, it isn't very important that you take up the perfect "horse stance" and then do a perfect roundhouse kick. If you end up in hospital, leaking blood from half a dozen stab wounds it doesn't matter if you "took the stance perfectly". As Bruce Lee once said "On the street, no on knows you are a black belt".

In the alley, facing a knife thrust, using a garbage can lid to block the knife and knock the attacker out cold will work fine, nevermind that in Shotokan Karate there are no "Garbage Can Lid Kata".Of course if you've practiced your forms (and other practices like kumite) well for many years, you won't be sweatily flailing around, panting for air and driven by adrenaline, like an untrained person will. You will be in control, using the minimal amount of movement and force and stay balanced all the time , and be aware of things like distance, force, balance, vector of attacks and so on.

You can even learn something from the combat situation and incorporate it into your practice ("hmm fighting in the dark in a cluttered alley is quite different from sparring in well lighted uncluttered dojos so how do I learn to fight better in such conditions?"). Who knows, you may be able to distill your experiences to create a "dark room kata" for others to learn from. And the practitoiners of this "dark room kata" wil in turn diverge from its fixed patterns in their own combat situations. And that is exactly as it should be.

One problem with software development is that "practise" and "performance" are mixed up thoroughly. Both generally happen within the context of a project someone else is paying for. A software developer learns "on the job". It is often impossible to change this state of affairs, but it is helpful to distinguish the "practice" and "performance" aspects , the kata and combat, the etude and the concert.

When you are learning extreme programing, do exactly what Kent Beck teaches in the "white book". When you are actually developing software it doesn't matter if you aren't "exact". If you have difficulties in applying a particular practice, make note and use what works. Later you can think of whether the problem is one of insufficient practice of the technique (in which case you practise more) or whether the technique itself is a misfit for your circumstances (in which case you modify the technique or discard it entirely). There is no intrinsic merit in force fitting the combat situation into the framework of your "school" of fighting. You'll just get stabbed.

There is no "perfect kata" which will reward rigid adherence with universally victorious combat ability. There is no "perfect methodology" which will guarantee success if it is followed "perfectly". Extreme Programming (to use one particular methodology) is not magic. Is is just the encoding of what Kent Beck (and others) learned by trying to get better in the projects they worked on. By all means learn from Kent. Don't worship him (or XP). Ignore the high priced consultants. Your situation and requirements are likely different from what Kent faced. Nobody cares what methodology you follow as long as you write code that delivers value. Arguments about "Shotokan Karate is better than Shaolin Kung Fu" are on the same level as "XP is better than RUP (or waterfall for that matter)".

Combat is not practice. A dojo is not an alley. And vice versa.

The 'Cult Of the MBA'

2006-06-20T15:58:00.000+05:30

In Joel Spolsky's article, there is one sentence that wish I had written

The cult of the MBA likes to believe that you can run organizations that do things that you don't understand.

On second thought though, the ideas behind that sentence look a little bit more complex than they appear. I have seen managers (used interchangeably with "folks who have mbas" for the rest of this post) have made a (positive) difference. Maybe it is just that competent people, with or without an MBA, do make a difference in situations including them. So the question becomes "Is there a significant differential advantage conferred by an MBA?" Somehow I doubt it.(I am talking of differntials of capability, not differntials of social standing or ability to climb organizational ladders) I have seen too many clueless morons with MBAs screw up situations behind repair.

Taking another vantage point, for a company that is essentially about doing cutting edge, innovative things (with or without software) and redefining the way the world works, it would be insane to hand the driving wheel over to a (non engineer) MBA. Which is probably why Larry and Sergei agonized over selecting their CEO for so long, and why Jobs is btter for Apple than Sculley. OTOH if company is involved in "software services" or "offshoring" or whatever, maybe its makes sense to hand over the reins to an MBA who can then 'manage "resources" ',"scale up operations" etc.

And anyway "cults" are not confined to MBAs. On the geek side of the fence we have the Cult of Apple, The Cult of Agile etc. Nothing new here folks, move right along.

I do NOT use Yahoo 360

2006-06-20T13:21:00.000+05:30

so please don't "add me as a friend" in Y360 :-)

I get 2 or 3 mails everyday saying that "Mr/Ms X has requested that you add them as a friend". The thought is appreciated. Seriously. but there is NO point in adding you guys to an account I do not use.

I use practically nothing from Yahoo except a webfacing email id and Yahoo messenger and the latter is for historical reasons, not because I am impressed with its quality. Those of you who know me enough to be "added as friends" please use either my gmail id or ping me on yahoo messenger or skype or... :-) .

I hope I didn't come across as arrogant but I really do NOT like yahoo's flashy ad riddled pages or its Y360 service. Please understand. Thanks in advance.

:-)

Compilers, TDD, Mastery

2006-06-19T20:04:00.000+05:30

I was talking to a friend yesterday about the upcoming compiler project when he asked me a question "Will you be coding the compiler in an Agile fashion? I mean using TDD etc?". This turns out to be an intriguing question.

I got sick of the "silver bullet" style evangelization of the Agile Religion and formally gave up "agile" some time ago. However I do believe in programmer written unit tests being run frequently and that is part of "agile". What I do not believe in is the notion of the design "emerging" from the depths of the code like the Venus of Milo from the waters by just repeating the "write-test, write code, refactor" cycle, a practice otherwise known as TDD.

Anyway, I decided to google for compiler s and tdd and came up with some absolute gems.Here's the first.

On the comp.lang.extreme-programming mailing list, I came across this absolutely hilarious exchange.

In the midst of a tedious discussion on whether XP scales to large projects, "mayan" asked,

I am not asking what kinds of companies are doing XP work - I believe a lot of them are, and successfully. What I am asking is anyone using XP for large size/high complexity/high reliability problems. To be more specific - stuff like optimizing compilers, OS kernels, fly-by-wire systems, data-base kernels etc

Ron Jeffries, one of the "gurus" of agile, replied,

. I'm not aware of any teams doing compilers or operating systems using XP, but having done both, I'm quite sure that they could be done with XP, and the ones I did (at least) would have benefited from it, even though they were successful both technically and in the market.

Aha this looked interesting! Someone actually thinks a compiler can be written in a XP (and presumably TDD) fashion. Mayan issued a challenge to Ron which looked like this

Excellent: lets talk about the following problem: write an optimizing back-end for a C compiler (assume we purchased the front-end from someone). How would we use XP, or adapt XP to it?

Some problems with compiler back-ends:

- its fairly clear what has to be done (core {intermediate-language/instruction selection/register-allocation/code-generation} + set of optimizations); its fairly clear what order they have to be done in (first the core, then a particular sequence of optimizations)

- you don't really need customer input. Having a customer around doesn't help.

- you can't demo very much till you have a substantial portion of the code written - the core (say about 30-60klocs) - no small (initial) release

- you had better get your IL (intermediate language) data-structures right - if you pick ASTs or 3 or 4 address form, you will do fine for the basic "dragon-book" optimizations, but later on you will either run into severe problems doing more advanced optimizations, or you will have to rewrite your entire code base [probably about 100-150klocs at this time]. Is this BUFD?

- you had better think of performance up front. Toy programs are easy; but a compiler has to be able to handle huge 100kloc+ single functions. Many heuristics are N^2 or N^3. Both run-time efficiency and memory usage end up being concerns. You can't leave optimization till the end - you have to keep it in mind always. It also pretty much determines your choice of language and programming style.

- TDD may be applicable for some of the smaller optimizations; on the other hand, for doing something like register-allocation using graph coloring, or cache blocking - I wouldn't even be able to know where to begin.

- The basic granule (other than the core) is the optimzation. An optimization can be small (constant propagation in a single basic block) or large (unroll-and-jam, interprocedural liveness analysis). The larger ones take multiple days to be written. Integrating them "often" is either pointless (you integrate the code, but disable the optimization) or suicidal (you integrate the code, but it breaks lots and lots of tests; see below). Best case: integrate once the optimization is done.

- Its not easy to split an optimization into subproblems; so typically one (programmer/pair) works on an optimization. For the larger ones, if it needs to be tweaked, or fixed, the unit that wrote it is the best unit to fix it. The overhead to grok a couple of thousand lines of code (or more!) vs. getting the original team to fix it is way too high.

- Testing, obviously, is a lot more involved. Not only do we have to check for correctness, we have to check for "goodness" of the produced code. Unfortunately, many optimizations are not universally beneficial - they improve some programs, but degrade others. So, unit testing cannot prove or disprove the goodness of the implementation of the optimization; it must be integrated with the rest of the compiler to measure this. Further, if it degrades performance, it may not be a problem with that optimization - it may have exposed something downstream from it.

- Typical compiler test suites involve millions of lines of tests. They tend to be run overnight, and over weekends on multiple machines. If you have a badly integrated optimization, you've lost a nights run. And passing all tests before integration is, of course, an impossibility. Even a simple "acceptance" set of tests will check barely a small percentage of the total function in the compiler.

Hmmm.....does this still look a lot like XP to you? I can see that at least 1/3rd of the XP practices being broken (or at least, severely bent).

Based on your experience, do you disagree with any of the constraints I outlined?

Mayan

I expected Ron to post a reply explaining how all the above can fit into the Agile/XP framework, but there was only [ the sound of crickets chirping].

On the XPChallengeCompilers page, Ron repeats his claim that "I've written commercial-quality compilers, operating systems, database management systems ". Hmm. Yeah. Whatever. He doesn't make any useful points about how one would actually go about doing something like this.

On the same page, Ralph Johnson has (as one would expect) a more thoughtful and articulate view point about how XP would apply to writing a (simple) compiler, but he focusses more on the fact that

" DoTheSimplestThingThatCouldPossiblyWork is true for compilers, as well. My problem is that people seem to think that the simplest thing is obvious, where my experience is that it is often not obvious. One of the things that makes someone an expert is that they KNOW the simplest thing that will really work."

Now this makes sense. There is no hint of the design "emerging" from "average" developers grinding through the tdd cycle and there is a strong hint that you do have to understand the domain of a compiler well before you can even conceive of the "simplest thing". He concludes with

"The XP practices will work fine for writing an E++ compiler. However, I think there will need to be some other practices, such as comparing code to specification, as well as appreciating the fact that you sometimes must be an expert to know what simple things will work and which won't".. Ahh the blessed voice of rationality.

Another area where Agile falls flat on its face is when dealing with concurrency. See the XP Challenge concurrency page for the flailing attempts of an "agilist" to "design by test" a simple problem in concurrency. Tom Cragill proposed a simple synchronization problem and Don Wells (and some of the agile "gurus") attempted to write a test exposing the bug and didn't succeed till Tom pointed the bug out.

Coming back to compilers, there are projects like the PUGS project, led by the incredible Autrijus Tang, in which unit tests are given a very high priority. I couldn't find any references to the PUGS design "evolving" out of the TDD cycle however. It seems as if they build up a suite of programmer written tests and run it frequently.I can see how that practice would be valuable in any project. Accumulating tests and an "automated build" are the practices I retained from my "agile" years. AFAIK the pugs folks don't do "TDD", expecting the design to emerge automagically.

This whole idea of "emergent design" (through TDD grinding) smacks of laziness, ignorance and incompetence. Maybe if you are doing a web->db->web "enterprise" project you can hope for the TDD "emergence" to give you a good design, (well, it will give you a design :-P) but I would be very surprised if it worked for complex software like compilers, operating systems, transaction monitors etc. Maybe we should perusade Linus to "TDD" the Linux kernel. TDD-ShmeeDeeDee! Bah! :P

Update: Via Jason Yip's post, Richard Feynman's brilliant speech exposes what exactly is wrong with the practice of "Agile" today. An excerpt

I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they've arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head to headphones and bars of bamboo sticking out like antennas--he's the controller--and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.

[DevNote] Interpreter To Compiler

2006-06-17T17:21:00.000+05:30

I generally use a wiki (moinmoin in case anyone is interested) on my laptop to keep notes on various projects I work on. However the wiki has not been transferred from my old laptop to my new one. With all the travellng I am doing, this won't get done till the next month or so. Meanwhile I'll use this blog to record (some of) my dev notes (when i can get net access ...sigh.....). Since Blogger doesn't support tagging or categories, I'll prepend a "[DevNote]" string to the title of the post so readers can ignore the post if they wish. These notes are likely to be cryptic and incoherent and most readers should just skip them.

To transform an interpreter to a compiler,

Rewrite interpreter in CPS form.
Identify Compile time and Runtime actions(CPAs and RTAs)
Abstract RTAs into separate 'action procedures'.Use thunks to wrap any weird control structures where necessary
Abstract Continuation Building procedures (CBPs)
Convert RTAs and CBPs into records. Now the eval on the parse tree spits out a datastructure.
Write an evaluator for the resulting data structure making sure to (a) implement register spilling (b) 'flatten' code to generate "assembly like" code.
TODO : investigate the pros and cons of Bottom Up Rewriting Systems (BURS) vs the scheme above vs generating a (gcc) RTL like intermediate language
TODO : investigate the effects on stack discipline and garbage collection when the various schemes are adopted . Which is more convenient?

StockHive goes live

2006-06-01T11:19:00.000+05:30

I am still internet challenged and will be offline for the next month or so, but this is worth commenting on.

My friend Yogi has launched StockHive, an online tool for folks investing in India's stock markets.

If you are an investor, you should definitely look at StockHive. For the techies amongst you the site is built on Rails.

Blog Lockdown

2006-05-11T15:09:00.001+05:30

I will be travelling for the next couple of months and will have zero network access. Athe commenting facility on this blog have been disabled. I will re enable them when I get back to civilization in mid July. Au Revoir

The Decline and Fall Of India

2006-05-10T19:15:00.000+05:30

The present government has proposed an utterly idiotic caste-based reservation in higher education and in private industry. In other words, get ready to fill in a religion/caste form to determine if you'll get an IIT admission, get a job, get promoted etc.

Atanu says it much better than I can. Read. Learn.

Well, I vote with my feet. I am getting out of this stupid country next year.

[tongue attached to cheek mode on]I wonder if China issues citizenships to foriegners. Of course I'd be arrested in 10 days for "anti communist beliefs" but it would be fun while it lasted. Might be worth considering if China ever became a democracy[tongue attached to cheek mode off]

Abandoning Agile

2006-05-08T12:26:00.000+05:30

My friend Yogi said in a chat session recently, "I havent seen your blog entry renouncing all things agile!"

Yogi, your wish is my command. here it is.

Bill Caputo said it much better than I ever could. So I'll borrow from his blog entry.

"...In short,... I am formally retiring from Agile Evangelism (something I lost the heart for some time ago), and simply becoming an Agile Private-Citizen. Thus, I expect that the tone of my writing will shift somewhat as I no longer care (and truthfully haven't for some time) whether the world adopts XP -- or even whether anyone does -- I am simply interested in building software to the best of my ability, working with others of like mind, and flat-out outperforming others using the techniques and tools that I have learned how to use over the past several years. That anyone familiar with XP would immediately recognize its strong impact on the nature of our software development project is almost (but not quite) irrelevant to that goal. ..."

Of course the difference between Bill's position and mine is that he still continues to use XP. He just doesn't evangelize it. The kind of problems on which I'll be working for the rest of the year (and hopefully, for the rest of my life) shift the context of development so much that most of the practices and assumptions of XP/Agile just don't make sense anymore. The only surviving practices are the constantly growing test suite (no more test first) run periodically and a very attenuated Continous Integration step. Where Bill is an Agile Private Citizen, I am an Agile Ex-Citizen.

I have also stepped out of the Agile Software Community Of India, a non profit society of which I am a founding member. I may still need to attend one final meeting to elect the next year's office holders, and help resolve some residual organizational issues but that's it. I am done with all things Agile.

It is a great relief. Agile is "hot" these days and all sorts of weirdos crawl out of the wooodwork to scramble on the "Agile" platform and caper and scream to attract "agile consulting" assignments. The politics is sickening. Now I can just shut these folks out and focus on what is more important, writing good software.

Grounding "exotic" technology - The Banker's story

2006-05-07T18:15:00.000+05:30

The machine learning system I was called in to design for a Telecom Fraud detection company is moving into production. I was told yesterday that the testing folks are "amazed at the high accuracy of the new codebase" and my client is extremely satisfied with the improvements. I was really happy to hear this. Nothing makes a programmer happier than news that his code/design is making a substantial contribution in the domain for which it was written.

My clients were leaders in their domain even before we added the new ML capability. With the enhancements (not limited to the fraud detection engines, they've added a lot of functionality in other areas) they should wipe the floor with the competition. None of their competitors has a fraud classifier system as powerful. And we have implemented only a small fraction of what could be done.

This datapoint just confirms by belief that powerful technology allied to a strong management vision can lead to massive value addition. What made the difference is the adoption of a key metric(erroneous classification/million cell phone calls) as the measure of the software's quality and the almost total freedom given to the implementation team (which included some of the best folks in the company). We had NO political or turf battles and the laser like focus of the manager in charge on the key metric helped us to evaluate alternate strategies and select the most appropriate ones. This (tying the software developed to key business metrics) is one way of "grounding" business software and preventing thrashing. (More explanation of this conmcept coming up in future blogs).

Back to Banking. One of the nuggets that came out of our conversation(see last blog) was the fact that most Indian banks still do manual verification of signatures on cheques. This has been automated to a large degree a long time ago in most banks in the West. The algorithms for this are available in the literature. Large chunks of software that made up the original solution are publicly available. All it needs is for someone to put together a hardware/software combination to provide this functionality and the banks would jump at it.

Why is this important? Here is the Industry context. In 2009, the Indian Banking sector will be opened to Foreign banks . (It is semi-open right now). That would mean(assuming India is doing well economically) that tidal wave of foreign banks would arrive and the existing banks would face real competition.

The scariest part of this impending shift in the landscape, from the point of view of Indian Banks is the massive technological advantage that western banks enjoy (relative to the Indian banks). And in the Indian banking scenario, market share is almost directly in proportion to the technology level of each bank.

Here is an example. For years, only ICICI offered an "at par cheque" facility, ie, the ability to present a cheque drawn on an account in another city and get it cashed instantaneously (vs. waiting for few days for the cheque to be couriered to the city in which the account resides). Thousands of businessmen did not have an alternative but to bank with ICICI because no one else had anything similair.

By the time the other banks caught up with this technology, ICICI came up with a feature by which transfers between accounts could be transacted on the net (even if the target account was in another bank). As of today no one else has this feature(unless they go through ICICI) and this gives them an edge.

I forget the domain term for this. But the point is that the ranking in the technical arms race almost directly determines the marketshare. And this marketshare is what is under threat when the big boys land with their deep pockets and massive mainframes. So even something as simple as automated cheque signature verification could provide an edge in the impending blood bath in the banking sector.

So there. Now you have an idea on how to make millions. If you do, cut me a cheque for the idea. :-)

A Thought Experiment

2006-05-07T14:07:00.001+05:30

I recently had a conversation with a friend who works in ICICI, one of the largest banks in India. Contrary to my "image" I am deeply interested in Business and the "enterprise"; I belive the practice of outsourcing the software at the heart of an enterprise to body shops flogging mediocre programmers is indefensible. My friend was kind enough to enlighten me on how banking really works and the context in which Indian Banking operates. The next few blog entries will be devoted to insights which spun off rom that conversation. First though, I'd like to focus on the gaps between image,ideal and reality

As with all industries (e.g the outsourced software "industry") there is an image, ("intellectually challenging work done by extremely bright people trying to change the world" in the case of the Indian software industry) which people outside the industry believe and a not-so-cool reality which only insiders are aware of. The reality of India's software "industry" is that most "hackers" working in it are comparable to the illegal immigrants from Mexico who come to swanky Wall Street offices early in the morning to vaccum and dust. The difference is that the average mexican immigrant who does these jobs does not fool himself into thinking he is doing "cutting edge" work.

The interesting thing is that beyond these two perceptions ("What does IndustryX look like from the outside?" and "What is IndustryX really like (from the inside)?") there is another question "What could IndustryX be like if we tried to really deliver superior customer value and create first class working environments and did first class work?".

Very few companies are trying to envision and implement the last question in any industry,whether it be software, banking or anything else.

Let us do a thought experiment and envision a future in which India did less of the "janitorial" programming, and becomes a true "software super power". In such an imaginary point in the multiverse,

India would be home to innovative companies like (say) Apple and Google
At least one popular language (like, say, Ruby), Operating Sytem (like Linux) , or widely used tool (like say, emacs) would be invented by an Indian.
People all over the world would expect a newly announced fantastic advance in software to have an Indian (vs an American) origin. At least afew "iconic" thought leaders (of the calibre of Linus Torvalds or Steve Jobs) would be Indian.
Many Indians would be publishing the "killer books" in various subfields of software theory and practice. Take the best technical book you have read. Now imagine it having an Indian author to see what I mean.
India would have annual events like the DARPA Robotics Car Race, with relentless pushing forward of the frontiers of knowledge and possibility every year.
The most talented and ambitious students from other countries would apply in droves to Indian Educational Institutions and the selection commitee would have the luxury to pick and choose the best.
Harvard and MIT would learn from IIMs and IIT and try to imitate their methods and "best practices".
Bangalore would be the California of the world, with the most innovative people , companies, and ideas finding their natural home there.
People all over the world would want to immigrate to India and acquire Indian citizenship. Our consulates in the USA and UK would be forced to impose quotas on work and other immigrant visas.
The core of a new software system would be designed and implemented in Bangalore and the "janitorial" work would be "outsourced" elsewhere. (Yes, I see you ROTFL :-) I told you this is all imagination. A man can dream can't he? :-))

Ah well, time to stop dreaming and do some useful work. But the future is not all doom and gloom. There are a few faint glimmers of light in an otherwise dark horizon. More on that later.

Everything Old is New Again - The Ruby DSL hype

2006-04-18T16:08:00.001+05:30

Update: I've received a few mails asking how to go about creating dsl s "the right way" in Ruby. If you are in Bangalore, I suggest you attend the Ruby DSL talk at the Agile India 2006 by my friends Badri and Sudarshan (both Thoughtworkers). Such things are best learned by demonstration than reading about it and the Badri and Sudarshan are probably the best people in Bangalore to learn from.

If you are not in Bangalore find someone who knows and ask him to teach you. :-)

[Rant] These things have been true from the dawn of programming languages

A language with a built in eval function allows you to transform code into a parse tree at runtime.
To the degree that the language has a uniform/flexible syntax, this transition between code and data is painless
All language transforms consist of two steps . First you convert a string into a data structure called a "syntax tree" . This is called parsing. Then various transformations can be done on this data structure to produce the effect you want.
Thus the string "2 + 3" can be converted into a datastructure like "{ operator: "+", operand1:"2", operand2:"3"}".
Then you can define various transforms that work on (or "visit" in oo terms) the components of the data structure and do whatever you want.
e.g : (pseudocode) def print(anAst) puts anAst.operand1,anAst.operator,anAst.operand2 end
def printReversePolish(anAst) puts anAst.operand1,anAst.operand2,anAst.operator end
def evaluate(anAst) return anAst.operand1 + anAst.operand2 end
anAst = parse("2 + 3")
print(anAst) => 2 + 3
printReversePolish(anAst) => 2 3 +
evaluate(anAST) => 5
and that is all there is (conceptually) to any langauge transform.
Ruby "DSL" s allow people who haven't studied the fundamentals of computer science to think they are discovering something new and amazing. Yes, Ruby allows a somewhat seamless mixing of "pure" ruby and "dsl" code, but this is just as true for any langauge with a built in eval and clean syntax. (lisp, forth, even smalltalk). The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely from ruby syntax)

To understand ruby "dsl" s, ignore all the hype (and the forthcoming stream of books explaining how ruby's 'dsl ability' is the next silver bullet that will save enterprise programmers , give their arid lives meaning, and solve world hunger), understand how interpreters work (read "The Essentials Of programming Languages") and meditate on these concepts in ruby - class eval, object_eval, method_missing. if you understand the basics of how languages work, ruby's "dsl" is some (very) old wine in some sparkling new bottles.

The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face.

Ok , biting your tongue and walking away quietly is almost as good.

DSLs like any other programming technique, when used appropriately gives some beneficial effects. But it is no panacea and a casual perusal of the net will reveal a bunch of atrocious dsls . Beware the hype and snake oil. [end Rant]

Learning from a Master{Class}

2006-04-08T18:01:00.000+05:30

I attended a series of lectures on technical topics today and here are my "take away points" in no particular order.

"Service Oriented Architecture" is an empty buzzword label designed to squeeze money out of unsuspecting clients
Good ideas (e.g: "role playing" a presentation) needs to be executed flawlessly to be effective and cannot be substitute for a belief in what one is speaking about.
Having one set of people selecting topics and another set of people being "volunteered" to speak is a terrible way of attempting to deliver a great speech/session.
Having original insights arranged in a logical structure is key to a good speech. Presentation skills provide a final finish.
Insight arises from taking the time to rigorously think about what is important, and by asking "why" for things everyone takes for granted.
Here is a way to test your "hacker quotient" (assuming you are a programmer). How much of your code is written by you wanting to code it vs having to code it?
If a series of sessions is to be arranged on a single theme, care should be taken to have them rise to a peak, in terms of quality of content and speakers. An abrubt decline in (relative) quality drives the audience away (fast).
If you don't have a good, genuine question to ask the speaker after a speech, for God's sake, shut up and sit down. Nobody thinks you are smart if you ask vague, inchoate and stupid questions

The logistics and organization was flawless. Congratulations to all concerned. I hope the forthcoming Agile India Conference does as well.

The Rest Of The Year - Plans

2006-04-05T20:08:00.000+05:30

I just rolled off a Telecom Fraud Detection project that saw the successful deployment of one of my core foci - Artificial Intelligence. My next project in 2006 will be in my next major focus area - compilers.

I can't talk too much about this in public yet but I can say that till about August middle/September beginning (tentative deadline) I will be building a production quality compiler for some custom hardware. Large parts of this project (if not all of it) will be open sourced at the end of the project, once the code base is stable enough. Parallelly I hope to help one of my friends bring out a product, and do a serious upgrade of my mathematics skills.

Life is good.

[Context Switch]

If any readers of this blog want to meet me, the Agile India Conference, (where Rajesh and I are doing a linked pair of talks on Robotics) is a good chance to do so.

I shall be talking about language design issues in robotics and Rajesh, on the problems of implementing custom firmware . There are many other interesting sessions on the agenda ( I particularly look forward to KD's speech on the impact of Culture on Agile Practices - knowing KD, this promises to be a real blast), so if you are in Bangalore (or can get to Bangalore, don't miss the fun. (Registration will open later tonight).

Update : Registration for Agile India 2006 is now open.

The Storm Before the Calm

2006-03-31T15:23:00.000+05:30

As of May the 6th of 2006 (the final day of the Agile India Conference, where I am speaking on "The Design And Implementation of Robotics Languages"), I am completely abandoning all connections to the "offshored software" world. No more agile/extremeprogrammng/tdd, no more j2ee, no more "offshore teams" delivering "superior business value" using "best practices" crap.

Paul Graham said,

"If it is possible to make yourself into a great hacker, the way to do it may be to make the following deal with yourself: you never have to work on boring projects (unless your family will starve otherwise), and in return, you'll never allow yourself to do a half-assed job. All the great hackers I know seem to have made that deal, though perhaps none of them had any choice in the matter."

Now whether I am or will ever be a "great hacker" aside, "you'll never work on boring projects, and in return, you'll never allow yourself to do a half-assed job." seems a fantastic principle to live by (it helps that I don't have a family I need to keep above the starvation line - Oh the blessed joys of bachelorhood :-) ).

Once you cross the age of 30, you are in the second half of your life. It is horrible enough that I spent one decade caught up in the "offshored software" paradigm. One decade - think of it and weep. A decade of dedicated practice would make you world class in anything you choose. One could be a world class athlete, musician, martial artist, dancer,writer, film maker... whatever! And I spent it writing "plonk this data off a database and put it on a set of dinky webpages" systems.

Shakespeare got it right when he made Macbeth say "And all our yesterdays have lighted fools, The way to dusty death."

And it is a very seductive bargain - in the beginning - money for nothing, multiple trips abroad (ignoring the fact that you are roughly in the position of a Sepoy in the Old army of the British Raj, occasionally sent abroad to shed your blood in the Empire's mysterious wars), all the trappings of a "successful" life. Time blurs and you wake up to find yourself either a second grade "project manager" type implementing commands from across the seas (and trying desperately to convince yourself that you are now in a superior position compared to the lowly blue collar techies, having forgotten what little you knew) or yet another burned out "techie" having no skills beyond "j2ee" or "perl and php".

Spending any more time in that world would mean I am frittering away what remains of my life. The Red Pill came just in time, methinks.

The number of genuinely good programmers and the amount of true innovation in India is astoundingly low, for all the talk of being the "21st century superpower". The "pseudo geek" profile many people unwittingly take on is very low on substance and very high on deception.

But it is not all doom and gloom. The less something exists, the more the opportunity to create it.

The mistake,I've found, is in thinking, like many Non Resident Indians do, that "I'll go home next year (and meanwhile, endure another year of meaninglessness)". Of course "next year" never comes. If one plans to "go home", the time to do it is right now.

From Snow Crash,

" Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. If my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, and devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad."

I am way past 25, but "devoting my life to being bad" sounds good anyway.

The Importance of Having Friends Who Disagree

2006-02-25T16:54:00.000+05:30

Paul Graham says, in one of his essays (emphasis mine) "....Why do you need other people? Can't you just think of new ideas yourself? The empirical answer is: no. Even Einstein needed people to bounce ideas off. Ideas get developed in the process of explaining them to the right kind of person. You need that resistance, just as a carver needs the resistance of the wood.

This is one reason Y Combinator has a rule against investing in startups with only one founder. Practically every successful company has at least two. And because startup founders work under great pressure, it's critical they be friends....."

In my experience, this need for "friendly resistance" extends to far more than creating startups. Every time you have a new idea, you need people you can bounce it off. To get any real benefit out of this process you need people with a complex combination of characteristics.

They should

have firm (but not rigid) opinions on their own
have logical reasons for those beliefs and be able to articulate them clearly
are driven by ideas and not ideology
not attach their egos to their opinions.
be willing to concede a valid argument even if it forces them to possibly re-examine their beliefs
know how to listen

Given all that, it is extremely difficult to find such "friends who disagree". I have been fortunate to have quite a few people around me, who have major disagreements with me, but are still friends (e.g see the discussion I had with KD in the comments section of my last blog entry).

Many people make the mistake of sorrounding themselves with people who think exactly like them and reinforce every idea or prejudice they have. This is a bad mistake and will often end up distorting the reality you see. One needs the corrective bucket of cold water in one's face once every so often. The only problem with surrounding yourself with bright people who think differently is that you may occasionally find that one of your ideas isn't as hot as you think it was or that one of your deeply held convictions is just plain wrong. This is dificult for some people because they make the rightness of their ideas a validation of their worth as persons. Blogging is a great way to expand this circle. But it can work both ways and one may end up with a "fan club" that just reinforces your prejudices.

So do a quick test right now. Make a list of your closest friends/acquintances/advisors. Then make a separate list of those who think totally differently from you, but are worth listening to anyway. Look at the intersection of these lists.

You may be surprised.

Yet Another Math Milestone

2006-02-21T20:06:00.000+05:30

A few days ago, I discovered an error in the mathematics of a paper (on neural network optimization) being prepared for publication by a very eminent scientist. My drawing attention to this discrepancy in the proof has led to a total recasting of the approach to the problem and I will be now listed as a co-author of the paper.

Hmm yeah. Whatever. So how is this significant?

Well this is the first time I have used my skill in mathematics (vs my skill in programming) to contribute significantly to a scientific effort. In an old blog post I had theorised that the acquisition of "mathematical thinking" would follow a four step path. I said ..

the first step in mastering math is be to learn to read the notation. just like learning the syntax of a new programming language

the second is to grasp the reality expressed by the notation at a gut level , like understanding the paradigm and patterns lying underneath a programming language , like ,say ,beginning to grok "oo"

the third is to use that understanding to create new possibilities

the fourth is to use a programming language to embody and refine those possibilities , thus creating programs that do what has never been done before.

On this scale, after more than a year of very hard work, I seem to be half way through the third step. I am now able to "think in math" just as (good) programmers are able to "think in objects"(or functions, or whatever the underlying abstraction of the language is) to a point where I can examine the line of a mathematical algorithm and extrapolate it in into new directions to limited distances. As of today I am not able to think "freewheelingly" in math and my ability to think programatically far outstrips my ability to think mathematically, but these two modes of thinking seem to be duals of each other.

Programmatic thinking somehow seems to investigate the "structure" (and the structural integrity) of a solution to a problem while mathematical thinking focusses on the abstract essence of a problem and the "degree of rightness" of a solution.

One is not a substitute for other. Thus I am very doubtful that "merciless refactoring", for example, gives any real insight into the nature of a problem, like many "agilists" claim. At best it gives an insight into the structure of a proposed solution. This grants only an indirect perspective onto the problem space. I have seen many excellent programmers who were relentless practitioners of "agile" work in a domain (say retailing) for years and still not really understand the "essential nature" of the domain (vs creating good solution structures). In other words they function like draftsmen with no artistic spark, repeating the same patterns endlessly without any genuine understanding or insight. Frequent "domain hopping" , whereby you work on a retail project for a few months, then move to "banking" then to "insurance" and so on, helps to deepen this focus on purely programmatic skills till every project becomes a blur of "billable hours" and "database->gui->database" options.

The scientist/ mathematician/domain expert on the other hand often understands a domain very well and can isolate problems and even propose solutions but is often unable to translate those visionsinto reality for lack of professional level programming skills. To use the art analogy, this is like being able to visualize exquisite paintings, without the required skill to transfer that vision to paper.

The traditional solution to this seems to be for the domin expert and programmer to engage in "conversation" ( 'onsite customer' anyone? :-) ). But in my view this is like drawing a picture by putting a "visualizer" (who can't draw) and a "draftsman" (who doesn't have artistic insight)in the same room and asking them to produce a painting. It might work for Identikit pictures but you won't get any works of art.

Coming back to the original idea, Mathematics,like programming is very much a meta skill, in that one can use the expertise across multiple domains and problem sets. Unlike programming, mathematics often helps the practitioner to grasp the underlying essential nature of a domain.

To conclude, I am considering switching my (long planned) PhD to Applied Mathematics (from Computer Science), the reasoning being that I can always learn programming on my own while using the formal structure of a PhD program to hone my mathematical skill. Alternatively I could specialize in a field that combines mathematics and programming (like AI or Robotics). However that decision is still some way off. In the coming months, I hope to deepen my knowledge of both mathematics (Topology and Group Theory if anyone is interested) and programming (more AI and Programming Language Theory). I am now more convinced than ever that a combination of mathematical and programming ability is worth acquiring.

Wir sind alle Dänen

2006-02-11T11:05:00.000+05:30

This is the rare "political" post. Those not interested, move right on.

Enough is enough.

In my opinion it is foolishness to initiate a "clash of civilizations". But if there is a battle line forming by virtue of some people insisting that their religious laws should apply to people of other religions (or no religion) then I know on which side I stand.

Anyone has the right to be "offended". That does not give them the right to kill, burn and issue threats. Fwiw, I do not believe in Islam, Mohammed, his "sacredness" or infallibility, etc etc. The people who protest most at this "disrespect" have laws in their own countries that officially forbid the practice of any other religion. If you carry a Bible or Bhagavad Gita into Saudi Arabia, it will be seized and shredded at the airport. How dare they protest against any "disrespect"? First practice this "respect to other religions" and then complain your religion isn't respected.

Kennedy once said "Two thousand years ago the proudest boast was civis Romanus sum. Today, in the world of freedom, the proudest boast is 'Ich bin ein Berliner.' All free men, wherever they may live, are citizens of Berlin, and, therefore, as a free man, I take pride in the words 'Ich bin ein Berliner!'"

Today, all free men, no matter where they live, are citizens of Denmark.

Wir sind alle Dänen.

Update: This post has generated a lot of feedback mostly by email (some people seem afraid to express themselves in public. hmmm...) .

One comment on this blog (subsequently deleted by the author) said "There's also the trap of 'they don't respect my religion/beliefs/whatnot so why should I respect theirs?'. That's the point where civilized conversation breaks down." .

This is a little naive. You can't have a conversation with people who don't follow the norms of "civilized" conversation and are not interested in a conversation. This most often happens with people who are convinced that their truth (religion/prophet/Holy Book/ideology/ whatever) is infallible. There should be no tolerance of the intolerant.

From an article by the editor of Jyllands-Posten (the Danish newspaper that published the cartoons) (emphasis mine)

"Has Jyllands-Posten insulted and disrespected Islam? It certainly didn't intend to. But what does respect mean? When I visit a mosque, I show my respect by taking off my shoes. I follow the customs, just as I do in a church, synagogue or other holy place. But if a believer demands that I, as a nonbeliever, observe his taboos in the public domain, he is not asking for my respect, but for my submission. And that is incompatible with a secular democracy."

So yes, this kaffir maintains that anyone who demands rights and respect that he is not willing to give to others is a hypocrite.

And while I may not (or may) be impressed with Islam or its Prophet, (that is my privilege in a free society), my stance is not anti-Islam but anti-hypocrisy.

People (of any religion) who say "respect my religion" should extend respect to other religions (and to atheists). There is no one way "civilized debate".

Old Laptop, Meet New Laptop

2006-02-04T00:55:00.000+05:30

As you can see I am not too much of a photographer (the laptop screens are much duller in the photos than in real life. I have no clue why.) but the new IBM ThinkPad Z60m (on the right) is a very nice machine, much more so than the (now ancient) Compaq Presario 2100 (on the left).

Indians are cheap, not free!

2006-02-01T01:36:00.000+05:30

When you are an independent consultant, you see a lot of crazy people. The most dreaded monster in this bestiary is that of the Non-Resident-Indian-Trying-To-"Develop Products"-In-Bangalore.

The game generally goes like this.

Some desperate Indian working in the USA on a regular job (note that he is NOT a venture capitalist, businessman etc) gets what he thinks is an "Awesome" idea. The he discuses this with some equally clueless friends and they hammer out some technology choices (j2ee , .net whatver). After a few weeks/months of this they decide they want the software done "cheap". Even companies which charge very little money per developer hour (say TCS) wouldn't give these folks the time of the day so they ask their friends in Bangalore to reccomend "top grade" independent developers.These poor innocents recomend people they know to be good developers and have left the corprorate rat race to work independently.

Now the fun begins. The Independent Consultant (IC from now) gets a call from the USA saying "MR X who you worked with in Company ... reccomended you as a top notch developer. We have this awesome idea that we need you to develop in "asp /.net" within 3 months. We also are looking for junior developers, testers and maybe a Project Manager. Do you think you can help us"? The consultant, if he has been independent for a while, has seen this dance many tims before and immediately asks "Do you guys have any money or is all this wishful thinking?"

There are a few different answers to this. Sometimes it is "yeah we can pay 5$ per hour for the first 24 weeks " ( I am NOT joking!!!). Or alternatively , "you see we are just starting out blah blah so we don't have much ready cash but we have all these highflying VCs lined up and if you can work free for the first 6 months then you have millions of dollars in equity".

Duh . Yeah Right.

Some of the more ... emmm .. subtle folks wants the IC to work for them without knowing what the "cool idea " is. After all a "cool idea" is very valuable right? We once encountered an "idea man" who offered to join our company as a "Project Manager" ("since it has been a long time since I coded anything") while looking for funding and in the meanwhile he would let us pay him an "industry standard" salary.

Stupidity is the one thing which REALLY has no limts.