Blog Lockdown

I will be travelling for the next couple of months and will have zero network access. Athe commenting facility on this blog have been disabled. I will re enable them when I get back to civilization in mid July. Au Revoir

The Decline and Fall Of India

The present government has proposed an utterly idiotic caste-based reservation in higher education and in private industry. In other words, get ready to fill in a religion/caste form to determine if you'll get an IIT admission, get a job, get promoted etc.

Atanu says it much better than I can. Read. Learn.

Well, I vote with my feet. I am getting out of this stupid country next year.

[tongue attached to cheek mode on]I wonder if China issues citizenships to foriegners. Of course I'd be arrested in 10 days for "anti communist beliefs" but it would be fun while it lasted. Might be worth considering if China ever became a democracy[tongue attached to cheek mode off]

Abandoning Agile

My friend Yogi said in a chat session recently, "I havent seen your blog entry renouncing all things agile!"

Yogi, your wish is my command. here it is.

Bill Caputo said it much better than I ever could. So I'll borrow from his blog entry.

"...In short,... I am formally retiring from Agile Evangelism (something I lost the heart for some time ago), and simply becoming an Agile Private-Citizen. Thus, I expect that the tone of my writing will shift somewhat as I no longer care (and truthfully haven't for some time) whether the world adopts XP -- or even whether anyone does -- I am simply interested in building software to the best of my ability, working with others of like mind, and flat-out outperforming others using the techniques and tools that I have learned how to use over the past several years. That anyone familiar with XP would immediately recognize its strong impact on the nature of our software development project is almost (but not quite) irrelevant to that goal. ..."

Of course the difference between Bill's position and mine is that he still continues to use XP. He just doesn't evangelize it. The kind of problems on which I'll be working for the rest of the year (and hopefully, for the rest of my life) shift the context of development so much that most of the practices and assumptions of XP/Agile just don't make sense anymore. The only surviving practices are the constantly growing test suite (no more test first) run periodically and a very attenuated Continous Integration step. Where Bill is an Agile Private Citizen, I am an Agile Ex-Citizen.

I have also stepped out of the Agile Software Community Of India, a non profit society of which I am a founding member. I may still need to attend one final meeting to elect the next year's office holders, and help resolve some residual organizational issues but that's it. I am done with all things Agile.

It is a great relief. Agile is "hot" these days and all sorts of weirdos crawl out of the wooodwork to scramble on the "Agile" platform and caper and scream to attract "agile consulting" assignments. The politics is sickening. Now I can just shut these folks out and focus on what is more important, writing good software.

Grounding "exotic" technology - The Banker's story

The machine learning system I was called in to design for a Telecom Fraud detection company is moving into production. I was told yesterday that the testing folks are "amazed at the high accuracy of the new codebase" and my client is extremely satisfied with the improvements. I was really happy to hear this. Nothing makes a programmer happier than news that his code/design is making a substantial contribution in the domain for which it was written.

My clients were leaders in their domain even before we added the new ML capability. With the enhancements (not limited to the fraud detection engines, they've added a lot of functionality in other areas) they should wipe the floor with the competition. None of their competitors has a fraud classifier system as powerful. And we have implemented only a small fraction of what could be done.

This datapoint just confirms by belief that powerful technology allied to a strong management vision can lead to massive value addition. What made the difference is the adoption of a key metric(erroneous classification/million cell phone calls) as the measure of the software's quality and the almost total freedom given to the implementation team (which included some of the best folks in the company). We had NO political or turf battles and the laser like focus of the manager in charge on the key metric helped us to evaluate alternate strategies and select the most appropriate ones. This (tying the software developed to key business metrics) is one way of "grounding" business software and preventing thrashing. (More explanation of this conmcept coming up in future blogs).

Back to Banking. One of the nuggets that came out of our conversation(see last blog) was the fact that most Indian banks still do manual verification of signatures on cheques. This has been automated to a large degree a long time ago in most banks in the West. The algorithms for this are available in the literature. Large chunks of software that made up the original solution are publicly available. All it needs is for someone to put together a hardware/software combination to provide this functionality and the banks would jump at it.

Why is this important? Here is the Industry context. In 2009, the Indian Banking sector will be opened to Foreign banks . (It is semi-open right now). That would mean(assuming India is doing well economically) that tidal wave of foreign banks would arrive and the existing banks would face real competition.

The scariest part of this impending shift in the landscape, from the point of view of Indian Banks is the massive technological advantage that western banks enjoy (relative to the Indian banks). And in the Indian banking scenario, market share is almost directly in proportion to the technology level of each bank.

Here is an example. For years, only ICICI offered an "at par cheque" facility, ie, the ability to present a cheque drawn on an account in another city and get it cashed instantaneously (vs. waiting for few days for the cheque to be couriered to the city in which the account resides). Thousands of businessmen did not have an alternative but to bank with ICICI because no one else had anything similair.

By the time the other banks caught up with this technology, ICICI came up with a feature by which transfers between accounts could be transacted on the net (even if the target account was in another bank). As of today no one else has this feature(unless they go through ICICI) and this gives them an edge.

I forget the domain term for this. But the point is that the ranking in the technical arms race almost directly determines the marketshare. And this marketshare is what is under threat when the big boys land with their deep pockets and massive mainframes. So even something as simple as automated cheque signature verification could provide an edge in the impending blood bath in the banking sector.

So there. Now you have an idea on how to make millions. If you do, cut me a cheque for the idea. :-)

A Thought Experiment

I recently had a conversation with a friend who works in ICICI, one of the largest banks in India. Contrary to my "image" I am deeply interested in Business and the "enterprise"; I belive the practice of outsourcing the software at the heart of an enterprise to body shops flogging mediocre programmers is indefensible. My friend was kind enough to enlighten me on how banking really works and the context in which Indian Banking operates. The next few blog entries will be devoted to insights which spun off rom that conversation. First though, I'd like to focus on the gaps between image,ideal and reality

As with all industries (e.g the outsourced software "industry") there is an image, ("intellectually challenging work done by extremely bright people trying to change the world" in the case of the Indian software industry) which people outside the industry believe and a not-so-cool reality which only insiders are aware of. The reality of India's software "industry" is that most "hackers" working in it are comparable to the illegal immigrants from Mexico who come to swanky Wall Street offices early in the morning to vaccum and dust. The difference is that the average mexican immigrant who does these jobs does not fool himself into thinking he is doing "cutting edge" work.

The interesting thing is that beyond these two perceptions ("What does IndustryX look like from the outside?" and "What is IndustryX really like (from the inside)?") there is another question "What could IndustryX be like if we tried to really deliver superior customer value and create first class working environments and did first class work?".

Very few companies are trying to envision and implement the last question in any industry,whether it be software, banking or anything else.

Let us do a thought experiment and envision a future in which India did less of the "janitorial" programming, and becomes a true "software super power". In such an imaginary point in the multiverse,

1. India would be home to innovative companies like (say) Apple and Google
.
2. At least one popular language (like, say, Ruby), Operating Sytem (like Linux) , or widely used tool (like say, emacs) would be invented by an Indian.
.
3. People all over the world would expect a newly announced fantastic advance in software to have an Indian (vs an American) origin. At least afew "iconic" thought leaders (of the calibre of Linus Torvalds or Steve Jobs) would be Indian.
4. Many Indians would be publishing the "killer books" in various subfields of software theory and practice. Take the best technical book you have read. Now imagine it having an Indian author to see what I mean.
5. India would have annual events like the DARPA Robotics Car Race, with relentless pushing forward of the frontiers of knowledge and possibility every year.
6. The most talented and ambitious students from other countries would apply in droves to Indian Educational Institutions and the selection commitee would have the luxury to pick and choose the best.
7. Harvard and MIT would learn from IIMs and IIT and try to imitate their methods and "best practices".
8. Bangalore would be the California of the world, with the most innovative people , companies, and ideas finding their natural home there.
9. People all over the world would want to immigrate to India and acquire Indian citizenship. Our consulates in the USA and UK would be forced to impose quotas on work and other immigrant visas.
10. The core of a new software system would be designed and implemented in Bangalore and the "janitorial" work would be "outsourced" elsewhere. (Yes, I see you ROTFL :-) I told you this is all imagination. A man can dream can't he? :-))

Ah well, time to stop dreaming and do some useful work. But the future is not all doom and gloom. There are a few faint glimmers of light in an otherwise dark horizon. More on that later.

Everything Old is New Again - The Ruby DSL hype

Update: I've received a few mails asking how to go about creating dsl s "the right way" in Ruby. If you are in Bangalore, I suggest you attend the Ruby DSL talk at the Agile India 2006 by my friends Badri and Sudarshan (both Thoughtworkers). Such things are best learned by demonstration than reading about it and the Badri and Sudarshan are probably the best people in Bangalore to learn from.

If you are not in Bangalore find someone who knows and ask him to teach you. :-)

[Rant] These things have been true from the dawn of programming languages

1. A language with a built in eval function allows you to transform code into a parse tree at runtime.
2. To the degree that the language has a uniform/flexible syntax, this transition between code and data is painless
3. All language transforms consist of two steps . First you convert a string into a data structure called a "syntax tree" . This is called parsing. Then various transformations can be done on this data structure to produce the effect you want.

   Thus the string "2 + 3" can be converted into a datastructure like "{ operator: "+", operand1:"2", operand2:"3"}".

   Then you can define various transforms that work on (or "visit" in oo terms) the components of the data structure and do whatever you want.

   e.g : (pseudocode) def print(anAst) puts anAst.operand1,anAst.operator,anAst.operand2 end

   def printReversePolish(anAst) puts anAst.operand1,anAst.operand2,anAst.operator end

   def evaluate(anAst) return anAst.operand1 + anAst.operand2 end

   anAst = parse("2 + 3")

   print(anAst) => 2 + 3

   printReversePolish(anAst) => 2 3 +

   evaluate(anAST) => 5

   and that is all there is (conceptually) to any langauge transform.

4. Ruby "DSL" s allow people who haven't studied the fundamentals of computer science to think they are discovering something new and amazing. Yes, Ruby allows a somewhat seamless mixing of "pure" ruby and "dsl" code, but this is just as true for any langauge with a built in eval and clean syntax. (lisp, forth, even smalltalk). The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely from ruby syntax)

To understand ruby "dsl" s, ignore all the hype (and the forthcoming stream of books explaining how ruby's 'dsl ability' is the next silver bullet that will save enterprise programmers , give their arid lives meaning, and solve world hunger), understand how interpreters work (read "The Essentials Of programming Languages") and meditate on these concepts in ruby - class eval, object_eval, method_missing. if you understand the basics of how languages work, ruby's "dsl" is some (very) old wine in some sparkling new bottles.

The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face.

Ok , biting your tongue and walking away quietly is almost as good.

DSLs like any other programming technique, when used appropriately gives some beneficial effects. But it is no panacea and a casual perusal of the net will reveal a bunch of atrocious dsls . Beware the hype and snake oil. [end Rant]

Learning from a Master{Class}

I attended a series of lectures on technical topics today and here are my "take away points" in no particular order.

1. "Service Oriented Architecture" is an empty buzzword label designed to squeeze money out of unsuspecting clients
2. Good ideas (e.g: "role playing" a presentation) needs to be executed flawlessly to be effective and cannot be substitute for a belief in what one is speaking about.
3. Having one set of people selecting topics and another set of people being "volunteered" to speak is a terrible way of attempting to deliver a great speech/session.
4. Having original insights arranged in a logical structure is key to a good speech. Presentation skills provide a final finish.
5. Insight arises from taking the time to rigorously think about what is important, and by asking "why" for things everyone takes for granted.
6. Here is a way to test your "hacker quotient" (assuming you are a programmer). How much of your code is written by you wanting to code it vs having to code it?
7. If a series of sessions is to be arranged on a single theme, care should be taken to have them rise to a peak, in terms of quality of content and speakers. An abrubt decline in (relative) quality drives the audience away (fast).
8. If you don't have a good, genuine question to ask the speaker after a speech, for God's sake, shut up and sit down. Nobody thinks you are smart if you ask vague, inchoate and stupid questions
.

The logistics and organization was flawless. Congratulations to all concerned. I hope the forthcoming Agile India Conference does as well.

The Rest Of The Year - Plans

I just rolled off a Telecom Fraud Detection project that saw the successful deployment of one of my core foci - Artificial Intelligence. My next project in 2006 will be in my next major focus area - compilers.

I can't talk too much about this in public yet but I can say that till about August middle/September beginning (tentative deadline) I will be building a production quality compiler for some custom hardware. Large parts of this project (if not all of it) will be open sourced at the end of the project, once the code base is stable enough. Parallelly I hope to help one of my friends bring out a product, and do a serious upgrade of my mathematics skills.

Life is good.

[Context Switch]

If any readers of this blog want to meet me, the Agile India Conference, (where Rajesh and I are doing a linked pair of talks on Robotics) is a good chance to do so.

I shall be talking about language design issues in robotics and Rajesh, on the problems of implementing custom firmware . There are many other interesting sessions on the agenda ( I particularly look forward to KD's speech on the impact of Culture on Agile Practices - knowing KD, this promises to be a real blast), so if you are in Bangalore (or can get to Bangalore, don't miss the fun. (Registration will open later tonight).

Update : Registration for Agile India 2006 is now open.

The Storm Before the Calm

As of May the 6th of 2006 (the final day of the Agile India Conference, where I am speaking on "The Design And Implementation of Robotics Languages"), I am completely abandoning all connections to the "offshored software" world. No more agile/extremeprogrammng/tdd, no more j2ee, no more "offshore teams" delivering "superior business value" using "best practices" crap.

Paul Graham said,

"If it is possible to make yourself into a great hacker, the way to do it may be to make the following deal with yourself: you never have to work on boring projects (unless your family will starve otherwise), and in return, you'll never allow yourself to do a half-assed job. All the great hackers I know seem to have made that deal, though perhaps none of them had any choice in the matter."

Now whether I am or will ever be a "great hacker" aside, "you'll never work on boring projects, and in return, you'll never allow yourself to do a half-assed job." seems a fantastic principle to live by (it helps that I don't have a family I need to keep above the starvation line - Oh the blessed joys of bachelorhood :-) ).

Once you cross the age of 30, you are in the second half of your life. It is horrible enough that I spent one decade caught up in the "offshored software" paradigm. One decade - think of it and weep. A decade of dedicated practice would make you world class in anything you choose. One could be a world class athlete, musician, martial artist, dancer,writer, film maker... whatever! And I spent it writing "plonk this data off a database and put it on a set of dinky webpages" systems.

Shakespeare got it right when he made Macbeth say "And all our yesterdays have lighted fools, The way to dusty death."

And it is a very seductive bargain - in the beginning - money for nothing, multiple trips abroad (ignoring the fact that you are roughly in the position of a Sepoy in the Old army of the British Raj, occasionally sent abroad to shed your blood in the Empire's mysterious wars), all the trappings of a "successful" life. Time blurs and you wake up to find yourself either a second grade "project manager" type implementing commands from across the seas (and trying desperately to convince yourself that you are now in a superior position compared to the lowly blue collar techies, having forgotten what little you knew) or yet another burned out "techie" having no skills beyond "j2ee" or "perl and php".

Spending any more time in that world would mean I am frittering away what remains of my life. The Red Pill came just in time, methinks.

The number of genuinely good programmers and the amount of true innovation in India is astoundingly low, for all the talk of being the "21st century superpower". The "pseudo geek" profile many people unwittingly take on is very low on substance and very high on deception.

But it is not all doom and gloom. The less something exists, the more the opportunity to create it.

The mistake,I've found, is in thinking, like many Non Resident Indians do, that "I'll go home next year (and meanwhile, endure another year of meaninglessness)". Of course "next year" never comes. If one plans to "go home", the time to do it is right now.

From Snow Crash,

" Until a man is twenty-five, he still thinks, every so often, that under the right circumstances he could be the baddest motherfucker in the world. If I moved to a martial-arts monastery in China and studied real hard for ten years. If my family was wiped out by Colombian drug dealers and I swore myself to revenge. If I got a fatal disease, had one year to live, and devoted it to wiping out street crime. If I just dropped out and devoted my life to being bad."

I am way past 25, but "devoting my life to being bad" sounds good anyway.

The Importance of Having Friends Who Disagree

Paul Graham says, in one of his essays (emphasis mine) "....Why do you need other people? Can't you just think of new ideas yourself? The empirical answer is: no. Even Einstein needed people to bounce ideas off. Ideas get developed in the process of explaining them to the right kind of person. You need that resistance, just as a carver needs the resistance of the wood.

This is one reason Y Combinator has a rule against investing in startups with only one founder. Practically every successful company has at least two. And because startup founders work under great pressure, it's critical they be friends....."

In my experience, this need for "friendly resistance" extends to far more than creating startups. Every time you have a new idea, you need people you can bounce it off. To get any real benefit out of this process you need people with a complex combination of characteristics.

They should

1. have firm (but not rigid) opinions on their own
2. have logical reasons for those beliefs and be able to articulate them clearly
3. are driven by ideas and not ideology
4. not attach their egos to their opinions.
5. be willing to concede a valid argument even if it forces them to possibly re-examine their beliefs
6. know how to listen

Given all that, it is extremely difficult to find such "friends who disagree". I have been fortunate to have quite a few people around me, who have major disagreements with me, but are still friends (e.g see the discussion I had with KD in the comments section of my last blog entry).

Many people make the mistake of sorrounding themselves with people who think exactly like them and reinforce every idea or prejudice they have. This is a bad mistake and will often end up distorting the reality you see. One needs the corrective bucket of cold water in one's face once every so often. The only problem with surrounding yourself with bright people who think differently is that you may occasionally find that one of your ideas isn't as hot as you think it was or that one of your deeply held convictions is just plain wrong. This is dificult for some people because they make the rightness of their ideas a validation of their worth as persons. Blogging is a great way to expand this circle. But it can work both ways and one may end up with a "fan club" that just reinforces your prejudices.

So do a quick test right now. Make a list of your closest friends/acquintances/advisors. Then make a separate list of those who think totally differently from you, but are worth listening to anyway. Look at the intersection of these lists.

You may be surprised.

Yet Another Math Milestone

A few days ago, I discovered an error in the mathematics of a paper (on neural network optimization) being prepared for publication by a very eminent scientist. My drawing attention to this discrepancy in the proof has led to a total recasting of the approach to the problem and I will be now listed as a co-author of the paper.

Hmm yeah. Whatever. So how is this significant?

Well this is the first time I have used my skill in mathematics (vs my skill in programming) to contribute significantly to a scientific effort. In an old blog post I had theorised that the acquisition of "mathematical thinking" would follow a four step path. I said ..

1. the first step in mastering math is be to learn to read the notation. just like learning the syntax of a new programming language
2. the second is to grasp the reality expressed by the notation at a gut level , like understanding the paradigm and patterns lying underneath a programming language , like ,say ,beginning to grok "oo"
3. the third is to use that understanding to create new possibilities
4. the fourth is to use a programming language to embody and refine those possibilities , thus creating programs that do what has never been done before.

On this scale, after more than a year of very hard work, I seem to be half way through the third step. I am now able to "think in math" just as (good) programmers are able to "think in objects"(or functions, or whatever the underlying abstraction of the language is) to a point where I can examine the line of a mathematical algorithm and extrapolate it in into new directions to limited distances. As of today I am not able to think "freewheelingly" in math and my ability to think programatically far outstrips my ability to think mathematically, but these two modes of thinking seem to be duals of each other.

Programmatic thinking somehow seems to investigate the "structure" (and the structural integrity) of a solution to a problem while mathematical thinking focusses on the abstract essence of a problem and the "degree of rightness" of a solution.

One is not a substitute for other. Thus I am very doubtful that "merciless refactoring", for example, gives any real insight into the nature of a problem, like many "agilists" claim. At best it gives an insight into the structure of a proposed solution. This grants only an indirect perspective onto the problem space. I have seen many excellent programmers who were relentless practitioners of "agile" work in a domain (say retailing) for years and still not really understand the "essential nature" of the domain (vs creating good solution structures). In other words they function like draftsmen with no artistic spark, repeating the same patterns endlessly without any genuine understanding or insight. Frequent "domain hopping" , whereby you work on a retail project for a few months, then move to "banking" then to "insurance" and so on, helps to deepen this focus on purely programmatic skills till every project becomes a blur of "billable hours" and "database->gui->database" options.

The scientist/ mathematician/domain expert on the other hand often understands a domain very well and can isolate problems and even propose solutions but is often unable to translate those visionsinto reality for lack of professional level programming skills. To use the art analogy, this is like being able to visualize exquisite paintings, without the required skill to transfer that vision to paper.

The traditional solution to this seems to be for the domin expert and programmer to engage in "conversation" ( 'onsite customer' anyone? :-) ). But in my view this is like drawing a picture by putting a "visualizer" (who can't draw) and a "draftsman" (who doesn't have artistic insight)in the same room and asking them to produce a painting. It might work for Identikit pictures but you won't get any works of art.

Coming back to the original idea, Mathematics,like programming is very much a meta skill, in that one can use the expertise across multiple domains and problem sets. Unlike programming, mathematics often helps the practitioner to grasp the underlying essential nature of a domain.

To conclude, I am considering switching my (long planned) PhD to Applied Mathematics (from Computer Science), the reasoning being that I can always learn programming on my own while using the formal structure of a PhD program to hone my mathematical skill. Alternatively I could specialize in a field that combines mathematics and programming (like AI or Robotics). However that decision is still some way off. In the coming months, I hope to deepen my knowledge of both mathematics (Topology and Group Theory if anyone is interested) and programming (more AI and Programming Language Theory). I am now more convinced than ever that a combination of mathematical and programming ability is worth acquiring.

Wir sind alle Dänen

This is the rare "political" post. Those not interested, move right on.

Enough is enough.

In my opinion it is foolishness to initiate a "clash of civilizations". But if there is a battle line forming by virtue of some people insisting that their religious laws should apply to people of other religions (or no religion) then I know on which side I stand.

Anyone has the right to be "offended". That does not give them the right to kill, burn and issue threats. Fwiw, I do not believe in Islam, Mohammed, his "sacredness" or infallibility, etc etc. The people who protest most at this "disrespect" have laws in their own countries that officially forbid the practice of any other religion. If you carry a Bible or Bhagavad Gita into Saudi Arabia, it will be seized and shredded at the airport. How dare they protest against any "disrespect"? First practice this "respect to other religions" and then complain your religion isn't respected.

Kennedy once said "Two thousand years ago the proudest boast was civis Romanus sum. Today, in the world of freedom, the proudest boast is 'Ich bin ein Berliner.' All free men, wherever they may live, are citizens of Berlin, and, therefore, as a free man, I take pride in the words 'Ich bin ein Berliner!'"

Today, all free men, no matter where they live, are citizens of Denmark.

Wir sind alle Dänen.

Update: This post has generated a lot of feedback mostly by email (some people seem afraid to express themselves in public. hmmm...) .

One comment on this blog (subsequently deleted by the author) said "There's also the trap of 'they don't respect my religion/beliefs/whatnot so why should I respect theirs?'. That's the point where civilized conversation breaks down." .

This is a little naive. You can't have a conversation with people who don't follow the norms of "civilized" conversation and are not interested in a conversation. This most often happens with people who are convinced that their truth (religion/prophet/Holy Book/ideology/ whatever) is infallible. There should be no tolerance of the intolerant.

From an article by the editor of Jyllands-Posten (the Danish newspaper that published the cartoons) (emphasis mine)

"Has Jyllands-Posten insulted and disrespected Islam? It certainly didn't intend to. But what does respect mean? When I visit a mosque, I show my respect by taking off my shoes. I follow the customs, just as I do in a church, synagogue or other holy place. But if a believer demands that I, as a nonbeliever, observe his taboos in the public domain, he is not asking for my respect, but for my submission. And that is incompatible with a secular democracy."

So yes, this kaffir maintains that anyone who demands rights and respect that he is not willing to give to others is a hypocrite.

And while I may not (or may) be impressed with Islam or its Prophet, (that is my privilege in a free society), my stance is not anti-Islam but anti-hypocrisy.

People (of any religion) who say "respect my religion" should extend respect to other religions (and to atheists). There is no one way "civilized debate".

Old Laptop, Meet New Laptop

As you can see I am not too much of a photographer (the laptop screens are much duller in the photos than in real life. I have no clue why.) but the new IBM ThinkPad Z60m (on the right) is a very nice machine, much more so than the (now ancient) Compaq Presario 2100 (on the left).

Indians are cheap, not free!

When you are an independent consultant, you see a lot of crazy people. The most dreaded monster in this bestiary is that of the Non-Resident-Indian-Trying-To-"Develop Products"-In-Bangalore.

The game generally goes like this.

Some desperate Indian working in the USA on a regular job (note that he is NOT a venture capitalist, businessman etc) gets what he thinks is an "Awesome" idea. The he discuses this with some equally clueless friends and they hammer out some technology choices (j2ee , .net whatver). After a few weeks/months of this they decide they want the software done "cheap". Even companies which charge very little money per developer hour (say TCS) wouldn't give these folks the time of the day so they ask their friends in Bangalore to reccomend "top grade" independent developers.These poor innocents recomend people they know to be good developers and have left the corprorate rat race to work independently.

Now the fun begins. The Independent Consultant (IC from now) gets a call from the USA saying "MR X who you worked with in Company ... reccomended you as a top notch developer. We have this awesome idea that we need you to develop in "asp /.net" within 3 months. We also are looking for junior developers, testers and maybe a Project Manager. Do you think you can help us"? The consultant, if he has been independent for a while, has seen this dance many tims before and immediately asks "Do you guys have any money or is all this wishful thinking?"

There are a few different answers to this. Sometimes it is "yeah we can pay 5$ per hour for the first 24 weeks " ( I am NOT joking!!!). Or alternatively , "you see we are just starting out blah blah so we don't have much ready cash but we have all these highflying VCs lined up and if you can work free for the first 6 months then you have millions of dollars in equity".

Duh . Yeah Right.

Some of the more ... emmm .. subtle folks wants the IC to work for them without knowing what the "cool idea " is. After all a "cool idea" is very valuable right? We once encountered an "idea man" who offered to join our company as a "Project Manager" ("since it has been a long time since I coded anything") while looking for funding and in the meanwhile he would let us pay him an "industry standard" salary.

Stupidity is the one thing which REALLY has no limts.

To Teach is To Learn Twice ... And More

I am sometimes asked by clients to implement Artificial Intelligence based solutions to problems involving massive datasets, real time requirements etc. A non trivial part of this work is to transfer the knowledge of AI algorithms and the underlying maths to the client's programmers. After doing this a few times, I now have a clearer idea of some of the difficulties involved and some partial solutions for them.

1. Problem 1 : Transferring Mathematical Intuition Using AI effectively often demands a deep understanding of the mathematics underlying whatever AI 'paradigm'/algorithm you use. For e.g. to even think about whether Neural Networks are a good solution for a problem, one needs to understand Linear Algebra at very intuitive level. To many people maths == a set of equations or symbols to be manipulated. And this manipulation and re arrangement of concepts to achieve precise effects is something programmers are naturally good at and so this tendency is even more deeply rooted in good programmers.

   For e.g., a lot of us have learned the equation F = m * a , where 'F' is Force, 'm' is mass and 'a' is acceleration. And to get through our exams we treat this as some kind of pluggable system in which two quantities are known (or can be derived from the given data) and the third has to be derived. This looks fairly simple. To check whether you understand the reality underlying the equation ask yourself this - "Does Force cause acceleration? Or Vice versa? Or both? Why? When?".

   The use of equations to *calculate* a quantity is different from being able to think effectively in terms of the concepts underlying the equation. To use mathematics effectively one has to constantly translate between the world of mathematics and the domain of the problem. Very few people(including mathematicians) feel the need to acquire this skill. People who use mathematics to get work done (e.g. Physicists/ Astronomers/ Race Car Designers) acquire this skill out of sheer necessity. Forcing oneself to translate (and asking students to translate) back and forth between English(without using mathemetical terms) and Mathematics is a valuable exercise. (try doing this for a simple differential equation and you will see what i mean)

   Thus, it isn't possible for someone to grab a few equations out of the latest AI/Pattern Recognition/Robotics etc book and apply them straight away to solve real world problems. To even know what is possible takes an understanding of the mathematical undepinnings of the various "families" of AI algorithms and this intuition takes a long time(at least for folks like me) to acquire. If programmers are just shown some algorithms or mathematical proofs, it is very unlikely they will be able to program an effective system or even maintain a system to meet changing environments. This is complicated by the fact that intuition may take a few years to acquire but has to be transferred in a week or less.

   While there is no perfect solution to this, I find that one can transfer large "chunks" of intuition through carefully selected (sequences of) motivating examples. In my first "iteration" of teaching, I said (in effect), "o.k. here is the (basic) theory. This is the algorithm in pseudocode. Use this and you will get the results you want". Everyone seemed happy but the whole effort "thrashed " for quite a while before delivering the (astoundingly effective) results.

   So these days, I try to instill an understanding of the theory distinct from the programming effort. More of a "teach how to fish" while giving the student enough "fish" so he doesn't starve till he learns to fend for himself.

   Thus instead of just throwing out the algorithms for (say) prediction and monitoring of data streams using Markov models, I start with real world examples of prediction vs monitoring and slowly feed in the maths (equations first, trivial programs next, then proofs, then real world programs). Given sharp "students" (as most of the programmers I interact with on a daily basis are, (Thank God!)) this works suprisingly well. I sometimes feel I am spinning a rope bridge across yawning chasms but the notion of a "slice" through a system works just as well in mathematics as in agile "story card" based implementation. Once a few cables are thrown across, programmers feel brave enough to explore the abyss a few feet at a time without too much assistance.

2. Problem 2 : Getting beyond "right" and "wrong" Sometimes in spite of my best attempts at simplifying and communicating, my "students" (I think of them as peer programmers rather than as students, hence the quote marks) sometimes misunderstand and "screw up" the concepts and write strange programs or advance baffling arguments. In the beginning my response was "That is NOT what I said! THIS is what you need to do. PLEASE look at the examples/notes/code. Aaaargh!!!! Not Again! ".

   A more effective way is to try to figure out what the student's mental model is, faulty though it be. So now, instead of reacting to a totally off track argument by (mentally) banging my head on the nearest wall, I say to myself "What he says makes sense to him. So what mental/model thought processes has he adopted which causes him to see the world in a fashion that makes this argument logical?". The moment I can identify this precisely I am able to come up with an illuminating example that demonstrates the error. And sometimes it turns out they are on to something important and it is my perceptions that need fixing. "Learn Twice" indeed.

3. Problem 3 : Ability to Do != Ability to Teach Sometimes the "teaching" gets so fatigue-inducing that I think to myself "I could have coded up all this in one tenth the time it taks me to communicate to someone else". My client, (being wiser than I) insisted on the teaching approach. What I have realized that many people (including Yours Truly) know how to do many thngs but are not often able to explain how they do them or teach others to do likewise. When you teach others you have to know each concept in a crystal clear fashion and also grok all the inteconnections between the concepts. Teaching often involves re-arranging concepts in increasing order of complexity and interdependence, seeking real life and programming examples that illustrate each facet of a concept with great clarity and slowly transitioning into complex real world problems and solutions. After teaching others, I understand many things at a much deeper level than I used to.

   That being said, teaching consumes massive amounts of time. I have to work 10 hours or more to prepare for a three hour "lecture". Thus "teaching" conflicts with "doing". For the time being, I'll focus on being a programmer more than a "teacher" type but I wonder how others manage.

There is a lot more I could write but this post is long enough. In another entry I will look into the difference between "Real world" and "Toy" AI.

Some people really get it..

from Xooglers

Sergey once asked a large assemblage of Googlers what our greatest corporate expense was. “Health insurance!” was one answer shouted back. “Salaries!” “Servers!” “Taxes!” “Electricity!” “Charlie’s grocery bills!,” came back others. “No,” said Sergey. “Opportunity cost.” He explained that the products we weren’t launching and the deals we weren’t doing threatened our economic stability more than any single line item in the budget.

How insightful is that? No wonder Google has no problems hiring good people.

Laptop Wars 3 - And the Winner is

an IBM -Lenovo Thinkpad z60m. The model I chose is a marginally less powerful (and less expensive) than the one reviewed in the article I linked to, but I have added some goodies like a hot swappable battery. I paid less than 2k $ for a jazzed up to the gills Thinkpad.

The macbook finally lost out because (a)I don't want to pay good money for beta testing apple's new hardware architecture (The G4 powerbook is too underpowered, and (soon to be) abandoned by Apple anwyay, not to mention terrible display issues on some of them which Apple pretends not to notice). (b)reliability issues with a macbook are likely to be life sucking given the atrocious customer service here in India. If I lived in the USA or Europe I may have been more tempted by the macbook.

IBM has excellent customer service in Bangalore and I have taken out a 3 year warranty. (130$ vs 340$ for the macbook). The Thinkpad outmatches the powerbook in sheer ruggedness. Hopefully in a couple of years Apple will have worked the kinks out of their new hardware choices and I will wait till then to drink the Apple Kool Aid.

My new laptop is on its way to my friend (who lives in the USA) and I should get my hands on it in early March and I will post my experiences installing Linux on the new laptop then.

Laptop wars 2 - Macbook Pro Vs IBM Thinkpad

A brief update.

I am now thinking of buying either a MacBook Pro (if it comes out before Feb 28, the day my friend - who will bring it to Bangalore- leaves the USA) or an IBM/Lenovo Thinkpad (extremely well suited to Linux - multiple sources confirm that suspend to disk and power management work well and the Thinkpad has a reputaion for rugged construction). The Macbook will cost me about 2600$ (including taxes) and the Thinkpad should be about 1200$. The trouble with the macbook of course is that it is based on new hardware and Apple has historically had problems with new hardware with the 12 in iBook G3 having a particularly horrendous history.

In either case, I'll be buying the machine from the USA. Prices in India are insane with a surcharge of almost 25% on any model, the macbooks being particularly pricey. For that kind of price differential I could buy a 20 in cinema display in addition to the macbook. Anyone returning from the USA can bring in a laptop with no customs duty thus making it cheaper by about 25-30%. Duh! Am I the only one who thinks that is a particularly stupid way to run an economy? And India is going to be the 21 st century SuperPower? Yeah Right!

The Agile Religion and the need for Merciless Pragmatism

Does anyone else get the feeling that "Agile" is now a religion with its holy books, many patriarchs, its churches, and commandments? ("Thou shalt work in pairs"). It is just a set of useful practices, master them, adapt them, use what works, discard the rest and get on with life.

The moment 'agile' becomes some kind of religion with itinerant preachers of the Holy Word attempting to convert the unwashed heathen to the worship of the True God, its time to step back and take a hard look at what is being attempted. Kent Beck made a list of practices which he found useful. Later a whole bunch of marketing was thrown at it and we have the present situation where people speak of "true agile" , "distributed agile" "xp 2nd edition vs xp 1st edition" and so on.

Another factor often ignore is that most of the "agile" methodologies (to use the word loosely) have their origins in the world of enterprise programming where teams of programmers wrestle with coding up systems for banks, insurance companies etc. Thus there are a lot of assumptions built into the extreme programming and other agile methods which simply don't hold in other contexts.

As a simple example, think about "Customer" and "onsite customer" in the context of a massive open source effort like the Linux Kernel. These terms just don't make sense. Any efforts to twist the meaning of the word to mean "the community" etc just robs the word of any meaning.

There is an even deeper notion of the separation of "what to program" (the customer) from "how to program" (the coder).A "luminary" who spoke on Agile recently said "Programming is all about taking knowledge from others and converting it to code". (Really! I am NOT making this up!)

"Take knowledge from others". Bzzzt! Cluelessness Alert! This may be true in the practice of consulting/business app/enterprise programming etc today but not necessarily elsewhere (Kernel hacking, scientific programming, compilers, embdded programming etc)

I think this view of the programmer as some kind of "coding body" with the "domain knowledge" residing elsewhere is deeply embedded into many "schools" of "agile". More about this in another blog entry but this separation of "what" and "how" has many built in assumptions about the context in which programming occurs.

There are whole worlds of programming outside the "enterprise" world where the 'agile practices' apply very tenuously, if at all. And even within the enterprise programming domain, agile /xp/ whatever-the-latest-buzzword should be constantly (re) evaluated, adapted and modified, not adopted and propogated mindlessly (I have been guilty of such behaviour in the past and use this blog post to unreservedly apologize to my victims).

If I want some kind of religious experience I can go to my nearest Descent-Of-The-Holy-Spirit-Scream-and-Yell-And-Speak-In-Tongues- Church. When I go to work I want logic, pragmatism and rationality.

I have adopted the "constantly growing test suite","refactoring" and "continous integration" ideas from XP to my work and jettsioned the rest. It is simply impossible to "pair program", do "test driven design" etc in the context in which I work.

TDD is, in retrospect, an insane approach to design. But more about that in another blog entry.

To conclude, the "agile" schools of programming do have many useful ideas. But "agile" is neither a scientific theory (like Relativity) nor some kind of Divine Revelation. They are just a list of practices which work(ed) for some people in some contexts. Beware of "Agile Consultants", "Agile Enablers", etc. Lock that chequebook away and do some focussed thinking and experimenting. Pragmatism trumps religious fervor any day.

PowerBook Vs Linux Laptop

I have always wanted a powerbook. And with some "out of band" money coming in from a consulting assignment, I was planning to buy one. I actually wanted to buy one pronto and if I were not waiting for Steve Jobs's address tomorrow, I would probably have bought one.

After I got home I was reading some reviews of the PowerBook and came across this.

WTF?

I have to pay Apple a premium price so they can dump bad hardware on me? My Compaq Presario 2100 has been treated very roughly over the last 3 years and still works without a hitch.

I am now inclined towards getting a good Linux laptop and am now waiting for my friend (and Kernel Hacker) Mridul Jain to reccomend a good laptop for Linux. I particularly require hibernate (to disk not to RAM) to work well and flawless power management.

Later in the day I'll investigate the display problems in detail. I am hearing more and more horror stories about Apple's machines which makes me very leery of buying them in a country with terrible consumer protection laws.

Update : the PowrBook is definitely out of the picture. Linux forever!