Yet Another Math Milestone

A few days ago, I discovered an error in the mathematics of a paper (on neural network optimization) being prepared for publication by a very eminent scientist. My drawing attention to this discrepancy in the proof has led to a total recasting of the approach to the problem and I will be now listed as a co-author of the paper.

Hmm yeah. Whatever. So how is this significant?

Well this is the first time I have used my skill in mathematics (vs my skill in programming) to contribute significantly to a scientific effort. In an old blog post I had theorised that the acquisition of "mathematical thinking" would follow a four step path. I said ..

1. the first step in mastering math is be to learn to read the notation. just like learning the syntax of a new programming language
2. the second is to grasp the reality expressed by the notation at a gut level , like understanding the paradigm and patterns lying underneath a programming language , like ,say ,beginning to grok "oo"
3. the third is to use that understanding to create new possibilities
4. the fourth is to use a programming language to embody and refine those possibilities , thus creating programs that do what has never been done before.

On this scale, after more than a year of very hard work, I seem to be half way through the third step. I am now able to "think in math" just as (good) programmers are able to "think in objects"(or functions, or whatever the underlying abstraction of the language is) to a point where I can examine the line of a mathematical algorithm and extrapolate it in into new directions to limited distances. As of today I am not able to think "freewheelingly" in math and my ability to think programatically far outstrips my ability to think mathematically, but these two modes of thinking seem to be duals of each other.

Programmatic thinking somehow seems to investigate the "structure" (and the structural integrity) of a solution to a problem while mathematical thinking focusses on the abstract essence of a problem and the "degree of rightness" of a solution.

One is not a substitute for other. Thus I am very doubtful that "merciless refactoring", for example, gives any real insight into the nature of a problem, like many "agilists" claim. At best it gives an insight into the structure of a proposed solution. This grants only an indirect perspective onto the problem space. I have seen many excellent programmers who were relentless practitioners of "agile" work in a domain (say retailing) for years and still not really understand the "essential nature" of the domain (vs creating good solution structures). In other words they function like draftsmen with no artistic spark, repeating the same patterns endlessly without any genuine understanding or insight. Frequent "domain hopping" , whereby you work on a retail project for a few months, then move to "banking" then to "insurance" and so on, helps to deepen this focus on purely programmatic skills till every project becomes a blur of "billable hours" and "database->gui->database" options.

The scientist/ mathematician/domain expert on the other hand often understands a domain very well and can isolate problems and even propose solutions but is often unable to translate those visionsinto reality for lack of professional level programming skills. To use the art analogy, this is like being able to visualize exquisite paintings, without the required skill to transfer that vision to paper.

The traditional solution to this seems to be for the domin expert and programmer to engage in "conversation" ( 'onsite customer' anyone? :-) ). But in my view this is like drawing a picture by putting a "visualizer" (who can't draw) and a "draftsman" (who doesn't have artistic insight)in the same room and asking them to produce a painting. It might work for Identikit pictures but you won't get any works of art.

Coming back to the original idea, Mathematics,like programming is very much a meta skill, in that one can use the expertise across multiple domains and problem sets. Unlike programming, mathematics often helps the practitioner to grasp the underlying essential nature of a domain.

To conclude, I am considering switching my (long planned) PhD to Applied Mathematics (from Computer Science), the reasoning being that I can always learn programming on my own while using the formal structure of a PhD program to hone my mathematical skill. Alternatively I could specialize in a field that combines mathematics and programming (like AI or Robotics). However that decision is still some way off. In the coming months, I hope to deepen my knowledge of both mathematics (Topology and Group Theory if anyone is interested) and programming (more AI and Programming Language Theory). I am now more convinced than ever that a combination of mathematical and programming ability is worth acquiring.

Wir sind alle Dänen

This is the rare "political" post. Those not interested, move right on.

Enough is enough.

In my opinion it is foolishness to initiate a "clash of civilizations". But if there is a battle line forming by virtue of some people insisting that their religious laws should apply to people of other religions (or no religion) then I know on which side I stand.

Anyone has the right to be "offended". That does not give them the right to kill, burn and issue threats. Fwiw, I do not believe in Islam, Mohammed, his "sacredness" or infallibility, etc etc. The people who protest most at this "disrespect" have laws in their own countries that officially forbid the practice of any other religion. If you carry a Bible or Bhagavad Gita into Saudi Arabia, it will be seized and shredded at the airport. How dare they protest against any "disrespect"? First practice this "respect to other religions" and then complain your religion isn't respected.

Kennedy once said "Two thousand years ago the proudest boast was civis Romanus sum. Today, in the world of freedom, the proudest boast is 'Ich bin ein Berliner.' All free men, wherever they may live, are citizens of Berlin, and, therefore, as a free man, I take pride in the words 'Ich bin ein Berliner!'"

Today, all free men, no matter where they live, are citizens of Denmark.

Wir sind alle Dänen.

Update: This post has generated a lot of feedback mostly by email (some people seem afraid to express themselves in public. hmmm...) .

One comment on this blog (subsequently deleted by the author) said "There's also the trap of 'they don't respect my religion/beliefs/whatnot so why should I respect theirs?'. That's the point where civilized conversation breaks down." .

This is a little naive. You can't have a conversation with people who don't follow the norms of "civilized" conversation and are not interested in a conversation. This most often happens with people who are convinced that their truth (religion/prophet/Holy Book/ideology/ whatever) is infallible. There should be no tolerance of the intolerant.

From an article by the editor of Jyllands-Posten (the Danish newspaper that published the cartoons) (emphasis mine)

"Has Jyllands-Posten insulted and disrespected Islam? It certainly didn't intend to. But what does respect mean? When I visit a mosque, I show my respect by taking off my shoes. I follow the customs, just as I do in a church, synagogue or other holy place. But if a believer demands that I, as a nonbeliever, observe his taboos in the public domain, he is not asking for my respect, but for my submission. And that is incompatible with a secular democracy."

So yes, this kaffir maintains that anyone who demands rights and respect that he is not willing to give to others is a hypocrite.

And while I may not (or may) be impressed with Islam or its Prophet, (that is my privilege in a free society), my stance is not anti-Islam but anti-hypocrisy.

People (of any religion) who say "respect my religion" should extend respect to other religions (and to atheists). There is no one way "civilized debate".

Old Laptop, Meet New Laptop

As you can see I am not too much of a photographer (the laptop screens are much duller in the photos than in real life. I have no clue why.) but the new IBM ThinkPad Z60m (on the right) is a very nice machine, much more so than the (now ancient) Compaq Presario 2100 (on the left).

Indians are cheap, not free!

When you are an independent consultant, you see a lot of crazy people. The most dreaded monster in this bestiary is that of the Non-Resident-Indian-Trying-To-"Develop Products"-In-Bangalore.

The game generally goes like this.

Some desperate Indian working in the USA on a regular job (note that he is NOT a venture capitalist, businessman etc) gets what he thinks is an "Awesome" idea. The he discuses this with some equally clueless friends and they hammer out some technology choices (j2ee , .net whatver). After a few weeks/months of this they decide they want the software done "cheap". Even companies which charge very little money per developer hour (say TCS) wouldn't give these folks the time of the day so they ask their friends in Bangalore to reccomend "top grade" independent developers.These poor innocents recomend people they know to be good developers and have left the corprorate rat race to work independently.

Now the fun begins. The Independent Consultant (IC from now) gets a call from the USA saying "MR X who you worked with in Company ... reccomended you as a top notch developer. We have this awesome idea that we need you to develop in "asp /.net" within 3 months. We also are looking for junior developers, testers and maybe a Project Manager. Do you think you can help us"? The consultant, if he has been independent for a while, has seen this dance many tims before and immediately asks "Do you guys have any money or is all this wishful thinking?"

There are a few different answers to this. Sometimes it is "yeah we can pay 5$ per hour for the first 24 weeks " ( I am NOT joking!!!). Or alternatively , "you see we are just starting out blah blah so we don't have much ready cash but we have all these highflying VCs lined up and if you can work free for the first 6 months then you have millions of dollars in equity".

Duh . Yeah Right.

Some of the more ... emmm .. subtle folks wants the IC to work for them without knowing what the "cool idea " is. After all a "cool idea" is very valuable right? We once encountered an "idea man" who offered to join our company as a "Project Manager" ("since it has been a long time since I coded anything") while looking for funding and in the meanwhile he would let us pay him an "industry standard" salary.

Stupidity is the one thing which REALLY has no limts.

To Teach is To Learn Twice ... And More

I am sometimes asked by clients to implement Artificial Intelligence based solutions to problems involving massive datasets, real time requirements etc. A non trivial part of this work is to transfer the knowledge of AI algorithms and the underlying maths to the client's programmers. After doing this a few times, I now have a clearer idea of some of the difficulties involved and some partial solutions for them.

1. Problem 1 : Transferring Mathematical Intuition Using AI effectively often demands a deep understanding of the mathematics underlying whatever AI 'paradigm'/algorithm you use. For e.g. to even think about whether Neural Networks are a good solution for a problem, one needs to understand Linear Algebra at very intuitive level. To many people maths == a set of equations or symbols to be manipulated. And this manipulation and re arrangement of concepts to achieve precise effects is something programmers are naturally good at and so this tendency is even more deeply rooted in good programmers.

   For e.g., a lot of us have learned the equation F = m * a , where 'F' is Force, 'm' is mass and 'a' is acceleration. And to get through our exams we treat this as some kind of pluggable system in which two quantities are known (or can be derived from the given data) and the third has to be derived. This looks fairly simple. To check whether you understand the reality underlying the equation ask yourself this - "Does Force cause acceleration? Or Vice versa? Or both? Why? When?".

   The use of equations to *calculate* a quantity is different from being able to think effectively in terms of the concepts underlying the equation. To use mathematics effectively one has to constantly translate between the world of mathematics and the domain of the problem. Very few people(including mathematicians) feel the need to acquire this skill. People who use mathematics to get work done (e.g. Physicists/ Astronomers/ Race Car Designers) acquire this skill out of sheer necessity. Forcing oneself to translate (and asking students to translate) back and forth between English(without using mathemetical terms) and Mathematics is a valuable exercise. (try doing this for a simple differential equation and you will see what i mean)

   Thus, it isn't possible for someone to grab a few equations out of the latest AI/Pattern Recognition/Robotics etc book and apply them straight away to solve real world problems. To even know what is possible takes an understanding of the mathematical undepinnings of the various "families" of AI algorithms and this intuition takes a long time(at least for folks like me) to acquire. If programmers are just shown some algorithms or mathematical proofs, it is very unlikely they will be able to program an effective system or even maintain a system to meet changing environments. This is complicated by the fact that intuition may take a few years to acquire but has to be transferred in a week or less.

   While there is no perfect solution to this, I find that one can transfer large "chunks" of intuition through carefully selected (sequences of) motivating examples. In my first "iteration" of teaching, I said (in effect), "o.k. here is the (basic) theory. This is the algorithm in pseudocode. Use this and you will get the results you want". Everyone seemed happy but the whole effort "thrashed " for quite a while before delivering the (astoundingly effective) results.

   So these days, I try to instill an understanding of the theory distinct from the programming effort. More of a "teach how to fish" while giving the student enough "fish" so he doesn't starve till he learns to fend for himself.

   Thus instead of just throwing out the algorithms for (say) prediction and monitoring of data streams using Markov models, I start with real world examples of prediction vs monitoring and slowly feed in the maths (equations first, trivial programs next, then proofs, then real world programs). Given sharp "students" (as most of the programmers I interact with on a daily basis are, (Thank God!)) this works suprisingly well. I sometimes feel I am spinning a rope bridge across yawning chasms but the notion of a "slice" through a system works just as well in mathematics as in agile "story card" based implementation. Once a few cables are thrown across, programmers feel brave enough to explore the abyss a few feet at a time without too much assistance.

2. Problem 2 : Getting beyond "right" and "wrong" Sometimes in spite of my best attempts at simplifying and communicating, my "students" (I think of them as peer programmers rather than as students, hence the quote marks) sometimes misunderstand and "screw up" the concepts and write strange programs or advance baffling arguments. In the beginning my response was "That is NOT what I said! THIS is what you need to do. PLEASE look at the examples/notes/code. Aaaargh!!!! Not Again! ".

   A more effective way is to try to figure out what the student's mental model is, faulty though it be. So now, instead of reacting to a totally off track argument by (mentally) banging my head on the nearest wall, I say to myself "What he says makes sense to him. So what mental/model thought processes has he adopted which causes him to see the world in a fashion that makes this argument logical?". The moment I can identify this precisely I am able to come up with an illuminating example that demonstrates the error. And sometimes it turns out they are on to something important and it is my perceptions that need fixing. "Learn Twice" indeed.

3. Problem 3 : Ability to Do != Ability to Teach Sometimes the "teaching" gets so fatigue-inducing that I think to myself "I could have coded up all this in one tenth the time it taks me to communicate to someone else". My client, (being wiser than I) insisted on the teaching approach. What I have realized that many people (including Yours Truly) know how to do many thngs but are not often able to explain how they do them or teach others to do likewise. When you teach others you have to know each concept in a crystal clear fashion and also grok all the inteconnections between the concepts. Teaching often involves re-arranging concepts in increasing order of complexity and interdependence, seeking real life and programming examples that illustrate each facet of a concept with great clarity and slowly transitioning into complex real world problems and solutions. After teaching others, I understand many things at a much deeper level than I used to.

   That being said, teaching consumes massive amounts of time. I have to work 10 hours or more to prepare for a three hour "lecture". Thus "teaching" conflicts with "doing". For the time being, I'll focus on being a programmer more than a "teacher" type but I wonder how others manage.

There is a lot more I could write but this post is long enough. In another entry I will look into the difference between "Real world" and "Toy" AI.

Some people really get it..

from Xooglers

Sergey once asked a large assemblage of Googlers what our greatest corporate expense was. “Health insurance!” was one answer shouted back. “Salaries!” “Servers!” “Taxes!” “Electricity!” “Charlie’s grocery bills!,” came back others. “No,” said Sergey. “Opportunity cost.” He explained that the products we weren’t launching and the deals we weren’t doing threatened our economic stability more than any single line item in the budget.

How insightful is that? No wonder Google has no problems hiring good people.

Laptop Wars 3 - And the Winner is

an IBM -Lenovo Thinkpad z60m. The model I chose is a marginally less powerful (and less expensive) than the one reviewed in the article I linked to, but I have added some goodies like a hot swappable battery. I paid less than 2k $ for a jazzed up to the gills Thinkpad.

The macbook finally lost out because (a)I don't want to pay good money for beta testing apple's new hardware architecture (The G4 powerbook is too underpowered, and (soon to be) abandoned by Apple anwyay, not to mention terrible display issues on some of them which Apple pretends not to notice). (b)reliability issues with a macbook are likely to be life sucking given the atrocious customer service here in India. If I lived in the USA or Europe I may have been more tempted by the macbook.

IBM has excellent customer service in Bangalore and I have taken out a 3 year warranty. (130$ vs 340$ for the macbook). The Thinkpad outmatches the powerbook in sheer ruggedness. Hopefully in a couple of years Apple will have worked the kinks out of their new hardware choices and I will wait till then to drink the Apple Kool Aid.

My new laptop is on its way to my friend (who lives in the USA) and I should get my hands on it in early March and I will post my experiences installing Linux on the new laptop then.

Laptop wars 2 - Macbook Pro Vs IBM Thinkpad

A brief update.

I am now thinking of buying either a MacBook Pro (if it comes out before Feb 28, the day my friend - who will bring it to Bangalore- leaves the USA) or an IBM/Lenovo Thinkpad (extremely well suited to Linux - multiple sources confirm that suspend to disk and power management work well and the Thinkpad has a reputaion for rugged construction). The Macbook will cost me about 2600$ (including taxes) and the Thinkpad should be about 1200$. The trouble with the macbook of course is that it is based on new hardware and Apple has historically had problems with new hardware with the 12 in iBook G3 having a particularly horrendous history.

In either case, I'll be buying the machine from the USA. Prices in India are insane with a surcharge of almost 25% on any model, the macbooks being particularly pricey. For that kind of price differential I could buy a 20 in cinema display in addition to the macbook. Anyone returning from the USA can bring in a laptop with no customs duty thus making it cheaper by about 25-30%. Duh! Am I the only one who thinks that is a particularly stupid way to run an economy? And India is going to be the 21 st century SuperPower? Yeah Right!

The Agile Religion and the need for Merciless Pragmatism

Does anyone else get the feeling that "Agile" is now a religion with its holy books, many patriarchs, its churches, and commandments? ("Thou shalt work in pairs"). It is just a set of useful practices, master them, adapt them, use what works, discard the rest and get on with life.

The moment 'agile' becomes some kind of religion with itinerant preachers of the Holy Word attempting to convert the unwashed heathen to the worship of the True God, its time to step back and take a hard look at what is being attempted. Kent Beck made a list of practices which he found useful. Later a whole bunch of marketing was thrown at it and we have the present situation where people speak of "true agile" , "distributed agile" "xp 2nd edition vs xp 1st edition" and so on.

Another factor often ignore is that most of the "agile" methodologies (to use the word loosely) have their origins in the world of enterprise programming where teams of programmers wrestle with coding up systems for banks, insurance companies etc. Thus there are a lot of assumptions built into the extreme programming and other agile methods which simply don't hold in other contexts.

As a simple example, think about "Customer" and "onsite customer" in the context of a massive open source effort like the Linux Kernel. These terms just don't make sense. Any efforts to twist the meaning of the word to mean "the community" etc just robs the word of any meaning.

There is an even deeper notion of the separation of "what to program" (the customer) from "how to program" (the coder).A "luminary" who spoke on Agile recently said "Programming is all about taking knowledge from others and converting it to code". (Really! I am NOT making this up!)

"Take knowledge from others". Bzzzt! Cluelessness Alert! This may be true in the practice of consulting/business app/enterprise programming etc today but not necessarily elsewhere (Kernel hacking, scientific programming, compilers, embdded programming etc)

I think this view of the programmer as some kind of "coding body" with the "domain knowledge" residing elsewhere is deeply embedded into many "schools" of "agile". More about this in another blog entry but this separation of "what" and "how" has many built in assumptions about the context in which programming occurs.

There are whole worlds of programming outside the "enterprise" world where the 'agile practices' apply very tenuously, if at all. And even within the enterprise programming domain, agile /xp/ whatever-the-latest-buzzword should be constantly (re) evaluated, adapted and modified, not adopted and propogated mindlessly (I have been guilty of such behaviour in the past and use this blog post to unreservedly apologize to my victims).

If I want some kind of religious experience I can go to my nearest Descent-Of-The-Holy-Spirit-Scream-and-Yell-And-Speak-In-Tongues- Church. When I go to work I want logic, pragmatism and rationality.

I have adopted the "constantly growing test suite","refactoring" and "continous integration" ideas from XP to my work and jettsioned the rest. It is simply impossible to "pair program", do "test driven design" etc in the context in which I work.

TDD is, in retrospect, an insane approach to design. But more about that in another blog entry.

To conclude, the "agile" schools of programming do have many useful ideas. But "agile" is neither a scientific theory (like Relativity) nor some kind of Divine Revelation. They are just a list of practices which work(ed) for some people in some contexts. Beware of "Agile Consultants", "Agile Enablers", etc. Lock that chequebook away and do some focussed thinking and experimenting. Pragmatism trumps religious fervor any day.

PowerBook Vs Linux Laptop

I have always wanted a powerbook. And with some "out of band" money coming in from a consulting assignment, I was planning to buy one. I actually wanted to buy one pronto and if I were not waiting for Steve Jobs's address tomorrow, I would probably have bought one.

After I got home I was reading some reviews of the PowerBook and came across this.

WTF?

I have to pay Apple a premium price so they can dump bad hardware on me? My Compaq Presario 2100 has been treated very roughly over the last 3 years and still works without a hitch.

I am now inclined towards getting a good Linux laptop and am now waiting for my friend (and Kernel Hacker) Mridul Jain to reccomend a good laptop for Linux. I particularly require hibernate (to disk not to RAM) to work well and flawless power management.

Later in the day I'll investigate the display problems in detail. I am hearing more and more horror stories about Apple's machines which makes me very leery of buying them in a country with terrible consumer protection laws.

Update : the PowrBook is definitely out of the picture. Linux forever!

Startup !

It is official. My friend Rajesh and I just shook (virtual) hands over an interstate phone connection on an agreement to create a startup company.

Right now there is just the intent to create a startup. We were talking of various ideas and suddenly it was "Hey why don't we start a company and do this instead of talking about it?--Yeah ! makes sense! Let's do it. Pick a day to launch--How about April Fools Day?--I like it! ".

So don't ask "What are you building? " (I wouldn't tell you anyway :-)). We have, as always, many ideas being kicked around and both of us are quite ...errr... unstable in different ways so what we are doing will probably get nailed down hours before the launch of the company (slated for Aril 1, 2006 ,the Paperwork Gods consenting). I would expect that there will be a stealth mode for quite some time after the official launch. I have quite a bit of stuff to wind down before April.

I am looking forward to this. The dynamics of a Jedi/Sith alliance promises to be interesting. Our strengths and personalities are very complementary.

Oh yeah. I am finally getting that 17" PowerBook as well :-). This is going to be fun.

What is your company's M/E ratio?

In simpler words, how many managers do you have per engineer? Look around you . Count the number of managers in your floor/division/office. Count the number of people who actually write or test code. Divide the former by the latter. That is your M/E ratio.

Now reflect on this snippet from this very interesting essay

"...For instance, you can do what Google does, and give each manager 200 direct reports, rather than the classic 7 or 8, so your management overhead grows with the log base-200 of the number of engineers, rather than the log base-8. That's the kind of clever thinking that comes from hiring math-y Ph.D.s, you know..."

As a thought experiment, imagine that your organization hired brilliant engineers and every manager had 200 of them reporting to him and that your CEO, CTO , and the seniormost people in the organization were all PhDs or well known hackers. What do you think might happen? I believe most "don't rock the boat and get my huge salary for pushing paper" type managers would flee in horror. With 200 reports, they won't have the time to harass people and act like the Dilbertian PHB. They might , you know, really add some value! I think you will end up getting really good managers (no, "good manager" is NOT an oxymoron. They are rare but they do exist). And more importantly with log-base-200 growth rate, the empire building politics playing manger will be relentlessly squeezed out.

In the last company I worked at, this ratio was about 1/8. Would have been much better, in retrospect, to have had an M/E ratio of about 1/20 (at least). Something to keep in mind if I ever get to create a multi billion dollar company.

I think the biggest favor Google has done the world is to prove that an engineer centric, low manager count, ignore-wall-street-and-please-the-customers company can make a lot of money. So no matter what their eventual fate, you can be sure a new generation of entrepreneurs will be paying attention. And learning.

Monads And Causality

Geenerally I don't post "just links" on my blog. But this time I'll make an exception, because these are very lucid explanations to difficult questions.

Question 1 : How do I explain Monads to a programmer without resorting to mathematics (Category Theory)? (This in turn came out of an email exchange with Peter Van Roy about explicit state vs monadic state (as used in Haskell))

The answer is here. This is the best answer I've ever seen which skirts Category Theory and various Haskellisms completely.

Question 2 : How can you really know if one factor causes another? (This came out of a conversation I had with a programmer working on a probabilistic classifer [I am working with them on integrating Machine Learning into their product suite])

Here are two sets of slides explaining the approach pioneered by Dr Judea Pearl at UCLA. To oversimplify, a new operator is added to probability theory with a separate semantics. This is a bit beyond standard probability but quite enlightening once you "get it".

On admitting what you got wrong

from Jeffrey Shallit's blog.

....Blondlot's tale is a cautionary one. By contrast, I offer a case where the proper behavior was displayed, from Richard Dawkins' 1996 Richard Dimbleby lecture:

A formative influence on my undergraduate self was the response of a respected elder statesmen of the Oxford Zoology Department when an American visitor had just publicly disproved his favourite theory. The old man strode to the front of the lecture hall, shook the American warmly by the hand and declared in ringing, emotional tones: "My dear fellow, I wish to thank you. I have been wrong these fifteen years." And we clapped our hands red. Admitting you are wrong is a basic part of the mathematical and scientific ethic. ...

I think it ought to be part of one's personal ethic too.

So here is a (tongue firmly attached to cheek) list of what I was wrong about. Be warned, this is a jumbled list of "lessons learned" and is intrinsically very subjective. Don't expect any coherence.

1. Ruby has a perl flavor.
2. Ruby is way better than Java.
3. A software developer in Bangalore can't do innovative, interesting work and make good money.
4. Developing Enterprise Software, especially "outsourced" software is a worthwhile career.
5. A knowledge of Mathematics/Programming Language Theory -insert other 'deep' subject here- is unnecessary to build good software.
6. Mathematics is boring
7. There is plenty of time to find out what to do in life and there is plenty of time to do it.
8. Objects are the best way to design programs.
9. Programmers are logical people and not prone to delusions and religious mania.
10. Agile Methodologies will work (better than what they are already doing) for most teams/products.
11. Most managers have a clue about their work/organizations/what their jobs are all about.
12. It is impossible to make massive amounts of money doing nothing/ by making other people miserable/delivering negative value (see above).
13. All managers are technically clueless.
14. In practice, Management, specially in the offshored software world, really adds value and is not a big con game.
15. It is impossible to combine being a manager (to get those massive salaries while doing nothing important) and a techie (to retain a of self worth and give life meaning).
16. Dilbert is fiction.
17. One person can't change the world
18. You have to be extra ordinarily talented to change the world.
19. The software industry is immune to political correctness fads because it is populated by logical, edcucated people (witness the noise around "diversity" or "we need more women in software")
20. Engineers can't master business and are bad at "reading the market", "understanding business value" etc and need support in the form of analysts/managers.
21. Some behaviours are so morally dubious that people who indulge in them exist only in fiction . Even if such people exist in the real world, they are very rare and you won't encounter them.
22. If you know a person well, you can predict what he or she will or will not do.
23. Talented people have a greater probability of being morally upright.
24. In real life the good guys don't always win.
25. Believing in God is better than being an atheist/agnostic.
26. Being an atheist/agnostic is better than being a believer.

Duh, I'd better stop now :-)

The perils of Java companies

Joel Spolsky writes on the perils of Java schools.Read it. It is an excellent article.

Now dwell on the fact that many of the same conclusions could apply to you if you have never developed a piece of software in a language other than Java or C#. Have you ever created a 5000 line + program in a language other than Java/C#? Do you know (as in wrote some decent programs in) any paradigms other than OO? Do you really know OO? (If you've learned only Java you probably don't). Do you work in a Java(C#) company?

With a new year dawning, it is a good time to make a resolution to improve your programming skills. Alternatively you could just "climb the ladder" and slump into "project management" or "iso/cmm".

Year End Retrospective

Over the last year or so, I have worked alone. This was a nice change from being part of a "globally distributed team", attending stand up (and other) meetings, endless 'pairing' sessions, and other "agile" practices.

As the year draws to an end, I think it maybe worthwhile to explore and clarify what I learned and how I changed.

First, I am now utterly disgusted with 'off shored enterprise' coding. I was never really comfortable with the bland projects that wash ashore in this domain. Too often the projects were utterly boring, in spite of working with some of the best companies and people. The last year has been fantastic in terms of the challenge and complexity of programming and I hope I never have to work on a leasing/banking/insurance/blah database-to-web-and-back, C#-or-java, offshored-to-save-money-by-hiring-tonnes-of-cheap-coding_bodies project in my life again! So help me God!

Second, working on some extremely tough problems have taught me to tone down my enthusiasm for (a) "OO" and (b) "agile". Don't get me wrong, there is much that is valuable in both but these days I see their limits with painful clarity. Now, I see Objects as one (vs the only) way to tackle a problem and while I still follow some of the agile practices (like unit testing and Continous Integration), I am very skeptical of the value of "pairing" etc, outside the simplicities of the "enterprise app" domain.

Third, I am now very focussed on three major themes - (a)learning to create "tough" programs(kernel hacking is tough, creating a normal insurance app in j2ee is not), (b) applying "deep theory" to practical ends (applying Bayesian Classification to spam filtering vs design pattens/mock objects/what have you in a standard business app) and (c) working towards being *really* good at programming, in global terms (and not just be another cheap Indian offshore programmer, or even worse , manager).

More concretely, in the coming year I plan to deepen my knowledge of AI and Compilers and also learn to hack hardware,besides acquiring more mathematical understanding.

2006 looks glorious! Happy New Year all.

Update: Thanks to Teju's comment, I've thought about this a little more.Here is a quick checklist of questions

1. Is your project interesting? Do you feel exhilarated or bored when working on your project?
2. Are the people on your project there because of their expertise or are they there to increase billing? If you left the project would it be hard or easy to replace you? How unique are your skills?
3. Do you learn new things every day or do you just do the same things over and over again? (this is characteristic of most offshored "enterprise" work, imo)
4. If you had all the money you wanted, would you still choose to work on this project? Or would you do something else?

Update: Vivek Singh makes an excellent point. He says that complex enterprise systems that scale to m(b)illions of users are hard to create (well). I agree (while still believing that most apps that are outsourced to Bangalore have nowhere this level of complexity and are boring jsp/orm/db, "repeat these 3 steps a 100 times" type projects).

What Mr Fowler Really Said

Martin Fowler's post on "Humane Interfaces" has set off a maelstrom in the 'blogosphere'. (He has referred to some of the more interesting posts in his updates to his entry).

As is typical in the world of software, the original points were soon lost in a cacophony of arbitrarily drawn battle lines (java vs ruby, open vs closed class systems, Java's List interfaces vs LinkedList class etc).

This is what I understood from Martin's post. (Heavily paraphrased. Don't blame Martin for any butchery I commit on his prose)

...There are two approaches to designing interfaces. One I (Martin) label the "Humane" approach. Another is the "minimal" approach. Here is an example..{ } .. The distinctions between these approaches are worth thinking about. I (Martin) lean toward the humane approach...

As readers of this blog know, I am all in favor of subjecting opinions to logical scrutiny. But I fail to see how this can be any clearer. It seems fairly obvious that (a) the example used is just that - one example and (b) either approach can be used in any language. Where did "Java vs Ruby" come from? What am I missing?

[rant] I am getting fairly irritated at the lack of clear thinking prevalent in the software world. Aaargh! [end rant]

Why a Lexus is Not An Acceptable Ferrari

Eric Kidd explains on his blog why he thinks 'Ruby is an Acceptable Lisp'.

His thesis is debunked very effectively in the comments section of his blog entry. What I find fascinating is the thinking behind this position.

Let us look at how Eric tries to assert his position that Ruby is as dense (or occasionally denser) than Lisp. He first creates an example of a lisp function , then writes a ruby equivalent which is marginally shorter, while brushing off the capabilties of lisp (reader macros in this case) because "it is not commonly done"!

First , this thinking is fallacious. The fallacy is called "Biased Generalization". From the Nizkor Project,

Description of Biased Sample

This fallacy is committed when a person draws a conclusion about a population based on a sample that is biased or prejudiced in some manner. It has the following form:

1. Sample S, which is biased, is taken from population P.
2. Conclusion C is drawn about Population P based on S.

And this is the error. Eric takes a lisp function that can be expressed concisely in Ruby and simultaneously dismisses Lisp's built in capability to match and exceed that brevity. He ignores lisp functions that can not be expressed as well (see the sentence "Yes, yield is a special-purpose hack, and yes, it only works for functions which take a single lambda but.." -- recreating "yield" in Lisp is trivial anyway-- ) making his conclusion suspect.

Blindly equating conciseness to "power" is also suspect( I used to think this way as well, not too long ago) as is the (unsupported) argument that Ruby is 'more powerful' than Python.

The deeper problem is that, when you say "X is an acceptable Y", what you are saying is "Y is superior and we all know that and we would like to use Y, but because of reasons a, b and c we make do with X".

As someone who uses Ruby and Lisp every day (the days of hacking "enterprise" code with Java and C# are gone! Praise the Lord!), I prefer to think of them as distinct tools in my tool kit. A screw driver can be used as a hammer in an emergency but that does not make it an "acceptable" hammer.

Now I'll stop dissecting Eric's arguments (he has a right to say whatever he wants to on his blog. My irritation stems from seeing some ruby zealots (here in Bangalore) make ridiculous claims about 'the power of Ruby' while having no clue about its limits or alternatives) and focus on my notions of what makes Lisp 'more powerful' than Ruby (and vice versa!).

I consider a language superior to another based on what is possible in one that is not easily possible in another.

From this perspective the power of Lisp stems from the "code is data" concept. Thanks to the s-expression notation (the regularity of the syntax is what powers macros), code is malleable to a degree not possible in Ruby. Yes, you can jump through hoops to get some of the same effect in Ruby (e.g the ParseTree module) but nothing Ruby can do can match the full power of Lisp to manipulate code and the environment just like any other piece of data.

Thus if you are working on domains or projects that need this kind of raw power, use Lisp. Also, contary to what most "enterprise" or "agile" people might tell you , "Object Oriented " is just one paradigm amongst many. In Lisp, one can seamlessly switch from one paradigm to another and can mix and match pardigms to precisely map the problem at hand. While one can approximate some of this in Ruby with enough effort, the language leans unashamedly towards OO with a pinch of functional programming.

The more interesting question, of course is how Ruby is more powerful than lisp. The answer has 3 parts - libraries, community and ease of installation and use. Yes, from a strict 'linguistic' view point things like ease of installation are not language features, but from a user's point of view a language that is theoretically powerful but in practice needs days of configuration and compiling to get a substandard version of what Ruby offers out of the box is unusable. (compare Ruby's Rails framework with the patchwork of CL-SQL, Uncommon Web, etc frameworks). I have little sympathy for "but it is better in theory" arguments. The Lisp community, as compared to Ruby's, is extremely arrogant and unhelpful and hypersensitive to any criticism of their language. Harping on "ANSI standards" doesn't help when an easy to use, open source (no I am not willing to spend big $$ to buy a closed source implementation) implementation with comprehensive, bug free libraries for networking, database access and other "low brow" tasks is non existent.

Thus I think Eric's conclusion ("In 2005, I’d think long and hard before choosing Lisp over Ruby.") matches my experience, though for different reasons. You could substitute "Python" for "Ruby" and it would still be valid.

I don't think "ease of hiring" is an issue unless you are writing crapola "enterprise" code, in body shopping (oops make that "consulting") mode where "plug compatible programming units" is the reigning concept, "agile" rhetoric notwithstanding. If the programmer you hire can't get upto speed in either Lisp or Ruby in a couple of weeks you are wasting your money anyway. But that is the topic of another entry.

Thus the sensible thing to do would be to learn Lisp and Ruby (and Python, and Forth, and smalltalk , and C )well.They are fascinating languages and will teach you different things. Alternatively you could study one and use that exclusively. The utility of a language (and thus, its "power") depends on what you the programmer can do with it, not on some abstract notion of "power".

In other words, learn to use Ruby to maximum effect. Learn to use Lisp to maximum effect. Learn to use --your favorite language-- to maximum effect. This will give you an awareness of what each is good for and where each falls down. Whether Ruby is an effective (whatever that means) Lisp or vice versa is irrelevant. A Lexus may suit your needs better than a Ferrari but it is not an "effective Ferrari" (though it may be better than that overhyped Ford (I did not say "Java/C#" :-P ).

Availability Update

I am not available for consulting assignments till (at least) the end of March 2006. Before writing to me to enquire about availability, do read this first.

I have just started a very interesting project which attempts to integrate machine learning into a massively successful product suite.The size of the data sets and the business requirements make this fairly challenging.

This, when added to my existing commitments (including some very interesting Open Source work), leaves me no time to take on more work till end of Q1 2006.

How Agile Dies

Bill Caputo writes

Perhaps the Agile of today -- represented by things like the Microsoft TDD article, the attempts by many consulting firms to make Agile a methodology (instead of set of principles like it used to be), and the increasing adoption of "Agile" by big IT -- deserves this criticism, but the Agile movement I remember held their observation as a premise.

*That* agile movement seems to already be dying out, but its echoes might still carry the message (if not recognize the messengers)

When you see people offering to "enable" half wit teams for outrageous amounts of money, and "agile" becomes another merketdroid buzzword on the same level as 'ISO 9000 certified' or 'CMM Level 5', you know you'd better call for a priest to administer the last rites.

Many companies in Bangalore are eagerly "developing an agile practice", which (but of course!) goes hand in hand with ISO 9000/CMM Level X /Whatever other BS 'methodologies' are already in place.

One of my friends, who works in 'Big IT', recently responded to "So what do you do?" with "I work in Organized Crime. I am part of an international syndicate that cons businessmen out of insane amounts of money and delivers little value in return".

Heh!