Ravi Mohan's Blog

Monday, December 26, 2005

Year End Retrospective

Over the last year or so, I have worked alone. This was a nice change from being part of a "globally distributed team", attending stand up (and other) meetings, endless 'pairing' sessions, and other "agile" practices.

As the year draws to an end, I think it maybe worthwhile to explore and clarify what I learned and how I changed.

First, I am now utterly disgusted with 'off shored enterprise' coding. I was never really comfortable with the bland projects that wash ashore in this domain. Too often the projects were utterly boring, in spite of working with some of the best companies and people. The last year has been fantastic in terms of the challenge and complexity of programming and I hope I never have to work on a leasing/banking/insurance/blah database-to-web-and-back, C#-or-java, offshored-to-save-money-by-hiring-tonnes-of-cheap-coding_bodies project in my life again! So help me God!

Second, working on some extremely tough problems have taught me to tone down my enthusiasm for (a) "OO" and (b) "agile". Don't get me wrong, there is much that is valuable in both but these days I see their limits with painful clarity. Now, I see Objects as one (vs the only) way to tackle a problem and while I still follow some of the agile practices (like unit testing and Continous Integration), I am very skeptical of the value of "pairing" etc, outside the simplicities of the "enterprise app" domain.

Third, I am now very focussed on three major themes - (a)learning to create "tough" programs(kernel hacking is tough, creating a normal insurance app in j2ee is not), (b) applying "deep theory" to practical ends (applying Bayesian Classification to spam filtering vs design pattens/mock objects/what have you in a standard business app) and (c) working towards being *really* good at programming, in global terms (and not just be another cheap Indian offshore programmer, or even worse , manager).

More concretely, in the coming year I plan to deepen my knowledge of AI and Compilers and also learn to hack hardware,besides acquiring more mathematical understanding.

2006 looks glorious! Happy New Year all.

Update: Thanks to Teju's comment, I've thought about this a little more.Here is a quick checklist of questions

  1. Is your project interesting? Do you feel exhilarated or bored when working on your project?
  2. Are the people on your project there because of their expertise or are they there to increase billing? If you left the project would it be hard or easy to replace you? How unique are your skills?
  3. Do you learn new things every day or do you just do the same things over and over again? (this is characteristic of most offshored "enterprise" work, imo)
  4. If you had all the money you wanted, would you still choose to work on this project? Or would you do something else?

Update: Vivek Singh makes an excellent point. He says that complex enterprise systems that scale to m(b)illions of users are hard to create (well). I agree (while still believing that most apps that are outsourced to Bangalore have nowhere this level of complexity and are boring jsp/orm/db, "repeat these 3 steps a 100 times" type projects).


Tejaswi said...

Wish you a happy new year my friend.

A curious thing about Systems vs. Math programming is that Systems programming (Enterprise, Operating Systems, Fault Tollerant Distributed Systems, etc.) is inherently common-sense driven. You need to be aware of a few "technologies" that have been time tested, or are plainly obvious. The challenge comes in because of the human element in designing a huge system, with more than 1 person in a team, working on various parts, coping with change, etc, etc. which are not inherently parts of the problem.

In Math programming, the code is more than always a single for-loop, a few variables, some if conditions, and some arithmatic operators, and you are done. But what needs to be done with these operators is highly complex, and needs a mathematical understanding of the problem, the assumptions, etc.

So, as of now, I would rate system building in itself is an easier task than mathematical programming. But the people angle makes System building so hard, that it becomes almost intractable. One person cannot do it in sufficient time, 'n' people cannot do it without resorting to practices. Its the people angle that has made a simple system building concept into a billion dollar industry.

Of course, all this is not as simplistic as I opine here, and the two spheres of computing (better choice of word than programming, here, at least), intertwine many times and lead a hybrid angle to software on a whole. But even the hybrid angle is not devoid of people, and so we are perpetually screwed.

I kind of forgot my original point, but well...

Ravi said...

first of all I don't quite know whare the system *vs* 'math programming' came from!

Also I have nothing against working on a team. I am against *(me)* working on a team where people are hired not for their skills or abilities but as "coding bodies" to create massive offshored teams delivering dubious value.

I am not sure that having to work with people is a "screwing element".I think working with dumb pople, or even smart people on a dumb project really lowers your quality of life. The operative element seems to be dumb vs smart (people or projects). I am not very sure it is people vs alone.
To (ab)use ane xample I am sure very many people work on (say) the Linux Kernel. I am not sure if that would be as boring as some of our 'enterprise' apps.

I guess the whole point needs more thought!

Anonymous said...

I don't think hiring for "skills" and not for "abilities" is unique to an off-shore setting. It does get amplified in that setting. I have been going thro' that experience recently. I am neither using my skills or my abilities. And I am NOT in a offshore setting, nor are "most" of the people around me.

Call me an optimist, but I forsee more interesting work being done in India wrt software applications (mathematical, business and whatever else) than any other place in the world. The difference is that in India it will be done with the expectation of making huge riches later (typical VC money trying to hit jackpot) while in US it will still be driven by passion for technology or for building something unique even if there is no money at the end of it. The situation hopefully will change, but we Indians have tradiationally been less risk averse. We just need to get a paycheck at the end of the month.

But this change won't happen in any of the big firms relying on outsourced projects.

Hopefully kids graduating out of undergrad school will start thiking differently.


chandrakant said...

Very nice post. Quite a few things resonated with me here and I would like to point them out.
Firstly, I totally agree with toning down the OO and agile trumpet. After coding in Python and Ruby and dabbling a little bit with Scheme, I have come to realize that what we are writing is a basic set of instructions, so it doesnt make sense to arbitrarily take a higher-level concept like a "class" and say things like "everything must be object-based" etc. With Python, Ruby and Scheme I have noticed that we can write powerful programs using just functions and or modules (one other way of an arbitrary higher-level grouping). What matters in my opinion is how flexible the language is to accomodate a programmers thinking. I would say that the fact that java allows only classes as the most granular level is a pretty big limitation.
Secondly, I once read one of your blog posts that recommended reading a book called "Concepts Techniques and Models of Computer Programming". I didnt think of it that much at that time but after a while I ran into this paper called "Teaching Programming with the Kernel Language approach" by the same authors. I was fascinated by the concept of teaching programming language features by incrementally building a language, adding features as we go along - and I was hooked to Mozart. Now I have this urge to learn everything about compilers, kernels, processes etc. I am not from a computer science background so I pretty much started my programming career with just business application programming. Now I feel that although learning the lower lever details of parsing, compiling etc are not necessary to do a "good-enough" job, it is definitely required if you want to gain a solid understanding of programming.
So thanks for putting forth your ideas, it has made me think and reflect quite a bit. Out of the questions in your checklist, the most important are: "do you feel exhilarated to work on your project? Do you learn new things everyday?"
I do try to learn new things everyday but sadly they are not on my job. I hope to answer both with a big emphatic yes someday soon :)

Ravi said...

you rightly identify "risk aversion" as being teh root cause of INdia's misrable software innovation. Unlike you though I am not sos ure it will change fast. We'll see.

Chandrakant, If you need some really mind bending languages, try Haskell and/or Forth! Your brain will never be teh same again! You've been warned.