Sunday, November 04, 2007
The Algorithm Quotient of a Software Project
Still thinking of the DARPA Car Races, it strikes me that a project's dependence on algorithmic innovation for a successful outcome is in direct proportion to its "interestingness" froma developers point of view.
Think about it. All the cars that autonomously traverse 160 miles of desert or travel through an urban environment while obeying traffic rules have essentially the same hardware. The difference in performance is about how algorithmically sophisticated the *software* of each car is.
On the other end of the spectrum most "enterprise" projects are algorithmically trivial (I said "most" so if anyone's ego out there is tied to his knowledge of JSP or whatever, spare me the righteous indignation) and the most complex data structure used is often a hashtable.
I don't think this is always a valid heuristic, but it might be valuable to rank projects on the scale of the algorithmic complexity involved in each (which in turn probably depends on how close the project comes to exploring the unknown, as the DARPA race efforts do), and then by how compelling they are and see if there is a correlation.
In my experience, while enterprise projects have their own set of problems to be solved, they are rarely algorithmic in nature (vs economic or scheduling issues for e.g).
Another supporting factor might be to see if interviews for the interesting dev jobs focus heavily on testing algorithmic knowledge. At first glance this seems to hold as well. If you are interviewing for Google Research, you probably need to be great at algorithms. If you are interviewing for Wipro, not so much - a knowledge of the contemporary buzzwords (agile, xp, tdd, ruby dsl blah) should get you through.
Subscribe to:
Post Comments (Atom)
13 comments:
Ok, this time I tried to read the entry more carefully :)
I think what you're getting at makes a certain amount of sense. However, "iterestingness from a developer's point of view" is too broad a statement. It really does depend on the developer in question. I have the impression that your interests are very much in the realm of the most technically challenging problems you are capable of solving and to a large extent you are interested in solving these problems for their own sake - for the sake of demonstrating a neat hack.
As an analogy, we can try comparing software development with medecine. In software development, as in medecine, I believe it makes sense for all developers to first try to improve their skill in some fundamental areas: A certain amount of knowledge about writing maintainable code; a certain amount of knowledge of common algorithms and how to implement them; some hands on work with a range of different problems, say starting with business-type applications, moving to video games, and then trying to write something for a small embedded cpu; Once one has a baseline of skill, then one can specialize in the kinds of things one really wants to do. It's fair to say that a trauma surgeon in some areas is far more skilled than a general practitioner, but that doesn't mean that the trauma surgeon would necessarily make a very good GP.
@Vlad,
I largely agree with the "acquisition of base skills" idea before "specializing". So far so good.
I guess we differ on the need for "writing a business system" first. Imo you can learn these "base skills" on better projects from the get go, if one has a choice. There is nothing very unique about enterprise systems that teaches these skills better.
But, sure, no real harm done by trying a variety of projects. If one must work on enterprise systems, work for a year or so in a *good* enterprise software company like Thoughtworks to learn the maximum possible in the shortest period of time, from people who know hat they are talking about, then start working on what really interests you.
"to a large extent you are interested in solving these problems for their own sake - for the sake of demonstrating a neat hack."
I've heard this before :-) - the artificial distinction drawn between "technically interesting" vs "otherwise interesting" projects.
Some people say (I am NOT saying you - Vlad- is saying this) they do enterprise development because they value "delivering business value" more than technical accomplishment. 99% of the time people use this as an excuse for suboptimal technical skills.
To deliver business value one has to understand business.To understand business one has to practice business. The very few developers who understand (an area of) business just as well as or better than its fulltime practitioners may indeed have a valid point.
At least in the context of the kind of work done n Bangalore, enterprise developers work on whatever comes their way, leasing today, insurance tomorrow, energy trading the day after and spend years implementing "story cards" or whatever without the least understanding of the essence of and core issues in insurance or leasing or energy trading.
On this kind of "body shop" work, once the "core" basics are mastered, (transactions, databases, sessions some security and work flow, for most enterprisey projects) what remains is knowledge of particular frameworks and technologies (RoR, jsp, j2ee etc) which frankly aren't all that difficult to learn. And it is the very rare enterprise developer who *really* understands even the core issues - declarative programming and relational programming for databases, who has read Jim Gray's book on transactions and so on. Most enterprise devs "learn" by just cranking out boiler plate endlessly.
Hence the fact that this kind of "enterprise" work can be outsourced easily, while Google can't outsource its search algorithm research and you certainly can't outsource robotics work.
I have great respect for people who *choose to* work for years writing software for a particular business domain because they are interested in that domain and know the domain inside out.
I've met exactly two people like that though in more than a decade of working in software.
Hi Ravi,
When I suggested starting with business apps, what I really was getting at was the notion of "form-based apps." The only reason is that such programs let you actually get something moderately useful up and running quickly. Basically I think one should have some experiences developing a bunch of fairly different applications before going into the workforce. I taught some computer courses to 11-16 year kids at one point, and here are the types of things I remember them doing. It was all pretty neat, albeit the code itself tended to be rather ugly.
- the famous 'snake' video game
- an original fast-paced 80's
arcade game
- a simple contact manager that
used a file for storage
- a multi-player network game
with path-finding along the
lines of warcraft, but simpler
of course.
- an online pizza-ordering program
- an online chat program
- towers of hanoi, with
auto-solving
@vlad,
that's a neat list of projects for people just starting out. I think you ought to turn them into a blog post so people can link to it. I completely agree with the "write different things while learning" idea.
I think you're right that a lot of people who might say "Oh, I could do AI but I prefer doing business programming" are kidding themselves. High end technical work requires a lot of dedication. My experience suggests that the odds of someone doing the sort of work you aspire to get into are based on 1) talent; 2) opportunity; and 3) personality. Some people who have 1 and 2 may prefer to make money or move up the corporate ladder into positions of influence and leadership, and sometimes they just prefer not having the headaches of a more demanding job or they like being a bigger fish in a smaller pond as it were... It's like the doctor thing I was talking about - most GPs would probably not be able to become surgeons even if they wanted to, but some could, and ultimately being a GP is a respectable job if you are dedicated to it. Same with business programming, it can be a rewarding and challenging career - not the same level as AI research perhaps, but that's the way it goes...
"and sometimes they just prefer not having the headaches of a more demanding job or they like being a bigger fish in a smaller pond as it were..."
prefer is the key word in that sentence. As long as people actively choose, fully realizing the consequences, you get no argument from me.
Just wondering though how many programmers would say of themselves " I *couldn't do* [cool programming] even if i really wanted to? ". In my experience there is a lot of self delusion going around.
Also, I am not sure that the GP/Surgeon analogy *fully* holds.
Most good enterprise programmers (a good, though not all encompassing, chunk of those who work at Thoughtworks, say) *could* work on more interesting things if they wanted to and were willing to work towards it.
In Bangalore in particular, most people drift into enterprise programming without knowing what they give up or actively choosing enterprise programming. And that's a shame and a waste of talent and opportunity (Imo).
If India is ever to shake off the "outsourcing destination" tag and the image of "cheap but untalented" programmers, more people should work on cutting edge software. It is harder to do that from here, but not impossible.
You know I'm no expert, but I think India has enormous potential to develop a really innovative technical culture... But I would *very* tentatively venture to say that building up an entrepreneurial spirit and a willingness to "do stuff" may be perhaps more important than having the deepest technical wizardry... After all, look at many of the well known Internet companies in the US - Ebay, Yahoo, Amazon, Facebook, Hotmail (started by an american from India I believe!). These are all well within the "enterprisey developer" realm, albeit in their current incarnations they represent the higher echelons of such development. Certainly for India to build a culture analogous to what you find at MIT/Catech would be very nice too!
Vlad,
Good point.
Just a very minor quibble - . The first three companies you list (Ebay, Yahoo, Amazon) have *very* strong technical wizardry at the core of their systems and a lot of not so wizardous (?) code sorrounding the cores.
Facebook is a good example of what can be accomplished without *that* level of wizardry (though I am sure thay have interesting problems to solve) But working in php -> ugh!
Don't get me wrong, there have to be some talented people at say Amazon to architect their core system, but it's still essentially a shopping cart and the fundamental tool they use to manage things is putting everything in RAM. I don't want to trivialize what they do - the scale of it demands smart people to figure out how to get everything to work smoothly. Still, I would say it's a rather lower level of 'wizardry' than advanced AI techniques the study of quantum computation or advanced crypto... Even someone you admire like Norvig gives me the impression of being at the bottom of the heap where the true genius stuff starts to happen...
"Even someone you admire like Norvig gives me the impression of being at the bottom of the heap where the true genius stuff starts to happen..."
Vlad,
I hope you don't take this the wrong way, but you reaaaalllllyyyy have no idea what you are talking about here :-).
>I hope you don't take this the wrong way, but you reaaaalllllyyyy have no idea what you are talking about here :-).
I am sure I don't and I brought it up as a kind of thought experiment! I definitely didn't want to offend. I have read some of the stuff on Norvig's web site, and he seems like a very intelligent man. Compare with the stuff going on at www.scottaaronson.com just as an example though. Would you not say that there is again another level there?
@Vlad,
I think you misunderstood my remark.
I am ***not*** offended. Why should I be? I respect Dr N. I don't worship him. :-)
He doesn't need me to validate his achievements!
I guess I don't have this strict hierarchy of levels that you seem to hold in your head. Scott Aronson is very impressive. So is Peter Norvig. So is (say) Sebastian Thrun. Now who ranks where on your "coolness scale"? a PhD vfrom MIT vs a PHd from berkeley vs a PhD from Carnegie Mellon. Quantum Computing vs AI Vs Robotics? I guess I don't get it.
They all look like "very high level people" to me, beyond my ability to rank.
Maybe someone like Albert Einstein (soemone who changed a whole paradigm of science) is "cooler" but that is a fairly useless disticntion in this context.
There *is* a level difference between dumbasses like me and any of the people I mentioned above (and frankly this is the only "level gap" I am interested in). Between them, not so much. And if there *is* I can't make it out by any objective criteria.
You obviously have a different opinion (No harm in that - differences of opinion are valuable :-).
And since this is a fast deteriorating (in the sense of going into undecideable questions about very abstract phenomena, not in the sense of "you have no clue") conversation, I'll stop now.
"And since this is a fast deteriorating (in the sense of going into undecideable questions about very abstract phenomena, not in the sense of "you have no clue") conversation, I'll stop now."
very wise ;-).
Having said that, I applied the "ranking by algorithmic complexity" idea to some of my programming efforts.
The algorithmically complex ones (sadly, not a rarity, given the kind of work I do) were certainly the most valuable, though I wouldn't call them very "interesting" - I found some of those algorithms to be very hard to code!
But then that may be because I am not a professional programmer!
An interesting notion, this "ranking by algorithm"
Post a Comment