Ravi Mohan's Blog
Friday, April 27, 2007
IIT Grads ! (necessarily) == Good Programmers Redux
When I wondered some time ago why the best programmers didn't come from the population of IIT grads,
many people claimed I had all sorts of motives (ranging from jealousy to fear) for having such "heretical" thoughts.
Now some other folks seem to offer a degree of vindication.
From the latest Outlook magazine (You can read the first page of the article here - I am not sure how long the url will be valid). For those too lazy to click on the link, an excerpt
What has triggered the discussion within the fraternity is the concern expressed by two distinguished IIT alumni about the general decline in standards of excellence at the institutes in recent years. Their remarks have questioned the calibre of students who make it into the IITs by subjecting themselves to the killing rigours of coaching factories in places like Kota and Hyderabad. The alumni seemed to conclude that these products of coaching factories—who now form, according to Wikipedia, 95 per cent of students at IITs—had a blinkered approach to education, did
not recognise new ideas and had lost the spirit of inquiry and innovation. In short, elements that had built Brand IIT over the decades had now gone missing
The first salvo was fired by Tata Steel MD and IIT Madras alumnus B. Muthuraman. Speaking at a Ruby Union meeting of the pioneer batch organized at his alma mater in January 2007, he said that tisco was "not likely to recruit" IIT graduates any longer. He based his opinion on a recent interaction he had had with a set of final year students, who he said did not even know the authors of books they were supposed to have studied. He made many other related statements suggesting that IITs thrive only on their "past reputation" and that he actually preferred other college students who were amenable to company training.
Now before the frothing-at-the-mouth defenders descend on this blog, I am not saying that IIT grads are poor programmers. What I AM saying is that in India there is no correlation between programming ability and the school of the programmer. Any company that hires only IITians AND (boolean AND - do read this sentence carefully before sending off that hate mail) hopes to recruit impressive programming talent by doing so is asking for disaster.
An IIT is *not* Stanford or MIT or CMU. Not even close. If you doubt this, first take a look at a typical PhD paper (in say Robotics) from any IIT you choose vs say Stanford. Then tell me I'm wrong. Thanks in advance.
Tuesday, April 24, 2007
Learning From Sudoku Solvers
Ron Jeffries attempts to create a sudoku solver - here, here, here, here and here. (You really ought to read these articles. They are ummm...{cough} ...err.... enlightening.)
Peter Norvig creates a Sudoku Solver.
Compare. Learn.
Update: Discussion on reddit
somewhat peripherally related but similiar (this about the bowling game) discussion
The devgrind post
Update 2:
(Oct 2009) Peter Norvig spoke about this post in Peter Seibel's book, "Coders at Work". My reaction..
Update 3:
Peter Siebel author of "Coders at Work" weighs in. Some gems there.
"One thing I noticed, reading through Jeffries’s blog posts, was that he got fixated on the problem of how to represent a Sudoku board. He immediately started writing tests of the low-level details of a few functions for manipulating a data structure representing the 9×9 Sudoku board and a few functions for getting at the rows, columns, and boxes of the board. (“Boxes” are what Sudoku players call the 3×3 squares subsquares of the 9×9 board.)
Then he basically wandered around for the rest of his five blog postings fiddling with the representation, making it more “object oriented” and then fixing up the tests to work with the new representation and so on until eventually, it seems, he just got bored and gave up, having made only one minor stab at the problem of actually solving puzzles.
I suspect, having done a small amount of TDD myself, that this is actually a pattern that arises when a programmer tries to apply TDD to a problem they just don’t know how to solve. If I was a high-priced consultant/trainer like Jeffries, I’d probably give this pattern a pithy name like “Going in Circles Means You Don’t Know What You’re Doing”.
.....
(Norvig has) 7 definitions in 12 lines of code and he’s done with data representation. I’m not sure how much code Jeffries ended up with. In his fourth installment he had about 81 lines devoted to providing slightly less functionality than Norvig provided in the code we just looked at. In the fifth (and mercifully final) installment, he started adding classes and subclasses and moving things around but never presented all the code again. Safe to say it ended up quite a lot more than 12 lines; if he’s lucky it stayed under 120.
" Siebel's post is worth reading in its entirety.
Update 4:
Andrew Dalke's Problems with TDD (and the comments, some by eminent TDD proponents) reveal TDD's feet of clay.
Peter Norvig creates a Sudoku Solver.
Compare. Learn.
Update: Discussion on reddit
somewhat peripherally related but similiar (this about the bowling game) discussion
The devgrind post
Update 2:
(Oct 2009) Peter Norvig spoke about this post in Peter Seibel's book, "Coders at Work". My reaction..
Update 3:
Peter Siebel author of "Coders at Work" weighs in. Some gems there.
"One thing I noticed, reading through Jeffries’s blog posts, was that he got fixated on the problem of how to represent a Sudoku board. He immediately started writing tests of the low-level details of a few functions for manipulating a data structure representing the 9×9 Sudoku board and a few functions for getting at the rows, columns, and boxes of the board. (“Boxes” are what Sudoku players call the 3×3 squares subsquares of the 9×9 board.)
Then he basically wandered around for the rest of his five blog postings fiddling with the representation, making it more “object oriented” and then fixing up the tests to work with the new representation and so on until eventually, it seems, he just got bored and gave up, having made only one minor stab at the problem of actually solving puzzles.
I suspect, having done a small amount of TDD myself, that this is actually a pattern that arises when a programmer tries to apply TDD to a problem they just don’t know how to solve. If I was a high-priced consultant/trainer like Jeffries, I’d probably give this pattern a pithy name like “Going in Circles Means You Don’t Know What You’re Doing”.
.....
(Norvig has) 7 definitions in 12 lines of code and he’s done with data representation. I’m not sure how much code Jeffries ended up with. In his fourth installment he had about 81 lines devoted to providing slightly less functionality than Norvig provided in the code we just looked at. In the fifth (and mercifully final) installment, he started adding classes and subclasses and moving things around but never presented all the code again. Safe to say it ended up quite a lot more than 12 lines; if he’s lucky it stayed under 120.
" Siebel's post is worth reading in its entirety.
Update 4:
Andrew Dalke's Problems with TDD (and the comments, some by eminent TDD proponents) reveal TDD's feet of clay.
Thursday, April 19, 2007
Restored Comments
A quick perusal of blogspot's comment moderation page showed me that some comments were pending. Generally I get a notification in my gmail account, but for some reason these didn't make it.
Sorry.
All the comments have been restored. They are scattered across many posts. The best was Ryan Cooper's comment on "If Toyota outsourced .. ".
Ryan, I agree with most of what you say. I still think the Poppendieck's books are worthless. But reasonable people can disagree.
:-)
Sunday, April 15, 2007
Asok's Own Country
Recently, a friend (we'll call him M) invited me to his company's annual day party. First the CEO made a speech about how they (the folks who work at the company) were "creators of history and not just passengers". People clapped. Then the party proper started.
Well, sorta kinda. The Mistress of Ceremonies declared that "the bar is now open". One small problem though. You couldn't just walk up to the bar and get a beer. First you had to locate a "coupon distributor", get a "beer coupon" and then walk over to the bar and hand over the coupon and the bartender would hand you a beer. If you wanted another beer, you had to rejoin the line. (I am NOT kidding. This really happened. I was there, as were 200 or so others, so I have plenty of witnesses).
M, (by now thoroughly embarassed) joined the long line, and after inching forward for about 20 minutes, arrived at the head of a line and asked for "two coupons please". The person dispensing the precious scraps of paper. The Person Behind the Counter -PBC from now-, took a long and incredulous look and then said frostily "Only one coupon per head. sir".
M: "so, if a party of four want beers,all four have to line up and get their beers" PBC: "That's right" M: "ok... if they want another beer, they need to join the line again." PBC: "That's right sir!" M: "All four of them" PBC (beaming) : "That's right, sir! Here is your beer coupon" M(handing it back): "Do me a favor friend, Have a beer on me" (walks away).Creators of History? :-)
Sunday, April 08, 2007
Scribes and Programmers
A long time ago, there weren't too many people who knew how to write. So if John Ancient wanted something written he would visit the local scribe and pay a fee. I am not sure if the scribes ever formed a guild but I wouldn't be too surprised.
These days, if Joe Modern wants a program written, he needs to hire a programmer (or a bunch of programmers) who'll translate his needs into the appropriate incantations.
Imagine a world where a majority of people know programming - perhaps not enough to write an operating system kernel but enough to write a basic Rails application - say a ToDoList application. (I did say "imagine"). How would programmers evolve?
I think they will become largely "post technical" - a term used, sometimes pejoratively, to describe developers who've moved away from code-centric jobs. The term is mostly applied to folks who go on to become managers, an idea which is looked down upon by some developers.
In a world where most folks could program, post technicalism would be the norm. A large chunk of present day programming jobs would simply disappear. You can't get a job these days if you "know all 26 letters of the English alphabet and can form grammatically correct sentences in English", which is the level at which a large majority of "professional" developers "program". Knowledge of writing is required but not sufficient for a paying job today. A doctor needs to know how to write but he also needs to know much much more. Musicians, biologists, teachers - all could do with a basic knowledge of programming. Perhaps basic programming skill should be taught just like reading and writing are. Then the idea of writing a quick script to generate a graph from a database wouldn't be some kind of exotic skill , but a basic competence expected from educated people.
Would this disappearance (or submergence below the awareness level) of programming skills be a bad thing? Not really. Technological progress has rendered many jobs obsolete over the centuries and humanity has coped very well.
Also, there would still be a need for people having extremely high levels of programming ability, just as there are professions that need extremely high levels of writing/communication ability - think copywriters or diplomats. The folks who focus on algorithms and networks and compilers and other 'deep programming' would probably survive just fine.
Of course the writing analogy can't be stretched too far. Documents are created by one person or a few people at most, over fairly short periods of time. Most significant programming projects involve large groups of people working on the same codebase for significant amounts of time, sometimes decades. The main challenges (as measured in terms of completion and impact of the project) are about people issues - how to hire them, how to retain them, how to make them work together.
And this organizational difficulty of managing large groups of people and getting them to build something coherent is the space in which, at present, many snake oil vendors (think "lean " or agile") thrive. The presence of snake oil is often an indicator of a fundamental and intractable problem. So even if a large number of people knew how to program, the organizational aspects will need hands on resolution. Thus the "project manager" type person would be alive and well. The VB/j2ee/dotNet folks are the ones who'd disappear.
But is the "manager dude" the only evolutionary path available to code jocks? I think not. For example, the world of science and engineering (modulo software "engineering") would still be open. Yes, there would need to be a lot of study - mathematics and statistics and the sciences and particular engineering domains - you won't be able to grab a book called "Java X.Y in 24 hours" and get a well paying job. People wouldn't be able to make money doing the equivalent of writing other people's letters for them. But I suspect there will be a lot of fascinating problems to solve, even in our imaginary world.
Even in a world where everyone codes, the "path of the suit" would still remain the Dark Side. You would still need to think long and hard and try to balance gain and loss before setting out on that path.
Wednesday, March 28, 2007
Paul Graham On Enterprise Software
From his latest essay
"If you don't think you're smart enough to start a startup doing something technically difficult, just write enterprise software. Enterprise software companies aren't technology companies, they're sales companies, and sales depends mostly on effort."
I like the "just" in "just write enterprise software".
Tuesday, February 27, 2007
If Toyota outsourced its assembly line
to "consultants" who charged x$/hour/person_deployed, would we have a Toyota Production System today?
Think about it. Imagine someone like Wipro or Infosys providing the manpower (cheap offshore manpower if you will) for Toyota's assembly line. What would the probability be of Toyota coming up with a world beating manufacturing/design approach?
Many "Lean Software" advocates draw very tenuous relationships between a very unique achievement in the manufacturing world, try to spin their own practices (ahem - did I hear someone say " agile" ?) and say (in effect) "By the power of Toyota, Taichi Ohno and Kanban, you should do [consultant's favorite methodology practice item]".
I recently read a book on implementing Lean Software, by one of the "luminaries" of the methodology sales world , in which every chapter had the following pattern.
- A one paragraph example of how Cool Company [Toyota/Boeing/Google] etc did something cool [developed the 747, say].
- One aspect of this successful effort (failed efforts by the same companies, often using substantially similair methodologies, are ignored) - say, cross functional teams- vaguely resonates with "agile" or "lean software" principles.
- One or more teeny weeny "experience reports" from the author which sound like stories from the Brothers Grimm, full of archetypes and soundbite morals-to-take-home, of the calibre of "Don't talk to wolves on the way to Gramma's house"
- a "summary" that states point blank that software development could do much better if the unenlightened masses would adopt these ideas
Saturday, January 27, 2007
Tech Skills - An Investment Metaphor
A snippet of a (yahoo messenger) conversation with a friend who thinks differently.
"...
friend: i'm actually quite happy being a follower with technology... i'd rather build cool apps than hack frameworks. Its similar to my approach to investing... get on board a stock just as it makes new highs :-) and ride the wave.
..."
Viewing technical skill acquisition (including the choice of language/frameworks/tools) through a metaphor of investment of scarce resources (that would probably benefit from the use of a deliberate strategy to maximise returns) is illuminating.
The metaphor fits loosely, with plenty of holes in it. The "investment" made by a developer in a particular technology (and thus by implication in the underlying paradigm, community, toolset etc) is primarily the time he spends on mastering the technology, and secondarily the mental effort required to do so (Kernel hacking is much harder then Ruby on Rails). Mental effort is not strictly equal to time consumed. If you have 24 hours of free time you can use it to work through a RAILS tutorial or understand continuation passing interpreters. The latter requires significantly more mental effort/unit time.
Spending time, however, is very different from spending money. Money does not diminish when you don't actively spend it(ignoring the time value of money), but the time available to you diminishes at a constant rate till it is all gone whether you consciously spend it or not. With money, you can make a marginally suboptimal investment, watch it for awhile, then re extract most of the value and reinvest it in another instrument without too much effort, losing just the "time value" of the money. You can't decide to learn RoR for a year, then extract most of the time invested, and reinvest it in say Kernel Hacking. Time, once "invested" is gone forever.
Like most non professional investors in stocks and shares, developers mostly invest their time into particular technologies by default rather by deliberate strategy and many people find themselves having "6 years experience in C++ " or whatever, without knowing quite how they got there. So it is useful to look at the "correct" ways of investing in shares and try to extract any lessons to apply to tech skill acquisition.
My friend's technology strategy is "similar to .. get on board a stock just as it makes new highs .. and ride the wave." . In the world on Finance, such a strategy is called "technical investing".
When the objective of the analysis is to determine what stock to buy and at what price, there are two dominant methodologies.- value investing and technical investing.
from investopedia
- Fundamental analysis is a method of evaluating securities by attempting to measure the intrinsic value of a stock.
- Technical analysis is the evaluation of securities by means of studying statistics generated by market activity, such as past prices and volume. Technical analysts do not attempt to measure a security's intrinsic value but instead use stock charts to identify patterns and trends that may suggest what a stock will do in the future.
Saturday, January 20, 2007
Comics as Literature
"Comics" bring to mind Mickey Mouse, Archie, Superman and other "read and throw" reading. But there is no reason why the presence of pictures need to limit the range or quality of the story. Stretching one's mind, it is even possible to imagine truly excellent stories conceived and distributed in comic format.
"V for Vendetta" was originally released as a comic. The film's the director made it a "chick flick" with V falling in love with Evie (gag!! Anyone who had read the comic had their heads explode at this "romantic" development in the film).
Constantine's lush and rich multi-realm occult backstory was dumbed down into a monochromatic Christian good vs evil fairytale in the movie. Both stories are the worser for conversion into film.
If you'd like to experiment with comics for adults, try Neil Gaman's Sandman series or Alan Moore's Watchmen. The latter was the only comic ever to win a Hugo award or be featured in Time magazine's list of 100 best English novels. It is a really good book. Even within the standard comic book framework, read "The Killing Joke" or "Arkham Asylum - Living Hell" for examples of what a creative team can do within the constraints of established backstory. The former explores the dynamic of the Joker/Batman animus (The sequel of Batman Begins is rumored to be based on this comic , at least as far the Joker's characterization goes) and the latter explores what happens when a rogue investment banker attempts a "not guilty by reason of insanity" plea to avoid jail and lands in Arkham Asylum. Batman is seen in about 3 panels in the book. The rest of the story is set almost completely inside the walls of Arkham and in spite of a slightly uneven "summon the Devil" subthread is a good read.
PS : the one series you want to avoid is Alan moore's Promethea. This is a story full of feminist nonsense splattered over a half baked understanding of the Jewish Tree of Life, and even some (equally halfbaked) Indian mysticism!
Thursday, December 07, 2006
Shining Darkly
My friend Amit Rathore wrote an excellent article on transitioning from a developer to a manager.
Richard Hamming's famous speech has excellent advice on the (similar) manager vs scientist decision that many researchers struggle with.
" ...Question: Would you compare research and management?
Hamming: If you want to be a great researcher, you won't make it being president of the company. If you want to be president of the company, that's another thing. I'm not against being president of the company. I just don't want to be. I think Ian Ross does a good job as President of Bell Labs. I'm not against it; but you have to be clear on what you want. Furthermore, when you're young, you may have picked wanting to be a great scientist, but as you live longer, you may change your mind.
For instance, I went to my boss, Bode, one day and said, ``Why did you ever become department head? Why didn't you just be a good scientist?'' He said, ``Hamming, I had a vision of what mathematics should be in Bell Laboratories. And I saw if that vision was going to be realized, I had to make it happen; I had to be department head.''
When your vision of what you want to do is what you can do single-handedly, then you should pursue it. The day your vision, what you think needs to be done, is bigger than what you can do single-handedly, then you have to move toward management. And the bigger the vision is, the farther in management you have to go. If you have a vision of what the whole laboratory should be, or the whole Bell System, you have to get there to make it happen. You can't make it happen from the bottom very easily. It depends upon what goals and what desires you have. And as they change in life, you have to be prepared to change.
I chose to avoid management because I preferred to do what I could do single-handedly. But that's the choice that I made, and it is biased. Each person is entitled to their choice. Keep an open mind. But when you do choose a path, for heaven's sake be aware of what you have done and the choice you have made. money life and you takes your choice.
Even worse is the developer who really likes being a developer and is or can be very good at it, but chooses to become a mediocre, unhappy manager because "that is the only way to make more money". These people deserve pity, but no sympathy.
- Don't try to do both sides.
Tuesday, November 14, 2006
Thinking as a learnable skill
If I remember correctly, it was in one of Edward de Bono's books that I first encountered the idea that thinking might actually be a skill, and thus learnable (and improvable by practice).
Over the years, this belief has been reinforced and I was forced to pull it all together yesterday when a friend asked me "How can I improve the quality of my thinking"?
This is my answer, fwiw.
First learn critical thinking. Critical Thinking teaches you to evaluate an argument and separate chaff from grain. Suppose someone were to state, for e.g "Agile is cool and we'd like to adopt it in the next project", how do you react? To learn critical thinking one could go through a course in logic, learn all the different logical fallacies and then try to apply them to real life conversations and claims. But Critical Thinking - An Introduction offers a gentler path. Just make sure to skip most of Chapter 1. Just read Section 11.4, 1.2 and 1.3 an move on to Chapter 2.
I've never seen a better book that teaches one how to cut through the layers of BS and get to the core of an argument. The Craft Of Argument is a good book as well, though oriented more towards writing good papers and articles.
Second, learn Lateral Thinking. Dr. de Bono,the inventor of the method, has written many many useless books and a couple of brilliant ones. The latter contain some powerful ideas.
Dr de Bono's central contribution is the notion of a "thinking tool", analogous to tools like a hammer or a screwdriver - each tool having a context of use and a particular effect. The books to get are Lateral Thinking For Management(non management folks can ignore the "for management". It just means that he uses management situations to illustrate various thinking tools. Nothing prevents you from using them in your own domains) and Serious Creativity. Both are chock full of systematic practical lessons about how to think in different situations.
Third, I'd suggest a brilliant thinking tool - Thinking Gray - from A Contrarian's Guide To Leadership by Steven Sample. The book deals with a lot more than thinking, and in addition is very hard to find (Thank You Learning Journey, for getting me a copy) and so I'll briefly explain "Thinking Gray".
In essence, Thinking Gray (TG from now) asks you not to make a decision on something, or choose a side in an argument till you absolutely have to. Sounds simple doesn't it ? However, the human mind has tendency to converge to a decision and classify data into binary categories (good/bad, black/white, friend/foe) and take sides in an argument. For e.g everyone has an opinion about (say) the American invasion of Iraq. Some may think it a "good" thing and others may decide that the war is a "bad" idea. The TG practitioner, while listening to all the data, refuses to take a binary stand till it becomes absolutely imperative that he does so.
This refusal to decide lets you avoid three dangerous tendencies - (1) the tendency to filter incoming data to support a pre decided conclusion (2) the tendency to vacillate between two conclusions, depending on who you last spoke to (and how persuasive he was) and (3) most dangerous, the tendency to slant your beliefs towards what you think people around you believe.
Steven Sample precisely distinguishes TG from skepticism. The skeptic has two mental "buckets" - one labelled "I believe" and another labelled "I don't believe". The skeptic's policy is that nothing goes into the "I believe" bucket without proof or a logical, convincing argument. Everything, by default goes into the "I don't believe" bucket and stays there till logic or proof moves it into the "belief" b. The TG practitioner on the other hand, has no "buckets".
One thing the TG practitioner has to be careful of is that other people may get confused about your "true" beliefs and intentions and may think that you agree with what they say because you seem to understand what they say and ask clarifying questions and so on.
Thinking Gray can be enhanced by projecting the multiple conclusions forward into the future (thus providing a rich tapestry of possibilities) and backward into the past(allowing multiple interpretations) . Combined with Paul Graham's "running upstairs" or "staying upwind" concept, TG allows you to navigate an uncertain future in a stress free fashion and maximize productivity in the present. More on this later.
To generate ideas, in other words, to be creative, de Bono's thinking tools work very well. I recently came across an approach called TRIZ. The basic idea seems to be worth investigating, but during 2003/2004, TRIZ ( and the associated concept ARIZ ) became the fad of the day just as "agile" and "lean" seem to be today's. Thus there is a lot of nonsense written about them as various authors bought out books and a lot of consultants were selling "ARIZ enablement" and hiring themselves out as "ARIZ coaches". As a result, there is a lot of worthless literature out there and some effort has to be extended to dig out the core of ARIZ and TRIZ. As soon as I get some free time(a precious commodity these days), I will look into this.
To conclude, all these methods treat thinking as a learnable skill and the one problem with this model is that to get good at thinking, as with finding the route to Carnegie Hall, practice is the only way. And practicing thinking is an odd concept for a lot of people.
Sunday, November 12, 2006
Karma Capitalism?
Sometimes, I toy with the idea of getting an Ivy League MBA. And then I come across something like this (via Evolving Excellence).
...For the members of the Young Presidents’ Association, meeting in New Jersey, this was no ordinary leadership seminar. They were being imbued with the values of the Hindu philosophy of Vedanta, by its most venerable proponent, Swami Parthasarathy. ...
On the syllabus at Harvard, Kellogg, Wharton and Ross business and management schools is the Bhagavad Gita, one of Hinduism’s most sacred texts.
....
He began studying the Bhagavad Gita, and has spent the past 50 years building a multimillion pound empire through explaining its practical benefits to wealthy corporations and executives.
He has recently returned to India from America where — in addition to the Young Presidents’ Organisation — he lectured students at Wharton Business School and executives at Lehman Brothers in Manhattan. His tours are booked well beyond next year, and will include Australia, New Zealand, Singapore and Malaysia.
While traditional business teaching has used the language of war and conquest, Parthasarathy uses the Bhagavad Gita to urge his students to turn inwards, to develop what he calls the intellect, by which he means their own personal understanding of themselves and the world, and to develop their “concentration, consistency and co- operation”.
The technical folk do have their religious fads, but compared to such arrant nonsense, they are the very epitome of logic and reason.
If the Ivy League schools and Lehmann Brothers and other corporations believe this nonsense enough to enrich "guru" (gag!!) Parthasarthy by many millions of pounds, why should I get an MBA? Hmmm on the other hand, that is precisely the reason why I should get one?
If anyone reading this blog has an Ivy League MBA and/or works on the Street, could you please explain how such rackets work? I can't make any sense out of this. Any help appreciated.
Friday, October 27, 2006
Cracking the GRE
Update: Analytical Score updated.
I wrote the GRE a few days ago.
Scores -
- quant- 790/800
- verbal 800/800 (heh!)
- Analytical Writing -
awaited (I am not too worried about this).- 6.0/6.0
Wednesday, October 04, 2006
Listening to Steve Yegge (and skirmishing with Obie)
Steve Yegge's blog post titled "Good Agile, Bad Agile" seems to have ignited one of those periodic feeding frenzies in the blogosphere. The comparison he makes between Agile and Scientology seems to have touched a nerve in many "agilists". The comments on Steve's post reveal a good deal of defensiveness, google envy and the occasional reasoned response.
Summarizing Steve's "anti agile" points (from a very long post), he says
- The Chrysler Project, which was the exemplar for agile projects failed badly.
- Mandatory 100% pairing sucks bigtime.
- Agile isn't scientific. (at best, any proof for "improved productivity" is anecdotal).
- There are a lot of flaky "agilists" trying to be "coaches" and sell books and seminars.
- These "seminar agilists" create an artificially polarized world where any process that isn't Agile(TM) is "waterfall" or "cowboy".
- Agile is very date driven, ignoring the human variations in productivity.
- If anyone puts forward an alternative process , the "defenders" of agile use one of two responses to oppose it
- "all the good stuff he described is really Agile"
- "all the bad stuff he described is the fault of the team's execution of the process"
And Oh, Steve, keep writing!
Sunday, September 17, 2006
Spreadsheets and Self Sabotage A Case Study
This is a true story. Names have been changed to protect the innocent (and the guilty).
Once upon a time,long long ago, in a faraway land, there was this company, a typical enterprisey company we'll call BankCo (in reality the company doesn't work in banking, the domain is similar but different, and I am just obfuscating things as explained above). BankCo was running all its "enterprise" operations off a set of excel sheets and at some point decided to redo it "properly". The folks at BankCo were essentially businessmen, not programmers so they decided to get some consultants at the cutting edge of technology. Enter AgileCo. AgileCo decided to rewrite the business logic as a 3 tired OO app using all the latest blub technologies.
The result? the BHS project (in AgileCo most projects have three letter acronyms) has frustrated developers, grumbling clients and absolutely yucky code.
Now there were a lot of things the team did wrong. Fundamentally the project was doomed the moment the team was "volunteered" for such an aggressive schedule (committed to the client by the sales manager, "agile planning" notwithstanding) but the thinking might have been that "hey we just have a bunch of spreadsheets, how hard can it be to convert them into a 3 tiered Blub app" . The (non intuitive) correct answer is "potentially very very hard".
Spreadsheets are very subtle beasts. They just look very simple. There are these rows and columns of cells and even non programmers can quickly create useful "programs" so it should be fairly easy to translate a set of spreadsheets into OO programs right?
Wrong.
For one, a spreadsheet is essentially a functional program.[warning - pdf]
[Excerpt]
"It may seem odd to describe a spreadsheet as a programming language. Indeed, one of the great merits of spreadsheets is that users need not think of themselves as doing “programming”, let alone functional programming — rather, they simply “write formulae” or “build a model”. However, one can imagine printing the cells of a spreadsheet in textual form, like this:
A1 = 3
A2 = A1-32
A3 = A2 * 5/9
and then it plainly is a (functional) program. Thought of as a programming language, though, a spreadsheet is a very strange one. In particular, it is completely flat: there are no functions apart from the built-in ones(*). Instead, the entire program is a single flat collection of equations of the form “variable = formula”. Amazingly, end users nevertheless use spreadsheets to build extremely elaborate models, with many thousands of cells and formulae."
Simon goes on to state how he and his colleagues worked on removing the "procedural abstraction barrier" in the functional language embodied by a typical spreadsheet. (There are folks working on integrating more powerful functional languages and other paradigms into spread sheets.
The key point is that a spreadsheet is essentially functional. Besides, a spreadsheet propogates changes in values and formulae automatically, unlike imperative and oo languages (e.g if the cell A1 is set to 5 instead of 3 in the above sample, A2 automatically becomes -27).
Besides value propagation , the 2 dimensional spatial relationships between spreadsheet cells makes it unlike most programming interfaces in existence today.
" One of the most successful end-user programming techniques ever invented, the spreadsheet, uses what is essentially a functional model of programming. Each cell in a spreadsheet contains a functional expression that specifies the value for the cell, based on values in other cells. There are no imperative instructions, at least in the basic, original spreadsheet model. In a sense each spreadsheet cell pulls in the outside values it needs to compute its own value, as opposed to imperative systems where a central agent pushes values into cells. In some sense each cell may be thought of as an agent that monitors its depended-upon cells and updates itself when it needs to. As Nardi (1993)(Nardi 1993) points out, the control constructs of imperative languages are one of the most difficult things for users to grasp. Spreadsheets eliminate this barrier to end-user programming by dispensing with the need for control constructs, replacing them with functional constructs. "
[ Link
Thus, when you try to convert the functionality of a complex set of spreadsheets to a blub-dot-net program, you are, in essence translating from a functional language with value propagation to a very different paradigm (OO in this case) and also adding things like persistence and multi user interaction which a normal spreadsheet does not have to deal with. This project needs competent developers who have enough resources (including time) to do a proper job,and are expert in both functional and object oriented programming with a keen awareness of the chasm between the two paradigms and a knowledge of how to bridge it. You wouldn't have much use for a German to English translator who didn't know any German, would you?
Besides this technical misstep there are a host of others that make AgileCo's BHS project an interesting case study.
The first is the premature time commitment to clients, bypassing the essence of AgileCo's favorite planning methodology. This is the killer mistake. The second is an arbitrary team structure with unclear roles and shaky team buy in. The third factor worth discussing is how the dominant methodology at AgileCo discourages deep thought by developers and encourages them to grab a "User Story" and hit the keyboard as soon as possible. This eliminates the chances of success for finding a technical approach that attacks the core of the problem. The fourth is how a totally unnecessary onshore/offshore team split adds dysfunctional management, unnecessary paperwork and communication overhead. The fifth and most crucial insight is about how dumb management can totally screw up a project's chances of success beyond repair.
Postscript:- The team is now on deathmarch mode and works on Saturdays and some Sundays. Of course the managers don't work on weekends. The brighter developers are looking for other jobs.
(*) Most spreadsheets have some kind of extension mechanism. In Excel, for e.g. you can use VB script to write your own extensions. But this means (a) and end user has to learn a programming language and (b) subtle errors can creep in when you mix imperative and functional code. Debugging spreadsheets is a nightmare at the best of times.
Saturday, September 09, 2006
Rules For Failure #1 Create a group
I've been notcing that people who never get anything done seem to follow a few (anti-)patterns when they start off their projects. The most common one says "To get something done, first create a committee".
The key insight is that people may not think they are creating a committee; they may think that they are "discussing" their ideas with friends. But what happens is that there is a lot of discussion and no action. If there is some action, it tapers off soon enough.
This can be seen in various attempts to "learn design patterns" or "evangelize agile" or whatever. The dynamics of forming a group give an initial emotional high and a short lived illusion of progress. In the end, the effort dies down and it is time for the next fad.
With all the emphasis on "teamwork", not enough people can summon up the gumption to start working on something by themselves and make some progress before talking about it, or trying to get other people involved.
Consciously reversing this pattern seems to help. To master a programming language or framework , for e.g, just shut up and code. When the coding is finished release the code and do something else. Doing that long enough, you'll find you attract like minded people.
So these days, when people say things like "I am very interested in learning mathematics" my counter question is "so how many problems/proofs did you work through today"? Nothing else is important. Replace "working through problems/proofs" with the appropriate context dependent measures of progress.
Don't yap. Work.
Thursday, August 03, 2006
W[ear] Cloak Of Invisibility
The compiler project has kicked off and I have zero free time. I am spending all my time coding and reading.So .... I am totally unavailable.
I am off the net and the phones are off. There is no way you can reach me. I am travelling extensively as well (long story). Mail will (eventually) be answered.
End of bulletin.
Saturday, July 29, 2006
Indo Pak War - A Slightly Different Perspective
But Martin, Enterprise Software IS Boring
Martin Fowler writes about Customer Affinity, a factor he believes distinguishes a good enterprise developer from a bad one. So far,so good.
He says "I've often heard it said that enterprise software is boring, just shuffling data around, that people of talent will do "real" software that requires fancy algorithms, hardware hacks, or plenty of math."
This is almost exactly what I say.(Just remove the quotes around "real" ;-) ).
Martin goes on to disagree with this idea (which is fine) but then he says "I feel that this usually happens due to a lack of customer affinity." And this is where I disagree.
People who work on things like compilers and hardware hacks and "tough" algorithmic problems can have customers, just as the enterprise folks do. I know I have. The compiler I am writing now has a customer. The massively parallel neural network classifier framework I wrote a couple of years ago had a customer. The telecom fraud detection classifier cluster I worked on last year (along with some other talented programmers) is being used by a company which is very business (and customer) oriented. So the idea that a "customer" (and thus customer affinity) is restricted only to those folks who write database backed web apps is simply not true.
Math/algorithms/hardware hacks and the presence or absence of a "customer" (and thus, customer affinity) are orthogonal (you know, two axes at 90 degrees to each other) issues.
With the customer issue out of the way, let us examine the core issue - the notion that talented developers would prefer to do stuff like compilers and algorithms instead of a business app (by which I mean something like automating the loan disbursal processes of a bank).
The best way to get a sense of the truth is to examine the (desired) flow of people in both directions. I know dozens of people who are very very good at writing business software who yearn wistfully for a job doing "plumbing" like compilers and tcp/ip stacks, but I've never yet seen someone who is very very good at writing a compiler or operating system (and can make money doing so) desperately trying to get back to the world of banking software. A programmer might code enterprise apps for money, but at night, at home, he'll still hack on a compiler.
It's kind of a "Berlin Wall" effect. In the days of the cold war, to cut through the propoganda of whether Communism was better than Capitalism , all you had to do was to observe in which direction people were trying to breach the Berlin Wall. Hundreds of people risked their lives to cross from the East to the West but practically no one went the other way. Of course this could just mean that the folks didn't appreciate the virtues of people's power and the harangues of the comissars but I somehow doubt it.
To see whether living in India is better than living In Bangladesh (or whether living in the United States is better than living in India) look at who is trying to cross the barriers and in which direction.
Economic Theory suggests another way of finding an answer. Suppose you were offered say 10,000$ a month for say 2 years, to develop the best business app you can imagine and say, 8000 $ a month, for the same two years to work on the best systems/research project you can imagine. Which one would you choose? Now invert the payment for each type of work. Change the amounts till you can't decide one way or another and your preference for either job is equal. Finding this point of indifference (actually it is an "indifference curve") will teach you about your preference for "enterprise over systems". I believe that while individuals will choose many possible points as their "points of indifference" on that curve, a majority of the most talented debvelopers would prefer a lower pay for a good systems/research project in preference to an enterprise project, no matter how interesting. And to most developers, an enterprise project becomes interesting to the degree it needs "tough programming" skills (massively scalable databases say.. there is a good reason Amazon and Google emphasize algorithms , maths and other "plumbing" skills in their interviews and an equally good reason why a Wipro or a TCS doesn't bother testing for these skills).
The widespread (though not universal) preference of the very best software developers for "plumbing" over "enterprise" holds even inside Thoughtworks (where Martin works and where I once worked). I know dozens of Thoughtworkers who'd prefer to work in "core tech". While I was working for TW, the CEO,Roy Singham, periodically raised the idea of venturing into embedded and other non enterprise software and a large majority of developers (which invariably included most of the best devs) wanted this to happen. Sadly, this never actually came to pass.
A good number of thoughtworkers do business software to put bread on the table, or while working towards being good enough to write "plumbing", but in their heart, they yearn to hack a kernel or program a robot. (Another group of people dream about starting their own web appp companies). For most of these folks tomorrow never quite arrives, but some of the most "businessy" developers in Thoughtworks dream of writing game engines one day or learning deep math vodoo or earning an MS or PhD. And good people have left TW for all these reasons. And if this is the situation at arguably the best enterprise app development company (at least in the "consultants" subspace) one can imagine the situation in "lesser" companies.
The programmer who has the ability to develop business software and "plumbing" software and chooses to do business app development is so rare as to be almost non existent. Of course, most business app dev types are in no condition to even contemplate writing "hard stuff" but that is a topic for another day.
Paul Graham (admittedly a biased source) says (emphasis mine),
"It's pretty easy to say what kinds of problems are not interesting: those where instead of solving a few big, clear, problems, you have to solve a lot of nasty little ones....Another is when you have to customize something for an individual client's complex and ill-defined needs. To hackers these kinds of projects are the death of a thousand cuts.
The distinguishing feature of nasty little problems is that you don't learn anything from them. Writing a compiler is interesting because it teaches you what a compiler is. But writing an interface to a buggy piece of software doesn't teach you anything, because the bugs are random. [3] So it's not just fastidiousness that makes good hackers avoid nasty little problems. It's more a question of self-preservation. Working on nasty little problems makes you stupid. Good hackers avoid it for the same reason models avoid cheeseburgers."
Strangely enough, I heard Martin talk about the same concept (from a more positive view point, of course) when he last spoke in Bangalore. He explained how much of the complexity of business software is essentially arbitrary - how an arcane union regulation about paying overtime for one man going bear hunting on Thanksgiving can wreck the beauty of a software architecture. Martin said that this is what is fascinating about business software. But he and Paul essentially agree about business software being about arbitrariness. It is a strange sense of aesthetics that finds beauty in arbitrariness, but who am I to say it isn't valid?
Modulo these ideas, I agree with a lot of what Martin says in his blog post. The most significant sentence is
"The real intellectual challenge of business software is figuring out where what the real contribution of software can be to a business. You need both good technical and business knowledge to find that."
This is so true it needs repeating. And I have said this before.To write good banking sofwtare, for e.g. you need a deep knowledge of banking AND (say) the j2ee stack. Unfortunately the "projects and consultants" part of business software development doesn't quite work this way. I will go out on a limb here and say that software consulting companies *in general* (with the rare honorable exception blah blah ) have "business analysts" who are not quite good enough to be managers in the enterprises they consult for and "developers" who are not quite good enough to be "hackers" (in the Paul Graham sense).
Of course factoring in "outsourcing" makes the picture worse. By the time a typical business app project comes to India, most, if not all of the vital decisions have been made and the project moves offshore only to take advantage of low cost programmers, no matter what the company propoganda says about "worldwide talent", so at least in India this kind of "figuring out" is almost non existent and is replaced by an endless grind churning out jsp pages or database tables or whatever. It is hard to figure out the contribution of your software to people and businesses located half a world away, no matter how many "distributed agilists" or "offshore business analysts" you throw into the equation.
This is true for non businessy software also,though to a much lesser degree. The kind of work outsourced to India is still the non essential "grind" type (though often way less boring than their "enterprise" equivalents). The core of Oracle's database software for .e.g. is not designed or implemented in India. Neither is the core of Yahoo's multiple software offerings - the maintenance is, development is not. And contary to what many Indians say, this is NOT about the "greedy white man's" exploiting the cheap brown skins. You need to do hard things and prove yourself before you are taken seriously. Otherwise people WILL see you as cheap drone workers. That's just the way of the world).
And I say this as someone who is deeply interested in business. T'is not that I loved business less but I loved programming more :-).I'd rather read code than abusiness magazine but I prefer a business magazine to say, fashion news. I actually LIKE reading about new business models. I read every issue of The Harvard Business Review (thanks Mack) and have dozens of businessy books on my book shelf along with all the technical books. (I gave away all (300 + books) of my J2EE/dotNet/agile/enterpise dev type book collection , but that is a different story). I am working through books on finance and investing and logistics. I have friends who did their MBA in Corporate Strategy from Dartmouth or are studying in Harvard (or plan to do so). I have friends who work for McKinsey, and Bain and Co, and Booz and Allen, and Lehman Brothers. If I were 10 years younger (and thus had a few extra years to live), I'd probably do an MBA myself. And of course I have spent 10 years (too long, oh, too long) doing "enterprise" software. In my experience (which could turn out to be atypical),the "plumbing" software is way tougher and waaaaay more intellectually meaningful than enterprise software, even of the hardest kind. I might enjoy an MBA, but I will NEVER work in the outsourced business app development industry again even if I have to starve on the streets of Bangalore. I'll kill myself first. So, yeah I am prejudiced. :-). Take everything I say with appropriate dosages of salt.
And just so I am clear, I do NOT think Martin is deliberately obfuscating issues. I think Martin is the rare individual who actually chooses enterprise software over systems software and is lucky enough to consistently find himself working with the best minds and in a position to figure out the business impact of the code he writes. This is very different from the position of the average "coding body", especially when he is selected more for being cheap than because he knows anything useful ("worldwide talent", remember :-) ).
Btw, Martin's use of the word "plumbing" is not used in a derogatory sense (though I've seen some pseudo "hackers" take it that way, especially since the word makes a not-so-occasional appearnce in many of his speeches) but more in the sense of "infrastructure". If you are such a "hacker" offended by the use of the term, substitute "infrastructure" for "plumbing" in his article. That takes the blue collar imagery and sting out of that particular word, fwiw.
I think Martin has very many valuable things to say.
I just think he is slightly off base in this blog post and so I respectfully disagree with some of his conclusions.
But then again, it could all be me being crazy. Maybe enterprise software is really as fascinating as systems software, and writing a banking or leasing or insurance app really as interesting as writing a compiler or operating system.(and Unicorns exist , and pigs can fly).
Hmmmm. Maybe. Meanwhile I'm halfway under the wall and have to come up on the other side before dawn. Back to digging.
Wednesday, July 26, 2006
[Politics] Tip Of The Hat To Israel
I like to think I am "centrist" when it comes to politics and that the world has shades of grey and is not all black and white.
However, in the present Israel/Lebanon-Hezbollah conflict, I sympathize with Israel.
The key characteristic of a nation is that it has (or should have) a monopoly of violence within its borders. As far as I can make out, the Israeli withdrawal from Lebanon was conditional on the *state* of Lebanon reigning in the Hezbollah.
If the Lebanese turn over a chunk of their territory to murdering fanatics trying to impose an archaic religious code on the rest of the world, they can't cry foul when people who lose patience with Allah's foot soldiers invade that territory. Either they control the territory, in which case they should be in control of their borders, not an Islamic militia. Alternatively they don't control it, in which case they have no business screaming about Israeli agression or invasion.
The Israelis, could in an ideal world, have shown a more nuanced approach, but I guess their patience has its limits.
this blog entry says it best.
Subscribe to:
Comments (Atom)