An Apprenticeship Pattern

Dave Hoover maintains a list of what he calls Apprenticeship Patterns, which are essentially heuristics that help you navigate the transition from being an "apprentice" programmer to a "journeyman" (The terms "apprentice", "journeyman" and "master" come from the terminology of medieval guilds which used these terms to delineate increasing levels of competence at a craft) .

While I am not so sure if the "patterns" format really fits , Dave has a very impressive list of excellent advice . Please go and read them if you haven't (and buy the book when it comes out) .

Recently a friend asked me how to become "as good as you" ( bwaa ha ha haa ) in programming and I drew out a couple of "patterns " from what i told her .

Here is the first

Book Chain

To gain mastery in any given field the best way is often to read and work through a series of "classic" books in sequence .

Since the books we are speaking about are books about programming there will be plenty of code to write as well.

The problem is that there are often many many books about any particular subject and it is not easy for an "apprentice" to determine what to read and in what order . A person who has already walked the path is often able to guide an "apprentice" in constructing the book chain .Working through such a reccomended chain of books will save you years of flailing around.

Here are a couple of examples of book chains .

Topic : Compilers

First work through Language Processors in Java (by David Watt ) .

This book gives you a "vertical slice" through constructing an interpreter for a (fairly limited or "toy" )language .

Then work through Essentials of Programming Languages (Wand and Haynes) .This book gives you deep insights into how a langauge works "from the inside".

Follow this up with Van Roy and Haridi's "Concepts, Techniques and Models of Computer Programming " which dissects the various programming paradigms and shows how each paradigm is best suited to a particular set of problems and more importantly what the limitations are . Again Paradigms are best understood by getting an insight into how they work internally rather than as random rules like for example ,"use polymorphism in preference to inheritance" .

Next , read a book (and look at the corresponding source code ) that explains how a production quality compiler/interpreter works .

Three books that fit the bill are Lisp in Small Pieces, A Retargetable C Compiler - Design and Implementation and Compiling with continuations

Now you are all set to look into a real world compiler/interpretr/vm (say the Ruby Interpreter or the JVM ) .And once you get through that , you are a "master" (or very close to mastery as makes no difference) .

And now for a slightly less technical example

2. Extreme Programming

First read Kent Becks "Extreme Programming" to get an overall idea . Then read (and work through ) Test Driven Design by the same author. Learn to use a unit testing farmework like JUnit

Then read and work through Martin Fowler's "Refactoring " and Josh Kerievsky's "Refactoring to Patterns" , and optionally Ron Jeffries's "Adventures in C# " . Learn how to operate a Continous Integration Framework (like Cruise Control) and how it works, and then understand how Dependency Inversion works and learn to use a Mock Objects framework like JMock. .

Follow this up be learning how to break up required functionality into stories . To learn more about the XP process read Jim Highsmiths book on Agile Project management , and optionally the Scrum book .

And there you are .If you have learned (and more importantly applied ) all these concepts , you are an XP "master" .

The best way to get a "book chain" for a topic you intend to specialize in is to ask someone who is already expert in the field to give you a sequence of books to study.

A poor second option is to find the "best book to begin with " (for Artificial Intelligence , this is Norvig and Russell's "Artificial Intelligence, a Modern Approach", for example ) and pick and choose from the bibliography.

Light In Dark Corners

Updated to include an opinion on Google India

Joel Spolsky has written an excellent article where he explains why it makes no sense to hire mediocre programmers for Product Development.His thesis is that mediocre programmers just cannot hit the "high notes" that top grade programmers can (read the article) and since the cost of reproduction of software is essentially zero , software products live in a winner take all environment .

He also points out that this does not apply to non product software development . I quote ..

"Sadly, this doesn't really apply in non-product software development. Internal, in-house software is rarely important enough to justify hiring rock stars. Nobody hires Dolly Parton to sing at weddings. That's why the most satisfying careers, if you're a software developer, are at actual software companies, not doing IT for some bank. "

The programmer who is in the top 1% and prefers to create enterprise software vs product development (almost ) doesn't exist .

But beyond that there is one more factor that ensures that companies which create mostly enterprise software accumulate mediocre(and worse) programmers over time . And this is the "per hour" billing model, which immediately puts the goals of the consulting organization and the client organization at loggerheads .

The client organization gains by reducing the total number of developer hours (and the dollar per hour rate of the individual developer) . The consulting organization gains by increasing the number of developers and the dollar rate(which is mostly done by hiring people with greater "number of years of experience , there being no objective way to measure "goodness of value delivered" ) . Thus sheer economic logic dictates that client and consultant pull in opposite directions. Most of the dysfunctionality of enterprise software development arises from this fundamental anomaly .For example the relentless pressure most consulting oranizations face to add "resources" comes from this "increase the number of people on a project " (rather than "deliver more value " ) mentality. And this pressure eventually translates into terrible people getting entrenched in even the finest organizations . Then of course you need more "committees" to manage the mess that results, and new games come into existence , all resulting in the more talented people fleeing or giving up.

This downward pressure forces the client to outsource the work ("You say it takes 20 developers , 3 managers and 5 QA guys? ok but i want 15 developers,2 managers and the QA folks in India at India rates" ) .

Product development work (not maintenance!) is almost never outsourced.Product development companies do the initial development of projects wherever the really good programmers are (and pay them really well . The money involved would give most "enterprise" developers heart attacks) and then outsource the bug fixes, operations etc .(the spiels of "India/China/--insert cheap offshoring destination here-- has top grade talent and so we started an office in Bangalore/Shanghai/wherever" are trotted out to make sure the offshore developers feel happy and often have no basis but the decreased "dollar per hour" rate for mindless repetitive work) .

So the best programmers in developed countries have nothing to fear . The average devloper who can churn out endless jsp/xml/ejb etc needs to be terribly afraid. And rightly so .

The one possible point of satisfaction in enterprise software comes from really understanding a deep clientside problem and writing good code to fix this. Alas, even this is denied to offshore programmers who have to play many games (see Eric Berne's "Games People Play") and fight amongst themselves like dogs to get one of the coveted "onshore" assignments and get anywhere near the real clients(where they are still paid less than the local developers) .The only strategy(in the Game Theory sense) that has a positive payoff in the offshore developer's world is to join the ranks of management as soon as possible and leave "lowly" work (and most of the work is terrible) behind . Then you can enjoy a "money for nothing" existence (at the cost of having a "meetings, committees and paperwork" life and losing your technical skills, if any ).

There are some efforts going on to change the basis of payment from the insidious "dollar per hour" concept .But so far nothing has been very succesful because changing the game would mean that both clients and consultants would have to actually think of what they are doing and calculate the ROI of every feature they add(and God forbid that such heresies come to pass :-) ) .

Meanwhile developers , if they are hell bent on developing enterprise software, should probably join one of the better companies around and pray (very, very hard) you get a greenfield , challenging project.

But the question remains.If you want to work with the best people and have a satisfying life working with technology why flirt with Enterprise Software at all? Do you really (really really) get your kicks writing Banking Software ?

And even if you do, it may make sense to constantly re evaluate your commitment .Here is a simple way to do this .

When you get up in the morning, knowing you have to go to work , do you smile or grimace? And when you finish the average working day ,are you happy, having learnt something new and created good features ?Or has the daily grind just chewed up another 8 (more likely 12, if you are in Bangalore) hours of your life leaving you drained and exhausted ?

Update : Apparently Google does things differently (Why am I not surprised? ). I say "apparently" because I have no confirmation yet.I was told that Google's India Office is really about hiring only top notch talent and not outsourcing for cheap labour.Apparently anyone who makes it through their (very tough obviously) interview processes can select which office they want to work for .In other words at the end of your interview process you can choose to work out of,say California or Zurich . If true , this is absolutely incredible and needs wider publicity in Bangalore!

Anyway this blog entry was mostly about the yuckiness of most outsourced enterprise projects and not about product companies,although it remains true that most product development is done abroad . Google is about as far from "yucky" as you can get .

Help me Find a book please ( Kramik Pustak Malika by Prof. Vishnu Narayan Bhatkhande )

I looked for this book both online and offline,but couldn't find a copy.Basically the book is "a six volume work ..of major archival significance. It is basically a collection of notated compositions, grouped according to the Ragas in which they are composed. It contains 1849 compositions in 189 Ragas. To this day there is no compendium of traditional compositions which comes close to it for variety and accuracy. " according to this site .

As far as i can make out this book is quite easy to find in North India .Jaipur University even prescribes at as a textbook in their Music Degree courses . Alas, I am in the distant South(Bangalore) where the dominant musical system is the Carnatic system .If anyone can get a copy for me ,(or even have a copy they wish to sell), I will be glad to pay the cost and postage charges. Please contact me at magicindian AT yahoo DOT com and we can work out the arrangements.

Let us see if the blogosphere helps where traditional bookshops have fallen down .

So What Should I Write On My Blog?

Yesterday one of my friends called me with a simple question "What do i put on my (to be started) blog" ?

It is sometimes possible for one to be so used to technology that it is hard to conceive that a large number of people just dont' "get" it . As we shall see blogs mean vastly different things to different people and it is difficult to generalize across all these view points.A lot of people don't blog because they are confused by these myriad perspectives and hence are not sure what exactly to write on a blog . And that is a great loss .

Anyhow my friend was waiting for my answer on the other end of a long distance call so I had to organize my thoughts to reply and this is the result.Maybe it will help someone "get" blogging.

1. The simplest use of a blog is as a "web log" , in other words, a record (on the web) of things you do/encounter/experience.I believe this is how blogs were initially used .Entries like "I went out for dinner with X today and later popped in to see the X-Men movie. The movie was fun" or "Guitar class was tough . I just can't get the finger shift for CMajor-Bminor" belong to this category.

   This type of blog combines the record keeping benefits of an activity log with the publishing capability of the Internet. Thus the blogger's community of friends can keep track of what is happening in his life.

   An interesting variant of this is where people blog about the arrival of a baby , detailing information about doctor visits and ultra sound scans, all the way to photos of the various stages of the baby's growth("hey my baby looked at me today. wow! here's a photo".)

   Yet another variant is a project blog with entries detailing the progress (or otherwise) of a project and is often used in the open source community .

2. The second common way to use a blog is as commentary . The mode of posting used here is that of a critic or reviewer , passing judgement on things that interest him .Essentially each post is a review of a movie or book or piece of software, or a court judgement , or anything else that is of interest to the blogger .My friend ,for instance, asked if a chapter by chapter review of the Sanskrit to Malayalam translation of the Ramayana would make a good blog entry . The answer is "yes" .
3. The third way is to use a blog as a forum for thinking about ideas or distilling experience .This is essentially the above two forms of blogging plus the recording of extrapolatory thinking.In other words, various things that strike the blogger's interest are logged and then there is an additional facet of thinking about them .It is taken for granted that such thinking is incomplete .

   For example I learn about the Theory of Constraints and the three main metrics it uses (Throughput, Inventory, and Operational Expense) .I might write a blog entry just applying the idea to a software consulting company and trying to extrapolate a bit ("ok , so applying TOC to software, a programmer on the beach is Operating Expense. So are all the non billable managers, recruiting staff etc. Code written but not paid for is Inventory .Or is it ?If the customer doesn't pay can this code be re used or sold? If not then is that too Operational Expense?" etc.Note the theme of "thinking out loud".

   The blogs of most software developers have this quality.A commenting facility and a strong community of readers adds an extra dimension to such blogging if they help you to flush out inconsistencies in your thinking and/or add new ideas .

4. Another way to use a blog is as a forum for writing/thinking. The blogger sees himself as an author and a blog posting is treated as a literary form just like a magazine article or an essay .In this world view a blog entry is for very short pieces of writing which are not long enough to make it ento an essay or article. Martin Fowler's blog is an excellent example .

   The main difference from the "exploratory" mode above is that the final piece of writing is much more polished and definite for all its (comparitively) small size. Any thinking is done outside the blog entry itself.Comments may or may not be present (just as there are essays and articles without commenting in books and magazines). This type of blog is used by professional writers or thinkers (management consultants etc) where the blog is additional evidence of the writer's expertise and thus is more "professional" than the "exploratory" or "logging of personal experience" type blogs.

5. The fifth way of using a blog is to explicitly focus on building a community and sharing of thoughts within an interest group , something like a distributed news group . Thus you have "java blogs" or "home schooling blogs" . The additional concept here is that the size and quality of the community built up around a blog , with the number and quality of comments, trackbacks and pingbacks is a key metric of how successful a blog is .
   In this school of thought (which i don't subscribe to , by the way) a blog is primarily a community building tool, which enables you to reach out and connect to people who share your interests but are distributed across the world. A variant of this is to use a blog to evangelize a particular company or product.Taking this even further, there is the notion of competing with other bloggers for "ranking" in the blogosphere ( a fancy term for "all the world's blogs" ) .
   There are people who are professional bloggers and make huge amounts of money from their blogs , either through running ads on their blog sites, getting book contracts, or being sponsored by companies.
6. There is an associated concept of "corporate blogging" , that treats blogs as a new medium of communication between the various stakeholders of a company (employees, shareholders etc) . when you hear of someone being fired for "inappropriate blogging", or a company exposing a corporate blog for potential cutomers to know how cool it is, this is the facet at work. There is also a whole ecosystem of visionaries, consultants and snake oil salesmen(not all mutually exclusive) who try to sell blogging tools (often just jazzed up versions of freely available blogging tools priced fantastically to make them "enterprise" worthy ) to bewildered managers as the next silver bullet that will solve all corporate ills.

In practice all these modes intermingle and there are various ratios of all these approaches simultaneously present in most blogs.There is no "right way". Use whatever works for you .

I , for one, tend to see my blog as a place for self expression and improving my writing skills and am not too bothered about community building or "reaching out". My friend Rajesh is more concerned about communities and sharing of experience .
I guess it just shows I am more of an "anti social element" than he is :-).

But then this is not static either. I created this new blog explicitly because many people asked for commenting facilities .So maybe in a year or so I might focus on "community".Right now , for me , a blog is a combination of log of events(minor focus), an arena for exploring thoughts(major focus) and practicing writing(major focus) .

Blogging with Blogger

The last post turned out be quite popular and more than 20 people requested me to add a "comment facility " . The choice was between (1)run a mysql based blogging app on my website or (2) go with a hosted version .

I first looked at WordPress .I even installed and ran it .While it is a very impressive in terms of fucntionality , I really hate PHP and the thought of mucking around with PHP to make minor tweaks to the site is disgusting. I had to dig around in the template files('sidebar.php' for those who want to know) and change the feed indicator to remove an extra "feed:" string that gave a result of "not a valid protocol" when i tried to get the feed from bloglines. From what little I understand of this issue it seems that WP is on the "cutting edge" but the aggregators haven't caught up yet , or some such nonsense .
And of course once I tweak the template I am locked into it because the other templates still have this "feed:" rubbish hard coded in .

I next considered Typo . The biggest advantage Typo has is that its is written in Ruby, and built on top of Rails which add all kinds of goodness . If anyone is not convinced of the superiority of Ruby Vs Php I suggest they take a look at the source code of Typo and then the corresponding PHP code in WordPress. The only reason i didn't put my blog on Typo is that I have a sense it is evolving *very* fast and i would like it to reach a point of stability .
That leaves other yucky systems written in perl or hosted solutions where you don't have to bother too much about what is under the hood . Since I hate perl almost (but not quite) as much as i dislike PHP , I decided to look for a fairly minimal free hosted solution .Blogger meets all my needs . I am not sure it supports trackbacks and pingbacks but I am ambivalent about this whole "blogosphere / social text" meme anyway .All I need for now are permalinks and commenting (and i may yet turn the latter off ) and blogger has both .

The one thing I would like to have had are categories . But on second thought, I haven't subscribed to any category speciic urls . I may just spawn a new blog if i need one for a separate category .And knowing the quality of the people at google, I am not too concerned about the software crashing etc.

Anyway that's the story . I moved most (but not all) of my old blog entries here,tweaked the template a bit to make permalinks explicit and everything looks good .I still need to figure out how to put up an rss link . Bloger seems to use Atom .

But so far so good .

let us see how it goes .

Why Hackers Should Read History

It all started very innocently . A manager at one of the many software companies in Bangalore announces a scheme to "tax" Indians travelling abroad on the company's business .

The basic (at best naive, at worst foolish) idea was that Indians who go to the USA would "contribute" a percentage of their (dollar) earnings into a "common pool" and this "tax" would be distributed amongst all the poor unfortunate Indians who couldn't travel to the Land Of The Golden Mountain.

The net buzzed with mails about "American arrogance" (the Manager is American) and "whoever heard of a communist company? "(the CEO of the company is supposed to have socialist leanings).

On a parallel thread , a couple of naive Canadians who came to India for a short visit were interviewed by a Canadian reporter about their Indian experience and the result was a fairly funny (to me anyway) article in the Canadian Press about how their luggage almost got stolen and a few comments about the first-world-and-third-world-existing-side-by-side aspect of Indian life (at least in Bangalore) .Mixed in with all the naivety were a couple of nasty comments about how Canada (unlike India) was "a place where you were not measured by your class stature" (perhaps they meant "caste" instead of "class" ?)

In yet another encounter , a couple of scientists at the Indian Defence Department were very scathing about "barbaric" Americans and their "decadent" life styles and I found myself in the position of having to point out the many *good* things about America and Americans and the things that need improvement in India .

The funny thing was that,unlike me, the speakers themselves had never actually visited America .But they were certain that Indian "civilization" was way ahead of a rich but decaying America.

I generally don't blog too much about politics or other "sensitive" areas but i figure it is time to take a stand.

Here are the principles I follow when dealing with people .

1. The intrinsic worth of a person has nothing to do with race or nationality . It is all about how you think and what you can do and how well or badly you do it.
2. Some Americans are arrogant . This is a problem of belonging to the dominant culture at a given point of time . The Romans had this arrogance. So did the Persians , the Assyrians , the Mongols, and later ,the Spanish, the Venetians and the British . And if you go back to a time when the Indians had a flourishing civilization when Europeans lived like animals , I dare say some of those Indians were arrogant too.
   All this has nothing to do with the world as it is today . In the end if you are good at something other people need , you will be given due respect .
   And if you still don't like the way the world works you have the freedom to try to change it . The only guaranteed freedom is the freedom to try.
   The thing to do when you face arrogance based on the latitude and longitude of your birth (or skin color) directed at you is to ignore it (or smash it flat depending on how combative you feel) .
   Which brings me to
3. A large number of Indians are subservient to those in Authority , especially if that Authority Figure has a white skin . This comes of a combination of lack of self confidence , an excess of greed (to grab dollars, green cards , whatever ) and suffering from a simultaneous lack of guts and liquidity of the spine .
   When someone dreams up a totally foolish idea (a great defect in many people who end up as managers , not because they are any good at management,but because they are useless for anything else) the thing to do is to point out the absurdity in a rational manner ,irrespective of whether the proposer came from Madras, Manhattan or Mars.
   And pour scorn on it if you think it is appropriate to do so (though there are often more subtle ways ofgetting what you want).Often just a polite refusal to play along will do the trick . It also helps if you have the courage of your convictions and a feeling of self worth independent of what those "higher up the ladder" think of you.
4. All cultures have good points and defects . The "Superior Man" (as the I Ching would say ) recognizes and promotes the virtues of all cultures and opposes and suppresses their vices both within himself and his society .
5. All cultures have great men and despicable ones .
   Those who think Americans are "decadent" should learn about(confining ourselves to Politics) Abraham Lincoln (there is a lot more to his greatness than "freeing the slaves" which is about all most Indians know of this incredible man. Read Bruce Catton's three  part  history of the American Civil Warto understand Lincoln in his context ) , Roosevelt, and Jefferson.
   To see how fools constantly try to ruin the powerful nation these great men built ,read up on ( for eg, again confining ourselves to the political sphere),Clement Vallandigham,Gerald Nye,Edgar J Hoover and the inimitable Dubya.
   Dilbert's PHB lives on in history and politics just as much as in cubicle land .
   Westerners who pride themselves on their scientific prowess should meditate on the fact that even multiplication was hard before zero (and thus the decimal system) was invented in "primitive" India . And that the theory of Evolution is still anathema in major chunks of the "scientifically advanced" USA .
   Those who take pride in their dominant economic status should look to European Kosovo to see how easily civilization crumbles when Time's judgement rolls around. Some believed till the end , "That can't happen here . we are Europeans".
   If you look a bit deeper into the shakiness of the American Trade Deficit and potential Dollar crash ,you'll see how feeble such assumptions are and how fast the shoe could be on the other foot.
6. Indians who believe in America's "decadence" should try to actually get to know some of the History of India rather than prattle on about "five thousand years of civilization".
   The "Golden Age" preached by Hindu Fundamentalists is largely a myth(barely half a century old) created to counter feelings of inferiority vis a vis the West. Historically the "morality" and social structures of the periods in which India came close to being a dominating culture have significant similairities with the mores of the West . A good book to start with is The Gem In the Lotus,followed by The Emperors of the Peacock Throne .
   To figure out how to "defeat" the west , read Victor Hanson's(Hanson is a slightly over the top ,but nontheless brilliant ,writer),How The West Has Won (don't worry about the alternate title.It is the same book). Indian editions of most of these books are available , a situation in which the "cheap Indian" tag actually works in your favor!
7. To shift the context a bit, There are Indians ,some of them in the highest strata of society,who judge other Indians on the basis of caste, especially when it comes to things like marriage . There are Americans ,some of them in the highest strata of society, who judge other Americans on the basis of race , especially when it comes to things like marriage . Caste Based Discrimination is illegal in India except for government sanctioned Reservations . Race Based Discrimination is illegal in America except for government sanctioned Affirmative Action . All of which shows that both societies are not perfect and contain substantial chunks of deluded people and a comparitively fewer number of intelligent ones .
8. and finally understand that one of the great powers that moves the world is Stupidity . "Never ascribe to malice that which can be adequately explained by stupidity." .So said Napoleon , a European who shook his continent and a good chunk of the world, a man who while symbolising the quintessential westerner , was secretly most comfortable with the mores of the Orient .

Stupidity And Brilliance have nothing to do with culture. And yes , the "company enforced travel tax" is a tremendously stupid idea which tries to fix a deep and complex problem with spit and duct tape. But it could, in theory, be used a creative provocation to evolve a better one . Pretty unlikely . But who knows? If everthing fails , this will provide some consolation to the oppressed masses . Arise ! awake ! You have nothing to lose but your money and self respect!

Goodbye to Java

Dear Java,

These days I program mostly in Lisp and Ruby (with a smidgen of Erlang and a dollop of C ). Coming back to you from such incredibly powerful languages is painful at the best of times , but after an attempt to invoke your new generics feature to do something conceptually simple resulted in almost unreadable code , I decided to throw in the towel . (I won't go into the messy details here . A google search for 'java generics suck ' should turn up enough arguments (and counter arguments ) . Tim Bray , Bruce Eckel and others do a good job of debunking the "new java" . ) .I'll still be in touch with you to complete the AIMA code (the forthcoming release is in Java 5 btw ) but I'll be damned if I ever use you for any other project .

I stayed well away from your newest competitior C# so that's quite all right too .

And so dear Java , it's been nice knowing you . While you were never a stunning beauty like Lisp or smalltalk or C, your massive libraries endowed you with a certain charm for a while . But alas today you are fat and shapeless and you desperately need a deodorant .

Not to worry though . The millions of lines of atrocious "enterprise" code that belong to you will ensure that you won't lack for suitors for the next 10 years . And the fast depleting billions in Father Sun's coffers will prop up your bulk for a while .So be of good cheer. Even the undead Cobol manages to drag herself through the years .So you have nothing to fear.You'll probably outlast me.

Meanwhile this is one programmer who is now rediscovering the sheer joy of programming writing some meaningful code that actually exercises some grey cells .

Hasta La Vista, Baby and Take Care .

Regds,

Ravi

Levelling up in Math land

(Edited on Feb 4 2011), This entry is correct but is now obsolete.I rewrote this with more detail here. Please use that instead.

Concrete Mathematics is a fine book. But in the very first there is a bit of handwaving that goes on to eventually get seemingly magical and the novice who is using the book for self study (iow, someone like me ) can get lost very easily .

A good example is the "solving a rcurrence by the repertoire method " explanation on pages 12 - 15 ). A margin note by a student (an excellent idea to include these) warns us "Beware . authors expect us to figure out the idea of the repertoire method by seat-of-the-pant examples instead of giving us a top down presentation " .

Unfortunately that doesn't help a reader to figure out what is going on and how to use the "repertoire method " to solve a recurrence . I've been banging my head against this for a few hours and i think i found out how the repertoire method works . here it is .

I will use the same recurrence the authors use .

Recurrence R =

f(1) = alpha
f(2n) = 2 * f(n) + beta for all n greater than or equal to 1
f(2n + 1 ) = 2 * f(n) + gamma for all n greater than or equal to 1
to solve a recurrence using the repertoire method ,

step 1
make sure that the recurence is expressible as a sum of parameters multplied by functions of n

the above recurrence can be transforemd to f(n) = A(n)*alpha + B(n)*beta +C(n)* gamma (this is done on page 13 of the book so i don't repeat it here)

where A, B and C are functions on n and alpha beta and gamma are constants

step 2
p times , where p is the number of parameters (in the above case, alpha, beta and gamma so p = 3 ) *assume* that f(n) = a simple function of n and solve using the original recurrence to get an equation . eg: in the above recurrence there are three parameters , alpha , beta and gamma

so say
f(n) = 2^m, --> Eq(1) (EDITED on Feb 4, 2011 - This is actually proved by induction in the book so this guess for the value of f(n) , but this guess does work in providing the value of A(n) better than proving it by induction)
f(n) = 1 -->Eq(2) and
f(n) = n -->Eq(3) be your assumed functions .

solving R with Eq(1) gives A(n)= 2^m --> Eq(4)
Solving R with Eq(2) gives A(n) - B(n) -C(n) = 1 -->Eq(5)
and solving R with Eq(3) gives A(n) + C(n) = n --> Eq(6)

step 3
solving these 3 equations (eqs 4,5 and 6) gives us the values of the coefficients .

A(n) = 2^m
B(n) = 2^m -1 - l (the last is the letter "ell" )
and C(n) = l (the letter ell)
thus the closed form of the recurrence is

A(n) alpha + B(n) beta + C(n) gamma = 2^m * alpha + (2^m - 1 -l) * beta + l*gamma

now the question is how can we be sure that we guess the "right" functions in step 2 ? how do we know that f(n) = 1 is a "good " guess ?

the answer is that if you guess "wrong" you won't be able to solve for that equation and will end up with an impossible equality .

and that is how the repertoire method works!

happy repertoiring

None So Blind... The Agile India Conference Experience Report Part 2

Thanks to problems setting up the laptop I missed Brett's presentation. I listened to Mridul's talk on "Kernel Programming + Agile" and then it was my turn . We grabbed some lunch and then went to see if we could attend some seminars . Unfortunately all of the presentations in the afternoon were boring as hell . Most were about "combining waterfall and agile" Apparently there was a talk earlier in the morning about a "wagile" methodology . If you, the reader, just fell off your chair or threw up , you can imagine my reaction.

I think God is having His revenge on me for generally being disrespectful and specifically for using a Dilbert Cartoon as a presentation aide. when i looked at all those PHB types drone on and on about "waterfall and agile are complementary " , i felt i was inside a Dilbert strip . I think a lot of clueless managers in Bangalore have latched on to the "latest buzzword" and are busy "optimizing" (bwaaa ha ha !! this is a private joke . i actually met a (totally dilbertian) manager who said "my main skill is optimizing" if you mention this to Abey or me, we'll fall around in hysterics ) and "combining" it with whatever outmoded nonsense they know about "managing software teams" .

It is very frustrating to see how much traction this stupid idea is getting in Bangalore amongst people who don't want to change how they or their companies function but want to be "agile" so they can fool their clients into giving them more money.

I have said so before and i will say it again .

The primary facilitating factor for "going agile" is a culture that respects no hierarchy but that of competence and maintains an atmosphere of total openess and trust.

now how many companies in Bangalore work like this ? 3 ? 5 ? you get the idea .

In the evening the organizers asked me to be on a panel on "distributed agile " . There was this person on the panel (who shall remain nameless) who claimed that his "whole life is agile" (!!! ) and then went on to proclaim a lot of "waterfall " ideas (eg:" 'novices ' should have their yahoo messengers monitored by a 'senior' developer to make sure they don't say anything "dangerous" to a client" ).I have never heard such nonsense in my life . I really lost my cool and did what i could to rip his arguments apart.

The audience loved it :-D .

An artist sketching the panel drew me in a furious, scowling figure. If i can get a copy of the drawing from Manoj Bharadwaj i will certainly put it up here .

Because one of my friends was leaving Bangalore , i missed the second day of the conference .

too bad, because i was looking forward to hear Fred George, Dakshinamurthy Karra ,Simon Harris(author of Simian), Henry Jacob and Luke Barret speak .The last two seem to be focussed on Interaction Design in an agile fashion. Alas it was not to be .

All in all it is wonderful that such conferences are happening in India . If the Agile India Society can just keep going we may see many more such events .On the other hand it may just wither and die.

we'll see .

Slide, Slide on the wall... The Agile India Conference Experience Report Part 1

I attended (and spoke at) the Agile India Conference. My speech about applying XP to a large AI classifier system I am working on was well received . I used slides only to represent visualisation of complex data (and one for a dilbert cartoon).

I think most people use slides because

1. they need to structure their thoughts as they create their speech,
2. they can "lean on" the slides,use them as a psychological crutch while on stage,to take away some of the stress associated with speaking
3. that is the only way they know how to make a presentation. That is how they learned to do it

Though i was a fairly good speaker and debater in college ,soon after i got into the software "industry" (note the quotes) i fell into the habit of creating and using slides for presentations. Recently i read this , this and this .

I also noticed that the very best speakers , even in the software field, (example Martin Fowler ) do not use slides .

ok so here was a chance to face an audience without any shields again after many years . The surprising thing was how powerful the hold of PowerPoint was. Though,years ago, i had spoken to large audiences before, without slides,or even notes, now it felt very strange to do so without PowerPoint guarding my back !! this was scary !

anyway , an intial awkwardness aside, the speech went off fine . The best compliment was given by a student volunteer (the conference was held at the PESIT Engineering College) who declared that he was able to follow the talk easily, though it was about a highly technical topic.

Vivek Singh , whom i respect tremendously, even while disagreeing with him about most things , was there and he didn't throw his shoes at me . .

So i guess it all turned out fine .Thanks to the talk, I met a lot of interesting people , including a person who writes browsers for mobile phones .We had a very interesting conversation.

And i will never use PowerPoint again . What i need is something that would allow me to outline the structure of a speech as i am speaking, annotate it with audience questions(as recorded audio ? ), connecting links etc .A cross between an outliner,a multi media blogging tool and a brain map .. hmmm time to fire up squeak.

More about the conference in the next entry .

Mathematics An Epiphany

Today morning , I was at the Centre of Artificial Intelligence Research staring rather stupidly at a page covered with strange squiggles and cryptic sigils . I had gone there to "gather requirements" and the person in charge dropped a couple of papers from the IEEE transactions on Machine Intelligence on the table and proceeded to talk in a language that brought back faint memories of drowsy classrooms from oh so long ago .

Sample sentence = "To solve the problem defined in expression (1), a single unconstrained problem equivalent to expression(1) is obtained by defining the augmented Lagrangian function denoted by phi ".

doh...

After ten minutes or so of this, i put up my hand and said "what does that mean in English? i didn't understand anything". My friend Mridul caught on and both of us asked increasingly dumb questions till the scientist gave up the struggle and patiently explained the math to us term by term translating into English and into a 2 dimensional diagram . I was very skeptical that i would ever understand anything but ever so slowly the squiggles resolved into a stunning set of ideas laying out a new way to do machine learning. After few hours, we were nodding along, even chipping in at various points with ideas that made sense . I had gone from skepticism to bemusement to fascination to awe.

It is hard to state this bluntly but i came out of that meeting convinced that a programmer who doesn't know math is somehow crippled(so is a mathematician who can't program, but that is another story). The power of those equations ,drawing down a whole set of tremendously complex abstractions into clean lines of force that thrummed with power , and seized even our bumbling thoughts and flung them unceremoniously into a world of bewildering possiblities, seemed to have affected Mridul too.

The first thing he said after the meeting was "Damn ! we should learn math.If we could wield it the way these scientists do, we could create so many fantastic programs ".

We had a long discussion over lunch on what we had perceived and distilled it down to these basic points.

1. the combination of deep mathematical skill and expert level programming ability would be the ultimate "level up"
2. the first step in mastering math is be to learn to read the notation. just like learning the syntax of a new programming language
3. the second is to grasp the reality expressed by the notation at a gut level , like understanding the paradigm and patterns lying underneath a programming language , like ,say ,beginning to grok "oo"
4. the third is to use that understanding to create new possibilities
5. the fourth is to use a programming language to embody and refine those possibilities , thus creating programs that do what has never been done before.

Strange .In one day, I have gone from hating mathematics to bedazzled wonder and a craving to learn .

Life is so wonderful .

RunAwayScreaming

I came across Run Away Screaming

so.... having nothing better to do .. i wrote

When they sell you as cheap Indians
Then tell you "you are   cool"
And Power is for Caucasians
And the PM is a bigger fool
You want to RunAwayScreaming....

When they tell you "we do XP"
And tell you "bill more hours"
And ship code that is buggy
To folks in Ivory Towers
You want to RunAwayScreaming....

When you ask for a laptop
And they form a Committee
And talk and argue till they drop
About if you are worthy
you want to RunAwayScreaming....

when for every new situation
someone makes complex laws
and a PHB fascination
becomes the latest  Cause
you want to RunAwayScreaming....

When body count is the Holy Grail
And "suck up" the success path
More Process leads to  "How to scale"
and Logic invokes wrath
you want to RunAwayScreaming....

yes it sucks!! i know .

but this is my blog and i can write what I want ! so there!

Back To The Seventies

Sergei Dmitriev wrote an article on his blog about what he calls Language Oriented Programming. Sergei claims "It is time to begin the next technology revolution in software development, and the shape of this revolution is becoming more and more clear. The next programming paradigm is nearly upon us." Unfortunately the rest of the essay is not very successful in establishing this claim.

What he seems to establish is that languages like Java and C# are inadequate for serious programming and makes the developer jump through hoops to get things done and results in lots of verbiage.

Fair Enough. The rest of the article explains a series of necessary languages and the tools required for manipulating them . Brian Selinsky rebuts this line of thinking in his blog .

Brian has two objections.

First , he says that custom languages are difficult to understand because they introduce new concepts which have to be understood by the reader. I think he is partly right.Writing new languages for every minor concept change would end up with weird code. If some set of concepts is better expressed as a library , in frozen languages like Java that is a better option.

But on the other hand one does need languages that reflect the underlying concepts. Using an object based library to do what SQL does well, for example is stupid. (I have participated in a project in which the chief designer went object crazy and reinvented database concepts as an object library . Believe me it is really no fun) .

On the other hand when you do need to "build a language to reflect the domain" the best way to do this is with Lisp Macros .(Doubters should read Paul Graham's On Lisp ).The good thing about macros is the the full power of your lisp environment is available at all points of time.one does not need to rebuild tools for each language defined.

Brian goes on to say "It's quite possible that languages like Lisp failed to become popular (beyond a small community) partially because they encourage programmers to write domain-specific languages. This has a tendency to fragment the community into groups that have trouble communicating (a tower of babel effect). " This is simply not true . The failure of Lisp to "take over the world" has many reasons but this is the first time i've heard that the macro facility was the reason . These minor quibbles aside ,I agree totally with Brian. Asking the typical "enterprise and J2ee expert " programmer to hack macros (or for that matter to use "Language Oriented Programming" to define new languages) is a sure invitation to disaster but this is because you macros are a power tool. You wouldn't ask someone whose driving experience is limited to tricycles to drive a ferrari would you ?

Inventing a whole tool chain to re invent what was essentially complete in the seventies seems suboptimal to me.In other words , learn lisp first . If you have a lot of time on your hands write a lisp library that can call into java and use objects defined in java libraries. That will give you "Language Oriented Programming" (oh nonot another buzz word) without a "technology revolution" .

Spells and Software

How hard is the software you write ? Most "business" software is more a matter of repeating endlessly minor variations on a fairly simple theme. Most "product" software consists of fixing an endless stream of bugs.

But if one goes beyond the "normal " part of software (and sometimes one does need to write those endless jsp pages or whatever to put bread on the table - been there, done that ) one sees an endless panorama of exciting, mind blowing software that waits to be written. Most of us software developers are so buried in the soporific details of our "day jobs" that we soon forget the wonder that brought us into software.

My epiphany occured whan i was about 10. I had never seen a computer but one of my cousins had a small "Space Invaders" portable game. I couldn't believe that somone had actually WRITTEN it for me to play.

Software is the nearest thing to magic in the 21st century.

Your thoughts, converted into arcane spell like languages move money, machinery and people to do all sorts of wonderful things. And as with magic, there are Master Wizards and Spell Crafters who extend the frontiers of the art. There are tomes of knowledge which are freely available but remain cryptic "until the student is ready". And of course there are frauds and charlatans(no i did NOT mean "managers"! I actually know a few GOOD managers!All two of them :-P ) . See all the nonsense happening in the name of "test first" or "Extreme Programming" for example.

A few people can spout the jargon. Almost nobody can CODE test first.

Here is an example of some software that I think has a touch of magic about it.The folks at the Cortex brought us Clover and now they are going to bring us fisheye! I hope they make a ton of money off their lovely products.

The best advice I have ever recieved on the subject came from Martin Fowler . I was feeling fairly gloomy about "everyone is fairly happy with the status quo " , when Martin said "Why do you care what everyone else does ? If no one does cool things, there's all the more left for you to do".

A light bulb licked on in that instant inside my mind and has never gone off. These days I don't care anymore that I am an Aging Overweight Indian "software professional" . I can write code .And I can dream.

And in the midst of all the CMM ISO madness no one can take that away from me .

So i am learning LISP , running my own consultancy, planning my Ph D ,learning to play the blues.. All the "weariness of the heart" has dropped away and life is crammed full of interesting things to do and code to craft.

All because a Master Wizard , perhaps unknowingly, passed on a spell to me (and I freely pass it to you ).Four words.

What do YOU care?

The Fowler Effect and The Thief's justification

One of my friends said " You thoughtworkers , (conveniently ignoring the fact that I am no longer a Thoughtworker) always point at Martin Fowler and say 'See how cool he is. Aren't I cool too?' ".

Though this was intended as a joke, there is a kernel of truth to this.

And it is not about Thoughtworks( which is one of the finest companies in the world. I advise any non thoughtworker reading this blog to consider working at Thoughtworks for a year or two.You'll meet some very high calibre peple and learn an incredible amount about how to write good code)

I have seen people from Lucent and even IBM Global Services (!!) and Wipro (!!!) and God Forbid, TCS , walk around with their noses in the air .

There are a few Indians , thoughtworkers and others , who stand on their own feet, technically and otherwise . But there is also a tendency to "derive" one's own coolness from a company, a co worker etc , which I think is shameful .

On a related thread, yesterday I was speaking to a friend who was in a moral dilemma which was all about being attracted to ,what we in our college days used to call an an AMG (Another Man's Girl). To a degree the attraction was reciprocated and my friend was in a dilemma as to proceed further or not.

In my usual brutally direct fashion,I pointed out that the solution to poverty is not theft . You can't steal another man's girl just because you don't have one (or more :-P ) of your own.

Back to the Fowler Effect .If you are poor in "coolness" the solution is not to steal a bit of Martin's , but to generate some of your own !

In general I think what one ought to do is to strive to reach the levels of excellence exemplified by the person(s) you admire. That is not only personally fulfilling, it is also a fitting tribute to your idol.

Besides it puts the "coolness" where it belongs - inside you .

So, yes, I want to be known for my technical skills . Yes I want to speak at OOPSLA someday. I want to write books . And Martin Fowler has been and remains a BIG inspiration, as a person who has "walked the walk". And Thoughtworks has been very educational . But I don't want ANY respect for having worked in Thoughtworks or having worked with Martin . Just react what I do and who I am and I am more than happy. Having said that, I WILL dedicate the first book I publish to Martin and Thoughtworks!

A Tale Of Three Movies

The last few days saw me in the grip of a flu/fever/cough infection that drew my temperature up and enthusiasm down.

So the moment i showed some signs of living to fight another day, I thought i would go and see a movie to revive my spirits. So then I and a friend went off to see Master and Commander - The Far Side Of The World - apretty decent movie about British warship and its battles with a French enemy. More than the naval action though , the story is about the men who sail the ship and fight and die for her. So i was feeling fairly satisfied when I was persuaded to see one more movie and unluckily for me i chose Pooja Bhatt's latest movie called "Paap" (that's "Sin" for you non hindi speakers).

I would strongly urge foriegners who are looking to "understand India " to see this incredibly stupid film. For, in understanding Bollywood's inanities , you would get a glimpse into the hopelessly muddled psyche of the metro dwelling Indian.

First of all if the plot sound somewhat familair, it should be. The script is a rip off Of "Witness"(by Peter Weir, starring HArrison Ford). The hero John Abraham, looks suitably bewildered at various points in the film, but the heroine(Udita Goswami ) who is apparently trying to portray a "Virginal Yet Sensual Indian Woman Tempted By Forbidden Desire" falls flat.She apparently has some lama's trick whereby she can withstand the cold , because she is often shown jumping into freezing lakes with very little on. Jeez just make a soft porn film on something and just be done with it . If Pooja Bhatt wants some direction on how to do this in a "socially relevant" fashion , she should talk to Mira Nair. Spare us the horrendous pseudo philosphy.Mix in a lot of comical lamas and impropable dangling plot threads and you have some idea of what this hopeless movie is like . The best part of the whole experience was the audience , who were rolling in the aisles, laughing at the various "profound " moments in the film.

You know you'd better quit directing when you cook up a scene with "young girl quoting poetry to handsome hunk and hunk falling in love" and the audience is in splits and hooting at the nonsense on screen. As Stephen King once said " Horror and Comedy are the hardest genres to get right. Unless you do it right your horror becomes comedy and your comedy beomes horror". Well, in Pooja's hands Romance becomes rib ticking comedy and the "forbidden desire" motif looks like a couple suffering from simultaneous epilepsy.

Just to remove the bad taste, I put on a DVD of Sholay - a movie made in the 70's and arguably the best movie ever made in Hindi. After the horror of Paap, Sholay was pure bliss. A simple story well told .

A Pity Indian film makers never learn from their few successes.

Software@India

I am beginning to despair of serious design level software development work ever coming to India. The (profit = number of bodies * x $$) paradigm ensures that most work coming to India is of the "here is a framework and here is how you do X. Now do 100 of these. Thank you " type .Of course the framework design happens in the USA .

The same for management . Except a few people who run their own companies or are the CTO/CFO level, most "project management" in India seems to consist of taking orders from higher ups in the USA and "motivating" the technical folks.

So amidst all the rising euphoria about India being the next software destination I am pretty cynical about the whole "our jobs are going to India" clamor i hear from my American friends.

Believe me you folks are better off without these jobs. Go do something interesting with your lives ...

It is not that we don't have talented people in India. But most people i know work at a fraction of their potential. I would guess the reasons are a combination of the Great Indian Laidback Attitude and a lack of opportunity and initiative.

i'm depressed . :-(

The Thinking Programmer

I was reading Edward de Bono 's Serious Creativity yesterday. The key insight that occured to me is that deBono discusses "thinking tools" which are formal structure one uses for thinking through a problem or focus area.

Later i wrote some code for the Ruby port of HotDraw (implementing double buffering to eliminate the nasty flicker when Figures move on the DrawingView). When iwrote a test and then wrote teh supporting code, it struck me!

Programming is thinking too.

You "think in" a programming paradigm/language combination instead of in a human language but is is still thinking. And every one of deBono's thinking tools should be applicable to programming. .

And when you are programming without needing to think, then you are basically doing "manual labor".

I need to investigate this idea further ...

And today , I am doing some really dumb XML->XSL->PDF transformations,inspecting the finished doc visually to make sure that every dot is where its is supposed to be.Yes i know someone needs to do it but it'snot really very exciting work(to say the least).

Yuck, back to work....

The Five Levels Of Programming

Just a mental exercise to distinguish multiple "levels" of programmers.

Some of the observations are India and Bangalore specific but the various "types" should beidentifiable everywhere.

Level 1 : the Novice/the Charlatan At the very bottom we have the "great unwashed hordes" - people who studied computer science because "they said that's the best way to get a job" ".The focus for these people are to get into "better " companies (where "better" is defined as "bigger") and move up the career ladder to become a manager as soon as possible.This is the person who seriously believes in iso 9000/cmm level 5 etc as "improving productivity". .On the brighter side this level also include s a lot of people just out of school and who love to program. and are eager to change the world.

Level 2 : the expert This is the guy who knows everything aboout one particular piece of technology . It could be jsp, weblogic,C++ /the PIC microcontroller/whatever. This is the guy who'll tell you what's happening when a particular problem occurs with the technology he is an expert in.("oh just go to a config file foo and change blah to baz" ).Unfortunately in India, even this level of expertise is pretty thin on the ground. Not surprising when you see that most "IT" jobs don't want really intelligent people. (A friend's HR Manager actually asked him to select the "less talented people" when he conducted interviews beacuse the "hacker types won't like our company". Not very surprising when you get a job because you are cheap(er) than because you are good.) The distinguishing characterisitic of the expert is when people in his team /company say " hmm That's a difficult problem let's call in Mr.x "

Level 3 : The Thinker People at this level think about what they are doing and ask a great number of intelligent questions. Most often they use the simple "why" and other "dumb questions" to investigate . This stage is also marked with growing dissatisfaction about the state of the world and a feeling of "I could do better than that " when they see a piece of sub optimal code/design. (This is similair to how a lot of writers start their careers - they read some tripe and say to themselves "this idiot got published? Why I could do better.").In my experience though a lot of otherwise excellent people "freeze" at this stage and never complete the leap across the chasm that separates them from

Level 4:The Hacker This is the person who has gone beyond wishing to actually writing some code , or perhaps writing a book about what he knows.What distinguishes him from a true guru is that his expertise is narrower in scope and he often does not redefine software programming (or a sub area thereof). The charcteristic that distinguishes hime from the "lower" level is that he is well known within his area of expertise. A good example of this type of person are the tech leads of open source projects like Hibernate or the author of a book like "Regular Expressions".These are true "bearers of light".

Level 5:The Guru These are the Martin Fowlers, the Peter Norvigs , the Stallmans and the Torvalds, the von Rossums and the Larry Walls. What they do profoundly changes the world of programming and influences how the rest of the world programs.

'Nuff said

What If You Couldn't Program?

I think this is a question every programmer needs to ask himself. Suppose for a moment that you got hit by a cosmic ray that removed the "programmer muscle" from your cerebrum.What would you do ? Let me see... I hate marketing so that takes out a whole slew of professions. Practically anything would require extensive retraining / re certifications.A bit too old too back to school. I got it! I will be a writer! the kind that tells stories... Once upon a time there was a whole country of people who were "hackers" but most of them wanted to move on and become Project Managers .... sounds familiar somehow......