An Apprenticeship Pattern

Dave Hoover maintains a list of what he calls Apprenticeship Patterns, which are essentially heuristics that help you navigate the transition from being an "apprentice" programmer to a "journeyman" (The terms "apprentice", "journeyman" and "master" come from the terminology of medieval guilds which used these terms to delineate increasing levels of competence at a craft) .

While I am not so sure if the "patterns" format really fits , Dave has a very impressive list of excellent advice . Please go and read them if you haven't (and buy the book when it comes out) .

Recently a friend asked me how to become "as good as you" ( bwaa ha ha haa ) in programming and I drew out a couple of "patterns " from what i told her .

Here is the first

Book Chain

To gain mastery in any given field the best way is often to read and work through a series of "classic" books in sequence .

Since the books we are speaking about are books about programming there will be plenty of code to write as well.

The problem is that there are often many many books about any particular subject and it is not easy for an "apprentice" to determine what to read and in what order . A person who has already walked the path is often able to guide an "apprentice" in constructing the book chain .Working through such a reccomended chain of books will save you years of flailing around.

Here are a couple of examples of book chains .

Topic : Compilers

First work through Language Processors in Java (by David Watt ) .

This book gives you a "vertical slice" through constructing an interpreter for a (fairly limited or "toy" )language .

Then work through Essentials of Programming Languages (Wand and Haynes) .This book gives you deep insights into how a langauge works "from the inside".

Follow this up with Van Roy and Haridi's "Concepts, Techniques and Models of Computer Programming " which dissects the various programming paradigms and shows how each paradigm is best suited to a particular set of problems and more importantly what the limitations are . Again Paradigms are best understood by getting an insight into how they work internally rather than as random rules like for example ,"use polymorphism in preference to inheritance" .

Next , read a book (and look at the corresponding source code ) that explains how a production quality compiler/interpreter works .

Three books that fit the bill are Lisp in Small Pieces, A Retargetable C Compiler - Design and Implementation and Compiling with continuations

Now you are all set to look into a real world compiler/interpretr/vm (say the Ruby Interpreter or the JVM ) .And once you get through that , you are a "master" (or very close to mastery as makes no difference) .

And now for a slightly less technical example

2. Extreme Programming

First read Kent Becks "Extreme Programming" to get an overall idea . Then read (and work through ) Test Driven Design by the same author. Learn to use a unit testing farmework like JUnit

Then read and work through Martin Fowler's "Refactoring " and Josh Kerievsky's "Refactoring to Patterns" , and optionally Ron Jeffries's "Adventures in C# " . Learn how to operate a Continous Integration Framework (like Cruise Control) and how it works, and then understand how Dependency Inversion works and learn to use a Mock Objects framework like JMock. .

Follow this up be learning how to break up required functionality into stories . To learn more about the XP process read Jim Highsmiths book on Agile Project management , and optionally the Scrum book .

And there you are .If you have learned (and more importantly applied ) all these concepts , you are an XP "master" .

The best way to get a "book chain" for a topic you intend to specialize in is to ask someone who is already expert in the field to give you a sequence of books to study.

A poor second option is to find the "best book to begin with " (for Artificial Intelligence , this is Norvig and Russell's "Artificial Intelligence, a Modern Approach", for example ) and pick and choose from the bibliography.

Light In Dark Corners

Updated to include an opinion on Google India

Joel Spolsky has written an excellent article where he explains why it makes no sense to hire mediocre programmers for Product Development.His thesis is that mediocre programmers just cannot hit the "high notes" that top grade programmers can (read the article) and since the cost of reproduction of software is essentially zero , software products live in a winner take all environment .

He also points out that this does not apply to non product software development . I quote ..

"Sadly, this doesn't really apply in non-product software development. Internal, in-house software is rarely important enough to justify hiring rock stars. Nobody hires Dolly Parton to sing at weddings. That's why the most satisfying careers, if you're a software developer, are at actual software companies, not doing IT for some bank. "

The programmer who is in the top 1% and prefers to create enterprise software vs product development (almost ) doesn't exist .

But beyond that there is one more factor that ensures that companies which create mostly enterprise software accumulate mediocre(and worse) programmers over time . And this is the "per hour" billing model, which immediately puts the goals of the consulting organization and the client organization at loggerheads .

The client organization gains by reducing the total number of developer hours (and the dollar per hour rate of the individual developer) . The consulting organization gains by increasing the number of developers and the dollar rate(which is mostly done by hiring people with greater "number of years of experience , there being no objective way to measure "goodness of value delivered" ) . Thus sheer economic logic dictates that client and consultant pull in opposite directions. Most of the dysfunctionality of enterprise software development arises from this fundamental anomaly .For example the relentless pressure most consulting oranizations face to add "resources" comes from this "increase the number of people on a project " (rather than "deliver more value " ) mentality. And this pressure eventually translates into terrible people getting entrenched in even the finest organizations . Then of course you need more "committees" to manage the mess that results, and new games come into existence , all resulting in the more talented people fleeing or giving up.

This downward pressure forces the client to outsource the work ("You say it takes 20 developers , 3 managers and 5 QA guys? ok but i want 15 developers,2 managers and the QA folks in India at India rates" ) .

Product development work (not maintenance!) is almost never outsourced.Product development companies do the initial development of projects wherever the really good programmers are (and pay them really well . The money involved would give most "enterprise" developers heart attacks) and then outsource the bug fixes, operations etc .(the spiels of "India/China/--insert cheap offshoring destination here-- has top grade talent and so we started an office in Bangalore/Shanghai/wherever" are trotted out to make sure the offshore developers feel happy and often have no basis but the decreased "dollar per hour" rate for mindless repetitive work) .

So the best programmers in developed countries have nothing to fear . The average devloper who can churn out endless jsp/xml/ejb etc needs to be terribly afraid. And rightly so .

The one possible point of satisfaction in enterprise software comes from really understanding a deep clientside problem and writing good code to fix this. Alas, even this is denied to offshore programmers who have to play many games (see Eric Berne's "Games People Play") and fight amongst themselves like dogs to get one of the coveted "onshore" assignments and get anywhere near the real clients(where they are still paid less than the local developers) .The only strategy(in the Game Theory sense) that has a positive payoff in the offshore developer's world is to join the ranks of management as soon as possible and leave "lowly" work (and most of the work is terrible) behind . Then you can enjoy a "money for nothing" existence (at the cost of having a "meetings, committees and paperwork" life and losing your technical skills, if any ).

There are some efforts going on to change the basis of payment from the insidious "dollar per hour" concept .But so far nothing has been very succesful because changing the game would mean that both clients and consultants would have to actually think of what they are doing and calculate the ROI of every feature they add(and God forbid that such heresies come to pass :-) ) .

Meanwhile developers , if they are hell bent on developing enterprise software, should probably join one of the better companies around and pray (very, very hard) you get a greenfield , challenging project.

But the question remains.If you want to work with the best people and have a satisfying life working with technology why flirt with Enterprise Software at all? Do you really (really really) get your kicks writing Banking Software ?

And even if you do, it may make sense to constantly re evaluate your commitment .Here is a simple way to do this .

When you get up in the morning, knowing you have to go to work , do you smile or grimace? And when you finish the average working day ,are you happy, having learnt something new and created good features ?Or has the daily grind just chewed up another 8 (more likely 12, if you are in Bangalore) hours of your life leaving you drained and exhausted ?

Update : Apparently Google does things differently (Why am I not surprised? ). I say "apparently" because I have no confirmation yet.I was told that Google's India Office is really about hiring only top notch talent and not outsourcing for cheap labour.Apparently anyone who makes it through their (very tough obviously) interview processes can select which office they want to work for .In other words at the end of your interview process you can choose to work out of,say California or Zurich . If true , this is absolutely incredible and needs wider publicity in Bangalore!

Anyway this blog entry was mostly about the yuckiness of most outsourced enterprise projects and not about product companies,although it remains true that most product development is done abroad . Google is about as far from "yucky" as you can get .