So what's wrong if you aren't an engineer?

Nothing at all! To quote Reg Brathwaite (again! But the man has a way of using words that is very eloquent. I can't resist).

[This is the second in a series of four blog entries. If you haven't already you might want to read part one first.]

What's wrong with being a clerk? Nothing. It's only a problem if deep in your heart you despise clerks and you spend your life in denial about the career you have chosen. I wouldn't wish that on anyone, so I asked my readers to think about that carefully.

Likewise, we can argue about what activities from programming can or cannot be considered Engineering. But really, even if you don't do any Engineering, what's wrong with that?

Heh!.

My last blog post seems to have ignited a mini firestorm. Observing how people react to an idea is sometimes more fun than the original idea itself. Reg gets it. In one of his replies to a comment (do read his blog entry), he says

"the statement “~p implies ~q” says nothing about whether p implies q." .(typo corrected. Thanks Arne!)

This is a bit more subtle than it seems. Try substituting p = "You use mathematics" and q = "You are an engineer") and try working out "~p => ~q" and " p => q" ( => is implication and ~ is not). You may be surprised !

I am continually astounded at how many people respond to arguments or claims without doing a logical analysis of what's being said. (Note: I am using "argument" and "claim" (and other words like "theory") in the logical/scientific sense, NOT in the " I had this 3 hours argument with my wife. She asked me to wash the car and I refused. At the end I was shouting and she was in tears. My theory is that women are a different species" sense).

When I was a debater, in my younger days, one of the lessons I learned early is to understand that you don't counter an argument from your emotional or "gut" level, by calling your opponent names, or attributing motives to him (unless you are trying to be a politician, when these tactics do pay off).The way to counter an argument is to dissect its logical structure, and show it is invalid (in specific contexts, if required). Rhetoric by itself can be powerful (and many politicians know this), but when layered on top of a logically sound argument is devastating. There are very precise ways of doing this, going back many centuries, at least as far as Aristotle and Plato.

In my last blog post, I made the claim that most software developers are not engineers. Here, I'll make another claim, even more provocative. Most software developers don't understand logic either. You think I am wrong? Quick, (assuming you are a software developer) what is the difference between the "if .. then" construct in programming languages like java and the logical "if ..then" (aka implication, often denoted by =>) ? If you, a software developer, answered correctly without having to think about it, rest assured, you are in a minority.

Using logical implication you can say(assuming we are talking about this Earth and this time stream) "If Napoleon Bonaparte was born in Europe, the Sun rises in the East" and have it evaluate to true. But of course. What's so surprising? "If Napoleon Bonaparte was born in India, the sun rises in the West" or "If Napoleon Bonaparte was born in India, the sun rises in the East" also evaluate to True! :-D. ("If Napoleon Bonaparte was born in Europe, the sun rises in the West" evaluates to False.) What does the birthplace of Napoleon Bonaparte have to do with where the sun rises? ;-)

Confused? heh! Don't worry it is a most people get totally zonked when they see this example for th first time. The key is to realize that implication is not causation. (neither is correlation but that is another topic. See this debunking of a claim that "research supports the effectiveness of TDD" to see an example of correlation vs causation - The trick I pulled is of course that many variants of the English "If.. then" are different from the logical "if ..then". [1]). Many people learn the truth table of implication without really internalizing what it means.

When I teach programmers first order logic, this is a constant stumbling block. The solution is simple. I ask them to think of the logical "if X then Y" (where x and Y are booleans or boolean valued expressions) construct as equivalent to a programmatic "If X then Y else True". The "else True " is key. In other words, (thinking programatically) does X have a value of true? if so return (the truth value of) Y else return true. Apply this to the "Napoleon" arguments and you'll get the correct (logical) answer for all possible combinations.

Why is such a confusing notion important? Proofs are logical structures using the primitives of FOL. A large part of mathematics(and science) is proofs. From science comes engineering. and if you are not using mathematics you are not an engineer (ducks for cover ;-)).

There are 5 connectives (not, and, or, if (or implication) and iff (or double implication) ) and two quantifications (Universal and existential) in First Order Logic, which need to be mastered before one can go on to things like proofs and logical structure. That's the bad news. The good news is that working through a book on logic (and there are PLENTY of those) will teach you how to use logic.

Reg goes on to say (in the same comment stream)

I remind everyone that "exists x such that x ~member of E does not imply that for all x, x ~member of E."

In other words, the fact that not using math means you aren't an Engineer does not imply that using math makes you an Engineer, for whatever definition of math we agree on.

Exactly so!

The point of the last blog post wasn't "you suck you enterprise developer subhuman moron", but "Don't delude yourself". No more, No less. Do what you love. And have fun!

PS: Once you know logic, using to to construct (or deconstruct) an argument is trivial. But for those who want to make sound arguments without necessarily studying "raw" logic (I hope you are not a sw dev ;-)), take a look at "The Craft of Argument" by Joseph Williams and Gregory Colomb.)

[1] Wikipedia

Part Three of this series is here.

Ratcatchers and Engineers

Ever wondered if what you are doing is (software) engineering? Here is a heuristic.

If you don't use mathematics in your day to day work, you aren't (an engineer). All engineers (say those who build bridges, or space craft, or cars) make heavy use of mathematics and/or hard sciences like Physics on a regular basis.

Now, not being an engineer is ok. Being a carpenter or a plumber is a perfectly honorable choice as is being a musician or actor or teacher. If you enjoy being a carpenter/plumber/automobile mechanic, more power to you. You should do what makes you happy and puts bread on the table. That said, a craftsman is not an engineer. The guy in the garage who fixes your engine is not an automobile engineer who could design the next generation car. Not close.

This insight was triggered by Raganwald's "No Disrespect" blog post. I quote

...Let me tell you the cold, hard, truth. You aren’t going to like this, but I ask you to believe me when I say that I am telling you this for your own good:

There is a culture of pretending business programming is more than it is. Some of you calling for more Java in University may take false hope that I am on your side. You may think that the people arguing for Scheme, Haskell, and OCaml are elitists. Wrong. They do not have a problem. You are the one with a problem because you don’t want to tell all your friends you have a job as a clerk.....

We all know what the typical software "engineer" job ad looks like. A job ad for a real engineer would look like this. (Noe the absence of "10 years in java/dotNet/Ror" type crap. Note that he explicitly asks for a Phd (and tell you under what circumstances he will waive it)

The distinguishing trait of an engineer (and Werner's job description explicitly ask for this) is that he builds and works with mathematical models to design a real world effect or system. They also use other tools(simulations, prototypes, experiments etc) but (mathematical) modeling is key. A scientist,as distinct from an engineer, uses roughly the same tools to advance the state of knowledge without necessarily affecting the real world. The borders are fuzzy. You have scientist-engineers and engineer-scientists, as well as people who focus on "pure" science or engineering.

"Modeling" is a deep topic. Read the book I've referred to at the end of this blog for examples of how this works. Suffice to say If I am not building (say) algorithmic models to help me decide how to build my software, or to generalize, if I am not using "theory" on a day to day basis, I am not *engineering* anything. Modeling has a very precise meaning in Engineering and Science. (No, UML diagrams, or "story cards" are not engineering artifacts no matter what the methodology vendors say ;-) ).

Enterprise software is the least amenable to the modeling/engineering approach. There are exceptions but most "enterprise" developers are the equivalent of clerks, as Raganwald so eloquently points out. There is nothing wrong with being a clerk as long as you know you are one and are not deceiving yourself. Most enterprise software projects, in keeping with their clerical nature, are life draining. But hey if you like it, go for it.

Another friend of mine, who is a very talented programmer (he recently moved away from enterprise app development) , when asked why he changed the focus of his career, told me,(paraphrased) "Humanity has only a very limited amount of talented people. It is a crime against humanity to employ that talent to bug fix enterprise applications or futz around with Ruby On Rails deployment issues. I want to do something meaningful". (As you can see I have interesting friends :-) ).

To conclude, the title "Software Engineer" is (most of the time) a particularly deceptive one. To be accurate it should something like "Software Maintenance Worker" or "Software Handyman", but I guess it is easier to hire someone if his job title is "Rodent Officer" instead of "Ratcatcher". .

PS:- Please, before anyone feels offended and sends me hate mail or snarky comments, please read (something like) "The Idea Factory - Learning to Think at MIT". (This was the book that jolted me out of my complacency and set my feet firmly on the research/engineering path). Then actually think about it :-). Either be happy as a rat catcher or do something about it.

[This is the first in a series of four blog posts. Read part two and part three]

Dev Camp Bangalore

Some folks at my former employer, Thoughtworks, are planning a DevCamp in Bangalore on February 9th, 2008.

This is an excellent idea, well overdue. Barcamps in Bangalore are overrun by people trying to finance the next "social networking" (gag!!) or "web 2.0"(double gag) "startup" by finding some dumb non-technical vc who doesn't know what's going on, photography and movie clubs and so on. And what technical sessions do exist are of the "Introduction to X" variety where X element of { Ruby On Rails, Erlang, Fad-du-jour} mostly cut and pasted from the web.

The Dev Camp web page has these instructions prominently displayed (emphasis mine).

"However, please assume a high level of exposure and knowledge on the part of your audience and tailor your sessions to suit. Avoid 'Hello World' and how-to sessions which can be trivially found on the net. First hand war stories, in-depth analysis of topics and live demos are best."

This should keep the fakes away (touch wood). Also there will be a few participants from Thoughtworks, and if the past is any guide the Thoughtworkers sessions will be well worth listening to.

One of the tenets of a "camp" is that there are no passive participants. Given that this is meant for developers, I hope to attend some high quality sessions. And if I attend, in the spirit of Xcamp,

I'll be presenting too. I will speak on one of

1. Monads in Depth - what they are, the underlying mathematics, what they are good for and how to use them in your favorite language
2. Reinforcement Learning - Algorithms and Applications in Robotics
3. Proof Techniques - a tutorial for Developers
4. Vajra - a high performance Lisp for Robotics Programming

Of course the topic list is highly fluid and I just might end up speaking on something else, in keeping with the spirit of XCamp, but if you have a preference, send me email!

if any readers of this blog attend, stop by and say hello!

Sliding Into Scala

I've been dissatisfied with java for a while now, but to give the devil his due, it does hit a sweet spot. When you are looking for a combination of cross platform, fast, statically typed, easily deployable language with tonnes of libraries, there isn't much else available. But on the other hand the language itself is mind numbingly verbose and since most of my coding these days is in a combination of scheme and C, when I do switch back to java, it is as if I am suddenly running through quicksand, eclipse notwithstanding.

Generics (the implementation not the idea) was the first misstep in Java's evolution. It makes it easy to write code that is almost impossible to read. For the AIMA code for example, when I used pure Java 5, generics heavy code, students complained that they couldn't make out what the code was doing and so for code that other people have to maintain or extend, I end up writing Java in a style I call "1.4 +" - Java 1.4 + enums + new for loop + generics *for collections* (only). The actual type system of Java 5 is a fairly simple (even simplistic) one (once you've worked through some books like TAPL), but the syntax is atrocious. The present controversy on "closures" (closure != anonymous function and I am tired of programming illiterates abusing terms with clearly defined meanings but that is a rant for another day) convinced me that Java is on the wrong track and is well on its way to obsolescence. "Java is the new Cobol" indeed.

What COBOL never had was an open source cross platform VM that other languages could target and a few hundred thousand libraries. Java != JVM, in other words. Couldn't I just use another language on the JVM ? Well yes, but - I don't particularly like Ruby. Jython hasn't caught up with the latest version of Python. Both these languages have dynamic type systems and are slower than compiled java. So there isn't too much actual choice given the parameters I listed above (fast, statically typed ... ).

Enter Scala. I looked at Scala a year ago, tried some sample scripts, stumbled across a bug and gave up. But things have changed since then. I've been experimenting with Scala over the last few days and I am tremendously impressed. Besides writing small scripts to explore various language features, I've been rummaging through the code for the compiler and type checker (written in Scala of course). Martin Odersky has something that is all too rare in the software world today - a strong sense of design (I am looking at you, Ruby On Rails).Scala has too many brilliant features to go into any great detail here, (there are plenty of blogs that do go into detail see this or this for e.g) but in essence code compresses to almost nothing, the type system is brilliant, and pattern matching is something I've wanted in a jvm language for a long time. One can even do monadic programming in Scala! I am VERY impressed.

Now don't get me wrong, there are a few rough edges. For example one of the things it doesn't have (as of this moment) is a good reflection API. I tried to write a xunit clone to get my feet wet and got stuck on identifying all the methods that start with "test", for example. (It *can* be done by leveraging the underlying Java but not from native Scala). The build system is a mess (try building from source and watch the build fail for lack of heap space - this is unforgivable in 2008) and there is less of a focus on testing/regression than I'd like. And oh, if you are using the interpreter make sure you use jline ("sbaz jline" should do it- I don't know why this isn't part of the standard distribution).

But these are minor quibbles - Scala is a brilliant accomplishment. I think Sun should just declare Scala to be Java 8 and be done with it.

Managers consume what?

from this site,

....managers are voracious consumers of theory. In other words, they value ways to think about their world, and mental tools that will let them make decisions and predictions with a level of confidence higher than they get from experience and intuition alone.

Ummmm. Ok. :-P

Teaching Mathematics

I've just taken up an assignment to mentor one of my friends in maths/proof/logic.

My "student" has no background in maths or logic (if you ignore the negative knowledge gained by sitting through a bachelors degree in Engineering in India's f***ed up educational system), but is a good programmer.

The expected post condition is that my "student" will be able to read and work out mathematical proofs, design an experiment, analyze an existing research paper, work out an algorithm for a given task, understand basic calculus and linear algebra and in general gain the ability to conduct independent research in Machine Vision. I (and some others, like my friend Rajesh, for e,g.) have worked out some of these "how to"s over the last few years, mostly by brute force search of the landscape, discovering pitfalls by falling into them (and making a careful map, which it is now time to pass on).

"How to work out a proof" for example is rarely taught (well) in a classroom setting and I have had to synthesize knowledge from many sources to come up with a workable technique. I can save my "student" hundreds of hours of work.

Teaching is how one crystallizes one's knowledge. I've always enjoyed teaching but what I've done earlier was in a many to one to many format, involving speaking to a group of people in using a blackboard to derive a proof or a projector to display code.

The one to one format will be a new experience. I spent a few hours today working out a "syllabus" and it looks very good, even if I do say so myself :-)

Update : I plan to teach first order logic and proofs first, probably using books by Velleman, Polya, and Doets and Van Ejck.

You can't catch up

One of my friends recently told me (I have interesting friends :-)) "Every time I meet you, you have moved further ahead in [area of focus], so how do I catch up with you?"

The answer == "You can't, unless (a) I stand still and wait for you to catch up - Why should I? (b) You figure out how to move a few hundred percent faster than me (Good luck with that)"

Imo, the right thing to do with life is not waste cycles "catching up" with other people. I'd rather do something that is uniquely mine. Competing with oneself is hard enough without wasting bandwidth on what other people are doing or how they are "drawing ahead" or whatever.

Robotics and the future of warfare

I was speaking to one of my friends doing research in robotics in a major American university when it struck me that most of the researchers in robotics and machine learning are gentle pacifists who wouldn't want to hurt a fly, while one of the greatest uses of robots is going to be in real battlefields.

And I am not talking about robots engaging in non lethal activities like scouting or mine clearing, though that's how robots will initially appear on the battlefield. I am talking of robots (on land ,sea and air) armed with guns and other weaponry and increasingly deciding for themselves when to fire and what to fire at.

I believe that in wars of the future two major features will be asymmetrical war (what we call "insurgency" -blurring the boundaries between "civilians" and "soldiers") and increased participation by increasingly sophisticated autonomous robots/remote control weapons. (see this column for some thinking along these lines). I'll ignore the first aspect and focus on the second, in the interests of keeping the size of this post manageable.

Here is how robotics will develop in war. First, robots will engage in non lethal activities like mine clearing or I.E.D detection. (This is happening today) Then you'll see them accompany human combat units as augmenters and enablers on real battle fields. (This is beginning to happen) As robotics gets more and more sophisticated, they will take up potentially lethal but non combat operations like patrolling camp perimeters or no fly areas, and open fire only when "provoked" (This is beginning to happen too). The final stage will be when robotic weapons are an integral part of the battlefield, just like "normal", human controlled machines are today and make autonomous or near autonomous combat decisions. These robots will look nothing like the vaguely humanoid robots of Star Wars - they will have shapes suited to their roles - think tiny dragon fly / cockroach like systems to computerized tanks and helicopters.

And this type of robotic battle system will not be confined to the major superpowers. As of today, what gives robots an edge is not really dependent on hardware, but on sophisticated software, something that does not depend on superpower status. Someone in Korea or India or China, or even a semi military corporation can gain a temporary edge in robotics software comparatively easily (though the Red Queen Effect will kick in soon enough to close the gaps) sufficient to tip a battle or a war one way or another.

The funny thing is that quite a bit of the required technology is available, sometimes in embryonic form, today. "The future is here, just not widely distributed" - as William Gibson said. I know that the Indian defense forces have made substantial progress in battle field robotics systems (I can't be more specific - the details are classified so don't ask! :-D ), and what India is on the verge of doing, others are, too.

None of this is to claim that future combat will occur between machines, all clinical and detached with no blood being spilled. On the contrary.

Historically, every innovation in technology has been hailed as bringing in an era of bloodless warfare. When the airplane was invented, for example, many columns were written on how they made war obsolete because, obviously, with planes roaming the sky no army could make unobserved movements and so war was now futile! Dresden and the Blitz and Hiroshima were just around the corner.

My prediction is that war will become more lethal, with increasingly sophisticated autonomous weapons systems and ever more vicious asymmetrical warfare twisting the business of making "the other fellow die for his country" (or ideology or religion or ...) into newer and ever more horrific forms. And that is something that all of us working on robotics or machine learning should think about.

Update: First Armed Robots deployed in Iraq . While these are not really autonomous systems (they are remote controlled) and substantial technical challenges remain on the path to truly autonomous systems, people conducting research in robotics know how fast these barriers are falling. Those who are not researchers in robotics/ai may mull over the progress in autonomous navigation of vehicles from 2004 to 2007 as exemplified by the DARPA Grand Challenge. Now extend that rate of progress to areas other than navigation.

Books for 2008

These are the books I'll be working through in 2008.

Meet "Spineless"

Spineless is a small Django app my friend Manoj whipped up to help me keep track of my books. People generally borrow books from me and I have a hard time remembering who took what when. Even after giving away almost all my 300 or so java/j2ee/dotNet/"agile"/"enterprise" books, I still have a few hundred books lying around in Bangalore and more in Trivandrum. I have entered the first 50 or so books into the app (actually I type some keywords and the app looks up the books in Amazon and grabs the details and plonks them into a database). There has been some demand for something like this, so all who asked, enjoy! I'll keep adding books whenever I find the time (which is a commodity in very short supply right now :-( ).

One of my friends asked me to enter my book list into Library Thing, but I have a visceral dislike of "social networks". Maybe too strong a mental association with Ruby on Rails, I don't know exactly why.

And before you ask, the name doesn't really refer to books' spines or anything to do with books really. It has to do with a private joke about something someone we know said recently. Manoj liked it so much that he named the app after it. Mention "optimization" to Abey or me and we'll roll around laughing. Mention "Spineless" to me or Manoj and you'll get the same reaction. Commemorating stupid things said by intelligent people (or vice versa) is a bit of a tradition in my circle of friends. What's life without people to make you laugh ?

Fish Eye Plans

One of the emergent effects of my time management meta focus is that I have I call a "fish eye" plan.

I know in great detail what I will be working on in the immediate future (say the next few days). Next week, I know, but in less detail.I know what I'll be working on next month, but at a higher level of abstraction. Next quarter, even more abstract, but I *do* know what I'll be doing. So if you ask me what I'll be doing in (say) March 2008, I'll have an answer. Not many people can answer that question. This is a very refreshing change.

This is not to say that plans won't change. They will, but you know what they say about failing to plan.

What pleases me the most is that I didn't set out to do this. I just followed Dr Randy Pausch's "system" of time management (with many tweaks and some significant additions from elsewhere) and it all just fell out.

Life is good.

Question: Is anyone in Bangalore (anywhere in India will do at a pinch) doing significant programming in Haskell? Mail me please. Thanks in advance.

I won't go seal hunting with you

I told someone a few minutes ago "I think you are very capable and like you a lot but I don't trust you one inch". (Yeah I do say things like that in real life :-D).

After the (very unproductive) conversation was over, I was trying to recollect where that phrase/concept came from. It was certainly not an original thought/phrase and I was sure I read it somewhere. Turns out it was from Edward de Bono who said (paraphrased) "some languages have one word which says 'I like you very much, but I would not go seal-hunting with you'." so one can say (in that language) "I [word] you" where [word] has the meaning of the phrase above.

I forget the book he used the phrase in but iirc, the context was that sometimes if we lack the precise word for something, we need to repeat a long phrase.

This holds in technical conversation too. We don't say "change the structure of the code without changing the behaviour", we say "refactor". In the absence of the word "refactor" we would probably use the longer phrase, till someone got tired of it and invented the word.

Back to seal hunting. In this case, the context was one of doing a startup together and I suggested the phrase as an explanation why a few people don't "openly talk" to him, while the other people do talk to me "openly". Note that this doesn't mean that person X *is* untrustworthy. All it means is that I (and a few other people) *think* that in a crunch, X wouldn't stand by us and would do what's best for him vs what's best for us.

Is not trusting someone a problem? In a normal context, not at all. We all work with people whose competence we admire but we wouldn't want to trust them to keep our objectives and safety in mind when the stakes are high. We just add a double dose of caution, keep our guards up and are careful with words around that person, read the fine print very carefully and so on.

I am not very sure that would work in the context of a startup though. In a startup (from what I've read, not that I've ever founded a startup, though I've been employed in a couple) the pressure is so intense that one doesn't have the time to do all this armor donning and being careful. Founding startups is not like seal hunting -it is like hunting Great White sharks or the aliens from the movie "Predator".The founders absolutely *must* trust each other to do the right thing when the bullets (or harpoons or laser beams) are raining down.

Another thing that I noticed from the conversation was the use of phrases which don't really mean anything but are invitations to games (in an Eric Bernian, Transactional Analysis sense - the person proferring the invitation doesn't often know what he is doing, from a group dynamics perspective).

For e.g the conversation started with "I don't want to jeopardize our friendship but .. " and I was thinking "What friendship? we aren't friends - we were never friends- We are professional acquaintances who speak about technology occasionally and have met at a couple of parties".

To me a friend is someone who you have let into the "inner courtyards" of your life. You can have many positive, enriching relationships which are of a lesser intensity than a friendship, but the word does have a defined meaning and it didn't make sense in this context.

That opening phrase puzzled me for a while, but then I realized that he didn't really mean it either. It is just a phrase the meaning of which depends very much on the context of its usage, in this particular instance meaning something like "I am very uneasy about the fact that people don't relate to me like they relate to you. I am going to vent my feelings a bit but I don't know how you'll react to that, so I'll just use this phrase and hope you believe " maintain friendship" is the reason I am screaming".

Real friends don't need to use those kind of weasel phrases and would say something like "I don't know wtf you are doing, but I am concerned" or something to that effect. I try to avoid these meaningless phrases as much as possible, not because it is a particularly noble thing to do, but because I don't want to clutter a conversation with superfluities. "Say what you mean, mean what you say" is a surprisingly effective, not to mention efficient means of communication.

If you keep an open mind you learn something from the most trivial incident. I learned a few things today. Not too bad for a 20 minute conversation. Not bad at all.

A list of topics

My focus on Time Management has affected how I write on this blog. Earlier, when I felt strongly about something or when I was feeling bored, I'd just dash something off. This wreaks havoc with schedules and "getting things done",and I have often stopped writing for a few weeks, months whatever. But this is dissatisfying in its own way.

Now, when I think of something that I feel like writing about, I just add it to a list (on a wiki, if I am at a computer, in a small notebook if I am not) and carry on with what I am doing. When a "write a blog post" size chunk of free time comes around, 15-20 minutes, typically) and I am working on the comp, I just pick an item off the list and write about it.

The style of writing is different than it used to me because I am writing against a timer. More direct, less textured, which is probably the appropriate style for a blog post. The longer wait between conception and recording also allows the good ideas to percolate and the bad ones to show their unworthiness.

For anyone who is curious, my present topic list (this is a continuously mutating entity) looks like this.

1. San Fransisco vs Bangalore - the right place for software startups?
2. The Idea Bottleneck and how it really isn't [DONE]
3. Stealing a leaf from Paul Graham.
4. SICP Stumbling Blocks
5. A Billion Dollars And Then?
6. Topic List on a Wiki [DONE]
7. Practical SICP - How SICP helps in real life projects
8. Derivative Trading as Programming Pedagogy
9. Growing into a (technical) book
10. Memorizing Shakespeare (and Shaw)

This is just a record of things I might write about. But the time management meta focus seems to be paying off already. I am writing more frequently than usual, and not feeling stressed at all.

Someone stole my cell phone

so if you are reading this, and are on my phone list, please send me your phone numbers to repopulate the address book on my new cell phone.

Thanks in advance.

Update - I got a new sim card but decided not to get a new phone till I complete my research paper.Meanwhile, please use email.

The Idea Bottleneck That Isn't

One of my friends was talking about doing a startup in Bangalore. This is a conversation that comes up regularly. I tried to get a startup off the ground twice, but failed because the people who I wanted to work with didn't live in Bangalore and I didn't want to move.

Those attempts isn't what this post is about. Because this friend was talking about a startup, and wanted some people to work with him , I spent some time talking to other friends who might be interested (I am well connected to interesting, technically capable people who are hungry for success and money).

One question I keep hearing is "What about ideas? Do you have an idea?". I have never understood this question. Generating decent ideas has never been a problem for me. I could generate 20 workable ideas in 20 minutes. And repeat that exercise a few times before I need to pause.

The first time I encountered this "idea bottleneck" concept I didn't pay it much heed. But then someone I respect said I was not paying enough attention to "finding an idea" if we were to attempt that hypothetical startup. So I reached into my mental pile of ideas and gave him an idea to work with, while we work through other issues. He seems happy about it and sent me a link showing that a company with a similar idea (actually a trivial subset) had been acquired for 50 million $ cash.

Another friend heard of this and said "Dude, how can you give up your idea like that? What if he rips you off? Be careful"? First, the friend I told the idea to is a decent chap. Second, I don't really care if he (or anyone else for that matter) "rips me off".

My operating concept is that having ideas per se isn't all that important. It is the actual execution that matters and so I am not afraid of people "stealing" my ideas. It isn't as easy to do that as many people think. The idea I suggested to my friend, for example, sounds simple but there are all sorts of interlocking little problems that all have to be resolved perfectly for it to be successful. I can imagine someone spending a few months (and a few hundred thousand dollars) in an attempt to get it to work and not getting anywhere, because he isn't aware of what to tweak to get a particular effect or how to fix a subtle "error". In fact if anyone can execute the idea better, faster or more comprehensively than I can he is welcome to it.

Paul Graham says it better than I can (in news.ycombinator).

.... if we told everyone about everyone else's ideas, it wouldn't be as bad as you might think. Secrecy is not as important as beginning founders think, because (a) ideas are less valuable than they think, and (b) the most common form of death for startups is suicide, not being killed by competitors."

The last sentence says it all.

Update: One hour after this was written, the proposed startup project committed suicide :-D.

R.I.P :-D

Experiments with Time Management

After viewing Randy Pausch's lecture on Time Management, I've been experimenting with various time management practices. (yeah I know how management-ey and buzzword-ey that sounds, sorry). It helps that I am going through the busiest month in my entire life, so I am more motivated than usual to make this work.

The most successful habit I've adopted is to put a money value on my time. As Randy says in the first few minutes of his talk, people are generally willing to give you their time, but not their money. If you ask people to spend a quarter of an hour to help you arrange the furniture for a talk, they will happily do so, but they won't hand over 10 $ for no obvious reason.

Once you start putting a dollar value on your time, though, strange things happen. For example, I decided not to attend Barcamp. (I'd registered, I cancelled it). Say an hour of my time is worth say a measly 25 $ (being a cheap outsourced-to Indian programmer). Travelling to and from Barcamp, and attending 3 or 4 sessions would take, at a minimum 6 hours. 6 * 25 = 150 $.

Would I spend 150 $ to attend Barcamp? Not on your life! To be honest, there are other disincentives also - I really don't want to attend a meet which is so "implementation lite" - where there is so large a focus on things like search engine optimization (!) and blogging.

People actually think blogging is a big enough deal to form "collectives" about blogging? - there is even a "collective" for "radical ideas" -- yeah right, that makes sense .. NOT. The mind boggles. I consider blogging the intellectual equivalent of writing a post-it note. I cannot imagine self-identifying as a "blogger" and making a big deal out of it. God have Mercy.

Back to Time Management. Since I don't (in general) attend phone calls from unknown numbers/people when I am working, I don't have interruption issues. And once I concentrate on something, it is hard even for people (even if they are sitting right next to me) to jerk me out of the zone - I don't hear anything when I focus on work so they'd have to nudge me, and risk getting a laptop thrown at their heads) so I didn't benefit so much from that part of his talk. But the money value of time is an excellent perception to have. Would I spend 50 $ to have a good conversation with a friend/someone doing interesting things? definitely.

He is also right about keeping a time journal. The result of such an experiment is the most horrifying document you will ever see in your life. The good news is that with such an abysmal base, you can make substantial improvements in a very short order.

To round things off, a couple of snippets from a Phil Greenspun article.

Ask a wage slave what he'd like to accomplish. Chances are the response will be something like "I'd start every day at the gym and work out for two hours until I was as buff as Brad Pitt. Then I'd practice the piano for three hours. I'd become fluent in Mandarin so that I could be prepared to understand the largest transformation of our time. I'd really learn how to handle a polo pony. I'd learn to fly a helicopter. I'd finish the screenplay that I've been writing and direct a production of it in HDTV."

Why hasn't he accomplished all of those things? "Because I'm chained to this desk 50 hours per week at this horrible [insurance|programming|government|administrative|whatever] job.

So he has no doubt that he would get all these things done if he didn't have to work? "Absolutely none. If I didn't have the job, I would be out there living the dream."

Suppose that the guy cashes in his investments and does retire. What do we find? He is waking up at 9:30 am, surfing the Web, sorting out the cable TV bill, watching DVDs, talking about going to the gym, eating Doritos, and maybe accomplishing one of his stated goals.

Retirement forces you to stop thinking that it is your job that holds you back. For most people the depressing truth is that they aren't that organized, disciplined, or motivated.

This is so true it is not funny. And one doesn't have to retire to encounter this phenomenon.I know many people who drop out of the 9-5 rat race for various reasons ( startups, research , search for the meaning of life .. whatever), and then find that months and years pass with nothing substantial being achieved.

Another interesting suggestion from the same article.

publish a public Web diary of what you do every day, thus discouraging you from wasting time because you'll be ashamed to admit that all you accomplished yesterday was a 15-minute oil change and a trip to Target

That is a very interesting idea, but a little forbidding, exposing one's lack of productivity to the world. But I should probably grit my teeth and try it for a month or so. If you want to see what this would look like for someone who is really really good at what he does (vs humbler mortals like you and me) see John Carmack's dot plan files (warning - Flash, sorry I couldn't find an html document of these and I am running out of my "write a blog entry" time slot). There's even a theory that these (Carmacks' plan files) were the origins of the present day blogs, no matter what Dave Winer says.

While I don't quite have the guts to "dot plan" my puny daily efforts, I could probably list down what I plan to do each month on the first day of the month and a report on the last day of the month detailing what actually got done.

So, as an experiment, here is what I need to do this month (November 2007) (client work excluded - privacy concerns blah).

1. Research - Solve the last 2 issues, code, write up research results into a paper and send out for review. Hard deadline of 24th Nov
2. Write a Reinforcement Learning library for the friendly folks at at the DRDO. This is a a lot of work
3. Implement the algorithms in chapters 9-12 of AIMA. (First Order Logic and planning, for anyone working through the book). Release the next version of the AIMA Java code.

Even with the newly adopted time management practices, that is a tonne of work which brings me to -

Blogging Time Over. Back to Work.

Hackers Month of Learning Haskell aka X Y Month

The local newspapers are all going crazy about National Novel Writing Month. Apparently NaNoWriMo is some kind of cultural phenomenon in which random people commit to writing 50,000 words in a month, hopefully finishing a (substantial part of a ) novel.

I am not sure any worthwhile novels actually get written this way. If you won't do it anway, you probably won't do it just because other people are also doing it but the idea has its possibilities. The "national" doesn't make sense in a networked world, so we'll replace it with the variable X.

Instead of novel writing we could do something more fun so we'll call it Y.

Let X = { .bunch of adjectives...} Let Y = { ..bunch of verbs.. }.

so we could have "Personal Month Of Derivative Trading" or whatever.

From the nanorimo website,

Valuing enthusiasm and perseverance over painstaking craft, NaNoWriMo is a novel-writing program for everyone who has thought fleetingly about writing a novel but has been scared away by the time and effort involved.

I can certainly think of analogues in programming. many people want to learn Haskell, (say) but never get around to doing it because at any given point of time, there are more important things to do. maybe someone should declare December to be "Hackers Learn Haskell" month and ask anyone participating to write, say, 500 functions in Haskell in a month. Or work through SICP completely. Or whatever.

from the nonwrimo site, Because of the limited writing window, the ONLY thing that matters in NaNoWriMo is output. It's all about quantity, not quality. The kamikaze approach forces you to lower your expectations, take risks, and write on the fly.

Make no mistake: You will be writing a lot of crap. And that's a good thing. By forcing yourself to write so intensely, you are giving yourself permission to make mistakes. To forgo the endless tweaking and editing and just create. To build without tearing down..

Programmers write a lot of crap code anyway so at least that part won't be new.

The Algorithm Quotient of a Software Project

Still thinking of the DARPA Car Races, it strikes me that a project's dependence on algorithmic innovation for a successful outcome is in direct proportion to its "interestingness" froma developers point of view.

Think about it. All the cars that autonomously traverse 160 miles of desert or travel through an urban environment while obeying traffic rules have essentially the same hardware. The difference in performance is about how algorithmically sophisticated the *software* of each car is.

On the other end of the spectrum most "enterprise" projects are algorithmically trivial (I said "most" so if anyone's ego out there is tied to his knowledge of JSP or whatever, spare me the righteous indignation) and the most complex data structure used is often a hashtable.

I don't think this is always a valid heuristic, but it might be valuable to rank projects on the scale of the algorithmic complexity involved in each (which in turn probably depends on how close the project comes to exploring the unknown, as the DARPA race efforts do), and then by how compelling they are and see if there is a correlation.

In my experience, while enterprise projects have their own set of problems to be solved, they are rarely algorithmic in nature (vs economic or scheduling issues for e.g).

Another supporting factor might be to see if interviews for the interesting dev jobs focus heavily on testing algorithmic knowledge. At first glance this seems to hold as well. If you are interviewing for Google Research, you probably need to be great at algorithms. If you are interviewing for Wipro, not so much - a knowledge of the contemporary buzzwords (agile, xp, tdd, ruby dsl blah) should get you through.

Watching the DARPA Car Race

It is almost 3 in the morning here in Bangalore as I type this. I've been watching the webcast of the DARPA Car Race. This years challenge was for the autonomous ( == no human control) robot cars to accomplish three sub-missions in an urban environment, following all traffic laws (obeying traffic signs, recognizing and conceding right of way etc). This task is a magnitude tougher than finishing a 130 mile autonomous drive in the desert, with zero human control, which was the goal of the DARPA 2005 car race.

Stanford's entry, Junior, just pulled in to the finishing area. While it isn't yet clear who won (because this is not a first past the post race), I'd put my money on Junior. (His (its? :-) ) predecessor Stanley won the 2005 Grand Challenge. This is history being made before our eyes.

The sight of the cars driving themselves, with the steering visible through the windows turning with no human hand at the controls is .. exhilarating... exalting... too cool for words. This is **REAL** programming in action. The people who made this happen are just incredible. Congratulations to all!

How lucky does one have to be to live in these times?

Now to drag myself off to bed (and dream of autonomous vehicles).

Is that office work?

So, it is late at night and two of my friends are talking to each other. One is a programmer, a very good one, and the other is a manager, I have no idea how good a manager he is.

The programmer is multi tasking, carrying on a conversation in one mental thread and working on some code in another. As the conversation ebbs and flows, his attention also focuses more or less on the code as appropriate, sometimes muttering to himself, sometimes whooping out loud, sometimes ignoring the code and focussing on the conversation.

The manager gets more and more curious as to what exactly the programmer is doing and finally blurts out "Is that office work ? something you have to deliver tomorrow?" . The programmer looks confused and says " No, no it is just something that I am fiddling with". Now the manager looks confused.

And in that moment, *I* understood something. Many people think of programming as work. It sounds very normal and rational when you write it out like that, but most good programmers I know don't think of programming as something you "do at work". Programming is a means of expressing oneself, which also *happens to be* something people pay for. And this is lost on some people who can't imagine "working on" something outside the office..

Just to be clear this has nothing to do with managers vs developers. I can imagine a manager poring over company balance sheets or whatever to find a good investment opportunity and some random programmer who doesn't write a line of code outside office hours asking the same question and getting the same answer.

The difference is something more fundamental. People either have something they work on in their free time or they don't. Now that I think of it, a staggering percentage of the people I know would just switch on the TV, or pick up a book, or idly surf the web or pick up the phone and call someone when they have a couple of hours of free time. The person who picks up a guitar, or writes a program, or analyzes a balance sheet or writes an essay is much rarer than the first variety. Many people don't work *on* anything outside their office hours (and then wonder why "life is boring".

I am sure many people say "yeah so what?" but as someone who has many irons in the fire and is always "working on" something or the other, it was a big revelation to me that most people **don't** have more things to do in life than time to do them in.

One lives. One learns.

As usual, once I finish writing a blog entry I find someone expressing the idea better than I can.

Excerpts from Dan Russell's essay on marathon running (you should read the whole thing).

I don't know how many people have told me that I'm crazed. But the reaction I get when I tell someone I'm running in the Boston marathon is either a "that's wonderful -- how exciting!" or a "you must be out of your mind..."

Usually, it's the latter. And if the person I'm talking with is not a runner or bicyclist or someone who enjoys the pleasures of a workout, then the conversation usually turns toward the health hazards of rigorous exercise. I've been told to watch for vitamin depletion, unexpected calcifications, loss of sensation in the private bits, shin splints, and even unexplained loss of sleep. It's touching, really, especially the ones that are worried about vague causal factors like "getting the body too riled up..."

....So it's hard to explain sometimes why you'd even want to run so far. ..Don't mistake me, running the marathon will be tremendously exciting; there's nothing quite like the sensation of running into the chutes at the finish line, all of the cheers, hearing your name and time called out over the PA -- it's a little like being born and first love and finally getting permission to stop all at once. It's relief, desire and fulfillment borne over the past several months all condensed sweetly into that final few minutes of running.

But as someone one said about practicing the viola, "the epiphanies come in practice." Don't mistake me, running the marathon will be tremendously exciting; ......So I've been running six days a week for the past year just to make it to Boston. Or have I? Seems to me that running is about, well, not to put too fine a Zenish point on it, running. Like anything else, once you've put enough hours of practice into a skill, the doing of the skill transcends the normal, pedestrian everyday experience. Once you get past the first thousand hours or so, the act of running becomes larger than the little aches and pains of physical motion. When I run, I'm moving over terrain; I glide through topography; my body becomes the vehicle that transports me through space. This is an odd, almost out-of-body feeling that starts to happen when you run long miles for training. More than a few times I've run and had the feeling that it would be fun to explore down this road; so I turn, and run down this road. Natural enough, but when you're running 18 or 20 miles, these side roads and random ventures can be two or three miles long each. That kind of distance really begins to change the nature of the experience. You cross towns on a whim and run over sizeable chunks of real estate just to see what's on the other side. On more than one occasion I've run north around a mountain or decided to shift to another watershed during a run. It's the change in scale that's so quietly dramatic. You stop worrying about distance and begin thinking about time -- "if I go this way, will it take another hour?"...

Exactly! And that is what my programmer friend was doing. Exploring. What happens if I try this? And then that? The pleasure from the act of exploring is something that my manager friend, who has nothing comparable in his experience to validate the process against, doesn't quite understand. (Note: This doesn't make him a bad person). So he falls back to "Is that office work"? or in other words, "where are you running to? Can't you get a taxi ?"