And Miles To Go - Part One

This post could be titled "How Paul Graham's Writing Makes me Weep With Frustration With Its Elegance And Conciseness" but that is probably too long a title.

The point is still valid though. In his latest article, I find he has articulated in simple, crystal clear sentences, ideas I have been wrestling with in wretched futility for months.

Here is a simple example. When I went back to Thoughtworks for a visit recently, someone asked me "You have been doing some interesting ( == non j2ee, non enterprise) work for a year now So what did you learn?" And I said (among other things) "Math is important.". But then the thoughtworker asked "But isn't Math just another domain for the "analyst" to master? Any domain expert who deals with a domain will know what exactly he wants calculated and how. Why should we developers delve into the underlying theory? In other words isn't Math just a domain? "

I instantly saw that there were deep chasms dividing the world of enterprise software from the kind of code I write these days, but I couldn't find the words to bridge those chasms. I mumbled weakly that "Understanding math somehow makes your thinking better". Neither the questioner nor I were satisfied.

Now see how elegantly Paul expresses it(emphases mine). After saying that letting your mind wander is often a good source of ideas, he begins to wonder why that is so.

"....What happens when your mind wanders? It may be like doodling. Most people have characteristic ways of doodling.....Perhaps letting your mind wander is like doodling with ideas. You have certain mental gestures you've learned in your work, and when you're not paying attention, you keep making these same gestures, but somewhat randomly.In effect, you call the same functions on random arguments. That's what a metaphor is: a function applied to an argument of the wrong type....The habits of mind you invoke on some field don't have to be derived from working in that field. In fact, it's often better if they're not ... Are some kinds of work better sources of habits of mind than others? I suspect harder fields may be better sources, because to attack hard problems you need powerful solvents. I find math is a good source of metaphors good enough that it's worth studying just for that."

And there is what I should have said when people asked what good Math was. Math gives you more "primitives" to operate with and more ways of combining them. So do programming languages like Erlang and Lisp (which I was forced to turn to when the problems I was working on got too hard to handle with Java or Ruby). Math (and some programming langauges) are indeed "powerful solvents" that give you more ways of perceiving a problem, more choices on how to deal with those problems and overall a much richer field of possibilities.

Duh! I had to wait for Paul Graham to write that up and when I read the above quoted sentences I was shouting at myself "Exactly!! That is what I wanted to say"

Here is an ultra simple example of what I experienced. I needed to build a fairly complicated Neural Network with avery abstruse Training Scheme. For a long time I used the habits of thought I had picked up in years of "Object Oriented Thinking" and thought of a neural network as an "object having properties X and Y , with behaviours A and B , consisting of n Layer objects, each having properties blah and behaviour foo.. A Training Method is a Strategy Object that decides ..." and so on.

The people who I was talking to were primarily mathematicians and scientists and soon a communications gap yawned and we were all getting frustrated. Now I could have insisted they grok Objects or I could just learn the underlying math. I chose the latter and gritted my teeth, clenched my muscles and jumped into the Linear Algebra and Calculus needed to understand neural networks and .... found it quite interesting. Nothing like the nonsense poured down our throats in college. I was now using the best books, written by the most talented folks in te field, to learn, and I was applying the knowledge I gained to solve a very tough problem. And soon I saw that understanding Vector Spaces and Differential Equations allowed me to see neural networks in a new way. As I told a friend sometime later, a Neural Network is a set of Equations and every training scheme solves a problem in Topology. There is no way I could have come to that understanding without harnessing the underlying math. No amount of "Domain Modelling" or "Extreme Programming" or whatever would have given me the understanding needed to tackle the problem.

And that is why today when some people look at the work I do and say "dude , I don't want to study all this math stuff. Too dry for me. Lisp is too hard as well, leave alone all this concurrency stuff. But I want to learn AI too.I will just use TDD and Java", I just smile to myself. Unlike most "enterprise" work, in production strength AI (and most other "tough" fields I would imagine), not understanding the underlying domain deeply enough to acquire an intuition for what will work and what will not, just wastes time. Without the new perceptions opened up by grokking the "powerful solvents"(and it will take about a year of work to get to that point), one is just as blind as ever.

And it needed Paul Graham to make it all clear in about 5 sentences.

I have also come to see that the Analyst/Developer split is very dangerous, even in 'Enterprise' software and leads to substandard software. But that is a topic for another post.