- Be sure you want to be a researcher. And why. The answer to "why" is intensely personal. You have to find your own answers. I want to be a researcher because I want my working life to be intellectually rich. Enterprise programming is too boring. Back when I was building the umpteenth banking/retailing/insurance/leasing app, I found I could coast too easily and work with maybe 10% of my ability and still keep up with some of the best programmers in that space (Most of them were working at 10% capacity too. This led to endless "losers club" sessions. Some of them sold out and became project managers. And got more frustrated. Others just got frustrated). The nature of the work (at least the variety that washes up on the shores of the Great Outsourcing Destination), is an endless repetition of a few mundane tasks. The complexities of enterprise software are in my opinion arbitrary. Too many snake oil methodologies, not enough deep thinking. Lucrative, yes. Stimulating, no. In a nutshell, I was bored stiff. I actually like programming. I enjoy reading technical books. I read technical (and these days mathematical) articles or books on a plane jouney or just before sleeping. I chose programming as a career because of the intellectual challenge inherent in it - that a good chunk of it ended up writing horribly boring enterprise software is a tragedy we won't delve into. At least in the future, I would like my working life to consist of wrestling with significant problems, not assembling api's like a jigsaw puzzle to create systems that I don't care about. (hey if you like enterprise software , good for you. I don't. Please don't fill my mailbox with whiney comments about how Banking apps are really uber cool). As Douglas Comer says in his article (after a lot of warnings about not getting into a pHd Program for a the wrong reasons), "If you have the capability and interest, a research career can bring rewards unequaled in any other profession. You will meet and work with some of the brightest people on the planet. You will reach for ideas beyond your grasp, and in so doing extend your intellectual capabilities. You will solve problems that have not been solved before. You will explore concepts that have not been explored. You will uncover principles that change the way people use computers." I want all of that. I'll probably go through a PhD sometime in the future but think a lot of what Dr Comer outlines is well within the grasp of a dedicated individual with good programming skills and access to the internet (well, perhaps not the "work with the brightest people on the planet" part, unless you consider collaboration over the internet "working with"). I can live with that.
- Do the groundwork. This is avery hard step for someone who is working in the industry vs in academia. There is almost always a substrate of very technical (and often highly mathematical) material you have to digest before you can comprehend (leave alone work in) research areas. My interests are (broadly) in Artificial Intelligence (more specifically Machine Learning) and Compilers and Programming Language Theory. The former calls for a good grounding in Calculus, Linear Algebra, Information Theory and Algorithmics. The latter needs a good grasp of things like the (typed) lambda calculus and Automata Theory. Theoretically if you've gone through an engineering or science Bachelors degree (BTech or Bsc, particularly in Comp Sci), you are supposed to know a good chunk of that. Anyone who has done a Bachelors in India (excluding the IITs and IIScs) knows that most people study this stuff 2 days before the exams (which are too often memory oriented) and forget everything two hours after the exams. For those who doubt this, I suggest the following experiment. First get a moderately tough problem in say Calculus or Probability. Find someone in your office who has graduated from an Indian university in Engineering or Science with a good GPA . Ask them to solve the problem. Enjoy the reaction. Contemplate the fact that an "engineer" (or "scientist") is ignorant about Calculus. Indian technical education very often consists of memorizing formulae without getting an intuition for the concepts behind the formulae. Un(?)fortunately ,to work in research one has to understand the underlying math or science. Working in Enterprise Software makes this situation worse. In spite of the Industry's endless thirst for Engineering and Science graduates the cold hard fact is that you very seldom use any science or engineering when you work on the typical Insurance or Banking web app. You end up endlessly churning APIs (ORMs, Rails, hibernate blah) and methodology artifacts (e.g uml diagrams or storycards) and (more often than not) fitting chunks of horrendous code together with spittle and prayer. Work on such systems long enough and you forget what little you retained from those years of education. It took me amost two and a half years (after leaving that "enterprise" job) to acquire the background skillset. It may take you less time. But it will take some time. And if this time is spent in adition to your 9 to 5 job, be aware that this part of the work is a very very lonely path. You've achieved the necessary skill when you can read a paper from a contemporary journal or conference in your field, like you'd read a news paper article or a requirements document. Another milestone is when you lose your fear for mathematical notation and proofs :-)
- Find interesting problems to work on
This is the toughest step for a non academic. You literally have no starting point, no end point and no route connecting the two . As my friend Tejaswi (an MTech from IIT btw)confirmed in a private conversation, most Masters degree theses are suggested by the "guide" professors who oversee your work. Even for PhD students, who are expected to do a lot of unsupervised work and thinking, the input of these guides is a vital component of problem selection. Rightly so. You can save humungous amounts of time and effort if someone knowledgable points you in the right direction.
But what do you do if you are not an academic but still want to do research? You have no "guides". Talented researchers have tonnes of work already and are too busy to respond to whiney emails from wannabe researchers, so how is a poor ex-enterprise coder to find significant problems?
Here are some things that worked for me.
- (Once you've got a grip on the fundamentals) Work through the core textbooks in your chosen field with a particular focus on working through (and coding up) the end of chapter exercises. Every field has a set of "core books" written by the some of the best researchers in that field. Find a suitable undergrad text and get to work. (For AI, Artifical Intelligence , a Modern Approach is an excellent choice. For compiler theory something like TAPL or EOPL would be a good starting point).If you are diligent, you'll eventually go beyond these books and start working through graduate texts. Once you've got a couple of grad level texts under your belt, you are ready to find problems.
- Read contemporary papers. This is the KEY step to finding problems. Most (good) papers have a "work to be done" section which is a rich source of problems to attack. Most papers and conference proceedings are onlinethese days.All hail the Internet!
- Discuss your research with folks who are doing similair things. In a city like Bangalore, this is fairly tough, because most of the software "hackers" are doing cheap labor, not research, but maybe your uncle is a professor in one of the IITs or your cousin works in the ISRO (indian Space Research Organization, for those who don't know). Talk to these people. In my experience, they are glad to work with someone who is genuinely interested. If all else fails (shamelessly) tap your friends who are working on their Masters or Phd or post-doc degrees.
- Start a research notebook to record your thoughts and activities. I have found this practice VERY useful. As would a blog ( a private one is better imo) to record your daily progress(or lack of it). The habit of taking small steps daily (or biweekly or whatever) is very important.
- The essence of "finding a problem" is to ask a question that (a) hasn't been asked before and/or (b) has no answer yet. research is all about finding answers to these questions. Put like that, it doesn't sound *so* hard does it? (yeah well it DOES sound hard .. :-P )
- Be self directed. This is a KEY personality trait. In mosts corporate settings you always have someone telling you what to do, (by) when to do it and sometimes how to do it. This is similair to a Masters course. A PhD student has to be much more self directed , but he still has deadlines and guides and colleagues. Someone working alone, in his spare time has none of this. Unless you have iron discipline and can work steadily over long periods and tolerate boatloads of ambiguity, you might as well not start. Life is too short.
- Once you find a significant problem you need to solve it. And then publish the results. I don't have any advice to give on these steps because I am still working on my first research problem. However I am making steady progress. Some of the best people in the field have expressed interest in seeing how the results turn out. My research notebook is filling up nicely. I write code every day. So hopefully in about 3-6 months I'll have something to report. Watch this space.
Ravi Mohan's Blog
Thursday, April 19, 2007
A Rogue Researcher's Route Map
One of my resolutions for this year is to make the transition from coder to researcher. While the journey is still very much ongoing and the end point is very far away, I believe I have a sufficient number of bruises from falling into pits (and the callusses from digging myself out of them) to make a few notes that may help illuminate the first few miles for anyone attempting a similair journey. One of the problems of "doing research" is that it is mostly undertaken in the context of somone undergoing a PhD or other academic program. There are really no guides for the non-academic researcher who wants to do research for the fun of it or as an adjunct to his day to day work. Most of the online 'guides for the perplexed' assume you have been admitted into a stellar PhD program. This advice is useless to someone who is not in academia. What follows comes from my experience in trying to scale the walls of research. I believe that a formal academic program is the best way to acquire research skills. If you can manage an MIT PhD, go for it. But I also believe one can contribute to the sum of human knowledge without that background. I believe I can more or less do anything I want to. I am kinda stubborn that way. So take all my advice with a mountain of salt. What follows is written for the non academic researcher, working part time. Those in a full time academic program have much better resources and don't need my ramblings. With all those caveats in mind, here goes,