Ravi Mohan's Blog

Tuesday, February 27, 2007

If Toyota outsourced its assembly line

to "consultants" who charged x$/hour/person_deployed, would we have a Toyota Production System today?

Think about it. Imagine someone like Wipro or Infosys providing the manpower (cheap offshore manpower if you will) for Toyota's assembly line. What would the probability be of Toyota coming up with a world beating manufacturing/design approach?

Many "Lean Software" advocates draw very tenuous relationships between a very unique achievement in the manufacturing world, try to spin their own practices (ahem - did I hear someone say " agile" ?) and say (in effect) "By the power of Toyota, Taichi Ohno and Kanban, you should do [consultant's favorite methodology practice item]".

I recently read a book on implementing Lean Software, by one of the "luminaries" of the methodology sales world , in which every chapter had the following pattern.

  1. A one paragraph example of how Cool Company [Toyota/Boeing/Google] etc did something cool [developed the 747, say].
  2. One aspect of this successful effort (failed efforts by the same companies, often using substantially similair methodologies, are ignored) - say, cross functional teams- vaguely resonates with "agile" or "lean software" principles.
  3. One or more teeny weeny "experience reports" from the author which sound like stories from the Brothers Grimm, full of archetypes and soundbite morals-to-take-home, of the calibre of "Don't talk to wolves on the way to Gramma's house"
  4. a "summary" that states point blank that software development could do much better if the unenlightened masses would adopt these ideas

Most of these "agile/lean" consultants haven't written any significant software, nor managed any world changing software efforts. In direct contrast, the TPS "senseis" are often people who have worked for decades on extremely successful, innovative, *world class* products that they can point to ("see that Prius/Corolla/other prominent model, I was its Chief Designer").

Software seems to be the only area in which people with no history of creating significant software can go around teaching people who do write software how to do it.

If you try to pin these folks down on any specific practice or project they fall back on ever more woolly explanations. The most creative one I heard was that "Lean Software is about the principles of effective development not about practices (which admittedly are quite different for sw development vs manufacturing )".

That's ..errr..comforting? No one can question motherhood and apple pie statements, especially when they are explicitly divorced from concrete achievements or practices.

To get intelligent people to listen, a lean (or anyother buzzword) salesman should (a) deliver outstanding, world beating software, (b) do so consistently for a long period of time(how does 20 years sound?) and then talk to others about how he or she pulled off all these astounding projects. Most people who are outstandingly good at development or management often have better things to do than be "methodology consultants". Can you imagine Linus Torvalds becoming a "How to create succesful open source projects " consultant? Or Sergei Brin "consulting" on "How to create a succesful product company"?

People who can, do. It is only after "doing" for a very long time and consistently succeeding that they can pass on that knowledge if they choose to do so.

Having said all that, if anyone wants to make a lot of money (in Bangalore/India), the best thing to do these days is to get a "Certified Scrum Master" certification. A *lot* of dumb companies here are looking for such "masters" and are willing to pay good money. If you need the money, go for it I say. Be sure to get in (and out) before the fad fizzles.

All you have to do is pay up, sit through the seminar(s) and hey presto, you are now qualified to spread the Word. I get enquiries *every week* on whether I can reccomend someone for (or do some) "Agile Coaching". A "Certified Scrum Master" can make a killing in Bangalore today. As I said, you need to be fast. Lots of people have sensed the money and are lining up for the ScrumMaster certification.

But if you are on the other end of that pile of money, beware of consultants bearing buzzwords.

Is it possible to learn from Toyota's processes and apply them in software? Perhaps. But such an effort would be less about listening to conmen and more about doing lots of original thinking, reading (the original sources .. not "lean software" type blurbs) and a tonne of "doing" and reflection in *your* context.

And oh, don't believe anyone who charges you by the person hour or is trying to get you to outsource software development is sincere about "lean software". They aren't.


Ryan Cooper said...

This is a pretty bold post. While I do agree that paying out lots of $$$/hour does not free an employer from the responsibility of hiring someone who knows what they're talking about, I find it insulting that you assert that anyone who gets paid by the hour is automatically insincere.

I understand the frustration of watching companies trying to pay for a magic bullet. And I also understand the frustration at the feeding frenzy of incompetent consultants that arises from these situations. But concluding that there are no genuine experts who can help an organization become more effective is not valid.

And lastly, you seem to imply that it's impossible there are deeper parallels between different disciplines that go beyond the technical practices involved. This baffles me.

Ravi said...

"I find it insulting that you assert that anyone who gets paid by the hour is automatically insincere."

hmmm that was not the intent. I didn't say (blanket) "insincere" . I said "insincere about (true) lean software".

There is nothing morally wrong in charging by the hour or on any other terms the market will bear.
There *is* an element of conmanship about selling things you have no (real)c expertise in.

What I intended to convey was that anyone who charges by the hour is extremely unlikely to come up with the deep insight necessary to radically reduce teh expenses/cost of that process. Even in the best $/hour based consulting firms there is a tendencey to bill a minimum of 40 hours (or whatever per week) irrespective of the actual work done and add extraneosu people or hours as much as you can get away with.This is what is "anti lean".

The $/hour system sets up an economic incentive which makes it very hard (though not theoretically impossible) for the payee to focus on reducing the client's "value stream" cost.

"But concluding that there are no genuine experts who can help an organization become more effective is not valid."

How do you conclude that? I never said that such experts don't exist. I only said they need to have a long history of actually doing the things they claim they can do.

If you have any counter examples I'll be glad to listen.

"you seem to imply that it's impossible there are deeper parallels between different disciplines that go beyond the technical practices involved. This baffles me."

I explicitly state towards the end of the post that parallels and cross learning ARE possible if one applies thought and effort towards that end. I am suspicious of people whose onely "expertise" is in methodology selling (think "agile coach" of the conman variety) and don't have world class projects to their credit. Or are you saying that Scrum (or whatever) "certification" is valid proof of expertise?

Anonymous said...

""People who can, do. It is only after "doing" for a very long time and consistently succeeding that they can pass on that knowledge if they choose to do so.""

Talking about Martin Fowler ?

Ravi said...


"Talking about Martin Fowler ?"

Well not really. I was thinking of Taichi Ohno and Shegeo Shingo (pioneers in Lean Manufacturing)when I wrote that.

But yes, in the world of programming, Martin would fit the profile, as would folks like Paul Graham, Peter Norvig, Douglas Schmidt etc.

None of them (including Martin) make a living as "methodology sellers" or "agile coaches" or whatever, though I am sure each has strong opinions (probably very different from mine) about how to write good code and run succesful projects.

Martin, since he focusses on enterprise software is probably the clearest anti-hype example as contrasted to the nonsense that permeates that corner of the software world.

Anonymous said...

Oops !! Seems like I got the wrong message across.

I think Martin Fowler is full of crap...

Cant find one line of code that he's ever written. All hes done is talk / preach of what others have done. Ive attended a few of his talks. I agree, he talks great ! But over time you'd begin to notice something very strange. Hes always talking about something someone else has done...

"People who can do...". He's definitely not in this league. You're still in that reality distorted world that some people (especially Martin) pull you into. Question it, Search for it (the code) and you'll see for yourself !

Ravi said...

"I think Martin Fowler is full of crap..."

hmmm.. you have a right to your opinions as do I :-). Free world isn't it?

"Question it, Search for it (the code) and you'll see for yourself !"

But I *have* seen his code (and examples of design) when I was in TW (I actually learned quite a bit from it. I blogged about this code sometime ago.).

So now who is in the "reality distortion field"- you or me? :-) And why be "anonymous"? Step into the sunlight :-)

My perception of Martin is that he is primarily a chronicler, distiller, clarifier and communicator and not a methodology salesman or project manager for that matter.

Ever hear of Martin working as an "agile coach" anywhere? He has strong opinions on agile/ xp , some of which I don't agree with but thats as far as it goes.

Martin has a lot of good things to teach as regards **enterprise programming**.

Now if you are not in that world (I am not, these days), you don't have much to learn from him. (What about refactoring though? Sure it was parcticed here and there before he nailed it down in prose but do you think the book is a net gain or loss for programmers? I am genuinely curious)

The books on Refactoring and Enterprise Architecture (for e.g.)are classics and should be on every programmers bookshelf (imho).

Also he never claims to know things he doesn't have first hand experience of. Just attend one of his talks and wait for the local bozo to ask him some nonsense question. The brutal "I have no idea" answer is quite ummm entertaining.

So while acknowledging your right to form your own opinions, I must respectfully disagree .

The fact is that Martin has a very long (and succesful) history in enterprise software (thus fulfilling the "do consistently for a long time" criterion) knows what he knows and what he doesn't know and he is definietly not one of the methodology vendors. On what he does know , he is able to bring an erudition and clarity to bear which is very rarely matched in technical writing.

I try to learn from Martin what he is good at - enterprise architecture and communication. DHH for e.g. openly acknowledges his debt to Martin's AR pattern at the core of Rails. Quite an endorsement of competence if you ask me.

Of course, all the above is totally MY opinion, which while different from yours is probably worth considering.

It seems as if we both agree on the need for long and sustained excellence before you start teaching. That was the key point of the blog entry.

The disagreement is whether a particular individual fits that pattern in your perception. Kinda peripheral to the main argument. If you have a problem with Mr Fowler's fame and fortune, you'll have to deal with it on your own I'm afraid.
This isn't quite the place for that.

Any thoughts regarding the main ideas of the post (vs the deservedness or not Of Martin Fowler's fame)?

Anonymous said...

""Martin has a lot of good things to teach as regards **enterprise programming**.""

Thats not programming, thats plumbing. A total bummer can do that once he understand loops, basic containers and a lil bit of databases.

Look at the others in your previous list - Peter Norvig, Paul Graham... These are real programmers. Reading their works takes you into a different world. You feel the power of the subject, be it AI, Interpreters or Macros... You get into this wierd... (I dont know how to describe it, but hope you get the point..)

Now compare that to reading "Refactoring", Do you get into that same wierd... (whatever you call it..)

I have nothing against Martin myself, but I just don't think he belongs to the league that these guys do.

"Thompson and Ritchie", Geniuses. Forget the code, just the design of Unix was a tech marvel. Now, would you put Linus in the same league ?

As you very correctly pointed out in this post, The people who do, rarely have the time to talk or teach about it. Very rare exceptions to this basic fact. But this in no way implies that those who fill that void, take their places.

""And why be "anonymous"? Step into the sunlight :-)""

I'll get myself an account over here soon. For now, Im just a nobody wandering the web.... :)

Ryan Cooper said...

Ah, it's good to see you and I are in agreement more than I thought. :)

I would make the distinction between 'consulting firms' (a.k.a. body shops) with real consultants. I find they are completely different animals. Real consultants rely heavily on references and word-of-mouth, so generally there actually IS a financial incentive for them to work in their clients' best interests.

I agree that to see parallels we need to reflect about our context... but I always get the impression from agile writers that they are trying to convince me to do that. And frankly, that is what a good "agile coach" (I prefer the term "agile facilitator" these days) does as well. It's really too bad so many snake-oil salesmen have co-opted the phrase, but there's no way to avoid that without embracing heavyweight certification, which obviously creates a host of new problems.

I suppose I get defensive when people neglect to acknowledge that the snake-oil salesmen are not the entirety of the agile movement. Hmm, actually that isn't right. I get defensive when the acknowledgment is overshadowed by the railing against the snake-oil salesmen. I find an agile approach extremely effective, and I think these kinds of rants often give the wrong impression to people still using a waterfallish process (or no process at all).

For the record, I assume your post was referring to the Poppendieck's books on Lean. As far as I can gather, both the Poppendiecks DO have backgrounds in software development, so I see no reason to discount them as non-experts. Sure, that is not the entirety of their experience, and they use examples from other domains in their books. Because they have experience in both software development and other new product development, they are able to draw parallels. I appreciate the insight, but still hold it as my responsibility to determine which insights are useful to me in my context.


Ravi said...

HI anonymous,
I like the idea of "different kind of weird" feelings on reading different books. :-)

All right, you have a right not to like Mr Fowler. :-)

I think most of the difference that you feel between MF and the others are that he is the only one i quoted who works in "enterprise" software.
I lke your use of "plumbing" coz MArtin uses the word to describe *NON* enterprise software.

NO matter what you do in enterprise software it is very hard to have the same kind of "cool" that comes from being a pioneer in languages, AI or systems. I do think the practice of enterprise software is a "lesser" activity *when compared to* systems programming, Robotics etc. Thus far I agree. But if you subtract the coolness factor (or feeling of weirdness factor) of the systems vs enterprise effect what then?

Before we drop this (tangential to the main line of discussion imo) topic,
If you are focussing on"newness" There was plenty of work done in AI before Peter NOrvig wrote about it and plenty of work done in Lisp (even in macros) before PaulGraham wrote his seminal book. Not an argument, just somethin for you to think about.

As to whether I consider Linus to be in the same league as Richie et al, sure I do. from my pov, any minor differences in the "heights" reached is very orthogonal to the point I was making.

Solid implementations osmetime trump novelty.

Anyhow, thank you for your thoughts. You are the most articulate "anonymous" to comment on this blog!

Hat's tip and thank you!.

Anonymous said...

Salute my wise friend !

I enjoyed my stay here too.. Thank you for sharing your thoughts in this nice little world of yours...

Its time for me to move on, as I have a lot of ground to cover and so little time.. so little time...

Until we meet again, So long my friend.. so long...

Click, Click...

jfischer said...

"Software seems to be the only area in which people with no history of creating significant software can go around teaching people who do write software how to do it."

So true. That's a classic.

Jon Miller said...

As a Lean / kaizen / Toyota Production System instructor whose customers pay me in some cases based on hours or days, I read your post with much interest.

My coding experience is limited to a few months of Basic on a TRS-80, about 25 years ago. And enough HTML to format text in a blog.

I disagree that consultants need to know how to do the work they are trying to help improve. In fact, my lack of deep experience, and the so-called "fixed ideas" one develops through expertise, is a strength.

I have logged many hours of improvement teaching and doing in factories, offices, hospitals, etc. over the years but I can't claim to have spent a significant time in any industry or at any work, other than perhaps translation and teaching (a.k.a. consulting).

We are teaching the skills to observe processes, identify waste, and organize your work in ways that cut out this waste.

The customer defines value of the coded product, not consultants and not the coders. Lean posits that all else is waste.

So there are a few basic things like this that need to be explained and agreed upfront, or through practice. Once that's done, anyone can do kaizen to any process, with respect for people as a prerequisite.

"Software seems to be the only area in which people with no history of creating significant software can go around teaching people who do write software how to do it."

We (Lean instructors) try to help you write software better - not how to write it, or how to write better code. Process, not product.

Ravi said...

Thank You for the detailed comment.

You say "We are teaching the skills to observe processes, identify waste, and organize your work in ways that cut out this waste."

This is the core point of discussion. How did you acquire these skills that you "teach"?

I for one don't believe such an abstract ability to "observe processes " etc can be acquired without having hands on and indepth knowledge of a concrete process. If I am wrong, I am willing to be convinced.

Taichi Ohno and other Toyota sensei seem to have been automobile/mechanical/maunfacturing engineers with plenty of hands on experience. If you can point me at someone from Toyota who was simultaneously a pioneer of lean AND lacked deep engineering skill, that would certainly be worth discussing.

In the absence of hands on, *deep* experience in any creation activity, (manufacturing, software, film making, whatever) where does the ability to observe and analyze a process and apply the principles of "lean" coem from? from reading books? I am not trying to insult you, I am genuinely curious.

Yes, I can agree with the notion that the best people in any field may not know exactly how they do incredible things. Tiger Woods may not be the best teacher of golf, but anyone who teaches golf should be a very very good golf player if not necessarily championship material.

What I don't condone is people who've never played (good) golf in their lives and don't even know how to hold a club telling others how to do so.

Besides, Ohno and other pioneers of lean seem to constantly stress "hands on" experience and *independent thinking* (vs depending on people who've read a few books and have no experience themselves).

And if the lean consultant is "interpreting" Ohno (and other parctioners) to the unwashed masses, what prevents the masses from reading what Ohno (and others) have to say and think for themselves (vs depending on a consultant who admittedly has no industry experience)?

Analogous to reading the Bible for yourself and decide how its teachings apply to your marriage(say) vs having a celibate, unmarried priest tell you how to be married "biblically".

This "I will teach you things I have no experience in actually doing" idea is (imo), dangerous and facile.

I've encountered it before when i was chatting with a management consultant (who is a good friend). She had great difficulty in pinnng down the value proposition of a (Mc Kinsey style) management consultant. That doesn't mean that there is no such value. It is just that I've never heard a good explanation.

Steven Spielberg talking of how to direct films is one things. I doing the same is laughable, irrespective of what books I've read about film making.

I am now going through your website. Some good writing there!


Ravi said...

@ the anonymous person who sent me the TOI article,

do you think you could express your view point in a sentence or two? The article you sent me is (quite) long and could be making a number of different points. BLog spot doesn't give me the ability to edit a comment in any detail I can only accept it or reject it.


Vladimir Levin said...

I think Kent Beck and Ward Cunnigham qualify, don't they? As far as I know they promote agile practices. Perhaps Robert Martin qualifies too. Sadly people jump on the agile bandwagon without really understanding what agile is about. To a great extent agile software development is just about common sense trumping preconceived notions of what software development should be like. I think you're just sick of the silicon snake oil salesmen being up to their old tricks of jumping on any bandwagon that lets them make a buck, but that's just the way it is - it has nothing to do with any particulat practice, method, or technology. It's just life, as Douglas Adams might say. I don't think companies like google have any special tricks to develop software better. They just try to hire the best and brightest and treat them well, which is always a sensible strategy.

Ravi said...

Maybe you are conflating "agile" and "lean"? The post is about those who try to sell the success of Toyota in manufacturing as some kind of half assed exemplar for software projects, **not** about "agile" software in general. I've written about "agile" elsewhere This post i specifically about the "lean" con men who try to jump on the agile wagon when it suits them (the first book of a series of two on lean sofwtare does this) and jump off it when they see it facing a backlash (the second book).

The "agile " folks at least have some basic credentials in having written some significant (if fairly small) pieces of software (JUnit and say the Wiki or FIT).

But since you mention Kent and Ward let us take a closer look at their achievements and see what we really have to learn from them.

"I think Kent Beck and Ward Cunnigham qualify, don't they?"

Would they? let's think about this a bit.

Kent's "significant software" is JUnit (ignoring Erich gamma for this discussion). He certainly has the credentials to tell us how to write small frameworks like JUnit and about the importance of unit testing your sofware and applying design patterns etc.

Should writing Junit make Kent the authority on how to run complex projects with large teams?

The C3 project,worked on by some or all of the agile gurus, is at best a very marginal success. The circumstances sorrounding its closure is very murky. What is undeniable is that from the *clients* perspective it was a flat failure and the joke in Chrysler is that agile was used "once and only once". Does that make agile worthless? Not at all. I have seen teams succeed with that approach. And also fail. As you rightly point out success in a software project depends primarily on hiring good people, not on the process they use.

Ward of course is known for his enormous creativity (wiki etc). Does that translate into writing complex/cutting edge/deep code? Take a look at the innards of FIT sometime and tell me that that is the best architecture possible for that functionality. Pay particular attention to the "node navigation" part.

Imo, Ward's (undisputed) genius is in the fertility of his mind and the creative solutions he comes up with and not in the methodology space. If he were to talk of how he comes up with these ideas, I'd gladly listen. If he starts "advocating agile" for *my projects* I'll certainly switch off.

But anyway, Ward or Kent "advocating agile" would be almost orthogonal to my points about (a) "lean software" being a half assed attempt to leverage off the toyota/TPS brand succes stories without substantial achievement in the software field to back it up
(b) the genral phenomenon of people without significant experience *in software* telling people who do write software how to go aboout doing this. The only analogy I've seen in another field is business consultants fresh out of school "advising" businessmen how to run their businesses. In any other profession (say medicine, or law or science) if you or I (as representatives of people without any deep experience or achievements in those fields) were to tell them how to conduct an operation or structure a legal argument we'd be at best laughed out of the building.

Paul Graham has this nice point somewhere about how to judge a field by seeing how many of the top practitioners are teachers. In software *some* of the best of the best are teachers (in top universities) but they most certainly are NOT methodology salesmen. And that was my point.

Vladimir Levin said...

Ah I see. Yes, I missed the point about lean vs. agile. I've read the Scrum book and I thought it was good. I paid more attention to the first part and not so much to the second part where they try in detail to draw that parallel between lean manufacturing and software. I thought it was an interesting point to make though, because high ceremony/up front design-based methodologies seem to use manufacturing and civil engineering as analogies for software development. For me it was poignant to point out that manufacturing itself can also be done in a more empirical way. That said, I am not a big fan of such "consultants" you speak of either. I think their main talent is making the executives who pay them feel good about themselves.

On a bit of a tangential note. a good friend of mine worked for a number of years on the Solaris kernel team at Sun. I talked to him about agile (interestingly he was the one who introduced me to Kent Beck's white book years before but I didn't get it at the time). He seemed to think that agile methods might have been applicable to their work but I don't think anything was ever tried. While I have a great deal of respect for the people who do this kind of heavy lifting programming, I think there is a lot of inertia in such environments and just because they keep doing the same stuff and continue to be successful at it doesn't mean there isn't room for improvement.

Anonymous said...

"But I *have* seen his code (and examples of design) when I was in TW (I actually learned quite a bit from it. I blogged about this code sometime ago.)."

Pretty curious as to how you saw his code when you were in TW.....

Ravi said...

how does it matter how?
How is the "how" relevant?