Ravi Mohan's Blog

Tuesday, April 18, 2006

Everything Old is New Again - The Ruby DSL hype

Update: I've received a few mails asking how to go about creating dsl s "the right way" in Ruby. If you are in Bangalore, I suggest you attend the Ruby DSL talk at the Agile India 2006 by my friends Badri and Sudarshan (both Thoughtworkers). Such things are best learned by demonstration than reading about it and the Badri and Sudarshan are probably the best people in Bangalore to learn from.

If you are not in Bangalore find someone who knows and ask him to teach you. :-)

[Rant] These things have been true from the dawn of programming languages

  1. A language with a built in eval function allows you to transform code into a parse tree at runtime.
  2. To the degree that the language has a uniform/flexible syntax, this transition between code and data is painless
  3. All language transforms consist of two steps . First you convert a string into a data structure called a "syntax tree" . This is called parsing. Then various transformations can be done on this data structure to produce the effect you want.

    Thus the string "2 + 3" can be converted into a datastructure like "{ operator: "+", operand1:"2", operand2:"3"}".

    Then you can define various transforms that work on (or "visit" in oo terms) the components of the data structure and do whatever you want.

    e.g : (pseudocode) def print(anAst) puts anAst.operand1,anAst.operator,anAst.operand2 end

    def printReversePolish(anAst) puts anAst.operand1,anAst.operand2,anAst.operator end

    def evaluate(anAst) return anAst.operand1 + anAst.operand2 end

    anAst = parse("2 + 3")

    print(anAst) => 2 + 3

    printReversePolish(anAst) => 2 3 +

    evaluate(anAST) => 5

    and that is all there is (conceptually) to any langauge transform.

  4. Ruby "DSL" s allow people who haven't studied the fundamentals of computer science to think they are discovering something new and amazing. Yes, Ruby allows a somewhat seamless mixing of "pure" ruby and "dsl" code, but this is just as true for any langauge with a built in eval and clean syntax. (lisp, forth, even smalltalk). The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely from ruby syntax)

To understand ruby "dsl" s, ignore all the hype (and the forthcoming stream of books explaining how ruby's 'dsl ability' is the next silver bullet that will save enterprise programmers , give their arid lives meaning, and solve world hunger), understand how interpreters work (read "The Essentials Of programming Languages") and meditate on these concepts in ruby - class eval, object_eval, method_missing. if you understand the basics of how languages work, ruby's "dsl" is some (very) old wine in some sparkling new bottles.

  • The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face.

    Ok , biting your tongue and walking away quietly is almost as good.

    DSLs like any other programming technique, when used appropriately gives some beneficial effects. But it is no panacea and a casual perusal of the net will reveal a bunch of atrocious dsls . Beware the hype and snake oil. [end Rant]

  • 8 comments:

    Anonymous said...

    I agree with your main point but you've missed what makes people excited about Ruby DSLs. It's not that you can parse some text into an AST and then interpret the AST, it's that you can define a DSL *without* having to parse text into an AST.

    Any wierdy-beardy Smalltalk or Lisp programmer will tell you that this has been around for a long, long time. In fact, most experienced programmers will tell you that defining languages in which to express the problem and it's solution is the very essence of programming.

    Yes laugh in their faces. But point them at Abelson and Sussman's Structure and Interpretation of Computer Programs, Paul Graham's On Lisp or any number of books about Smalltalk or functional programming.

    Ravi said...

    you say ,

    "But point them at Abelson and Sussman's Structure and Interpretation of Computer Programs, Paul Graham's On Lisp or any number of books about Smalltalk or functional programming."

    which part of "read 'essentials of programming languages' don't you understand?


    you say
    "t's not that you can parse some text into an AST and then interpret the AST, it's that you can define a DSL *without* having to parse text into an AST."

    again which part of

    'The built in "eval" takes care of the parsing (as long as the syntax does not vary too widely ffrom ruby syntax)" don't you understand?'

    And of course you twist my words.
    I said

    "The next time someone tries to tell you how he created this uber cool "dsl" in ruby AND that this takes some fundamentally new technique, laugh in his face"

    note the three letters in bold? the "AND"? hopefully you learned what "and" meant in primary school?

    It is not the designing or use of dsls which warrant derision , but the claim that is a fundamentally new technique, which exposes ignorance of basic CS.

    This is exactly the kind of clueless reaction I was warning readers against. Must be nice that you are protected by your anonymity form your own stupidity.

    Read the damn post before posting vapid rebuttals.

    Regards,

    Anonymous said...

    I see that the usual anonymous idiots have already surfaced :-)

    You are right to excoriate anyone who claims that ruby's dsl is a fundamentally new programming tecnique. It is no such thing, at least not to people who have had a basic computer science education, let alone any experience with lisp or smalltalk.

    However, it may be "new" for someone whose only experience in programing is confined to the j2ee or dotNet stacks. If prisoners who've been clapped in irons appear bewildered or ecstatic on being released into the prison courtyard for some exercis in the sunlight, perhaps that is all right :-)

    I think your problem is that you consistently overestimate the intelligence and programming skill of the average enterprise programmer. Herein the UK, the very best programmers don't work on banking software, they get hired by google. Expecting the average database drone to understand compilers and language processing and compiler theory may be too optimistic :-).Let them part with their money and buy the "Enterprise DSL With Ruby" style books. The book authors have to eat :-)

    EOPL is an awesome book btw. Your reccomendation is well thought out. If there is one book that cuts through "dsl" hype (or any other buzzwords about languages), EOPL is it.

    Anonymous said...

    Who is this Joe Williams, who derides the average enterprise programmer ? Provide a link please.Let me read your posts ! Kindly mention your worthiness before ridiculing others.
    And yeah, Do you work for Google ? I am happy to stay anonymous as i neither work for google nor ridicule the average !!

    Anonymous said...

    dear Anonymous Idiot,

    You might want to at least make sure your post makes sense before submitting it.

    "I am happy to stay anonymous as i neither work for google nor ridicule the average"

    This kind of claptrap as a substitute for logic almost guarantees you are a bad programmer my anonymous friend. If you want to stay anonymous, do so. That has nothing to do with "work for google" or "ridicule the average".

    No field of work which requires application of logic, would hire you if the above is sample of your thinking. Read up on the "Ad Hominem" fallacy. Criticize what people say, not who they are.

    As for me, I don't have a blog (so you can't "read my posts") or a web page of my own. Last I checked, neither was mandatory to surf the web (and post blog comments). And I am a scientist, not a programmer.

    Now to my comments. I said "Expecting the average database drone to understand compilers and language processing and compiler theory may be too optimistic".

    Now which part of this statement do you disagre with. Why? If you have a logical counter argument, (vs "Joe Williams doesn't have a blog nyah nyah") let us hear it.

    Ravi said...

    whoops.

    This is degenerating into commenters calling each other names again.

    "Anonymous" did post a reply

    here is about the only publishable sentence

    "The gist of what i said is that posting a comment under the garb of anonymity is no different from posting one with a non blogger name."

    The rest came perilously close to insulting whole nations. ("Englishmen... ").

    None of that on this blog please.

    Anyway I am not very sure I agree with the idea that what id people chose is particularly important *as long as* they make sensible arguments.

    There are good reasons to avoid a blogger id (or prefer "anonymous"). Having said that someone who puts his name on a comment (even without a blogger id) is preferred to one who stays 'anonymous', especially if he/she also writes to me offline or is otherwise known to me.


    I had hoped that 'anonymous' would take up joe's challenge to identify precisely that he objected to in his *main* argument(== "you can't expect the average enterprise programmer to understand compiler theory" ) and kick start a good discussion.

    I don't care (much) what id people use. I do care what they post here using that id makes logical sense and targets a good discussion.

    oh well,

    Anonymous said...

    Joe said: "You are right to excoriate anyone who claims that ruby's dsl is a fundamentally new programming tecnique."

    I'd maintain that Ravi is right to contradict or correct them.

    As to 'excoriate'? Well, that's a question of politeness and decency.

    I believe the tone of Ravi's post borders on smug. But since I am commenting on the style of the post and am engaging in a variety of 'ad hominem', I guess I am just about as guilty as he is :) I suppose I mention all this, because there's no way that I'd treat someone in the way that's being advocated, because I was once that way -- the wide-eyed convert. Not all of us have come to programming via academia or with any strong CS background. Some of us learn it later in our careers. But, since he qualifed it with [rant] I'll assume Ravi just wants to get it off his chest.

    Anyways, the _content_ of Ravi's post is dead-on, and lucid. It boils down the essence of the metasystems out there, and I'll probably use the same wording when explaining the topic to others.

    I do have an itch, which cannot be scratched by the typical CS/math analysis of the situation. That itch is the question: why do certain languages permit embeddable DSLs better than others.

    For a very long while, I had accepted the conventional wisdom that ONLY very-simple syntactical systems with consistent and thorough paradigms could produce these internal DSLs. Smalltalk and Lisp are the poster children of this philosophy: lisp for its near-zero syntax to AST impedance, and Smalltalk for its consistency and access to the Compiler & Stack as objects.....

    But then Ruby proved that it could be done too. And by 'done' I mean 'done easily' and in a way that seems to encourage programmers to use it. There's no better indicator to success than results: there are clearly more embeddable DSLs in the Ruby world than in a lot of the big enterprise languages.

    So what is it about Ruby that permits this? It seems to take the opposite tact than a lisp; indeed a lisp purist would certainly disparage its philosophy: tons and tons of syntactic sugar. And yet it has gotten a lot of the same power. (don't get me wrong, when push comes to shove, Lisp will always when out in expressive power)

    Just something to think about...

    Daniel Eklund

    Ravi said...

    "Not all of us have come to programming via academia or with any strong CS background. "

    I haven't anyway :-).

    I majored (if spending four years running after girls and rarely attending class can be called "majored") in Industrial engineering and have no formal training in CS and am very much a self taught programmer.

    Having said that, I get irritated when *basic* computer science is "rediscovered" and worse, repackaged as the next silver bullet(as is beginning to happen with ruby dsls).

    "But, since he qualifed it with [rant] I'll assume Ravi just wants to get it off his chest"

    Oh Absolutely. I was doing some technical oversight for a venture capitalist here and someone essentially repackaged html as a ruby "dsl" and wanted funding because of his "cutting edge technology." [hit head on wall]

    Ruby *is* a marvellous language. "DSL" is a valuable technique to have in your arsenal.


    But DSLs are NOT a "fundamentally new" technique. It just appears that way to ignoramuses.


    "why do certain languages permit embeddable DSLs better than others"

    I would say presence of eval + (relative) simplicity of syntax + the language would enable this. To make it *easy* though some part of the language substarte would need to be exposed in the language itself.


    Ruby is very much a "son of smalltalk" and though the syntax is not as regular as that of smalltalk or lisp , the uniform use of objects throughout and modelling the language substrate as objects (ruby's weird notion of "metaclass" notwithstanding) enables the meta hackery, imo.

    Thanks Daniel for a lucid comment.