Multiple contexts for a DSL.

In the last two blogs I showed why and how you might hide your Java code with a DSL. I mentioned how one of the strengths of DSL's is that it separates the concerns of describing data from actions on that data. Here is a simple example.


Two things are obvious. First, this is a todo list. Second, I'm likely to lose some data if my harddrive fails. The todo reads fairly well. Now let's try associating an action to the data. We'll print the items.

With a Java mindset you would probably parse the data and write it out. Maybe someone who knew you were a Java developer provided you XML for the same data. Then you just have to deal with one of several painful XML API's rather than write a parser yourself. If you don't want to parse it at all then you can be the parser and put the data directly into your code.

Since this todo list description is actually a DSL internal to Ruby we can deal with it very easily. We just need to associate actions to keywords in the language like "todo" and "item." Here is how we do that.


When we execute the todo list with these methods defined we get this.

To do list:
mow the lawn
write blog about this
back up harddrive

This quickly gave us the results we wanted, without taking the time to parse our custom todo language. We just associated behavior with parts of the DSL and there it was. In Ruby, yield is used to execute the block that was passed in. In this case three calls to item() were made when todo() yields to the block of items in the DSL. We see all three descriptions because item() is defined as printing out the item description. A specific grouping of bindings like this is what we refer to as a context. We can do anything we can think of with this todo list just by evaluating it in a different context.

We can see only high priority items by changing the behavior associated with item().

To do list:
mow the lawn

Maybe we would like to see all past due items.

To do list:
mow the lawn
write blog about this

Let's do something really different and count how many items we have.

3

As you can see, evaluating this todo list description in different contexts is a pretty easy way to use our data. We have a lot of control over the situation since our data is in the same form as our code. This is much closer to how a LISP hacker would deal with the data than how a Java/XML developer would.

The tree DSL that was the subject of my last two blog entries had only one context. We could build the tree from the description, but we would also like to assert that a structure is part of a tree and query that structure. So you probably see where I am going with this. Let's add an assert context and a query context for our tree DSL.

This is a bit harder than our trivial todo list example. Our tree DSL is a little different than most languages because it doesn't have any keywords that we can associate specific behavior with. All the words are used to describe tree node names. The build context implementation makes heavy use of method_missing() that was added directly to the Java TreeNode class with JRuby. We are going to do the same thing here, we just want to have a choice about what method missing does. Let's review what we had before.


Having the behavior on the TreeNode directly keeps us from having to write a tree visitor. Most of this method is centered around recursively walking the tree. First we try to find the child with the name we are looking for. We build it if we can't find it. Then we evaluate any left over sub-structure on the child. Interestingly enough, for an assert and a query the only change needed is what action to take if the child is missing. Let's make that into an injectable strategy.


This is a more general purpose TreeNode. We can inject an action to happen when a named child is not found. That action will have the parent and the name of the missing child as parameters. Notice that if no action is injected then method_missing becomes private so that it can't be abused. But how do we inject the actions, and what exactly will they do?


We extend the Tree class that we used to create our structure last time. Build, assert, and query all create procs that will be used as the child missing actions. In each of them three steps occur. First, the action is set on the TreeNode class. Then the structure is evaluated with that action in place. Finally, the action is cleared.

I would like a way of doing this without actually storing the action on the TreeNode class. This implementation wouldn't work well with multi-threading, and I feel dirty causing side-effects like changing the missing child behavior of all TreeNodes temporarily. I could also take some of the repetition out, but I left it because it makes the important concepts more obvious at first glance. For our current requirements this implementation will get us by. Hopefully we have good unit tests so we can easily refactor this later.

The build action does exactly what was done before by creating the missing child. Assert returns a list of discrepancies that were seen while trying to compare the structures. When assert returns an empty list, it means the whole structure that was asserted is indeed in the tree. A query throws an exception because we were purposefully looking for something, and if it wasn't there it is a problem. Lets see this in action.


After using this a little, I see that I'd like to have a wildcard to use in place of a TreeNode sometimes. This would basically be a call to getChildren() on the Java object backing that node. I can still call the Java method directly you know.


These all do the same thing. The whole point of the DSL is to make this read well, so I'd really rather see something like this.


This is really only useful for query. I can accomplish this improved readability simply by defining the * method on TreeNode to delegate to getChildren() like so.


Now I see an even more convenient way to use this new wildcard feature. I want to be able to use it in the middle of a path rather than just at the end. This will open up a world of new possibilities.

Now I can build, assert, or query similar parts of the tree that belong in parallel forks at once. I implemented this by adding a method to Array itself, which delegates any method called on the Array to all elements in it. Java people tend to have a problem with adding methods to a core class. They just don't know how to handle things like this because Java doesn't let you do anything powerful since it might be dangerous.


Now I have a very concise and powerful way to manage my trees. I can build, query, and make assertions on them way more effectively than I could do in Java. I also use the same tree description DSL for all of these actions. Even if you cannot use Ruby or these techniques in production, it is ideal for prototyping and unit tests. I've greatly increased test coverage using this strategy on a project with a model that was difficult to create, inspect and manage. Not only that, but the DSL enabled other team members to write tests with much less ramp up time on the underlying architecture.

How to hide your Java with a DSL using JRuby

Last time I showed some Java code for building a simple tree. Then I showed a DSL for representing that tree. The point was basically to make you want to create such a DSL. This time I am going to show you how to make the DSL actually do something. We are going to write all the code that will allow that DSL to build a tree. If you've never seen it, or don't remember, please look back at the code from the last post. It's important to see the java code we started with and the DSL that we are going to implement. It's kind of like those before and after pictures you're always looking at. Seriously, come back when you are done, and this will make more sense.

It's not odd to start with the DSL. In fact I usually start with a language that would be ideal and then modify it just enough so that it is valid Ruby syntax. It is important to understand the language features and know how much you can get away with as far as syntax. As you begin implementing it, your language will probably change. In the end you hopefully end up with a concise and powerful abstraction.

Ruby has a lot of features that make it ideal for internal DSL's. Literal Arrays and Hashes are simple and very useful here, even though we won't use them today. We are going to focus on blocks and method_missing. Note that all of these concepts are missing from Java.

We are going to decorate our Java TreeNode class with JRuby. That's right, we are going to slap behavior implemented in Ruby directly on our Java class. And we are going to make it do things that can't even be done in Java. But first I want to introduce the IRB.

The IRB is a command line that you can execute Ruby in. With JRuby you get a JIRB, which is the same thing except you can execute Java there too. Sweet, interactive Java! As a Java developer, it is worth learning about JRuby just for this new ability. It's a more efficient way to explore how Java classes work than implementing main methods all over the place. I have also prototyped new behavior on my Java objects very effectively through the JIRB. The following examples are what I typed into the JIRB to test my ideas for the implementation of the DSL. If something doesn't work out, you just keep typing. The JIRB lets you try something and see the results very quickly. I have never felt so close to my Java code.


This first piece of new behavior we added to TreeNode is a good example. The new line character is different in Java and Ruby so we switch them out in the toString method. We had to require the jar for this class. Then we assign the qualified class name to a constant for convenience. After that adding a new method to our Java class is done as if it were Ruby. We could call toString, but JRuby allows us to call the Java methods with Ruby style so we call to_string instead. Note that is is completely unnecessary for the DSL. I only did it because I already had a toString that wrote out in the DSL format, and I wanted it to look right in the JIRB. The next example is where the meat is.


This code does most of the work for us. Method_missing is a special method in Ruby. A call to method_missing is what happens if you call a method that doesn't explicitly exist. Java wouldn't even compile if you did something like this. In this case we find or create a TreeNode with the name of the method. This lets us represent a path in our tree like this: shape.ellipse.circle

As opposed to this in Java.


So this is 1 line of our DSL rather than 22 lines of Java. Even if your developers don't know Ruby it should be obvious that they are going to have an easier time writing this immediately.

This path stuff is cool, but it's not what I promised. Since our data structure is a tree, we want to express multiple children for each TreeNode. To do this we are going to use blocks. Blocks are a way to pass executable code into a method. A good example of using a block is the call to the find method in the first line of our method_missing way up there. For each child in the Collection returned by getChildren() on this TreeNode we are checking to see if it has the same name as the method name that invoked method_missing.

Blocks will also allow us to describe a tree structure. The way this works is that method_missing gets called recursively. Each time it will find or create the named TreeNode, and then it allows that TreeNode to evaluate the remaining structure. All methods in Ruby can be passed a block. Blocks are placed after a method call and fit within '{...}' or 'do...end.' I chose the brackets because I liked the way it made the language look, but either would actually work. Nesting the blocks allows for a natural tree structure. The block can be accessed in the method with an extra parameter. In this case &block. Don't get too worried about the '&' just yet, it is the concepts that are important. The second to the last line deals with constructing the children. If a block which represents substructure of the tree is given, it is evaluated on our child. The way that we recursively traverse the described children is the most complicated and important part of the whole thing. Now we can build a tree described like this.


For readability it is nice to leave off the parens on the methods called. Everything that is a word in this DSL is actually a method call that invokes method missing, which creates a TreeNode that is added as a child. There is a bit of a chicken and egg problem with this example though. Once we have a TreeNode we are rolling, but how do we get our hands on the root TreeNode, in this case "shape?" Here is a way to start the whole thing off that is consistent with our DSL.


To make this work we just have to implement Tree to create our first TreeNode, and then hand the rest of the hierarchy off to it.


One of my first Ruby projects was a DSL, and it was not a bad way to learn the language. I have taken these concepts further in production code by describing a complicated meta model with frames and slots, effective dating, and internal rules. The ideas presented here scaled well to the more complicated project. The performance was also acceptable. There are a few problems that I neglected to mention because their solutions are straight forward. One of the more noteworthy is the possibility that you would have to encode and decode the names of TreeNodes into valid method names if they weren't already. This is one of the costs of using an internal DSL. That's fine with me because it is much easier than writing a parser. In the next entry I would like to show how this DSL can be used for more than just building the tree.

Why hide your Java with a DSL using JRuby?

The Java vs. Ruby dialogs are everywhere. I intend to show how you can use the complementary strengths of the Ruby language and the Java platform. This basically amounts to trying to avoid the Java language as much as possible, while leveraging the JVM and libraries. There is enough information around about the merits of Java, so I am going to focus on how Ruby can enable Java with new super powers.

The problem with the Java language is the lack of expressiveness. If you are naive enough to think that Java has no real problem here then do two things. First, keep reading while I give an example. Second, work with a better language for a while. Be warned that once you start down this path you will be permanently changed.

This will work a little like a magic trick. We will start with some Java code. As usual, java will look awkward and feel clumsy. Then we will use JRuby and some concepts that are missing from Java to make all the ugliness disappear. By the end we will have code that people can read or write without knowing a thing about Java or programming at all. Here's the Java code, check it over and make sure it's authentic.


This is simply a node in a tree. Sure it is contrived for this example, that's the point. It knows its name and the nodes below it. We can add children to it. This would be useful for building up a little hierarchy like this.


Java got the job done, but that's ugly. In fact I bet you skipped right over that code block. That's okay, I'll write it out like this so that you can easily see that it is a hierarchy of shape types.


Now the nested structure of shapes is pretty obvious at a glance. This is actually valid Ruby code. No matter how much more Java than Ruby you have written, our DSL is going to immediately be more readable to you. This is the point where it should start to dawn on you that this isn't about Java vs. Ruby, it's about using the tools available effectively.

A DSL is just an abstraction like a good API. While an API can insulate you from details with simpler concepts, a DSL can insulate you from unintuitive syntax and language features as well. Ruby has the ability to do this way better than Java can.

If you are just concerned with the business of using a hierarchy of things then you don't need to know Ruby or even Java now. The fact that this DSL is internal to Ruby, implemented in JRuby, and delegates to existing Java is just implementation details to the user. That means they don't need to know. Only the author of the DSL needs to know Ruby. Ruby does a lot for us since we don't have to write a parser and can use existing tools like IDE's.

One concern about DSL's is the creation of lots of mini languages and the investment to learn them. But there are mini languages everywhere. Picking up a DSL is similar to learning a new API or XML dialect.

In this case the DSL can potentially do much more than the original Java code. The DSL is focused only on describing the data structure while the original Java is tied to the action of building the structure. Separation of concerns here is useful because it lets you use the description for actions other than building. For example, I've used a similar DSL for making assertions about the structure of a tree and also writing queries against it. This works out especially handy in unit tests where you need a convenient way to express an initial structure and an expected value.

Hopefully I've convinced you that this is genuinely useful stuff. Java is ugly, and sometimes it is just better to hide it. A Ruby DSL can be a nice alternative. Now that the why is out of the way, next time I will focus on the how.