Taking JavaScript Seriously in Grails (AngularJS and CoffeeScript)

  Whenever I write JavaScript I throw all software craftsmanship discipline out the window. There are no unit tests and very little structure to the code. It's just me and the browser refresh button until we get it right. If you are lucky you may have jQuery to provide a nice CSS selector based api that somewhat shields you from browser quirks. Maybe you even kept your project from devolving to the point where every page is basically its own JavaScript app with no hope of reuse between them. Even then JavaScript is still likely a second class language in your project. Surprisingly, this state of affairs is normal. I want to take my client side scripting just as seriously as any other code in my Grails web app. I want tests, structure, and less boilerplate, and I am going to use AngularJS and CoffeeScript to do that.

  I started with a new Grails 2.1 app and implemented the excellent AngularJS tutorial on top of it. Once I got it working I turned all the JavaScript code into CoffeeScript. Overall these technologies work very nicely together. Of course there were a few tricks I learned along the way and I will point those out. Grails and AngularJS are both MVC web frameworks that make heavy use of dependency injection and have great support for testing. There is some overlap that I will discuss later, but for the most part Grails stays on the server side and AngularJS stays on the client side. AngularJS actually feels like a more Grailsy way to handle the view layer than what Grails provides out of the box.

Grails

  The Grails side of this is pretty standard. I've used the handy resources plugin to manage my client side static resources like images, js, and css files. It allows you to define dependencies between these resources and then bring the required ones into a page just by specifying the top level of the hierarchy. There is a Grails Controller (shown below) with two kinds of actions. There are pass through actions (list and show) that defer to AngularJS, and end points for restful calls (query). PhoneService is a Grails service that handles the actual data. In a real application it would be backed by a proper data store, but here I hardcoded the actual JSON from the AngularJS tutorial.

AngularJS

  Though there is an AngularJS plugin for Grails, I didn't see any reason to use it. The Grails resources plugin makes using AngularJS so simple, and I like having direct control over which version of AngularJS I use and where the files are put.

  Much like Grails, AngularJS has its own set of controllers (for page manipulation), services (for general purpose tasks), and dependency injection to tie everything together. The DI feels a lot like Groovy with the way that injection takes place by name automatically. If you look deeper it starts to look a lot like Guice. Below are the Angular controllers used in this application to get JSON data and binding it to the page via $scope.

CoffeeScript

  There is much more to say about AngularJS, but before I do I'd like to mention that the AngularJS code that I am showing has been converted to CoffeeScript. I find CoffeeScript to be much more pleasant to use that JavaScript (and more Groovy like). The conversion was relatively straight forward, but there were a few gotchas around scoping of variables due to CoffeeScript's lexical scopes and implicit function safety wrapper. As you can see in the above example it is necessary to prefix controllers with an @ to elevate their scope. Also my AngularJS service test shown below uses a 'setup' variable to give the tests access to objects created in the 'beforeEach' section.

  I used the Grails CoffeScript plugin for this, but it was an older version that I found useful. There is a handy way that one can configure arbitrary CoffeeScript files to be converted to JavaScript. I liked this approach because I was content to let the same mechanism convert my CoffeeScript tests that is converting the rest of my CoffeeScript code. The newer version of the plugin doesn't seem capable of this. Being new to CoffeeScript, I also appreciated seeing the generated JavaScript.

Grails and AngularJS Together

  There are limitless ways that you could combine Grails and AngularJS. By posting this I hope to gain some new insights. I favor Grails quite a bit here just because I am more comfortable with Groovy. Think of this example as a Grails app which delegates to an AngularJS sidecar specializing in putting the final touches on the view layer.

Grails vs AngularJS responsibilities

Grails	AngularJS
URL mapping for full page transitions	client side data binding and dom manipulation (single page changes)
provides RESTful services	makes the AJAX calls
application concerns (overall file layout, server and client side dependencies, building and packaging, control center for all commands)	JavaScript organization and separation of concerns
server side DI with Spring	client side AngularJS DI
server side unit, integration and overall functional tests with Spock	JavaScript unit testing with Jasmine

  Rather than use AngularJS routing, I let Grails handle it (mostly because I am more familiar with Grails). You can see in PhoneController above that the pass through actions act very much like AngularJS routes. They map a url to a template (by Grails convention) and an AngularJS controller (by passing an explicit value to the template). In this case the template is a Grails .gsp.

  Grails and AngularJS share the templates which in this case are .gsp files. There are two passes to transform these files now. First Grails compiles the .gsp on the server and applies tag libraries, then AngularJS applies it's bindings and directives on the client. This is where AngularJS shines and I let it do most of the work. One thing to remember is that since Grails encodes the output of tag libraries, you can't use the AngularJS binding inside them. For example...

  Getting Grails and AngularJS to communicate effectively is simple once you realize that Grails params and AngularJS $scope are both the state that the respective controllers care about. So I found a way to stamp the Grails controller's params onto the AngularJS controller's $scope using ng-init. The main.gsp is a layout that I apply to all my Grails .gsp files. It is basically a place to put common elements that will appear on every page. You can see how I am taking this opportunity to always set the ng-controller (which was specified by name in the Grails controller) and pass the Grails params into it with ng-init.

  AngularJS provides resources for making rest calls. This is a great way to communicate with the Grails backend. To remove a lot of boilerplate code I created a service that constructs resources based on Grails URL conventions. You can see in the AngularJS controllers above where I make use of the service named 'Grails' to get phone data. The code for this service is below. It simply substitutes Grails controller, action and id values for their placeholder in the url. It will default to values found in $scope (which were copied from the Grails params if you remember).

Unit Tests

  Grails itself offers great support for writing tests, especially with the 2.0+ enhancements that allow better Spock integration. I must say that testing my code with Spock is bliss with its concise syntax, built in mocking, and intuitive assertions.

  The thing about AngularJS that really got my attention is the promise that I can be in a similarly happy place when writing JavaScript tests. And it delivers! This is no doubt because there are test gurus behind AngularJS. Jasmine allows my unit tests to have a familiar structure coming from Spock. The DI that is core to AngularJS allows me to isolate my logic and mock what I need to. The tests are run by js-test-driver which needs to connect to a browser. Your web app doesn't have to be running, js-test-driver just uses the browser as an engine to execute the javascript (and you can execute in multiple browsers in parallel). There is an IntelliJ plugin for js-test-driver that lets me run my tests as if they are JUnit. There is even JUnit output that I can use in Jenkins. And surprisingly code coverage for JavaScript (but I lost this when I converted to CoffeeScript). It was trivial to create a unit test for my AngularJS service described above that makes rest calls to Grails. I merely instantiate the service, mock the backend, and assert that the right calls would be made based on the scope and overrides.

Functional Tests

  Functional tests run against your app externally, usually through some type of browser automation. AngularJS provides end-to-end testing to accomplish this. These tests are described with Jasmine just like AngularJS unit tests. They are executed in the browser and are blazingly fast. AngularJS e2e tests are awesome, but I used a Grails based solution instead.

   Geb is a Groovy way to automate a browser and it works well with Spock to write Grails functional tests. It's basically a Groovy DSL on top of Selenium/WebDriver. Even though it is not as fast as AngularJS e2e tests, I am more familiar with these tools and I like the page object pattern that Geb uses to separate browser automation in the test from the specific markup of the page.

Above you can see the definition of a page. Most of the CSS selectors that address page elements can go here. Aside from avoiding repetition of these selectors, it makes the tests less brittle. When there are changes to the markup you just make a corresponding change to your page definition and all your tests still pass. There's no need to change the test itself unless the behavior of your test conceptually changes.

  Below you can see the related test. The setup method navigates to the specific page and the tests use the page object definitions to manipulate and make assertions about the page. Note how I am able to pass 'phoneThumb' an index so that I can click arbitrary thumbnail images. I think this makes for very readable and robust tests. Converting the AngularJS e2e tests from the tutorial to Geb was very intuitive.

  Running the Geb tests was not so straight forward. There is a Geb plugin for Grails that does some basic setup of Geb inside of Grails, and there is tighter integration to be had by following this tip from the Geb author. The plugin allows you to run the tests from the Grails command line, but I had to resort to a bit of a hack to run the tests from my IDE. Basically I needed a way to configure Geb outside of the Grails app.

  Speaking of Geb configuration, I have a few observations about my choice of driver. I tried HtmlUnitDriver, FirefoxDriver and ChromeDriver. HtmlUnitDriver just didn't work for my tests. FirefoxDriver worked out of the box, but was slow. ChromeDriver was fast (much closer to AngularJS e2e speeds) but there is a minimal amount of setup to be done on your machine first.

Benefits

  AngularJS brings a lot of the same good things to my client code that Grails brings to my server side code (structure, DI, tests). CoffeeScript makes my client code look and feel more like Groovy (less boilerplate, many common idioms). Overall my client side scripting is now a first class citizen and sits right beside the rest of my code (literally and figuratively).

Code

  All the code is on Gtihub. Feel free to use this as a starting point and send pull requests when you figure out improvements.

Todo

• Run in jenkins build on headless server and capture test failures (bonus points for code coverage)
• Use Testacular rather than js-test-driver to run unit tests.
• Replace CSS and HTML with Less and Jade respectively.
• Full CRUD with RESTful Grails endpoints.

Multiple contexts for a DSL.

In the last two blogs I showed why and how you might hide your Java code with a DSL. I mentioned how one of the strengths of DSL's is that it separates the concerns of describing data from actions on that data. Here is a simple example.


Two things are obvious. First, this is a todo list. Second, I'm likely to lose some data if my harddrive fails. The todo reads fairly well. Now let's try associating an action to the data. We'll print the items.

With a Java mindset you would probably parse the data and write it out. Maybe someone who knew you were a Java developer provided you XML for the same data. Then you just have to deal with one of several painful XML API's rather than write a parser yourself. If you don't want to parse it at all then you can be the parser and put the data directly into your code.

Since this todo list description is actually a DSL internal to Ruby we can deal with it very easily. We just need to associate actions to keywords in the language like "todo" and "item." Here is how we do that.


When we execute the todo list with these methods defined we get this.

To do list:
mow the lawn
write blog about this
back up harddrive

This quickly gave us the results we wanted, without taking the time to parse our custom todo language. We just associated behavior with parts of the DSL and there it was. In Ruby, yield is used to execute the block that was passed in. In this case three calls to item() were made when todo() yields to the block of items in the DSL. We see all three descriptions because item() is defined as printing out the item description. A specific grouping of bindings like this is what we refer to as a context. We can do anything we can think of with this todo list just by evaluating it in a different context.

We can see only high priority items by changing the behavior associated with item().

To do list:
mow the lawn

Maybe we would like to see all past due items.

To do list:
mow the lawn
write blog about this

Let's do something really different and count how many items we have.

3

As you can see, evaluating this todo list description in different contexts is a pretty easy way to use our data. We have a lot of control over the situation since our data is in the same form as our code. This is much closer to how a LISP hacker would deal with the data than how a Java/XML developer would.

The tree DSL that was the subject of my last two blog entries had only one context. We could build the tree from the description, but we would also like to assert that a structure is part of a tree and query that structure. So you probably see where I am going with this. Let's add an assert context and a query context for our tree DSL.

This is a bit harder than our trivial todo list example. Our tree DSL is a little different than most languages because it doesn't have any keywords that we can associate specific behavior with. All the words are used to describe tree node names. The build context implementation makes heavy use of method_missing() that was added directly to the Java TreeNode class with JRuby. We are going to do the same thing here, we just want to have a choice about what method missing does. Let's review what we had before.


Having the behavior on the TreeNode directly keeps us from having to write a tree visitor. Most of this method is centered around recursively walking the tree. First we try to find the child with the name we are looking for. We build it if we can't find it. Then we evaluate any left over sub-structure on the child. Interestingly enough, for an assert and a query the only change needed is what action to take if the child is missing. Let's make that into an injectable strategy.


This is a more general purpose TreeNode. We can inject an action to happen when a named child is not found. That action will have the parent and the name of the missing child as parameters. Notice that if no action is injected then method_missing becomes private so that it can't be abused. But how do we inject the actions, and what exactly will they do?


We extend the Tree class that we used to create our structure last time. Build, assert, and query all create procs that will be used as the child missing actions. In each of them three steps occur. First, the action is set on the TreeNode class. Then the structure is evaluated with that action in place. Finally, the action is cleared.

I would like a way of doing this without actually storing the action on the TreeNode class. This implementation wouldn't work well with multi-threading, and I feel dirty causing side-effects like changing the missing child behavior of all TreeNodes temporarily. I could also take some of the repetition out, but I left it because it makes the important concepts more obvious at first glance. For our current requirements this implementation will get us by. Hopefully we have good unit tests so we can easily refactor this later.

The build action does exactly what was done before by creating the missing child. Assert returns a list of discrepancies that were seen while trying to compare the structures. When assert returns an empty list, it means the whole structure that was asserted is indeed in the tree. A query throws an exception because we were purposefully looking for something, and if it wasn't there it is a problem. Lets see this in action.


After using this a little, I see that I'd like to have a wildcard to use in place of a TreeNode sometimes. This would basically be a call to getChildren() on the Java object backing that node. I can still call the Java method directly you know.


These all do the same thing. The whole point of the DSL is to make this read well, so I'd really rather see something like this.


This is really only useful for query. I can accomplish this improved readability simply by defining the * method on TreeNode to delegate to getChildren() like so.


Now I see an even more convenient way to use this new wildcard feature. I want to be able to use it in the middle of a path rather than just at the end. This will open up a world of new possibilities.

Now I can build, assert, or query similar parts of the tree that belong in parallel forks at once. I implemented this by adding a method to Array itself, which delegates any method called on the Array to all elements in it. Java people tend to have a problem with adding methods to a core class. They just don't know how to handle things like this because Java doesn't let you do anything powerful since it might be dangerous.


Now I have a very concise and powerful way to manage my trees. I can build, query, and make assertions on them way more effectively than I could do in Java. I also use the same tree description DSL for all of these actions. Even if you cannot use Ruby or these techniques in production, it is ideal for prototyping and unit tests. I've greatly increased test coverage using this strategy on a project with a model that was difficult to create, inspect and manage. Not only that, but the DSL enabled other team members to write tests with much less ramp up time on the underlying architecture.

How to hide your Java with a DSL using JRuby

Last time I showed some Java code for building a simple tree. Then I showed a DSL for representing that tree. The point was basically to make you want to create such a DSL. This time I am going to show you how to make the DSL actually do something. We are going to write all the code that will allow that DSL to build a tree. If you've never seen it, or don't remember, please look back at the code from the last post. It's important to see the java code we started with and the DSL that we are going to implement. It's kind of like those before and after pictures you're always looking at. Seriously, come back when you are done, and this will make more sense.

It's not odd to start with the DSL. In fact I usually start with a language that would be ideal and then modify it just enough so that it is valid Ruby syntax. It is important to understand the language features and know how much you can get away with as far as syntax. As you begin implementing it, your language will probably change. In the end you hopefully end up with a concise and powerful abstraction.

Ruby has a lot of features that make it ideal for internal DSL's. Literal Arrays and Hashes are simple and very useful here, even though we won't use them today. We are going to focus on blocks and method_missing. Note that all of these concepts are missing from Java.

We are going to decorate our Java TreeNode class with JRuby. That's right, we are going to slap behavior implemented in Ruby directly on our Java class. And we are going to make it do things that can't even be done in Java. But first I want to introduce the IRB.

The IRB is a command line that you can execute Ruby in. With JRuby you get a JIRB, which is the same thing except you can execute Java there too. Sweet, interactive Java! As a Java developer, it is worth learning about JRuby just for this new ability. It's a more efficient way to explore how Java classes work than implementing main methods all over the place. I have also prototyped new behavior on my Java objects very effectively through the JIRB. The following examples are what I typed into the JIRB to test my ideas for the implementation of the DSL. If something doesn't work out, you just keep typing. The JIRB lets you try something and see the results very quickly. I have never felt so close to my Java code.


This first piece of new behavior we added to TreeNode is a good example. The new line character is different in Java and Ruby so we switch them out in the toString method. We had to require the jar for this class. Then we assign the qualified class name to a constant for convenience. After that adding a new method to our Java class is done as if it were Ruby. We could call toString, but JRuby allows us to call the Java methods with Ruby style so we call to_string instead. Note that is is completely unnecessary for the DSL. I only did it because I already had a toString that wrote out in the DSL format, and I wanted it to look right in the JIRB. The next example is where the meat is.


This code does most of the work for us. Method_missing is a special method in Ruby. A call to method_missing is what happens if you call a method that doesn't explicitly exist. Java wouldn't even compile if you did something like this. In this case we find or create a TreeNode with the name of the method. This lets us represent a path in our tree like this: shape.ellipse.circle

As opposed to this in Java.


So this is 1 line of our DSL rather than 22 lines of Java. Even if your developers don't know Ruby it should be obvious that they are going to have an easier time writing this immediately.

This path stuff is cool, but it's not what I promised. Since our data structure is a tree, we want to express multiple children for each TreeNode. To do this we are going to use blocks. Blocks are a way to pass executable code into a method. A good example of using a block is the call to the find method in the first line of our method_missing way up there. For each child in the Collection returned by getChildren() on this TreeNode we are checking to see if it has the same name as the method name that invoked method_missing.

Blocks will also allow us to describe a tree structure. The way this works is that method_missing gets called recursively. Each time it will find or create the named TreeNode, and then it allows that TreeNode to evaluate the remaining structure. All methods in Ruby can be passed a block. Blocks are placed after a method call and fit within '{...}' or 'do...end.' I chose the brackets because I liked the way it made the language look, but either would actually work. Nesting the blocks allows for a natural tree structure. The block can be accessed in the method with an extra parameter. In this case &block. Don't get too worried about the '&' just yet, it is the concepts that are important. The second to the last line deals with constructing the children. If a block which represents substructure of the tree is given, it is evaluated on our child. The way that we recursively traverse the described children is the most complicated and important part of the whole thing. Now we can build a tree described like this.


For readability it is nice to leave off the parens on the methods called. Everything that is a word in this DSL is actually a method call that invokes method missing, which creates a TreeNode that is added as a child. There is a bit of a chicken and egg problem with this example though. Once we have a TreeNode we are rolling, but how do we get our hands on the root TreeNode, in this case "shape?" Here is a way to start the whole thing off that is consistent with our DSL.


To make this work we just have to implement Tree to create our first TreeNode, and then hand the rest of the hierarchy off to it.


One of my first Ruby projects was a DSL, and it was not a bad way to learn the language. I have taken these concepts further in production code by describing a complicated meta model with frames and slots, effective dating, and internal rules. The ideas presented here scaled well to the more complicated project. The performance was also acceptable. There are a few problems that I neglected to mention because their solutions are straight forward. One of the more noteworthy is the possibility that you would have to encode and decode the names of TreeNodes into valid method names if they weren't already. This is one of the costs of using an internal DSL. That's fine with me because it is much easier than writing a parser. In the next entry I would like to show how this DSL can be used for more than just building the tree.

Why hide your Java with a DSL using JRuby?

The Java vs. Ruby dialogs are everywhere. I intend to show how you can use the complementary strengths of the Ruby language and the Java platform. This basically amounts to trying to avoid the Java language as much as possible, while leveraging the JVM and libraries. There is enough information around about the merits of Java, so I am going to focus on how Ruby can enable Java with new super powers.

The problem with the Java language is the lack of expressiveness. If you are naive enough to think that Java has no real problem here then do two things. First, keep reading while I give an example. Second, work with a better language for a while. Be warned that once you start down this path you will be permanently changed.

This will work a little like a magic trick. We will start with some Java code. As usual, java will look awkward and feel clumsy. Then we will use JRuby and some concepts that are missing from Java to make all the ugliness disappear. By the end we will have code that people can read or write without knowing a thing about Java or programming at all. Here's the Java code, check it over and make sure it's authentic.


This is simply a node in a tree. Sure it is contrived for this example, that's the point. It knows its name and the nodes below it. We can add children to it. This would be useful for building up a little hierarchy like this.


Java got the job done, but that's ugly. In fact I bet you skipped right over that code block. That's okay, I'll write it out like this so that you can easily see that it is a hierarchy of shape types.


Now the nested structure of shapes is pretty obvious at a glance. This is actually valid Ruby code. No matter how much more Java than Ruby you have written, our DSL is going to immediately be more readable to you. This is the point where it should start to dawn on you that this isn't about Java vs. Ruby, it's about using the tools available effectively.

A DSL is just an abstraction like a good API. While an API can insulate you from details with simpler concepts, a DSL can insulate you from unintuitive syntax and language features as well. Ruby has the ability to do this way better than Java can.

If you are just concerned with the business of using a hierarchy of things then you don't need to know Ruby or even Java now. The fact that this DSL is internal to Ruby, implemented in JRuby, and delegates to existing Java is just implementation details to the user. That means they don't need to know. Only the author of the DSL needs to know Ruby. Ruby does a lot for us since we don't have to write a parser and can use existing tools like IDE's.

One concern about DSL's is the creation of lots of mini languages and the investment to learn them. But there are mini languages everywhere. Picking up a DSL is similar to learning a new API or XML dialect.

In this case the DSL can potentially do much more than the original Java code. The DSL is focused only on describing the data structure while the original Java is tied to the action of building the structure. Separation of concerns here is useful because it lets you use the description for actions other than building. For example, I've used a similar DSL for making assertions about the structure of a tree and also writing queries against it. This works out especially handy in unit tests where you need a convenient way to express an initial structure and an expected value.

Hopefully I've convinced you that this is genuinely useful stuff. Java is ugly, and sometimes it is just better to hide it. A Ruby DSL can be a nice alternative. Now that the why is out of the way, next time I will focus on the how.