Turtle Graphics DSL Implementation, part 2

In the previous post, I described the first part of how Napili implements a simple Turtle Graphics DSL.  Please follow the link to download the whole program, if you want to see the code in context.  This time, we’ll cover the Binding class.

As I’ve talked about before, the Binding class provides default values to variables.  I’ve also previously discussed Property Dispatch, as well as a trick in using the Binding to work around the problem of optional parenthesis only being possible for methods with arguments.  I’ll go over all of that again here, but just in case you’re joining us for the first time, rereading those older posts might be handy.

The BasicBinding class in Napili serves several purposes – it creates the Turtle object, and serves it out to either the user directly or via the BaseScript (as I covered in the previous post).  It also handles the “out” Binding variable (more on that later).  And it handles no-arg methods on the Turtle object.  Let’s go through the class, section by section, and we’ll discuss how it does that.

private static class BasicBinding extends Binding {
  Turtle turtle

  public BasicBinding() {
    super();
    turtle = new Turtle()
  }

This part should be self expanitory – the constructor  creates the Turtle object which is used to draw, and saves that for later use.

The next method, getVariable, provides the Binding variables to the user scripts.

  public Object getVariable(String name) {
    if (name == 'out') {
      return NapiliOutput.getPrintWriter();
    } 

out is a special value for Bindings – it’s the value that’s used by the println() method.  So, by supplying our own value for it, we can redirect user println calls from going to System.out (and the console) and instead send the output to somewhere more useful and visible to the user. In this case, that redirection happens to an output tab in the UI.

    if (name == 'turtle') {
      return turtle;
    }

The turtle variable is used in the BaseScript file we covered last time.  It’s also present for the user to manipulate directly – which could be handy for advanced users who want to create two or more turtles.

    // treat all no-arg methods on Turtle as binding variables with side effects
    if (turtle.metaClass.respondsTo(turtle,name)) {
      return turtle.metaClass.invokeMethod(turtle, name)
    }
    return super.getVariable(name)
  }

The next method, setVariable, isn’t anything special – since all the code will stop working in a strange and mysterious way if the value of turtle is overridden, don’t allow that to happen.

  public void setVariable(String name, Object value) {
    if ("turtle".equals(name)) {
      NapiliOutput.println('Unable to set "turtle" to value' + value)
      return;
    }
    super.setVariable(name, value);
  }
}

And that’s it.  I’ve now gone over all the DSL magic in the Groovy Turtle Graphics program.  Hopefully you found this relatively clear – if not, don’t hesitate to ask questions.  

Turtle Graphics DSL Implementation, part 1

Last post, I described a Turtle Graphics DSL, as implemented in my sample program Napili.  (Please download it if you want to see the full code I’m discussing here).  This time, I’ll discuss how to implement it.  It’s surprisingly easy – though long enough that I’m going to tackle it in chunks, this is just Part 1.

First, to run the program, there’s the following snip of code inside a Java class (which is triggered when the user clicks the Run button):

Class clazz = Class.forName("org.netdance.napili.language.BasicRunner"); 
InvokerHelper.getMetaClass(clazz).invokeStaticMethod(clazz, "run", new Object[]{});

All this does is use Groovy’s dynamic invoker structures to invoke BasicRunner.run() – this code is only there so I can swap out the Groovy DSL classes at a later time, when I’ll discuss things like security.  I could have done the same thing with Java’s reflection classes, but if you already have Groovy in your class path, I heartily advise using InvokerHelper, there’s lots of handy stuff in there, and it’s considerably easier than using Java’s native reflection library.  One thing to be aware of, though, if you do:  it’s in the package org.codehaus.groovy.runtime, which means that the Groovy team reserves the right to change the APIs whenever they need to – which could make an upgrade of your Groovy version painful (though it’s not likely that InvokerHelper.getMetaClass(Class) is going to change anytime soon).

There are three classes that make up the language portion of the code.  The first, and by far easiest, is the BaseScript (you’ll remember, I hope, that I already discussed BaseScripts).  Written in Groovy, it becomes fairly trivial:

package org.netdance.napili.language
abstract class TurtleDelegateBaseScript extends Script {
    def methodMissing(String name, args) {
        turtle."$name"(*args)
    }
}

This uses a different method of Method Dispatch than I’ve previously discussed – he methodMissing method is called by Groovy whenever there’s no method found which matches a given name.  Since we don’t provide any methods at all, that means that pretty much every method called is going to go through this method.  (Since Scripts define other methods which will be inherited, like run and getBinding, and those won’t go through methodMissing, but that’s a problem for another day).  

The meat of the program is just a single line, in bold above.  I’ve already described what that kind of call does when I discussed Method Dispatch, but it’s worth going through one more time.  The first word, turtle, is interpreted as a Binding variable.   So the Groovy script goes into the Binding class (which I’ll describe shortly), and fetches the value corresponding to turtle.   Next, the “$name” is substituted with the method name called, which will then call the method name on the Turtle object.  Lastly, * is called the spread operator, and serves to break up the list of arguments into individual objects.   At runtime, a command like forward 100 will thus be transformed into something like: getBinding().getVariable("turtle").forward(100) - moving the turtle forward 100 steps.

The next portion is the BasicRunner itself – you’ll recall that we called the static public run method of that class, above.  It looks something like this:

static def run() {
  String scriptStr = Napili.code;
  def config = new CompilerConfiguration()
  config.setScriptBaseClass("org.netdance.napili.language.TurtleDelegateBaseScript")
  def ic = new ImportCustomizer();
  ic.addImports('javafx.scene.paint.Color')
  def ti = new ASTTransformationCustomizer([value: 15L],TimedInterrupt)
  config.addCompilationCustomizers(ic, ti)
  def shell = new GroovyShell(config)
  try {
    def script = shell.parse(scriptStr)
    script.setBinding(new BasicBinding())
    script.run()
  } catch (Exception e) {
    // error handling here
  }
}

In this method, first you fetch the script String to execute from the passed class.  Then, create a CompilerConfiguration object.  To the configuration object, set the BaseScript to the TurtleDelegateBaseScript just described, above.  Add a new import of javafx.scene.paint.Color, which will be applied to every script, so the user can say things like pencolor Color.PURPLE without having to do an import themselves. 

The next line, with it’s creation of an ASTTransformationCustomizer, looks a bit mysterious, but it’s effect is very straightforward – it sets a timeout on the script of 15 seconds.  After that, the script will throw an exception.  How it does this is a bit of an involved explanation, but we’ll have a lot more to say about AST Tranformations in a future post.

Then, as I’ve previously covered, just create a GroovyShell, parse the script, add a Binding (as I’ve also previously covered), and run the script.  If the user makes an error in the program, it’ll be caught as an Exception in the catch block.

The Binding itself isn’t terribly complicated either, but this is getting long, so I’ll continue on to describe the Binding section in the next post.

A Turtle Graphics DSL

In my previous post, I discussed the example program that I wrote to exercise the things I’ve described so far.  Today I’ll discuss what the DSL will actually look like – as with most programming tasks, designing your DSL before implementation is almost always advised.

There are a couple of different ways that you can use to describe languages, the most popular (and my favorite) being BNF, but for our purposes, that’s certainly overkill.  Instead, let’s just do this informally.

First, we’ll allow all existing Groovy operators and control structures – there are good reasons to strip those out on occasion, but this isn’t one of them.  We’ll cover how to do that at a later date.  What this means is that structures like while (condition) {block} and if (condition) {block} will work just fine.  We’ll also be able to create methods inline, just as you can with Groovy scripts.  And standard comments (i.e., // and /* */) will also work as expected.

We’ll need the following commands to create an environment that allows for turtle graphics:

• Turtle / Movement Commands
• forward number 
• move the turtle forward by number of steps
• back number
• move the turtle backward by number of steps
• right number
• turn the turtle number of degrees to the right (clockwise)
• left number
• turn the turtle number of degrees to the left (counter clockwise)
• home
• return the turtle to the starting position, without drawing any lines
• Pen / Drawing Commands
• pendown
• set the pen to draw when the turtle moves
• penup
• set the pen to not draw when the turtle moves
• pencolor Color
• change the pen color to Color
• Animation / Demo Commands
• show
• show the turtle icon
• hide
• hide the turtle icon
• speed number
• change the speed at which the turtle draws, larger is faster

Keywords/methods are in bold, parameters are in italics.  Since we need to decide what number means, we can define it as a double, and since Groovy will auto convert int to double, we’re done with that.  Color is a little more problematic – ideally, we’d make it easy to use and understand, and have a whole bunch of code to do that, but since we’re currently just making a sample program, we’ll just use javafx.scene.paint.Color.

 

If you download the Groovy Turtle Graphics program Napili, you’ll see that all of these methods are already implemented in org.netdance.napili.Turtle.  I’ll talk about how to turn those methods into a DSL in the next post.  For now, here’s a sample program that you can run using these commands, along with the standard stuff that Groovy provides:

 

def circle(size) {
    45.times {
        forward size
        right 8
    }
}

speed 3
penup
forward 250
pendown
pencolor Color.PURPLE

12.times {
    circle 10
    right 30
}

home
hide

This program draws a series of interlocking circles, like you’d get from the child’s toy Spirograph.   We define a method (circle), set the speed a bit higher to make it all draw quickly, move forward without drawing to center the design, set the pen color to purple (PURPLE is one of many predefined values in javafx.scene.paint.Color), then draw 12 circles, turning 30 degrees after each.  At the end, we park the turtle in the home position and then hide it.  If you’re unfamiliar with Groovy, Integer.times(Closure) is a method that Groovy places on all values of Integer (and primitives are auto boxed to their class type in Groovy) – it just executes the closure as many times as the value of the integer – remember, parenthesis are optional for methods with arguments, so the circle method is really the equivalent of Integer.valueOf(45).times({forward size; right 8;}) - I trust you’ll agree that the format I used is actually easier to read, once you’re used to it.

See you next time, where I’ll talk about how to turn the methods on Class Turtle into the DSL above.

DSLs – What we’ve covered so far…

At this point, we’ve covered many of the basics of creating DSLs in Groovy:

• What DSLs are, and why you’d use them
• A quick introduction to Groovy, part 1 and part 2
• Dynamically executing Groovy Programs
• Creating binding variables (the nouns of your language)
• Creating new methods (the verbs of your language)
• Delegating behavior via Method Dispatch and Property Dispatch

With that, we’re ready to tackle a real example…  Which I’ll detail in the next post.

Property Dispatch on Groovy Objects

In my last post, I covered Method Dispatch on Groovy Objects.  The next topic is similar, Property Dispatch.

In Groovy, you can specify properties on objects in the same way that you can specify fields in Java:  With the “.” operator.  I.e., if I have an object named “Test”, and it has a class variable named “test”, you can access it with the phrase “Test.test” (just like Java).

As I already mentioned in my Introduction to Groovy, Groovy provides automatic creation of accessors and mutators (get and set).    That’s actually an oversimplification, but we’ll leave it at that for now.  As a result, consider the following Groovy class:

class GetSet {
   def firstField = 1
}

This can be used in the following way, inside a Java class:

GetSet gs = new GetSet();
System.out.println(gs.getFirstField());
gs.setFirstField("test");
System.out.println(gs.getFirstField());

That’s because, under the covers, void setFirstField(Object) and Object getFirstField() are added to the Groovy class.  Note that because the class field is declared as def, the type of these messages is Object.

While that’s handy in Java, in Groovy, we can do so much more with properties.  You can access them with dot notation:

def gs = new GetSet()
gs.firstField = 2.5
println gs.firstField

You can access them with get/setProperty (which is part of GroovyObject, the base object for all objects in Groovy):

def gs = new GetSet()
gs.setProperty('firstField','test2')
println gs.getProperty('firstField')

And you can even access a Map of all the properties on the object, and operate on that:

def gs = new GetSet()
println gs.properties.firstField

This last example is interesting since it’s using a few features of Groovy: the properties property is actually a shortcut to the getProperties() method, which returns a Map.  In turn, you can access the values of the map by using the key in dot notation.  So this is actually a shortcut for the following Java code: 

GetSet gs = new GetSet();
System.out.println(gs.getProperties().get("firstField"));

While all that’s handy, that’s still not dispatch.  In order to handle properties dispatch, there are two different methods we can use.  Consider the following code:

GetSet gs = new GetSet();
gs.newprop = 'new value'
println gs.newprop

newprop doesn’t exist in GetSet – we want to handle adding it at runtime.  The first method to do so is to override missingProperty(String) and missingProperty(String, Object).  The first method handles gets of properties that don’t exist, the second handles sets.  So the following code handles unknown properties seamlessly:

Map props = [:]
    
def propertyMissing(String name, value) {
    props[name] = value
}
    
def propertyMissing(String name) {
    return props[name]
}

In Groovy, Maps can be initialized to be empty with [:] and the values in that map can be accessed with the syntax pattern map[key].  So, in this code, if the property doesn’t exist, it will be added into an internal Map if you are setting it, and that value will be returned when the property is later used.  If the property is accessed before it’s set, a null is returned.  Of course, we could check for existence, and throw an exception if we desired to. 

This method will only add to the existing properties, however.  If there’s already a property (a class instance field, for instance) on the class, such as firstField in our example, this code is never touched.

If instead you want to completely override the access to fields, you can instead override getProperty(String), setProperty(String, Object) and getProperties() for good measure, if desired.  Here’s an example:

Map props = [:]
    
void setProperty(String name, value) {
    props[name] = value
}
    
Object getProperty(String name) {
    return props[name]
}
    
Map getProperties() {
    return props
}

This code actually does almost the same thing as the previous example, except that now, any instance or class fields are hidden from property access.    Meaning that the following code prints null:

GetSet gs = new GetSet();
println gs.firstField

But note that the following code will still print 1:

GetSet gs = new GetSet();
println gs.@firstField

This is because we’re using the “Java field accessor” in the second example – the .@  notation bypasses get/setProperty methods, and directly act on the underlying Java field.

Dispatch is just a short step away from the above code.  Instead of using the internal Map of properties, you could instead route the property lookup anywhere – an xml file, a database, or any other class or computation.

So, now that we’ve looked at method and property dispatch, we’ll run through some ways to use these in a DSL next time.  Until then…

 

Method dispatch on Groovy objects

 

As I promised my post about optional parenthesis, today I’m going to talk about method dispatch in  Groovy – specifically, dynamically dispatching methods on Groovy objects via the invokeMethod mechanism.  We’ll tie it into DSL creation in a later post.

The key method to know about for method dispatch is GroovyObject.invokeMethod(String name, Object args).  This method is located on GroovyObject, and is called when the method you called isn’t present on the object.

Consider the following Groovy code:

class Test {
  def invokeMethod(String name, args) {
    println "called invokeMethod $name $args"
  }

  def test() {
    println 'in test'
  }
}

def t = new Test()
t.test()
t.bogus('testing!',1,2,3)

 

This code prints out the following text, when run in groovyConsole:

in test
called invokeMethod bogus [testing!, 1, 2, 3]

the first call goes to the test method, as usual.  The second call, instead of resulting in a MissingMethodException, will route through the invokeMethod method.

This can be handy for all kinds of things, but the use I’ve put it to is as a dispatcher, routing calls to an underlying class or set of classes – here’s an example:

 

class One {
  static def test1() {
    println 'in test1'
  }
}

class Test {
  def invokeMethod(String name, args) {
    if (One.metaClass.respondsTo(One, name, *args)) {
      return One."$name"(*args)
    }
  }
}

def t = new Test()
t.test1()

This prints out 

in test1

Now, this example is flagrantly using a few tricks in Groovy, so let me mention how they work…  (Feel free to skip this paragraph if you’re content with this being a magic incantation.)   The first bolded line, with the respondsTo method, will retrieve the metaClass of the One class, which acts in a similar way to the Class class, but allows additional capabilities.  The respondsTo method of MetaClass takes, as it’s first argument, the actual object (or class, for static methods) that you’re checking, the name of the method, and then the arguments.  The asterisk in front of the “*args” argument is called the “spread operator”, and will derefrerence the array of args to instead be a list of arguments placed at the end of the method call.  The second bolded line is using the String substitution capability of Groovy’s double quoted Strings (called GStrings) to dynamically execute the method called name, again passed the arguments with the spread operator.  An additional wrinkle is that respondsTo actually  returns a ArrayList – which is treated as false by Groovy if empty, and true if not.  Now, you could do all this in Java, via reflection and other code, but the pain of doing so really makes this something you should probably do in Groovy.

There’s one other case we should probably cover – if your class implements the marker interface GroovyInterceptable, it won’t just intercept methods it can’t find – it will intercept all method calls on the class, including methods that are actually there.  That unfortunately means that we can’t directly call the method from within the invokeMethod though – instead, you must use the metaClass to invoke the method, getting around that interception.  Here’s how:

class One {
  static def test() {
    println 'in One.test'
  }
}

class Test implements GroovyInterceptable {
  def firstTime = true
  def invokeMethod(String name, args) {
    if (firstTime) {
      firstTime = false
      return One."$name"(*args)
    }
    return metaClass.getMetaMethod(name,args).invoke(this,args)
  }
  def test() {
    println 'in Test.test'
  }
}

def t = new Test()
t.test()
t.test()

 

This prints out:

in One.test
in Test.test

 

We’ll leave the explanation of how all this works for another time – the MetaClass is used pretty heavily in all kinds of Metaprogramming in Groovy, and we’ll cover it pretty extensively as we go on – but it’s use should be obvious from context.  Note that unlike our other use of args, we do not need to use the spread operator.  But keep the MetaClass in mind, because when it comes to adding methods dynamically at runtime, it’s the way to go.

Next up, we’ll talk about the related topic of Property Dispatch.

Word order and Groovy Parsing

 

In my previous post on Optional Parenthesis in Groovy, I mentioned using the optional parenthesis to make natural looking chains of commands.  But I realize that I overlooked mentioning a really important limitation with this approach, namely, how Groovy will parse these parentheses lacking statements.

Consider this code:

one two three four five

Since Groovy is a dynamic language, this statement will need to figured out by Groovy at compile time with almost no available context.  Is the token one a method, or a binding variable?  That information isn’t available to the compiler, so Groovy has to guess.  Here’s how to figure out what Groovy will do:

Fire up groovyConsole from a command line which has the GDK in it’s PATH.

Type in the line above.

From the menus, select Script -> Inspect AST

In the lower pane of the frame that pops up, you’ll see what Java code is going to be generated by the Groovy parser.  Take special note of the run() method – that’s the part that actually contains the code we typed in, turned into the code below:

public java.lang.Object run() {    
    return this.one(two).three(four).five
}

This really tells us all we need to know about how Groovy’s going to parse a String of words – it’s going to always follow the pattern:

method(param).method(param).variable

Where the first two words are going to be treated as a method name and a parameter (from a binding variable or locally scoped variable), respectively, while any odd instance will be a variable.

So, while you can use optional parentheses to construct a grammar, you should be aware that there are significant restrictions on how you can do so.

Laforge on DSLs in Groovy

Groovy expert Guillaume Laforge has posted on DSLs in Groovy, using lots of neat Groovy tricks to make the language sing. If you’re curious, check it out. I’ll probably be covering some of his more interesting tricks later in these pages. (At the rate I’m going, much, much later.)

Using optional parenthesis in Groovy to your advantage

Now that we’ve discussed adding default methods to your scripts with a BaseScript, it’s time to talk about one way to hide the fact that you’re calling a method:   Groovy allows you to omit parenthesis in cases where there is a parameter, and there is no ambiguity in use.

We can use this optional parens feature of Groovy to create more natural looking DSLs, and with judicious use of method chaining, we can create a simple grammar with very little code.  Let’s say that I want to create a simple DSL that moves a marker around in a confined space, and that the language should look similar to Logo, with a series of commands all in a line.  Since I don’t enjoy trigonometry,  I’ll forgo the usual two vector (x,y) space of Logo for a simpler single line.  That means than instead of needing to allow for commands like “forward 10 right 90 forward 6″, I can just use two verbs – “forward” and “back”.  We’ll add one more, “traveled”, to report total progress.  So, to define our DSL in a simple manner, I want to accept commands like “forward 10 forward 6 back 3 traveled”.

So, we can start by creating a simple BaseScript with the three verbs (let’s do it this time in Groovy, I’m feeling lazy):

 

abstract class BaseScript extends Script {

  int distance = 0;
    
  def forward(int i) {
    distance += i
    println "moved forward "+i
    return this
  }
    
  def back(int i) {
    distance -= i;
    println "moved back "+i
    return this
  }
    
  def traveled() {
    println "total distance traveled " + distance
    return this
  }

}

 

The script is storing the distance as an instance variable, and then all the methods are acting on that variable in some way.  Also, note that each method returns the current script object, enabling method chaining.

Once this is set up, we can use the following code to execute a script:

  CompilerConfiguration config = new CompilerConfiguration();
  config.setScriptBaseClass("groovybasescriptlogo.BaseScript");
  GroovyShell shell = new GroovyShell(config);
  Script script = shell.parse("forward 10 back 3 forward 5 traveled()");
  script.run();

 

The output when we run this is as you’d expect:

moved forward 10
moved back 3
moved forward 5
total distance traveled 12

 

But note there’s a catch here:  because traveled is a method without a parameter, there still needs to be a set of parenthesis at the end.  The reason why is also a clue as to how we can fix it.  Without the parenthesis, the traveled keyword will be interpreted as a binding variable instead of a method.  Since we’ve already covered how to handle Binding variables, we can change the code by using a Binding class.  First, eliminate the first bolded line in the BaseScript – that changes the distance variable from an instance variable on the Script class to a Binding variable.  You can also remove the traveled() method from the BaseScript as well, or leave it in there, to cover the case of using a parenthesis.  Then create a Binding class like so:

 

public class SimpleBinding extends Binding {
    
  int distance = 0;
  Script script;
    
  public SimpleBinding(Script script) {
    super();
    this.script = script;
  }

  public Object getVariable(String name) {
    if ("distance".equals(name)) {
      return distance;
    }
    if ("traveled".equals(name)) {
      System.out.println("total distance traveled " + distance);
      return script;
    }
    return super.getVariable(name);
    }
        
  public void setVariable(String name, Object value) {
    if ("distance".equals(name)) {
      distance = (Integer)value;
      return;
    }
    super.setVariable(name, value);
  }
    
}

 

So, the distance variable moves from the BaseScript to the Binding.  In the setVariable, add the ability to set the distance variable.  That covers the usage in the BaseScript.  We’ve also moved code around traveled to the Binding as well:  Now, when traveled is used without a parenthesis, it will go to the binding, print out the message, and still return the Script object (which is passed into the Binding at creation).  To use this code, just modify the calling code like so:

 

  CompilerConfiguration config = new CompilerConfiguration();
  config.setScriptBaseClass("groovybasescriptlogo.BaseScript");
  GroovyShell shell = new GroovyShell(config);
  Script script = shell.parse("forward 10 back 3 forward 5 traveled");
  script.setBinding(new SimpleBinding(script));
  script.run();

 

The output remains the same.  Doing this now eliminates the need for the trailing parenthesis on the traveled keyword.  That concludes this topic – feel free to ask any questions, below – I know this example is a little more involved than previous ones, but the concepts are still pretty simple, even if there’s more code on the page.  The next posting will cover a different way to handle adding verbs dynamically, via the invokeMethod mechanism.

 

Adding default methods with a Groovy BaseScript

Having talked about nouns in our DSL by adding new variables from the Binding, it’s time to talk a little bit about verbs – there’s a number of ways to do this in Groovy, so let’s start by looking at a BaseScript for Scripts.

A BaseScript is, effectively, the super class for the Script that’s executed dynamically from GroovyShell.    To use a BaseScript, create an abstract class that extends Script, implementing the methods you want to enable in your scripts.  Here’s a very simple example:

package basescripttest;

import groovy.lang.Script;

abstract public class BaseScript extends Script {
    void hello(String name) {
        System.out.println("hello "+name);
    }
}

This BaseScript will provide a single method to your script: hello – taking a single argument, a String.  Note that you could just as easily use a Groovy class for the same purpose:

package basescripttest

abstract class BaseScript extends Script{
    def hello(name) {
        println "hello $name"
    }
}

Which does the same thing.  To use it in a program, use the CompilerConfiguration class, which allows various modifications of how the GroovyShell parser acts at runtime.  Note that you need to pass the fully qualified name of the class as a String to the setScriptBaseClass.

CompilerConfiguration config = new CompilerConfiguration();
config.setScriptBaseClass("basescripttest.BaseScript");
GroovyShell shell = new GroovyShell(config);
Script script = shell.parse("hello('Jim')");
script.run();

 

So, to recap, we’re executing the following script:

hello('Jim')

Which is outputting the following text to stdout:

Jim

In this example, we just use the (now default provided) hello function in the BaseScript from within the provided class. This is a very simple example – we’ll go into more detail next time.