I really like Jasmine as a testing framework — I’ve converted my last two teams to using it and I really like how easy and expressive it makes testing. However, I have some very minor pain points around using Jasmine for larger codebases that include a lot of little objects with very isolated behaviors. jasmine-mocks aims to address issues that arise when writing Jasmine specifications for this kind of codebase.

Creating Mock Objects From Prototypical Classes

jasmine-mocks allows you to easily create a mock instance from a prototypical class (defining a constructor as a function and methods as functions on the constructor’s prototype). This removes the need for you to explicitly define functions on mock objects.

var Dog = function () {};
Dog.prototype.bark = function () {
 alert('bark!');
};

describe('DogTricks', function () {
  var mockDog = mock(Dog);
  var tricks = new DogTricks(mockDog);
  tricks.whistle();

  expect(mockDog.bark).toHaveBeenCalled();
});

Mocks That Can Emit Events

JavaScript frameworks allow firing events on arbitrary objects (two examples: Node’s EventEmitter and Closure’s EventTarget). jasmine-mocks allows you to make mock objects that inherit real behavior to allow testing these objects without instantiating a real class.

var Dog = function () {};
Dog.prototype = Object.create(EventEmitter.prototype);
Dog.prototype.giveBiscuit = function () {
  ++this.numBiscuits;
};

describe('DogTricks', function () {
  var mockDog = mock(Dog, EventEmitter);
  var tricks = new DogTricks(mockDog);

  mockDog.emit('roll over');

  expect(mockDog.giveBiscuit).toHaveBeenCalled();
});

Basic Matching Functionality

Spies are great, but they don’t allow you to be certain you are passing the right values to them. If you want to make sure that your Jasmine spies are being called with the right arguments you need to add extra asserts, something that shouldn’t be necessary.

jasmine-mocks allows you to easily create matchers that give your spies different functionality when different arguments are passed in. The main benefit to this is not returning good values when bad arguments are passed in.

describe('DogFeeder', function () {
  var mockDog = mock(Dog, EventEmitter);

  when(mockDog.likesFood).isCalledWith('biscuit').thenReturn(true);
  when(mockDog.likesFood).isCalledWith('kibble').thenReturn(false);

  var dogFeeder = new DogFeeder(mockDog);
  dogFeeder.addFood('biscuit);
  dogFeeder.addFood('kibble);
  dogFeeder.feed();

  expect(mockDog.feed).toHaveBeenCalledWith('biscuit');
});

The source code is available on Github. I hope that you find it useful!

This project is a standard ColdFusion backend (CF8) that sits in front of a SQL Server database. Rows are fetched from the database and displayed in a JQuery DataTable widget — one of the main reasons of using DataTables is the Excel export.

Sometimes we fetch a lot of data to export (on the order of 5MB or more). This is transferred back to the browser as JSON. For most of our requests to the server, we use CFAjaxProxy to instantiate JavaScript objects that make requests to the server on method invocation:

<cfajaxproxy cfc="package.model.BackendService" jsclassname="BackendService"></code>

<script type="text/javascript">
  var service = new BackendService();
  service.fetchAllTheData();
</script>

The specific issue I was having was that Firefox died while parsing the returned data with a console error in cfajax.js. Digging into I found that the issue was related to how the browser escapes returned JSON data.

I replaced the method invocation with a call to JQuery’s JSON handler:

  proxy = new BackendService();
  // OLD CODE
  proxy.fetchAllTheData(arg1, arg2, arg3);

  // NEW CODE
  $.getJSON("/package/model/BackendService.cfc",
    {
      method: "fetchAllTheData",
      returnFormat: "json",
      argumentCollection: JSON.stringify(
        {
          argName1: arg1,
          argName2: arg2,
          argName3: arg3
        }
      ),
      _cf_clientid: window._cf_clientid
    },
    function (data) {
      proxy.callbackHandler(data);
    });

This had the happy effects of fixing the bug (always good) while using a more mainstream technology (JQuery 1.6+ over years-old CF8 ajax.js) for browser/server interaction.

What’s a MUSH, and Why MUSHCode?

On a MUSH, the game world is populated by objects, and each object has a unique reference number (the dbref). Objects have attributes, which can contain either data or code. Certain attributes can listen for commands entered by the player, which triggers code execution. For example, many MUSHes have a +who command, showing logged in players along with handy information (time logged in, location, what they are doing).

Wizards have the permissions to create objects and change any attributes on any object, anywhere. In practice, Wizards build the systems that power the MUSH’s roleplaying through writing MUSHCode. MUSHCode is a layer on top of the PennMUSH server that provides things like background approval, roleplaying nominations, adjucated combat — all the things necessary to take a tabletop game online.

Why is MUSHCode Interesting

Code is the same as data — data is stored on objects side-by-side with code. We are already very close to Lisp!

There are two main code types of MUSHCode: commands and functions. Commands are the only way to start code execution, and these are very powerful — some fundamental things are only available as commands (namely, destruction of an object). Functions are much more versitile and in general mode of your MUSHCode is made
up of functions.

Here’s an example of a function:

th iter(1|2|3|4,add(##,1),|)
2 3 4 5

Here’s an example of a command:

@pemit me=Hi there!
Hi there!

Some commands and functions have overlapping functionality but during my time writing MUSHCode I found functions much more useful. When it was time to reach for behavior that was not exposed in a function, it was time to use a command — otherwise commands were code entry points and functions would end up doing most of the work.

Javelin, uberdev of PennMUSH, wrote a guide that talks about a lot of these things, and adds a lot more technical detail.

Functions have a stack limit — you can’t do too much work in a function or you’d run into errors. We ran into this a lot building our Star Wars MUSH.

The Greatest Game Never Released

I spent something like a year working, on and off, on a Star Wars MUSH that was never released. I think this happens a lot in MUSH-land — people have high ambitions about the kind of games that should be built. Some games only work once you have a critical mass of code and I fear our concept was a little high on that front. What we did build
seems amazing years later.

• Virtualized object system that allowed users to buy/trade/create fake objects for roleplaying purposes (mostly weapons — lightsabers, blasters)
• Mostly complete space combat system implementing most of the Star Wars d20 space system — turn-based space combat including movement, damage, collisions, missiles
• Viewscreens that linked ships and emitted events back and forth between the two
• Character generation and d20 roleplaying implementation
• Puppet system for Game Masters to create and command objects as puppets
• All the usual MUSH CRUD applications for opening/closing/commenting on issues

We apparently even wrote a bug database for keeping track of all the problems with the code I wrote!

Example Code

Here’s the code that powered interstellar transport. This is a pretty simple object relatively — a character goes to a location, sees that there is a transport node, and uses this object to teleport from one planet to another (while showing them the amount of time that this would take in-game to discourage planet-hopping for the sake of it).

PARENT: Intergalactic Transportation Node(#690TWn)
Type: THING Flags: WIZARD NO_COMMAND
An intergalactic transport node.  See [ansi(hy,+help transports)] for information as to their use.
Owner: Pozzo(#270POWwerACMcJ)  Zone: *NOTHING*  Building Chips: 10
Parent: *NOTHING*
Powers: 
Channels: SpaceMonitor
Warnings checked: 
Created: Thu May 20 20:53:07 2004
Last Modification: Sun Mar 18 17:53:42 2007
CMD_+TRANSPORT [#270R]: $^\+transport (.+)$:@switch [u(fn_cmd.+transport,%1)]=0,{th [syspemit(%#,%qz)]},1,{th [u(fn_cmd.do_+transport)]}
CMD_+TRANSPORT_LIST [#270R]: $^\+transport\/list$:@pemit %#=[u(fn_cmd.+transport_list)]
FN_CMD.DO_+TRANSPORT [#270]: [syspemit(%#,You take [setr(g,a transport from [ansi(hy,[name(loc(%!))])] to [ansi(hy,[name(%qd)])].  The trip takes [setq(h,u(fn_get_travel_time_between_systems,u(zone(loc(%!))/fn_get_system),u(zone(squish(%qd))/fn_get_system)))][iter(timestring(mul(%qh,60,60)),if(and(f(strmatch(##,0*)),f(strmatch(##,*m)),f(strmatch(##,*s))),##),,)] of IC time)]; please RP accordingly.)][tel(%#,%qd)][remit(loc(%!),[name(%#)] leaves on a transport.)][remit(%qd,[name(%#)] arrives on a transport.)][cemit(spacemonitor,%n takes %qg.,1)]
FN_CMD.+TRANSPORT [#270]: [setq(z,[setq(d,iter(u(fn_get_other_nodes),if(strmatch([squish(u(fn_get_name,##))],%0*),##),,))][if(f(%qd),%0 does not match any transport node.,if(gt(words(%qd),2),%0 matches more than one possible transport node; please be more specific.))])][f(%qz)]
FN_CMD.+TRANSPORT_LIST [#270]: [header(Transport Node: [u(get_name)],,hw,,b)]%r[ljust(ansi(g,Destination:),30)][ljust(ansi(g,Destination),15)][ansi(g,Travel Time (IC hours))]%r[iter(setr(l,iter(lattr(v(trans_db)/*_nodes),remove(xget(v(trans_db),##),loc(%!)),%b)),[ljust(u(fn_get_name,##),30)][ljust(if(isdbref(setr(g,u(zone(##)/fn_get_system))),u(%qg/get_name),%qg),15)][u(fn_get_travel_time_between_systems,u(zone(loc(%!))/fn_get_system),u(zone(##)/fn_get_system))],,%r)]%r[header(,[setr(w,words(%ql))] Destination[if(gt(words(%ql),1),s)],,hw,b)]
FN_GET_NAME [#270]: [setq(a,squish(rest(name(%0),:)))][if(hasattr(%0,display_name),xget(%0,display_name),if(strmatch(name(%0),*:*),%qa,if(strmatch(name(%0),*--*),after(name(%0),--%b),name(%0))))]
FN_GET_OTHER_NODES [#270]: setr(l,iter(lattr(v(trans_db)/*_nodes),remove(xget(v(trans_db),##),loc(%!))))
FN_GET_REGION_FOR_SYSTEM [#270]: [extract(grab(xget(v(hyper_obj),system_region_mapping),[edit(%0,%b,_)]|*),2,1,|)]
FN_GET_REGION_NUMBER_FOR_SYSTEM [#270]: [member(xget(v(hyper_obj),space_regions),extract(grab(xget(v(hyper_obj),system_region_mapping),[edit(%0,%b,_)]|*),2,1,|))]
FN_GET_SYSTEM [#270]: [if(hasattr(loc(loc(%0)),wing),loc(loc(loc(%0))),if(xget(loc(%0),ship),loc(loc(%0)),ulocal(zone(loc(%0))/fn_get_system)))]
FN_GET_TRAVEL_TIME_BETWEEN_SYSTEMS [#270]: [setq(0,if(isdbref(%0),squish(rest(name(%0),:)),%0))][setq(1,if(isdbref(%1),squish(rest(name(%1),:)),%1))][mul(extract(extract(xget(v(hyper_obj),space_travel_times),ulocal(fn_get_region_number_for_system,%q0),1,|),ulocal(fn_get_region_number_for_system,%q1),1),2)]
GET_NAME [#270]: [name(loc(%!))]
HYPER_OBJ [#270]: #405
TRANS_DB [#270]: #691

• CMD_+TRANSPORT is the entry point. You type in +transport LOCATION to be transported to a location (specified by name).
• FN_CMD.+TRANSPORT does validation: take the list of other transport nodes. Get their names. Make sure that the string that the user typed matches one and only one of them.
• FN_CMD.DO_+TRANSPORT performs the action. It’s worth examining this one more closely.

[syspemit(%#,
 You take [setr(g,a transport from [ansi(hy,[name(loc(%!))])] 
 to [ansi(hy,[name(%qd)])]. The trip takes
 [setq(h,u(fn_get_travel_time_between_systems,
         u(zone(loc(%!))/fn_get_system), 
         u(zone(squish(%qd))/fn_get_system)))]

 [iter(timestring(mul(%qh,60,60)),
    if(and(f(strmatch(##,0*)), 
           f(strmatch(##,*m)),
           f(strmatch(##,*s))),##),,)] of IC time)]; please RP accordingly.)]

[tel(%#,%qd)]
[remit(loc(%!),[name(%#)] leaves on a transport.)]
[remit(%qd,[name(%#)] arrives on a transport.)]
[cemit(spacemonitor,%n takes %qg.,1)]

• setr sets a local variable and returns that value for later use. There doesn’t appear to be a reason to save the name of the leaving location here — looks like dead code!
• setq sets a local variable and returns nothing. Here we set register ‘h’ to the travel time (in hours) between the two systems. This is mostly used to make the code more ‘readable’ by breaking out separate concepts into their own computation.
• The travel time between systems is computed by calling the fn_get_travel_time_between_systems on the ‘systems’ where the different transport nodes are located. This is an involved computation that I won’t get into.
• We multiply %qh by 60 minutes and 60 seconds to get the number of seconds of travel time. (Needed to pass to timestring, which returns a nicely formatted list of days, hours, minutes, and seconds from a num of seconds.)
• The iter over a timestring is just plucking the ‘hours’ out of a string of the format 0d 8h 47m 21s in a particularly awful way.
• tel actually teleports the player to the other location.
• remit sends a message to the former and current location of the players.
• cemit sends a message to a channel.

Here setq forms the let-mechanism of the language. iter forms the standard mapping function, with the argument passed in being code that takes an argument of ## — a lambda!

Good Things

MUSHCode made me a much better programmer, and I didn’t even notice it was happening while it was being done. While I was working during the day on my coursework in graduate school and undergraduate, in the evenings and weekends I would hack away on building the systems to power this MUSH.

The everything-is-just-a-string mentality of MUSHCode made adapting to functional languages a breeze — in the iter() example above, the second argument is code that will be executed as the list of iterated through, substituting ## for the current argument.

I came up with a code style that was my own that still looks pretty clean today. The work that I did was also a huge help in breaking down product requirements and developing a technical plan — the kind of thing that we do every day as software developers.

The thing that I find so amazing looking back is how clearly learning this ‘language’ influenced my development as a programmer … even while I didn’t think I was doing anything of the kind!

When developing in dynamic languages like Python and Ruby, it is even more important to provide a comprehensive unit test suite describing the low-level contract of the system than usual. You have no compiler step to save you from yourself — the interpreter assumes that you know what you’re doing, leading you to rely on tests more than ever.

I’m going to talk about my favorite way to write tests for dynamic languages. I think it hits a sweet spot that is viscerally satisfying — a sweet spot that is only possible because of the absence of a compilation phase.

In JavaScript

One of the coolest features of JavaScript is that, because everything is in the browser, you can tie a unit test framework into a web page. The one I’m most familiar with is Jasmine. In writing Jasmine tests, you slowly grow a Spec Runner, an HTML file that points to your source files (Calculator.js) and your JavaScript specification files (CalculatorSpec.js).

The speed at which Jasmine specifications run is extremely satisfying, making the traditional TDD cycle go very fast.

• Write a test
• Reload browser, see failure (RED BAR)
• Write the functionality for that test
• Reload browser, see success (GREEN BAR)
• Refactor, check that all specifications still succeed

In other languages, I can end up impatient with my compilers (especially when using a hog like Maven or when changing one header file causes an entire C++ project to be recompiled) and will sometimes delay writing tests knowing that it will take a ridiculous amount of time to see even the red bar.

Why the Browser?

I think that my generation has been conditioned to interact with the world through the browser. Check your favorite news site/RSS feed, read through all the articles — when you run out, you want to see if there are more … refresh the browser.

Reloading the browser to see what’s new has a powerful psychological effect. After years of web browsing I have been conditioned to refresh the browser to see what new goodies await.

When I see a failure in a browser-based testing environment, I want to fix it. I will always have a much deeper emotional commitment to typing the ‘reload’ key combination on my browser than I ever will to JUnit output.

In Python

I’ve just finished up work on the initial version of Greenbar, a server that runs in the browser and runs nosetests on the specified directory every time you reload the page.

Here is a failed test:

Here is a successful test:

I hope that Greenbar is useful — I will definitely be trying using it for my next Python project.

If you haven’t tried out browser-based TDD, you are missing out — the rapid cycle makes developing extremely satisfying and TDD generates a test suite that allows for you to refactor your code without fear of breaking anything.

Other Similar Projects

• Jasmine (JavaScript)
• PHPUnit Test Report (PHP)
• MXUnit (ColdFusion)

The web revolution occurred without me. I only started writing web applications a little over a year ago (PHP, then Python, then Ruby, then .NET MVC and Java). I still have a lot to learn on how to build a web application, but I think I understand the basics now.

If you’re like me and have been fundamentally a ‘backend’ programmer most of your life — if you’ve programmed more algorithms than applications and haven’t ventured into the web application world, you have probably missed out on a vital piece of the new technology stack.

It’s time for you to learn JavaScript.

JavaScript Is Everywhere

If you write an application in JavaScript, everyone can use it — technical, non-technical, developer or consumer. Fully featured web applications are the direction that the web is going towards, and JavaScript is starting to invade the server through Rhino, SpiderMonkey and Node. (Also, CommonJS is attempting to define a non-browser API.)

It’s not as obvious that JavaScript has any advantage on the server, but if a standard for file/socket interaction that can run on both the server and the client catches on, watch out: languages don’t necessarily win because of technical merit, they win through inertia. Running different parts on your application in different places on the same technology is a powerful argument for a technology stack.

JavaScript Is Just Like Any Other Language

You get to take with you all of the skills you’ve learned from developing in other languages. Are you a Java developer? JavaScript has some familiar syntax and some basic object-oriented boilerplate will help you along until you’re ready to tackle exactly what a ‘prototype’ is. Are you a functional programmer? JavaScript gives you first class closures. Libraries like Underscore allow you to use functional programming concepts in your applications. (sadly, no macros yet, though).

JavaScript Is Getting Better

Libraries like JQuery, Prototype, and Google Closure Library abstract the browser-specific ugliness of the Document-Object-Model API. Doing a jQuery.ajax means you don’t have to wonder if XMLHttpRequest is defined or not.

Libraries like Google’s V8 are working to bring the speed of virtual machines (like Java’s JVM) into the browser. Faster client applications means a more responsive GMail, Google Docs, Google Plus.

Developers are even starting to step in where the ECMA standards body is not, by providing tools for common language features like dependency management (RequireJS, Sprockets) and static type-checking (Closure Compiler).

There is a ‘good JavaScript’ movement determined to write better JavaScript code in the same way that unit testing/clean code has aspired to make developers write better C++/Java/C# code.

JavaScript faces a number of interesting challenges in the path to writing better code: inconsistent DOM API, lots of applications are basically GUIs (difficult to test in the first place), entrenched programmer beliefs about JavaScript not being a ‘real’ application.

It’s Easy To Learn

Everyone already has a read-eval-print loop for playing around with JavaScript syntax — it’s their browser! Google Chrome lets you turn on developer tools, go straight to the console and start hacking out JavaScript code. There are also many in-browser JavaScript development environments available for you to choose from to get immediate feedback.

There are a number of great resources available to help you learn JavaScript the Language, not simply JavaScript the browser toy. Right now my favorite is Eloquent JavaScript — it introduces the language by itself without appealing to any libraries like JQuery [1]. I especially like the Object Oriented Programming chapter — it makes it very clear how to build objects in a way that’s familiar from my years and years of Java/C++ programming.

Once that’s finished, I like JavaScript: the Good Parts and the JavaScript Garden for rounding out the parts of the language that are a little more foreign to me.

Very Flexible For Testing

Because anything can be redefined at runtime, it’s easier to create seams to test poorly architected code. At work, we’ve redefined the jQuery.ajax function to test parts of our code that interacts with the server from the client.

Jasmine is my testing framework of choice for JavaScript (in part because that’s what I used first). As a developer, the really satisfying thing for me in writing JavaScript tests is how fast they take to run. Control-Shift-R to reload the browser, all tests ran in 0.075 seconds. This makes doing TDD for JavaScript really easy (easy enough so that I’ll remember to do it!).

Conclusion

I’m not a JavaScript expert yet, but I’m trying to get better. If you’re also looking to improve I really recommend that your next project be in JavaScript. It doesn’t have to be a web application — it can be anything. It might not even talk to the browser at all.

JavaScript is just a language like any other. You still need to write tests. You still need to write small methods, small objects, and make sure you have seams for testing through dependency injection. Prototype-based inheritance is a little strange at first, but it’s not really too hard once you take some time to see what it’s doing.

As developers, we need to take all the great things we’ve learned about agile development and apply them to JavaScript development. Having a common platform in the browser and an expanding list of full-featured JavaScript applications means that JavaScript has already won.

[1] I like JQuery a lot but it’s very important to understand what in your JavaScript code is JavaScript, and what is JQuery. JQuery is just too useful for Getting Things Done — this blurs the lines and sometimes makes me forget, yes, JavaScript has for loops!

A major code smell is using IEEE 754 floating point numbers when an accurate representation is required. Most tasks I deal with in programming customer-facing webapp require accurate representation of units and I believe most programmers are not clear on what the built-in floating point types for a language actually provide.

Doubles Do Not Store Precise Values

Let’s check Wikipedia. A 64 bit floating point number is represented in the computer by:

• 1 sign bit
• 11 exponent bits
• 53 ‘signficands’

You get 53 bits of precision, stored in a binary format. Specifically, if \(e\) is the exponent, \(s\) is the sign, and \(b_0, \ldots, b_{53}\) are the bits of the significand, the number \(d\) is stored as

$$d = (-1)^{s} * 1.b_0 b_1 b_2 \ldots b_{53} * 2^{e-1023}$$

This is great if the number you are storing is expressable as a whole number or a finite sum of binary fractions. Unfortunately, most numbers do not satisfy this. In fact, pretty much any number doesn’t satisfy this.

Let’s check out the Clojure REPL to see what’s going on underneath the surface of java.lang.Double.

user=> (Double/toHexString 2.0)
"0x1.0p1"
user=> (Double/toHexString 0.5)
"0x1.0p-1"
user=> (Double/toHexString 0.2)
"0x1.999999999999ap-3"
user=> (Double/toHexString 0.3)
"0x1.3333333333333p-2"

The numbers \(0.2\) and \(0.3\) have inexact representations on the machine level because they are not expressable as a finite sum of powers of 2. Notice that the powers of 2 (\(2.0\) and \(0.5\)) do just fine.

Don’t Cast Doubles to Floats

This just makes a bad decision worse — you are truncating away significant digits by converting from a 64 bit representation into a 32 bit representation.

Never, ever, ever use Floats or Doubles for Money

Money needs to be precise. IEEE 754 digits were never intended to serve as precise values. Every language has an arbitrary precision arithmetic library. Use that library instead.

Still bad is using floats and doubles in measuring things that result in a payment, or measure a payment. Prefer integer/long values for this when possible, measured in atoms — the smallest unit of measurement in your system (bytes of bandwidth in, teraflops, compute cycles, whatever).

If you are writing code that involves money, never ever use floats or doubles.

Do the Right Thing

Racket has the concept of an exact (arbitary precision) and inexact (IEEE 754) number. Unfortunately exact numbers can become inexact numbers when involved with certain arithmetic operations.

For Java, use BigDecimal. For Ruby, use BigDecimal. For Haskell, use Data.Decimal. For C++, consider GMP.

However, XML/JSON is just a document. DSLs (domain-specific languages) are programmable documents. In Java and C#, libraries like Fluent NHibernate, Mockito, and JMock wrap library interaction as a psuedo-DSL, though still recognizably in their original language. In Ruby, Rails and Chef allow for specifications that seem completely divorced from their original language.

Clojure is a LISP port for the JVM and CLR. In LISP, macros allow for completely reprogrammable syntax, but they are one of the more advanced features of the language involving some more esoteric syntax. (In honesty usually I don’t feel like I need to change language syntax for my programs!)

Example Application: Restmock

Restmock is a server for serving mostly static content, indexed by a configuration file. The original reason for it was during the development of a frontend for a REST backend. During the development lifecycle for a frontend application, there is a lot of minor work that doesn’t a semantically rich backend and Restmock was developed in order to allow programmers to move faster during these tasks.

Serving static content is not a difficult task and is easily accomplished with many microframeworks [2] in a lot of different language. I wanted a flexible application that could have a web ‘core’ for which would not change often, but which could change its behavior based solely on a external configuration file. Specifically, I didn’t want to have to change any code in the event that I needed to add another document.

Why Clojure? At work, my team programs primarily in Java. By bundling a server together with a configuration (with lein uberjar [3]), I could distribute a mock service backend that would work out of the box, without any need to install a new compiler or interpreter.

Configuration Take 1: XML

My first take on how to configure Restmock was to provide an XML file programmatically specifying routes.

<route>
  <id>can retrieve all the kittens</id>
  <request>
    <path>/kittens</path>
    <method>:get</method>
  </request>
  <response>
    <type>text</type>
    <config>
      <text>Some adorable kittens!</text>
    </config>
  </response>
</route>
<route>
  <id>can't make a new kitten</id>
  <request>
    <path>/kittens</path>
    <method>:post</method>
  </request>
  <response>
    <type>status</type>
    <config>
      <status>422</status>
    </config>
  </response>
</route>
<route>
  <id>can update a kitten</id>
  <request>
    <path>/kittens/([0-9]+)</path>
    <method>:put</method>
  </request>
  <response>
    <type>status</type>
    <config>
      <status>202</status>
    </config>
  </response>
</route>
<route>
  <id>xml representation of a kitten</id>
  <request>
    <path>/kittens/([0-9]+)</path>
    <method>:get</method>
  </request>
  <response>
    <type>file</type>
    <config>
      <path>cute-kitten.xml</path>
    </config>
  </response>
</route>

XML-phobics may cringe, but really this is fine. It let me achieve my initial goal to distribute a useful server to other team members. However, it is very inflexible. There is no way to handle server state; you will always return the same file on the same request. There is no way to test that your backend is doing the right thing with regards to PUT, POSTs, DELETEs without checking the logs.

Configuration Take 2: Clojure Macros

What we are really doing in the above XML is building a server. There are a few nouns used to build this server:

• Request criteria: Does an incoming request match this route?
• Response: What should I respond if a request matches my request criteria?

Requests, responses, and the specific types of request criteria and responses end up being the new syntax of our server specification using a Clojure DSL. The equivalent specification of the above XML file in a macro-based DSL looks like:

(routes
 (route "Can retrieve all the kittens"
        (request (uri "/kittens")
                 (method :get))
        (response (text "Some adorable kittens!")))
 (route "Can't make a new kitten"
        (request (uri "/kittens")
                 (method :post))
        (response (status 422)))
 (route "Can update a kitten"
        (request (uri "/kittens/([0-9]+)")
                 (method :put))
        (response (status 202)))
 (route "Kitten XML"
        (request (uri "/kittens/([0-9]+)"))
        (response (xml-file "cute-kitten.xml"))))

What should this look like when it is converted into code?

• Request criteria should be a function that takes a request object and returns either true or false.
• Responses should either be HTTP response objects or functions from request objects to HTTP response objects (in case we want to change how responses look for similar requests, i.e. return the ID as part of the XML for a request to kittens/1).
• A collection of routes tries each route’s request criteria until one is matched, and then returns the response associated with that route.

Request Criteria Macros

(defmacro uri
  "Specifies a criteria of matching a URI"
  [path]
  `(fn [req#]
     (if (nil? (:uri req#))
       false
       (not (nil? (re-matches (re-pattern ~path)
                                     (:uri req#)))))))

(defmacro method
  "Specifies a criteria of matching a HTTP request's method"
  [method]
  `(fn [req#]
     (= ~method (:request-method req#))))

(defmacro request
  "Specifies a list of criteria to match a request on"
  [& criteria]
  `(fn [req#]
     (reduce #(and %1 %2)
             (map #(% req#)
                  (list ~@criteria)))))

In the above, there is some unusual syntax associated with making a macro:

• The ` at the start means to replace the macro with literally the following syntactic form.
• ~ means to evaluate a symbol.
• req# tells the interpreter to always expand req to a different variable name in different locations to prevent namespace collisions.
• ~@criteria means to take all the arguments represented by the symbol criteria and change them from a list back to a set of arguments.

The request macro first converts each of its criteria into a function from requests to booleans and returns a function from requests to booleans that requires all request criteria to be true. (It may look a little complicated, but it is “just” a left fold over the boolean values of applying the mapping function to each request!)

Response Macros

In contract to request criteria macros, response macros are very straightforward.

(defmacro response
  "Specifies a response handler"
  [handler]
  `(fn [req#] (~handler req#)))

(defmacro text
  "Specifies a text response handler"
  [text]
  `(text-handler ~text))

(defmacro xml-file
  "Specifies a xml file handler"
  [file]
  `(xml-handler ~file))

text-handler and xml-handler are functions that just wrap the HTTP request with the appropriate headers, set Content-Type, set an appropriate status, etc.

The Routes Macro

(defn route-handler [req]
  (matching-uri-handler
   (list default-route-handler)
   req))

(defmacro routes
  "A routes is a collection of route handlers"
  [& routes]
  `(defn route-handler [req#]
     (matching-uri-handler (list ~@routes) req#)))

Wait, what’s going on here? The routes macro is defining a name, and not a generic name — a very specific name, route-handler, that already has a definition.

The reason we define route-handler is so that the interpreter knows it is a valid symbol. The routes macro then redefines that symbol to be that of our server specification. This allows the web server to use the route-handler as the function from requests to responses.

Loading the DSL has a trick to it because of how Clojure namespaces interact:

(defn load-restmock-config [file]
  (do
    (binding [*ns* (find-ns 'restmock.dsl)]
      (load-file file))))

Clojure doesn’t let macros from one namespace define symbols in another. Therefore, in order to use our route-handler trick, we need to, when we load the DSL as a config file, locally bind the namespace to be the one that route-handler is defined in. This is a little technical but not too ugly.

Conclusion

This is the same configuration, but it is more flexible. Shared criteria can be defined once outside of the routes declarator and used within the specification. You can write stateful interactions and criterias (more on this coming later!). The good things about the XML configuration approach are still around: structured configuration that is external to the application.

Macros are an advanced feature of Lisp-like languages, but it is worth taking the time to wade through some more obscure syntax in order to understand when they are the correct solution to a problem.

Notes

[1] And yes, sometimes that configuration is in XML.
[2] Sinatra (Ruby), Tornado (Python), Dancer (Perl), Ring (Clojure)
[3] https://github.com/technomancy/leiningen

Throughout this post I will use the running example of evaluating an abstract syntax tree for an imperative language. Once a file has been lexed and parsed, the structure that you are left with is an abstract syntax tree, an internal data structure that encompasses what your program semantically means.

Evaluating an imperative language like JavaScript, Java, C, C++, etc involves separating out:

• expressions that evaluate to a value: 1 + 1, ptr->field.
• statements that change state or transfer control: a = 1, if (!ptr) { log_error(); }.

Statements contain expressions: right-hand-side of the assignment a = 1 is a integer literal.

JavaScript contains first class functions:

var a = function (i) {
    // anonymous function
    return i * 2;
}

The enclosing function is an expression (a value that can be assigned and passed as an argument to another function). The body of a function is a set of statements. Therefore to evaluate expressions, we must evaluate statements — and to evaluate statements, we must evaluate expressions.

This creates a mututal dependency between the two concepts. Languages handle mutual recursion differently — generally the more static checking/code generation that is done, the more annoying a concept this is to solve.

• C and C++ require forward definitions of functions
• It just works in Java as long as the two mutually recursive functions are part of the same compilation unit
• Ruby, Python, and JavaScript don’t care because they don’t check your code for undefined symbols before executing

Functional languages treat functions as values, and so in most languages, if two functions depend on each other they must be defined together. This is not always the case, but it is the case for the two main statically typed functional languages, OCaml and Haskell.

I believe that most of the time static checking is the right thing and so when we hit these kind of obstacles we should try our best to write code in the correct fashion.

Take 1: Mutually Recursive Functions

A standard approach to dealing with mutually recursive concepts is to write a mutually recursive functions. Evaluation is then a series of functions:

• val evaluate_expr : state -> expression -> value * state
• val evaluate_stmt : state -> statement -> state

let rec evaluate_expr st expr = 
  match expr with
   | ExpFn (args,body) -> as_value expr
   | ExpFnCall (rator,rands) -> 
         let (ExpFn (args,body), st') = evaluate_expr st rator and
             evalled_rands = List.map evaluate_expr st' rands in
             evaluate_stmt (subst body evalled_rands args) st'
   | (* other stuff here *)
and evaluate_stmt st stmt  =
  match stmt with 
   | StmtSeq (s1,s2) -> evaluate_stmt (evaluate_stmt s1) s2
   | StmtVarDecl (var,exp) -> 
           let (val,st') = evaluate_expr exp st in
               update st' var val 
    (* more statement types etc *)

Most of the time when I have written an interpreter in OCaml this has been the pattern that I’ve used. It is straightforward and good for small language specifications.

However, as the language you are interpreting gets bigger, you have more problems. Because of the recursive nature of the above setup, all of the parsing functions must be declared in the same compilation unit. With 10-15 different types of expressions and 20 types of statements — not a ridiculous setup in most modern languages — your logic is very tied together and you will end up with some very large files.

Large files are bad and need to be avoided. We need to constantly be asking ourselves: why are these different pieces of logic in the same file/class/module?

Take 2: Mutually Recursive Modules

In a functional programming language the main unit of abstraction is a mathematical function from inputs to outputs. (Contrast this with other languages like Java/C# where the main unit of abstraction is an object — or C, where the language enforces hardly any units of abstraction at all.)

Modules group functions together and allow for information hiding. Here’s an example signature for our expression and statement evaluators:

module type Expr = 
sig
   val evaluate_expr : state -> expression -> value * state
end
 
module type Stmt =
sig
   val evaluate_stmt : state -> statement -> state
end

When modules connect with each other, the Statement evaluation module can deal with the Expression evaluate module only through a signature, rather than requiring it to know precisely which function is being invoked.

Our above code thus looks like:

module ExprEvaluator : Expr (S:Stmt) =
struct
  let evaluate_expr st expr = 
    match expr with
     | ExpFn (args,body) -> as_value expr
     | ExpFnCall (rator,rands) -> 
         let (ExpFn (args,body), st') = evaluate_expr st rator and
             evalled_rands = List.map evaluate_expr st' rands in
             S.evaluate_stmt (subst body evalled_rands args) st'
     | (* other stuff here *)
end
 
module StmtEvaluator : Stmt (E:Expr) =
struct 
  let evaluate_stmt st stmt  =
    match stmt with 
     | StmtSeq (s1,s2) -> evaluate_stmt (evaluate_stmt s1) s2
     | StmtVarDecl (var,exp) -> 
           let (val,st') = E.evaluate_expr exp st in
               update st' var val 
      (* more statement types etc *)
end
 
(* instantiate modules *)
module rec StmtEvaluatorImpl = StmtEvaluator(ExprEvaluatorImpl)
and ExprEvaluatorImpl = ExprEvaluator(StmtEvaluatorImpl)

Here ExprEvaluator and StmtEvaluator are functors: a term’s more esoteric than it actually is. A functor is simply a module (group of functions) that talks to a signature (a module interface). It’s like if you had to parameterize your Java class with the interfaces it used to make an interface Impl.

First set up your modules with the signatures that they talk to, and then parameterize your modules via applying the functors — similar to dependency injection through Spring in Java or Python.

The recursive module syntax above binds the instantiations to each other, allowing us to completely separate the expression evaluation logic from the statement evaluation logic. This is a feature since OCaml 3.07. There is an equivalent setup in Haskell.

Conclusion

I think that functional languages are very good at selling the basic problems that they solve: many operations that we end up doing are more easily expressed as list operations, monads, lazy computation. However, as they get bigger some of these more advanced language features (modules, mutual recursion) pop up and it is important to embrace and understand these concepts just as much as ‘simple’ language concepts like first class functions.

Recursive modules may not be for every problem, but there are a lot of places where they are the right way to separate out dependencies between different abstract concepts. We should use abstraction and information hiding in functional languages as much as we stick to abstraction and information hiding in object oriented languages.

I firmly believe that in order to be a good programmer you need to have been a really awful programmer first. However, you need to be a special kind of awful programmer:

• don’t listen to anyone
• don’t try to figure out why something was done the first time
• don’t write any tests
• don’t ever clean anything up, ever (putting // TODO: refactor comments everywhere gives extra demerits)
• just get it working and then ship it

Once you have done everything wrong, then the real learning can begin. This is where the benefits of a classical computer science education stand out: you have learned essentially the full stack of the computer, the whys and traceoffs of each level — at least, in theory (don’t ask me to calculate how a cache lookup happens in an M-bit N-size L2 cache).

I’ve written a lot of code that I’m not proud of that got me to where I am today. Luckily my thesis code is a special brand of awful that makes it an eternal spring of crappiness to revisit over and over again.

If I was to be punished for my programming sins, the list of offenses would definitely include:

(Not a comprehensive list)

Way too many data structures

protected Map<JLiftVarLabel, Node> variableToNodeMap;

protected MultiMap<JLiftVarLabel, JLiftVarLabel> directVariableAffectMap;
protected MultiMap<JLiftVarLabel, JLiftVarLabel> directVariableAffectingMap;

protected MultiMap<Node, Label> specialSinks;
protected Map<Node, JLiftVarLabel> nodeToVariableMap;

protected Map<JLiftVarLabel, Collection<JLiftVarLabel>> cacheVarLabelAffectMap;
protected Map<JLiftVarLabel, Collection<JLiftVarLabel>> cacheVarLabelAffectingMap;

protected Map<Node, Collection<Node>> cacheNodeAffectMap;
protected Map<Node, Collection<Node>> cacheNodeAffectingMap;

protected DFSGraph<JLiftVarLabel> varGraph;
protected DFSGraph<JLiftVarLabel> mirrorVarGraph;
protected CacheCallBack<JLiftVarLabel> callBackObject;

protected Map<JLiftVarLabel, Label> lvarMap;

private final Map<JLiftVarLabel, JLiftLabelConstraint> variableToConstraintMap;

I’m sure there’s some reason that the class needed 14 private members, but I can’t remember it any more!

Lots of commented out code

/*
private boolean isSinkConstraint(LabelConstraint c) {
    return containsPairLabel(getSinkLabelForConstraint(c));
}

private boolean isSourceConstraint(LabelConstraint c) {
    return containsPairLabel(getSourceLabelForConstraint(c));
}


private Label getSinkLabelForConstraint(LabelConstraint c) {
    // tau <= c <<-  c is the sink
    // v == tau <<-  tau is the sink
    if (c.kind() == LabelConstraint.LEQ || c.kind() == LabelConstraint.EQUAL)
        return c.rhs();
    // v ==_{def} tau ||--> tau <= v <<-- v is the sink (a define should not be a sink)
    else if (c.kind() == LabelConstraint.DEFINE)
        return c.lhs();

    return null;
}

private Label getSourceLabelForConstraint(LabelConstraint c) {
    // v == tau <<-  tau is the source
    if (c.kind() == LabelConstraint.EQUAL)  {
        return c.rhs();
    }
    // tau <= v <<-  tau is the source
    if (c.kind() == LabelConstraint.LEQ)
        return c.lhs();
    // v ==_{def} tau ||--> tau <= v <<-- tau is the source
    if (c.kind() == LabelConstraint.DEFINE)
        return c.rhs();

    return null;
}

private boolean foundPairLabel;

private boolean containsPairLabel(Label lhs) {
    foundPairLabel = false;

    LabelSubstitution ls = new LabelSubstitution() {
            @Override
                public Label substLabel(Label L) throws SemanticException {
                if (L instanceof PairLabel) {
                    foundPairLabel = true;
                }
                return L;
            }
        };

    try {
        lhs.subst(ls);
    } catch (SemanticException e) {
        throw new InternalCompilerError(e);
    }

    return foundPairLabel;
}
*/

But I might need it later! (and I probably did at some point, commenting and uncommenting things until they worked again)

What the hell are you trying to do?

public JLiftCallHelper(Label receiverLabel,
                       Receiver receiver,
                       ReferenceType calleeContainer,
                       JifProcedureInstance pi,
                       List actualArgs,
                       Position position) {
    super(receiverLabel, receiver, calleeContainer, pi, actualArgs, position);
    this.receiverLabel = receiverLabel;
    this.calleeContainer = calleeContainer;
    if (receiver instanceof Expr) {
        this.receiverExpr = (Expr)receiver;
    }
    else {
        this.receiverExpr = null;
    }
    this.actualArgs = new ArrayList(actualArgs);
    this.pi = pi;
    this.position = position;
    this.callChecked = false;

    if (pi.formalTypes().size() != actualArgs.size())
        throw new InternalCompilerError("Wrong number of args.");
}

Demerits:

• Throwing an exception from a constructor
• Incomprehensible choice of constructor arguments
• instanceof check
• setting something to null if the instanceof check fails
• boolean flag initialized to false on object creation (explicit state! agh!)
• recasting a List into an ArrayList (I guess I wanted to use this as a vector, despite it starting with no structure?)
• raw Java pre-1.5 types (ArrayList and List instead of ArrayList<> and List<>)

Really this is awful code and it’s doing less than any of the other snippets here (including the commented block).

At least the method signature problems were inherited from the superclass.

Copy and Pasted Code

public class JLiftConstructorDecl_c extends JifConstructorDecl_c {

	@Override
	public Node typeCheck(TypeChecker tc) throws SemanticException {
		if (!((JLiftTypeSystem) tc.typeSystem()).allowSmallLeaks())
			return super.typeCheck(tc);
		else {
			// HACK: copy/pasted from ConstructorDecl_c
			Context c = tc.context();
			TypeSystem ts = tc.typeSystem();

			ClassType ct = c.currentClass();

For this one I at least had an excuse. C extends B extends A; all of them implement foo. Sometimes we want C to call A’s foo, sometimes we want to call B’s foo. Unfortunately, Java’s inheritance model doesn’t support that. Solution: copy and paste some code!

Method that’s Way Way Way Too Long

let constraint_set_to_flowgraph cl lattice = 
  match cl with 
    | [] -> G.create ()
    | _  -> 
      let atoms = timeThunk (fun _ -> atoms_in_conslist cl) (SimpTimer.record_time "get atoms in constaint list") in 
      let _ = 
	begin
	  Printf.printf ("%d atoms\n") (LexpSet.cardinal atoms);
	  flush stdout
	end
      in
      let g = G.create ~size:(LexpSet.cardinal atoms) () in
      let get_sub_lv_node s lexp =
	match lexp with 
	  | (LVar (lv,t)) -> LVar (lv,s :: t) 
	  | _ -> raise (Failure "cannot get sub lv node for non-lvar")
      in
      let get_incoming_lv_node = get_sub_lv_node `Incoming and
	  get_outgoing_lv_node = get_sub_lv_node `Outgoing in
      let get_incoming_node a = 
	match a with 
	  | LVar (ExpLVar l,t) -> get_incoming_lv_node a
	  | _ -> a 
      and
	  get_outgoing_node a = 
	match a with 
	  | LVar (ExpLVar l,t) -> get_outgoing_lv_node a
	  | _ -> a
      in
      begin
	timeThunk 
	  (fun _ ->
	    LexpSet.iter 
	      (function 
		| LVar (l,t) -> 
		  (match l with 
		    | ExpLVar e -> 
		      let incoming_l = get_incoming_lv_node (LVar (l,t)) and
			  outgoing_l = get_outgoing_lv_node (LVar (l,t)) in 
		      let nodeId = id_for_lexp incoming_l in
		      let incoming_tag = GraphLabel.tagForNode incoming_l and
			  outgoing_tag = GraphLabel.tagForNode outgoing_l in
		      begin
				      (*Printf.printf ("incoming: %d, outgoing: %d\n") incoming_tag outgoing_tag;*)
			timeThunk (fun _ -> (G.add_vertex g incoming_tag;
					     G.add_vertex g outgoing_tag))
			  (SimpTimer.record_time ~silent:true "graph building (add_vertex)");
			let edgeWeight = match nodeId with 
			  | Some id ->
			    if (StringSet.mem id (!XmlConstraintReader.nvNoDeclassify)) then
			      infinity_hack
			    else
			      (match (weight_for_lexp (LVar (l,t))) with
				| None -> 1
				| Some n -> n)
			  | None -> 1
			in
			G.add_edge_e g (G.E.create incoming_tag edgeWeight outgoing_tag);
					  (*Printf.printf("\tadded vertices to graph\n");
					    flush stdout;*) ()
		      end
		    | _ -> 
		      let lvar_tag = GraphLabel.tagForNode (LVar (l,t))
		      in
		      begin
			timeThunk (fun _ -> G.add_vertex g lvar_tag) ((SimpTimer.record_time ~silent:true "graph building (add_vertex)"));
				      (*Printf.printf ("added tag for %s\n") (lexp_to_string (LVar (l,t)));
				        flush stdout;*) ()
		      end
		  )
		| Label n -> 
		  let lab_tag = GraphLabel.tagForNode (Label n)
		  in
		  G.add_vertex g lab_tag
		| Join (_,_) -> raise (Failure "graph contains non-atomic node"))
	      atoms)
	  (SimpTimer.record_time "building graph (add nodes)");
	Printf.printf ("added each atom to the graph as a node -- graph size %d nodes\n") (G.nb_vertex g);
	flush stdout;
	timeThunk 
	  (fun _ -> 
	    List.iter
	      (fun (Leq (le,rhsA)) ->
			  (* add variable connections -- note that don't care if something flows to top or from bottom! *)
		let lhsAtoms = atoms_in_lexp le in
		LexpSet.iter 
		  (fun a ->
		    let add_edge_thunk = 
		      fun _ -> 
			let e = G.E.create
			  (GraphLabel.tagForNode (get_outgoing_node a)) 
			  (infinity_hack) 
			  (GraphLabel.tagForNode (get_incoming_node rhsA))
			in
				  (*Printf.printf ("edge: (%s,%s)\n\tconstraint: %s\n") (lexp_to_string a) (lexp_to_string rhsA) (cons_to_string (Leq (le,rhsA)));*)
			G.add_edge_e g e 
		    in
		    match a with
		      | Label id -> if (LatticeGraph.is_bottom lattice id) then () else add_edge_thunk ()
		      | _ -> add_edge_thunk ())
		  lhsAtoms)
	      cl)
	  (SimpTimer.record_time "building graph (add edges)");
	Printf.printf ("added edges between each atom to the graph -- graph size %d vertices, %d edges\n") (G.nb_vertex g) (G.nb_edges g);
	flush stdout;
	g
      end

I have no idea what this function is doing either.

No Unit Tests

Nothing here because I didn’t write anything!

(I did end up with several integration-level tests that were essentially glued-together bash scripts but that will not come to my rescue here)

Being a Recovering Horrible Programmer

Well, I’d like to think the moral is that nobody’s perfect and that everyone writes bad code and that if I had to do it all over again I’d be sure to fix all the sins that I’ve documented here, do it more cleanly, etc., etc., but that’s a lie.

I had to get stuff done and I did it. The code wasn’t pretty but it worked well enough to generate publishable results.

However, all this bad code ended up being the best kind of teacher:

• I didn’t realize how important modules and small classes were until I had classes that spanned hundreds of lines
• I didn’t learn how important short methods were until I had to debug a bunch of long methods
• I didn’t learn how important regression tests were until I changed something in the morning and had to spend the afternoon restoring my compiler
• I didn’t realize how important distinct units of computation were until I spent about a month chasing down performance problems in my OCaml code. (These problems were determined to either be in a library or my integration with the library. I gave up and rewrote the OCaml code in C++ in about a week.)

Throughout all this I could never blame anyone except for myself. I let the code get too big, too unwieldy, too impenetrable to change, and the programmer that I was always cleaning up after was me.

Lessons learned. I regret nothing! (except for not writing unit tests)