Jython + dom4j = High octane development

Weblog of Kent S Johnson

Python has strong support for data structures

Lists and dictionaries are built in to the language. It is very easy to create and use ad-hoc, heterogeneous data structures. Iteration and lookup for both lists and dictionaries are supported by the syntax of the language. This benefits all kinds of programs because lists, dictionaries and compound data structures are useful everywhere.

This strong support for structured data really shines in data-driven programs, because it is easy to specify and use the actual data. Complex data structures can be created statically, in declarations, or dynamically, in code. Unlike XML or other text-based data formats, no user-written parser is needed - the Python runtime parses the data for you and yields a native data structure. Using the data is supported by the core language.

Python has first-class functions

In Python, you can have standalone functions (not part of a class). Functions can be put in a list and called later, so a data structure can include callback functions. And a function can return another function as a value, so you can create a function that binds some variables while leaving others unassigned.

For example suppose you have a function adder that takes two arguments and returns their sum:

def adder(x, y):
    return x + y

Now you can make a function factory - given a value n, it returns a function of one argument that adds n to its argument:

def makeAddN(n):
    return lambda x: adder(x, n)

So a call makeAddN(5) returns a function that adds 5 to its argument.

This is a little hard to explain but it is very powerful. Do you ever miss function pointers from C or C++? Python can do everything you can do with a function pointer and more. I will show you a real-world example of this below.

In addition to Python's benefits, Jython provides easy access to Java libraries, both Java standard APIs (Swing), third party libraries (dom4j, Velocity, Jetty, etc) and our own Java code (Meccano1 codebase with additions for Meccano2)

Example: Meccano audit engine

The Meccano audit engine takes advantage of Jython's flexible data and first-class functions, and dom4j's integrated XPath engine, to make a very flexible, table-driven course audit tool.

The core of the engine is a recursive data structure consisting of XPath expressions and audit rules. The structure can be defined by these production rules:

A RuleItem is either

• A callable (a function or function-like object) which is called with the current dom4j node as its only argument
• A pair (tuple) containing an xpath expression and a RuleItem to apply to all the nodes that match that expression, or
• A list of zero or more RuleItems

The function which applies a set of rules is short and sweet - just 12 lines of code. It is passed a RuleItem structure and a dom4j node to apply the rules to. At each step it looks to see if it has a callable, a pair or a list, and takes the appropriate action:

def walkByRule(ruleItem, data):
    if callable(ruleItem):
        # This is an actual rule, just call it on the current data
        ruleItem(data)

    elif type(ruleItem) == type( () ):
        # ruleItem is a tuple ( xpathexpression, childRuleItem ) 
        # childRuleItem is applied to each node selected by the xpath expression
        childExpr, childRuleItem = ruleItem
        for child in data.selectNodes(childExpr):
            walkByRule(childRuleItem, child)

    elif type(ruleItem) == type( [] ):
        # A list of rules, evaluate it recursively
        for rule in ruleItem:
            walkByRule(rule, data)

    else:
        print 'Unknown rule type:', ruleItem

Here is a short snippet of the actual audit rules table. At this point the current node is a Learning Point. The first part of the snippet is an xpath expression that selects only Learning Points whose type is "summary". This is followed by a list containing two rules. The first rule evaluates an XPath expression againt the current node. The particular expression checks to see if the summary point is the last point. The second rule checks for the existence of an exercise point in the same topic, which is not allowed.

('.[@type="summary"]',  [
    xpathRule('count(following-sibling::LearningPoint) = 0',
       'Summary Learning Point must be the last Learning Point in a Topic'),

    limit('count(../LearningPoint[@type="software_simulation" and Slide/@type="exercise"]')', 0,
       'Topic with Exercise Point must not contain a Summary Learning Point'),
] ),

Note that the rules are actual function calls. How is this possible? The rule engine requires that the rules be instances of functions with one argument.

Well, xpathRule is a function that returns a function of one argument. To see how this works, first look at the function that implements the actual rule. It is a pretty straightforward function that evaluates an XPath expression against a data node and logs an error if the result is not 'true':

def _xpathRuleImpl(data, xpathExpr, desc):
    ''' data.valueOf(xpathExpr) must be 'true' '''
    result = data.valueOf(xpathExpr)
    if not result == 'true':
        _logError(desc, data)

To turn this function of three arguments into a function of one argument, we create a new function that binds the xpathExpr and desc args and just needs the data argument. That is exactly what xpathRule does:

def xpathRule(xpathExpr, desc):
    return lambda data: _xpathRuleImpl(data, xpathExpr, desc)

So the call to xpathRule in the audit data actually creates the function that will be called by the rule engine. The call to limit() does much the same thing.

The complete table of audit rules contains over 100 rules. They are all created using the same simple building blocks. Many of the rules reuse xpathRule, limit and other generic rules. In a few cases, special-purpose functions check a single specific rule.

In Meccano, the same core rule engine is used to create word counts. In this case, the rules at the leaves of the rule data are instructions for how to count the text in the corresponding node. Similar techniques are used in the part of Meccano that builds the visual representation of the dom tree for display in the GUI.

I hope these examples have given you some idea of the power of Jython and dom4j when used together for data-driven programming.

Detailed example

Here is a complete DomUtil module containing walkByRule and an extensive example: