Python Rocks! and other rants 2004/4

Weblog of Kent S Johnson

Simplicity Rules

Preaching to the Choir

My two "Why I love Python" articles have been wildly popular. Much of their popularity is due to being mentioned in the Daily Python-URL. But I have to wonder, why do Pythonistas so enjoy reading about why Python is great? And how can I reach the Java programmers where I work and convince them to try Python?

Why I love Python 2

Python makes it very easy to build complex data structures. One place this is handy is with data-driven programming.

For Meccano I wrote a simple walk-by-rule engine. It walks the tree of domain data and applies callbacks at indicated points. The walk is driven from a tree structure that can be quite large and deeply nested. (I have written about the rule engine before.)

Here is a simple example using some of the same techniques. As you read the example, imagine that the list of rules might be hundreds of lines long and deeply nested. Later on I will indicate some other ways the example might be extended.

Assume we are given a dictionary and we are to print its contents in a nested form where the nesting and order of keys in the output is given by the location of dictionary keys in a structure built from nested lists.

The essential idea of the problem is to use a staticly defined nested list to drive the formatting.

data = { 'a':1, 'b':2, 'c':3, 'd':4, 'e':5 }

formatData = [ 'a', [ 'b', [ 'd', 'e' ], 'c' ] ]


def output(format, indent):
  for item in format:
      if type(item) == type([]):
          output(item, indent+2)
      else:
          val = data[item]
          print '%*s%s: %s' % (indent, ' ', item, val )

output(formatData, 0)

a: 1
 b: 2
   d: 4
   e: 5
 c: 3

import java.util.HashMap;
import java.util.Map;

public class Structure {

   static Map data = new HashMap();
   
   static {
       data.put("a", new Integer(1));
       data.put("b", new Integer(2));
       data.put("c", new Integer(3));
       data.put("d", new Integer(4));
       data.put("e", new Integer(5));
   }

   static Object[] struct = {
       "a", 
       new Object[]{
           "b", new Object[]{
               "d", "e"
           },
           "c"
       }
   };

   public static void output(Object[] format, int indent) {
       for (int i = 0; i < format.length; i++) {
           Object item = format[i];
           if (item instanceof Object[]) {
               output((Object[])item, indent+2);
           }
           else {
               Integer val = (Integer)data.get(item);
               for (int j=0; j<indent; j++)
                   System.out.print(' ');
               System.out.println(item + ": " + val);
           }
       }
   }
   
   public static void main(String[] argv) {
       output(struct, 0);
   }
}

Wired News: Diebold May Face Criminal Charges

If you think development of voting machines is somehow different from other software/hardware projects you should read this article.

Diebold, manufacturer of voting machines used in California, has been decertified by the state and referred to the state attorney general for possible civil and criminal charges under state election law.

The details sound very familiar to projects I have worked on that were struggling to meet a deadline. For example, a new peripheral was installed days before the election that was still being debugged. The peripheral failed in two counties causing the polls to open late. In addition, "Diebold...installed uncertified software on its voting machines in 17 counties without notifying state officials or, in some cases, even county officials who were affected by the changes."

This behavior is appropriate for a trade show, not a federal election. It's no way to run a democracy.

Agile Prophecies of Dr Seuss

Everything I need to know I learned from The Cat in the Hat.

Just read it :-)

Don't Repeat Yourself

Don't Repeat Yourself and its special case Once and Only Once are two of the most important principles of good development. Read this essay for more.

Why I love Python

The code I wrote last night to build a Map of Maps shows one reason why I like Python so much - it is so easy to work with collections!

I wrote a sample app that shows the same thing in Java and Python. The requirement is to take a list of triples of strings in the form

[ language code, item id, localized string ]

and build a two level Map from language code => item id => data triple. Both examples include a simple test driver which prints:

This is a test
C'est un essai
no data
Another test
no data

import java.util.*;

public class Test {

  private Map _map = new HashMap();
  
  public Test(String[][] data) {
      // Convert the input data to a two-level Map from language code => course ID => locale data
      for (int i = 0; i < data.length; i++) {
              String[] itemData = data[i];
              
          String lang = itemData[0];
          
          Map langMap = (Map)_map.get(lang);
          if (langMap == null) {
              langMap = new HashMap();
              _map.put(lang, langMap);
          }
          
          String id = itemData[1];
          langMap.put(id, itemData);
      }
  }
  
  public String lookup(String lang, String id, String defaultData) {
      Map langMap = (Map)_map.get(lang);
      if (langMap == null) return defaultData;
      
      String[] itemData = (String[])langMap.get(id);
      if (itemData == null) return defaultData;
      
      String title = itemData[2];
      if (title == null || title.length() == 0)
          return defaultData;

      return title;
  }
  
  
  public static void main(String[] args) {
      String[][] data = {
          { "en", "123", "This is a test" },
          { "fr", "123", "C'est un essai" },
          { "es", "123", "" },
          { "en", "345", "Another test" }
      };
      
      Test test = new Test(data);
      
      System.out.println(test.lookup("en", "123", "no data"));
      System.out.println(test.lookup("fr", "123", "no data"));
      System.out.println(test.lookup("es", "123", "no data"));
      System.out.println(test.lookup("en", "345", "no data"));
      System.out.println(test.lookup("fr", "345", "no data"));       
  }
}

class Test:
  
  def __init__(self, data):
      # Convert the input data to a two-level Map from language code => course ID => locale data
      self._map = {}
      
      for itemData in data:
          lang, id = itemData[:2]
          self._map.setdefault(lang, {})[id] = itemData


  def lookup(self, lang, id, defaultData):
      itemData = self._map.get(lang, {}).get(id)
      if  not itemData:
          return defaultData
      
      return itemData[2] or defaultData
  
  
if __name__ == '__main__':
  data = [
          [ "en", "123", "This is a test" ],
          [ "fr", "123", "C'est un essai" ],
          [ "es", "123", "" ],
          [ "en", "345", "Another test" ]
      ]
      
  test = Test(data);

  print test.lookup("en", "123", "no data")
  print test.lookup("fr", "123", "no data")
  print test.lookup("es", "123", "no data")
  print test.lookup("en", "345", "no data")
  print test.lookup("fr", "345", "no data")

What happened to the Python part?

10x speedup ain't bad!

I am working on some of the worst code I have seen in years. About the best thing I can say for it is, it's mercifully short - 800 lines of Java with eight comments. (The second best thing is that it isn't in VB. The last truly horrible code I worked on was a VB monster with a 4000-line loop and case statement at its heart.)

The main loop is 250 lines. I don't think the author understood recursion; there are four separate stacks that maintain state in the loop! I'm not sure yet but I think they will all be replaced by recursive calls.

The loop builds a 5 MB XML structure in a StringBuffer, then writes it out to a file! Um, why not just write it to a file directly? Well, the StringBuffers are on one of the stacks so I have to sort that out first.

It has wonderfully readable conditional code like this:

if (isNoLocale() == false) ...

And the clincher - the program uses 16,285 localizations. They are keyed by language code and id. So how do you think the program was accessing this data? It put it all in a List and searched sequentially for a match!!! Yikes!

When I first ran the program it took 191 seconds to create the file.

So today's wins were to - Refactor the CSV reader part of the code into a separate class and clean that up, including changing the line items from Maps to Lists. Time to generate the file: 119 sec. - Build a two-level Map so the localizations can be looked up directly instead of by exhaustive search. Time to generate the file: 19 seconds!

And what does this have to do with anything, anyway? Well, I have to rant every once in a while or I will have to rename my blog :-)

And what is the way out of this mess? Refactoring and unit testing, of course! I have three test files containing all the live data used by the program. Every time I make a change I regenerate them and test with XmlUnit. Now I can refactor without fear to get the program to a point where I can understand it.

Poor code structure is a performance issue! Because if you can't understand it, you can't find the bottlenecks. You can't even profile code that is all in one method.

I saw the same thing with the 4000-line VB loop. I started factoring out common code and eventually I could see where the performance problems were and do something about them.

J2EE Design and Development, again

I really like this book. It is full of sensible, practical advice based on real experience.

For example Chapter 3 is about testing J2EE applications. The chapter starts with an introduction to Test-driven development, JUnit and best practices for testing in general. Then the author reviews several freely available testing tools including Cactus, ServletUnit, HttpUnit and Web Application Stress Tool.

Each of these tools is presented in the context of testing one aspect of a J2EE application. Short code snippets illustrate the use of the tool.

I have a hard time putting this book down. Really! OK, so I'm a geek. I admit it. And he's not Michael Crichton. But I enjoy reading the author's opinions and advice and I am learning a lot.

Alternative Enterprise Architectures

Here are some thought-provoking articles about enterprise application architecture:

GraphViz

I've just discovered the GraphViz package from AT&T Research. I have heard of the package before. It was a release of pydot, a Python wrapper, that got me to look at it.

GraphViz makes it astonishingly simple to create nice-looking graphics. For example, to make a package dependency graph of the Java packages in Meccano, I created this graph definition:

digraph G {
  size = "3,3";
  node [shape=box,fontname="Helvetica"];
  blasterui -> blaster;
  blasterui -> swing;
  blaster -> converter;
  blaster -> writer;
  writer -> mockcourse;
  devscript -> coursedata;
  devscript -> word;
  converter -> devscript;
  converter -> mockcourse;
  mockcourse -> coursedata;
  coursedata -> util;
  server -> editor;
  server -> writer;
  editor -> util;
}

I saved the definition in a file called dep.data. Next I ran dot from the command line with

dot -Tpng dep.data > dep.png

Here is the resulting image:

There is much, much more you can do with this package but this shows how easy it is to get a useful result.

pydot puts a Python wrapper around GraphViz. This could be useful for creating graphs from a program. For hand-created graphs it might be easier just to write the data file by hand as I did for the example.

I had to make a few changes to pydot.py to get it to work under Windows. There were problems with the PATH separator, the name of the .exe files, and text vs binary files. Here is a diff from version 0.9 that shows the changes I made:

Compare: (<)C:\Downloads\Python\pydot-0.9\pydot.py (26989 bytes)
   with: (>)C:\Python23\Lib\site-packages\pydot.py (27121 bytes)

117c117
<     for path in os.environ['PATH'].split(':'):
---
>     for path in os.environ['PATH'].split(os.pathsep):
121a121,122
>             elif os.path.exists(path+os.path.sep+prg + '.exe'):
>                 progs[prg]=path+os.path.sep+prg + '.exe'
727c729
<         dot_fd=open(path, "w+")
---
>         dot_fd=open(path, "w+b")
763c765
<         out=os.popen(self.progs[prog]+' -T'+format+' '+tmp_name, 'r')
---
>         out=os.popen(self.progs[prog]+' -T'+format+' '+tmp_name, 'rb')

Continuous Design

I have written before about Growing a design. Key to the success of this technique is keeping your code clean using principles such as Don't Repeat Yourself and You Aren't Going to Need It.

In this article, Jim Shore chronicles his experience with this process. I particularly like the sidebar Design Goals in Continuous Design which summarizes much of what makes this technique work.

Inversion of Control Frameworks

Whenever you hide an implementation class behind an interface, you have the problem of instantiating the concrete instances of the interface and giving them to the client code.

There are several ways to do this:

• The client code can instatiate the instance directly
• The instance can be stored in a global resource such as a singleton or registry
• The code that instantiates the client can also create the instance and pass it to the client
• An instance can be created from a global property using reflection

Each of these techniques has disadvantages:

• If the client creates the instance then you can't substitute a different implementation without changing the client, and the benefit of using the interface is reduced.
• Using a global registry, singleton or property makes the client depend on the global facility which makes testing and reuse more difficult.
• Reflection is complicated when the instance had dependencies of its own, for example it needs configuration data or depends on other interfaces.

A solution to this problem that is gaining popularity is to use a framework with support for Inversion of Control (or, as Martin Fowler calls it, Dependency Injection). With this technique, client code can be written with no dependencies on global resources. The framework takes care of initializing the required instances and providing them on demand.

Martin Fowler has an article that explains the technique. Two frameworks that use this technique are Spring and PicoContainer.

XPath and dom4j