from __future__ import * 23.1.2004

aeve-o-rama vol.2

This is the second in a finite series of articles about the development of aeve.

...Continuing from where I left off:

aeve.util

I didn't quite finish this one out last time (I think I was tired). aeve.util is generic could-be-anywhere Python code.

aeve.util.ShortBitmask

This is probably the least generally useful module in aeve.util, but the code is portable and it doesn't say Apple anywhere, so I figured this would be the right place for it, but it could or perhaps should be in aeve.terminology. The aete resource format (yes, that documentation is really from 1996) uses these 2-byte "flags" everywhere. In AEUserTermTypes.r ¹, they're arrays of boolean pairs, and that's the representation I chose to use. ShortBitmask is an int subclass, with a metaclass that does the dirty work. It's not meant to be used directly, but to be again subclassed and provided with a __pairs__ member that enumerate the properties. None is a placeholder property for a reserved bit. Here's what it looks like in action:

>>> from aeve.util.ShortBitmask import ShortBitmask
>>> class EventReplyFlags(ShortBitmask):
...   __pairs__ = (
...     ('replyRequired', 'replyOptional'),
...     ('singleItem', 'listOfItems'),
...     ('notEnumerated', 'enumerated'),
...   )
...
>>> erf = EventReplyFlags(1 << 15 | 1 << 13)
>>> erf.replyRequired
0
>>> erf.replyOptional
32768
>>> erf.singleItem
16384
>>> erf.enumerated
8192
>>> erf
40960

I'm using the masks themselves instead of booleans for a reason, I forget what it is at the moment, but I certainly had one. In any case, it's equivalent to what it would be as a boolean and it's faster to just return the integer without the conversion (not that I care). This odd hack is only so I don't have to carry around a bunch of per-flag-type constants later on (in aeve.runtime and aeve.compiler). It does make it harder (but still possible) to create runtime types by hand, as they expect ShortBitmask subclasses and not ints (the same goes for NamedTuple vs. tuple). I'm willing to trade that "convenience" for readable code.

aeve.constants

aeve.constants is probably one of the most and least useful parts of aeve. Basically what I did was take the AppleEvents constants from Python itself (bgen generated from the C header), converted it all over to aeve.util.Enumerations, and grouped it into classes that I could understand (for my reference, not used in code).

Before (these aren't marked as related.. they're mixed in with the other 958 lines of constants):

keyDirectObject = FOUR_CHAR_CODE('----')
keyErrorNumber = FOUR_CHAR_CODE('errn')
keyErrorString = FOUR_CHAR_CODE('errs')
keyProcessSerialNumber = FOUR_CHAR_CODE('psn ')
keyPreDispatch = FOUR_CHAR_CODE('phac')
keySelectProc = FOUR_CHAR_CODE('selh')
keyAERecorderCount = FOUR_CHAR_CODE('recr')
keyAEVersion = FOUR_CHAR_CODE('vers') 

After:

class AEEventParameterKeywords(FourCharCodeEnumeration):
    """Keywords for Apple event parameters"""
    keyDirectObject = FourCharCode('----')
    keyErrorNumber = FourCharCode('errn')
    keyErrorString = FourCharCode('errs')
    keyProcessSerialNumber = FourCharCode('psn ')
    keyPreDispatch = FourCharCode('phac')
    keySelectProc = FourCharCode('selh')
    keyAERecorderCount = FourCharCode('recr')
    keyAEVersion = FourCharCode('vers')

Yes, my code does needlessly use FourCharCode (since it will get turned into the mixed-in FourCharCode __baseclass__ anyway when the metaclass creates it). My excuse is that I was using regular expressions to do all this, and didn't feel a need to chop it out. I also compared the standard MacPython version (it used an older version of Universal Headers) with the OS X headers and I actually had to add in some missing OS X related constants, such as the typeApplicationBundleID, typeKernelProcessID, and typeMachPort addressing modes.

... to be continued

PyObjC, WindowServer, and rasterizing PDFs

We had a visitor to the MacPython Channel yesterday that had heard Quartz (really, CoreGraphics) was a really fast way to work with PDFs on OS X. His specific application was quickly to generating preview jpegs of very large (10mb) dynamically generated press-ready PDFs on the OS X Server 10.2 platform (dual 1ghz Xserve). His current solution was using GhostScript as the rasterization engine, but the color matching was off and it was grinding his poor server to a halt. MacPython to the rescue!

His first attempt was to figure out the CoreGraphics SWIG wrapper ¹. Unfortunately, he didn't know that it only worked on 10.3, and nobody in the channel was terribly familiar with the API beyond the examples from Apple and Andrew Shearer. That was eliminated as a potential solution.

The next (final, and working) attempt was to use CoreGraphics indirectly by way of Cocoa through PyObjC. Fortunately, Dinu Gherman had already done the hard work here, and had made available an open source pdf2tiff command line application that did nearly everything that was required. What was left was converting the script to output jpeg instead.

Initially I had naively thought since there wasn't an explicit NSImageRep for JPEG, it would have to be done as a postprocess by way of PIL, ImageMagick, etc. Then, fortunately I remembered that NSBitmapImageRep was the answer: it can read/write JPEG, PNG, TIFF, and more! We quickly changed the code (it was a three or four line change to pdf2tiff in all) to support JPEG by using NSBitmapImageRep. After we had that working, he wanted to tweak the quality options. Of course, the name of the properties key (NSImageCompressionFactor) was right in the documentation, so he plugged it in and everything just worked.

There is one caveat though, Cocoa requires WindowServer access. WindowServer is the process that makes the GUI tick in OS X, and runs as the currently logged in user. It communicates with these processes via a mach port, with permissions such that only the current logged in user, or root, can access it. So this means, that in order for his script to work on his server, it needs to be setuid root (or setuid the logged in user, and have a particular user logged in all the time). Hopefully someday Apple will help us out here and allow us to run a WindowServer in a virtual framebuffer.. reminds me of when I had to run Xvfb on Linux in order to use Java's AWT stuff to render out images on a server, but these was some 5 years ago, so things may be different.

In any case, he says the previews look better and it runs much faster. I don't have any numbers, but in my experiences with GhostScript and OS X's PDF engine (especially in 10.3!) I would guess it's at least one order of magnitude faster ;)