from __future__ import * 2004/1

Transitioning

I resigned from my job today so that I can find something that excites me. I'll be spending a lot of time with open source while I figure out what's next for me, so expect to see a lot of cool stuff here soon.

If you need a consultant/freelancer in the near term, especially if you're in the NYC area, let me know.

Or, if you represent a badass small company looking for a good full time hire, I may also be interested... but probably only if you're in NYC or the SF bay area (unless you can sell me on why I should move somewhere else).

More Stackless

Worked on Stackless some more this weekend. It should compile as Stackless.framework on OS X now, so pretty soon (when it's stable and all) you will be able to have stackless and python installed at the same time without having to worry about version mismatch errors, DYLD environment variables, or whatever.

pickle those iterators

I made a small patch to the Stackless 2.3 branch today: iterator pickling. It now pickles/unpickles all of the private iterator types (that I know of) in Python's builtins: dictionary-iterator, listiterator, rangeiterator, tupleiterator. I'm sure it will be in CVS pretty soon.

Python 2.3.3 Stackless 3.0 040119 (#5, Jan 25 2004, 11:26:07) 
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pickle import loads, dumps
>>> a = iter(range(10))
>>> a
<stackless.listiterator object at 0x3d0d50>
>>> a.next()
0
>>> a.next()
1
>>> b = loads(dumps(a)) 
>>> a.next()
2
>>> b.next()
2

Today in MacPython - snippets

Here's some links and snippets of interest from today on the MacPython Channel:

• READ THE FAQ! PLEASE! :)

• Squeak - when do we get GUI tools like this for MacPython?

• unicodedata - u'\N{DEGREE SIGN}' is the same as unicodedata.lookup('DEGREE SIGN')

• Zope - How is it different from Apache+PHP? The Plone installer is a good way to try it out on the Mac.

• ZODB - It doesn't use ExtensionClass anymore! But you still have to subclass Persistent.

• bundlebuilder - How do I make a pygame bundle? It's still hard (but I'm working on that). Python+pygame would be a great language to use for the iDevGames 21 Days Later contest, but the bundles are beyond the 2.1 MiB compressed limit! It's possible to make small bundles, but the means to do so are not currently available (but ask me in two weeks if you are participating!)

• IDEs - Vim, Emacs, SubEthaEdit and Xcode do syntax highlighting, PyOXIDE is an up and coming Cocoa IDE.

• Chemistry - PyQuante looks like a good way to learn some quantum chemistry.

• PyObjC - it rocks, but you should already know that. A 1.1 release should be coming out Very Soon Now, but has been set back by sourceforge CVS issues.

• wxPython still has bugs on the Mac

• XUL and Renaissance are XML-based-GUIs

• Enthought Python is a good way to get a scientific/engineering enabled version of Python. We don't have that yet for the Mac, but we should!

• Opportunity landed, but NASA TV is only available via Real (yuck!)

• New PackageWishList node on the wiki, use it!

aeve-o-rama vol.2

This is the second in a finite series of articles about the development of aeve.

...Continuing from where I left off:

aeve.util

I didn't quite finish this one out last time (I think I was tired). aeve.util is generic could-be-anywhere Python code.

aeve.util.ShortBitmask

This is probably the least generally useful module in aeve.util, but the code is portable and it doesn't say Apple anywhere, so I figured this would be the right place for it, but it could or perhaps should be in aeve.terminology. The aete resource format (yes, that documentation is really from 1996) uses these 2-byte "flags" everywhere. In AEUserTermTypes.r ¹, they're arrays of boolean pairs, and that's the representation I chose to use. ShortBitmask is an int subclass, with a metaclass that does the dirty work. It's not meant to be used directly, but to be again subclassed and provided with a __pairs__ member that enumerate the properties. None is a placeholder property for a reserved bit. Here's what it looks like in action:

>>> from aeve.util.ShortBitmask import ShortBitmask
>>> class EventReplyFlags(ShortBitmask):
...   __pairs__ = (
...     ('replyRequired', 'replyOptional'),
...     ('singleItem', 'listOfItems'),
...     ('notEnumerated', 'enumerated'),
...   )
...
>>> erf = EventReplyFlags(1 << 15 | 1 << 13)
>>> erf.replyRequired
0
>>> erf.replyOptional
32768
>>> erf.singleItem
16384
>>> erf.enumerated
8192
>>> erf
40960

I'm using the masks themselves instead of booleans for a reason, I forget what it is at the moment, but I certainly had one. In any case, it's equivalent to what it would be as a boolean and it's faster to just return the integer without the conversion (not that I care). This odd hack is only so I don't have to carry around a bunch of per-flag-type constants later on (in aeve.runtime and aeve.compiler). It does make it harder (but still possible) to create runtime types by hand, as they expect ShortBitmask subclasses and not ints (the same goes for NamedTuple vs. tuple). I'm willing to trade that "convenience" for readable code.

aeve.constants

aeve.constants is probably one of the most and least useful parts of aeve. Basically what I did was take the AppleEvents constants from Python itself (bgen generated from the C header), converted it all over to aeve.util.Enumerations, and grouped it into classes that I could understand (for my reference, not used in code).

Before (these aren't marked as related.. they're mixed in with the other 958 lines of constants):

keyDirectObject = FOUR_CHAR_CODE('----')
keyErrorNumber = FOUR_CHAR_CODE('errn')
keyErrorString = FOUR_CHAR_CODE('errs')
keyProcessSerialNumber = FOUR_CHAR_CODE('psn ')
keyPreDispatch = FOUR_CHAR_CODE('phac')
keySelectProc = FOUR_CHAR_CODE('selh')
keyAERecorderCount = FOUR_CHAR_CODE('recr')
keyAEVersion = FOUR_CHAR_CODE('vers') 

After:

class AEEventParameterKeywords(FourCharCodeEnumeration):
    """Keywords for Apple event parameters"""
    keyDirectObject = FourCharCode('----')
    keyErrorNumber = FourCharCode('errn')
    keyErrorString = FourCharCode('errs')
    keyProcessSerialNumber = FourCharCode('psn ')
    keyPreDispatch = FourCharCode('phac')
    keySelectProc = FourCharCode('selh')
    keyAERecorderCount = FourCharCode('recr')
    keyAEVersion = FourCharCode('vers')

Yes, my code does needlessly use FourCharCode (since it will get turned into the mixed-in FourCharCode __baseclass__ anyway when the metaclass creates it). My excuse is that I was using regular expressions to do all this, and didn't feel a need to chop it out. I also compared the standard MacPython version (it used an older version of Universal Headers) with the OS X headers and I actually had to add in some missing OS X related constants, such as the typeApplicationBundleID, typeKernelProcessID, and typeMachPort addressing modes.

... to be continued

PyObjC, WindowServer, and rasterizing PDFs

We had a visitor to the MacPython Channel yesterday that had heard Quartz (really, CoreGraphics) was a really fast way to work with PDFs on OS X. His specific application was quickly to generating preview jpegs of very large (10mb) dynamically generated press-ready PDFs on the OS X Server 10.2 platform (dual 1ghz Xserve). His current solution was using GhostScript as the rasterization engine, but the color matching was off and it was grinding his poor server to a halt. MacPython to the rescue!

His first attempt was to figure out the CoreGraphics SWIG wrapper ¹. Unfortunately, he didn't know that it only worked on 10.3, and nobody in the channel was terribly familiar with the API beyond the examples from Apple and Andrew Shearer. That was eliminated as a potential solution.

The next (final, and working) attempt was to use CoreGraphics indirectly by way of Cocoa through PyObjC. Fortunately, Dinu Gherman had already done the hard work here, and had made available an open source pdf2tiff command line application that did nearly everything that was required. What was left was converting the script to output jpeg instead.

Initially I had naively thought since there wasn't an explicit NSImageRep for JPEG, it would have to be done as a postprocess by way of PIL, ImageMagick, etc. Then, fortunately I remembered that NSBitmapImageRep was the answer: it can read/write JPEG, PNG, TIFF, and more! We quickly changed the code (it was a three or four line change to pdf2tiff in all) to support JPEG by using NSBitmapImageRep. After we had that working, he wanted to tweak the quality options. Of course, the name of the properties key (NSImageCompressionFactor) was right in the documentation, so he plugged it in and everything just worked.

There is one caveat though, Cocoa requires WindowServer access. WindowServer is the process that makes the GUI tick in OS X, and runs as the currently logged in user. It communicates with these processes via a mach port, with permissions such that only the current logged in user, or root, can access it. So this means, that in order for his script to work on his server, it needs to be setuid root (or setuid the logged in user, and have a particular user logged in all the time). Hopefully someday Apple will help us out here and allow us to run a WindowServer in a virtual framebuffer.. reminds me of when I had to run Xvfb on Linux in order to use Java's AWT stuff to render out images on a server, but these was some 5 years ago, so things may be different.

In any case, he says the previews look better and it runs much faster. I don't have any numbers, but in my experiences with GhostScript and OS X's PDF engine (especially in 10.3!) I would guess it's at least one order of magnitude faster ;)

strings-ing CoreGraphics

I was poking around to see if it was possible to run a separate virtual WindowServer (for non-root daemons to use without a logged in user). Of course, this isn't supposed to be possible, and certainly doesn't seem to be.

However, I did find some interesting things:

• WindowServer is just a bootstrap binary, it calls a single function in the CoreGraphics framework to start up.

• Probably has libpng, libz, libjpeg statically linked (and something to read LZW, but I'm not familiar with those headers).

• These might be possible command line options:

  • ServerEnvironment

  • -debug

  • console

  • -daemon

  • -exit

  • -virtual

  • -virtualonly

• All of that -psn and CFM launching garbage has to do with CoreGraphics (I see 'Joy!' and '-psn' in here)

• It has a reason to check to see if the CoreGraphics server and client have the same endian (maybe a remote mach port on the LAN?)

• It can probably log a whole bunch of fun debugging stuff to /tmp (probably with -debug)

• Somehow there's a reason to have the string "Photoshop 3.0" in there?

The quickest possible way to get readline in Panther Python

python `python -c "import pimp; print pimp.__file__"` -i readline

Done! Don't even need to download the MacPython addons, though you should.

aeve-o-rama vol.1

I've been working on totally rewriting aeve the past well, too long. This week I've made some huge progress. This is the beginning of a series of blog entries where I'm going to talk about the different components of aeve. I will probably only talk about ones that are finished, or mostly finished :) Today I'm just going to stick to the ones that have nothing in particular to do with Apple Events.

aeve._hacks

aeve._hacks is a package where I store things I shouldn't have to do, and it is the source of any import time side-effects that aeve has. Currently the only things in there are the Python 2.4 CVS versions of applesingle, and pprint. applesingle was replaced because it throws a string exception and because it has warnings. pprint was replaced because the Python 2.3 version won't pretty-print subclasses of pretty-printable types. Both of these hacks happen immediately on import of the aeve package, but nothing happens if you happen to be running a build of Python 2.4.

aeve.util

aeve.util is my repository for generic usable-elsewhere cross-platform Python code. I am pretty sure that I'm done with all of the modules in here.

aeve.util.Enumerations

This is one of my favories, and is strangely enough inspired by a similar construct in C#. In C# you can add metadata to just about anything (I think it's called attributes). You can, of course, do this in Python, but it's less of a "meta-protocol" than it is in C# (which has syntax support for the feature). aeve carries around a lot of enumerations whose enumerators need to carry around metadata (such as enumeration it came from, or its name). It works by making a per-enumeration subclass of the desired base type that has a modified __new__ and __repr__, and carries around a reference to the parent enumeration. The base type is typically int but in aeve it's usually a FourCharCode. FourCharCode is a str subclass that will __new__ from an int by struct.pack. The reason for this (as opposed to regular str) is that new-style Apple header files use integers to represent four char codes that contain macroman characters, presumably so that the header files are unix-tool friendly (7-bit unambiguous good ol' american ASCII). Enough of that, this is how it looks:

>>> from aeve.util import Enumerations
>>> class MyEnum(Enumerations.Enumeration):
...   a = 1
...   b = 2
... 
>>> MyEnum.a
MyEnum.a
>>> MyEnum.a + 4
5
>>> MyEnum.fromName('a')
MyEnum.a
>>> MyEnum.fromValue(1)
MyEnum.a
>>> MyEnum.fromValue(1).enumeration
<class '__main__.MyEnum'>

aeve.util.NamedTuple

I've talked about this one before, but I just want to let Just know that I took all the lambdas out ;) NamedTuple is used to build magically delicious tuple trees out of terminologies. I wrote this so that I could introspect the objects (used in aeteViewer: pdf screenshot), but primarily because using named attributes instead of arbitrary indexes is a whole hell of a lot nicer to look at. It's now relatively easy to tell what aeve.compiler.core is doing, for example.

aeve.util.descriptors

Here's where I'm going to stash all of the generic descriptors I need. So far, I've only used one generic descriptor: metamethod. This has been discussed previously, I used Phillip's implementation.

aeve.util.microfailure

microfailure is a homebrew version of twisted.python.failure. I just use it to capture exceptions and pass them along "safely" without ignoring them. It's used by aeve.terminology.resfinder and aeve.compiler.core.

aeve.util.namemangling

namemangling makes strings safe for python names/atributes by appending underscores and replacing unicode with ascii. This is so it doesn't clash with a keyword or builtin. It will do substitution of particular unicode characters such as u'N{DIVISION SIGN}' (aka ÷) with pythonic ascii names like 'div'. If it doesn't have a built-in translation it will use unicodedata.name, or at worst case, it will use the hex representation of the unichar.

... to be continued

linkedin.com

(Not Python Related)

linkedin.com seems to be like the "friendster for business contacts". I think the idea is pretty interesting, and could even serve to replace/augment traditional job searching for two reasons: most business seems to happen through personal contacts anyways, and it can serve as a "trust metric" (a la advogato).

As far as the implementation goes, it seems these guys have a decent enough idea about how to build a web application. The site is nearly all text (but not XHTML, it's loose HTML4 transitional). There's three things that stand out, it runs on Solaris/Netscape-Enterprise (old school), the URLs are short (screams of custom software), and (nearly?) all traffic goes over SSL! I don't know how much traffic they get, but the site is sure a lot faster than friendster and I haven't seen any JDBC errors yet ;)

A quick search shows that I don't seem to actually know anyone currently on, so feel free to sign up or invite me. My email is pretty easy to get ahold of if you don't already have it.

SolarWolf 1.4 on OS X

I've packaged up Pete's latest version of SolarWolf for OS X. It should work on 10.2 and 10.3 and should not have any external dependencies (even Python). This version adds some new items, mines, and all new graphics. It also seems like the starting position of your ship is less deterministic, but maybe I was just distracted by the new sprites :)

Until the SolarWolf page is updated: SolarWolf-1.4.dmg (4.3mb)

PEP 326 -- A Case for Top and Bottom Values

PEP 326 gets my vote, with one condition: change cmp.high and cmp.low! Why not just use max and min respectively? This follows the discussion Josiah Carlson and I had on his livejournal.

For the uninitiated, PEP 326 proposes that Python should have builtins that represent the smallest and largest possible object values regardless of object type. The smallest value will always return -1 for cmp unless it is compared with itself, in which case it returns 0. It's rather obvious what the largest value will do :)

Abusing __getitem__ is fun

class SliceAbuser(object):
    def __init__(self, value):
        self.value = value
        self.items = []

    def __getitem__(self, items):
        if not isinstance(items, tuple):
            items = items,
        append = self.items.append
        for item in items:
            append((item.start, item.stop))
        return self

It doesn't look like much, right? Well, this hack basically allows you to use dict syntax on a __getitem__! Check this out:

s = SliceAbuser
resource, index, help, style, more, moreindex = 'resource', 'index', 'help', 'style', 'more', 'moreindex'
o = s(resource)[
    'index.html': s(index),
    'help.html': s(help),
    'style.css': s(style),
    'more': s(more)[
        'index.html': s(moreindex),
    ],
]

Yeah, it's a whole sitemap in Python syntax. Instead of strings, pretend that resource, index, etc. are all whatever kind of Resource objects your web framework uses (I'm thinking in Twisted). Don't believe me? Look at this:

def printSiteMap(obj, parents = ()):
    print '    ' * len(parents), '/'.join(parents), '->', obj.value
    for key, value in obj.items:
        printSiteMap(value, parents+(key,))
printSiteMap(o)

-> resource
   index.html -> index
   help.html -> help
   style.css -> style
   more -> more
       more/index.html -> moreindex

WhatOS: a free open source embedded system development solution

This looks interesting! I don't typically post this sort of stuff, but I haven't seen it "on the blogosphere" yet. This is directly quoted from a pypy-dev post by Christian Tismer:

Then, Giorgio pointed me to a website today. It makes a really gorgeous claim:

WhatOS is a free open source embedded system development solution. It provides a complete set of tools for creating high-quality, reliable embedded systems. These include: a real-time operating system (RTOS) generator, a simulator for testing and debugging generated systems, and tools for interacting with systems remotely after they have been embedded.

So I thought I should roll this through this list. No idea if this makes sense, so far. http://www.sticlete.com/whatos/

enum { typeWiki = 'Wiki' }

I ended up doing a high level overview of Apple Events / AppleScript in a post on pythonmac-sig this morning. In response to Jack's suggestion on the MacPython channel to add this information to the FAQ, I've expanded, edited, and wikified it as AppleScript. I'll add the FAQ entry sometime later.