from __future__ import * 2004/2

DAAP Authentication Reverse Engineered

This is only related to Python in that I have an unreleased DAAP client/server that I put on hold because Apple had created an authentication protocol. It's currently Twisted based, and I may or may not be releasing it in the near future.

The client authentication protocol has finally been reverse engineered by David Hammerton. Apparently, it's MD5 based, and uses the URL, "Copyright 2003 Apple Computer, Inc." (does this remind anyone else of IBM, or what?), as well as the hex digest of a hash generated from various strings that would be in iTunes for other reasons (HTTP headers, DAAP content codes, etc).

The reverse engineered algorithm stores a 256 entry table of precalculated digests, but only uses one of them in particular, so it sure looks like he's taken a debugger to iTunes. This is interesting to note because it isn't even normally possible, as gdb crashes if you try and attach to it. It's highly likely that this was a deliberate measure by Apple to protect their FairPlay DRM and iTunes Music Store, but obviously isn't enough to stop a determined college student with plenty of reverse engineering experience under his belt.

Update: I shouldn't wake and post. Of course he doesn't have the iTunes-debugging problem, because HE DOESN'T USE A MAC. Duh. Hell, he probably runs iTunes for Windows under some flavor of Wine (he works on the commerical fork).

How not to design a search field

Many of Apple's applications have search fields, such as Help Viewer, Panther's finder, and Mail. However, if you've tried to use them, you probably find that they yield bizarre and typically useless results. I actually RTFM'ed, and here's why it sucks:

• It searches by word prefix (!!) and not by word.

• It uses english logical operators and parentheses (and, or, not) to build queries.

• The default search is or for all words.

• There is no way to search for phrases.

• You can't combine multiple types of queries (e.g. "From: Bob and Ippolito, Subject: mail and sucks")

There is, thank god, an undocumented workaround for first issue: it actually does search by word, not prefix, if you are using a logical operator. Therefore, to find emails that mention your friend ann, but you don't want to get hits on annual, announcement, etc. then you should search for "ann and ann".

I suppose you could use Mail's rule engine to do an ad hoc "advanced search" if you really needed to, but that sure sounds like a lot of work. I'd love to see regexes, but I think I'm better off waiting for hell to freeze over.

The dict.popitem conspiracy

I was creating PyProtocols interfaces for dict-like objects today, and I got stuck trying to decide whether dict.popitem is worthy of including in the interface for a mutable mapping. I have never used it, and I couldn't remember ever seeing it in use, so I checked the standard library. Other than objects that expose the same interface as dict, it seems that popitem is used once and only once in the entire standard library:

def pop(self):
    """Remove and return an arbitrary set element."""
    return self._data.popitem()[0]

I guess it might be vestige from before Python had a decent iterator protocol (though Set doesn't have the excuse of being so old).. but it's pretty ridiculous that popitem is used more often to expose dict-like interfaces than it is used to actually do something.

MarchPython

From stackless.com:

24-Feb-04: Stackless Sprint in Berlin

A sprint on Stackless Python will be done In Berlin, March 10-14. We are planning to work on auto-scheduling, more softswitching support, documentation, examples, Zope, refactoring Stackless into configurable layers, and more.

I'll be flying out to participate, and hopefully also to meet up with the MacPython crew in Amsterdam for a few days.. followed immediately by a week of PyCon.

Open a new terminal right here

Terminal.app and Vim are essentially my IDE, for better or for worse, and I often find myself starting up a new terminal window in the current directory. Here's the script I use (which could easily be a tcsh alias instead, but whatever):

tfork:

#!/bin/sh
set PWD=`pwd`
osascript -e "tell application \"Terminal.app\" to do script \"cd \" & quoted form of \"$PWD\""

Usage:

[crack:~/bin] bob% tfork
(a new window at ~/bin pops up in front)

Trac

Trac is a "minimal" Subversion Frontend/Wiki/Issue Tracker/Task Manager/etc. written in Python that revolves around a Subversion repository and a SQLite database. I've taken a quick glance (at the website and source code), and here are my thoughts:

Pros:

• Verison Control!

• Far, far, far less ugly than almost all of the others

• It's not sitting on top of some monster like Zope/Plone/etc.

Cons:

• The source code mixes tabs and spaces (what the hell are they thinking)?

• Lots of inline SQL code, no abstraction on top of SQLite.

• It is based on CGIs, with no fcgi/asyncore/Twisted/etc integration that I can see, so it's not likely to scale and may have database locking issues (either corruption, or not scaling because the database is locked per-process).

• The Wiki engine is based on MoinMoin syntax not reStructuredText, and it has no hooks to docutils.

• When they say minimal, they were probably comparing themselves with Plone. It certainly doesn't sound minimal from the description!

That said, I think something like this is sorely needed, but it doesn't sound appealing enough just yet to compell me to dump roundup, MoinMoin, viewsvn, etc.

graphviz on the Mac, LaunchServices

For an excellent example of what it takes to do a great port of UNIX software to OS X, look no further than Glen Low's port of graphviz for OS X. For good reasons to take a look at graphviz, check out John Schull's Weblog, especially entries such as Outline to graphviz.

This port and GUI front-end adds features such as rendering via the native OS X APIs (finally, good fast PDFs!), and it can actually use kqueue on OS X 10.3 to automatically re-render when a file changes!

Not only does at add a slew of cool new features (most of which I didn't mention), but all of the CLI tools are wrapped up in a nice little application bundle; making installation and removal painless, and relocation no longer becomes a problem.

If all OS X with significant UNIX heritage were distributed this way, one might think it would become difficult to add them all to your PATH. Fortunately, that's just not true ¹. Here's a nice little snippet of LaunchServices ² code that will return the full path of an application by name regardless of where it happens to be or if it has moved ³:

#!/usr/bin/python
from LaunchServices.Launch import LSFindApplicationForInfo
from Carbon.CoreFoundation import kCFURLPOSIXPathStyle
# this would normally come from LaunchServices.LaunchServices
# but bgen isn't perfect, and doesn't make this constant a
# four char code.
kLSUnknownCreator = '\x00\x00\x00\x00'

def pathForName(appName):
    """
    Returns the file system path for an application by name.
    NOTE: You should include the suffix, as in "Python.app".
    """
    fsRef, cfURL = LSFindApplicationForInfo(kLSUnknownCreator, None, appName)
    return cfURL.CFURLCopyFileSystemPath(kCFURLPOSIXPathStyle).toPython()

if __name__ == '__main__':
    import sys
    if len(sys.argv) == 1:
        import os
        raise SystemExit, "Usage: %s SomeApplication.app" % (
            os.path.basename(sys.argv[0])
        )
    for appName in sys.argv[1:]:
        print pathForName(appName.decode('utf8')).encode('utf8')

Usage looks like this (note: I use tcsh not bash):

[crack:~/bin] bob% setenv PATH ${PATH}:`./findapp.py GraphViz.app`/Contents/MacOS
[crack:~/bin] bob% rehash
[crack:~/bin] bob% which dot
/Applications/GraphViz/Graphviz.app/Contents/MacOS/dot
[crack:~/bin] bob% dot --help
dot: option -- unrecognized

Usage: dot [-Vv?] [-(GNE)name=val] [-(Tlso)<val>] <dot files>
 -V          - Print version and exit
 -v          - Enable verbose mode 
 -Gname=val  - Set graph attribute 'name' to 'val'
 -Nname=val  - Set node attribute 'name' to 'val'
 -Ename=val  - Set edge attribute 'name' to 'val'
 -Tv         - Set output format to 'v'
 -lv         - Use external library 'v'
 -ofile      - Write output to 'file'
 -q[l]       - Set level of message suppression (=1)
 -s[v]       - Scale input by 'v' (=72)
 -y          - Invert y coordinate in output

How to patch Python's standard library without touching vendor files

I've been using 2.3.0 for quite some time because it's the version Apple provides with OS X 10.3. This works out just fine most of the time, even though 2.3.3 is the latest version. However, it's definitely possible to hit a nasty stardard library bug or performance issue that has already been fixed.

For example, the _strptime module in 2.3.0 didn't cache locale settings. This misfeature makes it (in my experience) about 30 times slower than it should be (~20 dates/sec here); completely unacceptable.

Short of using PYTHONPATH (a total kludge, IMHO), it's not obvious how one would go about overriding (without overwriting) bits of the standard library; site-packages and any pth files are appended to sys.path long afterwards. Fortunately, someone already thought of this and placed an obscure feature into site.py: pth files can import arbitrary python code!

So now all you have to do is create a folder for standard library fixes, a python module to inject it into sys.path, and a pth file to make that happen whenever you start an interpreter. Here's mine:

/Library/Python/2.3/2.3_FIXES.pth

import _2_3_FIXES

/Library/Python/2.3/_2_3_FIXES.py

import sys
sys.path.insert(0, "/Library/Python/2.3_FIXES")

Now create /Library/Python/2.3_FIXES and drop all the standard library updates you need right in there. I suggest fetching _strptime.py from 2.3.3 right away ;)

PackageManager repository update

Another update to my OS X 10.3 Package Manager repository

New packages:

• PySQLite 0.5.0, Python wrapper for SQLite, an embeddable SQL database engine (linked statically against SQLite 2.8.12 configured with --enable-utf8).

pth files are your friend

Why is it that many Python packages insist on littering your site-packages folder? It's not very hard to make a setup.py that has its own little microcosm referenced by a pth file, and it makes the user's life so much better. See Numeric, PIL, or PyObjC for decent examples.

Bonus: the user can easily uninstall, cleanly upgrade, or even move the package to an alternate location simply by moving it and changing the path in the pth.

Update: It looks like this is done with the extra_path argument to setup(...). I would've mentioned it, but I didn't realize that it wasn't documented outside of the source, mailing lists, and files in an old CVS attic!

Apple is your Sysadmin

/usr is not yours (except for /usr/local).

/System is not yours.

Please, if you are installing your own software or "upgrading". DO NOT PUT IT IN APPLE'S LOCATIONS! If you do, you just broke your operating system. There is a good chance any patch that Apple provides may not work, existing and new software may not work, etc.

If you want to "upgrade" some software or service, put it in /usr/local or /Library. Change your PATH if you need to, maybe something in /etc, but god damnit don't rename, delete, or replace anything in Apple's space! In doing so, you are making your life harder, as well as annoying the heck out of all of us software developers who are writing software that depends on a particular operating system. When you screw with anything that lives on those directories, you are no longer running an operating system that we are willing to support.

/usr/local has been around for decades for this reason. It wouldn't be still be around if it wasn't a better idea than replacing your OS vendor's files on a whim.

Daikon

Daikon is a pretty neat idea.. it seems sort of like an empirical way to do stuff like type inference, but officially it dynamically detects program invariants. It would be nice if we had something like this for Python to help us write better tests and possibly optimize code. For some bizarre reason, Daikon 1 was written in Python, but 2 and 3 were Java. Even still, it has never analyzed Python code.

And if you think about doing a version for Python, don't bother porting it; the Daikon license is bogus. "Daikon is made available for education, research, and evaluation purposes (not for direct commercial gain), and there is no warranty". Besides, it would probably take more time to figure out which license a port/front end would be subject to (I see four or five mentioned!) than to just rewrite the damned thing.

NEXT_ROOT lives again

If you are wondering how to compile stuff on 10.3 such that it's probably backwards compatible with OS X 10.2, you'll want to set the following environment up (after having installed the 10.2 SDKs, of course):

setenv NEXT_ROOT /Developer/SDKs/MacOSX10.2.8.sdk
setenv MACOSX_DEPLOYMENT_TARGET 10.2

This, of course, is only useful if the whole environment you're trying to make standalone is already 10.2 compatible. The Python 2.3.0 that comes with OS X 10.3 is most definitely not. At some point, one of the MacPythonistas will probably put together a little "chrooted" environment for building 10.2 compatible --standalone'd software (but don't hold your breath).

PackageManager repository update

Another update to my OS X 10.3 Package Manager repository

Updated packages:

• numarray updated from 0.7 to 0.8, now using Apple's Accelerate framework for LAPACK and BLAS

New packages:

• pCFITSIO 0.99.0, a SWIG wrapper for CFITSIO (the source looked to be in a pretty sad state, but someone asked for it.. no guarantees).

bundlebuilder2 -- coming soon to a mac near you

For the past.. while.. I've been slowly hacking at a refactoring of bundlebuilder that uses my (unreleased) macholib package. Here's what the new bundlebuilder will do:

• (complete) In addition to automatically finding modules when in --semi-standalone and --standalone mode, it will also find non-System dylibs and frameworks!

• (complete) It properly rewrites mach-o headers on everything such that they have load commands referencing @executable_path/../Frameworks rather than their original paths.

• (in progress) Configurable detection of packages that should be included whole, such as pygame, because it references data files via __file__ (breaking zip import).

• (planned, to be released separately) a PyObjC GUI for bundlebuilder2, that makes all of this even more configurable and easy to use.

For the interested, macholib is basically a Python-rewrite of the /usr/include/mach-o headers, which a subset of the functionality from otool and install_name_tool. Originally I had released something called potool that attempted to do these things, but I had made a few wrong assumptions, so I rewrote it all as macholib.

Honestly, the only things I can think of that otool has that I don't are symbolic disassembly, symbol table dumps, and useless things like printing a nice description of the threadstate load commands. I actually do have a mostly-finished symbolic PPC disassembler written in Python, but it's an ugly port of some of GCC's guts and I doubt it'd be a good idea to unleash it on the world :)