Peter's Blog 2004/8

No email after syslog-ng death

Back to work after weeks holiday and only 4 emails? The culprit was syslog-ng which died the day after I left work (software efficiency) so email systems refused to work. Did

/etc/init.d/syslog-ng restart

and email started to flow. Well, trickle is a better word: spamassassin is not exactly lightning fast.

Gmail seems more and more attractive. Web based email is useful, I never bothered with it before because my email provider's server is sluggish. Cannot say that about google.

One4All with Sky Plus

I got the key magic codes to program up the play, pause, fast forward, rewind and record buttons of my One4All four (URC 7740). By default these buttons are programmed up for things like 'Box Office', 'Services' etc. Any Sky Plus user will tell you that Play, Pause and Fast Forward are far more useful. There aren't enough buttons on the remote for it to do everything. I might be able to program up a different device code (e.g. use DVD for sky plus and program the buttons differently) but I can't be bothered. Now I won't have to keep the real Sky Plus remote with me just to record programs.

The instructions to reprogram the buttons are:

1. Press the relevant device key

2. Press and hold the MAGIC key until the red light flashes twice.

3. Press 9 9 4. The red-light flashes twice

4. Press the MAGIC key once.

5. Enter the 5-digit function code (provided by customer service, where applicable).

6. Press the key, to which the function will be e.g. A.V. key, the red-light flashes twice.

The useful extra codes for device 1170 (Sky Plus) are:

00243 Pause
00707 Play
00528 Record
00339 Search Forward
00683 Search Reverse
00526 Sky

GMail

Decided my email setup was still not complex enough so I bought a trendy Gmail invite on ebay for £1.30 (a bargain for a new toy) and set up my server to forward all mail to it. This way I can try gmail out while still archiving messages safely at home. I did this by putting the following in the exim .forward file:

unseen deliver <blah>@gmail.com

This forwards everything but keeps handling the messages on the server.

GMail so far seems neat. Observations:

• Runs quite fast, subjectively faster than Thunderbird, dovecot etc.

• Cannot compose html email

• I sent an email to a friend who replied. I replied to the reply and Gmail bounced with this cryptic error:

  Technical details of failure:
  PERM_FAILURE: SMTP Error (state 13): 550 Relay not permitted - (ACL violation)
  

  Like a lot of email server error messages, this is a little lacking in detail. Which server was trying to relay and why?

• Gives very nice threaded presentation of messages. Would be great if I could reply to replies.

Freeview

Installed a freeview box at mum's house. It was a Goodmans's thing, £55. This gives about 32 digital tv stations and a load of digital radio stations. Some observations:

• Installed very easily once I remembered to plug the aerial in.

• Came with a SCART cable so TV did not need tuning to the box.

• Was going in minutes

• Extra TV stations like ITV2, Sky news, etc. Not an amazing choice but ok for £55 downpayment and no monthly fees.

• Some stations like UK Gold, E4 and Discovery are in the program guide but you have to pay a monthly subscription.

• Picture was noticibly better quality: much crisper, better contrast, compared to old analogue signal.

• Picture is either 16:9 in 4:3 letterbox or 16:9 full screen. Hence on her 4:3 tv she has to use letterbox.

• Box locked up once while I was showing it to her: had to power cycle. Tried updating firmware but it already had the latest version. Taught mum how to power cycle, just in case.

Procmail works but no email

After 5 days away I checked my email and was shocked to find absolutely none. Very odd. After a mild panic I worked out that procmail had diligently filtered my email into the correct mail folders but Thunderbird was only set up to poll the inbox folder. To get Thunderbird to check subdirectories for new email you have to set the properties for each folder and enable 'check this folder for new messages'. Hopefully in a new version they will add a default action to check all folders.

Spamassassin seems to be doing a good job with the spam filtering. Out of 8 spams, no false positives, 7 had been tagged by pyzor, the remaining spam had lots of silly defects:

4.2 MIME_BOUND_DD_DIGITS   Spam tool pattern in MIME boundary
3.1 MSGID_SPAM_CAPS        Spam tool Message-Id: (caps variant)
0.1 MIME_HTML_ONLY         BODY: Message only has text/html MIME parts
0.7 MIME_HTML_NO_CHARSET   RAW: Message text in HTML without charset
1.2 MISSING_MIMEOLE        Message has X-MSMail-Priority, but no X-MimeOLE
0.0 FORGED_AOL_TAGS        AOL mailers can't send HTML in this format
3.9 RCVD_DOUBLE_IP_SPAM    Bulk email fingerprint (double IP) found
1.7 HTML_MIME_NO_HTML_TAG  HTML-only message, but there is no HTML tag
4.3 FORGED_MUA_AOL_FROM    Forged mail pretending to be from AOL (by From)
1.8 FORGED_AOL_HTML        AOL can't send HTML message only
1.1 MIME_HTML_ONLY_MULTI   Multipart message only has text/html MIME parts
0.1 MISSING_OUTLOOK_NAME   Message looks like Outlook, but isn't

Do the spammers do anything as simple as test their spam against spamassassin? Hope not.

State of Play

Sky Plus box seems to be fine after hard disk upgrade. I've been away and we've only watched one episode of 'Family Affairs' but that had no problems at all. During the 30x fast forward the box used to stutter a bit, the poorer the recording the worse the stutter, sometimes pausing for a few seconds. Now the stutter is gone. The recording capacity has doubled, 2 films and 5x30 minute soaps use 15% of the disk space.

I bought a 512M Dimm for PC and plugged it in. No way will the PC recognise it as 512M, insists it is 256M. I tried:

• Flashing BIOS

• Swapping slots

• Putting in a company of 256 and 128M DIMMs

• Swearing at it

• Changing FSB from 133MHz to 100Mhz

• Trying a real video card with it's own memory instead of the on-board one that uses system ram.

No joy. For now I am living with 512M (real 256 + 1/2 512). I blame VIA for their naff chipsets, I kick myself for buying another one.

Sky Plus Fix

The playback of recordings on my Sky Plus box was seriously degrading: almost every recording had glitches where playback suddenly jumps back a few seconds and compression artifacts are all over the picture. At it's worst the sound goes or makes lots of noise. This was worse during a particularly dramatic episode of Family Affairs, spoiling my wife's veiwing pleasure.

So I've put the 80G hard disk from my old PC into it. For the record this is a Western Digital WD800BB-75DKA0. I took the following steps:

• Swapped disks

• Powered up box while holding down 'Backup' button on front panel. This caused a download of the firmware, giving an alarming 'Don't remove mains for at least 10 Minutes' display on the TV.

• After looking dead and ignoring the remote for a while it came to life and started downloading listings.

• Went to 'Services' menu and pressed 4 for the 'system setup' menu, '01 Select' for the secret housekeeping menu and 8 to reformat the hard disk.

• Box again went dead for a minute or so.

• On again, using remote, and downloaded listings.

• Turn to BBC 1, press live pause and marvel that it worked.

The secret with this is to stay calm and be patient when the box looks dead.

Now, fingers crossed that the drive lasts longer than the 6 days that the accursed IBM Deskstar survived (P366).

Ashford

My first chance to photo blog from starbucks. Sunny day in ashford, not hot like it has been recently. Global warning means horrible humidity in this part of the world. Not sure if this is better than rain.

It's my birthday on monday so i'm trying to think of a tech birthday present for myself. More memory for my pc is top of the list. It has only 256Mb and i ought to upgrade it before sdram becomes obsolete.

Hello from Dallas

Got a version of Visual Traceroute free on a cover disk. It's a souped-up traceroute that shows a world map and gives a host of peripheral information.

It confirms that this blog is hosted in Dallas, Texas and is on a Debian linux box. These days you don't always know where the servers may be.

Ooh Er, it even shows the town I live in if I search for my home server. Do I want the world to know?

My friend zugz uses NTL World for hosting and it seems he is hosted in Canada. He owes me some maple syrup.

Maildir Email archiving

I've written a python script to handle email archives. This script scans my Maildir directories and moves messages that are older than a directory-dependant age to a compressed archive. I will run this from cron.

#!/usr/bin/python
#
# Walks through Maildir store, compressing and deleting old email messages.
#
import email
import os
import time
import bz2

strRootMailDir = '/home/pcw/Maildir'

#
# Folders that can be pruned
# Defined as a dictionary where keys are directory names and values 
# are maximum age in weeks
#
oPrunable = { '.IT.Auto.ISA Server Alert': 2,
        '.IT.Auto.Message Deleted': 2,
        '.IT.Auto.Sophos Alert': 2,
        '.IT.Auto.Sweep Report': 2,
        '.IT.Auto.Virus': 2,
        '.INBOX': 12,       # can sit in inbox for 3 months
        '.Sent': 4}

#
# Callback from os.walk to prune a particular directory.
#
def ProcessDir( strArg, strDir, strNames):
  #
  # Split tail part of directory name.
  # This should be cur, new, tmp etc.
  #
  strPartDir, strTailDir = os.path.split( strDir)

  if strTailDir != 'cur':
    return

  #
  # Get maildir mail folder name.
  #
  strPartDir, strMailDir = os.path.split( strPartDir)

  #
  # Make sure folder is in list that are subject to pruning.
  #
  if not oPrunable.has_key( strMailDir):
    return

  #
  # Go through the files.
  #
  for strFile in strNames:
    strPath = strDir + '/' + strFile

    #
    # Get file creation time. Don't look in message itself as sent time 
    # is untrustworthy. Process old files
    # 
    oStat = os.stat( strPath)
    nFileTime = oStat.st_ctime
    if time.time() - nFileTime > (3600 * 24 * 7 * oPrunable[strMailDir]):
      #
      # Attempt to parse message. Make sure it is a mail message.
      #
      oMessage = email.message_from_file( open(strPath))
      strFrom = oMessage['from']

      #
      # Archive old email comressed.
      # Shove in existing BZ2 archive in mbox format.
      # The mbox is readable by the python mailbox module.
      #
      oBZ2 = bz2.BZ2Compressor()
      oBZ2.compress( 'From %s  %s\n' % (strFrom,
            time.strftime( "%a %b %d %H:%M:%S %Y", time.localtime( nFileTime))))

      oBZ2.compress( open( strPath).read())
      oBZ2.compress( '\n')

      #
      # Create Archive directory, per maildir, per month.
      #
      strArchiveName = strRootMailDir + '/Archive.%s.%s.bz2' % (
                      strMailDir[1:],
                      time.strftime( '%b%Y', time.localtime( nFileTime)))
      open( strArchiveName, 'a').write( oBZ2.flush())

      #
      # Delete the file.
      #
      os.unlink( strPath)

os.path.walk( strRootMailDir, ProcessDir, '')

Procmail and Spamassassin

I've set up procmail and spamassassin everywhere. For an easy life I have it set up thusly:

• fetchmail scans POP servers
• exim processes received mail
• exim .forward file passes messages to procmail, e.g. (.forward)

pipe "/usr/bin/procmail"

• procmail filters messages through spamassassin, e.g. (.procmailrc)

:0fw: spamassassin.lock
* < 100000
| spamassassin

# Mails with a score of 15 or higher are almost certainly spam (with 0.05%
# false positives according to rules/STATISTICS.txt). Let's put them in a
# different mbox. (This one is optional.)
:0:
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
.Spam.Definitely/

# All mail tagged as spam (eg. with a score higher than the set threshold)
# is moved to "probably-spam".
:0:
* ^X-Spam-Status: Yes
.Spam.Possibly/

• procmail delivers messages to appropriate maildirs
• dovecot provides IMAP

There are ways to get exim to live with spamassassin directly, without procmail e.g.

• hook it directly into the virus scanning hooks of exim.
• put in a pipe so messages go through exim, out to spamassassin, back into exim via a second process and out through a different rule.

I put procmail into the equation for a few reasons:

• I find many procmail recipies on the internet, not many exim filter recipies.
• The exim online documentation is comprehensive but turgid and you have to totally understand how an MTA works to grasp any of it. The procmail documentation I have found is far more accessable.
• The .forward->procmail->spamassasin is simple and clean.

I haven't bothered setting up a server daemon for spamassassin which would speed up the email handling. Checking each email takes about 15 seconds but this is ok, I can wait that long.

Observations:

• DNS Block Lists (NOT RBL's) seem to work as DNS servers. Nice for getting through firewalls.
• Spamassassin takes advantage of Vipuls Razor and Pyzor which are essentially databases of known spam messages, but unfortunately these don't work through firewalls
• Procmail is nice: I've set it up to do all my delivery filtering.
• For procmail delivery rules, it is important to remember to put a trailing / if the target is a maildir directory. This has caught me a couple of times.
• procmail can handle spaces in maildir directory names if you use quotes, e.g.

:0
* ^From:.*Somebody
".Messages from somebody/"

• Nice being able to test spamassassin from the command line:

spamassassin -D < mailfile

• It's taken me a few hours to learn that there are four s's in assassin. It seems to take an eternity to type.

Gentoo emerge Error

For a while now my emerges have all been reporting an error:

--- !found obj /usr/share/doc/gcc-3.3.2-r5/i686-pc-linux-gnu/libstdc++-v3/html/e
xt/lwg-defects.html
Traceback (most recent call last):
  File "/usr/bin/emerge", line 2605, in ?
    unmerge("clean", ["world"])
  File "/usr/bin/emerge", line 1852, in unmerge
    retval=portage.unmerge(mysplit[0],mysplit[1],portage.root,mysettings,unmerge
_action not in ["clean","prune"])
  File "/usr/lib/portage/pym/portage.py", line 2494, in unmerge
    mylink.unmerge(trimworld=mytrimworld,cleanup=1)
  File "/usr/lib/portage/pym/portage.py", line 5318, in unmerge
    mymd5=perform_md5(obj, calc_prelink=1)
  File "/usr/lib/portage/pym/portage.py", line 2485, in perform_md5
    return perform_checksum(x, calc_prelink)[0]
  File "/usr/lib/portage/pym/portage.py", line 354, in perform_checksum
    return fchksum.fmd5t(filename)
IOError: [Errno 5] Input/output error: '/usr/share/doc/gcc-3.3.2-r5/i686-pc-linu
x-gnu/libstdc++-v3/html/ext/lwg-active.html'

This error occurred at the end of emerge when it was trying to delete old packages so it was not too bad (it had already installed the new stuff).

Looking it detail, it transpired that the file in question, lwg-active.html, indeed is corrupt: catting it gives an IO error as well.

Deleting the file and running 'emerge clean' carried on past this until it found another corrupt file. In all about 20 files gave this error, mostly files called Changelog*.gz in various directories.

Next task is to run 'reiserfsck' but that involves booting from a floppy or CD and mounting the drive read only. fsck gives no errors. Depending on what reiserfsck says do I:

• change filesystem
• change hard disk

I cannot live with an unreliable file system. I'd rather use FAT.

Buttons

Found a neat program for generating buttons and wallpapers.

Drupal Wiki Formatting

I have been fiddling for days on getting decent wiki formatting working in Drupal. My requirements are modest (ahem):

• Bullet points entered typing a simple asterisk and a space: no toolbars, nothing more elaborate.
• New style hyperlinks, far easier than restructuredtext e.g:

    [go here|htttp://www.here.com]

• Facility to put in raw html, specificly source code snippets that have been syntax highlighted using vim.
• No need to edit snippets, such as inserting indents in every line, just adding a start and end tag.

Drupal's problem, acknowledged in the forums, is that the system of filtering that is used to convert the raw text to the formatted html does not really work very well: the filters step on each other's toes. My specific problem was that code snippets I put in would be altered by later filters, specificly text in square brackets would be converted to hyperlinks that didn't work. The wiki module also managed to interfere with itself: every line was transformed unless an exclamation mark was put at the start of each line: a tedious amount of extra editing.

I tried a number of options:

• try to get the latest phpwiki source to work with Drupal. I got this to a state where it was kinda working but it continually messed up external links. The code was hard work, complex object model, no useful comments (only FixMes). I gave up on this as my head was hurting from banging on a brick wall.
• try using textile. I didn't like this ones syntax.
• try using bbcode but the syntax is worse, virtually html using [] instead of <>.
• try using the markdown module but this is not free for commercial use.
• hack on the existing wiki module, adding the concept of blocks that are not processed by other forms of markup within the wiki module itself.

I've gone for the last option, I've added two forms of block tags:

<verbatim> tag

html entities are escaped so < > & etc can all be used:

	$text = str_replace("<p>%::%</p>","",$text);
	$text = str_replace("%::%</p>","",$text);
	$text = str_replace("%::%","",$text);
	$text .= $DontScrewWithSquareBrackets[9];
	$two = 1 & 1;

<rawhtml> tag

insert raw html, e.g. pre syntax-highlighted code

<?php
// $Id: wiki.module,v 1.8 2004/02/08 22:39:38 tdobes Exp $
/**
 * wiki.module - mostly taken from <http://phpwiki.sourceforge.net/>, v1.2.2
 */

function wiki_link($type) {
  if ($type == "system") {
    menu("wiki", t("Wiki Text Formatting Rules"), "wiki_page", 0, MENU_HIDE);
  }
}

function wiki_page()
{
  if (!user_access("access content")) {
    print theme("page", message_access(), t("Access Denied"));
    return;
  }

I'm not entirely happy with this as a solution, beating on the phpwiki code would have been better as that is a far more complete language. Maybe I will give that another try?

Update: I beat some more and did fix my problem. However, I came across other problems:

• The vertabim tag is equivalent to my code tag, it does not allow raw html to pass through
• phpwiki allows certain raw html tags to pass through, for example b, small, etc but not the font tag. The regexp matching that it uses is restrictive and does not allow tags with attributes past. This could not be fixed without hacking on the phpwiki code itself. I don't want to use a library I have to hack on directly. I could work around it by subclassing but what if the base classes get changed (ref broken PyDS problems).
• phpwiki has a 'RawHTML' plugin but that applies to the whole page, not sections thereof. Also this did not work as it relied upon the larger phpwiki structure rather then the bit that converted a string to html.
• phpwiki folk are paranoid about javascript security holes in raw html. I want this for intranet so anyone who breaks things gets sacked.
• there are also purists who think the markup should be enough and html is a cop out. They are probably not people who like to push boundaries, they prefer to erect them.

I'll stick with my hacked wiki module until something better comes along.

Parasites