Understanding Nothing

If there is one “scientific” discovery I am proud of, it is the discovery of the habit of writing without publication in mind. I experience it as a liberating habit: without it, doing the work becomes one thing and writing it down another one, which is often viewed as an unpleasant burden. When working and writing have merged, that burden has been taken away.

The habit is also liberating in a much more profound sense. I consider, for instance, EWD975 on the Theorem of Pythagoras – of all theorems! – a major acievement of last semester. It was an unadulterated pleasure to write it; one of its recipients called it “absolutely bloodly marvellous”. Had I only written with publication in mind, it would never have seen the light of day. Not only would aforementioned recipient have been denied his enjoyment, I would have been so too: what I sent out was Sunday’s version, which I could only write after the discovery that I could still improve on Saturday’s version. The only way to discover that a neglected or ignored topic is worth writing about is to write about it.

The EWD-series seems a little unusual. If it is sufficiently unusual to represent a novel style of doing science, the development of that style may be one of my major contributions.

– EWD1000

An interesting thread over at kuro5hin on the tribulations one person had when writing a filter to convert XML documents to LaTeX using XSLT (with examples of how some of the transformations would be handled using Perl or Haskell instead).

My initial reaction when faced with problems such as the one described in this post (lots of highly repetitive code) is always to use a different language to generate the code – e.g. use Perl to generate the repetitive XSLT. Then if you want to add a new substitution, just add it in Perl, and generate the resulting XSLT.

I still think that I’ll probably end up going down that path for the Music Database (probably using Template::Toolkit to process the .xsl files when they’re requested), but first I’m going to see how far I can stretch the abstraction in XSLT itself.

In the story above, several people pointed out better approaches to the problem, which make the ‘correct way’ much more readable and maintainable. But it’s still not great.

XML, per se, is more verbose than I’d usually like to deal with, so I’m probably going to end up with some level of abstraction into this – but I don’t want to get there too early.

This paper sets out to show that XSLT is a true functional language, by implementing 35 of the most common functions that you would encounter in such a language (foldl, map, minimum, sum, sumTree etc)! It’s long (the PDF version is 76 pages) – mostly because of the code: just because XSLT can implement all these things, that doesn’t mean it’s easy, simple, elegant or clean!

I’m enjoying playing with the new musicdatabase chain of CDDB data -> MySQL -> SQL -> Class::DBI -> FireCore -> Template::Toolkit -> XML -> XSLT -> HTML -> CSS -> display, but XSLT is jumping out as by far the most verbose, ugly, and unreadable link in that chain. It hasn’t been around enough yet, or been used widely enough yet, for there to be much written on on the speed/cost of development, or maintainability.

In theory, the really complex stuff should be sufficiently abstractable to allow for building up quite powerful libraries. Although XSLT doesn’t support higher order functions directly, it’s possible to simulate them, as this paper (and its examples) shows. If this sort of thing can be turned into a “sufficiently encapsulated hack”, then it may indeed be possible to write XSLT relatively quickly and neatly.

I’m trying to learn how to abstract common elements of XSLT away. For example, the MusicDatabase’s shiny new XML output gives the run-length of a CD, or a track on it, in seconds. But, for output, we’d really like to show it as minutes and seconds (i.e. 308 would become 5:08).

This is fairly simple to do:

  <xsl:value-of select="format-number(runlength div 60, '#')"/>:
  <xsl:value-of select="format-number(runlength mod 60, '00')"/>

However, I don’t want to have to repeat that everywhere, a) because I’m lazy and don’t want to have to go to the effort of finding the file it’s in every time I want it, and cutting and pasting from there, and b) because I worked that out myself on my second day of playing seriously with XSLT, and I’ll probably discover later that there’s a much better way, or that it doesn’t cope properly with some fringe case, and I won’t want to have to go through and change it everywhere it’s used (see Laziness above). Oh, and c) because it’s always good practice to do this sort of stuff, for lots of reasons that never seemed as convincing as laziness.

Initial research into this persuaded me that there were two main approaches: parameterisation (see Creating Generic XSLT Transforms), or named templates (see The Tao of Recursion: Named Templates in XSLT, or Math and XSLT [with its 'Calculate pi using Leibnitz recursive named template']).

Named templates seemed the best way to go, so I added a secs_to_mins template:

  <xsl:template name="secs_to_mins">
    <xsl:param name="secs" />
    <xsl:value-of select="format-number($secs div 60, '#')" />:
    <xsl:value-of select="format-number($secs mod 60, '00')" />
  </xsl:template>

And then called it:

  <xsl:call-template name="secs_to_mins">
    <xsl:with-param name="secs" select="runlength"/>
  </xsl:call-template>

(It didn’t work at first, as I’d left the $ signs off, but it everything was fine once I realised that).

Then all I had to do was move it into its own file, and place at the top of any XSL file that wants it:

  <xsl:import href="lib/secs_to_mins.xsl" />

Wahey! Now I can build lots of nice components to play with.

Now I just need to find a good way to play with XSLTunit or equivalent to build up a test suite for these…

I had an neat “CSS” idea today that I’ve never actually seen anywhere: giving your <body> tag an ID or CLASS. Then, if you have a stylesheet that operates across your entire site, you can style the same element in different ways on different pages by setting up rules along the lines of:

  A.cdlink { color: red; }

  #compilationPage A.cdlink { color: green; }

Simple and effective, but strangely absent, as far as I can see, from all the CSS books I have …

I spent most of today trying to integrate AxKit with Template::Toolkit.

Unfortunately there isn’t really a lot of information on this available that I could find. I think most people see them as either/or approaches, rather than complementary. There is a mailing list for the two combined, but it’s fairly quiet – and seems to be more interested in using TT as a replacement for “XSLT”. I wanted to come at it from the other end – use TT to generate the “XML”, and then apply the XSLT to that XML using AxKit.

So, I decided to just blunder into it all, and see if I could make it work. It really didn’t seem like it should be that difficult.

My first attempt was to merely chain the two together:

  <Location ...>
     PerlHandler MDB::Site->handler AxKit
  </Location>

Surprisingly, everything worked first time. I had TT output a calculated page, containing <?xml-stylesheet href=”/xslt/test.xsl” type=”text/xsl”?>, and AxKit happily used this to transform the page into a lovely “HTML” rendition. Or so I thought, until I tried it in IE and it fell over. After a lot of head-scratching I realised that actually AxKit wasn’t even getting near my output, and that Mozilla was applying the transformation client-side! IE was attempting to do the same, but failing.

After some more digging I realised that FireCore was set up to return DONE if it generated output, which would stop Apache even trying to fall through to the AxKit Handler. The server merrily sent out XML and my browser merrily transformed it. And whilst this would be perfect in a few years time, it’s not much use now, when most browsers can’t be relied on to do this properly. I wanted to do it server side.

So I changed FireCore to return OK instead of DONE, falling through to AxKit when it was done. But, when you chain together two PerlHandler calls, Apache will just call each as if it was two separate requests – not in sequence. So, whilst FireCore generated the XML from TT just fine, AxKit died horribly complaining that the XML file didn’t exist, outputting a bizarrely hybrid XML/200 error page!

I couldn’t find anything obvious in the Apache or the AxKit docs, and a quick google search didn’t reveal anything either, so I hopped onto the #axkit channel on IRC, and asked there. But everyone must have been at lunch or something, as there was complete silence.

After a little more digging around I discovered that AxKit allowed you to set up your own Provider class for when your source wouldn’t just be on disk. I was slightly perturbed by the “don’t even consider doing this without asking on the mailing list about it first” warnings everywhere, but it was my best approach for now. It actually wasn’t too hard. I just made my own MDB::AxKit::Provider class my Handler, and had it pretend to be the Apache handler and ask MDB::Site for the output for that URL. (FireCore hadn’t really ever expect to be queried in this way, so I had to tease apart a few things that were too tightly coupled, but it was a fairly trivial change, and a worthwhile one anyway)

But then, the nice people of the AxKit IRC channel all arrived back from wherever they had been, and pointed me to Apache::Filter, which lets you build up a chain of Apache Handlers, each of which operate in turn on the previous one’s output. Even though I’d gotten my own sublcass working, this seemed like it would be much cleaner in the long run. Apache::Filter only works with modules which have registered themselves with this, but AxKit has thoughtfully done so, and so all I needed to do, instead of the fancy subclassing, was to tell FireCore to register itself with Apache::Filter also, and then set up my Apache config:

  PerlModule Apache::Filter
  AxConfigProvider Apache::AxKit::Provider::Filter

No actual code needing written is always a good thing, in my books.

And, hey presto! Everything worked. Well, in Mozilla anyway. IE complained: “The XML page cannot be displayed”, as there was an unclosed meta tag.

I examined my XML. I examined my XSLT. I couldn’t find such tag. I looked at the page source in my browser, and there was indeed an <meta content=”text/html; charset=UTF-8″ http-equiv=”Content-Type”> tag. I examined my XML again. I examined my XSLT again. No sign of that anywhere.

The nice people on #axkit set my straight again. The W3C say that there should be such a tag, so AxKit helpfully inserts it if you haven’t set one up. No-one could work out why IE would care though. After a lot of head-scratching and playing with different things, I discovered that it worked in one IE window, and not in another! It seems IE had decided to cache the fact this file was XML much earlier when the transformation wasn’t happening at all, and was ignoring the fact that it was now HTML. Grrr.

I still didn’t like that my page wouldn’t validate as XHTML however, so it was suggested that I explicitly set an output method in my XSL: <xsl:output method="xml" media-type="text/html" />

And now, everything was wonderful. Apart from the HTML all being one big long line that you had to scroll horizontally forever to read if you wanted to view the source. Another slight modification later, and everything was even more wonderful: <xsl:output method="xml" media-type="text/html" indent="yes" />

They even showed me where to find all this information myself :)

So, now that Marc seems to have the search working again, and I know how to get everything working in an XML-ish fashion, we can back on track with getting the MusicDatabase back up an running again!

Jeremy wants to know how to get Use.Perl journals’ RSS feeds. Yes, the documentation is incorrect (or at least out of date). The way that seems to work at present is:

http://use.perl.org/journal.pl?content_type=rss&uid=10878

To get the number (10878) from the name (matts) involves going to the person’s journal page (http://use.perl.org/~matts/journal/), and hovering over one of the little yellow smileys and reading the uid from the status bar at the bottom of your browser!

No, it’s not what I would call easy, simple or obvious! :)

Also, the RSS feed produced uses <dc:date> as the identifier for the last update (most other blogging tools use <lastBuildDate>), so my fancy “when was this page last updated” tool needed modified for them also…) [Later: Doh! The "date" is NOW, not the last update. Doesn't seem like there's any way to get last update time from the RSS at all...]

More braindead software alert.

Last night I was at a friend’s house and we decided to order pizza from Domino’s. I very rarely order from them, so I can never remember which pizza it is I like – it’s either the “Hot and Spicy” or the “Tandoori Hot”, but I never know which. So, as we didn’t have a menu, I decided to look it up online.

The Dominos web site is one of the most dysfunctional sites I’ve come across in quite some time:

1. They don’t have a menu on-line.

I couldn’t believe this, and still can’t. I still think there must be something I’m just missing. But I just can’t find the menu. I can find out how many calories etc are in each type of pizza in their “Food Guide”, but nowhere actually tells me what toppings are on each pizza!

2. You can’t look at the list of products until you register

You can get a list of products in the on-line ordering section – but not until you give them your postcode, house number, name and phone number (each on a separate page). [Note, as I write this I can't even get to this section as my nearest store is closed - even though you can schedule deliveries for later!]

3. You can’t see what you’re ordering

I was impressed that you can order a half-and-half pizza, each with a different set of toppings. I was much less impressed that once it goes in your ‘basket’ it appears solely as a half and half pizza and doesn’t show you your toppings. If you trust the site this sort of thing probably isn’t a problem, but when I don’t, I’m always worried that the order will just be placed like this and the store will have to ring me and ask what was on each half!

4. There’s no way to give special instructions

I don’t like onions on my pizza. I thought that the one I was ordering had onions – but wasn’t sure (see point 1!). I wanted to write in a little box somewhere: “Please don’t put onions on the Tandoori Hot side!”. But there was no such little box, and as far as I could see, no way at all to let them know this short of ringing the store after placing the order – which would pretty much defeat the point of online ordering

4b. There’s no way to give delivery instructions

Similarly there was no way to say something like, “When you deliver, don’t go the front door as we’re not in the house, but are sitting round the back in the garden”.

5. They don’t take credit cards

One of the things about ordering online is not having to worry about always having cash. Not in Domino’s case. You can say that you’ll pay the delivery guy. Or you can pay by debit card. But not credit card. Even though the store takes credit cards if you walk in or phone them.

6. The store ignores your order

One hour later, and considerably hungrier, I rang the store to say “our pizza hasn’t arrived yet”. They took my details and said they couldn’t see the order and was I sure I didn’t order it from another store. When I explained I ordered on line she went silent for a moment and then said “Oh yes, there it is. No one saw that!” Then there was some muffled mumbling and another voice came on, saying “Good evening. This is the manager, how many I help you?”

6b. They don’t know how to handle problems

I hate it when people pass you to their manager without telling you they’re going to, and seemingly without telling their manager what’s going on either. I don’t want to have to repeat myself to several people without knowing why. I hate it even more when the manager then lies to you because they don’t realise that the original person has already told you the truth. In this case the manager attempted to explain that they were having “technical difficulties” and the order had only just come through. And so she did probably the only thing that she’d been trained to do, and offered the food for free. At least I got to give my “no onions” instruction!

They may know how to do delivery on-time, but Domino’s certainly haven’t got to grips with this whole e-business thing.

I really hate misleading error messages.

Radio just started complaining, when I tried to make the last post: [Macro error: The file "C:\Program Files\Radio UserLand\www\#prefs.txt" wasn't found.]

Of course C:\Program Files\Radio UserLand\www\#prefs.txt is actually there. A bit of googling reveals that lots of people have gotten this message, and the common solutions seem to be “reinstall radio”, “stand on your head, pull some hair out, go have breakfast, and come back”. A classic transient bug!

In other cases it seems to be that Radio’s braindead parser gets confused if you have too many quotation marks in a post. In my case changing double quotes to single quotes did the trick. Of course changing them back to double quotes after it had worked also worked. *shrug*.

Radio Userland is great when it works, but when it doesn’t it gets really annoying really fast.

Woolpert at times sounds like a Zen master. “Focus every day on being better. Every time management has a meeting, they should be reviewing improvements in process, not sitting around and talking about dead historical data.” What counts, in other words, isn’t just gathering information but implementing changes — and that too must be built into the company’s everyday operation.

This is a company that raised its on-time delivery of cement from 68% to 95% by working with Domino’s Pizza to find out how they did it!