Understanding Nothing

I’ve seen a few links to this new report, but they all seem to focus solely on the deliberately provocative headline numbers (“bugs cost the US Economy $60bn per year”). It’s hard to tell whether the actual numbers are real or not (they’re extrapolations of extrapolations!), but there are two quite interesting relative numbers reported:

Firstly: over 50% of these costs are incurred by users. This means that the other 50% comes from increased rework etc before the software gets released. I don’t know what the ratio is between the cost of a bug “in the wild” and the cost of rework from finding the bug, but if they were the same that would mean that 50% of bugs don’t get found until after release.

Secondly: 30% or so of these costs are “low hanging fruit” that could be avoided through implementing relatively simple testing procedures.

It still amazes me that people think that quality costs more. In software, perhaps more than any other area, quality should bring the cost down (unless, of course, you fall for the common fallacy that quality means process).

Yesterday we built our first page. Because we had to set up some database mappings it may have seemed more complicated that it actually was. If we wanted to add another page today that viewed a CD in a different way, all we would have to do is add another line (7) to the config_info in Site.pm:

5
6
7
8

  sub config_info {qq{
      /view/cd gets MDB::CD
      /show/cd gets MDB::CD
  }}

And now we just create another template in page/show/cd.tt.

Today we’ll build another similar page, for artist details.

In the database schema we had yesterday, we assumed that “artist” was a field in the CD table. In reality this would usually be a foreign key to an artist table, with, say, ‘artistid’ and ‘name’ columns. So let’s migrate our system to that.

First, we create the MDB::Artist class, telling it what table the data lives in, and setting up the has many relationship back to the CDs by this artist:

1
2
3
4
5
6
7
8
9

  package MDB::Artist;
 
  use base 'MDB::DBI';
 
  __PACKAGE__->set_up_table('artist');
  __PACKAGE__->has_many(cds => 'MDB::CD');
 
  1;
</code>

Then we let the CD class know that its ‘artist’ column doesn’t contain the artist name, but rather a pointer to the artist table (line 7):

1
2
3
4
5
6
7
8
9

  package MDB::CD;
 
  use base 'MDB::DBI';
 
  __PACKAGE__->set_up_table('cd');
  __PACKAGE__->has_many(tracks => 'MDB::Track');
  __PACKAGE__->has_a(artist => 'MDB::Artist')
 
  1;

Now, any time we ask a cd for its artist we’ll get back an MDB::Artist object instead of a plain string,

So, in our template, where we previously had

<h2>[% cd.artist %]</h2>

we now need

<h2>[% cd.artist.name %]</h2>

And now we can build our artist page. Again it’s very simple. We just add it to our config_info (line 8):

5
6
7
8
9

  sub config_info {qq{
      /view/cd gets MDB::CD
      /show/cd gets MDB::CD
      /show/artist gets MDB::Artist
  }}

And now we can create our artist template, page/show/artist.tt:

  [% META browser_title = "Artist details" %]
  <h1>[% artist.name %]</h1>
  <p>CDs:
     <ul>
       [% FOREACH cd = artist.cds.sort('year') %]
       <li>
         <a href="/show/cd/[% cd.id %]">[% cd.title %]</a>
       </li>
       [% END %]
     </ul>
  </p>

And of course we should really go back to our CD template and link the artist name to their own page:

    <h2>
      <a href="/show/artist/[% cd.artist.id %]">[% cd.artist.name %]</a>
    </h2>

And now we have a simple browsing mechanism from a CD to its artist to all that artist’s CDs.

Scott does the math(s) with all the 9s. Reminds me of the If 99% Were Good Enough lists. (I can’t find the one I used to have, but this one is the same idea). Of course, it wouldn’t surprise me in the slightest if every one of the items on this list actually happened; things like “20,000 incorrect drug prescriptions will be written this year” not only sound plausible, but probably understated to me…

The problems of course, are that most people believe that 99% is good enough, and most businesses don’t even strive for anything that good. And that every extra 9 you want usually costs about 10 times as much as the previous one, and most people can’t justify the investment.

At BlackStar, I like to think that the extra 9s easily paid off in customer loyalty, industry awareness, and press coverage.

Day 13 of Mark’s Accessibility Tips is “Include Real Links”.

This is a personal bugbear, but not for any of the reasons Mark lists. Me, I hate javascript links because I use Mozilla with my middle mouse button bound to “open link in new tab”. Which, of course, only works with real links. JavaScript links don’t want to open in a new tab for me, they usually want to spawn a pop-up window. But I don’t want lots of windows. That’s why I switched to Mozilla. Now I can still have lots of pages open at once without my windows explorer bar being completely unreadable.

Unless you decide that what I really want is popups.

We already have a database built of the freedb data (it’s a little out of date, but we can resync that later). So the basic version of the first page is fairly simple. We’ll add a page to display the details of a given CD, and use it as an example of how FireCore works.

To construct our Model we use Class::DBI. We’ll see more detail of what’s possible with it later, but for now all we need to do is use it to set up a basic class to represent the CD table. As we’re using MySQL, we’ve set up our MDB::DBI class as a subclass of Class::DBI::mysql, and told it how to connect to the database. Then we can use the fact that Class::DBI::mysql knows how to query the database for the schema of a table, and the only code we need to write to represent the CD is:

1
2
3
4
5
6
7

  package MDB::CD;
 
  use base 'MDB::DBI';
 
  __PACKAGE__->set_up_table('cd');
 
  1;

Then we add a line to the Site configuration to create the page to view the cd:

1
2
3
4
5
6
7
8
9

  package MDB::Site;
 
  use base 'FireCore::Site::Decl';
 
  sub config_info {qq{
      /view/cd gets MDB::CD
  }}
 
  1;

Each page that we wish to create for the site will get a line in the config_info.

The sytax for this is quite simple. Here we’re saying that the page “/view/cd” will take an additional argument (e.g. /view/cd/12039) which will represent an MDB::CD object. FireCore will take that argument, validate it, retrieve the MDB::CD with that ID, and pass it to the template.

Then all we need to do is set up the template, which will be “page/view/cd”. Tomorrow we’ll look at setting up some basic abstractions for the pages, but for now we’ll just create a simple HTML page.

FireCore automatically creates header and footer templates that will surround each page, so we don’t need to worry about those, other than to set the page time, so we create a template like:

  [% META browser_title = "CD details" %]
 
  <h1>[% cd.title %]</h1>
  <h2>[% cd.artist %]</h2>
  <p>Length: [% cd.length %]</p>

FireCore uses Template::Toolkit as its templating engine, so we enclose our template directives inside [% %]. As we get passed an MDB::CD object this will be available as ‘cd’ inside the template. Because it’s a Class::DBI object, it will automatically have all the columns of the table available as methods on the object. So to print the title of the CD, we simply say: [% cd.title %]. Now we can visit our CD page and make sure everything looks OK.

The most obvious thing missing now is track information. That lives in a different table, so we need to set up a Track class:

1
2
3
4
5
6
7

  package MDB::Track;
 
  use base 'MDB::DBI';
 
  __PACKAGE__->set_up_table('track');
 
  1;

Simple. And then, we need to tell the CD class that it has a relationship with the Track class (Current versions of MySQL don’t store this sort of information in the schema, so we can’t auto-detect this):

1
2
3
4
5
6
7
8

  package MDB::CD;
 
  use base 'MDB::DBI';
 
  __PACKAGE__->set_up_table('cd');
  __PACKAGE__->has_many(tracks => 'MDB::Track');
 
  1;

The only addition is line 6, where we tell the CD class to create a method ‘tracks’ which will return a list of all tracks whose ‘cd’ column is the same as our primary key: i.e. it will execute the SQL:

  SELECT *
    FROM track
   WHERE cd = $cdid

Template Toolkit gives us a looping construct that lets us then insert all the tracks into our template (ordered, of course, by their position on the CD):

  <p>Tracks:
     <ol>
       [% FOREACH track = cd.tracks.sort('position') %]
       <li>[% track.title %]</li>
       [% END %]
     </ol>
  </p>

And now we have our simple CD details page. Beyond setting up some basic Class::DBI representations in a purely declarative manner, note that we didn’t write a single line of Perl code. In fact, once you have your tables all mapped in this manner, which in practice is usually part of the set-up phase of the system, rather than done piece by piece like this, you can create quite a lot of your site without ever writing any Perl.

FireCore is much harder to actually set up than it should be. Once it’s all configured for a site it’s wonderful (as we’ll see later), but actually getting to that point is much too complex. There’s a whole range of things that need to be remembered: configuring the database, adding the requisite controls to the web server config files, setting up the directory structure for the templating system, creating the basic Perl subclasses for this site, setting up the revision control systems etc, etc.

This ends up taking at least half a day, and there’s always things that get forgotten or being done not quite right. Unfortunately we can’t even use a deferred install model where each bit only gets set up as it’s needed, as all of these are needed from the outset. We really need to find a way to automate this as much as possible. Most of it’s just cut’n'paste’n'tweak anyway, replacing the domain names and paths with the correct values. Something that asks a few pertinent questions and then just goes and makes everything happen could reduce this sequence to a matter of minutes rather than hours.

I wrote a few days ago about building systems bottom up. We’re hoping to apply this approach to The Music Database. And we’re going to document our sucesses and failures as we go.

The system we’re using for this is called FireCore. I’ll talk in more detail about it later, but for now I’ve written an Introduction explaining the basic concepts: Read more…

In 1997, after leaving NIWEB, but before starting BlackStar, I launched The Music Database. It was a simple concept, with big plans. The Internet Movie Database (IMDB), was one of the most useful sites on the Web [at this point it hadn't yet been sold to Amazon]. But there was no equivalent for music. A few noble attempts had been made, but they relied on the population of the net building them up from scratch, and none had gained the critical momentum necessary.

And then I had an interesting idea. I’d been using CDDB for a while to save me having to type the tracklists in for the CDs I was listening to on my computer. A few emails later I’d become an official mirror of the CDDB, and was busily converting their information from a series of flatfiles into a relational database. Then I put together a simple web based browse and search engine, and we got a local designer to knock up a simple design for it. The Music Database, version 1.0, was alive.

At this point there was over 50,000 CDs in this database, which made it much larger than most of the versions trying to bootstrap from nothing. Of course, as all the data was entered by people just to see it appear in their CD players, not in a database, there was little consistency of spelling, or even adherence to the CDDB’s rather loose specification for how info should be entered. But, we had a couple of media trainee placement students with us for several months, and we got one of those to start cleaning the data up. The nine different spellings of Einstürzende Neubauten were consolidated; multiple pressings of the same CD that differed only in the last song running two thousands of a second longer were removed. And the daily feed of updates and new CDs from CDDB got larger and larger. People started linking to us. And the traffic kept growing.

We had two features that made the initial version a success. Firstly not only was information on the band you were interested in probably there, you could actually find it. We were pretty much the only music site that was able to find things by “The The”. Secondly, you could see what compilation albums an artist had appeared on. Most music sites treated compilation albums as second class, only really listing what they classed as “significant” appearances – usually on soundtracks of big films.

But as it grew we started getting swamped by emails. People started pointing out corrections, which we could fix, but we preferred to get them to get a CDDB aware player, put their CD in the player, and resubmit the changed version back to CDDB. Then not only would we get the fixes, but everyone who used the player. But we also got lots of emails from people saying “How do I get a copy of this CD”. We initially linked to NTK, one the nascent online retailers, but then we noticed that most of the CDs that people were asking about weren’t actually available.

So we launched MDB version 2. This time it had a simple button beside every CD allowing visitors to buy or sell their CD. There was no fancy eBay style transactional engine behind this. People just entered the price they’d sell at, or the price they’d buy at, and these details, and the user’s email address would be displayed to anyone else who was interested. Thousands of CDs were listed within the first few weeks, and the levels of email dropped to a sensible volume again.

We had great plans for version 3 that would allow users to start entering extra information for the CDs, to start turning this into a great service, equivalent to IMDB.

Then we got a call from the aforesaid retailer, asking if we could license them the data we were using. We explained that it was mostly just CDDB data, and that they could license it from them. They explained that they had looked at CDDB and it was the fact that we had “cleaned” the data up that interested them. We sought some legal advice and learned more about copyright law for data than anyone should ever need to know. Although legally we probably could license the cleaned up data, it didn’t seem right. CDDB was GPLed, and the data had always been listed as such also, although this notice had recently disappeared from the download files, possibly because the GPL couldn’t apply to data.

We eventually came up with an agreement that kept both sides happy. We would license the database of Valley – the distributor that most of the online retailer used as a dropshipper, and write a piece of code that could turn the CD’s unique fingerprint (using a different algorithm from CDDB, whose approach had significant clashing issues with shorter CDs) into a reference into the Valley database.

Unfortunately CDDB weren’t happy. Despite our assurances that we weren’t licensing “their” data on, and despite us sending NTK to talk to them to see if there was anything they could license directly, they accused us of going behind their back and stealing their information, and subsequently removed our feed.

We had several ideas for how to get around this, but instead we took the money we’d made from the deal with NTK, and started BlackStar. Anni, whom we’d just employed to keep cleaning up the data, and contact lots of music sites telling them about MDB and getting them to link to it, handled the customer care for BlackStar instead. She was phenomenal in this role, and almost single handedly responsible for setting the standard of care that made BlackStar legendary.

BlackStar grew so quickly we didn’t have time to go back to the MDB, which, starved of new information grew staler and staler over time.

NTK went on to merge with CDnow. CDDB went on to become a commercial organisation, and launched CDDB2, leaving many of their early supporters and partners feeling screwed over. The freedb was launched as an alternative non-commercial version. It now has over half a million CDs.

And now the time is right to bring back The Music Database, built, this time on Freedb. We’ve a few new plans for it, but mostly the plan remains the same. There still is no music equivalent to IMDB. It’s time to change that.

In March 1998 we launched BlackStar. Everyone we talked to said “It’ll never work”. Many of them said “if this internet selling thing will work at all, Virgin will do it, and you can’t compete with them.”

When we got our first round of VC funding in the summer of 1999 people said “Now you have a chance to build your brand before Virgin move on line”.

When we got our second round in 2000 people said “now you get to consoldidate your position before Virgin arrive.”

Now Virgin have arrived.

They don’t seem to sell videos or DVDs. And they really don’t get what this Internet thing is all about:

The CD Search and buy is a small slice of what you’ll find in our stores and is just the start of what we plan on line. In time, we’ll be offering you the whole range of products here that you can also find in store.

Yesterday when everyone was oohing and aahing about Dave Sifry’s Moveable Type hack to show related stories using the Google API, I thought “that’s nice … hmmm … it shouldn’t be hard to do that in Radio. I’ll try to knock that up at the weekend.”

Jake’s beaten me to it.

Now to retitle all the posts that don’t bring back good results! And read all the interesting relating articles that I haven’t seen before!