Understanding Nothing

A wonderful, but horrific, tale of customer service escalation from Dell [make sure to read it, blog-style, from the bottom up]. A customer had been waiting for a part for a year! CS rep, who resolved the issue in 10 minutes, escalated a complaint straight to Michael Dell, as per the customer’s wishes. Michael Dell starts probing what happened from the top, and the chain moves down from VP CS, to assistant, to a Senior Resolution Specialist, to an e-Services Customer Care, to (presumably) supervisor, who balls the CS rep out for daring to escalate directly to the top rather than going through the normal process. Even though the normal process had completely failed and the customer had had to buy a new laptop almost a year ago just to be able to do their job!

I’ve never understood this concept that senior executives don’t have time to listen to customer complaints. In many ways it’s the best way to get a handle on how the business is really doing. Every complaint that comes in to your company is a sign that something that you thought was working actually isn’t. Unfortunately most businesses don’t take their complaints and actively ask “What do we have to do to make sure this doesn’t happen again”. And I’m coming more and more to believe that this simple question is the single most important differentiator between a good company and a bad one.

[via Mike Daisey]

Software engineers have long told their bosses and clients that they can have software “fast, cheap, or right,” as long as they pick any two of those factors. Getting all three? Forget about it! But United Space Alliance (USA), NASA’s main shuttle support contractor, had a mandate to provide software that meets all three criteria.

[via Simon Willison]

This is a nice article on how using the right tool for the job [in this case Python] can shrink your development time significantly. It’s a great piece of advocacy for the Python folks, but also for the whole a dynamic scripting language as a “real” language approach.

Unfortunately it reinfornces the usual Perl issues: “Without a lot of documentation, it is hard to grasp what is going on in Java and C++ programs and even with a lot of documentation, Perl is just hard to read and maintain.”

I know no-one believes me, but I’ll just yell it again anyway: It doesn’t have to be like that! Well written Perl can be as readable and maintainable as any other language.

Unfortunately most of it isn’t. It’s too easy to written terrible Perl. And it’s too acceptable to write terrible Perl. And unless software manages start insisting that their developers’ Perl is as readable and maintainable as it would have to be in any other language, then we’re always going to hear stories like this.

Good software, in Humphrey’s view, “is usable, reliable, defect free, cost effective and maintainable. And software now is none of those things.”

A further thought on this. I’ve seen lists like this many times now. Often they even leave “maintainable” out – “fast, cheap and right” is the usual mantra.

But the one that almost everyone always ignores is extensibility.

Most software is built on pre-existing software. Throwing the existing code away and starting from scratch rarely happens. New versions are built on old versions. And newer versions built on that. Eventually the code becomes a hulking tangled mess of special cases, exceptions, add-ons, hack-ins and brain numbing control flow.

And so programmers get tempted by the “throw it away and start again” syndrome. And other programmers write articles on how that’s the worst thing you can possibly do.

But yet very few people seem to write articles on how to build your software in a way to deliberately facilitate building on top of it later.

Of course people think that Analysis and Design and Specifications and all those sorts of things are important. But this is different. Those things can only help you build what you already know you’re going to need. Once that’s worked out most software is then built in the “it works, we’re done” way I talked about earlier. Occasionally, it even gets refactored into something readable.

Here we’re talking about what Paul Graham refers to as Bottom Up Programming. Every time you need to write something you stop and think “I have to write X … hmmm … that’s quite tricky … but … it would really be quite easy if someone had already written Y and I could just build on that”. Then you go build Y (or if you’re writing in Perl, you go off to CPAN and discover that someone else has written it for you!). Of course, building Y should make you think “that would be quite easy if someone had already written Z”. And so on.

By the time you’ve worked your way back up the stack you’ll have the original “X” built, but you’ll also have a suite of helper code that will enable you to build many other components trivially later.

It’s often been noted in computer programming that the best programmers are an order of magnitude (or more) quicker than average programmers. But no-one ever explains why. I never quite grokked this until recently. The crucial point is that the difference isn’t obvious straight-away. Give the super programmer and the normal programmer the same initial task, and there’ll probably not be that much difference. Occasionally the super programmer will even be slighter slower.

But ask them both to now build Phase 2 on top of their code, and watch the super programmer leave the average programmer in the dust. They’ll have built so many useful tools doing the first phase that if this new requirement is even vaguely similar to the first they’ll be done in no time — probably whilst the average programmer is still untangling the logic of the existing code to work out where it’s best to add the IF statements.

In the last year I’ve worked with numerous clients who wanted web sites built. They’ve known that what they were asking for now was just Phase 1, and that in 3 months time or so they’d want Phase 2. And then Phase 3. Not one of them seemed to realise that how Phase 1 was built would effect how much Phases 2, 3, 4, 5, etc would cost. No-one asked about this when we were pitching for the job. None of them seemed to believe that a later phase of what they saw as similar complexity could be expected to be cheaper or quicker than any prior phase.

They assumed that if Phase 1 cost £20,000 and took a month, then 5 phases would cost £100,000 and take 5 months. Most of the companies pitching for the work, probably quoted for it as if this would have indeed been true.

But if done right, the economics could really be closer to:

In total, they’d end up paying £38,000 instead of £100,000, which would presumably be very nice for them. But more significantly, in my view, they’d be finished in 2 months instead of 5. And each new phase they wanted to introduce could be done in a day instead of a month, at a fraction of the cost they’d originally expected. If they had the ability to keep coming up with improvements and new ideas for things to build, they could accelerate away from their competition at a phenomenal rate.

But virtually no-one really believes this is possible. The common wisdom is that systems get kludgier and dirtier over time. It always takes more time to add equivalent functionality rather than less.

At BlackStar, the first significantly large system I built, this was certainly the case. We didn’t build bottom up, and we paid the price. Over time it took longer and longer to add new functionality. The code base became so brittle that every time we fixed a bug in one place it made new bugs appear somewhere else. So we introduced more stringent procedures. This, of course, made the quality of code produced rise dramatically, but it slowed everything down even more.

We’re about to launch a major new project, potentially of comparable complexity to what we did at BlackStar. The question now is whether we can manage to reverse this trend. In four years time I want it to take significantly less time to add major new functionality that it does today.

I’ll try to document how we get on.

Good software, in Humphrey’s view, “is usable, reliable, defect free, cost effective and maintainable. And software now is none of those things.” [via FuzzyBlog]

This article covers a lot of the same ground as an article I was reading yesterday by Jerry Weinberg in Understanding the Professional Programmer. He compares the “if it compiles it must be OK” approach to students who learn that their essays are being graded by a computer which takes off marks every time it finds a spelling mistake or error in the grammar. Very quickly you’ll end up with a class of students who have perfect spelling and grammar, but who have no idea how to actually communicate ideas in writing.

Thankfully we’re not quite that bad and I don’t think many people can seriously believe that “it compiles, so we’re done” is true. But I’ve started noticing a distubing trend of people misusing the XP approach of “the tests all pass, so we’re done”.

Kent Beck is fond of saying “Make it run, make it right, make it fast.”

Usually this is taken to be a warning against premature optimisation (after all that’s where the quote is mostly commonly quoted from). But it’s also a warning that you’re not done when it manages to run. You still need to make it right. I’m always amazed by how few programmers have ever read the Refactoring book – or are even aware of the concept.

Note to self: When you find an interesting article, discover it crashes your broswer when you try to print it, and have to spend about 15 minutes trying to find it again via Google because you can’t remember what it was called and you have to try about 6 different search sets of search terms, then, when exactly the same thing happens all over again, make a note of the URL somewhere.

Aside from the main thrust of this article (expressing HTML programmaticaly), there’s a very interesting categorisation of web documents. Nørmark goes further than the traditional static vs dynamic division, showing that there are at four types of pages:

1. Static – Stored pages written directly in HTML, bound at document edit time.
2. Generated – Stored pages translated from another language to HTML, bound at document generation time
3. Calculated – Calculated pages generated from a program at document access time
4. Dynamic – Calculated pages generated from a program at browse time

He only really deals with static pages, but by expressing these in a Scheme syntax he opens the way for others to show the combination of this with Scheme to create the calculated pages.

Michael Dubelko was founder and CEO of DVD Express, which he set up after Cannel Studios, where he was president, was sold to Fox News.

In an interview with DVD Hotseat he was asked “At what rate – compared to new releases – do people actually order backcatalog titles? Are back catalog titles dead or do they sell at a steady rate?”

Bizarrely the company didn’t know this sort of information – even though it was over two years old at the time, and had turned over $16m the previous year: “We don’t have any statistics on this, but I believe it’s over 50% at this stage.”

Is it any wonder the company couldn’t succeed if they couldn’t even tell things like this?

Reminds me of the scary conversations we had with the Founder/CEO of another big-name US on-line DVD retailer we were considering buying, who not only couldn’t answer, but seemingly couldn’t even understand the relevance of questions about customer retention, repeat purchase rates, product split etc. That company no longer exists either. No surprise there then.

According to Hoovers:

“DVD Express merged with Maximum Holdings, which included a network of gaming Web sites and GameFan Magazine, in late 1999. The company filed for Chapter 11 bankruptcy in March 2001 and sold substantially all of its assets to Express Media Group the following June. Investors include Maximum’s founder David Bergstein and other top executives, as well as DVD Express founder Michael Dubelko and British game maker Eidos.”

If I’m reading this correctly it seems that Express Media Group was basically the same people as the bankrupt company, (which at the time of the merger had been valued at $500m). (Eidos paid $55m for a 12.6% stake).

The Google cache still has the Press Release from the relaunch in September 2001:

Express Media Group, Inc. recently purchased the assets of Express.com, Inc. With new owners, a new management team, and a precisely focused business model, the company looks to be profitable quickly … DVD Express has left the dark days of e-commerce behind them, moving forward with a plan that will benefit both the Internet industry and its burgeoning customer base.

That version never survived the 2001 Holiday Season.

The new site still claims to be owned by Express Media Group. I don’t give it much chance either.

Back in the days when they were pretty good, I bought some DVDs from DVDexpress.com

Today I got an email from DVD.com claiming: “We have acquired DVD Express and will be continuing where the company has left off.”

Further: “DVD.com also knew that with the drastic changes in online retail, DVD Express had difficulties meeting the expectation and needs of its customers.”

And to finish: “With our new management and site we thought it would be a perfect time to say hello as the only true online DVD provider.”

So, who are DVD.com?

Well, from what I can discover, DVD.com always has been DVDexpress!

The Wayback Machine’s earliest snapshot is from Feb ’98, but is a surreal “Favorite Brands” site for JET-PUFFED;® Marshmallows!

Their November 1998 site is the DVDexpress site. It’s branded as such, and is how I remember DVD express. I assume it was simply a server alias.

By October 1999 it has been developed into its own distinct information site – but still with the DVDexpress logo at the top, and credited at the bottom as “A DVD EXPRESS information source”.

By March 2000 DVDexpress has rebranded itself simply as Express.com, and DVD.com follows suit uses the new Express.com logo. But it’s still obviously the same company.

In March/April 2001 the DVD.com information site seems to have disappeared, and merely redirected people to the Express.com home page.

By May 2001 it was a server alias again, showing the Express.com site, as it continued to be until earlier this year, when Express.com and all its various domains shut down.

So what’s going on?

It seems to me that DVDexpress, which struggled over the last few year years, and seemingly collapsed in January, has decided to try to relaunch itself. It probably has indeed got new management. And quite possibly new investors. But instead of being honest about what’s going on, it’s tried to disguise it as a take-over by DVD.com, who are obviously meant to sound like a bigger, stronger, player.

Not a very promising restart in my books…

Everyone seems to be pointing to Meg’s column “What We’re Doing When We Blog”.

The “a-ha” moment for me was the paragraph:

Freed from the constraints of the printed page (or any concept of “page”), an author can now blog a short thought that previously would have gone unwritten. The weblog’s post unit liberates the writer from word count.

As Jonathon keeps pointing out “blogging can offer infinitely more than journalism”. I think this is one of the major differences. Blogging is mostly just sharing thoughts. Thoughts that, if you had to write 400 words on, whould never escape your own head. But in a format that allows, encourages, and even expects you to write less than 100 words, the thought gets written.

Of course, most of these thoughts probably aren’t really worth sharing.

But they’re out there. Waiting for someone to pick them up and expand on them or refute them.

I recently re-read a lot of Paul Graham’s writings on lisp, and decided this time I was actually going to find out more about his concept of using closures to simulate web pages acting as subroutines (in Lisp in Web-Based Applications).

The two ‘definitive’ papers on the topic seem to be by Christian Queinnec: The Influence of Browsers on Evaluators or, Continuations to Program Web Servers, and Inverting back the inversion of control or, Continuations versus page-centric programming.

The basic concept is that thinking of a web site as a set of pages is the wrong approach – instead you should think of it as an extended computation, where interactions with users suspend the computation and later resume it. When implemented as continuations the compuation can be reified automatically.

This seems to be one of those subtle, but significant, changes in mindset that could change how you even approach building a major site.