Understanding Nothing

I had a fascinating discussion yesterday about Freedom of Information statistics. Apparently, in the country in question (which shall remain anonymous as the actual details here are less important than the general concept), the percentage of requests that are being rejected has gone up significantly over the last year. It seems that many people think that this is a bad thing and shows that transparency is on the decline. My immediate reaction, on the other hand, was exactly the opposite: I assumed that this was a good thing, and transparency must be much greater now than before!

To explain, let me first step back to my days at BlackStar — at the time Europe’s largest online DVD retailer. Amongst my jobs there was tracking the response times to customer support emails. But, the flip side of that was to keep a close eye on what sorts of emails we were getting, to work out what we could do to no longer need to get those emails. Every week we would see which thing was taking our care staff most time to deal with that could most easily be minimised by making the site work better in the first place. Lots of people emailing us to see if they could cancel their order? Well, then we needed to push the ability for them to cancel it themselves out as late as possible: not just from when it started to be packed, but right through until the Royal Mail delivery truck actually picked it up. Lots of people emailing us to see if we were selling an item that didn’t seem to be listed on our site? Then we found and licensed a database of every video and DVD that had ever been released, even if they were no longer available. Lots of people now emailing us confused as to why were were listing an item on the site if they couldn’t buy it? Then we came up with a service to track down used copies for people. This process repeated itself time and time again and led us to have the most customer-friendly ecommerce site in Europe. However, the natural result of this was that over time the average customer support email got more complex. The low-hanging fruit were constantly being picked off so that no-one would ever again need to ask those sorts of questions. So, by definition, most of the requests we were getting were becoming significantly harder to answer, would take longer to deal with, and had a much higher chance of being unresolvable.

I view Freedom of Information requests in much the same way. Having to make a request should be an exception. Each one is highlighting an area where a government department should have been more proactively transparent. A perfectly functioning system would never require any Freedom of Information requests, as everything that would be released on request has already been published anyway. Most countries aren’t at that stage yet, but there’s a definite trend in that direction, helped greatly by initiatives like data.gov in the US and data.gov.uk in the UK. Both of these countries (and, presumably, any others doing likewise), aren’t entirely sure yet what the most important datasets to release are, though. Patterns of Freedom of Information requests can provide great insight into that. If certain types of information are being requested more than others, that’s probably a good sign that that’s something that would benefit greatly from pre-emptive disclosure.

So, to return to the original question of statistics: what would it mean if a department (or indeed entire country) rejected 100% of all Freedom of Information requests? Well, I guess it could mean that it was the most secretive, refusing to release anything at all. But it could also mean that it’s the most open, and each request made is being rejected either because the information is already publicly available anyway, or because it’s amongst the subset of information that is exempt from disclosure. The raw numbers tell us almost nothing. As always, the truth requires much more digging behind the statistics.

Timothy Fritz has a very interesting blog post on Continuous Deployment at IMVU (subtitled “Doing the impossible fifty times a day”), detailing how all committed code gets automagically pushed to their cluster of servers assuming it passes all tests. One very nice aspect of it is that the change is first put live on only a small set of their machines. Then if there’s any significant variation in a series of metrics tested on those machines (load average, errors generated, etc.) the revision is automatically rolled back rather than pushed to the remainder of the cluster.

In the comments someone raises the question of how such a system can work when database schema changes are required, describing this as the “achilles heel of partial cluster deployment”.

At BlackStar we didn’t have a system anywhere near this advanced, but we did have a requirement to have as close to zero downtime as possible and so we needed to come up with a system for putting database changes live in a way that couldn’t break code in the meantime.

One of the most common schema changes in an evolving system is the gradual migration of all 1-1 relationships to 1-many or many-many. (Someone recently posited that a database archaeologist could tell the age of a system by how many 1-1 relationships still existed. I can’t remember who or where, though. Leave a comment and I’ll credit them.)

So, for example, when you start out, it’s common to have an ‘email’ column in a ‘user’ table. Eventually, though, it will become necessary to handle a user who needs to use two or more different email addresses. The obvious solution is to split out an ‘email’ table, migrate all the existing data into it, and update the code to use that table instead of the ‘user’ table. However, when you can have different machines potentially running different versions of that code (the “before” and “after” versions) simultaneously, then you have problems. If you put the database schema changes live first, then the “before” versions will suddenly break. If you put the code live first, then the “after” version won’t work until you change the schema. In an environment where down-time is acceptable, then you just turn everything off, make the schema changes, push the new code, and you’re fine. But what to do when it isn’t?

Well, then you need to do everything in stages:

1. First, you need to create the new table. No code uses it yet, it’s simply a schema change, so you can safely make it go live.
2. Once deployed, you change any code that writes email address to write to both places. Users are still only allowed a single email address, but now that gets inserted into both the ‘user’ table and the ‘email’ table. Under normal circumstances such duplication is bad, but it’s only a temporary measure. Everything will be properly normalised when we’re finished.
3. Once that code is successfully live everywhere, you can then run a migration on all the existing data. Any new email addresses being added in the live system are being added to both tables, but before we can change any code to read from the new table, we need to make sure it’s comprehensive. So all pre-existing addresses need to be migrated. For a simple case like this you can probably use run a single SQL command; for more complex scenarios you may need a more involved script – but for those you may be better off breaking it down in to a series of migrations like this.
4. Once you’re sure that both tables are perfectly in sync, and are staying that way, you can start to migrate all code that reads email addresses to use the new table. This doesn’t have to happen all at once. In a well factored system the scope of this change should be very small, but in reality you’re likely to have code strewn all over the place that reads this data. But the doubled data source means they can gradually be eliminated one by one without blocking any other changes. (At BlackStar we generally made such changes very quickly as we couldn’t put the new functionality we wanted live until we were complete, but we also had a couple of cases where it was a much longer process that took several months to change all the code to use the new table).
5. Eventually, when you’re sure that no code reads from the old table you can remove the old code that writes to it, leaving, of course, the code that writes to the new table.
6. Once that’s live everywhere, you can delete the column. Or, if deleting a column takes too long on your system and might cause some downtime, you can just delete all the data from it, record by record if needs be. (Or, of course, you can apply a similar multi-step approach to create a new user table without this column, migrate all the code to use it instead, and then delete the old one.)
7. Now you have a system that, on the outside, functions identically to when you started – users can still only have a single email address. But that is no longer true of the underlying data schema. So you can now take whatever code imposes this restriction and fix it to allow for multiples without worrying about bringing the database into sync.

It’s a much more involved process, but at every step everything is consistent no matter which version of the code is active on a given server, everything continues to run safely, and there’s no need for any down time.

The actual time that it takes to get from stage 1 to stage 7 depends not only on how long it takes to develop the code changes, but also the gap between each deployment. If you only deploy changes once a week, it can take a few months to work through all the steps. If, however, you can get to a position where you can safely deploy multiple times per day, then you can of course be complete much much quicker. And if you only deploy once a month, or even once a quarter, well, then you have even bigger problems.

So, earlier this evening I got email from someone I haven’t heard from in about 10 years, telling me that I was featured in last weekend’s Sunday Tribune alongside an article about BlackStar, in a “What I did after I made millions”(!) piece.

So, I dutifully trotted off to the Tribune website to see if I could find the article, only discover that they don’t place their articles online like any other sensible newspaper. Instead they’ve teamed up with NewsStand who promise to delivery print versions of newspapers to your desktop. It seemed that I could get Sunday’s edition for $2, so I decided to indulge my narcissism and give it a go.

The first hurdle to a potential customer was the notice that the minimum charge was $3.50. Making people spend money they don’t actually want to is a really, really bad thing, but the dollar is ludicrously weak, so I decided to persevere.

Then the next page tells me that actually the previous page was a lie. The minimum charge is $10 – but as a special offer to new customers they’ll let me only pay them $5. This actually defies comment.

However, the small print told me that I could claim back any unspent portion of my account at any time, so I figured I’d give them that $5. But first I had to download the software. Oh yes – they have proprietary software for viewing the newspapers! A whopping great download that took 6 minutes even on my spiffy ADSL (after I managed to get to actually start downloading – their JavaScript doesn’t work on Mozilla, of course).

Finally I could download Sunday’s paper. An even bigger download that took over 15 minutes. Thankfully I could start viewing it before it had finished.

This is where it gets really bad.

Their special software is really just an image viewer – every page is just a huge image that, even with the software at full-size, can only fit half a page onto my screen. I eventually worked out how to scale the pages down so I could see two pages at a time, but then I needed to zoom in eleven(!) times to make an article readable (which of course involves clicking on the magnifying glass in the toolbar, and then clicking again on the area of the paper you want to zoom in on).

They do have a search button, but my vanity search didn’t reveal any results. I figured this was because it hadn’t downloaded the whole issue yet, but even when that finished I still couldn’t find anything.

So I spent about 20 minutes painstakingly going through every page. Nada. After all that pain, I couldn’t find the article.

There was an ad though for Interface – a new 8 page IT supplement as part of today’s paper.

But of course that’s not included with the online edition – even though every other section seems to be.

*sigh*

Now for the fun of attempting to get a refund through their woefully confusing on-line support system.

Anyone have a copy of the real paper?

IT Week this week references the Web Effectiveness Report 2003, which, amongst other things, reveals that only 19% of web site managers surveyed review their log files to look for problems with their site.

When we originally built BlackStar we were fairly novice Perl programmers. We knew that all Perl should really have “strict” mode and “warnings” turned on – so we made sure we did that. But as long as everything ran correctly, we didn’t really pay much attention to the warnings that were emitted.

But after a while we realised that the profusion of verbiage cluttering up our logs was highly distracting, and would ensure that nobody really paid them much attention – and the useful information about real problems would be obscured.

So we decided to clean this up, to ensure that anything appearing in the log would most likely be a symptom of a real-life actual problem, of which someone should probably be notified straightaway.

This was much easier to decide than to actually implement, however. We were introducing all manner of new quality procedures at the time, so it was relatively easy to at least decree that any new code, or any alteration of existing code, should be free from warnings. This way we could at least halt the growth of new warnings (although with site visits growing 30-40% per month, the volume of messages was still growing quickly). We even added into the programming schedule a little time devoted to removing some of the most egregious offenders, which probably removed about 50% of the warnings.

The slow clean up of having the old code gradually tidied in passing wasn’t going to get us anywhere fast enough though. So we introduced a new system that was disarmingly simple, but remarkably effective. Every night a process collated the error logs from across the various web servers, and generated a report of the ten most common problems. This report would be emailed to the entire programming team, and the next day one person would take responsibility for ensuring that the top item on that list vanished.

This approach proved very effective and within a few months the volume of warnings had decreased dramatically. It was so successful that we started to apply the approach to many other areas of the business. Any problem that was monitored over time was a candidate for it – and with an entirely web based business we had a lot of data to monitor. Someone became responsible for checking the list of the most common search terms that returned no results, and adding a re-mapping so that that search would automatically be transformed into what the person probably wanted. Someone else would ensure that the most visited DVD page that didn’t provide cross-reference to the VHS of the same item was given that linkage. Someone in the warehouse would upload the cover of the most visited item in stock that hadn’t previously been scanned.

Most of these tasks took less than 5 minutes. They would rarely be a top priority in amongst the constant fire fighting, growth-pains, and breakneck pace of developing new features and systems in the crazy world of exponential growth. In normal circumstances none of these things would even have appeared in someone’s list of top 10 priorities. But the simple action of ensuring that each day the top occurrence of each problem was removed created a staggering cumulative effect.

The art of time management is usually a matter of ranking your items by importance and urgency, and prioritising according to how high things appear on each axis. But most books, articles and seminars on the topic stop there. Spending a few minutes each day doing something that is neither particularly important, nor particularly urgent, but that has a beneficial outcome, has value. When everyone in an organisation is doing likewise, and those tasks are automatically selected based on their potential benefit, that value can be enormous.

The UK on-line DVD retail market is getting interesting again. Even ignoring imports, the Region 2 market was for a year or two pretty much a 2 horse race between Play.com and CDwow.

Play.com are based in Jersey and can take advantage of the no-VAT loophole. Recently, however, they’ve been gradually raising their prices, foolishly believing that customers won’t notice.

CDwow are in Hong Kong and ship from various locations around Europe (recently Sweden). [They're also under investigation from the BPI for their music imports]. They’re usually cheaper than Play but carry a narrower range.

For a long time Amazon was completely out of it – relying on their name more than their price. But with the introduction of their super saver shipping to the UK they suddenly became competitive – as long as your order was over the £25 threshold. Coupled with their 20% discounts on new releases – particularly box sets – this often made them the cheapest.

Now BlackStar have decided to become competitive again. They’ve finally brought back free postage to the UK and Ireland, and have started shipping low value items from Jersey.

So now it’s worth shopping around again. To take an example, the new Season 1 and Season 2 combined DVD box-set of 24 is now £54.99 at Play.com, £53.99 at BlackStar, £52.99 at CDwow, and £49.99 at Amazon.

Chicago is £13.99 at all of them, except Play.com where it’s £14.99 (but will incur postage at Amazon unless it’s part of a bigger order).

Teachers Season 1 is £17.99 at Play, £15.99 at Amazon and BlackStar and unavailable at CDwow.

And in back catalogue, The Pillow Book is £9.99 at Amazon, £8.99 at BlackStar and £7.99 at Play.

And there’s a few new kids on the block with cheap prices as well: PriceStorm are based out of Guernsey rather than Jersey, and DVDplus has been resurrected as part of the new THE. And ChoicesDirect are always worth checking out; they still have the 20% automatic pre-order discount, and are often the cheapest if you’re looking for something on VHS.

And that’s not counting everyone’s special offers…

Interesting times, indeed.

At the Emerging Technology Conference I heard Amazon staff say several times that Amazon is (or is striving to be) the most customer centric company on Earth. Every time I hear this it irks me more and more because I know that my dealings with Amazon have generally been pretty poor.

I don’t doubt Amazon’s sincerity in this regard, but I think they’ve got their terminology slightly wrong. I think they’re actually striving to be the most “customers centric” company rather than “customer centric”. Of course this doesn’t sound as good – in fact it just sounds wrong – but I think it’s closer to the truth.

From talking to lots of other Amazon customers, and from some of my experience with them, for the most part Amazon “Just Works”. They’ve put a lot of effort in to making sure that the general customer experience is right. But if you happen to fall through the cracks somewhere, everything falls apart and they really can’t cope.

Their customer care staff are notorious for not reading your email carefully and latching on to the first phrase in it that they have a stock answer for. I’ve had cases where this happened 3 or 4 times about the same matter before finally finding someone prepared to actually take the time to understand the problem.

When their systems work (and that’s probably 99.9% of the time now), they’re great – but it’s when they don’t that the customer relationship is at the most risk – and Amazon are terrible at handling this.

When we built BlackStar we decided to take a different approach. Because we didn’t really have a lot of money or time to make our systems work quite as well, we aimed for making 90% of orders flow smoothly – and made sure that our Customer Care staff were able to deal with all the cases that fell through the cracks. Most of the effort went into dealing with individual customers one by one rather than the abstract concept of all customers.

A few interesting things happened from this:

Firstly, we got to see which areas we were spending the most time dealing with, and so had a clear indication of where to spend our development resources.

Secondly, we found that customers actually seemed to like this approach. I’ve heard it said that a customer whose complaint is handled well ends up being more loyal than a customer with no complaints – and this definitely seemed to be the case. Our customers seemed impressed with the level of individual attention they received, and that paid off for us.

Thirdly, we found that this was even more true for our “power users”. The customers who tended to buy the most from us, tended to have the most problems, but by dealing well with them, we actually got to know them. It turned out that most of them had previously shopped with competitors, and when they’d managed to fall through the cracks there they’d received terrible service and promptly started shopping elsewhere. These customers were even more impressed with how we handled such cases as they were able to compare it with what they’d experienced elsewhere, and many of them remained customers for a very long time.

It’s seems that there’s two approaches you can take to dealing with problem cases. You can either put your efforts into ensuring they never happen (as Amazon seem to do), or accept that they will happen and put your efforts into being able to handle them well when they do (as we did at BlackStar). In reality you need to work at both, but I maintain that the payoffs are much greater from the second approach.

Whether you can ever attain that if you don’t treat Customer Care as a core competency, and outsource it, as Amazon do, is an interesting question …

BlackStar.co.uk recently redesigned their website to supposedly be XHTML 1.0.

Although it doesn’t actually validate, it has drastically reduced the download time of pages – particularly the front page.

However, someone seems to have listened to too many branding experts or something. Their homepage mentions the full URL of the site 191 times! Of course, this is all solely in the HTML source so any potential branding impact only applies to the freaky people who actually read the source of web pages…

The one simple act of making all the URLs absolute rather than relative bloats the file size of the page by over 19%. That’s over 4Gb of completely needless traffic per million page views. They must have money to burn on bandwidth or something…

Dave Cross:

Note to Play.com’s IT department – delivery address is an attribute of an order, not an account. It’s ok to have a default delivery address as an attribute of the account, but a customer must be able to override that on any particualr order.

Of course BlackStar.co.uk handles all of this stuff. Just a pity that it’s so ludicrously expensive for almost everything now…

There was an interesting article in April’s issue of Management Science (the paper linked to has a 2000 date, but seems to be the same paper), on how customers behave in invisible queues (such as when they email on-line retailers, or have to wait for a call-center call to be answered).

The authors note research that show that customers adapt their behaviour to their perceptions of what the wait time should be, formed through accumulated expecience, offset by how important their query is to them.

Contrary to common belief, they show that customers can and do change their expectations and accepted delay time based on perceived system performance – in one study of abandoned calls to a call center, they discovered that 38% of calls were abandoned across all delays at peak times between 100 seconds and 240 seconds.

And so they spend a lot of time with graphs and formulae (and circles and arrows and a paragraph on the back of each one…) showing how to model customers’ patience levels.

This is all well and good of course, but seems to miss the more important question of how to actually set customers’ expectations properly in the first place.

At BlackStar we implemented an auto-response system to incoming emails that said something like “Your email is 24th in our queue, and so we should be able to answer it within 50 minutes.”

It was a fairly simple approximation, with no complex predictive behaviour at all – it merely averaged the time taken to answer each of the last 10 emails. But it was good enough. At peak times, or when we were flooded with emails, the time people were told they’d have to wait rose, and we got a lot less customers deciding we hadn’t answered quickly enough and sending another email, or even telephoning (setting off a terrible vicious cycle that could sometimes take days to work our way back out from).

Loads of customers commented on how wonderful it was, several competitors approached us asking how they could buy the technology, and it was mentioned in at least 3 or 4 press reviews as a sign of how customer friendly we were.

We didn’t need or do any fancy impatience modelling – we just knew that the best way to keep the customers happy was to tell them what you were going to do, and then do it.

Scott’s back, with a Marketing 101 piece on how Great Customer Service is Great Marketing. I couldn’t agree more.

Back in 1998, when we decided to launch BlackStar, we had $10,000 from our MusicDatabase deal with NTK. I coded the site, Jeremy designed it, and Darryl cut all the deals. We had one salaried staff member – Anni, the best customer care person in the universe. We certainly had no money for advertising for at least 6 months, and even then we could only stretch as far as ads in Empire and Total Film – hardly the media saturation we embarked on when we suddenly found ourselves with millions in VC money a couple of years later. But we maintained a 35% month-on-month growth rate for each of those first 18 months, and I’d have no hesitation in saying that our Customer Care was the number one factor behind that.

What many people seem to miss is that there are two main types of customer service – particularly in a web environment. There’s both the “dealing with customers who contact you”, and the “making it so that customers don’t have to contact you”. Amazon, who have the chutzpah to declare themselves as the most customer focused organisation on the planet, are great at the second, but terrible at the first. If you slip through the cracks of their system, they’re lost. Several times I’ve been engaged in a cycle of emails with service staff who just don’t understand my problems and assume I mean something completely different because they know how to deal with that.

At BlackStar we bootstrapped our entire system, only expecting it to actually work 90% of the time, and so we gave our care staff wide flexibility in what they could do to solve problems. Whilst it might seem that this is a very expensive approach, it actually saved a lot of money. The last 10% of any system always accounts for 90% of the development cost, so we were able to implement things in a rough and ready way, put them live, deal with the cases where they didn’t work, and only bother solving them (well, eliminating 90% of those cases) when they were happening too often. This allowed us to make changes to our site and back-end systems practically in real time, enabling us to actually grow at that 35% per month rate.

In the process we not only built the best systems around (it only ever did what we and out customers actually needed it to do – not lots of ‘cool’ functions that someone had dreamed up and spent a year designing and implementing, only to find that no-one used them), but led to an approach to customer care that led to us winning several major awards, getting tons of press coverage, and having the happiest customers in the industry, who would gladly recommend us to everyone they knew.

The systems never worked “correctly” all the time, so there was never any attempt to hide behind them. Instead everyone was trained to just do whatever it took to sort out the problem. If that meant going down to Virgin or HMV and buying the video off the shelf to send to the customer, then that’s just what we did.

If it meant giving a customer a free video each month until our credit card processing abilities could include American Express, well, then that’s what we’d do. (It cost us about £30, forced us to sort out an obvious deficiency in our system, and turned an irate customer into a fan).

If it meant sorting out a problem that a US visitor had in getting the instruction manual for their home gym that they’d bought from Sears, well, we even did do that too. (The person never became a customer, but we got a nice write up in the Wall Street Journal out of it!)

We actively saw customer care as our major marketing tool. And it worked. I think many small companies know this instinctively. Word of mouth is the only marketing they can get. Big companies seem to forget it though. BlackStar certainly has.