Understanding Nothing

InfoWorld has an article about the decline in spending on web services. All well and good. However, there’s an interesting little aside in the middle:

“It’s tough out there,” said Fred Holahan [of] Silverstream, which was acquired last month by Novell. “IT managers are increasingly being asked to do more with less, and they’re facing a psychology that tells them, ‘Hey, you bought a ton of technology 18 months ago. Why not make that work?’” Unfortunately, Holahan argues, the savvy IT manager knows that the software and tools he bought a year or two earlier are not exactly cutting edge anymore, but his hands are tied to upgrade. Yet, for those IT shops still prepared to invest in new tools and application servers, it has become a buyer’s market.

Is it really that hard to build something with an 18 month old technology? Why does a ‘savvy’ IT manager really need to be investing in cutting edge tools anyway?

Maybe it’s just the case that these IT manager were oversold technology for technology’s sake 18 months ago, and it actually doesn’t do anything useful? What’s to stop them making the same mistake again – buyer’s market or not!

There was an interesting article in April’s issue of Management Science (the paper linked to has a 2000 date, but seems to be the same paper), on how customers behave in invisible queues (such as when they email on-line retailers, or have to wait for a call-center call to be answered).

The authors note research that show that customers adapt their behaviour to their perceptions of what the wait time should be, formed through accumulated expecience, offset by how important their query is to them.

Contrary to common belief, they show that customers can and do change their expectations and accepted delay time based on perceived system performance – in one study of abandoned calls to a call center, they discovered that 38% of calls were abandoned across all delays at peak times between 100 seconds and 240 seconds.

And so they spend a lot of time with graphs and formulae (and circles and arrows and a paragraph on the back of each one…) showing how to model customers’ patience levels.

This is all well and good of course, but seems to miss the more important question of how to actually set customers’ expectations properly in the first place.

At BlackStar we implemented an auto-response system to incoming emails that said something like “Your email is 24th in our queue, and so we should be able to answer it within 50 minutes.”

It was a fairly simple approximation, with no complex predictive behaviour at all – it merely averaged the time taken to answer each of the last 10 emails. But it was good enough. At peak times, or when we were flooded with emails, the time people were told they’d have to wait rose, and we got a lot less customers deciding we hadn’t answered quickly enough and sending another email, or even telephoning (setting off a terrible vicious cycle that could sometimes take days to work our way back out from).

Loads of customers commented on how wonderful it was, several competitors approached us asking how they could buy the technology, and it was mentioned in at least 3 or 4 press reviews as a sign of how customer friendly we were.

We didn’t need or do any fancy impatience modelling – we just knew that the best way to keep the customers happy was to tell them what you were going to do, and then do it.

Jon Udell talks about client- vs server-side scripting with Radio: Radio pages, for example, are dynamically generated but statically served — a strategy that I like very much on the whole, but sometimes chafe at when nifty features like TrackBack and your search extender show up in MT.

For some time now I’ve been considering moving away from Radio. I like a lot of its features, but whilst I can muddle my way around Radio’s macro system, I find it much easier to do things in Perl. And, as the pages are generated, rather than calculated, there’s lots of nifty things I’d like to do, but can’t.

And then it struck me, that actually I could solve both of these problems in one simple step – SSI.

So, I decided to try to implement one feature I’ve been wanting for quite some time: a “further reading”-type feature, like that at dive into mark etc, that tracks referrers back:

I turned SSI on in Apache for my blog directory, and wrote an Apache Logging Handler that throws the referrer information into a MySQL database. Then I wrote a simple CGI script that tallies up the referrers for a given page, and outputs them as HTML.

Then in my Day Template, I added the line:

<!--#include virtual="/cgi-bin/getlinks?date=<%longDate%>" -->

Now when Radio generates my page, it will include the SSI command in the static page, which in turn will get called at page request time, calculating the HTML to insert in the final document.

Hey presto! Server-side scripting, in Perl, with Radio.

(My only nit is that I have to pass a date with spaces in it as an argument to the CGI script. If anyone knows how I can reformat the ‘longdate’ macro output in Radio, I’d love to hear …)

Scott’s back, with a Marketing 101 piece on how Great Customer Service is Great Marketing. I couldn’t agree more.

Back in 1998, when we decided to launch BlackStar, we had $10,000 from our MusicDatabase deal with NTK. I coded the site, Jeremy designed it, and Darryl cut all the deals. We had one salaried staff member – Anni, the best customer care person in the universe. We certainly had no money for advertising for at least 6 months, and even then we could only stretch as far as ads in Empire and Total Film – hardly the media saturation we embarked on when we suddenly found ourselves with millions in VC money a couple of years later. But we maintained a 35% month-on-month growth rate for each of those first 18 months, and I’d have no hesitation in saying that our Customer Care was the number one factor behind that.

What many people seem to miss is that there are two main types of customer service – particularly in a web environment. There’s both the “dealing with customers who contact you”, and the “making it so that customers don’t have to contact you”. Amazon, who have the chutzpah to declare themselves as the most customer focused organisation on the planet, are great at the second, but terrible at the first. If you slip through the cracks of their system, they’re lost. Several times I’ve been engaged in a cycle of emails with service staff who just don’t understand my problems and assume I mean something completely different because they know how to deal with that.

At BlackStar we bootstrapped our entire system, only expecting it to actually work 90% of the time, and so we gave our care staff wide flexibility in what they could do to solve problems. Whilst it might seem that this is a very expensive approach, it actually saved a lot of money. The last 10% of any system always accounts for 90% of the development cost, so we were able to implement things in a rough and ready way, put them live, deal with the cases where they didn’t work, and only bother solving them (well, eliminating 90% of those cases) when they were happening too often. This allowed us to make changes to our site and back-end systems practically in real time, enabling us to actually grow at that 35% per month rate.

In the process we not only built the best systems around (it only ever did what we and out customers actually needed it to do – not lots of ‘cool’ functions that someone had dreamed up and spent a year designing and implementing, only to find that no-one used them), but led to an approach to customer care that led to us winning several major awards, getting tons of press coverage, and having the happiest customers in the industry, who would gladly recommend us to everyone they knew.

The systems never worked “correctly” all the time, so there was never any attempt to hide behind them. Instead everyone was trained to just do whatever it took to sort out the problem. If that meant going down to Virgin or HMV and buying the video off the shelf to send to the customer, then that’s just what we did.

If it meant giving a customer a free video each month until our credit card processing abilities could include American Express, well, then that’s what we’d do. (It cost us about £30, forced us to sort out an obvious deficiency in our system, and turned an irate customer into a fan).

If it meant sorting out a problem that a US visitor had in getting the instruction manual for their home gym that they’d bought from Sears, well, we even did do that too. (The person never became a customer, but we got a nice write up in the Wall Street Journal out of it!)

We actively saw customer care as our major marketing tool. And it worked. I think many small companies know this instinctively. Word of mouth is the only marketing they can get. Big companies seem to forget it though. BlackStar certainly has.

b-may wants to see more phrases from Macbeth in common use. Personally I’d like to hear more people describing themselves as having a mind full of scorpions, becoming borrowers of the night, and making sure they don’t become niggards of their speech.

And whilst alcohol certainly still provokes sleep and urine, I think it’s time that nose-painting came back into fashion…

Karen comments on the Tiny Perl Server Pages article in August’s Dr Dobb’s Journal, saying that it seems familiar as “we have implemented a system that works in an similar way to this one.”

Although on the surface this article may imply this, due to its comments on session management, MVC architechtures and database tables of pages etc (the article in the magazine has much more precursive discourse than the online version), but on closer inspection the TPSP code itself doesn’t really achieve any of this! In fact it’s just a poor imitation of PHP in the JSP/ASP mold.

FireCore (our website development) system is really at 180 degrees to this sort of system. To explain I’ll go back to 1995 when I wanted to start to add some dynamic elements to the websites I’d been building. At that time there were really only two approaches: SSI and CGI.

SSI (Server Side Includes) were ‘special’ HTML tags that I could use within my HTML to execute commands. Most commonly used with the ‘X-bit hack’, all you needed to do to change a plain HTML file to one that got pre-parsed by the server was ‘chmod +x pagename.html’.

Then to, for example, include today’s date, you would write HTML that looks something like this:

<p>Today's date is <!--#echo var="DATE_LOCAL" -->

Or to give the date this file was last modified:

<p>Last modified: <!--#flastmod file="index.html" -->

The other common use was to include the output of a different program (such as a hit counter):

<p>There have been <!--#include virtual="/cgi-bin/counter.pl"--> hits since 1st January 1995.

For simple changeable text within a web page this method was fine, and I used it quite happily for a few months. But it wasn’t enough for pages where the entire content would change – for example a search page. For that I was told I needed to learn CGI.

CGI is basically SSI inside out. Instead of your HTML containing some code that gets included, your write code and bind it in your webserver to be executed when a given URL is requested. Your code then gets executed and generates the HTML that will be emitted back to the browser.

In some ways this distinction is quite subtle, and basic ASP pages rarely differ that much from basic CGI pages. But as you start to do more advanced things the differences start to get greater, particularly as you start to deal with the problem of separating your presentation from your content. All sorts of people have attempted to describe how an MVC architecture can be implemented with a Server Page approach, but in general they all fall down as the View is driving everything rather than the Controller. With a CGI approach, you can get a better separation of concerns, as the CGI itself can act as the Controller.

Of course just as SSI has come a long way with ASP, JSP, PHP etc., CGI has also come a long way with many different approaches for each language. One of the most common approaches though, and the one that FireCore uses is mod_perl with Apache. Although many people run mod_perl almost exclusively with Apache::Registry, a simple wrapper around traditional CGI scripts, mod_perl is much more powerful than that. In FireCore’s case there are no CGI scripts at all. Each request gets handed to the FireCore::Request factory object, which examines the URI and loads the relevant Controller object which can examine the parameters passed, fetch the correct Model object(s) and pass them on to the View. The goal is that for most pages, all that is needed is the correct template file for the View to parse, but it is possible to intercept at any point in this, and do whatever you need.

Karen further asks, on this note, if all across the world companies that use Perl for building web based applications are all busy writing the same thing. Well, I don’t think this is at all a phenomenon unique to Perl (in fact I’d say there’s probably more code re-use in the Perl world than with most other languages), but I think it’s probably true. After all no-one else’s code could ever be as good/clear/specific to what we need as my code, could it? And some wheels are just meant to be triangular.

With FireCore we tried to avoid reinventing too many wheels! Really FireCore is just the initial mod_perl handler and a collection of useful generic Controller classes for database-backed web-pages (Viewer, Editor, Eraser, Uploader, Searcher etc.). The Model is handled by Class::DBI, and the View by Template::Toolkit – both well supported, actively maintained, widely used suites of modules available from CPAN.

This lets us build our sites really quickly, freeing up enough time to go reinvent some other wheels instead.

Brett Morgan references rebelutionary’s comment on rb3′s response to the Salon article on how many companies end up buying ‘solutions’ that Just Don’t Work(tm).

According to Brett, people honestly believed to get VC funding, you had to be running completely buzzword compliant. You had to be running Oracle on Sun boxes, switched with cisco switchgear, served using BEA, and load balanced using F5 load balancers.

Strangely, at BlackStar, we encountered most of this sort of thinking from advisors, partners and competitors – not from the VCs themselves. Yes, there were a few raised eyebrows, and probing questions, as to the scalability of our homegrown Linux/Apache/MySQL/Perl systems (not just for the site, but a fully integrated CRM / Warehousing etc. system), but when it came down to it they just trusted us to do the right thing and seemed quite impressed by just how cheaply we could do everything.

Probably the most vocal detractor of this sort of solution was David Friedensohn of BigStar, who told us at length how it would never scale, etc. Well, BigStar are pretty much defunct now, and from what I can see from their filings they ended up spending almost $17m on their web site to bring them just over $18m in sales. At BlackStar I’d be surprised if we spent $2m to build our systems to a level where they were supporting an equivalent level of sales…

Steve complains that this series hasn’t progressed in a few weeks. The next entry in the series was going to be a look at searching, using Class::DBI::mysql::FullTextSearch. But, when we used it we discovered that DBIx::FullTextSearch (which this uses) just doesn’t scale well. There’s well over 7 million tracks in the Music Database now, and searching them in that way just isn’t fun.

So, we spent a few weeks trying other solutions, and eventually settled on Lucene (which DBIx::FullTextSearch tries to replicate). It’s super lightning fast and solves all our problems. Of course, it’s in Java, so cue much playing with Inline::Java. We eventually got it working, but it’s still not tidied up to the extent that it’s a one-liner in FireCore, so I haven’t been able to write it up yet :(

I’ve also had a few people ask me questions about Class::DBI from this, so whilst we’re waiting, I’m going to put together a basic introduction to moving to Class::DBI.

Normal service should resume shortly. :)

When we first introduced mandatory code review at BlackStar, where another member of the development team had to review the code before it went live, we ran into the perennial problem of how long such reviews should take. Some people took twice as long as others, but this diligence would often pay off when they found a subtle bug that may have otherwise been missed.

In this article from Crosstalk, Robert T. McGann proposes a formula that can be applied to decide what rate of review/inspection is best for your organisation based on the cost of review vs the cost of later rework.

A graph of time spent vs excess labour time shows quite a clear “law of diminishing returns” in what McCann calls “Murphy’s Tongue”, named after Murphy’s Law “because any variation from the optimum, no matter how well intentioned, will increase development costs”.

Actually using the model to get an optimum “lines of code per hour” rate (91.29 in the example given!), is probably overkill for most organisations, particularly as you need to fill in over 20 variables that most non-CMM level 4 or 5 groups probably won’t know. And groups that can use it will need to adapt it slightly to cope with individual review speeds if reviews aren’t done by the entire team. But the basic idea is quite simple, quite obvious (once you’ve seen it), and, I would assume, quite rare.

Jeremy refers back to my response to his response to the OnLamp article, berating the fact that Radio doesn’t work with Movable Type’s TrackBack feature.

Well, I did a bit of digging and discovered David Watson’s post on how to make this work.

I tried it out, by inserting the following into the previous post:

<% scratchpad.s = tcp.httpClient (server:"jeremy.zawodny.com", path:"/mt/mt-tb.cgi?tb_id=43&url=http://www.tmtm.com/insanity/2002/07/14.html", ctFollowRedirects:"5"); string.httpResultSplit (scratchpad.s) %>

It seems to work, but seems to add multiple ‘pings’ to Jeremy’s trackback page. I’ll turn this into a radio macro once I discover how to work out what the URL of the current post will be, once created…