Understanding Nothing

Last weekend I braved a visit to the UK for Social Innovation Camp Scotland. My remit had been to be a roving expert, flitting from team to team, but I was so impressed by the first team I started working with that I ended up staying with them the whole weekend — right through to the point where we won!

Yesterday the News of the WorldSunday Times ran a rather hilarious scare-mongering piece about it. They quote Calum Steele, general secretary of the Scottish Police Federation, as saying: “the police service already have ways for the public to express dissatisfaction”.

The point that this so magnificently misses, however, is that those ways aren’t good enough. Local councils already have ways for the public to report potholes, graffiti, broken streetlights etc. — yet FixMyStreet flourishes. All public authorities already have ways for the public to make Freedom of Information requests — yet in just over a year WhatDoTheyKnow has already grown to handle about 10% of all such requests. The National Health Service already have ways for the public to provide feedback — yet over 7,000 people have preferred to use Patient Opinion (who have also managed to pull off the neat trick of getting the NHS to pay them to deliver complaints to them.)

There are many reasons why someone would prefer to use these sorts of sites rather than going directly. For some it’s purely practical: in many cases it’s much easier to visit a single easy-to-use site with a consistent interface rather than navigate the more, erm, interesting, waters of official sites, some of which still sport “Beware of the Leopard” signs.

For others it’s the communal nature of these sites, where others who have experienced the same problems can chip in with support and advice, or even just learn that they’re not alone.

But for me it’s all about how transparency reverses the balance of power. For too long too many government agencies have forgotten that they are meant to be our servants, not our masters. Before they will engage with us, they make us jump through hoops that do nothing but frustrate us, in the guise of making their lives somehow easier (though usually anyone with any inkling of business processes can’t help but wonder how it possibly ever could). And more often than not, complaints get the stonewall or runaround treatment, and those who persist often get little more than a bland not-quite-apology with no indication that anyone ever took the time to engage with the matter, and certainly no sign that anything might actually change as a result.

The simple act of moving all this out into the open changes things dramatically. Everyone knows that “what gets measured gets done”, and, in the UK at least, government bodies tend to be rather sensitive to what the public at large think of them. As such, rubbish that has been left in an alleyway for weeks has a habit of suddenly being collected rather quickly when there’s a public report of it on FixMyStreet for anyone browsing that Council’s page to view. Agencies tend to be less inclined to take 6 months to respond to Freedom of Information requests when anyone looking at their WhatDoTheyKnow page could see at a glance that they never meet the required timescales. (We’ve heard, for example, that the Information Commissioner’s Office love WDTK as now they get see all manner of patterns and common problems that are missed when only dealing with complaints that get escalated to them.)

Transparency is powerful, as the UK has learned dramatically over the last couple of months. And once it’s in place, it’s extremely difficult to remove it. A central proposition of the Open Source movement has been that “given enough eyeballs, all bugs are shallow“. I wish I were witty and wise enough to come up with an equivalent for Open Government (suggestions welcome!), but even without a catch-phrase the underlying idea still holds. Government in the open will, more often than not, be better government. In some parts of the world, this is a concept that still needs to be fought for. In the rest, where there’s at least a token agreement, even if (or perhaps especially if) it’s more honour’d in the breach than the observance, then join in. Create your own site. Shine some more sunlight. You don’t need permission. You’re already in charge. Just Do It.

It’s certainly been an eventful 6 months since I last posted.

Back in May we took over Ireland’s oldest ISP and have spent the last 6 months turning it around. They seemed to have an interesting business model whereby they would take all the revenue, give 50% of it to suppliers, 50% of it staff, and spend the other 50% on overheads.

I can’t say much more about it all here yet, as there are still six ongoing lawsuits, but I’m sure I’ll get to tell the stories some day. You really won’t believe some of them (like the story of the directors who locked themselves in their office and refused to talk to us…)

Conference Season was interesting this year, whilst all this was going on. We ended up having to skip Oscon after O’Reilly messed up our tutorials, but I got to go to FoafCamp and FooCamp in Amsterdam, the FOAF Workshop in Galway, Web 2.0 in SF, and of course we were hosting YAPC::Europe this year. I’m sure I’ll get to talk more about those later.

The takeover also meant we had to put most of our ongoing projects on hold for a while. Simon and Marc have both moved on to other things, and Marty, Karen, and I have been working full-time at UNITE.

We said we’d put pretty much everything else on hold for six months, and so it’s time to start digging some of those out again. Everything moves so fast that we’ve had to rethink some of them significantly. Twingle, of course, now has Gmail to contend with. That doesn’t worry us too much though, as we believe that Twingle’s value is a lot more than just search. More on that later too.

This blog is probably going to be different this time around too. I used to use it as a place to store interesting things I came across. I’ve now switched to using del.icio.us for that, so this will be much more about what I’m doing. Hopefully that won’t be too boring for everyone else.

I’m back from Etech, and waiting for the jetlag to kick in. I managed to stay awake until about 10pm last night, and awoke around 7am this morning, so hopefully I’ll get tired earlyish this evening, and work my way back into a normal sleep pattern, rather than the usual flying east jetlag where I can’t get to sleep until 4am. As I didn’t get to stop off on the east coast for a couple of days on the way back (as I’ve done for all my west coast visits in the last 6 or 7 years) I’m 8 hours out of sync rather than 5, so I’m not sure what difference that’ll make.

Etech itself was interesting, and fun. I found the Social Software tracks the most interesting, but was very disappointed in all the camera-phone sessions – I don’t know if it’s just a UK/EU difference, but I really didn’t come across anything interesting or new there.

Marc Smith and Donald Norman gave excellent keynotes, but Pertti Korhonen and William H. Janeway were really dull.

My main complaint about the conference as a whole is that most of the talks were too long, with not enough depth. 20 minutes would have worked much better than 40 for the vast majority of them – especially if there really was an abundance of great proposals.

This is a public service announcement. If anyone is ever thinking of staying in the Hotel Paix Republique in Paris, don’t. It’s possibly the worst hotel I’ve ever stayed in, and I’ve stayed in some terrible hotels.

I stayed there for YAPC::Europe last week; there was two hotels arranged, and this was the more expensive one – I dread to think what the other one was like. The lack of air conditioning was a major problem, and the room only became tolerable by actually going out and buying a large fan for the room. (The fact that the fan could cool the room down quite quickly was one of the few things in favour of the ludicrously small room.) Actually, there was supposedly air conditioning in the “reading room”, which was basically a 5 seater area with a small TV, where we spent most evenings playing poker. On the one occasion where they turned the air conditioning on, it was unnoticeable and we had to stealthily open the window again anyway.

The lack of shower came as quite a surprise, although if there had have been one, I’m sure that the towel policy would have been even more irritating than it was (they seem to have just enough towels to go around, so when room service cleans the room they take both the towels away, and bring them back in the evening).

I’m also not sure when I last had to sleep in a bed so small that I couldn’t lie on my back without dangling off the sides. Or had a ‘fridge’ in a hotel room that actually seemed to make things warmer rather than cooler. Or had hotel staff tell us we couldn’t play poker any more because the lights had to be turned off – only to discover that it was solely so the door staff could take over the area to watch TV.

And only having one key for each room gets annoying very quickly when you’re sharing. And it’s probably best not to think too much about the issues involved in reception staff happily giving your room key to anyone who happens to know your room number, without any identification…

Hopefully Google will now do its magic, and someone else at some other date will be spared this experience…

Basic Premise: Excessive hype hurts technology [Michael Masnick]

Lying With Statistics 101:

• “Killer App” - 96,000 results
• “Happy Customers” – 52,000 results
• “Profitable Company” – 15,000 results

I want:

• a successful product
• that meets a need
• that makes customers happy
• that makes money (somehow) (maybe)

If technology is intrinsically useful, then the Killer App will appear, almost by accident.

The Killer App is

• not just technology, but the application of it
• not the wheel, but the chariot, cart, bicycle, and car
• not the steam engine, but the steamboat and train
• not television, but Milton Berle

Don’t try to invent a killer app

• try to solve a need
• listen to users
• take advice, not dictation
• customers, customers, customers
• deliver what you promise – or more

Avoid:

• using the phrase “Killer App”
• vaporware
• believing what anyone says about you
• thinking that the most money wins
• thinking that the best technology wins
• not watching what customers want

[Michael mentioned a very interesting "Star Trek Comms Badge" device being used in some hospitals - need to find more info on this]

Bob Morris’ keynote started with a call to not only keep in mind history, but also to look at where we’re going. In the last hundred years $1000 worth of computational power has increased by 14 orders of magnitude – and is still accelerating. But, the cost of IT management has reversed – the TCO for storage is now two-thirds for management (your sysadmin) vs. one third for the storage itself (hardware and software). As well as the major costs of technology now being labour, that labour is also a major cause of availability problems. A recent survey of the cause of downtime reported the problem as being hardware 20% of the time, software 40%, and operator error the other 40%. And as eBay, AOL, etrade, Schwab etc have discovered in the last few years, that downtime can be very expensive. Technology needs to get better at managing itself.

The history of computing is a series of massive simplifications to the user experience that drives the technology (timesharing -> PCs -> GUI -> Web -> ???). But complex heterogenous infrastructures are hard: there are thousands of tuning parameters on hundred of components (a web site has to have properly configured firewalls, DNS services, caches, web servers etc.)

So, Morris wants to see autonomic systems that are:

• self-configuring (can adapt to their environment)
• self-optimising (can monitor, and tune)
• self-healing (can discover, diagnose, and react)
• self-protecting (can anticipate, detect, and protect)

Is this a pipe dream? Much of it is already happening – it’s just not holistic yet.

As an example take RDBMS query optimizers. By considering their environment, and considering the data they need to search, good optimizers can get 2 or 3 orders of magnitude performance increase (cf. compilers which are deemed to be good if they can get 20%-30% improvement). And now we’re starting to see learning optimizers, which can keep statistics on how they’re performing (disk space is cheap enough and plentiful enough now to actually keep large log files). Then they can make adjustments if they discover that they’ve gotten their cardinalities wrong.

For Homogenous Components Interacting, he gave the example of adaptive network routing, or high available clustering. He explained how Oceano‘s multiple-customer server farms uses virtualized hardware and virtualized software, to provide clustering for multiple web sites across shared servers. Although this is not a new technique, it’s still not common, as many clients still don’t really trust the security of sharing servers. However, costs are now reaching a point where the difference in price is significant enough to convince many people to swallow their fears!

He also describe recent advances in storage technology with collective intelligent storage bricks. These have much higher redundancy than RAID, and cool performance hotspots by taking proactive copies. With sufficiently improved sparing you can eliminate the need for repare actions for the life of the system. This has the added benefit of no longer needing a 2d packing structure, as you no longer need to be able walk around the machine to pull out drives. If you don’t need to replace the ‘bricks’ you can have a 3d structure, and with better cooling systems you can now get a petabyte of storage in a small cube.

He then proceeded to talk about the costs of managing client machines, used by non techies, which are usually around 50% of total time/cost, and promoted the idea of subscription computing. Many organisations have started to use this for customisation, or personalisation, or protection, or problem detection, or software updates, etc. But again it’s not that widespread, and not very holistic. IBM have recently started to reintroduce the old mainframe “hypervisor” concept to allow your machine to run multiple operating systems on the one machine, which Morris thinks will make this much easier.

Morris maintains that we need to focus on availability, maintainability, scalability, cost and performance. Systems need to be usuable by millions of people, but managed by half a person. This is a hard problem, which won’t be solved overnight, and needs the participation of academia, government and industry. More information on the project is available at http://www.ibm.com/research/autonomic

… it’s just not evenly distributed yet.

This quote from William Gibson was pretty much the text from which revivalist Tim O’Reilly preached his opening keynote for O’Reilly’s Emerging Technology conference. His basic thrust, now as always, is that if you want to spot the trends before they become mainstream, you need to watch the “alpha geeks”, where you’ll encounter magic (or at least something “sufficiently indistinguishable“), on a daily basis. Usually the entrepreneurs come after the geeks, and try to make these technologies work, in the wider sense. But there was a massive distortion recently, with too much being led by the focus of making money, and everyone trying to find the control points. Now (thankfully) the hype has died, and everything is percolating from the bottom again. The new internet operating system means that everyone can play, but this time they should learn to play better together. Everyone should figure out simple rules for co-operation, the control points shouldn’t throw their weight around, and everyone should strive to follow The Robustness Principle (aka the Golden Rule of the Internet: “Be rigorous in what your emit, but forgiving in what you accept”).

Marc Andreesen once famously denigrated Microsoft Windows as “Just a bag of drivers”. In Tim’s view this is completely correct, but profoundly wrong. That bag of drivers allows developers to forget about everything but those standards and APIs.

Tim warned that big companies should remember the fate of Lotus, who made a bet against the GUI with 1-2-3, whose market dominance crumbled in the face of Microsoft, who obviously bet very strongly for it. As a current example, he cited MapQuest, a true “killer app” of the traditional internet, but one which risks losing out to an application which allows you to extract the information you require (e.g. distances, for expenses claims, without the map or driving directions). In the future applications should aim to be part of the “bag of drivers” of the new Internet Operating System.

The schedule for The Perl Conference 6 has been announced. And, for me at least, it’s a disappointment.

When the Call For Participation was put out, the theme was announced as “Doing More With Less”. The heavy focus on Perl 6 and Parrot this year seems only to meet this if we take “less” to mean “not yet available”. I’ll be surprised if Perl 6 is in existence by TPC 7. At one level I’m looking forward to its arrival, but I don’t believe it’s anywhere near time to start making the lead talks of a conference.

For whatever reason there also doesn’t really seem to be very much else at the conference I’d really want to hear. Over on London.pm, my criticism of this led Randal Schwartz to challenge me to describe what I would want to see.

I thought about it for a day, and came back with the following. It’s not enough to build a whole conference around, but it shows the sorts of things that would have enticed me to go:

Doing More With Less Money

The obvious one for an Open Source conference. What are the open source equivalents to big dollar approaches. Some of this exists in the conference (an overview of the perl content management systems etc), but I’d have expected more (I’m surprised there isn’t something on RT, with its recent introduction as the bug-reporting arm of CPAN).

Doing More With Less Skill

(or some more ‘politically correct’ version of this that wouldn’t have made people think that attending was the equivalent of being seen with a shelf full of “… For Dummies” books.)

A lot of modules on CPAN have quite complex and arcane interfaces which provide you a lot of power, as long as you’re happy with closures and callbacks and anonymous data structures/subroutines etc, when actually quite a lot of perl programmers are frightened even of references.

Recently however quite a few people have been writing ::Simple modules aimed at providing a large subset of the functionality wrapped in an easy interface. I’d have liked to have seen a few talks on this sort of approach, possibly with a BOF for people who are interested in not just providing the sorts of Power Tools that let other developers do amazing
things, but providing a nice learning curve into them. (This doesn’t have to just be aimed at beginners: Damian could easily have done a bit on Filter::Simple and Attribute::Handlers which took exactly the same approach at a more advanced level…)

Doing More With Less Hassle

Even advanced and experienced Perl programmers spend a lot of time doing monotonous tasks again and again. Joel Spolksy explained in a recent article why he was moving Fog Creek gradually to .NET. One of the reasons he gave was that: “All the grungy stuff that takes 75% of the time creating web applications with ASP (such as form validation and error reporting) becomes trivial. ASP.NET is as big a jump in productivity over ASP as Java is to C.” What are the Perl equivalents of this?

Doing More With Less Time

One of the things I’ve found when building applications (usually web-based, but not always), is that most of the “heavy lifting” has been done before, and released to CPAN for me, saving me huge amounts of time. I still have to write a lot of glue code though, tying all these things together. And I know that lots of people have probably written almost identical (but probably much better) glue before me. I’d have liked to have seen some people talking about how they tie lots of packages together: perhaps a “How to Build a $20,000 website in an afternoon” session – my version would have been on how to tie Class::DBI, Template::Toolkit, CGI::Untaint, Class::DBI::FromCGI, Date::Simple and Spreadsheet::ParseExcelSimple together to provide a database-backed website based on information supplied by a client in Excel files. But I’d like to hear other people’s eqivalents.

Doing More With Less Power

I don’t really mean the “political” power question here (“My company is moving most of its development to Java / C# / whatever. I know I can get work done ten times faster in Perl – what should I do”), although that could be interesting too.

Instead, I’m talking about things like The Fractional Horsepower Webserver. Jon Udell presented a wonderful paper at TPC 2 (1998) on a desktop HTTP server that ran a distributed contact manager application. 4 years on, with desktop servers on the rise again, and peer to peer much more commonplace, what’s happening in this area?

For years the Perl world has been great at writing (unix) server based applications, but relatively poor at writing (windows-based) desktop applications. As the two continue to coalesce what great Perl-based desktop applications are being written using a local web-browser as their front end?