Understanding Nothing

Matt Wardman, asking what a “bicycling Parliament” would look like, compares the salary and benefits packages for Norwegian MPs to those in the UK. I’m perennially dismayed by how infrequently this sort of comparison takes place, particularly in Britain. It’s as if there’s a feeling of “We invented modern democracy and everyone should be studying us. What could we possibly learn from anyone else?”

I’ve recently been comparing the UK’s Freedom of Information laws to those of other countries, and the answer there, as always, is “quite a lot, actually” (Particular kudos on that one to Estonia where virtually all government information is automatically electronically published and doesn’t need to be specially requested.)

Generally, however, making such comparisons is trickier than it ought to be. Unless you can find a report from some organisation that published the results of an comparison of a particular area of information, it requires lots of searching through primary sources and trying to work out whether you’re comparing like to like. And that’s not including all the cases where what the laws say bears only a vague resemblance to what actually happens.

Wikipedia is a good starting place for high level information, and provides a basic (and generally well-referenced) comparison on economics, tax rates, and some legal topics (e.g. Freedom of Information and Age of Consent).  There are also good pages gathering country-specific information from a variety of sources (e.g. visa-free travel with a British passport), but on other topics (e.g. comparing the remuneration of MPs or equivalent around the world) the information is either well hidden or not gathered.

Is there some other source for this sort of comparative study, generally? If not, should there be, and if so where? Is it just a matter of seeding pages on Wikipedia for the relevant topic, and hoping people flesh it out and keep it up to date? Or is there a better alternative?

Timothy Fritz has a very interesting blog post on Continuous Deployment at IMVU (subtitled “Doing the impossible fifty times a day”), detailing how all committed code gets automagically pushed to their cluster of servers assuming it passes all tests. One very nice aspect of it is that the change is first put live on only a small set of their machines. Then if there’s any significant variation in a series of metrics tested on those machines (load average, errors generated, etc.) the revision is automatically rolled back rather than pushed to the remainder of the cluster.

In the comments someone raises the question of how such a system can work when database schema changes are required, describing this as the “achilles heel of partial cluster deployment”.

At BlackStar we didn’t have a system anywhere near this advanced, but we did have a requirement to have as close to zero downtime as possible and so we needed to come up with a system for putting database changes live in a way that couldn’t break code in the meantime.

One of the most common schema changes in an evolving system is the gradual migration of all 1-1 relationships to 1-many or many-many. (Someone recently posited that a database archaeologist could tell the age of a system by how many 1-1 relationships still existed. I can’t remember who or where, though. Leave a comment and I’ll credit them.)

So, for example, when you start out, it’s common to have an ‘email’ column in a ‘user’ table. Eventually, though, it will become necessary to handle a user who needs to use two or more different email addresses. The obvious solution is to split out an ‘email’ table, migrate all the existing data into it, and update the code to use that table instead of the ‘user’ table. However, when you can have different machines potentially running different versions of that code (the “before” and “after” versions) simultaneously, then you have problems. If you put the database schema changes live first, then the “before” versions will suddenly break. If you put the code live first, then the “after” version won’t work until you change the schema. In an environment where down-time is acceptable, then you just turn everything off, make the schema changes, push the new code, and you’re fine. But what to do when it isn’t?

Well, then you need to do everything in stages:

1. First, you need to create the new table. No code uses it yet, it’s simply a schema change, so you can safely make it go live.
2. Once deployed, you change any code that writes email address to write to both places. Users are still only allowed a single email address, but now that gets inserted into both the ‘user’ table and the ‘email’ table. Under normal circumstances such duplication is bad, but it’s only a temporary measure. Everything will be properly normalised when we’re finished.
3. Once that code is successfully live everywhere, you can then run a migration on all the existing data. Any new email addresses being added in the live system are being added to both tables, but before we can change any code to read from the new table, we need to make sure it’s comprehensive. So all pre-existing addresses need to be migrated. For a simple case like this you can probably use run a single SQL command; for more complex scenarios you may need a more involved script – but for those you may be better off breaking it down in to a series of migrations like this.
4. Once you’re sure that both tables are perfectly in sync, and are staying that way, you can start to migrate all code that reads email addresses to use the new table. This doesn’t have to happen all at once. In a well factored system the scope of this change should be very small, but in reality you’re likely to have code strewn all over the place that reads this data. But the doubled data source means they can gradually be eliminated one by one without blocking any other changes. (At BlackStar we generally made such changes very quickly as we couldn’t put the new functionality we wanted live until we were complete, but we also had a couple of cases where it was a much longer process that took several months to change all the code to use the new table).
5. Eventually, when you’re sure that no code reads from the old table you can remove the old code that writes to it, leaving, of course, the code that writes to the new table.
6. Once that’s live everywhere, you can delete the column. Or, if deleting a column takes too long on your system and might cause some downtime, you can just delete all the data from it, record by record if needs be. (Or, of course, you can apply a similar multi-step approach to create a new user table without this column, migrate all the code to use it instead, and then delete the old one.)
7. Now you have a system that, on the outside, functions identically to when you started – users can still only have a single email address. But that is no longer true of the underlying data schema. So you can now take whatever code imposes this restriction and fix it to allow for multiples without worrying about bringing the database into sync.

It’s a much more involved process, but at every step everything is consistent no matter which version of the code is active on a given server, everything continues to run safely, and there’s no need for any down time.

The actual time that it takes to get from stage 1 to stage 7 depends not only on how long it takes to develop the code changes, but also the gap between each deployment. If you only deploy changes once a week, it can take a few months to work through all the steps. If, however, you can get to a position where you can safely deploy multiple times per day, then you can of course be complete much much quicker. And if you only deploy once a month, or even once a quarter, well, then you have even bigger problems.

Banking in the UK is painful. Most people I know agree with this. Amongst those who have encountered other countries’ systems, it’s pretty much unanimous.

Estonia’s banking systems, on the other hand, are a pleasure to work with. All I needed to open an account was my passport (unlike the UK where you need 18 pieces of ID, 15 of which have to show a current address, dated within the last 3 days, as well as the original long form birth certificates of at least 5 grandparents), almost everything is done electronically, and everything Just Works™.

My latest delight was when I paid my internet bill yesterday. This is actually the first time I’ve ever had to do this, as I’ve just switched to a new ISP, and the service is now in my name, rather than previously when it was in the landlord’s name and paid with my rent.

As always, payment was a simple task. Log on to my internet banking account, click New Payment, enter the account number and amount (accepting all the other defaults, like when the transfer should happen [default 'now']), confirm my authorisation code, and off goes the money — in real time, no less.

However, the surprise was in what happened next. On the confirmation page I was given a message telling me that this payment was to an account that the bank recognised as one that people tend to pay to monthly, so perhaps I’d like to set up a direct debit?

Similarly to the UK, Estonia has both Direct Debits and Standing Orders. There are a variety of differences between the two, but generally a Standing Order is good for regular bills that are for a fixed amount (e.g. your rent), whereas a Direct Debit is better for a variable bill (e.g. utilities). The online version of most UK banks only let you set up Standing Orders though — for a Direct Debit you need to get the company to send you a physical sheet of paper, fill in your banking details, sign it, and then post it back to them. Electronic versions do exist, but they’re still the same amount of hassle, as they are an authorisation for a third party to take money from your bank, and so have to be supplied to your bank by that company, rather than you.

In Estonia the process is much, much, much simpler. When I clicked through to the page suggesting I set a direct debit up for my ISP bill, it had all the company and account details pre-filled. I only had two boxes to fill: the date on which the direct debit would leave each month (defaulting to today’s date), and the maximum amount that the company could take each month. Then just enter my authorisation code again, and it’s done. Very simple, and very effective.

It’s the simple things like this that I enjoy so much about Estonia. The UK seems to have an inbuilt desire to make absolutely everything overly complicated. People raise all sorts of reasons as to why the simple version will never work, and slowly but surely all the edge-cases take over and suffocate the original idea. But no-one ever seems to really notice that it’s happening, and at the end everyone is proud of the robust system that they’ve created, convinced that it couldn’t be any better. For some reason that doesn’t seem to happen so much in Estonia: the simpler solution tends to win out against the complex one. And, curiously enough, I pretty much never hit any problems, whereas in the UK I’m constantly falling through some gap or other in the massively over-engineered version that’s supposed to cope with every contingency.

I can’t quite work out where the difference comes from though. I thought at first that it was because Estonia got to start pretty much everything again from scratch less than 20 years ago, whereas the UK has to cope with lots of legacy systems and a “Can’t get there from here” problem. But now I think it’s deeper than that: that there’s actually a different philosophical mindset that changes how people building these sorts of systems tend to approach the issues. But I don’t really know what it is, or even how to find out. Pointers welcome!

Another great example of how to lie with statistics went by on the BBC Open Secrets blog this morning.

By law government authorities must reply to all Freedom of Information requests “promptly and in any event not later than the twentieth working day following the date of receipt”. Unfortunately many of them ignore the first half of this, and take it to mean that they can routinely take the full twenty working days to respond (I don’t have actual figures, but I suspect that graphing the responsese on WhatDoTheyKnow.com would show a disproportionately high percentage responding on the twentieth day).

The statement from the Home Office spokesman goes even further than this, though:

The Home Office is committed to responding to FOI requests within the timescales set by the Act but the complexity of our cases means that this will not always be possible. We have therefore set an internal target of 90 per cent which is achievable and challenging.

The arrogance here is breathtaking. In order to set targets that they believe they can achieve, they ignore the legal requirement, and set themselves a goal that involves breaking the law 10% of the time!

They then go on to boast about how much better they are now than a few years when they broke the law more than half the time. No doubt, if they manage to meet these new targets, at the end of the year there can be much rejoicing that “all targets have been met or surpassed”, and the small matter of the target being a completely ludicrous one in the first place can be quickly glossed over.

Recently, as part of my investigation into naming, I came across a curious entry in the Northern Ireland General Register Office FAQ:

• Question: Can I register my childs name in another language?
• Answer: You can register your childs name in any language you wish providing you use English characters.

I had already discovered that England and Scotland had no such restriction, so I was curious what the basis for it was in NI. Was it some sort of sectarian hold-over from a system that tried to discourage all those Irish names with their fadas and lenition? I could vaguely understand Chinese immigrants being forced to Anglicise their babies’ names, but would all the new Polish immigrants have to avoid all those ł’s and ś’s, or convert them to l and s, so that in generations to come there would be all sorts of pronunciation traps like with Menȝies?

Presumably some punctuation characters would be allowable, apostrophes and hyphens being fairly common, but what about others? Could someone have an ! or even an @ in their name?

So I emailed GRONI to find out.

At first they were mildly unhelpful, stating only: “The General Register Office will accomodate ‘accented’ character as long as the name registered is in english text. For example you asked if Irène would be acceptable. The General Register Office does accept such characters.”

So I asked about for a proper formal definition of what was acceptable or not, giving as a further example letters that might appear to be accented, but actually aren’t, like å, Ø, and š. Again the broader question was ignored and I was merely told that my examples “would all be ‘characters’ that the GRO could accommodate.”

Fearing a protracted game of Zendo, where I gave more and more elaborate examples and was left to puzzle out the rules, I gave up on the probing and asked whether it was a legal restriction (in which case, under what legislation), or purely a technical one in some system they use (in which case, an answer about what character set(s) or even Unicode code points would suffice).

After three further emails, all of which were completely ignored, I made a formal complaint. That was initially brushed aside, but eventually I got a response that said that, actually, names can be registered with “any Unicode character”, and although there is currently “a technical restriction on the signature field regarding Unicode characters”, this is “currently being investigated”.

So it seems things are better than I thought, and even 吳安娜 is allowable.

They still haven’t changed their FAQ though.

The internets are up in arms about a Florida man who has recently been sent to jail. On reddit the story even reached the front-page as: “Only in America? Man Jailed Without Bail For Being Too Poor to Resod Lawn”.

Generally the comments, whether on the original article, on reddit, on digg, or on various blogs that have also posted the story, appear to be completely idiotic. Almost everyone is up in arms that such a terrible thing could happen, with some even going as far as calling for the judge to be removed and the HOA president to be fired. If they’re lucky, that is. Some commentors seem to want either or both of them to be tarred and feathered instead.

Most seem to ignore the fact that the behaviour of dear Mr Prudente is much more imbicilic than any HOA rule. The facts are actually quite simple, if you can get past the over-sensational nature of most of the reporting. As part of the covenant for living where he does, Mr Prudente has a obligation to keep his lawn maintained in a certain manner. He entered into this agreement freely and has presumably complied with it for the last ten years, but has recently stopped doing so. He ignored a series of letters asking him to rectify this, so ended up being taken to court to enforce performance.

Rather than attending and pleading poverty, he chose to not even show up for the trial. In a civil matter the court process basically boils down to a simple formula: each side gets to make their case in turn, and the judge rules which is most persuasive. If you don’t bother to turn up, it’s taken as you not making any case at all, and the other side will pretty much always be more persuasive than you! So, unsurprisingly, he lost and was ordered to bring his garden into line within 30 days. A month later, having still not done so, he also had fees awarded against him, presumably after again not turning up to argue why they shouldn’t be.

In September, 3 months after his “Do this within 30 days” period had expired, having continued to completely ignore the order of the court, the judge found him in contempt. Judges generally don’t like being ignored. But the judge gave him a final chance, offering him another 30 days period to comply, and warning him of the consequences of continuing to just ignore it.

A month later, Mr Prudente discovered that judges aren’t joking when they tell you “Do this or go to jail.”

It’s really a simple system. When a court orders you to do something, you either comply or appeal. You don’t just stick your fingers in your ears and declare “Nah! Nah! I can’t hear you!”

Is the rule that requires homeowners to keep their grass maintained in a certain manner a stupid one? Possibly. I doubt I’d want to buy a house somewhere that has that sorts of requirement. But mostly that’s irrelevant. The “victim” here agreed to it. Should he go to jail for breaking it? Absolutely not. But he hasn’t. He’s gone to jail for repeatingly ignoring a court order. And should people go to jail for that? Absolutely.

I could blog every day about the insane laws that are being passed with alarming frequency in the US and UK, and rant about where the legal systems are running wild. But this isn’t one of those stories. It isn’t even close to one. This is simply: “Man sent to jail for continuing to refuse to do what a judge tells him.” Please find something better to be outraged about.

On a regular basis I go driving or cycling with a GPS in logging mode. This creates a nice big GPX file of where I was on a second-by-second basis.

Whilst on such travels, I often take photos. My camera timestamps these photos, but doesn’t have a built in GPS.

So, I want something that takes the photos, and the GPS tracks, and writes the GPS information into each of the photographs.

So far, so good – there is all manner of software out there to do this.

However, this relies on the clocks in my GPS and camera being in sync, which isn’t always true (and even if I were to remember to synchronise them every time I go out that doesn’t help with old photos which are really badly out of sync, such as from my round-Australia drive where I was crossing timezones on a regular basis and would often forget to update the time on my camera for several days).

GPSPhotoLinker, which I normally use, is almost good enough here, but not quite. It allows me to adjust the time on all my photos by a specified amount. But then I need to work out what that amount should be, and that’s a decidedly non-trivial problem.

Generally I can pinpoint fairly accurately where at least one photograph was taken, but then I need to manually find that point in my GPS tracks to work out what time I was really there at – which is also non-trivial as if I’ve stopped to take a photograph, and my GPS is logging every second, I’ll have a variety of points all roughly around the point. And it’s even worse if I’ve been past that same point several times. GPS Visualizer is quite helpful here as it will plot a GPX file onto Google Earth with timestamps as waypoints, but it’s super slow, and really doesn’t like big GPX files (of the type you might get if you drive 300+km a day logging every second…)

So what I really want is something that will take the GPS track, and a set of a photographs, and plot the photos on a map for where it thinks they were taken. Then I should be able to take one or more of the photos and drag them to where they were actually taken, and the software will use that new information to adjust all its timings and replot all the other photos. Then, when I’m happy they all look like they’re in roughly the correct location, it can write the GPS info to the photos.

Does such a thing exist?

If not, how difficult would it be to even write a little utility that does the plotting of photos onto Google Maps and then lets me “try out” different scenarios simply by giving it an offset and having it move all the photos accordingly, until I get a good enough result by trial and error? It doesn’t sound like it would be too difficult, but there are too many components of it I don’t know anywhere near enough about to actually do it.

Anyone any thoughts or pointers?

“The Home Secretary’s record on private information … displays a reckless disregard for the privacy of other people’s confidential information, matched with a determination to keep Government data just as secret as he or the Government choose at any one time. The Labour party believes that the balance should be struck differently and that it is the individual who needs protection in two ways. First, private information relevant to him or her should be protected against the state, and secondly, information possessed by the state that might result in detriment to the individual should be made available to that individual.”

– Roy Hattersley, Data Protection Bill debate, House of Commons 1983

Under English law, what dictates that you must have a name?

At first glance it seems like a stupid question: of course you have to have a name. But do you? And if so, from when?

You don’t get given a name the instant you’re born, so there’s obviously some length of time during which you’re entitled not to have one. My first thought was that it was to do with the birth being registered, which is controlled by the Births and Deaths Registration Act 1953. However, although all births must be registered within 42 days, there is no obligation to register a name at that time. At any time during the first 12 months the Registrar can add a name to the original record.

And although it might be possible to infer from this that you must therefore have to have a name by the end of your first year of life, it doesn’t actually say so. So if it’s a legal requirement it must come from somewhere else.

But where?

Suggestions welcome.

Today, within minutes of each other, I received two emails.

The first, from LoveFilm, opened: “Dear {CUSTOMER_INFO_FIRST_NAME}”.

The second, from Easyjet, greeted me: “NAME: [[Firstname]] [[Surname]]”.

Doesn’t anyone pay attention any more?