Understanding Nothing

Whenever I teach a Dreamweaver or JavaScript class, few of the students can type. I usually spend a minute or two trying to convince them that being able to type will not only save them time and improve their accuracy, but will totally alter their relationship with the computer, giving them control and confidence that they currently lack. I don’t know how a two-finger typist could even consider programming.

Yet I know that relatively few will invest a small amount of time and money to get the big payoff. The willingness of human beings to act contrary to their own best interests never ceases to amaze me.

We were recently discussing the same thing with regards to programmers and editors. A programmer spends a huge amount of their working day interacting with an editor of some description. Most of the programmers I know use vi. But most vi users I’ve encountered learn more than the basic keystrokes and commands needed to get their work done – often using 2, 3, or even 10 times as many keystrokes as needed (I’m always shocked how many vi users will ‘right arrow’ their way to the end of a line, and then hit ‘a’ and ” to start typing a new line, and who are then amazed when I tell them that not only will ‘A’ start appending to the end of the current line, but that ‘o’ is even easier than ‘A’.

Of course, my own ‘vi’ knowledge is a little rusty too. I learnt most of what I know 10 year ago, when it really was ‘vi’, not new-fangled clones like ‘vim’. I’ve gladly switched to ‘vim’ a long time ago, mostly for multi-level undo, and syntax highlighting, but I’d never really gotten around to learning all the new fancy things you can do.

Now I’m trying to learn one new command a day. I’m not totally convinced by folding yet, and it’ll take me a while to get used to visual selections, but ‘gq’ is nice, and I’ve got great plans to integrate ctags with our configuration management system.

I wonder how many employers though would be prepared to send their employees on an ‘Advanced vi’ course. Somehow I don’t think there’d be very many…

(I got 84wpm on the test – but mostly because it doesn’t allow backspacing. My typing style involves lots of mistakes which are corrected very quickly!)

I’ve spent the last two days in Damian Conway’s “Advanced Module Implementation” course. In it he steps through, line by line, the core code of his Attribute::Handlers, NEXT, Class::Delegation, Inline::Files, Filter::Simple, Perl6::Variables, Perl6::Currying, Hook::Lexwrap, Regexp::Common, and, yes, Acme::Bleach modules to show lots of different ways in which to build modules. And twist your brain in lots of painful ways along the way.

A key refrain along the way is that of Perl Aikido – having enough mastery of your craft to know how to get lots done, not by writing lots of code, but by doing just the right thing at the right time. Most of these modules are pretty much implemented in less than one screenful of code. And whilst some of it is insanely devious (such as the infamous goto , "$imposter" } }; line), much of it is just piecing together straightforward things in an unusual way (the much more subtle deviousness of Regexp::Common that doesn’t look at all scary until you try to follow just what is actually going on).

I’ve recently been noticing a glitch in my theory of extraneous code. Almost everyone knows that a beginner in any programming language writes far too much code. They don’t yet know all the idioms of the language, and try to replicate the approaches they know from other languages, even if this language has a built in manner for doing this. As you get better at writing the language you learn the better approaches, and your code gets shorter. However, I no longer think there’s a simple progression from this point to the ‘master’ level where modules like the above ones can be written in one screenful of code. I think that ‘expert’ programmers, still on their way towards ‘master’, actually revert to writing much too much code again. When faced with a difficult problem, they know enough of the nasty things they can do to attempt to stomp through the problem. Ironically someone with less knowledge would probably not attempt this, and try to find a way around the problem instead. An expert may have lots more tools in his toolbox, but it takes a master to know exactly which one is best for every job.

Employees at the Jim Beam bourbon distillery are being limited to four bathroom breaks per 8 1/2 hour shift, only one of which can be unscheduled. Extra trips to the bathroom can result in reprimands. Workers with six violations can be fired.

When will employers stop this sort of idiocy? The company claims that this had to be put in place “because some workers abused the privilege of unlimited bathroom breaks”. But, of course, fails to explain why the company couldn’t just deal with those employees instead of punishing everyone.

It was as if it was just a spur-of-the-moment, not-thought-through-properly, type of thing either. They actually consulted with a urologist first!

Pity they didn’t consult with someone with any sense, who might have pointed out just how much productive work someone would do in an hour when the only thing on their mind was just how much time was left until their next allowable bathroom visit…

[via b-may]

When we first introduced mandatory code review at BlackStar, where another member of the development team had to review the code before it went live, we ran into the perennial problem of how long such reviews should take. Some people took twice as long as others, but this diligence would often pay off when they found a subtle bug that may have otherwise been missed.

In this article from Crosstalk, Robert T. McGann proposes a formula that can be applied to decide what rate of review/inspection is best for your organisation based on the cost of review vs the cost of later rework.

A graph of time spent vs excess labour time shows quite a clear “law of diminishing returns” in what McCann calls “Murphy’s Tongue”, named after Murphy’s Law “because any variation from the optimum, no matter how well intentioned, will increase development costs”.

Actually using the model to get an optimum “lines of code per hour” rate (91.29 in the example given!), is probably overkill for most organisations, particularly as you need to fill in over 20 variables that most non-CMM level 4 or 5 groups probably won’t know. And groups that can use it will need to adapt it slightly to cope with individual review speeds if reviews aren’t done by the entire team. But the basic idea is quite simple, quite obvious (once you’ve seen it), and, I would assume, quite rare.

I’ve seen a few links to this new report, but they all seem to focus solely on the deliberately provocative headline numbers (“bugs cost the US Economy $60bn per year”). It’s hard to tell whether the actual numbers are real or not (they’re extrapolations of extrapolations!), but there are two quite interesting relative numbers reported:

Firstly: over 50% of these costs are incurred by users. This means that the other 50% comes from increased rework etc before the software gets released. I don’t know what the ratio is between the cost of a bug “in the wild” and the cost of rework from finding the bug, but if they were the same that would mean that 50% of bugs don’t get found until after release.

Secondly: 30% or so of these costs are “low hanging fruit” that could be avoided through implementing relatively simple testing procedures.

It still amazes me that people think that quality costs more. In software, perhaps more than any other area, quality should bring the cost down (unless, of course, you fall for the common fallacy that quality means process).

Good software, in Humphrey’s view, “is usable, reliable, defect free, cost effective and maintainable. And software now is none of those things.”

A further thought on this. I’ve seen lists like this many times now. Often they even leave “maintainable” out – “fast, cheap and right” is the usual mantra.

But the one that almost everyone always ignores is extensibility.

Most software is built on pre-existing software. Throwing the existing code away and starting from scratch rarely happens. New versions are built on old versions. And newer versions built on that. Eventually the code becomes a hulking tangled mess of special cases, exceptions, add-ons, hack-ins and brain numbing control flow.

And so programmers get tempted by the “throw it away and start again” syndrome. And other programmers write articles on how that’s the worst thing you can possibly do.

But yet very few people seem to write articles on how to build your software in a way to deliberately facilitate building on top of it later.

Of course people think that Analysis and Design and Specifications and all those sorts of things are important. But this is different. Those things can only help you build what you already know you’re going to need. Once that’s worked out most software is then built in the “it works, we’re done” way I talked about earlier. Occasionally, it even gets refactored into something readable.

Here we’re talking about what Paul Graham refers to as Bottom Up Programming. Every time you need to write something you stop and think “I have to write X … hmmm … that’s quite tricky … but … it would really be quite easy if someone had already written Y and I could just build on that”. Then you go build Y (or if you’re writing in Perl, you go off to CPAN and discover that someone else has written it for you!). Of course, building Y should make you think “that would be quite easy if someone had already written Z”. And so on.

By the time you’ve worked your way back up the stack you’ll have the original “X” built, but you’ll also have a suite of helper code that will enable you to build many other components trivially later.

It’s often been noted in computer programming that the best programmers are an order of magnitude (or more) quicker than average programmers. But no-one ever explains why. I never quite grokked this until recently. The crucial point is that the difference isn’t obvious straight-away. Give the super programmer and the normal programmer the same initial task, and there’ll probably not be that much difference. Occasionally the super programmer will even be slighter slower.

But ask them both to now build Phase 2 on top of their code, and watch the super programmer leave the average programmer in the dust. They’ll have built so many useful tools doing the first phase that if this new requirement is even vaguely similar to the first they’ll be done in no time — probably whilst the average programmer is still untangling the logic of the existing code to work out where it’s best to add the IF statements.

In the last year I’ve worked with numerous clients who wanted web sites built. They’ve known that what they were asking for now was just Phase 1, and that in 3 months time or so they’d want Phase 2. And then Phase 3. Not one of them seemed to realise that how Phase 1 was built would effect how much Phases 2, 3, 4, 5, etc would cost. No-one asked about this when we were pitching for the job. None of them seemed to believe that a later phase of what they saw as similar complexity could be expected to be cheaper or quicker than any prior phase.

They assumed that if Phase 1 cost £20,000 and took a month, then 5 phases would cost £100,000 and take 5 months. Most of the companies pitching for the work, probably quoted for it as if this would have indeed been true.

But if done right, the economics could really be closer to:

In total, they’d end up paying £38,000 instead of £100,000, which would presumably be very nice for them. But more significantly, in my view, they’d be finished in 2 months instead of 5. And each new phase they wanted to introduce could be done in a day instead of a month, at a fraction of the cost they’d originally expected. If they had the ability to keep coming up with improvements and new ideas for things to build, they could accelerate away from their competition at a phenomenal rate.

But virtually no-one really believes this is possible. The common wisdom is that systems get kludgier and dirtier over time. It always takes more time to add equivalent functionality rather than less.

At BlackStar, the first significantly large system I built, this was certainly the case. We didn’t build bottom up, and we paid the price. Over time it took longer and longer to add new functionality. The code base became so brittle that every time we fixed a bug in one place it made new bugs appear somewhere else. So we introduced more stringent procedures. This, of course, made the quality of code produced rise dramatically, but it slowed everything down even more.

We’re about to launch a major new project, potentially of comparable complexity to what we did at BlackStar. The question now is whether we can manage to reverse this trend. In four years time I want it to take significantly less time to add major new functionality that it does today.

I’ll try to document how we get on.

The schedule for The Perl Conference 6 has been announced. And, for me at least, it’s a disappointment.

When the Call For Participation was put out, the theme was announced as “Doing More With Less”. The heavy focus on Perl 6 and Parrot this year seems only to meet this if we take “less” to mean “not yet available”. I’ll be surprised if Perl 6 is in existence by TPC 7. At one level I’m looking forward to its arrival, but I don’t believe it’s anywhere near time to start making the lead talks of a conference.

For whatever reason there also doesn’t really seem to be very much else at the conference I’d really want to hear. Over on London.pm, my criticism of this led Randal Schwartz to challenge me to describe what I would want to see.

I thought about it for a day, and came back with the following. It’s not enough to build a whole conference around, but it shows the sorts of things that would have enticed me to go:

Doing More With Less Money

The obvious one for an Open Source conference. What are the open source equivalents to big dollar approaches. Some of this exists in the conference (an overview of the perl content management systems etc), but I’d have expected more (I’m surprised there isn’t something on RT, with its recent introduction as the bug-reporting arm of CPAN).

Doing More With Less Skill

(or some more ‘politically correct’ version of this that wouldn’t have made people think that attending was the equivalent of being seen with a shelf full of “… For Dummies” books.)

A lot of modules on CPAN have quite complex and arcane interfaces which provide you a lot of power, as long as you’re happy with closures and callbacks and anonymous data structures/subroutines etc, when actually quite a lot of perl programmers are frightened even of references.

Recently however quite a few people have been writing ::Simple modules aimed at providing a large subset of the functionality wrapped in an easy interface. I’d have liked to have seen a few talks on this sort of approach, possibly with a BOF for people who are interested in not just providing the sorts of Power Tools that let other developers do amazing
things, but providing a nice learning curve into them. (This doesn’t have to just be aimed at beginners: Damian could easily have done a bit on Filter::Simple and Attribute::Handlers which took exactly the same approach at a more advanced level…)

Doing More With Less Hassle

Even advanced and experienced Perl programmers spend a lot of time doing monotonous tasks again and again. Joel Spolksy explained in a recent article why he was moving Fog Creek gradually to .NET. One of the reasons he gave was that: “All the grungy stuff that takes 75% of the time creating web applications with ASP (such as form validation and error reporting) becomes trivial. ASP.NET is as big a jump in productivity over ASP as Java is to C.” What are the Perl equivalents of this?

Doing More With Less Time

One of the things I’ve found when building applications (usually web-based, but not always), is that most of the “heavy lifting” has been done before, and released to CPAN for me, saving me huge amounts of time. I still have to write a lot of glue code though, tying all these things together. And I know that lots of people have probably written almost identical (but probably much better) glue before me. I’d have liked to have seen some people talking about how they tie lots of packages together: perhaps a “How to Build a $20,000 website in an afternoon” session – my version would have been on how to tie Class::DBI, Template::Toolkit, CGI::Untaint, Class::DBI::FromCGI, Date::Simple and Spreadsheet::ParseExcelSimple together to provide a database-backed website based on information supplied by a client in Excel files. But I’d like to hear other people’s eqivalents.

Doing More With Less Power

I don’t really mean the “political” power question here (“My company is moving most of its development to Java / C# / whatever. I know I can get work done ten times faster in Perl – what should I do”), although that could be interesting too.

Instead, I’m talking about things like The Fractional Horsepower Webserver. Jon Udell presented a wonderful paper at TPC 2 (1998) on a desktop HTTP server that ran a distributed contact manager application. 4 years on, with desktop servers on the rise again, and peer to peer much more commonplace, what’s happening in this area?

For years the Perl world has been great at writing (unix) server based applications, but relatively poor at writing (windows-based) desktop applications. As the two continue to coalesce what great Perl-based desktop applications are being written using a local web-browser as their front end?