Understanding Nothing

Playing with del.icio.us yesterday, I came across the popular post “Six Ways on Getting More Done each day”. I certainly hope its popularity isn’t a sign that people are going to follow its ideals.

1. Focus high importance tasks first

“If you are choosing to watch TV over completing your project that is due tomorrow, you are definitely getting your priority wrong.” Actually if you’re unable to watch TV because you haven’t yet finished a project that’s due tomorrow, then you have a bigger problem. Granted, that problem might be that you work in an organisation that doesn’t give you enough time to do your job, but working late into the night to show actually you can finish things on an impossible deadline isn’t really going to help with that problem long term. Probably just the opposite. But assuming you had enough time to do the project in the first place, then why aren’t you finished yet? If it’s life quality you’re after, then being finished with time to spare wins by quite some margin every time. Kathy Sierra had an interesting post a few days ago about how people can’t be afraid and rational simultaneously. The fear of not getting a project finished on time is a very good way of making sure that the project isn’t actually done as well as it could be. When this happens most people tend to blame it on just not having had enough time to do it well. Kathy’s post made me realise that it’s not the lack of time, per se, so much as the inability to use that last minute time as well as normal.

2. Work smarter

Now, everyone knows that I’m a major proponent of *that*. But I’ve refined my approach quite considerably over time. These days I rarely think that it’s a net win to try to completely automate away something you do. Certainly not before you’ve even tried to do it. The amount of effort involved in doing this is almost always going to be greater than in just doing the task. Hackers tend not to notice this as when you’re expending the energy on something you like, rather than on a boring task, it feels like you’re getting more done. But in general you’re actually getting less done – just enjoying it more. These days I’m a much bigger proponent of automating away one part of the task each time you do it. In general this should be the part that you’ll get the biggest “bang for the buck” in automating based on how long it will take to automate vs how long you spend doing it each time. It’s a fairly tricky equation to get correct, so don’t get sucked in to trying to calculate accurately. Again that just distracts from doing it. When in doubt make the improvement that you know will take least time. If you’re doing the task often enough, you’ll make so many improvements that you’ll manage to get 90% of the task automated away quickly enough anyway. The other 10%, of course, is usually almost impossible to automate away sensibly. The systems that claim to do this usually just play a smoke and mirrors game where the work is instead diverted to some other part of the system. Usually a user who suddenly has to do some new task to work around the lack of fudgability in the system

3. Work faster

Again, people should imagine that I have no problem with this. I certainly agree that almost everyone who uses a computer regularly could benefit from learning to type faster. But I also agree with Dijkstra that, save for secretarial-type tasks, the bottleneck should usually be thinking, not typing. And thinking isn’t something that works well at a faster pace. If anything, for me at least, it prefers a slower pace. Of course, in a great many organisations, appearing to work hard, and appearing to work fast, are seen as good things. Appearing to be just sitting thinking is seen as a bad thing. And as for going for a walk around the park to think … Again, I much prefer the approach of trying to speed up everything you do a teensy little bit every time you do it, and relying on the cumulative effects.

4. Work harder

“Drop your TV watching session. Drop your tea break.” Erm. No. Not at all. If anything, take more breaks. Working constantly just means working less efficiently. I really didn’t think anyone believed this.

5. Concentrate and focus tasks

“Concentrate yourself on one task and only one task.” This is one I’m not sure about. There are certain types of tasks where this probably works well. But most people’s jobs are interrupt driven. Everyone like to complain about how terrible this is, and how much they’d be able to get done if they just got peace to do their job. I wonder how much more we’d all get done collectively if we learned to accept that interrupts are natural, and possibly even good, and learned to work effectively with them.

6. Avoid to make mistakes (sic)

“There aren’t any excuse if you are making the same mistake twice. Note it down as notes and remind yourself when you are doing similar tasks again.” I agree that mistakes are time consuming. But just noting them down to try to remember not to make them again is a huge mistake. Instead you need to identify how the mistake came to be made, and try to make it so that that sort of mistake /can’t/ be made again. Again this is subject to the same sort of rules as #2 above. If you’re making lots of mistakes you certainly don’t have time to stop everything each time and concentrate on the root cause. But you certainly have time to put even one barrier in the way of a future you (or a future someone else) going down that path again. As before, if it’s a common error, the entire path will very quickly get blocked up quite effectively.

Reading back through this, and thinking about my reactions to these points, I think the key to Getting More Done is, like most things, slow steady improvement. It’s not glamorous, it doesn’t sell books, and it’s certainly not easy. But it’s effective. If you can make a 1% improvement on something every day, then you’ll be twice as good in about three months, and over ten times as good in a year. This starts to add up fast. Of course, the more finely tuned you can get everything, the harder it becomes to find something else to squeeze out. But most of us have so much nonsense tied up with everything we do that that problem is usually years away!

One of the other things that’s changed quite dramatically in the past three months is the way in which I deal with my mail. I’ve been through a number of different mail clients and setups in the past, and have never quite found any approach that quite fits my needs. I was a long term fan of MH on unix, mostly because there was no real client getting in the way. Everything was just simple files on disk with a series of commands to allowed you to slice and dice them in interesting ways. (Actually it was really only one big command that was hard-linked to be called as different commands, changing its behaviour depending on the name it was called as!)

This meant you could chain your mail commands into a normal unix pipeline, allowing you to manipulate them in simple ways – as long as you’re comfortable with the unix command line, of course! And if MH didn’t offer something you needed, you could just write your own as simple shell or perl scripts. One of my first perl scripts was a recursive grep through my mail archives.

Eventually I was persuaded to move to mutt, the one true unix mail client, and it has served me well for past 7 or 8 years. Its scoring and colour coding became very useful during the years I started drowning in spam, and once I got spam detection tools working, the ability to set up keybindings to integrate mutt with those made my life much easier.

I’ve had a rather strange mutt set-up though. I never really mastered procmail, or Mail::Audit, or equivalent, so rather than filtering on ‘From:’ lines, I’ve always filtered on ‘To’ lines. (Having my own domain means I can have an infinite number of email addresses). Each of the many mailing lists I subscribe to, and each commercial site I have to sign up for, gets a unique address, which then get gatewayed to their own folders through a series of .forw.d files. I then have a little script triggered from my mutt startup, to inform mutt of their existence. I then read ‘groups’ of mail in turn, rather than having them all mixed up in one big box.

I’ve experimented with other mail setups from time to time, and have been using Thunderbird via IMAP as by secondary mail reader for a while – particularly for my commercial mail folders which tend to mostly be in HTML these days (and, though some will be horrified to hear me say it, usually better for it), and for dealing with attachments.

But two things have changed in my mail set-up recently, and it’s making me rethink the whole approach again.

Firstly, I’ve switched my spam detection to the new service we’ve set up in UNITE. This is proving much more accurate than bogofilter (which had gotten stuck around 85-90%), and is currently running at 98.8% on the mail that arrives in my box (and a much greater amount of spam is also stopped upstream, never even reaching me). It also operates a web-based quarantine, so I get almost no spam arriving in my inbox. All my fancy mutt bindings to tag, report, and delete spam are to very little avail any more, and with a much clearer inbox, my scoring and colour coding loses a lot of its value.

Secondly, I’ve moved almost all my mailing lists and commercial email to Gmail. As a general mail client, Gmail is disappointing. I don’t think I could use it as my primary tool. But its ‘conversation’ approach works surprisingly well for mailing lists. With judicious use of labels and filters, I can not only direct each list to a different “folder”, but actually allow multiple messages to be in multiple ‘folders’ (so that, for example, Class::DBI discussion on the Maypole mailing list can also appear in my CDBI list, along with any CPAN uploads with Class::DBI in their name). This also allows me to stay on some of the mailing lists I was going to unsubscribe from. There are a variety of lists I subscribe to but rarely read (although I still occasionally skim the threaded subject lines for the occasional post that looks interesting). These are usually related to technologies that I use from time to time, and whose lists don’t have good archives. Often, when I have a problem, I can then just search through my own archive of the list. Gmail, of course, makes this even easier than before.

With these two changes, I now have a situation that I haven’t seen for close to 15 years, where almost all the email arriving on my desktop is either personal or work-related, and almost all requires some action! This theoretically enables me to massively simplify my mail set-up (and then probably add a new layer of complexity in a different way!)

I’m currently thinking of reverting to a single inbox, and doing all my filtering after reading rather than before, probably to a variety of categorised ‘TODO’ folders, leaving my inbox as a real inbox, ideally emptying it every time I access it.

The fact that it’s taken me 15 years to get close to the position that the majority of email users have as their normal set-up probably says something interesting. I just don’t know what that is yet.

Joel’s Book of the Month this month is Robert Glass’s Facts and Fallacies of Software Engineering. This has been on my To Buy list for a while, as I keep stumbling across articles by Robert Glass that I like.

The book lists 55 Facts, all seemingly backed up by references. I really wish Amazon had “search inside” enabled for this book though, as I want to see what the references given are for Fact #2, “The best programmers are up to 28 times better than the worst programmers”. As I’ve written before, I believe the oft-cited Sackman experiment to be a fundamentally flawed source for this claim.

The chances of finding this book in Belfast are slim, so if anyone has a copy and can let me know what the sources listed against this fact are I’d be grateful.

IT Week this week references the Web Effectiveness Report 2003, which, amongst other things, reveals that only 19% of web site managers surveyed review their log files to look for problems with their site.

When we originally built BlackStar we were fairly novice Perl programmers. We knew that all Perl should really have “strict” mode and “warnings” turned on – so we made sure we did that. But as long as everything ran correctly, we didn’t really pay much attention to the warnings that were emitted.

But after a while we realised that the profusion of verbiage cluttering up our logs was highly distracting, and would ensure that nobody really paid them much attention – and the useful information about real problems would be obscured.

So we decided to clean this up, to ensure that anything appearing in the log would most likely be a symptom of a real-life actual problem, of which someone should probably be notified straightaway.

This was much easier to decide than to actually implement, however. We were introducing all manner of new quality procedures at the time, so it was relatively easy to at least decree that any new code, or any alteration of existing code, should be free from warnings. This way we could at least halt the growth of new warnings (although with site visits growing 30-40% per month, the volume of messages was still growing quickly). We even scheduled into the programming schedule a little time devoted to removing some of the most egregious offenders, which probably removed about 50% of the warnings.

The slow clean up of having the old code gradually tidied in passing wasn’t going to get us anywhere fast enough though. So we introduced a new system that was disarmingly simple, but remarkably effective. Every night a process collated the error logs from across the various web servers, and generated a report of the ten most common problems. This report would be emailed to the entire programming team, and the next day one person would take responsibility for ensuring that the top item on that list vanished.

This approach proved very effective and within a few months the volume of warnings had decreased dramatically. It was so successful that we started to apply the approach to many other areas of the business. Any problem that was monitored over time was a candidate for it – and with an entirely web based business we had a lot of data to monitor. Someone became responsible for checking the list of the most common search terms that returned no results, and adding a re-mapping so that that search would automatically be transformed into what the person probably wanted. Someone else would ensure that the most visited DVD page that didn’t provide cross-reference to the VHS of the same item was given that linkage. Someone in the warehouse would upload the cover of the most visited item in stock that hadn’t previously been scanned.

Most of these tasks took less than 5 minutes. They would rarely be a top priority in amongst the constant fire fighting, growth-pains, and breakneck pace of developing new features and systems in the crazy world of exponential growth. In normal circumstances none of these things would even have appeared in someone’s list of top 10 priorities. But the simple action of ensuring that each day the top occurrence of each problem was removed created a staggering cumulative effect.

The art of time management is usually a matter of ranking your items by importance and urgency, and prioritising according to how high things appear on each axis. But most books, articles and seminars on the topic stop there. Spending a few minutes each day doing something that is neither particularly important, nor particularly urgent, but that has a beneficial outcome, has value. When everyone in an organisation is doing likewise, and those tasks are automatically selected based on their potential benefit, that value can be enormous.

The economic corollary, cleverly exploited by Billy Beane, was that players who walked more often were being systematically undervalued by the market and could be had more cheaply. Software development has an equivalent to the base on balls: a module that doesn’t have to be written new because it already exists and can be reused.

We know that the best coders are far more productive than the norm. Arguably, that variance might lie in the patience, discipline, and research skills required to recycle rather than to reinvent. If so, finding ways to measure and value these qualities could yield a pivotal advantage.

– Jon Udell

If we hope to be able to measure productivity in some quantifiable way, we must first appreciate what it means to be “more productive.” If I say that programmer A is “five times” more productive than programmer B, I mean that programmer A can finish a programming task in one-fifth the time it takes programmer B to finish it. Stated differently, it would take a team of five B programmers to keep up with programmer A.

Let me go one step further and define the concept of “perfect productivity.” A programmer is perfectly productive if he can perform a task in the same time it takes to specify it. There is no fat in the implementation. The code corresponds directly to the specification of the problem. The obvious corollary to perfect productivity is perfect maintainability. Code that corresponds directly to the specification of the problem can be modified as quickly as the specifications are changed.

Watching a programmer practice perfect productivity is like watching a court reporter. Both continue typing until the speaker takes a breath. Then they both stop and wait. There is a direct translation between words (specification) and keystrokes.

Perfect productivity is, of course, a fantasy, as is perfection of any sort. Being a fantasy, however, does not eliminate it as a goal. It is always our goal, and it should be yours.

— Gary A. Bergquist, The Zark library of utility functions

This article by Moser and Nierstrasz points out that those who use Object Technology (such as OO Frameworks, or other collections of reusable components) commonly report increased productivity. However individuals do not magically become more productive by using these. Rather they are just as productive as before, but at higher levels of abstraction. Frameworks do not make developers more productive, they just reduce the amount of work needing to be done!

They also spend some time discussing how Function Points are not a good measure for OO development, as they were never designed to fit a meta-model other than CRUD database structures, and OO models are completely different to this. They also point out that Function Points do not capture the effects of reuse; that is, the availability of frameworks or libraries that address parts of the requirements. Instead of measuring total functionality delivered, we should be measuring incremental functionality beyond that which is provided by framework components.

They therefore claim that a good software size estimation method:

• avoids historical “magic numbers”
• takes into account the availability of reusable components and frameworks
• captures arbitrary kinds of functional requirements
• expresses possible links between functional and structural requirements
• can be applied early in the development process

They then set forward “The System Meter” as an alternative to Function Points, which they claim is less predictive, but can be applied earlier in the process.

Karen asks why we, as software creators, haven’t yet learnt about the things that people were writing about more than 20 years ago. This ties neatly into an article I was reading yesterday in the April 1998 issue of Computer by Ware Myers.

Citing figures gathered by himself and Putnam, Myers claims that:

A reality that is not universally acknowledged is that the general capability of software organizations extends over a truly enormous range. The difference between the productivity of individual developers is in the order of 10 to 20 times. The difference between the capability of software development organizations is at least two orders of magnitude greater – 100 to 200 times.

As with most of these figures, the result is based on comparing extremities. However Myers also shows that, on their scale of 33 levels, “an organization now about 5 index numbers below the mean can increase its process productivity by a factor of 5-11 times by improving not to the very top, but to five index numbers above average.”

He offers some suggestions as to ways that organizations could improve, such as establishing the feasibility of a large development task beforehand, citing figures that 65% of 1 Million+ LOC systems are cancelled before completion, as are 50% of 500KLOC systems, and 25% of 100KLOC systems. As Myers says, these approaches “seem fairly obvious, but many software organizations manage to ignore them.”

So, what’s the hang up? Myers cites 5 main reasons why we never learn:

1. Psychological hurdles: we’re resistant to change. Developers in particular are usually keen to get to the “meat” of the task – the actual coding – and hate spending time on messy preliminaries.
2. We don’t know about better ways: Myers kindly suggests that developers might not have had the time to seek out new approaches, whereas I think most developers just don’t have the inclination. However, Myers points out that Business Systems projects see, on average, 10% productivity gains per year, so not keeping up can have quite a large cumulative effect.
3. We’re Gun-Shy: Lots of the “better ways” promoted in the past haven’t delivered on their supposed effectiveness. Why should be believe any other ways?
4. We’ve tried and failed: Even the valid “better ways” are usually not easy to implement; they take champions, time, investment and persistence.
5. We’re stifled economically: we may not have the resources to devote to long-range productivity efforts

Personally I agree more with Karen: programmers love to program. And we love to solve problems. Looking up someone else’s solution just isn’t as interesting or exciting as trying to work through it ourselves. And writing less code? There’s no way that someone else’s code would be as good as, or do all the things that mine would!

There’s been a lot of debate on the Extreme Programming list recently about Tcl vs Java, mostly spurred by Dossy’s citation of the Jeff Hobbs quote: I’ve come to the realization that there are so many more Java jobs than Tcl jobs because it takes 10 Java programmers to do what I can do in Tcl. I’m only half-joking…

Interestingly I came across an article today in IEEE Computer, from Jan 1998, that compared the scripting languages (particularly Tcl) with the system languages (C, C++, Java et al). It covers most of the usual ground, but also includes an interesting chart of some applications that, for one reason or another, ended up being written in both types of languages:

Some of the reduction in effort is no doubt due to the TCL version usually being written second, but for at least one of the applications this wasn’t true. In many of the cases the TCL version also provided more functionality.

This paper from Barry Boehm and Philip Papaccio in IEEE Transactions on Software Engineering in 1988 is most notable for the size of its  bibliography, which has 126 references! It also contains some interesting insight into why they think that software costs are important:

1. Costs are big and growing: therefore any percentage savings will also be big and growing
2. Many useful products are not being developed: therefore reducing costs will provide more time to attack this backlog
3. Understanding and controlling costs can give us better software, not just more software

The four main strategies they give for improving software productivity are:

1. Writing less code
2. Getting the best from people
3. Avoiding rework
4. Developing and using integrated project support environments

The most significant influence on software costs in the number of source instructions one chooses to program. This leads to cost-reduction strategies involving the use of 4GLs or reusable components to reduce the number of source instructions developed; the use of prototyping and other requirements analysis techniques to ensure that unnecessary functions are not developed and the use of already developed software products.

The next most significant influence by far is that of the selection, motivation, and management of the people involved in the software process. In particular, employing the best people possible is usually a bargain, because the productivity range for people usually is much wider than the range of people’s salaries.