Understanding Nothing

So Mint has won the TechCrunch40 award. It’s good to see the world of personal finance software being shaken up again by companies like Mint and Wesabe – it could certainly do with it.

I’ve ranted before (many times, in fact) about how badly business accounting software sucks. I doubt it will surprise anyone if I say that I think that personal finance software is universally terrible too.

Whilst companies like Mint and Wesabe show a lot of promise, and are challenging some of the core assumptions of entrenched software like Microsoft Money and Quicken, I’m not going to hold my breath for them to become particularly useful to me any time soon.

I’m far from the ‘average customer’ for this sort of software, and some of the things I see as absolute requirements are going to be way down the priority list for many other people, but I’ve far too many other things to be expending my efforts on right now to build this myself, so I’m going to throw out my wishlist in the hope that someone, somewhere, might listen.

Firstly, all financial software I’ve seen gets one core fundamental thing completely back to front. They’re all account based. Before I can tell it that I spent some money I have to tell it what bank account or credit card or whatever I used. This is incredibly silly. The primary thing I’m interested in recording and tracking is my overall spending and financial status, not my individual account details – I can already get that information online for each of my accounts. Aggregating the data into one place is mildly useful, but isn’t even particularly important to me, never mind the key focus. I spend money. That that money happens to come from bank account A or B, or credit card X, Y, or Z is secondary at best. Sometimes it comes from several of them, like when I had to buy a replacement laptop recently and the price was higher than my daily limit on one of my bank cards so I had to split it over two. Most software makes me artificially split that into two transactions. It’s not. Deal with it.

This is partially related to the fact that I buy lots of things with cash. Most software lets you run a ‘cash’ account, but they don’t really take it seriously. For me it’s the default position. If I bought something, assume I paid cash. Let me say that actually I put it on a credit or debit card as an optional extra field. The new wave of tools are even worse at this than old ones, as they assume that most of my data entry can be eliminated by importing bank and credit card statements. Whilst that feature is quite nice, it’s only going to get a tiny percentage of the transactions I want to record. And even the ones it does get won’t be as detailed as I want. Because the other overarching difference between me and most users is the level to which I’m anal about data entry. My financial software needs to let me revel in that. I don’t just want to record that I spent €22.19 on groceries at the supermarket. I want line entries for each item. I want to know how much I spent on milk last year.

Trying to eliminate the need for data entry by fetching all my records directly from my bank isn’t going to work for me. I’d rather you put much more effort into making my data entry really fast. You already know from previous transactions that when I say I was shopping at Rimi and bought milk that it cost 9.50 EEK. Auto-completing ‘milk’ as I type it is useful, but you should also be guessing the amount I spent too. If you get it wrong it won’t slow me down from having to type it anyway, but when you get it right it speeds me up. It will also have the useful side-effect of letting me notice much quicker when they put their prices up!

Next: multiple currencies. Most financial software is written by people in the USA, and it shows. At worst it assumes everything is in US$. At best dealing with multiple currencies is usually an add-on, and quite a clumsy one at that. So far this year I’ve bought things in about ten different currencies. My primary bank account is multi-currency. I can have cash in it in about 25 different currencies simultaneously and transfer between them at will. I’ve yet to find any personal finance software that copes with this.

And, again, I shouldn’t have to select a currency before entering something, or, worse, set up a different account for each currency I use: ‘cash (GBP)’, ‘cash (Euro)’, etc. Let me set a default currency that, when I don’t give any more information, will be assumed, and then let me just type €20 or 20 EUR to override it. And by ‘or’ I mean my choice, not yours. Needless to say all the reporting should be able to cope with this. Please don’t tell me that last month I spent 38EEK + €4.20 + £1.19 on milk. Instead give me a proper total in any (i.e. all) of the currencies that I use.

And of course there are language issues too. When I’m entering my grocery shopping in whichever strange country I’m currently spending time, sometimes I can’t remember what one of the things I bought was. That’s OK. Let me enter “päeval seemned” now, and later, let me change it. But it’s important that it remembers, so that the next time I forget it automatically changes it again for me.

Or, alternatively, just let me emulate this myself through a hierarchy of categories. Then I can just put ‘Млеко’ in category ‘milk’ and have everything work itself out. But please don’t restrict me to some arbitrary number of categories for no reason. Or, worse, make me type the entire tree every time. Just let me type ‘Млеко’ and have it autofill the whole way up through milk -> drinks -> groceries, or whatever I’ve set up. In fact, if you implement this right I’ll be able to get even more anal and record the brand of milk too and have it cascade up. And, of course, I should be able to analyse the spending at any level of this tree, with drill down and roll up, over any time period of my choosing. That should go without saying, but you’d be surprised how bad most financial software is at letting you do things that should be obvious.

Being able to add comments to any transaction is fairly standard. Less common is per-line-item comments. What I really want is even fancier still: I want to be able to specify data fields that apply on a per item basis depending on what that item is. So when I buy Fanta at the supermarket I want to be able to say whether it was 1 litre, or 1.5, or 2 or whatever — in a way that lets me later query just how much of the stuff I drink a month. Or the average price I paid per litre. (Did I mention before that I want my financial software to let me be really anal? I mean this much more than you probably imagine!) When I buy books I want fields to magically appear for the author and title, for each book.

Similarly, when I buy dinner at a restaurant, it should prompt me to fill in the tip given. And let me get reports later on which restaurants I tip highest at. And of course it should cope with the scenario where I pay for the bill on my credit card and my dinner companion(s) gives me cash. And with me then marking my share of that bill as been a work related expense, and let me balance that off later against a particular repayment. With reports in the meantime of all my unclaimed and unpaid expenses, of course.

And, whilst we’re on balancing things off, don’t assume that all transactions happen instantaneously. Most systems aren’t as great as the Estonian ones. When I pay off a UK credit card from a UK bank account the money leaves my account on one date, and usually doesn’t reach my credit card until 3 or 4 days later. Yes, I want that to just be entered as one transaction, but I’d like it to be able to have different dates at each end, please. And the same for fees. Transferring money between different bank accounts in different countries often incurs fees at each end. I want to enter all that as one transaction, and just have the right thing happen. But that’s close to impossible in the account-based systems I’ve already ranted here about.

Lastly, my perennial bugbear of corporate accounting applies here too. I want to be able to apply a useful lifespan to certain things I buy. Unlike corporate software, in personal finance this shouldn’t be a default position. I don’t want to depreciate my carton of milk over three days, or assign a daily value to it. I’m not quite that anal yet. But I do want to say that the DVD player I bought lasted for 18 months. Or that my new phone should last me at least 12 months before I upgrade. (And of course correct this later when I don’t actually replace it for 15 months.) Then I can have the normal reports on both a purely cash in/out basis, and also something closer to a company’s P&L that pretends I paid for the DVD player at £5 a month rather than all up front. Done properly that gives a much better picture of my real spending, rather than having everything appear to fluctuate wildly every month based on a variety of non-recurring purchases.

Very little of this should be difficult. There aren’t really any complex algorithms here, and even the tricky UI issues (changing input fields based on other ones) are commonplace in major Ajax toolkits these days. It’s all just a Small Matter of Programming. Any takers?

Update 25/9: Yesterday I bought a bus ticket in София for €10 + 5lv (to clear out my Bulgarian currency before leaving). I know of no personal finance software that will allow me to enter that transaction.

Update 2009: More on this.

In his latest post, Joel explains how creating great software involves finding, and fixing, the tens of thousands of tiny things that should be better. The unfortunate side effect he describes is familiar to all who know me: “It takes a mindset of constant criticism to find them. You have to reshape your mind until you’re finding fault with everything. Your significant others go nuts. Your family wants to kill you.”

As an example (as if one were needed), just yesterday I was ranting to Casey about the Ikea website. Take a moment to visit it, and see if you can guess what drove me mad about it.

I’ll give a clue. It’s in this list:

• Norge (Norway)
• Österreich (Austria)
• Россия (Russia)
• Polska (Poland)
• Portugal (Portugal)
• România (Romania)
• Schweiz | Suisse | Svizzera (Switzerland)

Read more…

For the last few months I’ve been a slightly more than casual observer of the Maypole project. Although I don’t actually run any sites on Maypole, it’s based on rather a lot of my public code, and a lot of issues people raise on the mailing list aren’t really about Maypole, so much as Class::DBI or CGI::Untaint or Class::DBI::FromCGI or somesuch.

It’s also been interesting to watch as, although it’s not directly connected, Maypole seems to have been heavily influenced by Kasei’s internal FireCore framework which Simon had to work with for the guts of a year. He threw away lots of our mistakes, implemented some of the things we’d talked about but hadn’t yet gotten around to, and, being Simon, took some of the ideas much further than we’d imagined. So I was keen to see how it would hold up in the wild.

Of course one of the first things people wanted was to replace the dependency on Template::Toolkit, and be able to use any of the myriad of Perl templating systems. In hindsight I believe implementing this to have been the first major post-release flaw.

I’ve been a student of web abstraction frameworks for a few years now, and the siren song of many of them is the idea that depending on any given component is a problem. Every part should be capable of being swapped out for an equivalent. In database backed web-based systems in particular this means you shouldn’t be tied to any given database or database abstraction layer, or to any given templating system. I don’t know whether it’s a false hubris, or just a good old fashioned desire for world domination, but there’s a pervasive idea that the framework should be capable of doing anything and everything you desire.

But I’ve come to believe that exactly the opposite is true. Frameworks have most power when they aim to do one thing well, and become tightly entwined around the components that make them up. In Maypole’s case, its initial goal was to make it really trivial to build database backed web systems around Class::DBI and Template::Toolkit. Lots of people build such systems using precisely those components, and they all repeatedly face and solve the same problems, time and again. Maypole neatly bundles up and encapsulates one path of building such sites, freeing developers up from the drudgery to concentrate on the value-added business logic.

But converting such a framework to hot-swappable components restricts the amount of best practice than can be neatly encapsulated. Each RDBMS/OO mapping layer does some things better than others. Each templating system, likewise. A framework committed to one of each can play to their strengths. A framework allowing interchangeability has to reduce itself to lowest common denominator.

One the advantages Maypole had over FireCore was a set of default templates. Simon now regrets building these, as they were so good that people believed they should be able to use them for everything, when for most applications most people will need to write their own. But, tied to Class::DBI and Template::Toolkit, Maypole would be able to create much better default templates, making better use of CDBI’s introspection methods, and TT’s VIEWs, MACROs, and Plugins. In a more generalised world this becomes much harder, and the framework becomes blander and blander.

Nothing can be everything to everyone. Too many frameworks try, and end up being used by almost no-one other than the person who wrote it – mainly because they’re the only person who can ever find their way through the maze of twisty abstractions.

Behind my beliefs on this lies one often overlooked truth: frameworks are hard. They’re much more about philosophy than technology. Early design mistakes get magnified dramatically, usually in direct proportion to the level of abstraction you’re aiming at. And the more you build the framework in isolation from the sites you’re building using the framework, the more likely you are to make those mistakes. And once they’re made they’re really hard to recover from.

I once consulted for a company who had what we came to call “three legged cow syndrome”. They had developed a basic content management system that, like many of these things, had morphed into a product that could, of course, do anything any client (and particularly any potential client) wanted. And every time someone came with a request for something the system didn’t already do, they tweaked the system so that it could do that in future. After all, that’s just good practice.

Of course, each time they did this, they codified a hybrid of generic and specific requirements – mainly because they couldn’t know the difference.

Because the first client who ever wanted a certain feature had certain business rules associated with how that feature would work, the code assumed that every client who wanted that feature must work the same way. But of course, every client’s process was different. And, almost invariably, insane. It’s the nature of business systems. They evolve over time to account for all manner of foibles. As technologists we want to simplify these things, and refactor them all away.

But almost every real-world business has all manner of bizarrely crippled logic underlying their business rules. They’re all three legged cows. And in this company every client who later wanted the system to do something slightly differently (as they all did) was treated as if they were insane. Because everyone knows you don’t do that like that – you do it like this. And when they finally prevailed, the developers would have to spend weeks untangling the code to work out what was common and what was unique to each client.

The only way around this syndrome that I know of is to implement the first client’s additions in isolation, as some sort of standalone plugin or add-on. Then the second time someone comes in with something similar do the same, based on what you’ve learned the first time. The third time you can begin to factor out the commonality. Cautiously.

There’s been a lot of furore on this sort of issue recently in the emerging Maypole community. The new maintainer after Simon’s retirement attempted to take the framework too far too quickly. In some ways this speed has probably saved Maypole. By moving at such a rapid pace, no-one could keep up, and people started complaining. If he’d moved much slower, introducing his features one at a time, each one would probably have made seductive sense, and been accepted, until eventually there was nothing left. Frog boiling is a remarkably effective means of changing the direction of any project if you do it slowly and subtly enough.

But now there’s been a fork. Hopefully Sebastian can take the new Catalyst project off to where he wants to go, and Simon F can revert Maypole back to a really good CDBI+TT framework. And hopefully I can avoid being seduced too much, and steal the truly good ideas back for FireCore.

Many developers treat a delivery as a chance to say, “Here’s the software you asked for,” but they should be saying, “How’s this version?” This is why iterative development is so useful: It gives the business folks and developers numerous opportunities to refine their ideas and discover the system’s hidden potential. If we view deliverables as nouns, we tend to ship them and run. But if we treat a deliverable as a very – as something we do to help improve the way we add value to the business – we encourage communication, cooperation, and feedback between developers and customers.

– Dave Thomas and Andy Hunt, Verbing the Noun, IEEE Software, July/August 2003.

An interesting take on the age old debate, found over on the c2 wiki, in response to the suggestion that teaching computer science should pay much more attention to its applications in the real world:

I disagree. That’s SoftwareEngineering. CompSci should be about the theory and the limits of what’s possible at least as much as it is about coding. If we adopted that view about mathematics, for instance, we’d spend all our time on loan calculations, completing your tax return, and basic statistics

InfoWorld has an article about the decline in spending on web services. All well and good. However, there’s an interesting little aside in the middle:

“It’s tough out there,” said Fred Holahan [of] Silverstream, which was acquired last month by Novell. “IT managers are increasingly being asked to do more with less, and they’re facing a psychology that tells them, ‘Hey, you bought a ton of technology 18 months ago. Why not make that work?’” Unfortunately, Holahan argues, the savvy IT manager knows that the software and tools he bought a year or two earlier are not exactly cutting edge anymore, but his hands are tied to upgrade. Yet, for those IT shops still prepared to invest in new tools and application servers, it has become a buyer’s market.

Is it really that hard to build something with an 18 month old technology? Why does a ‘savvy’ IT manager really need to be investing in cutting edge tools anyway?

Maybe it’s just the case that these IT manager were oversold technology for technology’s sake 18 months ago, and it actually doesn’t do anything useful? What’s to stop them making the same mistake again – buyer’s market or not!

Good software, in Humphrey’s view, “is usable, reliable, defect free, cost effective and maintainable. And software now is none of those things.” [via FuzzyBlog]

This article covers a lot of the same ground as an article I was reading yesterday by Jerry Weinberg in Understanding the Professional Programmer. He compares the “if it compiles it must be OK” approach to students who learn that their essays are being graded by a computer which takes off marks every time it finds a spelling mistake or error in the grammar. Very quickly you’ll end up with a class of students who have perfect spelling and grammar, but who have no idea how to actually communicate ideas in writing.

Thankfully we’re not quite that bad and I don’t think many people can seriously believe that “it compiles, so we’re done” is true. But I’ve started noticing a distubing trend of people misusing the XP approach of “the tests all pass, so we’re done”.

Kent Beck is fond of saying “Make it run, make it right, make it fast.”

Usually this is taken to be a warning against premature optimisation (after all that’s where the quote is mostly commonly quoted from). But it’s also a warning that you’re not done when it manages to run. You still need to make it right. I’m always amazed by how few programmers have ever read the Refactoring book – or are even aware of the concept.

Note to self: When you find an interesting article, discover it crashes your broswer when you try to print it, and have to spend about 15 minutes trying to find it again via Google because you can’t remember what it was called and you have to try about 6 different search sets of search terms, then, when exactly the same thing happens all over again, make a note of the URL somewhere.

Aside from the main thrust of this article (expressing HTML programmaticaly), there’s a very interesting categorisation of web documents. Nørmark goes further than the traditional static vs dynamic division, showing that there are at four types of pages:

1. Static – Stored pages written directly in HTML, bound at document edit time.
2. Generated – Stored pages translated from another language to HTML, bound at document generation time
3. Calculated – Calculated pages generated from a program at document access time
4. Dynamic – Calculated pages generated from a program at browse time

He only really deals with static pages, but by expressing these in a Scheme syntax he opens the way for others to show the combination of this with Scheme to create the calculated pages.

I recently re-read a lot of Paul Graham’s writings on lisp, and decided this time I was actually going to find out more about his concept of using closures to simulate web pages acting as subroutines (in Lisp in Web-Based Applications).

The two ‘definitive’ papers on the topic seem to be by Christian Queinnec: The Influence of Browsers on Evaluators or, Continuations to Program Web Servers, and Inverting back the inversion of control or, Continuations versus page-centric programming.

The basic concept is that thinking of a web site as a set of pages is the wrong approach – instead you should think of it as an extended computation, where interactions with users suspend the computation and later resume it. When implemented as continuations the compuation can be reified automatically.

This seems to be one of those subtle, but significant, changes in mindset that could change how you even approach building a major site.