Archive

Posts Tagged ‘Wiki’

What is a spreadsheet-wiki?

June 3rd, 2009 4 comments

While I’m on the subject of products I really want to see, I would be remiss of me not to mention the spreadsheet-wiki. This one should already exist by now, and I hold myself largely responsible for it not — after all, I spent almost a year working with Dan Bricklin and Socialtext trying to make it happen. When we parted ways, I hoped to be able to continue the project, but for a variety of reasons that never came together either. There have, from time to time, been vaguely encouraging noises from Socialtext, but this still doesn’t seem like a high priority for them, and the information that leaks out from time to time implies they’re still going down a different path. I’ve deliberately held back from talking about some of this stuff to give them a chance to get something out, but it’s 18 months now since I left, any inside knowledge I had is long past its sell-by date, and I really want to see this come together from somewhere.

By far the most common response when I tried to explain to people what I was working on, and what a spreadsheet-wiki actually meant, was “Oh, you mean like Google Spreadsheets?” But Google, and their online spreadsheet rivals, aren’t really creating what I want. Google Spreadsheets is no more a spreadsheet wiki than Google Docs is a text wiki. Yes, they’re great for collaboration, but that’s only half the wiki story. The critical other ingredient on a wiki is the humble link. Even outside wiki-land the power of the hyperlink is still poorly understood and massively underrated. It’s the fundamental building block of the Web, but even still hasn’t lived up to anywhere near its potential. Almost everyone, when they talk or write about Wikipedia, focuses on the “Anyone Can Edit!” part (whether with awe or despair), but the vast majority of readers never edit anything—the key for them is that absolutely everything is a link:


My dream is that that could also be true for numbers.

Wikipedia, of course, is full of numbers. People can talk about them, change them, cross-reference them, and do all manner of wiki goodness with them. But that’s not enough. Those numbers currently live in splendid isolation. They can’t interact in a spreadsheety way.

There have been various attempts to fix this, generally involving embedding spreadsheets into wiki pages as a replacement for plain tables. But although that achieves the goal of being able to perform some basic calculations in-place, it’s no better than being able to embed an Excel sheet in a Word document. It doesn’t solve any of the well known problems with large spreadsheets (aka Spreadsheet Hell). In a spreadsheet-wiki the spreadsheet should not be a second class citizen, subservient to the wiki. Rather, the spreadsheet should itself contain wikiness. Forget simple single sheet spreadsheets; I’m talking here about hundreds or thousands of properly cross-linked sheets, all mutually feeding each other. Forget having to email around your monthly financial statements comparing actuals to budgets with everything gradually drifting out of sync as no-one is quite sure which is the master copy any more, and no ability to examine how you got to what you have. In a spreadsheet-wiki every number is a link. You can see where it came from, and where it’s being used. If an assumption changes, everything that depends on it automatically changes. No more wasting 3 weeks in a dead end because you were working from old numbers. No more wondering why that P&L entry for “Miscellaneous Expenses” was so high in March. No more wasted time collating projections and forecasts from department heads, harmonising them into a divisional budget for the upcoming year, only to have to redo the entire process 4 times when the CFO trims your budget, or the COO explains some of the impact of a new office opening in in September. Instead everyone can work on their own page, have the data pulled automatically into a series of other sheets, and have changes take effect universally and instantly whilst everyone hammers out the details — with, of course, full transparency of who changed what when (and hopefully why), and the ability to roll-back to any earlier stage.

Most of the technology to make this work already exists. There are some interesting issues when you start talking about thousands of inter-related sheets, but that can evolve when we see what the real usage patterns are. Making something come together that will show just how powerful a concept this is, is mostly just a matter of vision, SMOP, and tuits. Like any of my other ideas, I’d love to work on it, but I can’t build it on my own. If you’re interested in working with me on it, or just building it yourself and picking my brains from time to time, please get in touch.

My work here is done

February 14th, 2007 3 comments

When we took over Ireland’s oldest ISP, almost 3 years ago, we hoped that we would be able to turn it around in 6-12 months. Unfortunately it was in much worse shape than we had been led to believe, and it took a lot longer than planned.

After several tumultuous years, the beast has been tamed, and, barring unforseen trouble in the next couple of months, we’re about to post our second consecutive year of clear profits – an unparalleled feat in over 10 years of trading.

And with that I believe I have successfully made myself redundant, and can move on.

As longtime readers will know I have been using wikis for a long time, and most recently have been abusing the Semantic MediaWiki extension to layer a financial reporting tool on top of a wiki. As part of the discussion surrounding this Ross Mayfield of Socialtext had pointed me at Dan Bricklin’s wikiCalc, the project by the inventor of the original computer spreadsheet to merge that concept with a wiki.

Although I don’t believe that that would be able to do what I need yet to build a full financial system, or indeed to do much beyond creating simple web tables, the idea intrigues me greatly, and after a variety of discussions over the last month or so, I’ve decided to join forces with Dan and Socialtext to see what I can do to help it get there.

I’m really excited about moving back into startup land again, and working with some really great people who are actively engaged in trying to change the world.

And hopefully I don’t fall into the trap, seemingly all too common in this line of work, of being too busy actually creating “social software” to post here anymore…

Wiki Gardening

January 16th, 2007 1 comment

Chris Dent, whose blog is a treasure trove of interesting thinking about wikis, asks:

Even if you have everyone writing in blogs every day, how do you ensure that all those stories are distilled for information that is useful tomorrow, next month, next year and five years down the road?

This is a great question, and one that I have also been puzzling over a lot in my current employ. When we took over the running of Ireland’s oldest ISP a couple of years ago there was a huge information loss problem. So within the first week or so we set up RT to track customer enquiries, a blog for each member of staff to narrate their work, and a wiki to act as a basic customer management system and repository of useful information (it has since grown into much more, including our accounts package, but that’s a different story).

Now, we often have the opposite problem. I remember seeing the information somewhere, but can’t remember where it is – is it buried in a customer dialogue within RT? Did someone write it up on their blog? Was it added to the wiki?

In this version of the problem, ‘search’ probably is the simplest answer. But as Chris points out, search isn’t always the right answer. I would go further in my reasons why, however.

Search is useful when you’re looking for something, and you know what it is. Often you don’t have both halves of that. When a customer contacts you, for example, you should be able to pull up a single page of details about them where all the important facts will be listed: what level of support they have; what services they have; major problems on their account; key personnel etc. If a customer has recently had significant problems with their email, but this hasn’t been recorded on the wiki, and the person dealing with the customer now doesn’t know that, then they’re not even going to consider searching for information about it. Even if they have a vague memory of overhearing someone talking about some sort of problem with the customer’s account a month or so ago, they probably don’t even have enough information to search with.

It wouldn’t matter how great a search appliance we had, normal search just wouldn’t help here. This is where, as Chris points out, the process of wiki gardening comes in. Someone needs to tend to the wiki, carefully pruning back the less relevant information, and reshaping each page into its most useful form.

But this is a time consuming operation, and generally most people don’t have that sort of time. It’s hard enough trying to ensure that a summary of the key facts from each customer interaction just get copied over onto their wiki page, without also needing to spend another five minutes just tidying that page up. In larger organisations where call-centre staff are measured on how many queries they can handle per hour, the disincentive is much much stronger.

And, of course, any time where a human is copying information between two different computer systems, a giant red flag should pop up and scream that something really bad is going on.

I don’t have any great answers to this problem, but I wholeheartedly agree that enterprise wikis need to provide better tools for dealing with information stored outside themselves.

JotSpot did quite a lot of work in this area, providing two-way integration with Salesforce.com etc., but although that made for a cool demo, I don’t think that’s really what’s needed. Rather than just replicating the data stored elsewhere, the wiki needs to allow you to summarise what’s there, and then direct you off to view the full detail in situ. (Ideally searching within the wiki should pick up the full content though). But, crucially, there needs to be a way for the wiki to know when there’s un-summarised data needing handled, rather than users needing to remember to copy the data across.

Perhaps in the first instance it’s as simple as being able to add a little gizmo to a page that tells it how to find, for example, all RT tickets for this customer, which will then automatically list each ticket – initially in a default manner but where each entry can be edited in the traditional wiki way? As we move more and more into a world where different pieces of software can talk to each other with webservices, it should become easier and easier for this sort of information to be pulled across.

Semantic Mediawiki already provides an syntax for querying information within the wiki (although there’s no way yet to manually manipulate the results), so something like this could probably be repurposed to query information outside the wiki in a similar manner. Time to go ask on the semantic wiki list whether anyone’s working on anything like this. And I’ll certainly be paying close attention to how Socialtext (or anyone else, for that matter) tries to solve this issue.

Tags:

The wiki year

December 31st, 2006 1 comment

Although wikis have been around for quite a long time, each of the last few years has seen significant growth in awareness and usage, and also in money flow. 2006 has continued this trend: ‘wikipedia’ and ‘wiki’ both feature in Google’s Top 10 Searches of the year, Alexa shows Wikipedia rising to the 12th most visited site (now displacing eBay), with a reach that has trebled over the last year, and of course Google’s acquisition of Jotspot, Amazon’s investment in Wikia, and a flood of VC funding into new entrants such as Wetpaint.

So where do we go from here? Despite the surge in interest I think we’re still in the very early days.

Wikipedia has proven that, contrary to most people’s intuition and fears, it is possible to shape something of great value out of what appears to be complete chaos. Both Wikia and Wetpaint are doing a good job of extending this approach to allow the creation of intricately detailed sites on topics that only merit a page or two on a Wikipedia (such as Lostpedia). Although the majority of these sites will never gain the necessary critical mass, I believe we’ll see a massive increase in the number of these highly specialised mini-encyclopedia wikis over the next few years.

I also think that wikis will continue their steady march “inside the firewall”. Companies that have spent huge amounts of money on intricate content management systems for their intranets will be reluctant to switch, and traditional corporate paranoia will make lots of companies very wary of ceding so much control down to the “lower ranks”, but an ever increasing number of SMEs and progressive BigCos will continue to take the plunge.

The rise in corporate wikis will also drive a lot of necessary technology innovation. The last year has shown signs of this with the move to Wikiwyg, so that users no longer have to learn troublesome formatting commands. Corporate clients will continue to demand advances that make the technology more accessible to all employees, and also more useful for actual business needs as the initial excitement gives way to frustration at the things that are still too difficult to achieve. I suspect we’ll see some significant developments in the next year relating to finding information that is currently getting lost in the twisty maze of ad hoc content, and many improvements related to dealing with and integrating with more traditional document management (Word), as well as tabular and numeric data and charts (Excel). For the latter, WikiCalc appears to be leading the way at the minute and I suspect we’ll see some of that approach start to make its way into other products.

Along these lines I suspect we’re about to see an explosion of wiki-style technology start to appear in other types of applications, as the wiki ethos starts expanding beyond wikis themselves. We’ve already the very early stages of this, with Amazon and eBay attempting to find ways of harnessing customer involvement throughout their sites, and many software products basically delegating their help and support functions to a customer-powered wiki. Over the next few years, I believe the line between the supporting wiki and the actual product will become much more blurred in numerous products.

My own personal hope is that we’ll also see huge growth in support for structured data in a wiki environment: both in terms of the underlying technology and also in sites actually using it. It will be interesting to see what Google do with the JotSpot technology here (I believe the current approach leaves something to be desired, but the idea is good), and Semantic MediaWiki is continuing to move forward at an impressive rate. However, much as I might desire otherwise, I suspect we need another year or more of increased acceptance of the general wiki concepts before we really start to see the benefits of structured wikis emerging on a wide scale.

Tags:

Web 3.0 is made of people

November 15th, 2006 4 comments

John Markoff wrote in the NY Times earlier this week that Web 3.0 is coming. In some quarters he may as well have said that that sky was falling. The Web 2.0 moniker is still contentious in its own right, and enough scorn has already being poured on companies seeking investment for their Web 3.0 product, without someone trying to lay claim to the name now for the next phase of the Semantic Web.

Ross Mayfield is among the people pushing back on this, citing his earlier “headline only” posts: “There is no Web 3.0″ and “Web 2.0 is made of people”.

I find myself the rather interesting position of agreeing for the most part with both John and Ross.

For the most part, the whole version numbering of the Web is stupid. Websites in 2003 were wildly different from those in 1999, but no-one then was declaring us to be in the era of Web 1.7 or Web 1.2. But “Web 2.0″ let people make a break from the past. A lot of the confusion over what Web 2.0 actually means really stems from everyone having something different they wanted to escape. Two of the main splits were from web-designers and entrepreneurs. Web designers wanted to get away from an era of not being able to do anything interesting because they had to spend so much of their time supporting old crumbly browsers. Google Maps gave them a big example to cite when arguing with their boss that the technology has been available long enough and if anyone can’t use it, that’s their problem. Entrepreneurs wanted to bury the “internet bubble” hangover and start building new and exciting things again. For a few years VCs had been complaining that they had no way to get rid of their money, and that they might have to start returning funds, uninvested. Meanwhile Y Combinator was massively increasing the visibility of angel funding on the back of Paul Graham’s relentless evangelism of the start-up religion. Google and Yahoo were on just enough of a buying spree to make first time entrepreneurs believe that all they had to do was work like mad for three months and then flip. And with the bookshelves full of titles like “How To Build Amazon.com in Just 72 hours with Ruby on Rails” everything was primed for something to happen.

Naming is important. Most of these things had been ticking away for quite some time. Just calling the technical side “Ajax” sparked much more interest and development than having to explain what XHttpRequest could do for your JavaScript. Calling the business side “Web 2.0″ had much the same result, even though, in both cases, the terms are still mutating daily. The terms have great value, even if they’re mostly meaningless.

But I’m certainly not ready to say that anything should be called “Web 3.0″, and certainly not the Semantic Web. Mostly because I think it’s a silly thing to do, and I don’t want the Semantic Web being so easy to dismiss. I think both John and Ross miss the point on this front. Or, perhaps more accurately, they both see it, but don’t really understand it.

Because at its heart, the Semantic Web is more about people than machines. Stories about how, with a little work on setting up the right ontologies, you’ll be able to suddenly merge databases together with the magic fairy dust of RDF are rightly laughable, whether you’re painting grandiose visions of the entire web, or even integrating a single company’s client list with that of its new parent company. The biggest difficulty in any of these projects isn’t technical, and can’t be solved by computers. In the absence of any other identifying information, a computer can never reason out whether, as Clay Shirky asked, your John Smith is the same person as my John Q. Smith.

But whereas this type of problem created the brick wall for old school AI, we live in a new era, where as Aaron Swartz has pointed out, our choices aren’t just million dollar markup or million dollar code. We can now also harness million dollar users. Wikipedia is fast becoming the quickest way to find information on almost anything, including things that would otherwise not be seen as particularly encyclopedic. (I was on holiday in Estonia when the final Formula 1 race of the year was being held in Brazil and wanted to find out what local TV channel it would be shown on, and at what time.After half an hour of fruitless searching through a variety of TV listing sites, I discovered that the Wikipedia F1 page included a list of which channel broadcast the race in each country, and from there was quickly able to find what I needed.)

But Wikipedia is essentially unstructured. It has an increasingly elaborate templating system underneath that lets users create “infoboxes” of pseudo-structured data, but trying to write software to extract information from wikipedia pages is a step back to what extracting information from Amazon was like before they introduced web services – fragile screen-scraping destined to collapse at the tweak of a page layout.

If it were possible to extract structured information from Wikipedia, the possibilities would be endless. But, of course, extracting structure where there is none is back to million dollar code. There are, however, moves afoot to extend the Wikipedia syntax to allow for the semantic annotation of facts. This would allow the million dollar users to gradually layer structure over the information. This structure would then be queryable, both inside the wiki and outside. There would be no more need to maintain by hand all those “List” pages, like the “List of rivers of Africa“. Each river could just be annotated on its own page as being “in Africa”, and this page would be a simple query. The page for each river already includes information on what country it is in, what length it is etc. But this is purely human readable. With just a slight extension to the wiki syntax, this information can be annotated in-place and become machine readable. (In the most popular extension it’s somply a matter of extending “… is a river in [[Zimbabwe]]” to “… is a river in [[location::Zimbabwe]]”) Most users don’t even need to learn the syntax. Wikipedia has already shown that if someone creates the initial information a lot of other people will turn up to tidy and rearrange it to fit the current social norms.

This approach works bottom up, rather than top down. There’s no need to wait for a Standards Committee to construct an elaborate ontology. You can just create new relationship types on the fly, in true wiki fashion, and then work out later how to best describe the actual meaning of the relationship. Standards wonks can come along behind the normal users and annotate the meta-information at one remove to enable more complex queries or reasoning. Purists hate the idea, of course, and there are thousands of reasons why it could never work. But there are thousands of reasons why wikis in general don’t work, and why wikipedia can’t even conceivably work, and that hasn’t been much of a problem so far. Like most things in computing these days, the biggest issues are really social, not technical, and Wikipedia seems to have found a way to work despite, on indeed, perhaps because of, all their social problems. So at this stage I think I’d be willing to take my chances on rewriting the future, and place my bet on it actually being Wikipedia Takes All.

Tags:

Wiki Comparison

September 27th, 2006 No comments

Open Source Classroom has an excellent in-depth examination of most of the main wiki options that I’ve seen in quite some time. The requirements are a little different from most (collaboration amongst 10 and 11 year old school children), but the differences aren’t really that different. It’s not just children who don’t want to learn complex wiki markup, and many companies wouldn’t really want uncontrollable Google ads throughout their site either.

Mediawiki, Pbwiki, Jotspot, Wikispaces, Dokuwiki, Netcipia, Editme, and Wetpaint are all considered, and all come up short. You can read the review yourself to see why, and discover which one was eventually chosen.

Tags:

More on Accounting Wikis

September 23rd, 2006 1 comment

Karen has contributed some useful thoughts in response to my previous post about using Semantic Mediawiki as an accounting system.

One of the key things she highlights that I’d never really put my finger on before, is that traditionally recording financial information is a specialist skill – that of a bookkeeper. This made sense historically when there actually were physical day books and ledgers into which every transaction needed to be entered and then turned into a trial balance to be passed to the accountant. But technology isn’t meant to just replace paper with pixels, it’s meant to enable us to find new and better ways of performing tasks. For some reason bookkeeping software never managed this.

Most people can learn fairly quickly how to add a basic sales or purchase invoice to any financial system, or to record payments for either. This is because all software now hides the underlying double-entry nature of these functions. It’s a simple one-screen process. The software successfully hides the complexity by translating the user input into the full long-winded procedure that would otherwise be required.

But once you get outside this basic comfort zone, the learning curve gets very difficult very quickly. Buying capital equipment via hire purchase? Ouch! Better ask an accountant how to record that. Issuing an annual invoice that a customer pays monthly by direct debit? You need to make manual adjustments every month. Even something as commonplace as payroll is overly complex in most systems (even when you buy their expensive add-ons). Yes, accounting for payroll is complex. The company has to act as an unpaid tax collector, removing PAYE and NIC at source to be paid over to HMRC at a later date along with some extra Employer taxes. The employer and employee might also jointly contribute to a pension scheme, which needs handled separately again. Increasingly companies are also being asked to withhold other money from the employee at source, e.g. for child support. There are too many possibilities for accounting software to be able to hide all the complexity.

In traditional software packages, structure comes first. You can’t just record the data and hope to sort it out later. With a wiki, however, this is reversed. The structure and meaning can emerge later. As long as all data is recorded and annotated in a sensible way, we can work out how to deal with it later. If some new government Mandated Savings System was introduced that requires us to withhold 1% of an employee’s salary every month, to be held for them and paid to them only upon leaving the company, we wouldn’t need to wait for a software update or until an accountant worked out for us what the various balancing entries should be. Someone could just add an extra line to the salary payment entry ([[mss amount:=21.91]]), and working out how to actually account for that not only becomes Someone Else’s Problem, but, critically, can happen at a later date.

And if the structure turns out to be wrong, that’s not a problem either. It’s trivial for someone to just go through and change each entry, or even write a little script to automate this if there’s too much needing changed (with, of course, a full audit trail built in, for all those accountants having heart failure at this point).

It also means that any information that you want to capture for creating management accounts, rather than financial accounts, can be trivially recorded as well. Want to record which sales team is responsible for a sale? Which floor of your office this bottled water bill applies to? Which developer ordered this book from Amazon? Just do it. As soon as you annotate the information, with a new field, the field automatically springs into existence, and it’s instantly possible to query on it. Open a second office? Just inform everyone that they now need to say [[Office::Belfast]] or [[Office::Lisburn]], and you’re done.

The possibilities are, literally, infinite.

Tags: ,

Socialtext 2.0 raises the wiki bar

September 22nd, 2006 1 comment

Socialtext have just launched version 2.0 of their wiki product. I haven’t had a chance to actually play with it yet, but based on what I’ve read so far, and on watching the screencast of the new features, this is a phenomenal improvement over the previous version.

The original version too readily betrayed its roots in Ingy‘s ‘kwiki’ project. This was a really simple way to set up a wiki back in the distant past of 2003, and sparked a wave of plugin development for a while, but in recent years has fallen out of favour, with more and more people preferring MediaWiki for a Free Software Wiki.

I don’t know if Socialtext have built version 2 from scratch, or whether they’re still building on a kwiki core (the source is available, so I could presumably find out easily if I wasn’t feeling so lazy), but they’ve moved a long way beyond it now.

The most obvious addition is Wikiwyg mode, which is hardly surprising as they’ve been publicly working on that for quite a while now, and have teamed up with Wikipedia to try to make it happen there too. But there are lots of other nice features too. All the best ideas from the various other wiki products out there appear to have been neatly packaged up into a coherent whole with a lot of magic ajax dust to simplify many key tasks.

Tagging seems to be becoming popular on wikis now, and will hopefully displace the more hierarchical ‘Category’ approach of Mediawiki. Socialtext makes adding and removing tags very simple, especially with a del.icio.us style auto-suggest feature. Attaching files to a page also appears to be a nice feature, but I’d need to play with this to see how well the file handling is built in to other features (like search).

Of course there’s RSS everywhere, not just on recent changes and watchlists and the like, but also for any search query you want to watch over time.

Once nice feature from Kwiki has been retained: the ability to see inline what other pages are linking back to the page you’re currently on. Although most wikis have a “What Links Here?” feature, my anecdotal experience implies that hardly anyone ever uses it. Bringing this information right out front lets people stumble across interesting connections they might otherwise have been completely unaware of, which is a significant plus in a corporate or community setting.

The built in blog tool also looks interesting. There have been a few people playing with the wiki/blog combination for a while, but most just end with the worst of both worlds. On the surface Socialtext appear to have managed to get the best of both: pages that look and feel more like a blog than a wiki, but with the full power of a wiki backend – particularly for linking to other pages.

None of these features are particularly innovative: the real breakthrough is in just stealing lots of good ideas and tying them together really well. The one feature that did surprise me though, and maybe the key innovation (unless they snaffled it from some other wiki software I haven’t encountered yet), is the Dashboard.

In my experience the “Home Page” of a corporate wiki is generally terrible, and either remains fairly empty, or becomes little more than a mass of links to all the pages people wanted to create but didn’t know where to link them from. Socialtext have blown this concept away and replaced it with a Personalised home page for each user, displaying a dashboard of useful information: Recent updates, changes to your watched pages, a shared whiteboard, a personal (but not private) notepad, etc.: all the pages that people would either have to look at regularly anyway, or, more likely, forget to look at regularly. Bringing them together into a single page suddenly makes that page much more valuable, particularly, again, in a corporate setting.

Apparently there’s a whole bundle of new features under the hood too, particularly an API for interfacing with other software or building your own extensions, but I haven’t had a chance to explore these yet.

Unfortunately there doesn’t seem to be anything for storing structured information, which is now a key requirement for our corporate wiki, or I’d seriously consider switching to this. Here’s hoping for 3.0…

Tags:

Using a Wiki as an Accounting System

September 21st, 2006 11 comments

In the UK, the de facto accounting system for small business is SAGE. This is most unfortunate, as it is an appalling piece of software for many, many reasons. I can rant for hours on its many shortcomings, but today I’ll mention two of the biggest ones, and how we’ve been able to work around these using a wiki.

Firstly, its reporting is terrible. It can, of course, give you all the traditional accounting reports – P&L, balance sheet, cash flow, actual vs forecast, debtors list, creditors list, etc. etc.. In recent years they’ve even added the ability to produce these in HTML. However, there is no way to interrogate the output or investigate things further. When I see a financial report I want to be able to click on any piece of data on it to get further information. This is a classic case of a 90% solution. There are really complex tools available to let you slice and dice data any way you want, but in a small business, just making each number a link to a screen that shows you what makes up that total is generally both necessary and sufficient. SAGE fails miserably here. I’ve encountered many companies who went used to go through the process of getting their bookkeeper or accountant to print these reports on a regular basis, but never really understood what they were seeing, so eventually reverted to the traditional “gut feel” approach that most small business owners prefer, leaving formal reporting for their annual accounts and their semi-regular reports to the bank manager.

SAGE does allow you custom build your own reports, but as far as I’m aware there’s no facility to allow for the creation of “interactive” reports. It’s also highly likely that some enterprising company has built some sort of add-on to do this, as there’s an entire third-party ecosystem surrounding SAGE, but I’ve never found one.

This reporting problem falls into the “generally annoying, but I guess I can live with it category”. Historically my general approach has been to export the pertinent information as CSV and manipulate in Excel – what most companies seem to do when they get big enough to move up from having a simple bookkeeper to actually hiring an accountant.

The much bigger problem however — the one that makes me tear my hair out when having to work with SAGE — is how it deals with accruals. This is an accounting concept whereby you are aware of income or expenses that will apply to an given accounting period, but for which an invoice has not yet been raised. The classic example is a bill that is issued quarterly in arrears. A quarterly bill for £60 should really be entered as £20 for each of the three months. But when you go to close April’s accounts, that quarter’s bill hasn’t arrived yet, so obviously hasn’t been entered on the system. As such your accounts don’t accurately reflect the costs incurred in the month. So you have to accrue a £20 charge that can be reconciled when the bill finally arrives. Similarly, if the bill is quarterly in advance, you don’t want to just charge the full bill against the month when it arrives, so you assign parts of it to each of the next two months. SAGE makes this remarkably complicated. Bookkeepers and accountants have gotten so used to that they don’t really think about it, but watching the confusion of a new admin assistant learning how to do it drives me insane. As a result, many small businesses just don’t bother doing this and their monthly accounts swing wildly in the months where the bigger invoices hit.

One of my standard rants to accountants is about how, yes, double entry bookkeeping was a great invention in 12th Century, but it’s no longer a useful metaphor for users of a computerised accounting system that should be able to take care of such things by itself. Business owners and managers want to enter sales and purchases, note when these are paid, and then check it all off against the bank statements each month to make sure nothing untoward has happened. They don’t care, and shouldn’t need to care, about journals and ledgers and trial balances.

But there seems to be no such thing as “modern” accounting software. They’re all just computerised versions of an ancient system. Some manage to hide more of the underlying mechanisms more than others, but it doesn’t take much scratching to discover what’s really under the surface.

When we took over the running of Ireland’s oldest ISP a few years back, the accounts were a mess. Customers were billed monthly, quarterly, six monthly, and/or annually – some in advance, some in arrears, some in the middle of their billing cycle. How they were billed also bore little relation to how they actually paid – many of the customers who got annual bills actually paid monthly, for example. Suppliers, in turn, tended to invoice quarterly in advance. Large software development projects also carried a substantial Work in Progress element. It was a huge amount of work at each month end to enter all the necessary accruals and movements, and the resulting P&L and balance sheet were mostly incomprehensible due to the wild swings involved in booking or releasing money from customer pre-payments, often received months earlier, but which have to be treated as a liability until that month’s services have been delivered.

The simple solution to this problem is of course to just allow each invoice (whether sales or purchase) to have an associated ‘period’. A phone bill charges for the previous quarter’s calls, and the following quarter’s line rental. Most companies account for these separately so already split the bill and assign each a different nominal code. The trick is to also add the relevant dates to each: calls from April 06 – June 06, line rental from July 06 – September 06. The software can then do all the calculations it needs for accruals each month without the need for a hideously complex manual month-end process. SAGE, however, doesn’t allow you to do this. They’re not alone. Every accountant I’ve spoken to is completely unaware of any low-end financial software that does. But they all agree that it would make the accounting process much, much simpler for most companies. I can’t believe that no-one else has ever thought of doing this, so I’m left with the assumption that it’s just a giant conspiracy to create unnecessary jobs for millions of accountants and bookkeepers.

So, in frustration, we decided to use a wiki for our accounts instead. Mediawiki with the Semantic extensions makes for a very interesting data store, as I’ve written about previously. For basic accounting information there are only really a few key concepts: purchase invoice, sales invoice, payment made, payment received, bank lodgement, bank statement.

Each gets its own Category, and has a set of relationships and attributes defined. Each invoice has attributes for net, VAT, and gross amounts, a nominal code, an issue date, and a from date/to date pair for the period it applies to. Each also has a Supplier/Customer relationship which of course links to the page on the wiki where we’re already storing information for that company. A cheque received relates to one or more invoices, has attributes for the amount, and is related to a Bank Lodgement, which in turn is an amount on a date to a bank etc. A provisional bank statement can then be generated through an <ask%gt; query for each transaction applying to that bank in a given period. This can then be visually cross-referenced against the physical statement when it arrives to make sure that nothing is missing or erroneous. Most small companies are on cash accounting for VAT, so the bank statement query can also be set to also include the VAT portion for each transaction, and a simple sum provides all the information necessary for the quarterly VAT return.

Producing a P&L is trickier. The syntax isn’t really expressive enough to do this yet, and I’m not sure it ever will be. Even if it allowed the full Sparql syntax, as has been mooted, I suspect the computations to properly assign invoices over various accounting periods might be too complex. At the minute we produce these by querying the wiki’s underlying database directly with a Perl script that does all these calculations. This also means, of course, that each piece of data on the resulting report can be a link to a page explaining how it’s made up.

The process is, of course, error-prone. But it’s much, much easier to correct errors in a wiki than in SAGE. Mediawiki, of course, maintains a full revision history for every page, so there’s a built in audit trail that’s much easier to examine than the one in most financial systems. We’ve created a few scripts that run regularly to look for obvious errors, such as invoice items where the net + vat doesn’t equal the gross. The syntax is still too simplistic for some of the reporting we’d like, so until it provides aggregate functions, or until we learn how to use RDF better, we still have to do too much digging at the raw data with SQL. It’s very far from a complete financial system, as we haven’t really needed to deal with things like share capital (although asset purchases are simply a purchase invoice with a very long from date/to date range, so the bulk of a balance sheet is there). And it’s built on a fast changing base – Semantic Mediawiki is still in the early stages of development. So it’s certainly not something everyone could, or even should, do. But it has allowed us to remove a huge amount of recurrent admin work, see much more clearly what the financial status of the company is in real time, and drill down into that data where something seems puzzling.

All we need to do now is edit the ten years worth of historic data that we imported to assign the correct periods to each invoice…

Annotating Links

June 12th, 2006 No comments

I’ve recently added the Semantic Mediawiki extension to the Nigov wiki. This is a plugin that allows for the creation of semantic information within a wiki.

The basic principle is extremely simple. Rather than just creating a normal wiki link from one article to another, you annotate that link with information what sort of relationship it represents. This is similar to XFN, and other microformats, except they live in wikitext rather than HTML. Being a wiki, however, they don’t need to be predefined anywhere – the first time you create such a relationship it springs into being as a new “blank” relationship that you can describe or define further.

So, for example, on Nigov we store minutes from Belfast City Council meetings. One of the advantages of having them in a wiki, over the official Word documents, is the linking. The list of councillors who are in attendance can be linked to each of their individual pages. When minutes of previous meetings are approved, they can be linked to. With Semantic Mediawiki we can go even further. In a normal wiki there is no distinction between the link to a councillor who is in attendance, and the link to one who sent apologies. Now, however, we can annotate the links to differentiate between them. On the text of the wiki page for the minutes there is no obvious difference (other than the summary box at the bottom that shows all the semantic information known), but we now have two extra benefits.

Firstly, every page is now addressable as RDF. These can then be fed into a reasoner, or picked up by a semantic search engine. We’re in the early days of the semantic web so this isn’t really as useful yet as it will be in a few years time. But even now there’s still a major benefit: the internal query facility. As well as a built in “Simple Semantic Search”, you can also dynamically include the results of queries into other pages.

So, by annotating each councillor with links like: “[[Is councillor for::Belfast City Council]]”, we can now auto-generate the list of all councillors. Of course, we were always able to do this if we added them to a Category, but if we extend this with “[[electoral area:=Pottinger]]”. and “[[party member of::DUP]]”, we can pull that data into our resulting table as well.

Thus, our list of councillors page, rather than having to be maintained by hand, is as simple as just “ask”ing:

<ask format=”table”>
[[Is councillor for::Belfast City Council]]
[[electoral area:=*]]
[[party member of::*]]
</ask>

Most of the data in the Nigov wiki still needs to be annotated like this before we’ll be able to generate the interesting reports (how often do councillors attend meetings vs send apologies?), and we probably won’t be able to ask the really interesting questions (“have any councillors attended a meeting that allocated funds to an organisation they’re involved in where they didn’t declare a conflict of interest?”) until there’s a proper reasoning engine underneath, rather than the current SQL model.

But for a little extra work when adding links to a page, we get a pretty good payoff now as well as the potential of even more later without having to do any additional work then. Seems like a good deal to me.

Tags: ,