For the last few months I’ve been a slightly more than casual observer of the Maypole project. Although I don’t actually run any sites on Maypole, it’s based on rather a lot of my public code, and a lot of issues people raise on the mailing list aren’t really about Maypole, so much as Class::DBI or CGI::Untaint or Class::DBI::FromCGI or somesuch.

It’s also been interesting to watch as, although it’s not directly connected, Maypole seems to have been heavily influenced by Kasei’s internal FireCore framework which Simon had to work with for the guts of a year. He threw away lots of our mistakes, implemented some of the things we’d talked about but hadn’t yet gotten around to, and, being Simon, took some of the ideas much further than we’d imagined. So I was keen to see how it would hold up in the wild.

Of course one of the first things people wanted was to replace the dependency on Template::Toolkit, and be able to use any of the myriad of Perl templating systems. In hindsight I believe implementing this to have been the first major post-release flaw.

I’ve been a student of web abstraction frameworks for a few years now, and the siren song of many of them is the idea that depending on any given component is a problem. Every part should be capable of being swapped out for an equivalent. In database backed web-based systems in particular this means you shouldn’t be tied to any given database or database abstraction layer, or to any given templating system. I don’t know whether it’s a false hubris, or just a good old fashioned desire for world domination, but there’s a pervasive idea that the framework should be capable of doing anything and everything you desire.

But I’ve come to believe that exactly the opposite is true. Frameworks have most power when they aim to do one thing well, and become tightly entwined around the components that make them up. In Maypole’s case, its initial goal was to make it really trivial to build database backed web systems around Class::DBI and Template::Toolkit. Lots of people build such systems using precisely those components, and they all repeatedly face and solve the same problems, time and again. Maypole neatly bundles up and encapsulates one path of building such sites, freeing developers up from the drudgery to concentrate on the value-added business logic.

But converting such a framework to hot-swappable components restricts the amount of best practice than can be neatly encapsulated. Each RDBMS/OO mapping layer does some things better than others. Each templating system, likewise. A framework committed to one of each can play to their strengths. A framework allowing interchangeability has to reduce itself to lowest common denominator.

One the advantages Maypole had over FireCore was a set of default templates. Simon now regrets building these, as they were so good that people believed they should be able to use them for everything, when for most applications most people will need to write their own. But, tied to Class::DBI and Template::Toolkit, Maypole would be able to create much better default templates, making better use of CDBI’s introspection methods, and TT’s VIEWs, MACROs, and Plugins. In a more generalised world this becomes much harder, and the framework becomes blander and blander.

Nothing can be everything to everyone. Too many frameworks try, and end up being used by almost no-one other than the person who wrote it – mainly because they’re the only person who can ever find their way through the maze of twisty abstractions.

Behind my beliefs on this lies one often overlooked truth: frameworks are hard. They’re much more about philosophy than technology. Early design mistakes get magnified dramatically, usually in direct proportion to the level of abstraction you’re aiming at. And the more you build the framework in isolation from the sites you’re building using the framework, the more likely you are to make those mistakes. And once they’re made they’re really hard to recover from.

I once consulted for a company who had what we came to call “three legged cow syndrome”. They had developed a basic content management system that, like many of these things, had morphed into a product that could, of course, do anything any client (and particularly any potential client) wanted. And every time someone came with a request for something the system didn’t already do, they tweaked the system so that it could do that in future. After all, that’s just good practice.

Of course, each time they did this, they codified a hybrid of generic and specific requirements – mainly because they couldn’t know the difference.

Because the first client who ever wanted a certain feature had certain business rules associated with how that feature would work, the code assumed that every client who wanted that feature must work the same way. But of course, every client’s process was different. And, almost invariably, insane. It’s the nature of business systems. They evolve over time to account for all manner of foibles. As technologists we want to simplify these things, and refactor them all away.

But almost every real-world business has all manner of bizarrely crippled logic underlying their business rules. They’re all three legged cows. And in this company every client who later wanted the system to do something slightly differently (as they all did) was treated as if they were insane. Because everyone knows you don’t do that like that – you do it like this. And when they finally prevailed, the developers would have to spend weeks untangling the code to work out what was common and what was unique to each client.

The only way around this syndrome that I know of is to implement the first client’s additions in isolation, as some sort of standalone plugin or add-on. Then the second time someone comes in with something similar do the same, based on what you’ve learned the first time. The third time you can begin to factor out the commonality. Cautiously.

There’s been a lot of furore on this sort of issue recently in the emerging Maypole community. The new maintainer after Simon’s retirement attempted to take the framework too far too quickly. In some ways this speed has probably saved Maypole. By moving at such a rapid pace, no-one could keep up, and people started complaining. If he’d moved much slower, introducing his features one at a time, each one would probably have made seductive sense, and been accepted, until eventually there was nothing left. Frog boiling is a remarkably effective means of changing the direction of any project if you do it slowly and subtly enough.

But now there’s been a fork. Hopefully Sebastian can take the new Catalyst project off to where he wants to go, and Simon F can revert Maypole back to a really good CDBI+TT framework. And hopefully I can avoid being seduced too much, and steal the truly good ideas back for FireCore.