Of babies and bathwater (or: Why I love the Zope Component Architecture)
In defence of one of the great Python frameworks of the past decade.
I went on holiday for four days. When I came back and opened Thunderbird, I saw that there were 97 new emails in the zope-dev newsgroup (mailing list). Controversial debate, here we come.
This particular debate was instigated in reaction to an earlier thread by Chris McDonough, who was trying to bring some BFG API improvements back up to zope.component. I always value Chris' opinions on design and development - few people are as committed to software maintenance, programmer friendliness or high-quality documentation as he and his colleagues at Agendaless are. And even though Chris' proposal didn't quite fit in the end, he inadvertently re-kindled a debate about how we can improve the API of the Zope Component Architecture (ZCA), i.e. zope.interface and zope.component.
Recently, there's been a sense of backlash against the ZCA. It's too hard. It's too Zope-specific. It breaks expectations and has a sub-optimal API. This has sometimes spilled over into a generic mistrust of frameworks. Frameworks are too hard. They're too application-specific. Many of them are too esoteric and break expectations in one way or another. We all just want to write Python code with simple functions and simple data structures and maybe a decorator or two, dammit. None of the complexity that requires us to read documentation and learn new concepts.
Don't get me wrong. Many of these criticism are justified in part. ZCA over-use is a common affliction in some parts (and I shall certainly take my share of the blame). Over-generalisation is the enemy of discoverability. The ZCA, like all inversion-of-control frameworks, encourage separation of concerns to the point where you can't always find the implementation for something without understanding what registrations are currently in effect. Ian Bicking used the term "non-locality" of code, which is definitely something that ZCA espouses - even encourages - and this comes at a cost.
And yet. The ZCA rocks. It's an incredibly advanced and powerful way to build software. When I work in other environments (or other programming languages), I often miss it, and I sometimes end up re-inventing parts of it for my own needs. Zope, Plone, Grok, BFG and the other frameworks that use the ZCA constitute some of the largest and most complex codebases in the Python world. They are also some of the most extensible, customisable products in existence - probably in all of software. The ZCA sits at the heart of that, and has (ignoring a huge amount of legacy code that clouds the picture) facilitated a type of consistency in extensibility that is difficult to emulate.
There is no doubt also that the ZCA adds complexity. It demands that you swallow its core concepts up front (interfaces, adapters, utilities, events) and get to know them pretty well. It has a few nooks where people can do crazy things, usually doing as much harm as good.
But my take on it all is that the "framework backlash" is wrong. It seems to happen when people who've been working on high-complexity software starts building small and simple things, using elegant frameworks like BFG or Pylons. A whole application is just a couple of functions, a few dicts and lists, and maybe a page template or two. Why can't all of our code be like this? But of course it can't. When you start building software that is tens of thousands of lines of code, you need architecture, you need formalisms, you need common patterns and repeatable techniques. You need to cater for teams that change, skills that vary and codebases that get so big no one person can manage it all at once. You need to consider how your code will evolve, because the code you wrote first will be in dire need of refactoring by the time you're half way through your project.
That doesn't mean the ZCA is right for all situations. Rather, I'd argue that learning to use the ZCA effectively is a bit like learning a new programming language on top of Python. You need to "get" those concepts, just like you need to "get" functions, classes, for loops and if statements. You need to gain some experience and learn some common patterns and techniques. In short, you need to invest some time and become familiar with your tools. That investment will pay off for a big project, or a complex one. It probably won't if you're building a small, stand-alone application. And for the huge grey-zone in-between, you're going to have to make informed decisions.
For the ZCA itself, there is clearly an opportunity right now to evolve, and a degree of energy poised to make this happen. My personal wish-list would be the following:
- Improve the documentation. Chris McDonough pointed out that when you try to explain something, you'll find holes in your concepts. This tutorial is my latest attempt at explaining the ZCA concepts - it may be helpful, either if you'd like to learn more about the ZCA, or if you'd like to try to write canonical ZCA documentation (in which case, feel free to borrow liberally).
- Document clearly when the ZCA adds value, and when it may be overkill.
- Improve the API, but don't sacrifice backwards compatibility and don't break existing documentation. We've had a long education process to date, getting people familiar with the ZCA concepts. We need to avoid the kind of damaging mega-confusion that surrounded the whole Zope 2 / Zope 3 / Five debacle. Giving a new name to an existing concept or a new signature to an existing method is the kind of thing you do only to spite your users. Live up to past decisions, even if they weren't optimal, and find a pragmatic way forward rather than wish every effort can be turned into "green fields" development.
- Build better tools for introspection and debugging. The ZCA maintains a lot of runtime state about components, which could be used more effectively to help people see which registrations are in effect.
And finally, some advice for projects, such as Plone, that use the ZCA for extensibility and inversion-of-control : Don't let the ZCA be the first API that people see, and don't assume everyone has yet reached ZCA mastery. Think of the ZCA as a building block, not the end goal.
BFG does this very well. A particular feature may be implemented using the ZCA, in order to be allow different policies to be swapped in, say. But for the casual user of the framework, there is a helper method which performs the necessary ZCA lookups and presents a domain-specific, documented API for a particular purpose. And it documents its design decisions. That's maturity for you.
We're trying to do something similar with Dexterity and the ecosystem of tools around it. The same will hopefully be true for other technologies as we move towards Plone 5 and what we hope will be a much improved integrator learning curve. Stay with us. The ZCA certainly will.