Over the past couple of years, we've seen a dramatic shift in the way Plone software (and Zope software, and, really, Python software in general) is packaged. Our code is now broken into dozens of packages, pulled in via a complex set of dependencies, and managed via powerful tools like Buildout.
However, the part of this whole story that excites me the most is the humble README.txt, combined with the practice of turning it into the "long description" in setup.py that appears on PyPI.
I didn't always appreciate this quite so much, but that little bit of documentation is the minimum you must do if you release something. It is not a substitute for more narrative documentation, which helps people learn and understand how the pieces fit together. However, the package README is the first line of defence for reference documentation. Once people have an idea about how your software fits together, they will find the package and expect to be told enough about to use it without having to resort to code archeology.
Let's look at some examples. First, there's the narrative "how does it all fit together" type documentation. This needs to be kept separate, and must be discoverable independently of code structure. A good example of mine is the Dexterity manuals.
Next, a bad example: The README for plone.theme. Ironically, I was trying to remember how to hook a browser layer up to a CMF theme (the raison d'etre of plone.theme) and got rather annoyed by the total lack of information in that README. I then realised who wrote the package. D'oh.
A better example is collective.beaker, a package for using the Beaker session and caching management framework in Zope 2. Notice how you both find out what the package is for, and how it is used. There are installation instructions, too, and hints on things like testing, where this package has some unusual implications.
More recently, I've started doing this very seriously, for example in the as-yet unreleased update to plone.app.registry and the new-and-exciting plone.app.caching. My coding cycle is now to do the interfaces module first, then code and tests in tandem, and then documentation. I normally start writing the README while waiting for test runs to complete. This cycle is then repeated, so that I update and revise the README as I refactor code. Finally, I read the rendered README through once as a sanity check before doing a release to PyPI.
Importantly, the documentation for the packages above has helped make them substantially better. By trying to explain how they worked, I found troubling inconsistencies and omissions that I'd failed to spot in my unit tests or interfaces.
Put differently, if something cannot be documented properly, it's crap. Find out for yourself before you foist it on everyone else, as Chris McDonough would no doubt agree with.
I'll leave you with some practical advice:
In most countries, small businesses are the backbone of economic growth and employment. This is especially true in technology, where small businesses and startups also often drive innovation and quality improvements in their field.
I've had the opportunity to work for a couple of small (5-20 people) software development companies, as well as for some really big companies. More importantly, I've been in a position to help medium and large clients procure services from smaller vendors, advising on such things as the risk of relying on a smaller partner and the potential benefits of a closer working relationship and specialised expertise.
Sometimes, this type of partnership works very well, but it can also be pretty frustrating for all parties involved: the client, the vendor, and the hapless business consultant in the middle trying to smoothen out the process. That's usually down to details: differences in expectations, poorly written contracts, communication problems, or differences in working styles.
It's not always easy to give feedback individually, but here are some collected thoughts. These are not directed at anyone in particular. In fact, I think I've been lucky with the small businesses I've work with. Still, I'm sure many of us would recognise some of these challenges from projects past.
I'd bet that most small software businesses work primarily for clients that are larger than themselves, be that in the private or public sector. This has implications for the relationship between client and vendor. In particular, the client is exposed to more risk than you are (or at least they'll think that they are), because if you go belly-up, the client is unlikely to get their money back. This is not a bad thing, so long as both sides are aware of it. If the old adage that "the client is always right" holds true, a corollary may be that "a bigger client is always more important than you are".
Of course, that doesn't mean the client should be given free rein to boss you around. If you are honest about your size, the client needs to respect that. If they start asking for the impossible, such as exclusive access to people who are also needed on other projects, sit them down and talk to them about your business and why things are they way they are. If they can't hold up their end of the bargain, it's time to look for new clients.
The size mis-match sometimes becomes acute when a small business tries to sound bigger than they are. Nothing makes a client think "amateurs" like hollow attempts at grandeur or obfuscation. Personal favourites include:
Some people never say "no". The problem is if you tell your clients you can do everything, they expect you to be good at everything, too. And one big failure is going to loom larger in clients' memories than ten success stories. Tough luck. If you're not very good at requirements management, or web design, or PostgreSQL, don't tell them you can do it. Find a way to work with others who are good at it instead.
This also has to do with confidence in front of the client. If you can't explain to a client in a few short sentences what you and your company do best, start working on that right now. As in dating, a show of confidence can go along way towards sealing the deal.
The best focus statements I've seen tend to be:
Conversely, one of the worst ones I've seen went something like this: "Almost no matter what your problem, we'll find a way to apply Python and Zope to it". I paraphrase. And I forget where I read it. But this implies that you have a mighty hammer, and pity the projects that you may look upon as nails.
I submit that this is the kind of thing you need to decide up-front, and be brave enough to stick to it. Companies like IBM will sell you products (e.g. software licenses or hardware) as well as services (consulting, development), but they are huge and they keep those parts of their business quite separate. If you're a small company, you can sell products (i.e. deliver physical products or sell licenses), or you can sell services (i.e. work on projects). You can't do both. Or at least you shouldn't.
Selling things and selling services are quite different activities, that place different demands on a business. If you sell software licenses, clients are going to expect salespeople, professional documentation, rapid security updates, license management and bulk licensing deals, solid support, and continuous improvement to the product. Development requires a substantial up-front investment, as well as ongoing investment in maintenance and updates, marketing and so on.
If you are delivering projects, clients expect you to work on their project full time. As soon as you become successful as a consultant or contractor, you're not going to have time to work on your products. Unless you dedicate staff exclusively to supporting and developing your products, you're going to be caught between the rock of your clients and the hard place of your customers. And few small companies have the luxury of ring-fencing people like that.
Fundamentally, this is the distinction between "business-as-usual" work and "project" work. Mixing the two modes of working is hard enough in a big organisation. For a small business, it normally means working for your clients during the day and working on your products at night. That's not sustainable.
The attraction of being a "product company" is that it promises a stable revenue stream and frees you from the pain of project management and complicated clients. However, most companies struggle to reach sufficient sales quickly enough to make back their initial investment and still pay their staff. Project work at least pays straight away.
If you are working with open source, trying to sell licenses can be even more complicated. Using open source technologies is one of the great equalisers for small companies, who can credibly claim that the risk of going with a smaller vendor is counterbalanced by a community of people and companies familiar with the technology. Unfortunately, there won't be a community around your proprietary product, so you end up undoing one of the good arguments for people to buy your services. And that's before you get into the intricacies of copyright and license terms, made even worse if you ever blur the lines and use a customer project to build a product you then intend to sell to others. (Hint: don't do it.)
This one's related to the previous point, but it's harder to get away from support. If you build a piece of software for a customer, they are going to want support.
You probably can't get away from doing support, but you should decide whether you are going to make most of your money on project fees, or in support contracts, and configure your business accordingly. Clients understand the distinction, and will want to know: are you in it only for new development (in which case they will want strong assurances about continued support and, possibly, supportability by other vendors), or are you after a long-term contract (in which case they'll expect a discount on your rates up-front in return for a longer term contract).
You also need to make sure you dedicate enough resources to support on an ongoing basis. Your current support clients are liable to be your next project clients. That is, unless you bungle that crucial call that comes in just when your team is trying to finish off a project for another client. It can be difficult to balance "old" (and perhaps boring) support work with "new" project work, so make sure you build that into your plans from the beginning, or you'll end up on the nightshift again.
Pricing can be really hard. To set your prices, you need to understand something about what your customers expect, and what your competitors are charging, but both your clients and your competitors have an interest in keeping that information from you. You also need to know when to change your prices, and how to build that into your contracts.
There are some useful rules of thumb, though. First of all, you can charge less as a small company than can a big company. This has little to do with the quality of your work or the depth of your expertise. However, there is an inherent risk premium for the client associated with going with a smaller vendor. If you screw something up, chances are it will sting the client more than it will sting you. At least instinctively, most clients believe that bigger companies are less likely to screw up, and better able to back-fill in case of resource unavailability or skill shortages. To a certain extent, this is true. A big vendor working with a big client has more to lose, and will invest in keeping a relationship going. They will also hopefully have more checks and balances internally, and often more experience on which those checks and balances rest.
Cost is often a reason put forward by those in the client's procurement process wanting to go with a smaller vendor. If your costs come in as high as or higher than a "big" name, those champions will have a harder time explaining why your tender should win.
In a very unscientific estimate, I'd suggest that the best place to be is in the upper third of the price range for your local market. If you're the most expensive vendor, you better be damned good, because as soon as the client has to cut cost, they'll start at the top of the rates card and work their way down. If you're the cheapest bidder, they may question your seriousness. Cost can be a signal of quality, but don't confuse cause and effect: you can't charge more in order to appear like a gold-plated option. You'll be extremely closely scrutinised if you make that claim, and if your claim doesn't hold up, you'll be dropped like a stone.
You also need to be transparent about your pricing. Pricing can be very psychological. Many clients will be unhappy about paying a high day rate for a small vendor, even if the total cost is dwarfed by other projects. It simply feels wrong for the five-person company to charge more than IBM or SAP do for their top consultants. This can quickly turn into a charge that you are out of touch with your market.
You should have a rate card. For standardised services, like support, you can put this on your website. For projects, keep it confidential, but share it with your clients. Your rate card needs to set out a day or hourly rate for each type of resource you have, e.g. senior developer vs. junior developer. Clients also like discounts, so feel free to show them a rate card and then outline a price based on a discount.
No matter what you do, though, for the love all that is good and holy, produce a simple price table that makes it abundantly clear what the client has to pay, and when they have to pay it. If the client has to think (or bring out a spreadsheet) to figure out the total cost, you've lost. If it's not clear what's to be paid up front, and whether there is an annual cost involved for support, you've lost. For example, if you itemise "development" and "annual support", is support included in year 0? Is it payable from when the project begins or ends? What if there are multiple releases? "Creative", unclear pricing is probably the number one way to look unprofessional. You should have a single pricing model and stick to it, not leave it up to each tender to come up with some way of making money.
Finally, a word about pricing in its wider context: you need to measure your recovery. How many hours do your staff work? How many hours do you pay them for? How many hours' worth do you invoice the client for? And how many hours' worth do you actually get paid, on time?
If you don't have these or similar numbers readily to hand, you're not running your business properly. A lot of small companies fail to invoice regularly, or send invoices that contain errors. This is unforgivable and wreaks all sorts of havoc. If you've ever seen anyone try to bend their SAP finance system to accommodate a vendor that does unusual things, you'll know what I mean.
Ask yourself this: could you lower your prices and still make the same money if you actually managed to recover fees for all the hours you spent on a project? If the answer is yes, you should try to do that, because your customers will feel like you're a more predictable partner, you will appear more professional, your staff will feel like their work counts for more, and you are more likely to win work since you now have a cost advantage. Driving this type of efficiency in your own business is hugely important, and requires constant attention and measurement. And proper internal systems for things like time writing and invoicing. Don't skimp on these. And please don't try to build your own, unless you happen to also sell those types of solutions to clients.
Just as important as knowing yourself, is knowing your customers. This is where industry specialisation can help, although I think that often happens more by accident and word-of-mouth marketing than by design. Ideally, you want to walk into the first meeting with a new or prospective client armed with a solid understanding of who they are, and what they do. Do some research:
I've seen many a small business win out over a bigger rival because the client says, "they know us best". A close partnership and good personal relationships are your biggest assets when you're small, especially in the private sector.
Everybody hates writing tenders. They take way too much time, involve a lot of tedious questions, and the outcome is by no means certain. And without fail, they end up keeping you up into the wee hours the night before submission. However, your tender response is the first impression you leave with a new client. Some obvious things that help are:
The tender might get you in the door, but you've got to work hard to stay there. In general, I've found that there are two types of clients: those that have a project management office and expects you to work within their process, and those that expect you to get on with the job yourself. In either case, you'll get the blame if something goes wrong.
If the client has a process, learn it. You should ask for proper training, and figure out where to get support if you have questions. The last thing you want is to be remiss in some project management deliverable. The kind of people who call you up on such things are unlikely to be impressed by your programming skills.
If the client doesn't have a process, you need one. My advice would be to pick one that already exists and is well known, rather than trying to invent your own. Get your people certified in it if you can. Scrum is a personal favourite, but the most important thing is to have a process and sticking to it. It's important that your process includes a means of managing requirements throughout the lifecycle of a project, and gives lots of opportunities for interaction with and feedback from the client.
Every time you communicate with your client - by email, letter, phone or in person - you are either building on or taking away from their impression of you. That goes for every member of your team. If you slide into jargon, come across as flippant (rule of thumb: no humour in emails), or simply give the impression that you are out of touch, you're in trouble.
Also be aware that email is the most dangerous form of communication. A lot of problems in my career could've been avoided had I spoken to someone on the phone or in person, rather than by email. One of the biggest lessons I've learned is that open source developers have an entirely different way of reading and writing emails than clients. A lot of people don't take well to long emails. Some people don't feel it's obligatory to respond to emails, but will normally answer the phone and return messages. Businesspeople often don't understand inline quoting, either.
Finally, be professional at every turn. For example:
These are little things, but they can make a big difference in how you are perceived.
One of the great things about my current job is that we almost always insist on working on-site. The one project where we didn't spend much time on-site was also one of the hardest, and that was not an accident. Working on-site lets the client see the work you do. It allows you to meet people spontaneously or as needed to clarify requirements or other aspects of the project. It allows you to socialise and form relationships beyond the purely functional.
Working on-site can be difficult if you're doing development. Things like corporate firewalls often get in the way. And of course, not all clients are just down the road. Try to do it at least part-time anyway, especially in the early phases of the project.
Hire a lawyer. Make sure you understand every contract you sign. Make sure any contract you ask the client to sign has been drafted by a lawyer and doesn't contain anything "weird". Weird contracts are a great way to mire a project in delays, and can come back to bite you in ways you'd rather not know about.
I hope this list has been useful. I'm sure you could add many more. Feel free to use the comments to share your own experiences, or tell me that I'm wrong.
]]>Plone is true "community-driven" open source project. No-one is paid specifically to work on Plone, and yet it gets developed. The Plone Foundation protects Plone's intellectual property and handles other issues (like commissioning marketing materials and managing domain names), but takes a completely hands-off approach to development. And yet Plone development continues apace. No one company has a monopoly on development, and yet many small companies and individuals manage to self-organise into building the content management system you know and love (or love to hate).
How does this happen? How does the content management system, which you are implementing for your business or selling to your clients or using on a day-to-day basis, come to be?
That's right, open source software is built by contributors. People who write code and documentation, who test and report bugs, who evangelise and organise conferences and other events.
But who are these mythical contributors? By and large, they're people like you and me. They have needs to solve. They have projects to deliver. They have customers with requirements. And they have found out that by contributing to the project, they gain something.
There are several reasons why people contribute to open source (some of them outlined in my Master's thesis), but here are some of the most important ones in the context of Plone:
Contributions don't necessarily come in the form of code either. Some of the more valuable things you can do are:
Everyone, without exception, has the capacity to contribute. And yet, many are apprehensive. Let's count down the most common excuses.
Guess what, no-one else has any time either. Not one single person is paid specifically to work on Plone. Do you really think that you are so much more busy than everyone else?
This is of course a very valid excuse. Plone shouldn't come at the expense of your personal life or client deliverables. But this is not a zero-sum game. Time spent contributing has benefits. The smart Plone companies have realised that they owe their business in part to the community of contributors, and need be a part of that to stay relevant, to improve their standing in the market, and simply to have a platform to build on in the future.
Find out how you can contribute, and take it seriously. Many Plone companies have vowed to spend 5-10% of their time on community contributions. One way to do that is to send people to conferences and sprints where they can connect with others and work on Plone core.
Furthermore, the best Plone companies will find ways to improve Plone in the course of their customers projects, for example by writing re-usable components instead of "point solutions" where possible. Customers benefit here too, because they end up with more maintainable software that is understood by a larger community of vendors. The solution that is cheapest in the short term is rarely the cheapest over the total lifecycle of a project, and vendor lock-in (even to a vendor as nice as you and your company) is a risk not to be taken lightly. It's pretty likely that your customers chose open source and Plone at least in part for these reasons, so don't be afraid to make the case that contributing to the community makes sense for them as well as for you.
Trust me, if you know where to look, you'll find code in Plone written by programmers way less skilled or experienced than you.
This is probably the most common excuse, but it doesn't hold water. Everyone can contribute, and writing code is far from the only thing that is valuable. You are never going to get better unless you try. Perhaps you shouldn't try to perform major refactoring or change any fundamental behaviour the first time around, but bugs are fair game, and Plone has a well-established process for reviewing proposals for new features.
More importantly, you'll find that if you try to contribute, you will get a lot of help and kudos. Existing contributors are much more likely to help those who try to help themselves than those who throw up their hands and say they can't or won't (but still complain about bugs going unfixed). Ask in the IRC chatroom or the on the mailing lists. Make it clear that you're trying to do your bit. I for one will help you if I can. And before you know it, you'll be the one helping others get their contributions in. We all started somewhere.
Plone in particular has a low barrier for contributions. There are people who read the checkin mailing lists and will email you if you did anything wrong. In the words of Plone's co-founder Alexander Limi: It's better to ask for forgiveness than permission.
We have a tool for you: it's called Subversion.
Some people seem to think that getting a fix into Plone will take longer than doing it themselves, e.g. with a monkey patch in a separate product. This is entirely counterproductive. First of all, by keeping the fix out of its proper location, you create an additional maintenance burden. Secondly, you're likely to find yourself fixing the same thing again on your next project, especially if Plone has moved on and your patch is no longer 100% valid. Thirdly, fixing a problem with an override is almost almost more complex than fixing the original code. Finally, you are keeping others from benefitting from your changes, so you get none of the network effect benefits.
Fixing something properly is easy. Modern Plone projects are managed with Buildout, which in turn manages a set of Python eggs that make up the Plone distribution. All you have to do is check out a "develop egg" from Subversion for the package you need to fix. If you use the mr.developer extension, it's even easier, you can just add the svn path and run "bin/develop co plone.whatever". You make your fix there, and then commit (try to make sure the tests pass; you may also be asked to merge your fix to trunk if you're on a maintenance branch, but that is easy in almost all cases). Until there's a release, you'll need to keep running that egg from Subversion, but this is little different from running your own, unreleased/untested code.
Now, some people are nervous about running Plone packages form subversion. Luckily, the actively maintained Plone versions see a release every 2-4 weeks if there are changes to release. And since Plone is now just a set of eggs, you don't need to wait for a full Plone release. Ask the maintainer of the package you had to fix for a new release, and you can start using it right away with a single line in your [versions] block in your buildout.
In an case, the burden of fixing something "out of context" and managing that fix "forever" in a separate place is always going to be higher than doing it properly. You're also much more likely to get feedback and help from others if you contribute, so you'll know much sooner whether your fix is "right".
This one falls into three categories: Some people hate paperwork and can't be bothered to sign a legal document; some people are uncomfortable with the terms of the agreement; and some people work in organisations who are too paranoid to have their employees sign an open source contributor agreement.
There's little excuse for the first category: Plone's contributor agreement is a two page form written in pretty clear language. I think it took me five minutes to read it, sign it, and send it off.
If you are in control of your business, you also have little to worry about. So many other businesses have signed up that you are unlikely to have a unique problem. The contributor agreement is there solely to protect you and others like you. It is not a legal document designed to take away your rights. Anyone who's been involved in an open source project where intellectual property issues became a real, rather than a hypothetical, issue knows that this is a murky world that can lead to a lot of confusion. You should be thankful that people have invested a lot of time and pro-bono work in helping the Plone Foundation set up a constructive agreement.
If you are being told you can't sign, at least try to make the case. Ask others how they've overcome such obstacles. Most companies have ways and means if you only try.
Finally, although Plone's contributor agreement covers the bulk of code in the Plone release, it is not necessary to sign one to commit code to the Collective subversion repository, or to contribute in other ways.
In economics, this is known as the "free rider" problem. If we all had to pay the police directly for fighting crime, chances are an insufficient number of people would pay, since crimes prevented in your neighbourhood benefit you equally whether you paid the policemen who prevented them yourself, or only your neighbours did.
Some projects have attempted to address this in the same way that governments do when they tax citizens and use the revenue to fund public services (and duck houses): one company or organisation tightly controls development and charges some or all users of the software a fee to use the software or become certified solution providers. Some of that revenue is then ploughed into development.
Plone doesn't follow this model, and never will. You should be thankful. You have an opportunity to participate in a truly democratic process, to be part of a mature, open and friendly community, to have your voice heard and directly influence the future of the platform. You can bet your business on a technology, safe in the knowledge that many others have before you, and no-one can decide they want to squeeze you out of the market by denying your rights to the software. This is true open source. It's pretty amazing. You owe it to yourself to be part of it.
]]>Over the past few days, I've finally taken the time to learn a bit more about Distributed Version Control Systems (DVCSs). In particular, I've read both The Mercurial Book and Pro Git, as well as a few blogs and tutorials.
Most of the time, I work with Subversion, both in development projects at work, and when I contribute on Plone. Subversion is a VCS, but not a DVCS. There is a single, centralised repository, from which everyone checks out a local working copy of a given project. Subversion tracks any number of files, co-ordinating updates by multiple authors, and allows you to roll back to previous versions should you need to. It deals with merges, branches and conflicts, too, although not in quite as sophisticated a way as a DVCS.
DVCSs are different. Typically, there is no central repository. Everyone has a full copy of the repository, including the full revision history. Most commands are local, and changes need to be explicitly pushed to a remote server. Branches are a lot cheaper, and merges are easier. A DVCS supports a number of workflows (which can be mixed on an ad-hoc basis). You can have a single maintainer of the canonical version of a given project "pulling" from others' published repositories, or a model whereby people submit patches exclusively by email. There doesn't even need to be a canonical version at all, just a number of people with their own sandboxes, sharing code in a completely decentralised manner.
Mercurial and Git are both very powerful systems. Of the two, I prefer Mercurial, mainly because it seems a little easier to work with. Git is probably a bit more powerful (I like the "stash" feature in particular), but whilst the basic commands are relatively straightforward, I found myself worried that I'd need to read the documentation many more times before it became comfortable. Some features, like submodules, seem downright confusing, and there are some niggling inconsistencies in the way the command line interface works. Of course, it helps me that the Mercurial commands closely mimic those of Subversion.
But would I switch to them wholesale? Perhaps, but not yet. The DVCS model, and the enhanced support for branching and merging, is certainly attractive. However, I worry that the fundamental DVCS concepts are just a little too complicated. I normally have to teach version control basics (with Subversion) to the people who join my project teams. It's not always easy for people to grasp, and harder still to get into good working habits. Subversion at least simplifies the process, in that it can be explained by the analogy of a filesystem with history. There is very little "magic" going on. For example, a "tag" is just a copy of the code in the "/tags" directory, nothing more, nothing less. You can understand how you'd do it all manually if you had to.
By contrast, I think it would be difficult to explain a DVCS without reference to the underlying data model. Both of the books I read made heavy use of diagrams and examples to illustrate even pretty basic concepts, and felt it necessary to explain the repository data structures in some detail. You need to build a mental model that includes a commit history with branches and merges. You need to understand concepts like "heads", "tips" and "parents". The history is not even static - it can be changed, for example by "rebasing". If you don't have a degree in computer science or at least a good understanding of with such concepts as abstract data structures (e.g. linked lists) and pointers, as well as familiarity with the UNIX tool chain for managing diffs and patches, I think you're going to struggle to pick up a DVCS in an hour or two (the time I normally have to get people started). And if you don't understand the DVCS you're using well enough, I fear that the experience could become very frustrating. I don't get the feeling that either system is particularly conducive to trial-and-error. As soon as people start asking, "what the heck happened to my file?", I think you've lost.
This is a classic "power vs. simplicity" tradeoff. The very concepts of decentralised version control, branching, and merging are complex. There are a lot of edge cases. There are a lot of real-world use cases that require sophisticated solutions, especially when you start getting into projects that have a large number of contributors or a high volume of contributions.
The good news is that even if you're working with a project that uses Subversion for its repository, you can use both Git (via git-svn) and Mercurial (via hg-subversion) as a "super-client" for Subversion. You can have local commits and use the various DVCS tools, and then "push" your changes up to the Subversion server, which won't know the difference between this and any other client. I intend to start trying that out with Mercurial and hg-subversion shortly.
The next time I start a new/non-Plone project, I may also try to use Mercurial (or Git) only. And if I do, I'll almost certainly use BitBucket to do it. BitBucket (and its Git equivalent, GitHub) is a free (up to a point, but you can buy plans with more data allowance) service that enables anyone to set up Mercurial repositories for any number of projects.
The interesting thing, though, is that BitBucket and GitHib are a little bit like Facebook for programmers. Whereas traditional "project hosting" services such as Google Code (which supports both Mercurial and Subversion) is centred around the project, BitBucket and GitHub are centred around the people. That is, they both host repositories on URLs like http://<service>/<person>/<project>. There is no single location for a given project. Instead, you are encouraged to create your own fork of someone else's repository (there's even a button on the web GUI) if you want to work on that project, and then either request access to push your work back yourself, or ask the maintainer to pull it into their working copy.
I don't think this is the correct model for all projects. I suspect it is best suited for small, ad-hoc projects, personal projects, or projects that are not yet off the ground. In fact, I think this model could be actively damaging for a large, co-ordinated project such as Plone. Of course, you don't have to use this completely decentralised model, so that is not an argument against Git or Mercurial in itself.
I would say, though, to those working on Plone projects: please continue to use the shared infrastructure we already have, until such time that we are all ready to move to something different, together. Although it is not quite the same, working with a Subversion repository via Mercurial or Git is pretty powerful and gives you most of the benefits of having a fully DVCS-managed environment. By putting things in the places where other people are most likely to find them, and allowing those not ready to make the switch to work with the tool chain with which they are already familiar, you will encourage contributions and aid the cohesiveness of the community. When it comes to "core" Plone code, there is also a legal aspect to consider, in that the Plone Foundation owns code in the Plone repository, and that this code is protected by the Plone Contributor Agreement. That is an important factor in protecting Plone's intellectual property, and not something to be taken lightly.
Do I think Plone will eventually move to using Mercurial or Git? It seems a bit unlikely, given the amount of code we have in our Subversion repositories. I'm also worried that some of our integrators would find the Mercurial or Git learning curve a little bit too steep. We need to keep the barriers to contribution as low as possible. However, there clearly would be benefits too, so we should have a debate and see what the community wants. It seems a lot of other projects are moving to a DVCS, which is a sign that we should look into it, too. And in the meantime, I would encourage everyone to try out Mercurial or Git, or both.
]]>If you read this blog, you are probably a technologist, and quite possibly a software developer. If so, ask yourself this: When was the last time you learned something truly new? Not just another aspect of what you already know best (say, a Python developer reading up on a module in the standard library he hasn't used before), but something that puts you out of your comfort zone and stretches your ability to learn (like that same developer tackling Erlang).
If the answer is more than a month ago, stop reading and think about your prioritises for a moment. And think about how you came to be a technology professional in the first place.
Technology is all about learning new things. I remember when I first started playing with my mother's IBM 386 PC with MS-DOS 6 (yes, I know I'm young). I was terrified and excited. When I learned about DOS batch files, I thought I could build just about anything. With GOTO. And SHIFT. When I borrowed my friend's father's Visual Basic 6 manuals, I read them cover to cover and tried to build a better text editor. Oh, how I have learned the error of my ways, but that was an exciting time. It was how I got into technology, and became interested in programming. Half a dozen programming languages later, it's still one of my most important hobbies, and the cornerstone of my professional career. If I hadn't had the desire to learn new and unfamiliar things, I would probably have been an accountant. Yipes!
A colleague and friend once told me: if you're in technology, you're an absolute idiot if you're not spending at least 30 minutes every day learning something new. Your knowledge today is most likely obsolete in a few years' time. But more importantly, the aptitude for learning and passion for acquiring new skills is probably your most valuable professional attribute. Without it, you're just someone who learned a trade and now applies it day in, day out. There's nothing wrong with this, but it certainly limits your ability to grow professionally.
I think we all know this. I think we all remember the excitement of learning new things. And yet, programmers can be some of the most stubbornly set-in-their-ways people you'll ever meet. This goes beyond the pointless "religious" debates (vi vs. emacs; object oriented vs. procedural). It even goes beyond programming languages and paradigms. It goes right down to the tools we use to write our software.
I was prompted to write this by a discussion on the Deliverance mailing list about moving the code to GitHub or BitBucket, two "social coding" portals that use Git and Mercurial for version control respectively. The aim was to make it easier for people outside our community to contribute, and to present a bit more an outward face. Some developers also preferred these tools over Subversion, which is what we use now.
In response, one of the contributors basically said he wouldn't be interested in contributing if it moved away from Subversion. I don't know his reasoning, but for a moment I found myself in agreement. In fact, I've berated people who move code away from a Subversion repository to a Git or Mercurial one before (although not only for the reason I'm about to describe - more on that in a future blog post). I'm extremely comfortable with Subversion. It fits my brain and I use it on a daily basis. Having to learn something else sounded onerous. My urge was to resist it, so as to keep the project well within my comfort zone.
But then I thought: how utterly stupid a reason. I learned Subversion at one point too. Why should this be any different? Maybe Git or Mercurial really would be better? I didn't know them well, so I had few factual arguments. My resistance was instinctive: For the love of God, don't make me learn anything new!
The same applies everywhere. People freak out when you present them with a new framework or tool. I've spent weeks - if not months - building frameworks that I believe in, only to have people shoot it down because it's "one more thing to learn". (For sure, an excessive learning curve can be a big problem, but many of these "new" things are in fact about making old things easier for new people to learn). Programmers build up irrational hatreds for all sorts of things: XSLT (XDV, anyone?). XML. Java. In a grown up community like Plone's, we don't see too much of this (though perhaps it is just displaced by apathy), but peruse Slashdot or Reddit one evening, and you will find hordes of kids born ten years after K&R was written proclaiming the virtues of C and refusing to even look at C++, let alone Java. They learned that from someone.
This problem is of course not unique to software development, but I think it is a little hypocritical at times. We claim to be on the cutting edge of technology, and yet we often engage in great amount of navel-gazing. It leads to NIH (Not Invented Here) Syndrome. Rather than learn about what others have done and build on it, we prefer to build our own, with our own tools, designed in our own way, exactly the way we've always done it. Get off my lawn, you young whippersnappers!
What can be done about this? The advice about taking half an hour of each day to learn something new is a good one. After the Deliverance debate, I decided I was long overdue to learn about DVCS and found the excellent Mercurial Book, which I read online. I'm now reading Pro Git to compare. Learning new things is fun, and makes you a better developer. The ability to draw insights from a number of disparate sources is probably one of the main things your clients pay you for (or your boss values you for), whether you're a consultant, a developer or some other kind of professional. So keep growing your list of sources.
But we must also recognise some of the reasons people become insular. There are so many technologies and tools out there, one can't hope to learn about them all. "Choice overload" instinctively makes people retract to the familiar. Once we got jobs and families, we stopped having time to just surf the internet looking for "cool new things". It can also be hard sometimes to pick out the things truly worth investing time in.
Here, you can help your friends. When you find something worth learning about, Tweet liberally and blog about it. Pick out the best summary you can find, and show it to others. It is much easier to get started when someone gives you a recommendation and helps you filter out the noise. And if you've just built the next great piece of technology, you could do worse than to make it easy and fun to learn.
]]>I went on holiday for four days. When I came back and opened Thunderbird, I saw that there were 97 new emails in the zope-dev newsgroup (mailing list). Controversial debate, here we come.
This particular debate was instigated in reaction to an earlier thread by Chris McDonough, who was trying to bring some BFG API improvements back up to zope.component. I always value Chris' opinions on design and development - few people are as committed to software maintenance, programmer friendliness or high-quality documentation as he and his colleagues at Agendaless are. And even though Chris' proposal didn't quite fit in the end, he inadvertently re-kindled a debate about how we can improve the API of the Zope Component Architecture (ZCA), i.e. zope.interface and zope.component.
Recently, there's been a sense of backlash against the ZCA. It's too hard. It's too Zope-specific. It breaks expectations and has a sub-optimal API. This has sometimes spilled over into a generic mistrust of frameworks. Frameworks are too hard. They're too application-specific. Many of them are too esoteric and break expectations in one way or another. We all just want to write Python code with simple functions and simple data structures and maybe a decorator or two, dammit. None of the complexity that requires us to read documentation and learn new concepts.
Don't get me wrong. Many of these criticism are justified in part. ZCA over-use is a common affliction in some parts (and I shall certainly take my share of the blame). Over-generalisation is the enemy of discoverability. The ZCA, like all inversion-of-control frameworks, encourage separation of concerns to the point where you can't always find the implementation for something without understanding what registrations are currently in effect. Ian Bicking used the term "non-locality" of code, which is definitely something that ZCA espouses - even encourages - and this comes at a cost.
And yet. The ZCA rocks. It's an incredibly advanced and powerful way to build software. When I work in other environments (or other programming languages), I often miss it, and I sometimes end up re-inventing parts of it for my own needs. Zope, Plone, Grok, BFG and the other frameworks that use the ZCA constitute some of the largest and most complex codebases in the Python world. They are also some of the most extensible, customisable products in existence - probably in all of software. The ZCA sits at the heart of that, and has (ignoring a huge amount of legacy code that clouds the picture) facilitated a type of consistency in extensibility that is difficult to emulate.
There is no doubt also that the ZCA adds complexity. It demands that you swallow its core concepts up front (interfaces, adapters, utilities, events) and get to know them pretty well. It has a few nooks where people can do crazy things, usually doing as much harm as good.
But my take on it all is that the "framework backlash" is wrong. It seems to happen when people who've been working on high-complexity software starts building small and simple things, using elegant frameworks like BFG or Pylons. A whole application is just a couple of functions, a few dicts and lists, and maybe a page template or two. Why can't all of our code be like this? But of course it can't. When you start building software that is tens of thousands of lines of code, you need architecture, you need formalisms, you need common patterns and repeatable techniques. You need to cater for teams that change, skills that vary and codebases that get so big no one person can manage it all at once. You need to consider how your code will evolve, because the code you wrote first will be in dire need of refactoring by the time you're half way through your project.
That doesn't mean the ZCA is right for all situations. Rather, I'd argue that learning to use the ZCA effectively is a bit like learning a new programming language on top of Python. You need to "get" those concepts, just like you need to "get" functions, classes, for loops and if statements. You need to gain some experience and learn some common patterns and techniques. In short, you need to invest some time and become familiar with your tools. That investment will pay off for a big project, or a complex one. It probably won't if you're building a small, stand-alone application. And for the huge grey-zone in-between, you're going to have to make informed decisions.
For the ZCA itself, there is clearly an opportunity right now to evolve, and a degree of energy poised to make this happen. My personal wish-list would be the following:
And finally, some advice for projects, such as Plone, that use the ZCA for extensibility and inversion-of-control : Don't let the ZCA be the first API that people see, and don't assume everyone has yet reached ZCA mastery. Think of the ZCA as a building block, not the end goal.
BFG does this very well. A particular feature may be implemented using the ZCA, in order to be allow different policies to be swapped in, say. But for the casual user of the framework, there is a helper method which performs the necessary ZCA lookups and presents a domain-specific, documented API for a particular purpose. And it documents its design decisions. That's maturity for you.
We're trying to do something similar with Dexterity and the ecosystem of tools around it. The same will hopefully be true for other technologies as we move towards Plone 5 and what we hope will be a much improved integrator learning curve. Stay with us. The ZCA certainly will.
]]>In a previous life, I used the Hudson Continuous Integration server for Java projects. If you haven't heard of Hudson before, it's like Buildbot, but with a butler. It is also more user friendly and visually appealing.
If you haven't used continuous integration before, now is a good time to start. In any project, you should have a CI server set up that periodically runs your automated unit and integration tests. That way, you are notified immediately of any regressions you may have caused. This is particularly important in a team situation where you have multiple programmers working on the same code base. With CI, you can pinpoint which checkin broke the build (Hudson will tell you down to the revision, checkin message and author) and rectify it quickly.
Hudson will poll your source control system or react to external events (like you clicking a button or even an IRC message, if you have the right plugin installed), perform a build, and collect test results. It can distribute tests across several nodes, run builds in parallel, and integrates nicely with tools like Selenium and JUnit. Oh, and it can post your build status to Twitter.
Hudson is probably better known in Java circles, and has a bit of a Java slant, but the latest version is really rather nice. It's about as simple to get up and running as it could be, and it has a number of plugins, including ones that add the ability to execute Python scripts, various monitoring and reporting options, and that all-important Chuck Norris plugin:
"Chuck Norris doesn't need a debugger, he just stares down the bug until the code confesses."
And now, you can use it for your Plone project. Well, you always could, but collective.xmltestreport, a package I just released, provides a test runner that can output the types of XML test reports that Hudson expects to be able to produce pretty graphs and trending information about your tests.
It's pretty easy to set it all up. Download Hudson and start it up like so:
$ java -jar hudson.war
That will run Hudson on port 8080. See the Hudson documentation for more details on alternative ways of running the web server.
Then, log into the Hudson console and go to the "Manage Hudson" screen. Install a few plugins. I got plugins for:
The last one is particularly important, of course.
Then, create a new job. I used the "free-style" build template, specifying a Subversion URL to check out a buildout, a build trigger based on SCM polling, and an "execute shell" build step like:
cd dev # name of buildout bin/develop up # update mr.developer sources bin/buildout # re-build bin/test -x -s plone.dexterity # run tests
The build in question is the Dexterity development build. This has a [test] part which looks like this:
[test]
recipe = collective.xmltestreport
eggs =
plone.dexterity
# other eggs in this list not shown
extra-paths = ${zope2:location}/lib/python
defaults = ['--exit-with-status', '--auto-color', '--auto-progress']
The recipe is a special version of zc.recipe.testrunner, which installs the test runner from collective.xmltestreport. This is the runner that adds support for the -x command line option seen in the shell commands above.
Back in the Hudson configuration, enable "publish JUnit test results report" and set the report path to "dev/parts/test/testreports/*.xml". Here, "dev" is the name of the buildout). "parts/test" is created by buildout, and the "testreports" sub-directory is created by the collective.xmltestreport test runner when the -x command line option is given.
Now, activate a build (or wait for Hudson to do it for you). You can watch the console in real-time performing the build happen, and then explore test results and build trends afterwards. It's pretty intuitive and you can get a lot of information about when a test started failing, how long it has been failing for, and what has been happening to your code since.
There is of course a lot more to Hudson. You can set up various notifications, and create multiple builds that depend on one another. You probably also want to set up some notifications (e.g. via email) when the build breaks. We also used to have Hudson jobs for things like resetting the test database (which we would run ad-hoc) and performing a health check on a live website (which we would run every ten minutes, and then all hell would break lose when it failed).
If you need to see what Hudson is doing, look in ~/.hudson. In particular, you will find the source checkout in ~/.hudson/jobs/<name>/workspace, unless you specified an alternative workspace location (which is useful if you want two builds to work on the same checkout, though you need to be a bit careful).
]]>Update: Gave credit to Godefroid Chapelle, who started five.grok with Lennart Regebro.
For a while now, thanks to the efforts of people like Sylvain Viollon, Godefroid Chapelle, Lennart Regebro, Vincent Fretin and others, we've been able to use Grok techniques to register Zope adapters, utilities, event subscribers, views, viewlets, and so on in Zope 2 and Plone. This helps lower the Zope learning curve by largely taking ZCML out of the picture for component registration, allowing you to keep the code that makes up a component and the "wiring" for that component to be usable in the Zope Component Architecture (ZCA), together in one place.
Dexterity builds on this with additional grokkers, and the Dexterity documentation recommends using Grokkers for more basic Zope component registrations. You don't need to be using Dexterity to benefit from five.grok, however.
Today, I've published a detailed manual on using five.grok in Zope 2 and Plone. This also explains the fundamental Zope Component Architecture concepts with examples and analogies, so if you are a little rusty on your ZCA ABCs, and you haven't got Chapter 9 handy, you may find it a good read.
I've also openly invited the Grok project to plagarise this manual for their own purposes, should they find it useful.
You can find the manual here.
]]>Over the past year or two, the transition from "products" to "eggs" has taken hold. Almost no new third party products (and very few new releases of old products) are released as product tarballs, and it seems almost everyone is using buildout and eggs to distribute their code. This is good, and helps bring Plone's packaging story in line with that of other frameworks.
With eggs come namespace packages: the ability to organise code into namespaces. We see this in Plone itself, with namespaces like plone.* (packages like plone.memoize or plone.browserlayer), and nested namespaces like plone.app.* (plone.app.portlets, plone.app.contentrules) or plone.portlet.* (plone.portlet.collection, plone.portlet.static). A number of other "shared" namespaces have also popped up, such as collective.*, five.*, z3c.*, and so on, as well as namespaces used by specific organisations (jarn.*, slc.*).
Unfortunately, the rules for the use of namespaces are a bit fuzzy, and sometimes cause confusion. It's also difficult (and inadvisable) to change packages names after a package has been released and there are uses "in the wild", so getting the name right the first time is important. I'll try to articulate some of the unwritten rules here.
First of all, a small, but important point: For Plone packages, we tend to have a one-to-one mapping between package names and egg names, but this is by no means required.
For the avoidance of doubt, the egg name is the thing that is registered on PyPI and listed in the buildout.cfg eggs option or a setup.py install_requires line. An egg is an archive of Python code and other resources (like documentation or images). The Python modules included in the egg may or may not be organised into packages, and those packages may or may not be namespace packages.
A namespace package is like any other Python package (i.e. a directory with an __init__.py file in it), except that when a package is declared as a namespace package (which involves a couple of lines of boilerplate in the __init__.py), it is possible to have multiple eggs provide modules and sub-packages within that package.
As an example, Plone depends on packages such as plone.theme and plone.browserlayer. Both of these put sub-packages into the the plone namespace package. If plone was not a top level namespace package, the two eggs would clash (the package(s) from one egg would effectively hide all others using the same top level package name).
The convention of naming the egg after the top level package is useful because it makes it easier to find code: if you are looking for the code behind the module plone.theme.interfaces, you know to look in the egg called plone.theme. It's also one less thing to worry about naming. However, this is not enforced. If you installed the Paste egg, for example, you will get a number of packages in the paste.* namespace, all distributed in a single egg. Other packages, like PasteScript, provide further packages in the same top level namespace.
There are no hard and fast rules for package naming, but there are some rules of thumb based on accepted practices from Plone and other projects.
There are several namespace packages in use today. Below are some of the most commonly used "community" ones and their purpose.
Today sees the second release of Dexterity, the less-boilerplate, more-fun, better-documented content type framework for CMF and Plone.
This release includes full support for rich text fields with markup transformation and tag stripping, full support for WebDAV, External Editor and other file representations of content, as well as a number of bug fixes and minor improvements.
We hope you'll experiment with Dexterity in your own projects and give feedback and contributions to help close the last few remaining feature gaps with Archetypes and make the platform even better.
If you haven't already, take a look at the FAQ to learn more about Dexterity, the installation guide to get started, and the detailed manuals to see how easy it is to build content types with Dexterity.
]]>