An über-buildout for a production Plone server

This weekend, I decided to finally learn a bit more about nginx. I actually really like it. It's much easier to understand and configure than Apache is, and is reportedly also a lot faster, at least at high loads (I would love to have some equivalent of mod_svn, though).

Encouraged by this, I set about updating this server to use nginx. In doing so, I discovered the gocept.nginx recipe, which can configure nginx in a buildout. I decided to create a standalone über-buildout that would contain all the services I need on the server. You can find it at the end of this post.

Update #2 (supersedes #1): I've simplified the ZEO client setup a little using plone.recipe.zope2cluster. I think it could be made a bit more elegant with zc.recipe.macro, but that doesn't quite work as advertised when creating parts on the fly. I also removed buildout.eggtractor in favour of explicitly stating develop eggs (it's a production server, after all), and set up a specific versions block to pin zc.recipe.testrunner and zope.testing, which was causing some problems in some configurations.

The buildout sets up the following things:

• Plone 3.2
• Develop eggs go in src/, other eggs are listed in a separate section.
  
• A ZEO storage server.
• Two ZEO clients (it should be easy to add more).
• A load balancer for these that also takes care of virtual hosting. This uses a standalone nginx instance. I could've used pound or HAProxy or something similar, of course, but nginx was nice and easy to set up.
• A Varnish caching proxy.
• An nginx instance to go in front, with two virtual hosts: one for Plone, and one for a simple file server.
• A supervisor to control it all.

All you need is the buildout.cfg and the boilerplate bootstrap.py to get it going.

The buildout.cfg file is quite long, but I've tried to keep it well structured and commented. Furthermore, common settings such as hostnames, ports, usernames and passwords are kept in separate sections near the top.

On my server, I now have a small file called production.cfg that looks like this:

[buildout]
extends = buildout.cfg

[instance-settings]
user = zope-admin:password

[supervisor-settings]
user = supervisor-admin
password = password 

[plone-sites]
main = plone-instance-id

[hosts]
main = www.server.com
fileserver = files.server.com

[ports]
main = 80

[users]
main = www
cache = www
balancer = www
zope = www
supervisor = www

With this, I do:

$ python bootstrap.py # once
$ ./bin/buildout -c production.cfg
$ ./bin/supervisord

Actually, I lie slightly... I also had to do some chowning on the parts/, var/ and files/ directories so that they were owned by user www.

I can monitor all the servers using the very nice supervisor web GUI on 9001, and stop/start services there. To shut everything down in one go, I'd do:

$ ./bin/supervisorctl shutdown

I have a system startup script that basically does this.

The buildout also creates a logrotate file (parts/logrotate.conf) that can be used with the logrotate service by symlinking it to e.g. /etc/logrotate.d. This rotates the logs for all the services weekly (well, Varnish doesn't have a logfile, and supervisor does its own log rotation).

There are a few things that could be improved still:

• I'd love to have a way to add more ZEO clients with a configuration parameter, but this would require some more clever recipes and may make everything a bit too obtuse.
• Having all the configuration in buildout.cfg is nice. On the other hand, it means that changing the configuration means re-running buildout. This only takes a second, since the build-intensive operations are in separate parts that are not re-evaluated each time, but it's easy to forget. And of course, the recipes only go so far and may not allow you to edit all configuration options.
  
• I could have used Varnish's load balancing capabilities and skipped the separate load balancer. I'm not sure if this would give better performance. In the event, I wanted to experiment with a separate configuration. Also plone.recipe.varnish does not support load balancing at the moment, and so I would've had to use a custom file.
• Depending on whether the buildout is run as the effective user, you may need to do some chown'ing to get the right permissions on things. It'd probably be possible for buildout to do that, but I'm not sure it's a good idea to start messing with file permissions automatically like that. Perhaps for var/, parts/ and files/, though.

Here is the buildout. Feel free to use it if it's useful. I'd be grateful if anyone wants to turn this into a proper how-to on plone.org with a bit more background to explain what these pieces all do and how to use them.

# Deployment buildout
# ===================
# 
#  This buildout configures a number of servers:
#
#   - 'main', an nginx web server that may run on port 80
#   - 'cache', a varnish cache that is configured to proxy a Plone site
#   - 'balancer', an nginx instance used as a load balancer for ZEO clients
#   - 'zeoserver', a ZEO server
#   - 'instance1' and 'instance2', two ZEO clients
# 
# Log rotation is configured for all of these, except Varnish, which does not
# write a log file. The log rotation configuration file is in 
# parts/logrotate.conf and this needs to be symlinked into the main logrotate
# configuration.
# 
# In the 'main' nginx configuration, virtual hosting will be enabled for a
# single Plone site and a file server. This can be adjusted as required.
#
# Finally, a supervisor instance is set up. To start it all up, do:
#
#  $ ./bin/supervisord
#
# Go to http://localhost:9001 to see supervisor status.
#
# The configuration is fully contained within this buildout. Hostnames, ports
# and common options can be changed in the buildout sections at the top of
# of this file. More detailed configuration is contained in the relevant
# parts, further down the file.

[buildout]
newest = false
parts =
    zope2
    zeoserver
    
    instance1
    instance2
    
    nginx-build
    varnish-build
    
    fileserver-directory
    
    main
    cache
    balancer
    
    logrotate.conf
    
    supervisor
    zopepy
    omelette

develop = 
    src/Products.NuPlone

# Plone version - 3.2

find-links = 
extends = http://dist.plone.org/release/3.2/versions.cfg
versions = versions

# Version pins, above and beyond what the Plone 3.2 versions block says

[versions]
zc.recipe.testrunner = 1.1.0
zope.testing = 3.6.0

# Download urls - encapsulate Zope, Varnish and nginx versions

[downloads]
zope = http://www.zope.org/Products/Zope/2.10.6/Zope-2.10.6-final.tgz
varnish = http://downloads.sourceforge.net/varnish/varnish-2.0.2.tar.gz
nginx = http://sysoev.ru/nginx/nginx-0.7.30.tar.gz


# Default settings for ZEO clients. http-address is set with zc.recipe.macro

[instance-settings]
eggs = 
    Plone
    Products.CacheSetup
    Products.NuPlone
zcml =
products = 
user = admin:admin
zodb-cache-size = 5000
zeo-client-cache-size = 300MB
debug-mode = off
zope2-location = ${zope2:location}
zeo-client = true
zeo-address = ${zeoserver:zeo-address}
effective-user = ${users:zope}
http-address = $${:http-address}

# Default settings for supervisor

[supervisor-settings]
user = admin
password = admin

# Plone sites - used in VirtualHost configuration

[plone-sites]
main = blog

# Hostnames for various servers. 'main' is the public hostname.

[hosts]
main = www.example.com
fileserver = dist.example.com
cache = 127.0.0.1
supervisor = 127.0.0.1
balancer = 127.0.0.1
instance1 = 127.0.0.1
instance2 = 127.0.0.1

# Ports for various servers. 'main' is for the public hostname

[ports]
main = 8000
cache = 8101
balancer = 8201
zeo-server = 8301
instance1 = 8401
instance2 = 8402
supervisor = 9001

# OS users to drop to for various processes

[users]
main = optilude
cache = optilude
balancer = optilude
zope = optilude
supervisor = optilude

##############################################################################  
# 1. Zope - build, ZEO server, ZEO clients
##############################################################################

[zope2]
recipe = plone.recipe.zope2install
fake-zope-eggs = true
additional-fake-eggs = 
    ZConfig
    pytz
skip-fake-eggs =
    zope.testing
url = ${downloads:zope}

[zeoserver]
recipe = plone.recipe.zope2zeoserver
zope2-location = ${zope2:location}
zeo-address = ${ports:zeo-server}
effective-user = ${users:zope}

# Generate instances using zc.recipe.macro - we have one macro that
# contains the shared settings, 

[instance1]
recipe = collective.recipe.zope2cluster
instance-clone = instance-settings
http-address = ${hosts:instance1}:${ports:instance1}

[instance2]
recipe = collective.recipe.zope2cluster
instance-clone = instance-settings
http-address = ${hosts:instance2}:${ports:instance2}

##############################################################################  
# 2. Build nginx and varnish for later configuration
##############################################################################

[nginx-build]
recipe = zc.recipe.cmmi
url = ${downloads:nginx}

[varnish-build]
recipe = zc.recipe.cmmi
url = ${downloads:varnish}

##############################################################################  
# 3. Create directories
##############################################################################

[fileserver-directory]
recipe = ore.recipe.fs:mkdir
path = ${buildout:directory}/files

##############################################################################  
# 4. Configure front-end web server
##############################################################################

[main]
recipe = gocept.nginx
nginx = nginx-build
configuration = 
    user ${users:main};
    error_log ${buildout:directory}/var/log/main-error.log warn;
    worker_processes 1;
    daemon off; 
    events {
        worker_connections 1024;
    }
    http {
        # Proxy to Varnish cache
        upstream cache {
            server ${hosts:cache}:${ports:cache};
        }
        
        # Main server goes upstream to Varnish
        server {
            listen *:${ports:main};
            server_name ${hosts:main};
            access_log ${buildout:directory}/var/log/main-plone-access.log;
            location / {
                proxy_pass http://cache;
            }
        }
        
        # File server
        server {
            listen *:${ports:main};
            server_name ${hosts:fileserver};
            access_log ${buildout:directory}/var/log/main-dist-access.log;
            autoindex on;
            root ${buildout:directory}/files;
        }
        
    }

##############################################################################  
# 5. Configure Varnish for Plone
##############################################################################

[cache]
recipe = plone.recipe.varnish
daemon = ${buildout:directory}/parts/varnish-build/sbin/varnishd
bind = ${hosts:cache}:${ports:cache}
backends = ${hosts:balancer}:${ports:balancer}
cache-size = 1G
user = ${users:cache}
mode = foreground

##############################################################################  
# 6. Configure load balancer
##############################################################################

[balancer]
recipe = gocept.nginx
nginx = nginx-build
configuration = 
    user ${users:balancer};
    error_log ${buildout:directory}/var/log/balancer-error.log warn;
    worker_processes 1;
    daemon off; 
    events {
        worker_connections 1024;
    }
    http {
        upstream zope {
            server ${hosts:instance1}:${ports:instance1} max_fails=3 fail_timeout=30s;
            server ${hosts:instance2}:${ports:instance2} max_fails=3 fail_timeout=30s;
        }
        server {
            listen ${hosts:balancer}:${ports:balancer};
            server_name ${hosts:main};
            access_log off;
            rewrite ^/(.*)  /VirtualHostBase/http/${hosts:main}:${ports:main}/${plone-sites:main}/VirtualHostRoot/$1 last;
            location / {
                proxy_pass http://zope;
            }
        }
    }
    
##############################################################################  
# 7. Set up supervisor to run it all
##############################################################################

[supervisor]
recipe = collective.recipe.supervisor
port = ${ports:supervisor}
user = ${supervisor-settings:user}
password = ${supervisor-settings:password}
serverurl = http://${hosts:supervisor}:${ports:supervisor}
programs =
    10 zeo       ${zeoserver:location}/bin/runzeo                  true ${users:zope}
    20 instance1 ${buildout:directory}/parts/instance1/bin/runzope true ${users:zope}
    20 instance2 ${buildout:directory}/parts/instance2/bin/runzope true ${users:zope}
    30 cache     ${buildout:directory}/bin/cache                   true ${users:cache}
    40 balancer  ${nginx-build:location}/sbin/nginx [-c ${balancer:run-directory}/balancer.conf] true ${users:balancer}
    50 main      ${nginx-build:location}/sbin/nginx [-c ${main:run-directory}/main.conf]         true

##############################################################################  
# 8. Log rotation
##############################################################################

[logrotate.conf]
recipe = zc.recipe.deployment:configuration
text = 
    rotate 4
    weekly
    create
    compress
    delaycompress

    ${buildout:directory}/var/log/instance1*.log {
        sharedscripts
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/instance1.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/instance2*.log {
        sharedscripts
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/instance2.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/zeoserver.log {
        postrotate
            /bin/kill -USR2 $(cat ${buildout:directory}/var/zeoserver.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/main*.log {
        sharedscripts
        postrotate
            /bin/kill -USR1 $(cat ${main:run-directory}/main.pid)
        endscript
    }
    
    ${buildout:directory}/var/log/balancer*.log {
        sharedscripts
        postrotate
            /bin/kill -USR1 $(cat ${balancer:run-directory}/balancer.pid)
        endscript
    }
    
##############################################################################  
# 9. Debugging tools - preconfigured python interpreter and omelette
##############################################################################

[zopepy]
recipe = zc.recipe.egg
eggs = ${instance-settings:eggs}
interpreter = zopepy
extra-paths = ${zope2:location}/lib/python
scripts = zopepy

[omelette]
recipe = collective.recipe.omelette
eggs = ${instance-settings:eggs}
products = ${instance-settings:products}
packages = ${zope2:location}/lib/python ./