md.recipe.celery 1.1
Buildout recipe for installing celery for use with Zope's ZCA and configuring it using an ini file.
License: BSD

Introduction
============

Buildout recipe for installing Celery for use with Zope's ZCA and configuring
it using an ini file.

This recipe was originally based on collective.recipe.celery,
but differs greatly so that it now has become its own package. This recipe
defines a custom loader that can read celery configuration from an ini file.
This allows you to generate the configuration by using a template for
example, using collective.recipe.template. The custom loader also bootstraps
the ZCA automatically when a worker initializes.

You can use it in a part like this::

    [celery]
    recipe = md.recipe.celery
    eggs =
      ${buildout:eggs}
      celery[redis]
    celery_conf = etc/celery.ini
    logging_conf = etc/logging.ini
    zope_conf = etc/zope.conf


Supported options
=================

General options
---------------

eggs
    A list of additional eggs you want to make available to Celery. Use this to
    add additional dependencies such as ``kombu-sqlalchemy`` or the module(s)
    containing your task definitions.

celery_conf
    Location of the Celery configuration .ini file.

logging_conf
    Location of the file containing the python logging setup

zope_conf
    Location of the zope configuration file.


Configuration
=============

Example of an ini file that could be used by this recipe::

    [celery]
    BROKER_URL = sqlite:///celery_broker.db
    CELERY_IMPORTS = myapp.tasks,otherapp.tasks
    CELERY_RESULT_BACKEND = sqlite:///celery_results.db
    CELERY_RESULT_SERIALIZER = json

    [celerybeat:task1]
    # Execute every hour
    task = myapp.tasks.Task1
    type = crontab
    schedule = {"minute": 0}

    [celerybeat:task2]
    # Execute every 30 seconds
    task = myapp.tasks.Task2
    type = timedelta
    schedule = {"seconds": 30}

    [celerybeat:task3]
    # Execute at midnight
    task = otherapp.tasks.Task3
    type = crontab
    schedule = {"hour": 0, "minute": 0}

The [celery] section defines all the celery options. Every [celerybeat]
section defines an individual task.

Example usage
=============

Create a celery ini file holding your celery configuration::

    >>> write(sample_buildout, 'celery.ini',
    ... """
    ... [celery]
    ... BROKER_URL = sqlite:///celery_broker.db
    ... CELERY_IMPORTS = myapp.tasks,otherapp.tasks
    ... CELERY_RESULT_BACKEND = sqlite:///celery_results.db
    ... CELERY_RESULT_SERIALIZER = json
    ...
    ... [celerybeat:task1]
    ... # Execute every hour
    ... task = myapp.tasks.Task1
    ... type = crontab
    ... schedule = {"minute": 0}
    ...
    ... [celerybeat:task2]
    ... # Execute every 30 seconds
    ... task = myapp.tasks.Task2
    ... type = timedelta
    ... schedule = {"seconds": 30}
    ...
    ... [celerybeat:task3]
    ... # Execute at midnight
    ... task = otherapp.tasks.Task3
    ... type = crontab
    ... schedule = {"hour": 0, "minute": 0}""")

This file could be generated by a template such as collective.recipe.template.

Next create a logging config::

    >>> write(sample_buildout, 'logging.ini',
    ... """
    ... [loggers]
    ... keys = root
    ...
    ... [handlers]
    ... keys = console
    ...
    ... [formatters]
    ... keys = generic
    ...
    ... [logger_root]
    ... level = INFO
    ... handlers = console
    ...
    ... [handler_console]
    ... class = StreamHandler
    ... args = (sys.stderr,)
    ... level = NOTSET
    ... formatter = generic
    ...
    ... [formatter_generic]
    ... format = %(asctime)s %(levelname)s [%(name)s] %(message)s""")

Then we'll create a buildout that uses the recipe::

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = celery
    ... index = %(server)s/index
    ... find-links = %(server)s
    ... 
    ... [celery]
    ... recipe = md.recipe.celery
    ... celery_conf = celery.ini
    ... logging_conf = logging.ini
    ... zope_conf = zope.conf
    ... """% dict(
    ...     server=link_server))

Running the buildout gives us::

    >>> print system(buildout)
    Installing celery.
    celery: Creating directory /sample-buildout/parts/celery.
    celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
    Getting distribution for 'celery'.
    Got celery 2.3.1.
    Generated script '/sample-buildout/bin/celeryctl'.
    Generated script '/sample-buildout/bin/celeryd'.
    <BLANKLINE>

Check that we have the celery scripts::

    >>> ls(sample_buildout, 'bin')
    -  buildout
    -  celeryctl
    -  celeryd

Check that we got the celery loaders::

    >>> ls(sample_buildout, 'parts', 'celery')
    - loader.py
    - zopeloader.py

If we run the celeryd script, it prints out the config data::

    >>> print(system(join(sample_buildout, 'bin', 'celeryctl')))
    [('BROKER_URL', 'sqlite:///celery_broker.db'),
     ('CELERYBEAT_SCHEDULE',
      {'task1': {'schedule': <crontab: 0 * * * * (m/h/d/dM/MY)>,
                 'task': 'myapp.tasks.Task1'},
       'task2': {'schedule': datetime.timedelta(0, 30),
                 'task': 'myapp.tasks.Task2'},
       'task3': {'schedule': <crontab: 0 0 * * * (m/h/d/dM/MY)>,
                 'task': 'otherapp.tasks.Task3'}}),
     ('CELERYD_HIJACK_ROOT_LOGGER', False),
     ('CELERY_IMPORTS', ['myapp.tasks', 'otherapp.tasks']),
     ('CELERY_RESULT_BACKEND', 'sqlite:///celery_results.db'),
     ('CELERY_RESULT_SERIALIZER', 'json')]
    <BLANKLINE>

We can include additional eggs using the eggs option::

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = celery
    ... index = %(server)s/index
    ... find-links = %(server)s
    ...
    ... [celery]
    ... recipe = md.recipe.celery
    ... eggs =
    ...     other
    ... """% dict(server=link_server))

    >>> print system(buildout),
    Uninstalling celery.
    Installing celery.
    celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
    Getting distribution for 'other'.
    Got other 1.0.
    Generated script '/sample-buildout/bin/celeryctl'.
    Generated script '/sample-buildout/bin/celeryd'.
    Generated script '/sample-buildout/bin/distutilsscript'.

The recipe should handle updates as well, trigger this by pinning celery to
another version and checking one of the outputted scripts.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = celery
    ... index = %(server)s/index
    ... find-links = %(server)s
    ...
    ... [versions]
    ... celery = 2.3.0
    ...
    ... [celery]
    ... recipe = md.recipe.celery
    ... eggs =
    ...     other
    ... """% dict(server=link_server))

    >>> print system(buildout),
    Updating celery.
    celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
    Getting distribution for 'celery==2.3.0'.
    Got celery 2.3.0.
    Generated script '/sample-buildout/bin/celeryctl'.
    Generated script '/sample-buildout/bin/celeryd'.
    Generated script '/sample-buildout/bin/distutilsscript'.

    >>> cat(sample_buildout, 'bin', 'celeryctl')
    #!/Users/rwendt/Projects/python/bin/python
    <BLANKLINE>
    import sys
    sys.path[0:0] = [
      '/sample-buildout/eggs/celery-2.3.0-py2.7.egg',
      '/sample-buildout/eggs/other-1.0-py2.7.egg',
      '/sample-buildout/parts/celery',
      ]
    ***

We can control which scripts are generated using the scripts option.
If no value is given, then script generation is disabled::

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = celery
    ... index = %(server)s/index
    ... find-links = %(server)s
    ...
    ... [celery]
    ... recipe = md.recipe.celery
    ... scripts =
    ... """% dict(server=link_server))

    >>> print system(buildout),
    Uninstalling celery.
    Installing celery.
    celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.

    >>> ls(sample_buildout, 'bin')
    -  buildout

Let's create the celeryd script only::

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = celery
    ... index = %(server)s/index
    ... find-links = %(server)s
    ...
    ... [celery]
    ... recipe = md.recipe.celery
    ... scripts =
    ...     celeryd
    ... """% dict(server=link_server))

    >>> print system(buildout),
    Uninstalling celery.
    Installing celery.
    celery: Generated loader file /sample-buildout/parts/celery/zopeloader.py.
    Generated script '/sample-buildout/bin/celeryd'.

    >>> ls(sample_buildout, 'bin')
    -  buildout
    -  celeryd

=======
CHANGES
=======

1.1 (2013-12-19)
================

- Add missing files to Manifest.


1.0 (2013-12-18)
================

- Initial release. Based loosely on collective.recipe.celery.