hexagonit.recipe.cmmi 2.0
zc.buildout recipe for compiling and installing source distributions.
License: BSD
Keywords: development buildout recipe

********************************************
Recipe for compiling and installing software
********************************************

.. contents::

The recipe provides the means to compile and install source distributions
using ``configure`` and ``make`` and other similar tools. It is inspired by
the zc.recipe.cmmi_ recipe but provides more control over the build process.

Repository: http://github.com/hexagonit/hexagonit.recipe.cmmi

Clone URL: git clone git://github.com/hexagonit/hexagonit.recipe.cmmi.git

Issue tracker: http://github.com/hexagonit/hexagonit.recipe.cmmi/issues

Supported Python versions: 2.6, 2.7, 3.2, 3.3

Supported zc.buildout versions: 1.x, 2.x

Travis build: |travis|

.. |travis| image:: https://api.travis-ci.org/hexagonit/hexagonit.recipe.cmmi.png

.. _zc.recipe.cmmi : http://pypi.python.org/pypi/zc.recipe.cmmi

Change History
**************

2.0 (2013-04-07)
================

  - Dropped support for Python 2.4/2.5. Use the 1.x versions for those.
    Currently supported versions are Python 2.6, 2.7, 3.2 and 3.3. [dokai]

  - Python 3.2/3.3 support. [dokai, mmariani]

  - Tox support.
    https://github.com/hexagonit/hexagonit.recipe.cmmi/pull/7
    [msabramo]

  - Travis CI integration. https://travis-ci.org/hexagonit/hexagonit.recipe.cmmi
    See https://github.com/hexagonit/hexagonit.recipe.cmmi/pull/7
    [msabramo]

  - Automatically clean up a left-over compile directory from a previously
    failed run. See https://github.com/hexagonit/hexagonit.recipe.cmmi/pull/9
    [desaintmartin]


1.6 (2012-06-28)
================

  - Relicensed under the `3-clause BSD license <http://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_.28.22New_BSD_License.22_or_.22Modified_BSD_License.22.29>`_.
    [dokai]


1.5.1 (2012-05-21)
==================

  - PEP8 / Pyflakes cleanup.
    [dokai]

  - Quote ``--prefix`` paths to support whitespace characters in
    the path. Closes https://github.com/hexagonit/hexagonit.recipe.cmmi/pull/4
    [galpin]

  - Fixed importing of the hook scripts on Windows. Closes
    https://github.com/hexagonit/hexagonit.recipe.cmmi/pull/6
    [grzn]


1.5.0 (2010-12-17)
==================

  - Refactored the environment variable handling logic. Python versions prior
    to 2.6 have an issue clearing the environment variables using
    ``os.environ.clear()`` (See http://bugs.python.org/issue3227). [dokai]

    Instead of modifying ``os.environ`` directly we use the ``subprocess``
    module to run the commands in child processes which are given an explicit
    environment which is a copy of the current ``os.environ`` augmented with
    the per-part overrides. As a result, ``os.environ`` is no longer modified
    by this recipe.

    The `Python hook scripts`_ are passed the augmented environment dictionary
    as a third parameter.

    .. warning:: Existing hook scripts accepting only two parameters
                 continue to work but they do not have access to the modified
                 environment variables. To fix this they should be refactored
                 to accept the third parameter.

    See https://github.com/hexagonit/hexagonit.recipe.cmmi/issues/issue/1/#issue/1/comment/605362
    for details.


1.4.0 (2010-08-27)
==================

  - Added support for passing options to ``make`` with the new
    ``make-options`` option. See the `Installing a package without an autoconf
    like system`_ section below for an example. [dokai]

  - The ``--prefix`` parameter will be automatically given to the configure
    command if and only if

      - the ``configure-command`` is not used to specify a custom configure command and

      - ``--prefix`` is not given explicitly in the ``configure-options`` option.

    [dokai]

  - Removed the ``is_build_dir()`` heuristic.

    Previously the recipe inspected the contents of the downloaded package to
    determine if it contained the necessary files for building the package (it
    checked if files named ``configure`` or ``Makefile.PL`` existed) and gave
    an error message if they were missing. However, the recipe is useful for
    building many different kinds of software packages and checking for
    particular files limited its use severely.

    Now the recipe omits any checks for particular files in the downloaded
    package. It is recommended that you use the ``md5sum`` option in your part
    configuration to assert that you are downloading the package you expect
    to. [dokai]


1.3.1 (2010-08-23)
==================

 - Refactored the ``is_build_dir()`` helper method to make it easier to test
   and override in customized recipes. [dokai]

 - Fixed the handling of the working directory so that it is restored to the
   state before executing the recipe regardless whether the recipe was
   successful or not. Thanks to Jonathan Ballet for the report and preliminary
   patch. [dokai]

 - Fixed http://github.com/hexagonit/hexagonit.recipe.cmmi/issues#issue/1
   Environment variables defined in one part will no longer leak to other
   subsequent parts. [dokai]


1.3.0 (2009-09-20)
==================

  - Added new options ``environment-section`` and ``environment`` to control
    environment variables before executing the recipe.

  - Added a new option, ``prefix``, to override the installation prefix.
    Defaults to the previously hardcoded value of the part location.


1.2.0
=====

  - Added new ``configure-command`` option to control the command used to
    generate the ``Makefile``. This makes it possible to build slightly
    different packages, e.g. Perl projects where Makefile.PL replaces the
    configure script.


1.1.1
=====

  - Don't try to execute hooks if the option is an empty string. This
    will make it possible to disable hooks when extending existing
    parts.


1.1.0
=====

 - Added new option ``path`` to allow building and installing local
   source trees. The ``path`` option is mutually exclusive with
   ``url``.


1.0.1
=====

 - Fixed a bug with 'keep-compile-dir' option. The location of the
   compilation directory was not available through the
   ``options['compile-directory']`` option as documented.


1.0.0
=====

 - Initial public release.

Detailed Documentation
**********************

Supported options
=================

``url``

    URL to the package that will be downloaded and extracted. The
    supported package formats are .tar.gz, .tar.bz2, and .zip. The
    value must be a full URL,
    e.g. http://python.org/ftp/python/2.4.4/Python-2.4.4.tgz. The
    ``path`` option can not be used at the same time with ``url``.

``path``

    Path to a local directory containing the source code to be built
    and installed. The directory must contain the ``configure``
    script. The ``url`` option can not be used at the same time with
    ``path``.

``prefix``

    Custom installation prefix passed to the ``--prefix`` option of the
    ``configure`` script. Defaults to the location of the part. Note that this
    is a convenience shortcut which assumes that the default ``configure``
    command is used to configure the package. If the ``configure-command``
    option is used to define a custom configure command no automatic
    ``--prefix`` injection takes place. You can also set the ``--prefix``
    parameter explicitly in ``configure-options``.

``md5sum``

    MD5 checksum for the package file. If available the MD5
    checksum of the downloaded package will be compared to this value
    and if the values do not match the execution of the recipe will
    fail.

``make-binary``

    Path to the ``make`` program. Defaults to 'make' which
    should work on any system that has the ``make`` program available
    in the system ``PATH``.

``make-options``

    Extra ``KEY=VALUE`` options included in the invocation of the ``make``
    program. Multiple options can be given on separate lines to increase
    readability.

``make-targets``

    Targets for the ``make`` command. Defaults to 'install'
    which will be enough to install most software packages. You only
    need to use this if you want to build alternate targets. Each
    target must be given on a separate line.

``configure-command``

    Name of the configure command that will be run to generate the Makefile.
    This defaults to ``./configure`` which is fine for packages that come with
    a configure script. You may wish to change this when compiling packages
    with a different set up. See the ``Compiling a Perl package`` section for
    an example.

``configure-options``

    Extra options to be given to the ``configure`` script. By default
    only the ``--prefix`` option is passed which is set to the part
    directory. Each option must be given on a separate line.

``patch-binary``

    Path to the ``patch`` program. Defaults to 'patch' which should
    work on any system that has the ``patch`` program available in the
    system ``PATH``.

``patch-options``

    Options passed to the ``patch`` program. Defaults to ``-p0``.

``patches``

    List of patch files to the applied to the extracted source. Each
    file should be given on a separate line.

.. _Python hook scripts:

``pre-configure-hook``

    Custom python script that will be executed before running the
    ``configure`` script. The format of the options is::

        /path/to/the/module.py:name_of_callable

    where the first part is a filesystem path to the python module and the
    second part is the name of the callable in the module that will be called.
    The callable will be passed three parameters in the following order:

        1. The ``options`` dictionary from the recipe.

        2. The global ``buildout`` dictionary.

        3. A dictionary containing the current ``os.environ`` augmented with
           the part specific overrides.

    The callable is not expected to return anything.

    .. note:: The ``os.environ`` is not modified so if the hook script is
              interested in the environment variable overrides defined for the
              part it needs to read them from the dictionary that is passed in
              as the third parameter instead of accessing ``os.environ``
              directly.

``pre-make-hook``

    Custom python script that will be executed before running
    ``make``. The format and semantics are the same as with the
    ``pre-configure-hook`` option.

``post-make-hook``

    Custom python script that will be executed after running
    ``make``. The format and semantics are the same as with the
    ``pre-configure-hook`` option.

``keep-compile-dir``

    Switch to optionally keep the temporary directory where the
    package was compiled. This is mostly useful for other recipes that
    use this recipe to compile a software but wish to do some
    additional steps not handled by this recipe. The location of the
    compile directory is stored in ``options['compile-directory']``.
    Accepted values are ``true`` or ``false``, defaults to ``false``.

``environment-section``

    Name of a section that provides environment variables that will be used to
    augment the variables read from ``os.environ`` before executing the
    recipe.

    This recipe does not modify ``os.environ`` directly. External commands
    run as part of the recipe (e.g. make, configure, etc.) get an augmented
    environment when they are forked. Python hook scripts are passed the
    augmented as a parameter.

    The values of the environment variables may contain references to other
    existing environment variables (including themselves) in the form of
    Python string interpolation variables using the dictionary notation. These
    references will be expanded using values from ``os.environ``. This can be
    used, for example, to append to the ``PATH`` variable, e.g.::

        [component]
        recipe = hexagonit.recipe.cmmi
        environment-section =
            environment

        [environment]
        PATH = %(PATH)s:${buildout:directory}/bin

``environment``

  A sequence of ``KEY=VALUE`` pairs separated by newlines that define
  additional environment variables used to update ``os.environ`` before
  executing the recipe.

  The semantics of this option are the same as ``environment-section``. If
  both ``environment-section`` and ``environment`` are provided the values from
  the former will be overridden by the latter allowing per-part customization.

Additionally, the recipe honors the ``download-cache`` option set
in the ``[buildout]`` section and stores the downloaded files under
it. If the value is not set a directory called ``downloads`` will be
created in the root of the buildout and the ``download-cache``
option set accordingly.

The recipe will first check if there is a local copy of the package
before downloading it from the net. Files can be shared among
different buildouts by setting the ``download-cache`` to the same
location.

Example usage
=============

We'll use a simple tarball to demonstrate the recipe.

    >>> import os.path
    >>> src = join(os.path.dirname(__file__), 'testdata')
    >>> ls(src)
    - Foo-Bar-0.0.0.tar.gz
    - haproxy-1.4.8-dummy.tar.gz
    - package-0.0.0.tar.gz

The package contains a dummy ``configure`` script that will simply
echo the options it was called with and create a ``Makefile`` that
will do the same.

Let's create a buildout to build and install the package.

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ...
    ... [package]
    ... recipe = hexagonit.recipe.cmmi
    ... url = file://%s/package-0.0.0.tar.gz
    ... """ % src)

This will download, extract and build our demo package with the
default build options.

    >>> print(system(buildout))
    Installing package.
    package: Extracting package to /sample_buildout/parts/package__compile__
    configure --prefix=/sample_buildout/parts/package
    building package
    installing package

As we can see the configure script was called with the ``--prefix``
option by default followed by calls to ``make`` and ``make install``.

Installing a Perl package
=========================

The recipe can be used to install packages that use a slightly different build
process. Perl packages often come with a ``Makefile.PL`` script that performs
the same task as a ``configure`` script and generates a ``Makefile``.

We can build and install such a package by overriding the ``configure-command``
option. The following example builds a Foo::Bar perl module and installs it in
a custom location within the buildout::

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = foobar
    ... perl_lib = ${buildout:directory}/perl_lib
    ...
    ... [foobar]
    ... recipe = hexagonit.recipe.cmmi
    ... configure-command = perl -I${buildout:perl_lib}/lib/perl5 Makefile.PL INSTALL_BASE=${buildout:perl_lib}
    ... url = file://%s/Foo-Bar-0.0.0.tar.gz
    ... """ % src)

    >>> print(system(buildout))
    Uninstalling package.
    Installing foobar.
    foobar: Extracting package to /sample_buildout/parts/foobar__compile__
    building package
    installing package

.. _Installing a package without an autoconf like system:

Installing a package without an ``autoconf`` like system
========================================================

Some packages do not use a configuration mechanism and simply provide a
``Makefile`` for building. It is common in these cases that the build process
is controlled entirely by direct options to ``make``. We can build such a
package by faking a configure command that does nothing and passing the
appropriate options to ``make``. The ``true`` utility found in most shell
environments is a good candidate for this although anything that returns a
zero exit code would do.

We are using a dummy "HAProxy" package as an example of a package with only a
Makefile and using explicit ``make`` options to control the build process.

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = haproxy
    ...
    ... [haproxy]
    ... recipe = hexagonit.recipe.cmmi
    ... configure-command = true
    ... make-options =
    ...     TARGET=linux26
    ...     CPU=i686
    ...     USE_PCRE=1
    ... url = file://%s/haproxy-1.4.8-dummy.tar.gz
    ... """ % src)

    >>> print(system(buildout))
    Uninstalling foobar.
    Installing haproxy.
    haproxy: Extracting package to /sample_buildout/parts/haproxy__compile__
    Building HAProxy 1.4.8 (dummy package)
    TARGET: linux26
    CPU: i686
    USE_PCRE: 1
    Installing haproxy

Installing checkouts
====================

Sometimes instead of downloading and building an existing tarball we need to
work with code that is already available on the filesystem, for example an SVN
checkout.

Instead of providing the ``url`` option we will provide a ``path`` option to
the directory containing the source code.

Let's demonstrate this by first unpacking our test package to the filesystem
and building that.

    >>> checkout_dir = tmpdir('checkout')
    >>> import setuptools.archive_util
    >>> setuptools.archive_util.unpack_archive('%s/package-0.0.0.tar.gz' % src,
    ...                                        checkout_dir)
    >>> ls(checkout_dir)
    d package-0.0.0

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ...
    ... [package]
    ... recipe = hexagonit.recipe.cmmi
    ... path = %s/package-0.0.0
    ... """ % checkout_dir)

    >>> print(system(buildout))
    Uninstalling haproxy.
    Installing package.
    package: Using local source directory: /checkout/package-0.0.0
    configure --prefix=/sample_buildout/parts/package
    building package
    installing package

Since using the ``path`` implies that the source code has been acquired
outside of the control of the recipe also the responsibility of managing it is
outside of the recipe.

Depending on the software you may need to manually run ``make clean`` etc.
between buildout runs if you make changes to the code. Also, the
``keep-compile-dir`` has no effect when ``path`` is used.


Advanced configuration
======================

The above options are enough to build most packages. However, in some cases it
is not enough and we need to control the build process more. Let's try again
with a new buildout and provide more options.

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ...
    ... [build-environment]
    ... CFLAGS = -I/sw/include
    ... LDFLAGS = -I/sw/lib
    ...
    ... [package]
    ... recipe = hexagonit.recipe.cmmi
    ... url = file://%(src)s/package-0.0.0.tar.gz
    ... md5sum = 6b94295c042a91ea3203857326bc9209
    ... prefix = /somewhere/else
    ... environment-section = build-environment
    ... environment =
    ...     LDFLAGS=-L/sw/lib -L/some/extra/lib
    ... configure-options =
    ...     --with-threads
    ...     --without-foobar
    ... make-targets =
    ...     install
    ...     install-lib
    ... patches =
    ...     patches/configure.patch
    ...     patches/Makefile.dist.patch
    ... """ % dict(src=src))

This configuration uses custom configure options, an environment section,
per-part customization to the environment, custom prefix, multiple make
targets and also patches the source code before the scripts are run.

    >>> print(system(buildout))
    Uninstalling package.
    Installing package.
    package: [ENV] CFLAGS = -I/sw/include
    package: [ENV] LDFLAGS = -L/sw/lib -L/some/extra/lib
    package: Extracting package to /sample_buildout/parts/package__compile__
    package: Applying patches
    patching file configure
    patching file Makefile.dist
    patched-configure --prefix=/somewhere/else --with-threads --without-foobar
    building patched package
    installing patched package
    installing patched package-lib


Customizing the build process
=============================

Sometimes even the above is not enough and you need to be able to control the
process in even more detail. One such use case would be to perform dynamic
substitutions on the source code (possible based on information from the
buildout) which cannot be done with static patches or to simply run arbitrary
commands.

The recipe allows you to write custom python scripts that hook into the build
process. You can define a script to be run:

 - before the configure script is executed (pre-configure-hook)
 - before the make process is executed (pre-make-hook)
 - after the make process is finished (post-make-hook)

Each option needs to contain the following information

  /full/path/to/the/python/module.py:name_of_callable

where the callable object (here name_of_callable) is expected to take three
parameters:

    1. The ``options`` dictionary from the recipe.

    2. The global ``buildout`` dictionary.

    3. A dictionary containing the current ``os.environ`` augmented with
       the part specific overrides.

These parameters should provide the callable all the necessary information to
perform any part specific customization to the build process.

Let's create a simple python script to demonstrate the functionality. You can
naturally have separate modules for each hook or simply use just one or two
hooks. Here we use just a single module.

    >>> hooks = tmpdir('hooks')
    >>> write(hooks, 'customhandlers.py',
    ... """
    ... import logging
    ... log = logging.getLogger('hook')
    ...
    ... def preconfigure(options, buildout, environment):
    ...     log.info('This is pre-configure-hook!')
    ...
    ... def premake(options, buildout, environment):
    ...     log.info('This is pre-make-hook!')
    ...
    ... def postmake(options, buildout, environment):
    ...     log.info('This is post-make-hook!')
    ...
    ... """)

and a new buildout to try it out

    >>> write('buildout.cfg',
    ... """
    ... [buildout]
    ... newest = false
    ... parts = package
    ...
    ... [package]
    ... recipe = hexagonit.recipe.cmmi
    ... url = file://%(src)s/package-0.0.0.tar.gz
    ... pre-configure-hook = %(module)s:preconfigure
    ... pre-make-hook = %(module)s:premake
    ... post-make-hook = %(module)s:postmake
    ... """ % dict(src=src, module='%s/customhandlers.py' % hooks))

    >>> print(system(buildout))
    Uninstalling package.
    Installing package.
    package: Extracting package to /sample_buildout/parts/package__compile__
    package: Executing pre-configure-hook
    hook: This is pre-configure-hook!
    configure --prefix=/sample_buildout/parts/package
    package: Executing pre-make-hook
    hook: This is pre-make-hook!
    building package
    installing package
    package: Executing post-make-hook
    hook: This is post-make-hook!

For even more specific needs you can write your own recipe that uses
``hexagonit.recipe.cmmi`` and set the ``keep-compile-dir`` option to ``true``.
You can then continue from where this recipe finished by reading the location
of the compile directory from ``options['compile-directory']`` from your own
recipe.


Contributors
************

* Kai Lautaportti (dokai), Author
* Cédric de Saint Martin (desaintmartin)
* Marc Abramowitz (msabramo)
* Nicolas Dumazet (nicdumz)
* Guy Rozendorn (grzn)
* Marco Mariani (mmariani)
* galpin

Download
***********************