chevah-buildbot-configuration-builder 0.3.0
Configuration layer on top of Buildbot dict.
License: MIT

buildbot-configuration-builder
##############################

A configuration layer on top of Buildbot configuration dict.

You pass a simplified dict and it will return a new dict with all objects
required for Buildbot configuration.

This project tries to simplify buildbot configuration by defining a set
of common rules to be used by all builders. It reduced flexibility in favor
or simplicity.

Steps are just shell commands which take arguments and environment variables.

It provide a few specialized steps (ex to get the source code) but is not
designed for custom steps written in Python.

Once the buildbot configuration is defined as a simple dict it should be
easy for any member of the team to maintain it (add slaves or builders).

For configuration the Buildbot you will still need to read buildbot
documentation.

At this stage it is designed to be used for private buildmasters.

It provides the following features:

* Get single
* Multiple projects
* Slaves definition
* Multiple slaves for a builder.
* HTTP web status
* Try schedulers with username/password authentication
* Send build results to GitHub Pull Request state
* Send email notifications

I plan to add:

* Run steps in a Python virtualenv
* All defining Python version for virtualenv
* Scheduler based on GitHub PR commits
* Scheduler based on GitHub commits, to replace polling scheduler.
* IRC status... for example when master builds fail

It does not provide (patches welcomed):

* SVN, BZR, CVS...etc
* Get code from multiple sources


Travis CI comparison
====================

This project was inspired by Travis CI simple configuration file.
I wanted to have something similar for Buildbot.

There are already a few project which can convert a travis.yml into a
buildmaster configuration so this project is not designed this purpose.

I wanted multiple projects on a single Buildbot instance and with slaves
on multiple operating systems.

I also wanted the powers of `buildbot try` which allows to test a patch
without committing the changes.


Builders
========

The reason why you ended up with buildbot is to get a builder which will
run a set of steps, on a particular slave with option environment variables.

Normal buildbot builder are magically created based on an `environment` and a
set of steps. `environments` are created based on a set of slaves and a set
of environment variables.

Continue reading for info about slaves, environments, projects, steps.


Slaves
======

A slave is the normal buildslave which has a name, password, max_builds.
They a defined using the `slaves` root key.

You have the optional `DEFAULT` slave definition with settings for all slaves.

I hope that slaves are simple to understand, so here is an example::

    {
    OTHER_CONFIGS: {},

    'slaves': {
        # `DEFAULT` values are applied to all slaves and can be overwritten
        # for each slave.
        DEFAULT: {
            'password': 'password',
            'max_builds': 1,
            'notify_on_missing': 'infrastructure@chevah.com',
            },

        # We have 2 linux slaved with default configuration.
        'bs1c-lnx-ubuntu1404-x64-29': {},
        'bs1i-lnx-debian7-x86-37': {},

        # This is a windows slave which uses a different password.
        'bs1i-win-2012-x64-34': {
            'password': 'other-password'
            },

        # We have some linux slaves which can run many tasks.
        'bsmeta1b-lnx-ubuntu1204-x86-30': {'max_builds': 100},
        'bsmeta2i-lnx-ubuntu1204-x86-32': {'max_builds': 100},

        },
    }


Environments
============

The concept of `environment` is inspired by Travis CI environments.

You create an `environment` by associated a set of (similar) slaves with
a set of environment variables.

A slave can be part of multiple environments.

Environments are latest combined with projects and steps to create a builder.

You will define in a single place all the environments used by your projects.
In this way you can share similar environments (builders) between multiple
projects.
Later, in project definition you can choose what environments are associated
with a specific project.

The final builder generated from environment and the project will be named:
`PROJECTNAME-ENVNAME`

All values from the environment dictionary are copied to builders, with
the exception of the `slaves` list.

The builder will run only on one of the slaves associated with the environment,
based on a random rule.

To increase availability you can define multiple slaves for an environment,hence for a builder.

You have the optional `DEFAULT` environment definition with settings for all
environments.

In addition, the following environment values are always set:

* `CI=true`
* `BUILDBOT=true`
* `COMMIT=current_revision`
* `BRANCH=name_of_the_branch`
* `BUILD_NUMBER=buildbot_build_number`
* `BUILDER_NAME=name_of_buildbot_builder`
* `BUILD_DIR=directory_where_test_is_executed`
* `TEST_ENVIRONMENT=name_of_builders_environment`
* `TEST_ARGUMENTS=builder_test_property`

`TEST_ARGUMENTS` are not present in builds triggered by gatekeepers. This is
done on purpose to avoid variable builds for gatekeepers.
`TEST_ARGUMENTS` are copied in group builders, as well as skip steps.

The following environment variables are set if there is a property with the
same name, but in lowercase:

* `GITHUB_TOKEN`
* `GITHUB_PULL_ID`
* `CODECOV_TOKEN`
* `TEST_AUTHOR`


Example::

    {
    OTHER_CONFIGS: {},
    'environments': {
        # These env args are applied to all environments.
        DEFAULT: {
            'TEST_TYPE': 'normal'
            },
        # We want to run tests on Linux
        # but you can create environments based on distro / CPU, etc
        # When combined with project named `brink` it will create a
        # builder named `brink-linux`.
        'linux': {
            'slaves: [
                'bs1c-lnx-ubuntu1404-x64-29',
                'bs1a-lnx-centos7-x64-31'
                ],
            },
        # This will generated a special builder which will overwrite the
        # default environment variable `TEST_TYPE` and also add a new one.
        # For project `brink` it will create builder `brink-leaks`.
        'leaks': {
            'slaves': [
                'bs1c-lnx-ubuntu1404-x64-29',
                'bs1i-win-2012-x64-34',
                ]
            'TEST_TYPE': 'leaks',
            'CHEVAH_GC': 'yes',
            },

        # In case an environment has no extra environment variables you can
        # defined it as a list of slaves.
        # This will create builder `brink-meta-director`.
        'meta-director': [
            'bsmeta1b-lnx-ubuntu1204-x86-30',
            'bsmeta2i-lnx-ubuntu1204-x86-32',
            ],
        },
    }


Projects
========

You can use the same Buildbot installation for multiple projects.
Projects don't have direct access to slaves but rather use the environments
to run project's code on a slave.

A project defines a set of project specific options:

* `repo` - the url used to get project source
* `github_slug` - used to publish GitHub commit status
* `poll_interval` - number of seconds to wait for change source scheduler

It also defines a set of steps, a set of groups and a set of gatekeepers which
are explained later.

A `DEFAULT` project can be used do define default values for all projects.
These values will be used when a project does not specify a specific
configuration.

Here is an example for defining 2 projects. Steps, groups and gatekeepers
are explained later::

    {
    OTHER_CONFIGS: {}
    'projects': {
        DEFAULT: {
            'steps': [DEFAULT_STEPS_FOR_ALL_PROJECTS],

            'poll_interval': 60,
            },
        # `brink` project will use the default steps
        # will will check for changes on master much often.
        'brink': {
            'repo': 'http://git.chevah.com/brink.git',
            'github_slug': 'chevah/brink',
            'poll_interval': 30,
            'groups': BRINK_GROUPS_EXPLAINED_LATER,
            'gatekeepers': BRINK_GK_EXPLAINED_LATER,
            },
        # `compat` project has a different set of steps.
        'compat': {
            'repo': 'http://git.chevah.com/compat.git',
            'github_slug': 'chevah/compat',
            'steps': [COMPAT_SPECIFIC_STEPS],
            'groups': COMPAT_GROUPS_EXPLAINED_LATER,
            'gatekeepers': COMPAT_GK_EXPLAINED_LATER,
            },
        },
    }


Steps
=====

The following step types are available

* SLAVE_COMMAND - execute a shell command on slave
* SOURCE_COMMAND - get project code or apply patch
* MERGE_COMMAND - merge code with a branch

Default step type is SLAVE_COMMAND.

You can conditionally execute a step by using the `optional` configuration.
In this case it will be executed only when when `force_STEPNAME` property
is present on the builder and is not false.

The same set of steps are executed for all builders. In order to run
different tests based on different environments/builders you should dispatch
them bases on environment variable.

For example to run unit tests, pyflakes checker and documentation builder a
single `run_ci` shell command is used which should dispatch a specialized
command based on the environment variables.

Steps are defined inside the project's `steps` key::

    from chevah.buildbot_configuration_builder.builder import (
        MERGE_COMMAND,
        SOURCE_COMMAND,
        )

    {
    OTHER_CONFIGS: {}
    'projects': {
        'brink': {
            'steps': [
                # Get source based on project settings.
                {'type': SOURCE_COMMAND},
                # Merge with master.
                {'type': MERGE_COMMAND , 'branch': 'master'},
                # Option clean the build folder.
                {
                    'name': 'clean',
                    'command': ['bash','./paver.sh', 'clean'],
                    # This step is only executed when `force_clean=1`
                    # builder property is defined.
                    'optional': True,
                    },
                # Build dependencies
                {
                    'name': 'deps',
                    'command': ['bash','./paver.sh', 'deps'],
                    'add_environment': {
                        'SOME_VAR': 'some-value',
                        'OTHER_VAR': Interpolate('%(prop:builder)s'),
                        },
                    },
                # Run tests
                {
                    'name': 'test',
                    'command': ['bash', './paver.sh', 'run_ci'],
                    },
                ],
            },
        },
    }


Groups
======

You can group multiple builders into a group.
For example you can create a group named `pr` which will trigger all builders
required to validate a pull request commit and another group named
`post-commit` which will trigger all builders required to check the committed
code. Or you can group them in 'supported' and 'experimental' builders.

For each member of a group a dedicated builder is created. This builder will
execute the steps associated with this project. The builder will be named
PROJECT_NAME-ENV_NAME.

A builder can be part of multiple groups.

The group will have a dedicated group builder which will trigger a build for
all builders form the group and report the results once all builders are done.

The builder associated with GROUP_NAME for PROJECT_NAME will be named
PROJECT_NAME-group-GROUP_NAME.

Groups are defined inside the project's `groups` key::

    {
    OTHER_CONFIGS: {}
    'projects': {
        'brink': {
            'groups': {
                # This will create a builder named `brink-group-post-commit`
                # and we can use this builder to trigger multiple builders.
                'post-commit': [
                    # Here is a list with environment names, NOT slave names.
                    # This will created the following builders:
                    # brink-leaks, brink-linux-x86, brink-windows-x64
                    'leaks',
                    'linux-x86',
                    'windows-x64'
                    ],
                # This will create a builder named `brink-group-supported`
                # Since `leaks` builder is very slow we don't run to check
                # if a changes is ready for review.
                'supported': [
                    'linux-x86',
                    'windows-x64',
                    ],
                # This will create a group named `brink-group-unstable`.
                'unstable': [
                    'solaris-x86',
                    'freebsd-x64',
                    ],
                },
            },
        },
    }


Gatekeepers
===========

Gatekeepers are specialized builders which will not use the project's steps.

The following step types are available:

* SEQUENTIAL_GROUP - run all builders from a group one after another and wait
  for all builders to end.
* PARALLEL_GROUP - run all builders in parallel and wait for all to end.
* MASTER_COMMAND - run a shell command on master
* DIRECTORY_UPLOAD - upload a folder on master

By default gatekeeper builders are triggered by try schedulers. You can
change this to trigger the builder based on changes on a branch using
`'scheduler': 'master'` option.


Gatekeepers are defined inside the project's `gatekeepers` key::

    {
    OTHER_CONFIGS: {}
    'projects': {
        'brink': {
            'gatekeepers': {
                # Post commit builder which will run all tests
                # in sequential mode.
                'master': {
                    # Run this builder after changes are done on master branch.
                    'scheduler': 'master',
                    'stable_timer': 300,
                    'steps': [
                        {
                        # Trigger each builde from a group, one after another.
                        'type': SEQUENTIAL_GROUP,
                        # Name of the step show in logs.
                        'name': 'step name all',
                        # Name of the triggered group.
                        'target': 'post-commit',
                        },
                        {
                        # Upload local folder `dist` on buildmaster.
                        'type': DIRECTORY_UPLOAD,
                        'name': 'upload_production',
                        'source': 'dist',
                        'destination': '/srv/buildmaster/upload/production',
                        'optional': True,
                        },
                        {
                        # Execute shell command on buildmaster.
                        'type': MASTER_COMMAND,
                        'name': 'Fix permissions',
                        'command': ['chmod', '-R', '755', '/srv/buildmaster/upload'],
                        },
                        ],
                    'notifications': {
                        # Once tests on master are done notify everyone on
                        # both success or failure.
                        'email_all': ['dev@domain.com'],
                        },
                    },

                # Trigger tests and send result to GitHub for pull request
                # status when asked by buildbot try.
                'review': {
                    'steps': [{
                        'type': PARALLEL_GROUP,
                        'name': 'check review step',
                        'target': 'supported',
                        'set_properties': {
                            'codecov_token': '1234',
                            },
                        'copy_properties': ['github_pull_id'],
                        }],
                    'notifications': {
                        'email_all': [INTERESTED_USERS],
                        },
                    'github_send_status': True,
                    },

                # Trigger tests and merge in master on success
                # When asked by buildbot try.
                'merge': {
                    'steps':  [
                        {'type': SOURCE_COMMAND},
                        {
                            'type': PARALLEL_GROUP,
                            'name': 'all',
                            'target': 'all',
                            },
                        {
                            'name': 'merge-commit',
                            'command': ['bash', './paver.sh', 'merge_commit'],
                            },
                        ],
                    'notifications': {
                        'email_passing': ['dev@domain.com'],
                        'email_error': [INTERESTED_USERS],
                        },
                    },
                },
            },
        },
    }


Global buildmaster configuration
================================

You can directly define buildmaster configuration by using the `global` key
from root dict::

    {
    OTHER_CONFIGS: {}
    'global': {
        'title': 'ACME Project Buildbot',
        'db_url': 'sqlite:///state.sqlite',
        'buildbotURL': 'http://build.domain.com/',
        'titleURL': 'http://www.domain.com',

        'slavePortnum': 10089,

        'changeHorizon': 500,
        'buildHorizon': 500,
        'eventHorizon': 50,
        'logHorizon': 50,
        'buildCacheSize': 15,

        'logCompressionMethod': 'gz',

        'mergeRequests': True,
        },
    }


Web status
==========

Web status settings are defined in the `web` key of the root dict::

    {
    OTHER_CONFIGS: {}
    'web': {
        'interface': '0.0.0.0',
        'port': 10088,
        'htpasswd': '/srv/www/buildbot.passwd',
        'authorization': {
            'gracefulShutdown': True,
            'forceBuild': True,
            'forceAllBuilds': True,
            'pingBuilder': True,
            'stopBuild': True,
            'stopAllBuilds': True,
            'cancelPendingBuild': True,
            },
        },
    }


Email notifications
===================

Gatekeeper builders can be configured to send email notifications.

Email settings are defined in the `email` key of the root dict::

    def lookup(user):
        """
        Called with each responsible user of the build and should
        return the full email address.
        """
        return user + '@domain.tld'

    {
    OTHER_CONFIGS: {}
    'email': {
        'server': {
            'fromaddr': 'buildbot@domain.com',
            'relayhost': 'smtp.domain.com',
            'useTls': True,
            'smtpPort': 587,
            'smtpUser': 'stmp-user',
            'smtpPassword': 'stmp-pass',
            },

        # This is optional and use to convert buildbot users to
        # email addresses.
        'user_to_email_mapper': lookup,
        # Available placeholders for subject:
        #  result, projectName, title, builder
        'subject': '%(result)s %(builder)s',
        },
    }


Try schedulers and triggers
===========================

Each builder has an associated Try scheduler so you can use `buildbot try`
to run a patch or a revision.

Group builders and gatekeeper builders also have an associated try scheduler.

They are defined in the `try_scheduler` root key::

    {
    OTHER_CONFIGS: {}
    'try_scheduler': {
        # Port on which to listed for buildbot try requests.
        'port': 10087,
        # Credentials accepted by buildbot try.
        'credentials': [('try_user_name', 'try_user_pass')],
        # Default environment/slaves used to execute the try schedulers for
        # groups and other builders which don't have an explicit environment.
        'environment': 'meta-director',
        },
    }


GitHub Integration
==================

You can send Buildbot's results for a specific commit to GitHub.
To send them you will need a GitHub token.

Each project also need to define the GitHub slug. See `projects` documentation.

GitHub is configured using the `github` key from root dict::

    {
    OTHER_CONFIGS: {}
    'github': {
        'token': 'GITHUB-TOKEN-VALUE',
        },
    }


master.cfg integration
======================

This is the basic content of your master.cfg file::

    # You will need to import more things for here to define steps.
    from chevah.buildbot_configuration_builder.builder import (
        generate_configuration
        )

    config = {
        'global': {YOUR_DATA},
        'try_scheduler': {YOUR_DATA},
        'github': {YOUR_DATA},
        'web': {YOUR_DATA},
        'email': {YOUR_DATA},
        'slaves': {YOUR_DATA},
        'environments': {YOUR_DATA},
        'projects': {YOUR_DATA},
        }
    BuildmasterConfig = generate_configuration(config)