humilis 1.7.1
AWS cloudformation-based deployment framework
License: MIT

humilis
=======

|Build Status| |PyPI|

Helps you deploy AWS infrastructure with
`Cloudformation <https://aws.amazon.com/cloudformation/>`__.

This project is originally based on the
`cumulus <https://github.com/cotdsa/cumulus>`__ project. See
`CUMULUS_LICENSE <https://github.com/humilis/humilis/blob/master/CUMULUS_LICENSE>`__
for license information.

Installation
============

Install the `AWS CLI <https://aws.amazon.com/cli/>`__:

::

   pip install awscli

Configure the AWS CLI:

::

   aws configure

``humilis`` will use whatever credentials you introduced when
configuring your AWS CLI installation.

You can now install the latest “stable” version of ``humilis``:

::

   pip install humilis

or the development version if you prefer that:

::

   pip install git+https://github.com/humilis/humilis

After installation you need to configure humilis. To configure globally
for your system:

::

   humilis configure

The command above will store and read the configuration options from
``~/.humilis.ini``. You can also store the configuration in a
``.humilis.ini`` file stored in your current working directory by using:

::

   humilis configure --local

``humilis`` will always read the configuration first from a
``.humilis.ini`` file under your current work directory. If it is not
found then it will read it from your system global config file
``~/.humilis``.

Development environment
=======================

Assuming you have
`virtualenv <https://virtualenv.readthedocs.org/en/latest/>`__
installed:

::

   make develop

   . .env/bin/activate

Testing
=======

At the moment, most tests are integration tests with the AWS SDK. This
means that you will need to `set up your
system <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html>`__
to access AWS resources if you want to run the test suite.

::

   py.test tests

Quickstart
==========

Define your infrastructure environment following the examples in the
`examples
directory <https://github.com/humilis/humilis/tree/master/examples>`__.
Then to create the environment:

::

   humilis create examples/humilis-firehose.yaml

To update the environment after it has been deployed:

::

   humilis update examples/humilis-firehose.yaml

And to delete it:

::

   humilis delete examples/humilis-firehose.yaml

Humilis environments
====================

A ``humilis`` environment is just a collection of cloudformation stacks
that are required for an application. Instead of having a monolytic CF
template for your complete application, ``humilis`` allows you to define
infrastructure *layers* that are combined into an *environment*. Each
``humilis`` layer translates exactly into one CF template (therefore
into one CF stack after the layer is deployed).

Breaking a complex infrastructure environment into smaller layers has at
least two obvious advantages:

-  **Easier to maintain**. It’s easier to maintain a simple layer that
   contains just a bunch of `CF
   resources <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html>`__
   than serve a well-defined purpose.

-  **Easier to reuse**. You should strive to define your infrastructure
   layers in such a way that you can reuse them across various
   environments. For instance, many projects may require a base layer
   that defines a VPC, a few subnets, a gateway and some routing tables,
   and maybe a (managed) NAT. You can define a humilis layer with those
   resources and have a set of layer parameters (e.g. the VPC CIDR) that
   will allow you to easily reuse it across environments.

Environment anatomy
-------------------

An environment *definition file* is a
`yaml <https://en.wikipedia.org/wiki/YAML>`__ document that specifies
the list of layers that form your enviroment. The file should be named
as your environment. That is, for environment ``my-app-environment`` the
environment description file should be called
``my-app-environment.yaml``. The contents of the environment definition
should be organized as follows:

::

   ---
   my-app-environment:
       description:
           A description of what this environment is for
       layers:
           # The layers that you environment requires. They will be deployed in the
           # same order as you list them. Note that you can also pass parameters 
           # to a layer (more on that later).
           - {layer: name_of_first_layer, layer_param: layer_value}
           - {layer: name_of_second_layer}
           - {layer: name_of_third_layer}

Layer anatomy
-------------

Anything associated to a given layer must be stored in a directory with
the same name as the layer, within the same directory where the
environment *definition file* is located. If we consider the
``my-app-environment`` environment we used above then your directory
tree should look like this:

::

   .
   ├── my-app-environment.yaml
   ├── name_of_first_layer
   │   ├── meta.yaml
   │   └── resources.yaml
   ├── name_of_second_layer
   │   ├── meta.json
   │   └── meta.yaml
   └── name_of_third_layer
       ├── resources.json.j2
       └── resources.yaml.j2

A layer must contain at least two files:

-  ``meta.yaml``: Meta information about the layer such as a
   description, and layer parameters.
-  ``resources.yaml``: Basically a CF template with the resources that
   the layer contains.

Those two files can also be in ``.json`` format (``meta.json`` and
``resources.json``). Or you can add the extension ``.j2`` if you want
the files to be pre-processed with the
`Jinja2 <http://jinja.pocoo.org/>`__ template compiler.

Below an example of how a layer ``meta.yaml`` may look like:

::

   ---
   meta:
       description:
           Creates a VPC, that's it
       parameters:
           vpc_cidr:
               description: The CIDR block of the VPC
               value: 10.0.0.0/16

Above we declare only one layer parameter: ``vpc_cidr``. ``humilis``
will make pass that parameter to Jinja2 when compiling any template
contained in the layer. So the ``resources.yaml.j2`` for that same layer
may look like this:

::

   ---
   resources:
       VPC:
           Type: "AWS::EC2::VPC"
           Properties:
               CidrBlock: {{ vpc_cidr }}

References
==========

You can use references in your ``meta.yaml`` files to refer to thing
other than resources within the same layer (to refer to resources within
a layer you can simply use Cloudformation’s
`Ref <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html>`__
or
`GetAtt <http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-getatt.html>`__
functions). Humilis references are used by setting the value of a layer
parameter to a dict that has a ``ref`` key. Below an a ``meta.yaml``
that refers to a resource (with a logical name ``VPC``) that is
contained in another layer (called ``vpc_layer``):

::

   ---
   meta:
       description:
           Creates an EC2 instance in the vpc created by the vpc layer
       dependencies:
           - vpc
       parameters:
           vpc:
               description: Physical ID of the VPC where the instance will be created
               value:
                   ref: 
                       parser: layer
                       parameters:
                           layer_name: vpc_layer
                           resource_name: VPC

Every reference must have a ``parser`` key that identifies the parser
that should be used to parse the reference. There are also two optional
keys:

-  ``parameters``: allows you to pass parameters to the reference
   parser. You can pass either named parameters (as a dict) or
   positional arguments (as a list).

-  ``priority``: the parsing priority. Parameters with a lower value in
   ``priority`` will be parsed before parameters with a higher value.
   This allows some reference parsers to refer internally to other
   parameters within the same layer. For example, the ``lambda`` parser,
   when parsing templated lambda code, it uses previously parsed layer
   parameters as template parameters.

More information on the reference parsers that are bundled with humilis
below.

Available reference parsers
---------------------------

``layer_resource`` references
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``layer_resource`` references allow you to refer to the physical ID of a
resource that is part of another layer.

**Parameters**:

-  ``layer_name``: The name of the layer you are referring to
-  ``resource_name``: The logical name of the layer resource

**Example**:

Consider the following environment definition:

::

   ---
   my-environment:
       description:
           Creates a VPC with a NAT in the public subnet
       layers:
           - {layer: vpc}
           - {layer: nat}

Obviously the ``nat`` layer that takes care of deploying the NAT in the
public subnet will need to know the physical ID of that subnet. You
achieve this by declaring a ``layer_resource`` reference in the
``meta.yaml`` for the ``nat`` layer:

::

   ---
   meta:
       description:
           Creates a managed NAT in the public subnet of the NAT layer
       parameters:
           subnet_id:
               description:
                   The physical ID of the subnet where the NAT will be placed
               value:
                   ref:
                       parser: layer_resource
                       parameters:
                           layer_name: vpc
                           # The logical name of the subnet in the vpc layer
                           resource_name: PublicSubnet

When parsing ``meta.yaml`` humilis will replace this:

::

   ref:
       parser: layer_resource
       parameters:
           layer_name: vpc
           # The logical name of the subnet in the vpc layer
           resource_name: PublicSubnet

with the physical ID you need (something like ``subnet-bafa90cd``). You
can then use this physical ID in the ``resources.yaml.j2`` section of
the ``nat`` layer:

::

   {# Pseudo-content of layers/nat/resources.yaml.j2 #}
   resources:
       {# An Elastic IP reservation that will be associated to the NAT #}
       NatEip:
         Type: 'AWS::EC2::EIP'
         Properties: {}
       {# Custom resource deploying the NAT #}
       NatGateway:
         Type: 'Custom::NatGateway',
         Properties:
           {# The ARN of the Lambda function backing the custom resource #}
           ServiceToken: 'arn:aws:lambda:eu-west-1:XXXX:function:CreateNatGateway'
           {# Here we use the subnet_id reference defined in meta.yaml #}
           SubnetId: {{subnet_id}}
           AllocationId:
               Ref: NatEip

``environment_resource`` references
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``environment_output`` references allow you to refer to resources that
belong to other humilis environments.

**Parameters**:

-  ``environment_name``: The name of the environment you are referring
   to
-  ``layer_name``: The name of the layer you are referring to
-  ``resource_name``: The logical name of the layer resource

``layer_output`` references
~~~~~~~~~~~~~~~~~~~~~~~~~~~

``layer_output`` references allow you to refer to outputs produced by
another layer.

**Parameters**:

-  ``layer_name``: The name of the layer you are referring to
-  ``output_name``: The logical name of the output parameter

In general you should prefer using ``layer_output`` references over
``layer_resource`` references. The output parameters produced by a layer
define an informal *layer interface* that is more likely to remain
constant than the logical names of resources within a layer.

``boto3`` references
~~~~~~~~~~~~~~~~~~~~

``boto3`` references define arbitrary calls to
`boto3facade <https://github.com/InnovativeTravel/boto3facade>`__. The
latter is just a simpler facade interface on top of
`boto3 <https://github.com/boto/boto3>`__.

**Parameters**:

-  ``service``: The AWS service, e.g. \ ``ec2`` or ``cloudformation``.
   Note that only only AWS services that have a facade in
   `boto3facade <https://github.com/InnovativeTravel/boto3facade>`__ are
   supported.
-  ``call``: The corresponding facade method,
   e.g. \ ``get_ami_by_name``. The value of this parameter must be a
   dictionary with a ``method`` key (the name of the facade method to
   invoke) and an optional ``args`` key (the parameters to pass to the
   facade method). Best to look at the example below to understand how
   this works.
-  ``output_attribute``: Optional. If provided the reference parser will
   return the value of this attribute from the object returned by the
   facade method.

Below an example of a layer that uses a ``boto3`` reference:

::

   ---
   meta:
       description:
           Creates an EC2 instance using a named AMI
       # More stuff omitted for brevity
       ami:
           description: The AMI to use when launching the EC2 instance
           value:
               ref:
                   parser: boto3
                   parameters:
                       service: ec2
                       call:
                           method: get_ami_by_name
                           args:
                               - test-ami
                       output_attribute: id

``humilis`` will parse the reference using this code:

::

   # Import the Ec2 facade
   from boto3facade.ec2 import Ec2

   # Create a facade object
   ec2_facade = Ec2()

   # Make the call
   ami = ec2_facade.get_ami_by_name('test-ami')

   # Extract the requested attribute
   ref_value = ami.id

``file`` references
~~~~~~~~~~~~~~~~~~~

``file`` references allow you to refer to a local file. The file will be
uploaded to S3 and the reference will evaluate to the corresponding S3
path.

**Parameters**:

-  ``path``: The path to the file, relative to the layer root directory.

``lambda`` references
~~~~~~~~~~~~~~~~~~~~~

``lambda`` references allow you to refer to some Python code in your
local machine. If your code follows some simple conventions ``humilis``
will take care of building a `deployment
package <http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html>`__
for you, uploading it to S3, and the reference will evaluate to the S3
path of the deployment package.

**Parameters**:

-  ``path``: Path to either a completely self-contained ``.py`` file, or
   to the root directory of your lambda code. In the latter case your
   code needs to follow some simple conventions for this to work. More
   information below.

-  ``dependencies``: A list of dependencies to be included in the Lambda
   deployment package. Dependencies may be either pip installable
   packages, or paths to local Python packages or modules, or paths to
   local ``requirements`` files.

**Example**:

::

   ref:
       parser: lambda
       parameters:
           # Path to the root directory containing your lambda code
           path: dummy_function
           dependencies:
               # The Lambda code requires Pypi's pyyaml
               - pyyaml
               # It also requires a local package in this path
               - mycode/mypkgdir
               # And this local module
               - mycode/mymodule.py

which will evaluate to a S3 path such as:

::

   s3://[bucket_name]/[environment_name]/[stage_name]/[func_name]-[commithash].zip

**Code conventions**:

Following the example above, the contents of the layer responsible of
deploying the ``dummy_function`` lambda may look like this:

::

   .
   ├── dummy_function
   │   ├── dummy_function.py
   │   └── setup.py
   ├── meta.yaml
   ├── outputs.yaml.j2
   └── resources.yaml.j2

Basically all your code needs to be included under directory
``dummy_function``. In this case there is only one file:
``dummy_function.py``. External dependencies need to be specified in
your ``setup.py``.

``secret`` references
~~~~~~~~~~~~~~~~~~~~~

``secret`` references retrieve a secret using Python’s
`keyring <https://pypi.python.org/pypi/keyring>`__ module.

**Parameters**:

-  ``service``: The name of the service the secret is associated to.

-  ``key``: The key (e.g. the username) that identifies the secret.

**Example**:

::

   ref:
       parser: secret
       parameters: {"service": "mysqldb", "key": "adminuser"}

Custom Jinja2 filters
---------------------

Humilis defines the following `custom Jinja2
filters <http://jinja.pocoo.org/docs/dev/templates/#filters>`__:

-  ``uuid``: A random UUID. Example: ``{{''|uuid}}``.
-  ``password(length=8)``: A random password. Example:
   ``{{10|password}}``.

.. |Build Status| image:: https://travis-ci.org/humilis/humilis.svg?branch=master
   :target: https://travis-ci.org/humilis/humilis
.. |PyPI| image:: https://img.shields.io/pypi/v/humilis.svg?style=flat
   :target: https://pypi.python.org/pypi/humilis