haldane 0.3.0
A friendly http interface to the aws api
License: BSD
Keywords: aws api

=======
haldane
=======

a friendly http interface to the ec2 api

Requirements
============

- Vagrant
- VMWare Fusion
- 1 gigabytes of free ram
- Python 2.7.8 or below due to a bug in 2.7.9 SSL

Installation
============

    OS X Users are assumed to be running homebrew. Please set that up before continuing.
    Our makefile should install everything necessary to run this service within a Ubuntu 14.04 or Mac OS X 10.9+ environment.

On a VM
-------

Once you have vagrant and virtualbox installed, you can bring up a vm with the service installed:

.. code-block:: bash

    cd path/to/haldane
    vagrant up

By default, this will use ``vmware_fusion`` as the vagrant provider, though using you can also use ``virtualbox``:

.. code-block:: bash

    vagrant up --provider virtualbox

Manually
--------

You will need to setup a few system-level dependencies first. You can do this using the following make target:

.. code-block:: bash

    # install system-level dependencies first
    # may ask for sudo permissions to install certain packages
    make requirements

    # now create a python virtual environment with all the required packages
    # will ask for sudo permissions to install virtualenv if it is not available
    # the path is set to ``.virtualenv`` by default, though we respect
    # the following environment variables:
    # - ``WORKON_HOME``
    # - ``VIRTUALENV_PATH``
    make venv

    # activate the virtualenv:
    eval $(make venv-activate)

    # to deactivate the virtualenv, run the following command:
    deactivate

    # for more actions you can perform, just run the ``make`` or ``make help`` commands

Running
=======

You can ssh onto the box and run the webservice:

.. code-block:: bash

    vagrant ssh
    source .env && make server

The webservice will now be running and exposed to the host machine at ``http://localhost:5000``

> Note that you may wish to change the configuration in use. You can do so by modifying the .env.test file with your configuration.

Configuration
=============

All configuration is set via environment variables. The following environment variables are available for use:

- ``ALLOWED_IPS``: (Default: None) A comma-separated list of ip addresses that can basic authentication.
- ``ALTERNATIVE_AUTOSCALE_TAG_NAME``: (Default: None) A tag that can be used as an alternative to the AWS group name to categorize instances.
- ``AWS_API_VERSION``: (Default: ``2016-09-15``) The default api version to use when retrieving instance type.
- ``AWS_ACCESS_KEY_ID``: (Default: None) An AWS access key id
- ``AWS_REGIONS``: (Default: us-east-1) A comma-separated list of regions to query for.
- ``AWS_SECRET_ACCESS_KEY``: (Default: None) An AWS secret access key
- ``BASIC_AUTH``: (Default: None) A list of basic auth user/password combinations. The format for each is ``username:password``.
- ``BOOLEAN_AWS_TAG_ATTRIBUTES``: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set and converted into booleans.
- ``BUGSNAG_API_KEY``: (Default: None) An api key for reporting errors to bugsnag.
- ``CACHE_EXPIRATION``: (Default: ``180``) Time in seconds until a cached AWS api retrieval expires.
- ``CACHE_SIZE``: (Default: ``1024``) Max number of items to cache in the LRU cache. Can be safely set to 2.
- ``DEBUG``: (Default: ``0``) Whether to turn on debug mode or not.
- ``LISTEN_INTERFACE``: (Default: ``0.0.0.0``) The interface which the server will bind to.
- ``PORT``: (Default: ``5000``) Server port.
- ``SENTRY_DSN``: (Default: None) An DSN for reporting errors to sentry.
- ``TOP_LEVEL_AWS_TAG_ATTRIBUTES``: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set.

The AWS policy is fairly small, and an ``iam-profile.json`` is provided in this repository in the case that you wish to lock down permissions to only those necessary.

Endpoints
=========

- ``/``: Healthcheck
- ``/_status``: Healthcheck
- ``/amis?q=<query>&limit=<limit>``: List all amis owned by the user specified by the AWS credentials.
  - ``format`` (optional): If set to ``list``, turns ami attributes from an object indexed by the name key to a list of those objects. Can also be set to ``csv``. Defaults to ``dict``.
  - ``id`` (optional): An image id to filter by (eg. ``ami-21e750d9``)
  - ``query`` (optional): Substring to search ami names by before returning the resultset
  - ``region`` (optional): Filter to a specific region
- ``/instance-types/<api-version>``: List all instance types available for a specific api version (version is optional).
- ``/instances/<region>?q=<query>&limit=<limit>&status=<status>&group=<group>``: List all nodes
  - ``format`` (optional): If set to ``list``, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set to ``csv``. Defaults to ``dict``.
  - ``limit`` (optional): An integer to limit the resultset by
  - ``query`` (optional): Substring to search the ``name`` field by before returning the resultset
- ``/instances/group/<group>?region=<region>&query=<query>&status=<status>``: List all nodes grouped by autoscale group
  - ``format`` (optional): If set to ``list``, turns node attributes from an object indexed by the name key to a list of those objects.
  - ``query`` (optional): Substring to search node names by before returning the resultset
- ``/rds-instances/<region>?q=<query>&limit=<limit>&status=<status>``: List all nodes
  - ``format`` (optional): If set to ``list``, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set to ``csv``. Defaults to ``dict``.
  - ``limit`` (optional): An integer to limit the resultset by
  - ``query`` (optional): Substring to search the ``name`` field by before returning the resultset

Filters
-------

Field values that are *exactly* one of the following strings are transformed into their language "equivalents":

- ``none``
- ``nil``
- ``true``
- ``false``

    The values are lowercased before the check is performed.

The following attributes filters are globally available:

- ``availability_zone`` (optional): An availability zone to filter by (eg. ``us-east-1a``)
- ``id`` (optional): An instance id to filter by (eg. ``i-21e750d9``)
- ``name`` (optional): A name to filter by (eg. ``graphite-ec2-01``)
- ``region`` (optional): Filter to a specific region (eg. ``us-east-1``)
- ``status`` (optional): Filter to specific node status (eg. ``terminated``)

The following attribute filters are available for the ``/instances`` and ``/instances/group`` endpoints:

- ``elastic_ip`` (optional): Whether or not to filter to just instances with an elastic_ip
- ``group`` (optional): An autoscale group name to filter by
- ``image_id`` (optional): An image id to filter by (eg. ``ami-123abc4d``)
- ``image_name`` (optional): An image name to filter by (eg. ``BaseAMI``)
- ``instance_type`` (optional): Filter to a specific instance type (eg. ``t2.large``)
- ``instance_class`` (optional): Filter to a specific instance type (eg. ``t2``)
- ``instance_profile_id`` (optional): Filter to a specific instance profile id (eg. ``O34RQ3IUIO3FOUI3F``)
- ``instance_profile_name`` (optional): Filter to a specific instance profile name (eg. ``Production-Api``)
- ``ip_address`` (optional): Filter to a specific private ip address (eg. ``54.10.2.20``)
- ``private_ip_address`` (optional): Filter to a specific private ip address (eg. ``10.10.2.20``)
- ``vpc_id`` (optional): Filter to specific vpc (eg. ``vpc-8675309``)

The following attribute filters are available for the ``/rds-instances`` endpoint:

- ``allocated_storage`` (optional)
- ``auto_minor_version_upgrade`` (optional)
- ``backup_retention_period`` (optional)
- ``ca_certificate_identifier`` (optional)
- ``copy_tags_to_snapshot`` (optional)
- ``db_instance_arn`` (optional)
- ``db_instance_class`` (optional)
- ``db_instance_port`` (optional)
- ``db_instance_status`` (optional)
- ``db_name`` (optional)
- ``dbi_resource_id`` (optional)
- ``engine`` (optional)
- ``engine_version`` (optional)
- ``enhanced_monitoring_resource_arn`` (optional)
- ``license_model`` (optional)
- ``master_username`` (optional)
- ``monitoring_interval`` (optional)
- ``monitoring_role_arn`` (optional)
- ``multi_az`` (optional)
- ``preferred_backup_window`` (optional)
- ``preferred_maintenance_window`` (optional)
- ``publicly_accessible`` (optional)
- ``secondary_availability_zone`` (optional)
- ``storage_encrypted`` (optional)
- ``storage_type`` (optional)

Field Filtering
~~~~~~~~~~~~~~~

Fields in the response body can be filtered using the ``fields`` querystring argument. Fields are a ``comma-separated`` list of any of the attributes already returned. Tags cannot be filtered on a per-tag basis, though you may choose to include or exclude the ``tags`` attribute entirely.

.. code-block:: bash

    curl http://localhost:5000/instances?fields=id,image_name

Complex Filters
~~~~~~~~~~~~~~~

Filtering is performed *after* retrieving results from the EC2 API. The following are valid filters:

- ``exact``: performs an exact match on the value of the tag name
- ``in-list``: splits the tag value by comma and verifies that the passed value is in the resulting list
- ``is-null``: checks if the value is ``null``
- ``is-true``: checks if the value is ``true``
- ``is-false``: checks if the value is ``false``
- ``substring``: performs a substring match on the value of the tag name
- ``ends-with``: performs a match using ``endswith`` on the value of the tag name
- ``starts-with``: performs a match using ``startswith`` on the value of the tag name
- ``not-in-list``: splits the tag value by comma and verifies that the passed value is *not* in the resulting list
- ``not-substring``: performs an inverse substring match on the value of the tag name
- ``not-ends-with``: performs an inverse match using ``endswith`` on the value of the tag name
- ``not-starts-with``: performs an inverse match using ``startswith`` on the value of the tag name

A simple example is as follows

.. code-block:: bash

    curl http://localhost:5000/instances?substring.name=www

Tag Filtering
~~~~~~~~~~~~~

You can also filter by tags by using the ``tags.FILTER.TAG_NAME`` querystring pattern as follows:

.. code-block:: bash

    curl http://localhost:5000/instances?tags.exact.bootstrapped=true&tags.substring.Name=admin

You may also specify an exact match filter when omitting the ``FILTER`` section like so:

.. code-block:: bash

    curl http://localhost:5000/instances?tags.bootstrapped=true

Filtering by the status field
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Valid ``status`` values are as follows:

- ``pending``
- ``running``
- ``shutting-down``
- ``terminated``
- ``stopping``
- ``stopped``

.. code-block:: bash

    curl http://localhost:5000/instances?status=running

Errors
~~~~~~

If an invalid querystring argument is passed, a ``json`` response similar to the following will be sent from the service:

.. code-block:: javascript

    {
      detail: "Invalid region querystring argument passed.",
      status: 400,
      title: "Invalid argument passed"
    }

The following errors are possible:

- Invalid region querystring argument
- Invalid status querystring argument
- General EC2ResponseError

How it works
============

This services uses ``boto3`` underneath to provide an api listing all nodes available in a given EC2 account. It is intended to be used as a replacement for hitting the EC2 api directly, as that can be slow at times.

Caveats
=======

When using the ``/instances`` or ``/instances/group`` endpoints with a format of ``dict``, the response is keyed by the ``Name`` tag of the ec2 instance. If you have multiple servers with the same value for the ``Name`` tag, this may result in "hidden" servers. If a server is not running, it will be hidden by default, otherwise it may overwrite the previous entry. This will both be logged in the ``haldane`` logging output *and* will be surfaced as ``meta.hidden_nodes``, regardless of the endpoint being used.