pov-fabric-helpers 0.3
Fabric helpers we use at PoV
License: MIT

Fabric Helpers
==============

This is a collection of helpers we use in Fabric_ scripts.  They're primarily
intended to manage Ubuntu servers (12.04 LTS or 14.04 LTS).

.. _Fabric: http://www.fabfile.org/

.. contents::


Helpers (aka why would I want this?)
------------------------------------

APT packages:

- ``ensure_apt_not_outdated()`` - runs ``apt-get update`` at most once a day
- ``install_packages("vim screen build-essential")``
- ``install_missing_packages("vim screen build-essential")``
- ``if not package_installed('git'): ...``
- ``if not package_available('pov-admin-helpers'): ...``

User accounts:

- ``ensure_user("myusername")``

SSH:

- ``ensure_known_host("example.com ssh-rsa AAA....")``

Locales:

- ``ensure_locales("en", "lt")``

Files and directories:

- ``ensure_directory("/srv/data", mode=0o700)``
- ``upload_file('crontab', '/etc/cron.d/mycrontab')``
- ``generate_file('crontab.in', '/etc/cron.d/mycrontab', context, use_jinja=True)``
- ``download_file('/home/user/.ssh/authorized_keys', 'https://example.com/ssh.pubkey')``

GIT:

- ``git_clone("git@github.com:ProgrammersOfVilnius/project.git", "/opt/project")``
- ``git_update("/opt/project")``

PostgreSQL:

- ``ensure_postgresql_user("username")``
- ``ensure_postgresql_db("dbname", "owner")``

Apache:

- ``ensure_ssl_key(...)``
- ``install_apache_website('apache.conf.in', 'example.com', context, use_jinja=True, modules='ssl rewrite proxy_http')```

Postfix:

- ``install_postfix_virtual_table('virtual', '/etc/postfix/virtual.example.com')```
- ``make_postfix_public()``

Keeping a changelog in /root/Changelog (requires
/usr/sbin/new-changelog-entry from pov-admin-tools_)

- ``changelog("# Installing stuff")``
- ``changelog_append("# more stuff")``
- ``changelog_banner("Installing stuff")``
- ``run_and_changelog("apt-get install stuff")``

plus many other helpers have ``changelog`` and/or ``changelog_append``
arguments to invoke these implicitly.

.. _pov-admin-tools: https://github.com/ProgrammersOfVilnius/pov-admin-tools


Instance management API
-----------------------

All of my fabfiles can manage several *instances* of a particular service.
Externally this looks like ::

  fab instance1 task1 task2 instance2 task3

which executes Fabric tasks ``task1`` and ``task2`` on instance ``instance1``
and then executes ``task3`` on ``instance2``.

An instance defines various parameters, such as

- what server hosts it
- where on the filesystem it lives
- what Unix user IDs are used
- what database is used for this instance
- etc.

To facilitate this ``pov_fabric`` provides three things:

1. An ``Instance`` class that should be subclassed to provide your own instances

   .. code:: python

    from pov_fabric import Instance as BaseInstance

    class Instance(BaseInstance):
        def __init__(self, name, host, home='/opt/sentry', user='sentry',
                     dbname='sentry'):
            super(Instance, self).Instance.__init__(name, host)
            self.home = home
            self.user = user
            self.dbname = dbname

   and since that's a bit repetitive there's a helper

   .. code:: python

    from pov_fabric import Instance as BaseInstance

    Instance = BaseInstance.with_params(
        home='/opt/sentry',
        user='sentry',
        dbname='sentry',
    )

   which is equivalent to the original manual subclassing.

   (BTW you can also add parameters with no sensible default this way, e.g.
   ``BaseInstance.with_params(user=BaseInstance.REQUIRED)``.)

2. An ``Instance.define()`` class method that defines new instances and
   creates tasks for selecting them

   .. code:: python

    Instance.define(
        name='testing',
        host='root@vagrantbox',
    )
    Instance.define(
        name='production',
        host='server1.pov.lt',
    )
    Instance.define(
        name='staging',
        host='server1.pov.lt',
        home='/opt/sentry-staging',
        user='sentry-staging',
        dbname='sentry-staging',
    )

   (BTW you can also define aliases with ``Instance.define_alias('prod',
   'production')``.)

3. A ``get_instance()`` function that returns the currently selected instance
   (or aborts with an error if the user didn't select one)

   .. code:: python

    from pov_fabric import get_instance

    @task
    def look_around():
        instance = get_instance()
        with settings(host_string=instance.host):
            run('hostname')


Previously I used a slightly different command style ::

    fab task1:instance1 task2:instance1 task3:instance2

and this can still be supported if you write your tasks like this

.. code:: python

    @task
    def look_around(instance=None):
        instance = get_instance(instance)
        with settings(host_string=instance.host):
            run('hostname')

Be careful if you mix styles, e.g. ::

    fab instance1 task1 task2:instance2 task3

will run ``task1`` and ``task3`` on ``instance1`` and it will run ``task2`` on
``instance2``.


Usage
-----

Get the latest release from PyPI::

    pip install pov-fabric-helpers

and then import the helpers you want in your ``fabfile.py``

.. code:: python

    from fabric.api import ...
    from pov_fabric import ...


Usage as a git submodule
~~~~~~~~~~~~~~~~~~~~~~~~

You can add this repository as a git submodule

.. code:: bash

  cd ~/src/project
  git submodule add https://github.com/ProgrammersOfVilnius/pov-fabric-helpers

and in your ``fabfile.py`` add

.. code:: python

  sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'pov-fabric-helpers'))
  if not os.path.exists(os.path.join(sys.path[0], 'pov_fabric.py')):
      sys.exit("Please run 'git submodule update --init'.")
  from pov_fabric import ...


Testing Fabfiles with Vagrant
-----------------------------

I don't know about you, but I was never able to write a fabfile.py that worked
on the first try.  Vagrant_ was very useful for testing fabfiles without
destroying real servers in the process.  Here's how:

- Create a ``Vagrantfile`` somewhere with

  .. code:: ruby

    Vagrant.configure("2") do |config|
      config.vm.box = "ubuntu/precise64"  # Ubuntu 12.04
      config.vm.provider :virtualbox do |vb|
        vb.customize ["modifyvm", :id, "--memory", "1024"]
      end
    end

- Run ``vagrant up``

- Run ``vagrant ssh-config`` and copy the snippet to your ``~/.ssh/config``,
  but change the name to ``vagrantbox``, e.g. ::

    Host vagrantbox
      HostName 127.0.0.1
      User vagrant
      Port 2222
      UserKnownHostsFile /dev/null
      StrictHostKeyChecking no
      PasswordAuthentication no
      IdentityFile ~/.vagrant.d/insecure_private_key
      IdentitiesOnly yes
      LogLevel FATAL

- Test that ``ssh vagrantbox`` works

- In your ``fabfile.py`` create a testing instance

  .. code:: python

    Instance.define(
        name='testing',
        host='vagrant@vagrantbox',
        ...
    )

- Test with ``fab testing install`` etc.

.. _Vagrant: https://www.vagrantup.com/


Changelog
=========


0.3 (2016-09-11)
----------------

- ``register_host_key()`` now takes ``fingerprints`` so you can specify both
  MD5 and SHA256 fingerprints.

  Use either ``register_host_key(key, fingerprint=md5_fprint)`` or
  ``register_host_key(key, fingerprints=[md5_fprint, sha256_fprint])``.

- Low-level helper ``ssh_key_fingerprint()`` now takes ``force_md5`` so you
  can insist on MD5 instead of whatever OpenSSH gives you by default (which is
  SHA256 for modern OpenSSH).


0.2 (2015-08-06)
----------------

- New helpers:

  - ``git_update()``, ``register_host_key()``,
  - ``ensure_locales()``,
  - ``changelog_banner()``, ``run_and_changelog()``,
    ``has_new_changelog_message()``,
  - ``install_missing_packages()``, ``package_available()``,
  - ``upload_file()``, ``generate_file()``, ``ensure_directory()``,
    ``download_file()``,
  - ``install_postfix_virtual_table()``,
  - ``install_apache_website()``,
  - ``ensure_ssl_key()``.

- New optional arguments for existing helpers:

  - ``git_clone()`` now takes ``branch`` and ``changelog``.
  - ``ensure_user()`` now takes ``shell``, ``home``, ``create_home``, and
    ``changelog``.
  - ``install_packages()`` now takes ``changelog``.
  - ``changelog()`` now takes ``context``.
  - ``changelog_append()`` now takes ``context`` and ``optional``.
  - ``changelog_banner()`` now takes ``context`` and ``optional``.

- Increased safety:

  - all helpers check their arguments for unsafe shell metacharacters.
  - changelog() and friends quote the arguments correctly.

- Improved instance API:

  - allow ``str.format(**instance)`` (by making Instance a subclass of
    ``dict``).
  - allow instance aliases defined via ``Instance.define_alias(alias, name)``
    static method.

- Bugfixes:

  - ``ensure_postgresql_db()`` now works correctly on Ubuntu 14.04.
  - ``run_as_root`` now correctly handles ``env.host_string`` with no
    username part.

- New low-level helpers you're probably not interested in, unless you're
  writing your own helpers:

  - ``aslist()``, ``assert_shell_safe()``,
  - ``ssh_key_fingerprint()``,
  - ``render_jinja2()``, ``render_sinterp()``,
  - ``parse_git_repo()``,
  - ``generate_ssl_config()``, ``generate_ssl_key()``, ``generate_ssl_csr()``,
  - ``get_postfix_setting()``, ``parse_postfix_setting()``,
    ``add_postfix_virtual_map()``, ``add_postfix_setting()``,
  - ``run_as_root()``.


0.1 (2014-11-19)
----------------

- First public release.

- Helpers:

  - ``ensure_apt_not_outdated()``, ``package_installed()``,
    ``install_packages()``,
  - ``ensure_known_host()``, ``ensure_user()``,
  - ``git_clone()``,
  - ``ensure_postgresql_user()``, ``ensure_postgresql_db()``,
  - ``changelog()``, ``changelog_append()``.

- Instance API:

  - ``class Instance``, ``Instance.with_params()``,
    ``Instance.REQUIRED``, ``Instance.define()``.
  - ``instance._asdict()``.
  - ``get_instance()``.

- Low-level helpers you're probably not interested in, unless you're
  writing your own helpers:

  - ``asbool()``,
  - ``postgresql_user_exists()``, ``postgresql_db_exists()``.