bioscons 1.1.0
Extends the scons build tool for reproducible research in bioinformatics.
Requires Python: >=3.5

==========
 bioscons
==========

This package extends the scons build tool for the construction of
reproducible workflows in bioinformatics.

Documentation is available on github: http://nhoffman.github.io/bioscons/

Background
==========

Why does SCons make sense for reproducible bioinformatics pipelines?

* SCons has a sophisticated mechanism for determining dependencies,
  meaning that re-running SCons will only re-execute steps needing
  updating.
* The definition of sources and targets (see example below) defines a
  dependency graph that supports parallelization of tasks.
* Most of the work of pipelines is done by external programs that are
  easy to execute with a shell-like syntax for commands.
* On the other hand, SCons also allows the execution of arbitrary
  python code in creating your script, and thus one can leverage the
  power of the python standard library, Biopython, NumPy, etc in your
  script.
* Rather than dealing with a mess of filenames, subsequent steps in an
  SCons build are expressed in terms of *file objects*
* Steps in the pipeline are implemented as *Commands* which implement
  a shell command or a python function in a way that consistently
  channels inputs into outputs
* Provides multiple mechanisms for cleanly executing isolated steps of
  the workflow (for example, by previewing commands to be executed
  using ``scons -n``, and pasting a single command directly into the
  shell)
* SCons validates files and can fail incrementally

scons example
-------------

Here's a simple but complete example of a build script demonstrating
the execution of two commands::

  vars = Variables()
  vars.Add('data', help='source data', default='./data')
  vars.Add('out', help='output directory', default='./output')

  env = Environment(variables=vars)

  alignment = env.Command(
      target='$out/seqs.aln.fasta',
      source='$data/seqs.fasta',
      action='muscle -in $SOURCE -out $TARGET'
      )

  tree = env.Command(
      target='$out/seqs.tre',
      source=alignment,
      action='FastTreeMP -nt -gtr $SOURCE > $TARGET'
  )

Here we have defined some variables for the build environment (ie, the
input and output directories), and constructed an object (``env``)
defining the execution environment. The output of the first command is
used as the input to another (thereby explicitly defining a dependency
between the two commands), and for the most part the names and paths
of the inputs and outputs are abstracted away with shell-like variable
substitution rules. It's easy to build pipelines involving complex
dependencies that nonetheless remain extremely easy to read.

So, what does ``bioscons`` provide?
-----------------------------------

Mainly, integration with the Slurm job scheduler via a subclass of the
SCons ``Environment`` object. For example, here's how to modify the
above example so that each job will be dispatched to a slurm queue
with a specified number of cores requested for each job::

  from bioscons.slurm import SlurmEnvironment

  vars = Variables()
  vars.Add('data', help='source data', default='./data')
  vars.Add('out', help='output directory', default='./output')

  env = SlurmEnvironment(variables=vars, use_cluster=True)

  alignment = env.Command(
      target='$out/seqs.aln.fasta',
      source='$data/seqs.fasta',
      action='muscle -in $SOURCE -out $TARGET',
      ncores=4
      )

  tree = env.Command(
      target='$out/seqs.tre',
      source=alignment,
      action='FastTreeMP -nt -gtr $SOURCE > $TARGET',
      ncores=10
  )

But ``bioscons`` also provides some additional utilities for creating
an inventory of targets, timing actions, modifying file paths. See
http://nhoffman.github.io/bioscons/ for complete package
documentation.

Installation
============

dependencies
------------

* Python 3.5+
* scons 2.4+

installation scenarios
----------------------

Various installation scenarios are possible involving different
combinations of system package installers, pip, and virtualenv vs
system installs. We will describe only the recommended configuration
here, although others are possible. Note that although ``bioscons``
*should* work with scons 2.4+, ``scons`` itself is only compatible
with python 3 in versions > 3.0.0

Install both scons and bioscons to a virtualenv
-----------------------------------------------

We strongly recommend installing both this package and ``scons`` to a
virtualenv rather than to your system due to idiosyncrasies in the
``scons`` installation script, and the fact that an older version of
``scons`` is likely to be installed by package managers. This option
is available using Python 3.5+

Start by creating a virtualenv::

  python3 -m venv bioscons-env

Due to some quirks in the ``scons`` installation process, you must
ensure that ``pip`` is the most recent version, and ``wheel`` is
installed::

  source bioscons-env/bin/activate
  pip install -U pip wheel
  pip install bioscons

Take care that pip corresponds to the intended version of the python
interpreter; a safer option may be to use ``pip3``.

installation from source (for development)
------------------------------------------

::

  https://github.com/nhoffman/bioscons.git
  cd bioscons
  python3 -m venv bioscons-env
  source bioscons-env/bin/activate
  pip install -U pip wheel
  pip install -e .
  pip install -r requirements.txt  # to run tests, build docs

Defining the execution environment for reproducible pipelines
=============================================================

When intending to run the version of ``scons`` installed to the
virtualenv, it is a good idea to include the following directive in
your ``SConstruct``::

  venv = os.environ.get('VIRTUAL_ENV')
  if not venv:
      sys.exit('--> an active virtualenv is required')

It is best to define the ``$PATH`` used to locate executables that are
used within your pipeline.

Monitoring Slurm tasks                                                                                                                                                                                                                                                                                                                                                                        
======================                                                                                                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                              
A useful way to monitor a slurm queue on a Linux system is to use watch::

 watch squeue

For more information on managing Slurm tasks and installing Slurm
on your system go to https://slurm.schedmd.com/documentation.html