asimtote 1.1.0
Compare network device configuration files using contextual structures

ASIMTOTE
========

`asimtote` is a program which will compare two network device configuration
files (a _source_ and _target_ configuration, or "from" and "to") and produce a
file of commands (the _difference_ configuration) which will reconfigure a live
device running the source configuration into the target configuration.

The philosophy is that a primary configuration is maintained separately from
the devices' own configurations, most likely generated through some sort of
build process, and it is necessary to reconfigure live devices to match it.

Quick start
-----------

The program is run by supplying the platform name and the filenames of the
source and target configurations.  For example:

    asimtote ios current/router-fenway.cfg new/router-fenway.cfg

... this will write, to standard output, the commands required to update the
configuration.  For example, if `current/router-fenway.cfg` contained:

    vlan 100
     name APPLE

    vlan 200
     name BANANA

... and `new/router-fenway.cfg` is:

    vlan 100
     name ORANGE

... then the resulting output would be:

    no vlan 200

    vlan 100
     name ORANGE

    end

A third argument of the output filename can be specified.  There are also
options to process multiple configurations in a single run.

Operation
---------

The program does this in three main steps:

* a _command parser_ to read in the source configuration files (source and
  target) and store then in an internal data structure (a Python dictionary)

* a _difference calculator_ compares the two configurations to determine the
  differences - what is _removed_ and what is added or _updated_

* a _converter_ examines the differences and generates the commands required
  to reconfigure the device

The system also supports the ability to define 'rules' which can specify that
parts of the configuration can be excluded or included from comparison.  This
is useful if an element of the configuration needs to be changed outside the
managed system to investigate an issue or handle something that is not
supported.

Platforms
---------

Currently, Cisco IOS is the only concrete platform and only a subset so far
(the comparator is still being tweaked to handle this all relatively
straightforwardly, before all the commands are implemented).  There are
currently some odd commands and edge cases which are awkward to handle without
some improvements to the core process.

Currently, all this works using standard text-based configuration files.
Ideally, this would use an API to avoid having to parse legacy configuration
information and effect the changes, but text-based files were all that were
available at the time the program was started, and the use of text-based files
does have the advantage that it is easy for a network engineer to see what is
going to be changed.


The name
--------

This package was formerly known as `net-contextdiff` but has been renamed as it
is expected to drop the indented configuration difference mechansim (still
using context, but not via indenting).

The new name `asimtote` is a backronym for something like 'Autonomous System
Internetwork Management ...' but the rest has not yet been completed.  It's
similarity to 'asymptote' is deliberate as it aims to get a configuration into
a desired state but will probably never ever get there.

Command line
------------

The following command line options are available, which can customise its
operation, as well as a large number of options useful to debug the parsing
and conversion process:

    usage: asimtote [-h] [-r FILENAME] [-s PATH] [-e] [-E] [-q] [-O] [-R] [-C]
                    [-D] [-A] [-U] [-S [key ...]] [-P] [-V] [-v]
                    {ios} old [new] [output] [devicename ...]

Positional arguments:

* `{ios}` - the platform for the configuration files.

* `<old>` - the source (old / "from") configuration filename; `%` can be used
  to substitue in the name of the device, if multiple device configurations are
  being compared with the `devicename` argument.

* `<new>` - the target (new / "to") configuration filename; `%` can be used to
  substitute in the name of the device.

* `<output>` - write the conversion commands to the specified file, instead of
  standard output; `%` can be used to substitute in the name of the device.

* `<devicename> ...` - if specified, process the configurations for the listed
  device(s), subsituting the name of each one into the old, new and output
  options, where a `%` sign is used.

Optional arguments:

* `-h`, `--help` - show command line usage help

* `-r <filename>`, `--rules-tree-filename <filename>` - specifies the filename
  of a file containing the _rules tree_, described below.

* `-s <path>`, `--rule-spec <path>` - overrides the normal processing order of
  rules from the rules tree - by default it is:

      !rules:exclude:ALL
      !rules:exclude:%
      rules:include:ALL
      rules:include:%

  ... which specifies that the entry `rules.exclude.ALL` are to be excluded
  (the leading `!`), then the device-specific rules (`%` matches the device
  name), then only the rules in `rules.include.ALL` are to be included (no
  leading `!`), then the device-specific rules - this option is typically only
  useful during debugging.

* `-e`, `--explain` - add comments to the output, explaining what the path into
  the configuration dictionary was for each difference found.

* `-E`, `--reverse` - swap the source and target configurations, converting the
  target back into the source, which should undo a particular reconfiguration.

* `-q`, `--quiet` - when comparing the configuration for multiple devices,
  don't print out the name of each device, as it is processed.

* `-O`, `--no-output` - don't write out configuration changes to standard
  output or the output file - this is useful when testing parsing, comparison
  or conversion and the debug output shouldn't be cluttered with normal output.

* `-R`, `--dump-rules` - print the rules processed for including and excluding
  parts of the configuration; each call increases the verbosity of the output:

  * 1 - dump the complete rules tree (read with `-r`) and specifications
    (defined with `-s`).

  * 2 - in addition, print the rules matched by each specification.

* `-C`, `--dump-config` - print the source and target configuration
  dictionaries after parsing.

* `-D`, `--dump-diff` - print the remove and update differences, when comparing
  the configurations.

* `-A`, `--dump-all-diff` - print the differences across all configurations as
  a YAML tree.

* `-U`, `--dump-exclude-rules` - this is the same as `-A` but the format is
  slightly tweaked to incude the `exclude:<devicename>` levels so the output
  can be more easily included in a rules tree file read with `-r`.

* `-S <key> ...`, `--subtree-dump-filter <key> ...` - limits the output from
  the debug options to just the specified path into the configuration tree -
  this is useful when investigating just a part of the configuration.

* `-P`, `--debug-parser` - each occurrence of this option increases the
  verbosity of debugging on the operation of the configuration parser.

  * 1 - print matched command parser class including regular expression.
  * 2 - print command parsers that were checked but did not match.
  * 3 - print the matched fields in the command, passed as arguments to the
        method which stores the configuration in the dictionary.

* `-V`, `--debug-convert` - each occurrence of this option increases the
  verbosity of debugging on the operation of the difference converters:

  * 1 - print the matching converter name and action (update/remove/etc.) only.
  * 2 - print the conversion class and generated commands for each conversion.
  * 3 - print the old/new/remove/update arguments to the converter.
  * 4 - include any skipped converters (e.g. ones in contexts that were removed
        so weren't necessary).
  * 5 - include converters that didn't match any changes.

* `-v`, `--version` - show the program version number and exit

Rules tree
----------

If a difference between the inventory and live network needs to persist for a
period of time, the file rules-tree in the inventory/ directory can be used to
exempt parts of the live configuration (on the devices themselves) from being
changes by net-config.

The format of rules-tree is several levels in the form:

    rules:
      <action>:
        <device>:
          <item>: ...

The keys are as follows:

* `rules` is the literal string and identifies the ruleset

* `action` is either `exclude` or `include`: `exclude`s are processed, then
   `include`s (so include will only include things that haven’t already been
   excluded), although this order can be changed with the `-s` option:

   * `exclude` identifies things that are to be exempted from generating
     differences.  Normally this will be action to use.

   * `include` specifies that only the things listed are to be included.

* `device` is the name of the device to which the rule applies, or `ALL` for
  all devices

* `item` is the element of the configuration dictionary to which the rule
  applies - it can be something at the toplevel (e.g. interface for all
  interfaces) or something more specific - e.g.:

      interface:
        Vl789:
          - ip-access-group

Note that, as with any YAML file, no warning will be generated if a key at a
particular level is duplicated, but the behaviour is undefined.  For example:

    rules:
      exclude:
        dist-west:
          - interface
        dist-mwest:
          - hostname
        dist-west:
          - hostname

... this will specify `dist-west` twice and only one will likely take effect.

### Element specification

When specifying a configuration element to exclude or include, the path into
the configuration dictionary must be specified.  The dictionary is that built
by `asimtote`, when it parses and compares the configurations and is often
related to a name of a command used by the network device platform but will be
in a more hierarchical form.

To determine what the key should be used, `asimtote` can be run manually with
the `-I` option and a configuration containing the desired element supplied.
For example (note there will be a lot of output):

    $ asimtote -C ios current-configs/dist-west.cfg | less
    ...
    interface:
      Fo3/10:
        arp-timeout: 900
        description: CORE-MILL
        ip-address: 131.111.6.114 255.255.255.252
        ...

To create an rule excluding the description of this interface from being
changed, the following rule can be specified:

    rules:
      exclude:
        dist-west:
          interface:
            Fo3/10:
              - description

To exempt the entirety of interface `Fo3/10` from being reconfigured, the
higher level configuration element can be specified, e.g.:

    rules:
      exclude:
        dist-west:
          interface:
            - Fo3/10

To aid in the writing the rules, asimtote can be run manually with the `-U`
option:

    $ asimtote -O -U ios current-configs/%.cfg new-configs/%.cfg - dist-west
    dist-west
    rules:
      exclude:
        ALL:
          interface:
            Fo3/10:
              description: {}

... this will print out the rules to exclude the changes (and the name of the
router, which should be ignored).  These rules can be cut and pasted into the
`rules-tree` file, although take care to higher level elements are not
duplicated.

Note that the above is functionally equivalent to:

    rules:
      exclude:
        ALL:
          interface:
            Fo3/10:
              - description

Adding functionality
--------------------

The `Developing.md` file contains an overview about how the code works, and how additional functionality can be added.

The code itself has more complete, detailed documentation about how individual
classes and methods work.

Future developements
--------------------

Longer term, this script may be made unnecessary by other tools but, if it
remains, will probably be parsing configurations via something like REST and
JSON, rather than by reading legacy text configuration files.  It would also be
making changes using similar calls rather than generating legacy text
conversions.  The problem of making changes in the correct order and committing
them may still remain, though.

It would also be desirable to separate the parser as this could be useful
elsewhere.  At the moment, however, it's so intertwined with the converter this
would be tedious to maintain and keep in sync.  It should probably be made
easily usable as a separate component, however.