nuage-metroae-config 1.1.0
Template-based configuration tool for Nuage Networks VSD
Requires Python: >=2.7

# Nuage MetroAE Configuration

Nuage MetroAE configuration template engine.

Version 1.1.0

This tool reads JSON or Yaml files of templates and user-data to
write a configuration to a VSD or to revert (remove) said configuration.

## Overview

MetroAE config is a Python-based engine which can apply configuration to VSDs
via templates.  The templates provide an abstraction around the VSD
configuration model simplifying and validating the required data.  Data is
provided by the user in the form of Yaml or JSON files and is transformed and
applied to the VSD through the tool's templates.  The required data and proper
format for each template is defined by standardized JSON schema specifications.

## Installation

### Via Docker Container

There are several modes of operation for MetroAE config.  For most users, the
tool should be consumed via the MetroAE Docker container.  Using this method
ensures that all of the libraries are properly installed and the `metroae`
command-line tool can be utilized for config (as well as deploy).  Instructions
for the Docker container installation are available at:

https://github.com/nuagenetworks/nuage-metroae/blob/master/Documentation/DOCKER.md

If using the Docker container, all remaining installation instructions can be
skipped.

### As Python Library

The MetroAE config tool is completely implemented as a Python library.  As such,
the functionality of the tool can be integrated into other Python tools or
libraries through the use of the APIs.  To do this, the library can be installed
via Python pip package from pypi.org:

```
pip install nuage-metroae-config
```

The pip installation of the package will pull in all dependent libraries
automatically.  The following section of installing from Github can be skipped.
The use of the Library APIs is documented in the `Python Library API` section.

### From Github Python Source

Using the MetroAE config tool without the Docker container can be accomplished
by pulling the source from Github.  You may want to use this method if you are
comfortable working with Python environments and don't want the overhead of a
container.   The instructions for this case is described in the next section.
If using this method, you would be responsible for installing all of the
necessary prerequiste libraries and to obtain the VSD API specs and templates.

#### Installation of MetroAE Config from Github

This section describes how to install the MetroAE config tool from the Github
repo.  These steps will be necessary to use the metroae_config command-line
tool directly.  If using the MetroAE Docker container, this section should be
skipped.

The primary requirements for running MetroAE config are:
1. Installation of Python and the necessary packages.
2. Installation of the config engine python libraries
3. Access to the Configuration templates
4. Access to the VSD API Specification

#### Installation of Python and required packages.

Most linux distributions come with python bundled into the operating system.
Python 2.7 is required. Currently Python 3.0 is not supported.

To verify that python is available we can do the following:

*Ubuntu*

```
root@ubuntu:~# python --version
Python 2.7.5
```

*RHEL/Centos*

```
[root@rhel]# python --version
Python 2.7.5
```

To manage the python package installation we will use PIP. Install PIP via
apt-get or yum:

*Ubuntu*

```
root@ubuntu:~# apt-get install python-pip
...
```

*RHEL/Centos*

```
[root@rhel]# yum install python-pip
...
```

Most packages required by MetroAE config are available as part of the base
python functionality. However we will need to install the following:

- Bambou
- Jinja2 (min version 2.10)
- PyYAML
- lark-parser

Additional packages for unit-test

- mock
- pytest
- requests
- requests-mock
- lark_parser

The following packages are installed via the same method on both Ubuntu and RHEL/Centos.

*Ubuntu and RHEL/Centos*

```
[root@rhel]# pip install -r requirements.txt
...
```

#### Installation of configuration engine

Currently MetroAE config is available on github and will be installed via git
clone. Check the MetroAE home page for latest master branch location.

First we need to install git.

*Ubuntu*

```
root@ubuntu:~# apt-get install git
```

*RHEL/Centos*

```
[root@rhel]# yum install git
```

The repository is available via Github:

```
https://github.com/nuagenetworks/nuage-metroae-config
```

Depending on the authentication method with github, ie. SSH Key, or
Username/Password the git clone command line may change.


For username/password authentication

*Ubuntu and RHEL/Centos*

```
[root@rhel]# export GIT_SSL_NO_VERIFY=false
[root@rhel]# git clone https://github.com/nuagenetworks/nuage-metroae-config.git
Cloning into 'levistate'...
Username for 'https://github.com/nuagenetworks/nuage-metroae-config.git': sfiddian
Password for 'https://sfiddian@github.mv.usa.alcatel.com':
remote: Counting objects: 619, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 619 (delta 1), reused 0 (delta 0), pack-reused 610
Receiving objects: 100% (619/619), 178.01 KiB | 0 bytes/s, done.
Resolving deltas: 100% (398/398), done.
```


#### Installation of the VSD API Specifications

MetroAE Config requires a description of the VSD API to create, read, update and
delete the template contents into the VSD. We do this via reading the published
VSD API Specification. This specificaiton is opensourced and is available on
the public Nuage Networks github repository.



*Ubuntu and RHEL/Centos*

```
[root@rhel]# git clone https://github.com/nuagenetworks/vsd-api-specifications.git
Cloning into 'vsd-api-specifications'...
remote: Counting objects: 23428, done.
remote: Total 23428 (delta 0), reused 0 (delta 0), pack-reused 23428
Receiving objects: 100% (23428/23428), 4.98 MiB | 1013.00 KiB/s, done.
Resolving deltas: 100% (18163/18163), done.
```

## Command-line Tool

When using the MetroAE Docker container, all commands are accessed through the
`metroae` command-line tool as:

    metroae config ...

If you are using the Python source obtained through Github, then any reference
to `metroae config` should substitute the equivalent Python source tool:

    python metroae_config.py ...


## Parameters

MetroAE Config command-line tool usage:

    usage: metroae config [-h]
                          {create,revert,validate,list,schema,example,upgrade-templates,version,help}
                          ...

    Version 1.0 - This tool reads JSON or Yaml files of templates and user-data to
    write a configuration to a VSD or to revert (remove) said configuration. See
    README.md for more.

    positional arguments:
      {create,revert,validate,list,schema,example,upgrade-templates,version,help}

    optional arguments:
      -h, --help            show this help message and exit


    usage: metroae config create [-h] [-tp TEMPLATE_PATH] [--version]
                                 [-sp SPEC_PATH] [-dp DATA_PATH] [-d DATA]
                                 [-v VSD_URL] [-u USERNAME] [-p PASSWORD]
                                 [-e ENTERPRISE] [-lg]
                                 [datafiles [datafiles ...]]

    positional arguments:
      datafiles             Optional datafile

    optional arguments:
      -h, --help            show this help message and exit
      -tp TEMPLATE_PATH, --template_path TEMPLATE_PATH
                            Path containing template files. Can also set using
                            environment variable TEMPLATE_PATH
      --version             Displays version information
      -sp SPEC_PATH, --spec_path SPEC_PATH
                            Path containing object specifications. Can also set
                            using environment variable VSD_SPECIFICATIONS_PATH
      -dp DATA_PATH, --data_path DATA_PATH
                            Path containing user data. Can also set using
                            environment variable USER_DATA_PATH
      -d DATA, --data DATA  Specify user data as key=value
      -v VSD_URL, --vsd_url VSD_URL
                            URL to VSD REST API. Can also set using environment
                            variable VSD_URL
      -u USERNAME, --username USERNAME
                            Username for VSD. Can also set using environment
                            variable VSD_USERNAME
      -p PASSWORD, --password PASSWORD
                            Password for VSD. Can also set using environment
                            variable VSD_PASSWORD
      -e ENTERPRISE, --enterprise ENTERPRISE
                            Enterprise for VSD. Can also set using environment
                            variable VSD_ENTERPRISE
      -lg, --logs           Show logs after run


## Example Usage

Apply enterprise, domain and ACLs to VSD.

    $ metroae config create -tp sample/templates -sp ~/vsd-api-specifications -v https://localhost:8443 sample/user_data/acls.yaml

    Configuration
        Enterprise
            name = 'test_enterprise'
            DomainTemplate
                name = 'template_public'
                [store id to name domain_template_id]
            Domain
                name = 'public'
                templateID = [retrieve domain_template_id (DomainTemplate:id)]
                IngressACLTemplate
                    priority = 100
                    defaultAllowNonIP = False
                    allowAddressSpoof = False
                    name = 'test_acl'
                    defaultAllowIP = True
                    IngressACLEntryTemplate
                        networkID = ''
                        stateful = True
                        protocol = 6
                        description = 'Test ACL'
                        etherType = '0x0800'
                        statsLoggingEnabled = True
                        DSCP = '*'
                        priority = 200
                        action = 'FORWARD'
                        locationID = ''
                        destinationPort = '*'
                        locationType = 'ANY'
                        sourcePort = 80
                        networkType = 'ANY'
                        flowLoggingEnabled = True
                EgressACLTemplate
                    priority = 100
                    defaultInstallACLImplicitRules = True
                    defaultAllowNonIP = False
                    name = 'test_acl'
                    defaultAllowIP = True
                    EgressACLEntryTemplate
                        networkID = ''
                        stateful = True
                        protocol = 6
                        description = 'Test ACL'
                        etherType = '0x0800'
                        statsLoggingEnabled = True
                        DSCP = '*'
                        priority = 200
                        action = 'FORWARD'
                        locationID = ''
                        destinationPort = '*'
                        locationType = 'ANY'
                        sourcePort = 80
                        networkType = 'ANY'
                        flowLoggingEnabled = True

Revert (remove) objects configured during application, use -r option

    $ metroae config revert -tp sample/templates -sp ~/vsd-api-specifications sample/user_data/acls.yaml

## User Data

The templates within the MetroAE config tool are applied using data provided by the
user.  Each set of data values provided allows the creation of an instance of
configuration on the VSD.  The required format for the data of each template is
defined by a JSON schema describing the required fields, types and other
constraints on the data.

### File Format

The user data for MetroAE config can be in either Yaml or JSON format.  Each entry
in the file defines a template and the value sets to use against that template
to instantiate configuration.

    - template: Enterprise
      values:
        - enterprise_name: my_first_enterprise
          description: The first

        - enterprise_name: my_second_enterprise
          description: The second

In this example, two data sets are provided for the Enterprise template and the
result will be that the two enterprises will be created on the VSD.  Only the
enterprise_name is required, but description can be provided optionally.  The
JSON schema for the Enterprise template defines the data format.

### Children

Let's now add domains to the enterprise objects.  In the Domain template, note
that in addition to a domain_name, it also needs the enterprise_name to be
specified so that the domain can be associated with the correct enterprise.  We
could define our user data as follows:

    - template: Enterprise
      values:
        - enterprise_name: my_first_enterprise
          description: The first

        - enterprise_name: my_second_enterprise
          description: The second

    - template: Domain
      values:
        - enterprise_name: my_first_enterprise
          description: The first
          domain_name: my_first_domain

        - enterprise_name: my_second_enterprise
          description: The second
          domain_name: my_second_domain

This user data is fully valid and will create two enterprises each with one
domain.  However, we are duplicating the enterprise names and it would be
tedious and error prone if the names needed to change.  In order to simplify
nested data, children can be utilized as follows:

    - template: Enterprise
      values:
        - enterprise_name: my_first_enterprise
          description: The first
      children:
        - template: Domain
          values:
            - domain_name: my_first_domain

    - template: Enterprise
      values:
        - enterprise_name: my_second_enterprise
          description: The second
      children:
        - template: Domain
          values:
            - domain_name: my_second_domain
              description: Overriden description

In this case, each Domain is a child of a parent Enterprise.  As such, all of
the values from the parent are inherited in the children.  The children
domains automatically inherit the enterprise_name of the parent.  This also
applies for the description.

Note that values can be overriden if necessary as with the description of the second
domain.  The most specific value takes precendence.  To prevent ambiguity, any
template with child templates can only define one set of values.

### Groups

Another mechanism for reducing duplication in data is by using groups.  A group
entry is very similar to a template, except that no actual template is
instantiated.  Thus, the group defines a stand-alone data set.  Groups are
useful when common data sets need to be defined and referenced elsewhere.

    - group: first
      values:
        - enterprise_name: my_first_enterprise
          domain_name: my_first_domain

    - group: second
      values:
        - enterprise_name: my_second_enterprise
          domain_name: my_second_domain

    - group: ssh
      values:
        - protocol: tcp
          port: 22

    - group: html
      values:
        - protocol: tcp
          port: 80

    - template: Acl
      values:
        - acl_name: http_acl
          $group_domain: first
          $group_traffic: http
          action: permit

        - acl_name: ssh_acl
          $group_domain: second
          $group_traffic: ssh
          action: deny

In the above example, data sets for domains and traffic types are defined as
groups.  These are being applied to the http_acl and ssh_acl templates using
$group fields and referencing by group name.  The template will inherit all of
the values from any group that is referenced.  The result in this case is that
http_acl is created in the first domain and permits http traffic (tcp port 80)
and the ssh_acl is created in the second domain and denies ssh traffic (tcp
port 22).

Group definitions can be children or have children.  However, they should only
define one set of values.  The field name for group references must start with
the string "$group", but can have any suffix following it.  The purpose of the
suffix is to allow multiple group references without name collision.  Although
any suffix can be specified, it is recommended to choose one that is
descriptive of the reference meaning.

### Field and Value Lists

An alternative way of specifying values is by using a field list and set of value
lists.  This format is similar to Comma Separated Values (CSV) data and
can efficiently specify a large amount of data.

    - template: Acl
      fields: ['acl_name', '$group_domain', 'protocol', 'port', 'action']
      values:
          -   ['acl1',     'first',         'tcp',      22,     'deny']
          -   ['acl2',     'first',         'udp',      5000,   'permit']
          -   ['acl3',     'second',        'tcp',      80,     'deny']
          -   ['acl4',     'second',        'udp',      5002,   'permit']

A fields list must be specified to define the field name for each position of
the value lists.  The values data sets must be specified as lists with the same
length as the fields list.

## Listing Templates

The templates that have been loaded into the MetroAE config tool can be listed
with the following:

    $ metroae config -tp sample/templates --list

    Domain
    Enterprise
    Subnet
    Zone

## Generating User Data Examples

An example of user data for any template can be provided using the following:

    metroae config example -tp sample/templates Domain

    # First template set - Create a L3 Domain
    - template: Domain
      values:
        - enterprise_name: ""                      # (reference)
          domain_name: ""                          # (string)
          description: ""                          # (opt string)
          underlay_enabled: enabled                # (['enabled', 'disabled', 'inherited'])
          address_translation: enabled             # (['enabled', 'disabled', 'inherited'])

## Generating JSON Schemas

A JSON schema can be generated for the user data required for any template.
These schemas conform to the json-schema.org standard specification:

    metroae config schema -tp sample/templates Domain

    {
      "title": "Schema validator for Nuage Metro config template Domain",
      "$id": "urn:nuage-metro:config:template:domain",
      "required": [
        "enterprise_name",
        "domain_name",
        "underlay_enabled",
        "address_translation"
      ],
      "$schema": "http://json-schema.org/draft-04/schema#",
      "type": "object",
      "properties": {
        "underlay_enabled": {
          "enum": [
            "enabled",
            "disabled",
            "inherited"
          ],
          "title": "Underlay enabled"
        },
        "address_translation": {
          "enum": [
            "enabled",
            "disabled",
            "inherited"
          ],
          "title": "Address translation"
        },
        "domain_name": {
          "type": "string",
          "title": "Domain name"
        },
        "enterprise_name": {
          "type": "string",
          "title": "Enterprise name"
        },
        "description": {
          "type": "string",
          "title": "Description"
        }
      }
    }

## Query Tool

### Description

MetroAE config can be used as a generic data querying tool to extract
information from devices.  At the time of this writing VSD and ES are
supported.  The tool uses a simple language to describe the data that is to be
retrieved.  The data will be displayed to the screen in a YAML like format, but
it could (in addition) be written to a file.  The results are also returned as
a Python list when using query tool as a library.

### Usage

The query tool is used by specifying "query" as the action to `metroae_config`.
The parameters are mostly the same as other actions supported by
`metroae_config`.  The VSD specification path `-sp` is required if querying a
VSD.  The other VSD parameters (i.e. user, pass, URL, etc) also apply for
query in the same way as configuration creation actions.

The query tool also supports queries to ElasticSearch (ES).  In this case, a
new `-es` parameter has been added to specify the address of the ES to query.

The `query language` can be specified to the tool directly on the command-line
for easy manual use using the `-q` query parameter.  An example follows:

    metroae config query -v https://localhost:8443 -q 'enterprise.name'

    Device: Nuage Networks VSD 20.5.1
    - Shared Infrastructure
    - public
    - private
    >>> All actions successfully applied

Multiple queries can be specified on a single line using semicolon `;`:

    metroae config query -v https://localhost:8443 -q 'apps = SAASApplicationType.name; ents = enterprise.name'

    Device: Nuage Networks VSD 20.5.1
    apps:
    - Office365
    - SalesForce
    - WebEx
    ents:
    - Shared Infrastructure
    - public
    - private
    >>> All actions successfully applied

The language definition for query tool can also be specifed in a file to be
read.  One or more of these files (or a directory containing .query files) can
be specifed on the command-line for the tool to read and execute each file.
Newlines in files separate commands.  However, semicolons are still
supported in files as well.

    example.query:

    apps = SAASApplicationType.name
    ents = enterprise.name

    metroae config query -v https://localhost:8443 example.query

    Device: Nuage Networks VSD 20.5.1
    apps:
    - Office365
    - SalesForce
    - WebEx
    ents:
    - Shared Infrastructure
    - public
    - private
    >>> All actions successfully applied

### Language Definition

The query tool uses a `query language` to specify the data to be reteived from
devices.  This language is intended to be simple to read and use and is easily
extendable.

#### General Formatting

Each query to be performed is specified one per-line.  Alternatively and
equivalently, semicolons `;` can be used to separate multiple queries on a
single line.  Any query beginning with a hash `#` indicates a comment and will
be ignored.  Outside of strings, any whitespace is ignored.

#### Data Retrieval

Data retrieval commands are specified by providing nested object names
separated by dot `.` with the attribute as the last identifier.  A retrieval
traverses all instances of each level of the specified object tree and returns
the attribute value for every instance found.

The result is always a flat list of the attribute values.  Any heirarchy is
flattened.  If no results are found or there is an error (such as missing child
object) then an empty list is returned.  With this procedure, processing
results is consistent as there will always be a list.

For VSD:

    Enterprise.Domain.Zone.Subnet.name

The above (reading from right to left) returns a list of the names from all
subnets under all zones under all domains under all enterprises.  For VSD
queries, each object is a configuration object specified by entity name.  The
names here are case-insensitve although note that case may matter for other
devices (like ES).

For ElasticSearch:

    nuage_sysmon.disks.available

The above (reading from right to left) returns the available values for all
disks under the nuage_sysmon index.  For ES, the first object is the ES index
and then subsequent object names traverse the JSON objects stored in each ES
record.  Since ES encodes using JSON, the object names and attribures are all
case-sensitive as required by the format.

#### Attributes

The last identifier on a data retrieval is the attribute.  It determines which
value to extract from the objects traversed.  In previous examples, a single
value was extracted, however multiple attributes can be gathered by specifying
a comma-separated list in curly braces `{}`.  When retrieving multiple
attributes, the result becomes a list of dictionaries with key/value pairs
for each of the attributes.  A star `*` can be used to get all available
attributes.

Single attribute:

    Enterprise.name

    - Shared Infrastructure
    - public
    - private

Multiple attributes:

    Enterprise.{name,id}

    - name: Shared Infrastructure
      id: abcd-0001
    - name: public
      id: abcd-0002
    - name: private
      id: abcd-0003

All attributes:

    Enterprise.{*}

    - name: Shared Infrastructure
      id: abcd-0001
      bgpenabled: false
      dhcpleaseinterval: 24
      (Whole bunch of stuff omitted for brevity...)
    - name: public
      id: abcd-0002
      bgpenabled: false
      dhcpleaseinterval: 24
      (Whole bunch of stuff omitted for brevity...)
    - name: private
      id: abcd-0003
      bgpenabled: false
      dhcpleaseinterval: 24
      (Whole bunch of stuff omitted for brevity...)

#### Filters

Filters can be applied to each object of the data retrieval.  Filters are
enclosed by square brackets `[]` and are separated by and `&` characters.  The
choice of `&` as a separator enforces that filters are combined together as
logical AND operations.

The filter applies only to the level where it is specified.  The filters
manipulate the results from the level, usually by removing unwanted records.
Any subsequent object levels would operate on the filtered list.  For example:

    Enterprise[name=public].domain.name

The filter on enterprise will limit enterprise results to only a single
enterprise object (with name `public`).  However, let's say the `public`
enterprise had 10 domains.  Then the 10 name results for those domains would be
returned.

##### Range Filter

A range filter can be used to limit the start and max number of results at the
level.  The filter has a start index and an end index separated by colon `:`.
Indexes are 0-based positions of the list.  Thus 0 is the first and 4 would be
the 5th item in the list.  The items from the query are returned starting at
the start index and ending one before the end index.  Any range out of bounds
of the results are omitted.  Thus if the start index is after the end of the
results, an empty list is returned.

If the start index is omitted, a value of 0 is assumed (start at beginning).
If the end index is omitted, then the entire remainder of the list is returned.

A single index (without colon specified) can be used to return the single
item from the results at the specified index.  If this is out of bounds of the
results, an empty list is returned.

Negative indicies are allowed and count from the end of the list.  Thus -1 is
the last item and -5 is the 5th from last.

The behavior of the range filter is exactly the same as Python list slicing.

    Note that ES index queries do not support negative range indicies.  Use a
    reverse sort and positive indicies instead.

Examples using enterpises: `e1, e2, e3, e4, e5`

    enterprise[1:5].name
    - e2
    - e3
    - e4

    enterprise[1:99].name
    - e2
    - e3
    - e4
    - e5

    enterprise[:5].name
    - e1
    - e2
    - e3
    - e4

    enterprise[3:].name
    - e4
    - e5

    enterprise[99:].name
    []

    enterprise[3].name
    - e4

    enterprise[99].name
    []

    enterprise[-3:-1].name
    - e3
    - e4

##### Attribute Filter

Object results can be filtered by attribute values.  Only results where the
field value of the object matches the filter value will be returned.  The
format for these filters is the field name equal `=` value.  A list of values
can be provided enclosed in square brackets `[]` and separated by comma `,`.
This denotes that any result matching any of the list values will be returned.

Given these enterprises:

    - name: e1
      BGPEnabled: true
    - name: e2
      BGPEnabled: false
    - name: e3
      BGPEnabled: true

The following examples produce:

    enterprise[BGPEnabled=true].name
    - e1
    - e3

    enterprise[name=[e2, e3]].name
    - e2
    - e3

    enterprise[name=[e2, e3] & BGPEnabled=false].name
    - e2

    enterprise[name=[e1, e3] & BGPEnabled=false].name
    []

##### Sort Filter

The sort filter can be used to sort the results by an object field.  Note that
the sorting occurs only at the level where it is specified.  If there are
child objects under a sort fitler the children would not be sorted, but they
would be queried in the parent's sorted order.

Sorting is specified by the `%sort=` keyword followed by the field name of the
object to sort by.  There is also `%sort_desc=` to sort in descending order.

Given these enterprises:

    - name: e3
      dhcpleaseinterval: 1
    - name: e1
      dhcpleaseinterval: 2
    - name: e2
      dhcpleaseinterval: 3

The following examples produce:

    enterprise[%sort=name].name
    - e1
    - e2
    - e3

    enterprise[%sort=DHCPLeaseInterval].name
    - e3
    - e1
    - e2

    enterprise[%sort_desc=name].name
    - e3
    - e2
    - e1

##### Group Filter

A group filter can be specified to combine together results by the value of
another field.  This makes it easier to identify subsets of results given that
the query tool collapses them all into flat lists.

The group filter is specified by `%group=` followed by the field name to group
by.  When using groups, results are returned as a list of pairs.  The first
item of the pair is the group field value and the second is the list of results
matching the group field value.

For example, normally results are flattened and thus it is difficult to
determine which domain goes with which enterprise:

    enterprise.domain.name

    - domain1
    - domain2
    - domain3
    - domain4

However, using grouping the relationships can be determined:

    enterprise[%group=name].domain.name

    - - public
      - - domain1
        - domain2
        - domain3
    - - private
      - - domain4
    - - Shared Infrastructure
      - []

#### Variables

Results can be stored in variables for identification and use in later queries.
Variables are restricted to C-like names which cannot start with a number and
can contain capital letters, lower case letters, numbers or underscore `_`.

Variable assignments are performed by specifying the variable name then equal
`=` then the expression or data type to store in the variable.  The output of
assigments changes to a dictionary.

    enterprise_names = enterprise.name

    enterprise_names:
    - public
    - private
    - Shared Infrastructure

Variables are dynamically typed and can contain any data type.  More
information about data types will be provided in a later section.  Variables
can be dereferenced in later queries by specifying dollar `$` and the variable
name.  Variables that have been dereferenced but not yet defined will cause an
error.

Variables can be filtered:

    enterprises = enterprise.{name,id}

    enterprises:
    - name: public
      id: abcd-0001
    - name: private
      id: abcd-0002

    $enterprises[%sort=name]

    - name: private
      id: abcd-0002
    - name: public
      id: abcd-0001

    $enterprises[%sort=name].id

    - abcd-0002
    - abcd-0001

Variables can also be used as the values for filters:

    enterprises = enterprise.name

    enterprises:
    - e1
    - e2
    - e3
    - e4

    start = 2

    start: 2

    $enterprises[$start:]
    - e3
    - e4

    name = "e2"

    name: e2

    enterprise[name=$name].{name,id}
    - name: e2
      id: abcd-0002

Variable values can be specified on the command-line using the `-d` option.
The format is variable name `=` value. i.e. `-d name=enterprise1`.  When
variables are defined on the command-line, they override any assignment
within queries.  This allows defaults to be set inside query files and `-d` to
override with a specific value.

#### Data Types

The query language defines various data types to be specified.  Variables are
dynamically typed and can accept any of these types.

Strings are defined enclosed with single `'` or double `"` quotes.  Multi-line
blocks can be defined using triple single `'''` or double `"""` quotes.

    string = 'this is a string'
    string = "another has ' in it"
    multiline = """
    this has
    multiple lines
    """

Numbers can be specified as positive or negative integers.  Floating point
numbers are not supported.

    integer = 42
    negative = -4

For Booleans, `true` specifies true and `false` specifies false.

    is_true = true
    is_false = false

Lists can be formed using square brackets `[]` and separating items by comma
`,`.  Lists can contain any of the above data types even in combinations.
However, lists of lists are not supported.

    enterprise_names = ['e1', 'e2', 'e3']

#### Functions

There are a set of functions that can operate on expressions such as data
retrieval or variables.  These come in the form of function name followed by
the expression enclosed in parens `()`.

The `count` function returns the number of items in a list.  An error is
produced if the expression provided to count is not a list.

    names = enterprise.name

    names:
    - e1
    - e2
    - e3

    count($names)
    3

The `reverse` function returns the items of a list in backward order.  An error
is produced if the expression provided to reverse is not a list.

    reverse(enterprise.name)
    - e3
    - e2
    - e1

#### Combine

There is a `combine` operator which combines expressions together depending on
the data types.  The format is expression1 plus `+` expression2.  The
expressions can be variables, data type constants or data retrieval.

If both expressions are lists, then the combine operator will return a list
with all of the items of both.

    names = enterprise.name

    names:
    - e1
    - e2
    - e3

    $names + ["new1", "new2"]
    - e1
    - e2
    - e3
    - new1
    - new2

If both expressions are integers, then the combine operator will add the
values.

    count(enterprise.name) + 1
    4

If both expressions are strings, then the strings will be concatenated.

    filename = "output"
    filename: output

    path = "/tmp/" + $filename + ".txt"
    path: /tmp/output.txt

Any other combination of data types will result in an error from combine.

#### Actions

There are a set of actions that can change the state of later queries or
perform a task.  These come in the form of action name followed by a list of
comma-separated arguments enclosed in parens `()`.  Actions do not return any
value and they cannot be part of an assignment or expression.

The `connect` action sets up a new session with a device.  All subsequent
queries are directed to the new device.  The first argument to connect is the
device type, then the remaining parameters are specific to the device.  Using
connect allows data to be gathered from multiple sources in the same query
execution.

For VSD:

    connect("VSD", url, username, password, enterprise, cert_file, cert_key_file)

    Where:
        url (required)           : URL of VSD to connect to
        username (optional)      : Username to be used to connect (default "csproot")
        password (optional)      : Password to be used to connect (default "csproot")
        enterprise (optional)    : Enterprise to be used to connect (default "csp")
        cert_file (optional)     : Path to a certificate file (instead of password)
        cert_key_file (optional) : Path to a certificate key file (required if cert_file specified)

For ElasticSearch:

    connect("ES", address, port)

    Where:
        address (required) : Address of ElasticSearch to connect to
        port (optional)    : Port of ElasticSearch (default 9200)

The `redirect_to_file` action causes all subsequent queries to write results
to the specified file in addition to echoing to the screen.

    redirect_to_file("/tmp/output.txt")

The `echo` action turns on and off the printing of output of subsequent queries
to the screen.  This only affects the display of the queries and they still
operate normally in all other ways.  Variables would still be assigned and
output would still be written to file if redirected.  Echoing is on by default.
it can be turned off with the string `off` and turned back on with the string
`on`.  Disabling echoing is useful when huge amounts of data are expected to be
returned but it would clutter the screen.

    echo("off")

The `output` action turns on and off the writing of results to the screen,
redirect file and Python returned results.  Variables are still assigned when
output is off.  Turning off output is useful for gathering all intermediate
variables and then turning it on to write out a single final report.  Output is
on by default.  It can be turned off with the string `off` and turned back on
with the string `on`.

    output("off")

The `render_template` action outputs the text of a variable substituted
template.  The action takes a [Python Jinja](https://jinja.palletsprojects.com/en/2.11.x/)
template string as an argument.  Jinja templates contain tags enclosed by
double curly braces `{{` `}}` that reference variables to be substituted into
the text.  The variables assigned from queries are all available to be injected
into the text output.  Jinja templates support loops, conditionals, math and
other filters.  See the [Python Jinja](https://jinja.palletsprojects.com/en/2.11.x/)
documentation for more. Using Jinja templates, many file formats can be output
including nice text reports, HTML pages, CSV or JSON.

The `render_yaml_template` action is exactly the same as `render_template`
except that substituted variables are formatted properly for YAML/JSON output.
For example strings are quoted, booleans are converted to true/false, etc.

Example query file:

    output("off")

    connect("VSD", "https://localhost:8443")

    report_file = "report.txt"
    ent_names = enterprise.name
    ent_count = count($ent_names)
    licenses_expires = License.expirytimestamp

    template = """============== VSD Health Report ===========
    VSD:
        Enterprises: {{ ent_names | join(', ') }} ({{ ent_count }})
        License expiry: {{ ((licenses_expires[0] / 1000 - now) / (3600 * 24)) | int }} days
    ===========================================
    """

    redirect_to_file($report_file)
    output("on")
    render_template($template)

Resulting output to `report.txt`:

    ============== VSD Health Report ===========
    VSD:
        Enterprises: public, private, Shared Infrastructure (3)
        License expiry: 90 days
    ===========================================


## Python Library API

The MetroAE Config engine is written entirely using Python.  It can be integrated
into other Python-based tools using the library APIs via the pip package
`nuage-metroae-config`.  The following sections describe the use of the APIs.

### Template Store Class

The `TemplateStore` class is a repository for all of the templates loaded into
the engine.  It will be used by the `Configuration` class to apply template
actions to a device via a `DeviceWriterBase` class.

from nuage_metroae_config.template import TemplateStore

    Reads and parses configuration templates.

- read_templates(path_or_file_name)

        Reads and parses templates from either all templates in a
        directory path, or a single template specified by filename.
        Both yaml (.yml) and JSON (.json) files are supported.

- add_template(template_string, filename=None)

        Parses the specified string as a template in Yaml or JSON format.

- get_template_names(software_type=None, software_version=None)

        Returns a list of all template names currently loaded in store.
        If software_version and/or software_type is provided, names will
        be filtered by the specified version/type.

- get_template(name, software_type=None, software_version=None)

        Returns a Template object of the specified name.  If software_version
        and/or software_type is provided, template of specified version/type
        will be returned.

### Template Class

The `Template` class is a read-only class which provides information about a
template that was read in via the `TemplateStore`.

from nuage_metroae_config.template import Template

    Configuration template.  This class is read-only.

- get_name()

        Returns the name of this template.

- get_template_version()

        Returns the template version

- get_software_version()

        Returns a dictionary of {"software_version": "xxx", "software_type": "xxx"}

- get_schema()

        Returns the schema for the template variables in json-schema form.

- get_example()

        Returns example user-data for template variables in YAML format.

- get_documentation()

        Returns template documentation in MarkDown format.

- validate_template_data(template_data)

        Validates that the template_data provided matches the variables schema.
        Returns True if ok, otherwise an exception is raised.

### VSD Writer Class

The `VsdWriter` class is a derived class of the `DeviceWriterBase` that applies
configuration to a Nuage Networks VSD.  The writers are modular such that
new devices could be configured by defining derived device writer classes based
off of the common `DeviceWriterBase` base class.  The remainder of the engine
APIs use the generic base class.  This provides great extendability to
configure many different devices.  Note that the `VsdWriter` class also acts as
a reader for the `Query` class.

from nuage_metroae_config.vsd_writer import VsdWriter

    Writes configuration to a VSD.  This class is a derived class from
    the DeviceWriterBase Abstract Base Class.

- set_session_params(url, username="csproot", password=None, enterprise="csp", certificate=None)

        Sets the parameters necessary to connect to the VSD.  This must
        be called before writing or an exception will be raised.

- get_version():

        Returns the version running on the VSD in format:
            {"software_version": "xxx", "software_type": "xxx"}

- set_software_version(software_version):

        Sets the software version of the VSD.  This can be obtained via the
        get_version() method to get from the VSD itself or overwriten
        to a specific value.

- read_api_specifications(path_or_file_name):

        Reads the VSD configuration API specifications from JSON files
        in the specified path or file name.  This must be called before
        writing or an exception will be raised.

- set_logger(logger):

        Set a custom logger for actions taken.  This should be based on the
        logging Python library.  It will need to define an 'output' log level
        which is intended to print to stdout.

### User Data Parser Class

The `UserDataParser` class parses the YAML-based user-data format defined in
the `User Data` section of this document.  It is able to return the data in
template_name/data_dictionary pairs.  This data can be fed into the
`Configuration` class to render templates and apply configuration.

from nuage_metroae_config.user_data_parser import UserDataParser

    Parses Yaml or JSON files containing template user data

- read_data(path_or_file_name):

        Reads and parses user data from either all files in a
        directory path, or a single file specified by filename.
        Both yaml (.yml) and JSON (.json) files are supported.

- add_data(user_data_string):

        Parses the specified string as user data in Yaml or JSON format.

- get_template_name_data_pairs():

        Returns parsed data as a list of (template_name, data_dict) pairs.

### Configuration Class

The `Configuration` class puts all the above pieces together to configure a
device.  It requires a `TemplateStore` object with loaded templates.  User data
is applied to the `Configuration` object which can be taken directly from the
`UserDataParser` class or from pure Python data via another source.  Finally,
the configuration is applied, updated or reverted to a device using one of the
derived writer classes.

from nuage_metroae_config.configuration import Configuration

    Container for template instances.

- set_software_version(software_type=None, software_version=None)

        Sets the current software version of templates that is desired.
        If not called, the latest software version of templates will be
        used.

- add_template_data(template_name, template_data)

        Adds template data (user data) for the specified template name.
        Data is specified in a kwargs dictionary with keys as the
        attribute/variable name.  The data is validated against the
        corresponding template schema.  An id is returned for reference.

- apply(writer)

        Applies this configuration to the provided device
        writer.  Returns True if ok, otherwise an exception is
        raised.

- update(writer)

        Applies this configuration to the provided device
        writer as an update.  This means objects that exist will
        not be considered conflicts.  Returns True if ok, otherwise
        an exception is raised.

- revert(writer)

        Reverts (removes or undo) this configuration from the
        provided device writer.  Returns True if ok, otherwise
        an exception is raised.

- set_logger(logger):

        Set a custom logger for actions taken.  This should be based on the
        logging Python library.  It will need to define an 'output' log level
        which is intended to print to stdout.

### Query Class

The `Query` class allows information to be retrieved from a device through
device readers based on the `DeviceReaderBase`.  Note that the `VsdWriter`
class is also a reader derived off of this base and can be used for query.  The
information to be retrieved is defined by a "query language" which is described
in the `Language Definition` section of this document.  Results are returned
from the `execute()` of the query language text.  Also, any variables set during
the query can be retrieved with `get_variables()`.

from nuage_metroae_config.query import Query

    A class which can retrieve information from a device using a
    DeviceReaderBase reader object.  The information to be retrieved is defined
    by query language text.

- set_reader(reader)

        Set the primary reader object for performing queries on the device.
        The reader needs to be derived from the DeviceReaderBase class.

- get_variables()

        Returns a dictionary of all variables and values set during queries.

- register_reader(reader_type, reader)

        Register a reader object for each 'connect' type for performing queries
        on the device.  This allows devices to be switched within the query
        text.  The readers need to be derived from the DeviceReaderBase class.

- add_query_file(path_or_file_name)

        Reads and parses query sets from either all query files in a
        directory path, or a single query file specified by filename.
        Query files are expected to have .query extension

- execute(query_text=None, override_variables)

        Executes the query defined in query_text.  Variables in the query
        text can be overriden using the override_variables kwargs dict.  The
        results of the query are returned as a list with an entry for each
        query text line.  If query_text is None, then query text is pulled
        from query files defined by add_query_file.

- set_logger(logger)

        Set a custom logger for actions taken.  This should be based on the
        logging Python library.  It will need to define an 'output' log level
        which is intended to print to stdout.

### ES Reader Class

The `EsReader` class can be used as another reader with the `Query` class.  It
is derived from the `DeviceReaderBase` class and can retrieve information from
ElasticSearch.

from nuage_metroae_config.es_reader import EsReader

    Performs queries on ElasticSearch.  This class is a derived class from
    the DeviceReaderBase Abstract Base Class.

- set_session_params(address, port=None):

        Sets the parameters necessary to connect to the ES.  This must
        be called before reading or an exception will be raised.

## Library API Example - Config

    from nuage_metroae_config.configuration import Configuration
    from nuage_metroae_config.template import TemplateStore
    from nuage_metroae_config.user_data_parser import UserDataParser
    from nuage_metroae_config.vsd_writer import VsdWriter

    ENGINE_VERSION = "1.1.0"
    VSD_API_SPEC_PATH = "./vsd-api-specifications"
    TEMPLATE_PATH = "./templates"
    USER_DATA_PATH = "./user_data"
    VSD_URL = "https://10.0.0.1:8443"
    VSD_USERNAME = "username"
    VSD_PASSWORD = "password"
    VSD_ENTERPRISE = "csp"


    def setup_template_store(template_path):
        store = TemplateStore(ENGINE_VERSION)

        store.read_templates(template_path)

        return store


    def setup_vsd_writer(vsd_url):
        vsd_writer = VsdWriter()
        vsd_writer.add_api_specification_path(VSD_API_SPEC_PATH)

        vsd_writer.set_session_params(vsd_url,
                                      username=VSD_USERNAME,
                                      password=VSD_PASSWORD,
                                      enterprise=VSD_ENTERPRISE)

        device_version = vsd_writer.get_version()
        vsd_writer.set_software_version(device_version)

        return vsd_writer


    def setup_configuration(template_store, user_data_path):
        config = Configuration(template_store)

        parser = UserDataParser()
        template_data_pairs = parser.read_data(user_data_path)

        for data in template_data_pairs:
            template_name = data[0]
            template_data = data[1]
            config.add_template_data(template_name, **template_data)

        return config


    # Setup
    template_store = setup_template_store(TEMPLATE_PATH)
    vsd_writer = setup_vsd_writer(VSD_URL)
    config = setup_configuration(template_store, USER_DATA_PATH)

    # Apply the configuration from user data
    config.apply(vsd_writer)

    # Revert the configuration from user data
    config.revert(vsd_writer)


## Library API Example - Query

    from nuage_metroae_config.es_reader import EsReader
    from nuage_metroae_config.query import Query
    from nuage_metroae_config.vsd_writer import VsdWriter

    VSD_API_SPEC_PATH = "./vsd-api-specifications"
    VSD_URL = "https://10.0.0.1:8443"
    VSD_USERNAME = "username"
    VSD_PASSWORD = "password"
    VSD_ENTERPRISE = "csp"

    ES_ADDRESS = "10.0.0.2"
    ES_PORT = 9200


    def setup_vsd_query_tool(vsd_url):
        vsd_reader = VsdWriter()
        vsd_reader.add_api_specification_path(VSD_API_SPEC_PATH)

        vsd_reader.set_session_params(vsd_url,
                                      username=VSD_USERNAME,
                                      password=VSD_PASSWORD,
                                      enterprise=VSD_ENTERPRISE)

        device_version = vsd_writer.get_version()
        vsd_reader.set_software_version(device_version)

        vsd_query = Query()
        vsd_query.set_reader(vsd_reader)

        return vsd_query


    def setup_es_query_tool(es_address):
        es_reader = EsReader()
        es_reader.set_session_params(es_address, ES_PORT)

        es_query = Query()
        es_query.set_reader(es_reader)

        return es_query


    def get_subnetwork_usage(vsd_query):

        query_string = """
        subnets = Enterprise.Domain.Zone.Subnet.{name, netmask}
        zones = Enterprise.Domain.Zone[%group=name].Subnet.{name, netmask}
        d_count = count(Enterprise.Domain)
        s_count = count(Enterprise.Domain.Zone.Subnet)
        z_count = count(Enterprise.Domain.Zone)
        """
        vsd_query.execute(query_string)

        results = vsd_query.get_variables()

        domain_dict = dict()
        domain_dict_agg = dict()

        # [{"name": <subnet_name>, "netmask": <netmask>}, ...]
        domain_dict["subnets"] = results["subnets"]
        # [[<zone_name>, [{"name": <subnet_name>, "netmask": <netmask>}, ...]], ...]
        domain_dict["zones"] = results["zones"]
        domain_dict_agg["d_count"] = results["d_count"]
        domain_dict_agg["s_count"] = results["s_count"]
        domain_dict_agg["z_count"] = results["z_count"]

        return domain_dict, domain_dict_agg


    def get_enterprises_with_alarms(vsd_query):

        query_string = """
        Enterprise[%group=name].Alarm[severity="CRITICAL"].description
        """
        results = vsd_query.execute(query_string)

        # [[<enterprise_name>, [{"description": <alarm_descr>}, ...]], ...]

        enterprises_with_critical_alarms = dict()
        count = 0

        for result in results:
            ent_name = result[0]
            alarms = result[1]
            if len(alarms) > 0:
                enterprises_with_critical_alarms[ent_name] = alarms
                count += 1

        enterprises_with_critical_alarms['total_enterprises_with_alarms'] = count

        return enterprises_with_critical_alarms


    # Setup query
    vsd_query = setup_vsd_query_tool(VSD_URL)

    # Perform queries
    domain_dict, domain_dict_agg = get_subnetwork_usage(vsd_query)
    enterprises_with_critical_alarms = get_enterprises_with_alarms(vsd_query)