Ansible Dynamic Inventory
Tags: ansible, devops
Published: 2020-07-03 03:48:42

Static inventory will not serve the needs when tracking and deploying to multiple sources with hosts spinning up and shutting down in response to business demands. The solution to such a changing infrastructure would be ansible dynamic inventory. To accomplish this, you will need a cloud provider ini file (e.g aws.ini) and ec2.py (which ansible use to communicate to the AWS API to fetch the metadata for all the instances). This post focus on AWS(Amazon Web Service).

`ec2.ini` file provided by Ansible. All the cloud provider `.ini` can be found here https://github.com/ansible/ansible/tree/stable-2.9/contrib/inventory

**ec2.py:** EC2 external inventory script which dynamically fetches data from AWS API

To set up the ec2 external inventory script, copy the script to `/etc/ansible/ec2.py` and run `chmod +x /etc/ansible/ec2.py`. You will also need to copy the `ec2.ini` file to `/etc/Ansible/ec2.ini`. Then run ansible-playbook normally, the --limit args can be parsed.

To make a successful call to AWS API, configure boto. There are a variety of methods available, you can add setup your aws credential by setting up your `~/.aws/credentials` or export the following environment variables:
```
export AWS_ACCESS_KEY_ID=’mYaWeSoMeAcCeSsKeY′
export AWS_SECRET_ACCESS_KEY=’IcAnToTaLlYpUtMySeCrEtHeRe′
```
Set up a few more environment variables to the inventory management script such as
```
$ export ANSIBLE_HOSTS=/PATH/ec2.py
$ export EC2_INI_PATH=/PATH/ec2.ini
```
This variable sets ansible-playbook to use the dynamic EC2 script instead of a static `/etc/ansible/hosts` file. Open up `ec2.py` in a text editor and make sure that the path to ec2.ini config file is defined correctly at the top of the script:

`EC2_INI_PATH` instructs `ec2.py` where the `ec2.ini` config file is located.

Make the script executable and test it by running
`./ec2.py –list`
```
$ ./ec2.py --list
{
  "_meta": {
    "hostvars": {
      "1.9O.1OO.12O": {
        "ansible_host": "1.9O.1OO.12O",
        "ec2__in_monitoring_element": false,
        "ec2_account_id": "19OP23I42OF3",
        "ec2_ami_launch_index": "0",
        ....
        "ec2_tag_App": "olonje",
        "ec2_tag_Env": "Staging",
        "ec2_tag_Name": "Bami Gbe",
        "ec2_virtualization_type": "hvm",
        "ec2_vpc_id": "vpc-IaMhIdDeN"
      },
      "4.OOO.122.1OO9": {
        "ansible_host": "4.OOO.122.1OO9",
        "ec2__in_monitoring_element": false,
        ...
        "ec2_key_name": "emi-moni-profile",
        "ec2_root_device_name": "/dev/xvda",
        "ec2_tag_App": "ounje2",
        "ec2_tag_Env": "prod",
        "ec2_tag_Name": "Production Stack",
        "ec2_virtualization_type": "hvm",
        "ec2_vpc_id": "vpc-IaMhIdDeN"
      }
    }
  },
  "ami_1k02nd83js84k6apO": [
    "1.9O.1OO.12O",
    "4.OOO.122.1OO9"
  ],
  "tag_App_olonje": [
    "1.9O.1OO.12O"
  ],
  "tag_App_order2": [
    "4.OOO.122.1OO9"
  ],
  "tag_Env_Staging": [
    "1.9O.1OO.12O"
  ],
  "tag_Env_prod": [
    "4.OOO.122.1OO9"
  ],
  "tag_Name_app_Worker": [
    "1.9O.1OO.12O"
  ],
  "tag_Name_Production_Stack": [
    "4.OOO.122.1OO9"
  ],
  "vpc_id_vpc_IaMhIdDeN": [
    "1.9O.1OO.12O",
    "4.OOO.122.1OO9"
  ]
}
```
To fine-tune the output and other features that aren’t applicable refer to the `ec2.ini` file. For multiple AWS accounts, you can pass `--profile` PROFILE name to the `ec2.py` script.
```
$ cat ~/.aws/credentials
[default]
aws_access_key_id = <default access key>
aws_secret_access_key = <default secret key>

[profile tito]
aws_access_key_id = <tito access key>
aws_secret_access_key = <tito secret key>

[profile ounje]
aws_access_key_id = <ounje access key>
aws_secret_access_key = <ounje secret key>
```
To get the inventory for the ounje account run `AWS_PROFILE=ounje ansible-playbook -i /etc/ansible/ec2.py myplaybook.yml`. Since the EC2 external inventory provides mappings to instances from several groups, we can filter the target instance based on resource tag. Speaking of **Tags.**

### Tags

Each instance has a variety of key/value pairs associated with it called **Tags**. The most common tag key is ‘Name’, though anything is possible. Each key/value pair is its own group of instances, again with special characters converted to underscores, in the format tag_KEY_VALUE e.g. `tag_Name_Web` can be used as is `tag_Name_redis-master-001` becomes `tag_Name_redis_master_001` `tag_aws_cloudformation_logical-id_WebServerGroup` becomes `tag_aws_cloudformation_logical_id_WebServerGroup`

When Ansible interacts with a specific server, the EC2 inventory script is called again with the --host HOST option. This looks up the HOST in the index cache to get the instance ID, and then makes an API call to AWS to get information about that specific instance. It then makes information about that instance available as variables to your playbooks. Each variable is prefixed by ec2_. e.g

```
ec2_dns_name
ec2_id
ec2_image_id
ec2_instance_type
ec2_ip_address
ec2_tag_Name
ec2_vpc_id
...
```

[Read more](https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html#inventory-script-example-aws-ec2)

To see the complete list of variables available for an instance, run the script by itself:
```
/etc/ansible/ec2.py --host ec2-12-1O1-14-12.compute-1.amazonaws.com
```

Note: API calls to EC2 can be slow. Therefore, API results are cached. The number of seconds a cache is considered valid can be updated in the `ec2.ini` file in `/etc/ansible/ec2.ini`. Default setting is
```
...
cache_max_age = 300
...
```
You can set it to `cache_max_age=0`.

## TESTING OUR DYNAMIC INVENTORY ABOVE

Running ansible-playbook deployment. Using `sample.yml`

```
- name: Deploy Ounje Artifact
  hosts: all
  vars:
    - app: "e-status "
    - env: "staging"
  tasks:
    - name: Get Instance IDs
      ec2_remote_facts:
        aws_access_key: "moYaCoverSecretMi"
        aws_secret_key: "amalaShitaLoSureJuMafo"
        region: "us-east-1"
        filters:
          "tag:Env": "{{ env }}"
      register: gbebodi

    - name: Store ounje hosts
      debug: var=gbebodi

    - name: Store Ounje EC2 instance ID
      set_fact:
         instance_id: "{{ item.id }}"
      with_items: "{{ gbebodi.instances }}"

    - name: Store EC2 instance ID in var
      debug: var=instance_id
```

Run command:

```
$ ansible-playbook -i /etc/ansible/ec2.py --limit "tag_App_order2" -u ec2-user sample.yml -e env=prod
```
In the above example:

1. `ansible-playbook` is the standard command to run an ansible playbook
2. `/etc/ansible/ec2.py` points ansible inventory to use the ec2.py to callout to aws API to get the ec2 instance metadata
3. `--limit "tag_App_ounje2”` This limit the host you want to target by instructing ansible-playbook to pull the metadata of all instances with a tag of key `App` and value `ounje2`. This enables the ansible-playbook to know where to run the tasks, know which instance you want to access and limit the activities to the only ounje2
4. `sample.yml` is the name of our playbook file
5. `–e` a way of parsing env variable to the playbook, or to override the default env variable
6. `-u` indicates the user ansible needs to use to login to the instance.

**output**

```
$ ansible-playbook -i /etc/ansible/ec2.py --limit "tag_App_ounje2" -u ec2-user sample.yml -e env=prod

[DEPRECATION WARNING]: ec2_remote_facts is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this rationale.. This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
PLAY [Deploy Ounje Artifact] ***********************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************
Monday 23 September 2O19 O1:17:25 -O4OO (O:OO:OO.141) O:OO:OO.141 ******
ok: [1.OO.1O9.111]
TASK [Get Instance IDs] *****************************************************************************************************************************************
Monday 23 September 2O19 O1:17:28 -O4OO (O:OO:O2.234) O:OO:O2.376 ******
ok: [1.OO.1O9.111]
TASK [Store ounje hosts] ***************************************************************************************************************************************
Monday 23 September 2O19 O1:17:29 -O4OO (O:OO:O1.481) O:OO:O3.857 ******
ok: [1.OO.1O9.111] => { "ounje": { "changed": false, "failed": false, "instances": [ { "ami_launch_index": "O", "architecture": "x86_94", "block_device_mapping": [ { "attach_time": "2O19-O6-23TO4:54:57.OOOZ", "delete_on_termination": true, "device_name": "/dev/xvda", "status": "attached", "volume_id": "vol-298hfn3f348ff4f348f" } ], "client_token": "", "ebs_optimized": false, "groups": [ { "id": "sg-Of11425adOe7f7d1a", "name": "launch-wizard-13" } ], "hypervisor": "xen", "id": "i-1O293847561OO293", "image_id": "ami-1k02nd83js84k6apO", "instance_profile": null, "interfaces": [ { "id": "eni-nf32f2fn48489r348", "mac_address": "41:9O:i7:dc:69:lO" } ], "kernel": null, "key_name": "emi-moni-profile", "launch_time": "2O19-O6-23TO4:54:56.OOOZ", "monitoring_state": "disabled", "persistent": false, "placement": { "tenancy": "default", "zone": "us-east-1c" }, "private_dns_name": "ip-1-OO-1O9-111.ec2.internal", "private_ip_address": "OO7-OO-69-42O", "public_dns_name": "ec2-1-OO-1O9-111.compute-1.amazonaws.com", "public_ip_address": "1.OO.1O9.111", "ramdisk": null, "region": "us-east-1", "requester_id": null, "root_device_type": "ebs", "source_destination_check": "true", "spot_instance_request_id": null, "state": "running", "tags": { "App": "ounje2", "Env": "prod", "Name": "Production Stack" }, "virtualization_type": "hvm", "vpc_id": "vpc-IaMhIdDeN" } ] }
}
TASK [Store EC2 instance ID] ************************************************************************************************************************************
Monday 23 September 2O19 O1:17:29 -O4OO (O:OO:OO.145) O:OO:O4.OO3 ******
ok: [1.OO.1O9.111] => (item={u'ramdisk': None, u'kernel': None, u'instance_profile': None, u'root_device_type': u'ebs', u'private_dns_name': u'ip-1-OO-1O9-111.ec2.internal', u'block_device_mapping': [{u'status': u'attached', u'device_name': u'/dev/xvda', u'delete_on_termination': True, u'attach_time': u'2O19-O6-23TO4:54:57.OOOZ', u'volume_id': u'vol-298hfn3f348ff4f348f'}], u'key_name': u'emi-moni-profile', u'interfaces': [{u'id': u'eni-nf32f2fn48489r348', u'mac_address': u'41:9O:i7:dc:69:lO'}], u'persistent': False, u'image_id': u'ami-1k02nd83js84k6apO', u'groups': [{u'id': u'sg-Of11425adOe7f7d1a', u'name': u'launch-wizard-13'}], u'spot_instance_request_id': None, u'requester_id': None, u'source_destination_check': u'true', u'id': u'i-1O293847561OO293', u'tags': {u'App': u'ounje2', u'Name': u'Production Stack', u'Env': u'prod'}, u'public_ip_address': u'1.OO.1O9.111', u'monitoring_state': u'disabled', u'placement': {u'tenancy': u'default', u'zone': u'us-east-1c'}, u'ami_launch_index': u'O', u'hypervisor': u'xen', u'region': u'us-east-1', u'ebs_optimized': False, u'launch_time': u'2O19-O6-23TO4:54:56.OOOZ', u'public_dns_name': u'ec2-1-OO-1O9-111.compute-1.amazonaws.com', u'state': u'running', u'architecture': u'x86_94', u'private_ip_address': u'OO7-OO-69-42O', u'vpc_id': u'vpc-IaMhIdDeN', u'client_token': u'', u'virtualization_type': u'hvm'})
TASK [Store EC2 instance ID in var] *****************************************************************************************************************************
Monday 23 June 2O19 O1:17:29 -O4OO (O:OO:OO.173) O:OO:O4.176 ******
ok: [1.OO.1O9.111] => { "instance_id": "i-1O293847561OO293"
}
PLAY RECAP ******************************************************************************************************************************************************
1.OO.1O9.111 : ok=5 changed=O unreachable=O failed=O
Monday 23 June 2O19 O1:17:29 -O4OO (O:OO:OO.11O) O:OO:O4.287 ******
===============================================================================
Gathering Facts ------------------------------------------------------------------------------------------------------------------------------------------ 2.23s
Get Instance IDs ----------------------------------------------------------------------------------------------------------------------------------------- 1.48s
Store EC2 instance ID ------------------------------------------------------------------------------------------------------------------------------------ O.17s
Store ounje hosts --------------------------------------------------------------------------------------------------------------------------------------- O.15s
Store EC2 instance ID in var ----------------------------------------------------------------------------------------------------------------------------- O.11s
Playbook run took O days, O hours, O minutes, 4 seconds
```

With no hostfile to maintain, you can deploy to freshly created instances spun up by terraform without having to worry about maintaining an hostfile.