EDA for AWS operations
Tags: ansible, eventdriven, aws, redhat
Published: 2024-03-22 15:15:32

## Welcome

Great that you're here. Today, I will write a bit about Event-Driven Ansible and its possibilities for improving the operational efficiency of our AWS environment. But wait, what does it mean by Event-Driven Ansible? My favorite sentence from the documentation is "EDA provides a way of codifying operational logic," so in short words, we can code logic for responding to different events. For example, we can trigger playbook execution with Lambdas make our self-service process a bit more ops-oriented, or introduce ChatOps.

## Before we begin

### Native AWS services?

Seems that could be a question here. Why not use System Manager, as it is a dedicated service for operations? Or OpsWorks? Or even Lambdas. Let’s break it down now.

#### System Manager

As AWS has many complementary solutions with quite similar functionality, we will start with the most well-known, and easy to use. System Manager is agent agent-based solution for managing a whole fleet of instances, no matter whether cloud or on-prem, if an agent is installed System Manager can be used. Additionally, with the usage of [Automation](https://aws.amazon.com/systems-manager/features/#Automation), we can use more than 370 pre-defined tasks, written with PowerShell, Bash, or Python, and that is great. If we wanted to schedule servers update, we can do it, and execute this operation on the whole fleet at the same time(which is not the best idea), or on part based on tags and groups.
So what’s the challenge? System Manager isn’t declarative, If we would like to add a custom ssm-document for creating a new folder, we need to write for example: 

```yaml
---
schemaVersion: '2.2'
description: State Manager mkdir example
parameters: {}
mainSteps:
- action: aws:runShellScript
  name: make /opt/test
  inputs:
    runCommand:
    - sudo mkdir /opt/test && chmod -R 0440 && chown admin. /opt/test
```

It’s fine, but it's like taking a step back. For service restart, it’s a great solution, but refreshing our system or using complex functions becomes even more complex.

#### OpsWorks

The solution was designed to serve managed Chef/Puppet to customers, unfortunately, all of them will reach End of Life withe the end of May 2024. During the time of writing this article (1-half of March), you can read about it on the public service page. [OpsWorks](https://aws.amazon.com/opsworks/). 
So as a summary, nice solution unfortunately based on Chef/Puppet, not a SaltStack, also the idea of stacks could be a blocker for a multi-cloud environment. 

#### Lambdas 

Lambdas and the whole serverless ecosystem were called a “new big thing”, a lot of people advocate for using function as much as possible and everywhere. After a while, we realized, that Lambdas-based architecture is very hard to keep clean and simple, also can easily make our bill much higher. That is why we started using them when it makes sense. For example image processing, data processing, etc, but most importantly, functions can act as a glue, for everything, connecting services that aren’t natively integrated, executing tasks, or even [stopping our instance](https://blog.3sky.dev/article/instance-stopper/), when some criteria weren’t met. The thing is that it will be quite hard and not very time-efficient to use Lambdas for service restart tasks. The main reason is that functions operate on different logic layers, let’s say hypervisor level. However, server-level operation could be a challenge. 

## Initial infrastructure

First, we need to set up our base infrastructure and install the needed packages.
Let’s assume, that our infrastructure will be very simple. We will use just for simplification three small VMs, all of which will be placed in a private subnet, one will be acting as a bastion host, for communication with our ansible-runner, just to keep the whole setup a bit more “real world” and secure.


![Initial infrastructure](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jir8iilxoqep6wj9x2hz.png)


As it’s quite a straightforward task I will use AWS CDK for that, please take a look at the user-data part, which is a place where we’re installing EDA components:

```typescript
    userData.addCommands(
      `dnf --assumeyes install git maven gcc java-17-openjdk python3-pip python3-devel`,
      `echo 'export JAVA_HOME=/usr/lib/jvm/jre-17-openjdk' >> /etc/profile`,
      `pip3 install wheel ansible ansible-rulebook ansible-runner`,
      `echo 'export PATH=$PATH:/usr/local/bin' >> /etc/profile`,
      `mkdir /opt/ansible`,
      `git clone https://github.com/3sky/eda-example-ansible /opt/ansible/eda-example-ansible`,
      `cd /opt/ansible/eda-example-ansible && /usr/local/bin/ansible-galaxy collection install -r collections/requirements.yml -p collections`,
      `chown -R ec2-user:ec2-user /opt/ansible/*`,
    );
```

When the stack is ready we can connect with a regular `.ssh/config` file.

```shell
Host jump
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  IdentityFile ~/.ssh/id_ed25519_eda
  User ec2-user
  # this one could change
  # as its public IP
  Hostname 3.126.251.7

Host runner
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  ProxyJump jump
  IdentityFile ~/.ssh/id_ed25519_eda
  User ec2-user
  Hostname 10.192.0.99

Host compute
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  ProxyJump jump
  IdentityFile ~/.ssh/id_ed25519_eda
  User ec2-user
  Hostname 10.192.0.86
```

With that being done, we can focus on popular use cases.

## EDA configuration

As today’s article is an EDA introduction, we will focus on basic http listener and basic service configuration, next article will be focused on more complex event sources like [MSK](https://aws.amazon.com/msk/) or watchdogs.

### Server configuration

First let’s use a basic server setup, based on documentation:

```yaml
---
- name: Listen for events on a webhook
  hosts: all

  ## Define our source for events

  sources:
	# use Python range function
    - ansible.eda.range:
        limit: 5

	# use webhook with path /endpoint
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

	# use kafka
    - ansible.eda.kafka:
        topic: eda
        host: localhost
        port: 9092
        group_id: testing
```

Seems rather straight, right? First, we’re using the keyword `sources`, then with typical Ansible syntax, we’re providing the FQDN of the module. Also as you can see there are multiple event sources, all of which are described [here](https://ansible.readthedocs.io/projects/rulebook/en/stable/sources.html#event-source-plugins). In the case of introduction, I will be using web-hook endpoint. 

### Rules configuration

Now let’s define our first rules:

```yaml
---
- name: Listen for events on a webhook
  hosts: all

  sources:
	# use webhook with path /endpoint
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

  # define the conditions we are looking for
  rules:
	 
    - name: Rule 1
      condition: event.payload.message == "Hello"

  	  # define the action after meeting the condition
      action:
        run_playbook:
          name: what.yml

  	# As it's a list of dicts, we can add another rule
    - name: Rule 2
      condition: event.payload.message == "Ahoj"

      action:
        run_playbook:
          name: co.yml
```

### Execution phase

Great! Let’s assume, that both of our `run_playbook` looks very similar, and are simple:

```yaml
---
- hosts: localhost
  connection: local
  tasks:
    - debug:
        msg: "World"
```

Now, we’re good to go, and execute our `ansible-rulebook` command:

```shell
ansible-rulebook --rulebook eda-example.yml -i inventory.yml --verbose
```

Then let’s open another terminal window(or login into `bastion` host) and execute the testing http request:

```shell
curl -H 'Content-Type: application/json' -d "{\"message\": \"Test\"}" 10.192.0.92:5000/endpoint
```

Inside the log we can see:

```shell 
2024-03-20 20:42:26,859 - aiohttp.access - INFO - 10.192.0.92 [20/Mar/2024:20:42:26 +0000] "POST /endpoint HTTP/1.1" 200 158 "-" "curl/7.76.1" 
```

Then, let’s try met out conditional, by changing our message to “Hello”, and next to “Ahoj”.

```shell
2024-03-20 20:44:39,199 - aiohttp.access - INFO - 10.192.0.92 [20/Mar/2024:20:44:39 +0000] "POST /endpoint HTTP/1.1" 200 158 "-" "curl/7.76.1"

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "World"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2024-03-20 20:44:41,923 - ansible_rulebook.action.runner - INFO - Ansible runner Queue task cancelled
2024-03-20 20:44:41,924 - ansible_rulebook.action.run_playbook - INFO - Ansible runner rc: 0, status: successful
2024-03-20 20:46:14,240 - aiohttp.access - INFO - 10.192.0.92 [20/Mar/2024:20:46:14 +0000] "POST /endpoint HTTP/1.1" 200 158 "-" "curl/7.76.1"

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Svet"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
2024-03-20 20:46:16,620 - ansible_rulebook.action.runner - INFO - Ansible runner Queue task cancelled
2024-03-20 20:46:16,621 - ansible_rulebook.action.run_playbook - INFO - Ansible runner rc: 0, status: successful
```

Now we have a working example, of working EDA. Let’s focus on something more useful.

## Example use cases

As you probably notice already it’s more like REST API to Ansible, and you’re right. It could do that as well, as web-hook is an event as well. So let’s focus on basic administration tasks.

```yaml
---
- name: Listen for events on a webhook
  hosts: all

  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

    - name: Restart Server
      condition: event.payload.task == "restart"
      action:
        run_playbook:
          name: reboot.yml

    - name: Reload Nginx
      condition: event.payload.task == "reload-nginx"

      action:
        run_playbook:
          name: reload-nginx.yml
```

With the following rulebook, we’re able to restart the server if needed or reload nginx with the usage of REST API. That can be easily connected with ITSM systems like Jira or ServiceNow. However please note that there is no filter or tags implemented - based on [documentation](https://ansible.readthedocs.io/projects/rulebook/en/stable/actions.html), which means, we need a dedicated playbook for a particular group of hosts, or issues. 


## Summary

EDA seems to be a more and more popular topic among many organizations, which is why in my opinion is a good time to take a look at it. To make it even simpler to start with I prepared a whole ready-to-use project that contains:

- playground based on AWS
- all the playbooks and rules included in this article
- step-by-step README file

That is why please use this article just like an introduction, the whole magic should take place on your AWS account. Please play with rules and use-cases, the thing about possibilities which this pattern can deliver in your organization, or which aspect can improve. If you have any questions or need help don't hesitate and contact me. 
 
[Link to GitHub repository](https://github.com/3sky/eda-example)