Receiving Slack notifications when CloudTrail logging gets turned off
Tags: aws, security, serverless
Published: 2023-08-10 08:09:46

When someone turns off CloudTrail logs, it should be a warning sign, which we want to know when it happens. Luckily we can use EventBridge to route CloudTrail events to various targets and take the necessary steps to remediate the issue.

## 1. Problem statement

Amazon <a href="https://aws.amazon.com/cloudtrail/" class="link" target="_blank" rel="noopener noreferrer">CloudTrail</a> is the surveillance camera for our accounts. It records every API call that any users or roles make. If we have multiple accounts set up in AWS <a href="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_introduction.html" class="link" target="_blank" rel="noopener noreferrer">Organizations</a>, we can create a central **trail** in the management account. We can then enable logging to all accounts and all regions. Or, if we use <a href="https://aws.amazon.com/controltower/" class="link" target="_blank" rel="noopener noreferrer">Control Tower</a> to set up the account structure, we don't need to do anything because it will automatically create the trail.

It's needless to say that we should always have CloudTrail logging **turned on**. If someone turns off logging, we won't see who did what and when in our accounts. If anybody with the relevant permissions does anything, we will miss important information for our investigations.

This way, we want to know when someone turns off CloudTrail. We can also restart logging automatically if relevant to the business use case.

## 2. Solution overview

In this solution we will create an <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html" class="link" target="_blank" rel="noopener noreferrer">EventBridge</a> rule that listens to the `StopLogging` CloudTrail event. When we have a multi-account setup, it should be in the **management account** that contains the trail.

After logging has stopped, EventBridge will route a message to a dedicated <a href="https://slack.com/" class="link" target="_blank" rel="noopener noreferrer">Slack</a> channel with the user's name, account number and the time of the event. Optionally, it will also invoke a <a href="https://aws.amazon.com/lambda/" class="link" target="_blank" rel="noopener noreferrer">Lambda function</a>, which will re-enable logging in CloudTrail.

This post won't discuss how to create Slack channels and apps and how to enable incoming webhooks. I won't go into detail about creating Lambda functions and EventBridge rules either. Instead, I'll highlight some key points we want to consider when implementing the architecture.

I assume the solution's components (trail, Lambda function, EventBridge rule) are in the same region.

## 3. EventBridge

As discussed in the last paragraph, we'll create an EventBridge **rule** that listens to the `StopLogging` CloudTrail event.

Some key points to consider are the following.

### 3.1. Use the default bus

We should create the rule in the **default** bus because AWS services send their events there.

### 3.2. Creating a connection

The Slack channel webhook is an external API, so we should first create an **API destination**. Before doing that, we'll need a **connection**.

We can create a connection named `TestConnection` in the **Integration/API destinations** section on the EventBridge page. The new connection can look like this:

![Creating a connection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/78zgcknmnfq8v591zexa.png)

Slack uses a bot user to forward webhook notifications to the associated channel. The webhook endpoint authentication occurs with a **token**, which we must specify in the connection setup. The authorization type is `API Key`, the header's name is `Authorization`, and the value should be in `Bearer TOKEN` format. We can get the token on the Slack app page from the **Features/Oauth & Permissions** section.

![Authorized connection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qzwei7yrl6c408kw2hlu.png)

The status of the connections should now be `Authorized`. `Deauthorized` indicates that the connection setup was unsuccessful.

### 3.3. Creating an API destination

Next, we can create the API destination, and let's creatively call it `Test`.

![Creating an API destination](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sertwnhv3gxb3mpnnfxk.png)

The destination endpoint is the **webhook URL**, and we should select `POST` as the method. The `TestConnection` we created above is available now, and we can choose it from the dropdown.

If everything goes well, the API destination's status will be `Active`.

We have now prepared EventBridge to make authorized calls to the Slack channel's incoming webhook.

### 3.4. Creating a rule

We are ready to create the rule in EventBridge!

**Event pattern**

The event pattern should look like this:

```json
{
  "source": ["aws.cloudtrail"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["cloudtrail.amazonaws.com"],
    "eventName": ["StopLogging"]
  }
}
```

The `source` and `eventSource` properties are pointing at CloudTrail because we want to catch the `StopLogging` event. CloudTrail emits this event when someone turns off logging.

**Slack target**

Let's add the **first** target to the rule. The target type is `EventBridge API destination` for the Slack webhook endpoint, and we should select the one we created above. It's called `Test` in this example, and the corresponding connection (`TestConnection`) will auto-populate the relevant fields.

An API destination is a separate resource, so EventBridge will need permission to invoke it. EventBridge will create a new <a href="https://aws.amazon.com/iam/" class="link" target="_blank" rel="noopener noreferrer">IAM</a> role if we don't already have one. Its permission policy looks like this:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "events:InvokeApiDestination"
      ],
      "Resource": [
        "arn:aws:events:us-east-1:123456789012:api-destination/Test/*"
      ]
    }
  ]
}
```

**Input transformer**

Slack's incoming webhook expects an object with a `text` property. We should define what we want to see in the channel in this field. In this case, it's a plain string without fancy formatting: `USER_NAME/ROLE_NAME turned off CloudTrail logging in account 123456789012 at 2023-08-07T11:31:21Z`.

For this to happen, we should configure the **target input** and create a custom output in the **Additional settings** section on the **Select target(s)** page. The configurator uses JSON path, so the `Input path` field will be very similar to what we see in <a href="https://docs.aws.amazon.com/step-functions/latest/dg/input-output-example.html" class="link" target="_blank" rel="noopener noreferrer">Step Functions</a>.

If we want EventBridge to send the above message, we can write something like this:

```json
{
  "account": "$.account",
  "dateTime": "$.detail.eventTime",
  "user": "$.detail.userIdentity.principalId"
}
```

The values come from the <a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference.html" class="link" target="_blank" rel="noopener noreferrer">CloudTrail event</a> object EventBridge receives.

In the following textbox, the `Template` will define the message format:

```json
{
  "text": "<user> turned off CloudTrail logging in account <account> at <dateTime>"
}
```

The `<KEY>` element refers to the relevant property key defined in the input path object.

**Lambda function target**

If we want to restart logging automatically, we can add a **second target** that can be a Lambda function (see next section). It's an `AWS Service` target type, and there shouldn't be any problem finding it from the dropdown (provided that we have previously created it).

EventBridge will automatically add the `lambda:InvokeFunction` permission to the function's **resource-based** policy. The statement will allow the EventBridge rule to invoke the function.

## 4. Automating logging restart

Let's see some minimal <a href="https://nodejs.org/en" class="link" target="_blank" rel="noopener noreferrer">Node.js</a> code for the function that we can use to restart CloudTrail logging.

```js
import { CloudTrailClient, StartLoggingCommand } from "@aws-sdk/client-cloudtrail";

const client = new CloudTrailClient();

export const handler = async (event) => {
  const trail = event.detail.requestParameters.name;

  const input = {
    Name: trail,
  };
  const command = new StartLoggingCommand(input);

  try {
    const response = await client.send(command);
    return response;
  } catch (error) {
    console.error('Error: ', error.message);
    throw error;
  }
};
```

The only thing worth mentioning here is that we want to know the trail where we want to start the logging. We can get this information from the `event` object and store it in the `trail` variable. It will be the same event as discussed in the input path section.

To make the code work, we should add the `cloudtrail:StartLogging` permission to the function's **execution role**.

## 5. It should work!

We can now try the solution.

If we stop logging in CloudTrail, we should receive a message within a few seconds in the Slack channel specified in the webhook settings. If we add the second trigger to automate logging restart, we should see a green `Logging` indicator in the **General details** section on the CloudTrail page.

## 6. Considerations

Of course, Slack is not the only way to receive notifications. EventBridge integrates with many target services, both AWS and 3rd parties. For example, we can set up **email** or **phone text message** alerts in <a href="https://aws.amazon.com/sns/" class="link" target="_blank" rel="noopener noreferrer">SNS</a>, or add a different target if business needs require.

## 7. Summary

CloudTrail logging should be a key element in any company's security management. If we turn off logging, we won't have a history of what happened in our accounts. To meet security and compliance requirements, it might be a good idea to create notifications and trigger automated fixes in case someone turns off trail logging.

One option is to send Slack notifications from EventBridge and use a Lambda function to re-enable logging automatically.

## 8. Further reading

<a href="https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html" class="link" target="_blank" rel="noopener noreferrer">Creating a trail</a> - How to create a CloudTrail trail

<a href="https://api.slack.com/messaging/webhooks" class="link" target="_blank" rel="noopener noreferrer">Sending messages using Incoming Webhooks</a> - Enabling Slack webhooks

<a href="https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html" class="link" target="_blank" rel="noopener noreferrer">Getting started with Lambda</a> - How to create a Lambda function

<a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html" class="link" target="_blank" rel="noopener noreferrer">Creating Amazon EventBridge rules that react to events</a> - Everything about EventBridge rules AWS style

<a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destination-partners.html#eb-api-destination-slack" class="link" target="_blank" rel="noopener noreferrer">API destination partners - Slack</a> - Some guidelines for setting up Slack as an API destination in EventBridge

<a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html" class="link" target="_blank" rel="noopener noreferrer">Amazon EventBridge input transformation</a> - Lots of information on input transformation