AWS: CloudTrail overview and integration with CloudWatch and Opsgenie
Tags: aws, devops, todayilearned
Published: 2021-07-16 05:55:08

---
title: AWS: CloudTrail overview and integration with CloudWatch and Opsgenie
published: true
date: 2021-07-15 07:23:33 UTC
tags: aws,devops,todayilearned
canonical_url: https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/
---

![](https://cdn-images-1.medium.com/max/1024/1*8QppZuMnRQSngTE4H4UqzA.png)

AWS CloudTrail is a service for auditing AWS accounts events and is enabled by default.

It saves all actions that were done by a user, IAM role, or an AWS service via AWS Console, AWS CLI, or AWS SDK.

CloudTrail will write information about every API call, log in to the system, services events, and is an indispensable instrument for AWS account security.

Such events will be stored for 90 days, but you can configure a _trail_, that will store selected events to an AWS S3 bucket and can send them to AWS CloudWatch, and in CloudWatch we can configure Alarms to receive notifications.

These trails can be of two types: global, applied to all AWS Regions, and local for only one chosen region. See [How does CloudTrail behave regionally and globally?](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-regional-and-global-services)

So, in this post we will take a closer look at AWS CloudTrail and its features, will configure export to CloudWatch, and will create an alarm and notifications to Slack via Opsgenie.

- [AWS CloudTrail Events](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#AWS_CloudTrail_Events)
- [Management events](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Management_events)
- [Data events](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Data_events)
- [Insights events](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Insights_events)
- [CloudTrail trail](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#CloudTrail_trail)
- [Create a CloudTrail trail](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Create_a_CloudTrail_trail)
- [CloudTrail and CloudWatch integration](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#CloudTrail_and_CloudWatch_integration)
- [Create an AWS CloudWatch custom metric](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Create_an_AWS_CloudWatch_custom_metric)
- [Opsgenie and AWS Simple Notifications Service](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Opsgenie_and_AWS_Simple_Notifications_Service)
- [Create an AWS CloudWatch Alarm](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Create_an_AWS_CloudWatch_Alarm)
- [Opsgenie and custom fields from AWS SNS](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/#Opsgenie_and_custom_fields_from_AWS_SNS)

### AWS CloudTrail Events

See [What are CloudTrail events?](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-concepts.html#cloudtrail-concepts-events) and [Logging management events for trails](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html).

An _Event_ in CloudTrail is a record of any activity in an AWS account. Events can be three types: _management event_, _data event_, _insight event_.

#### Management events

Management events are anything about operations that are performed on AWS resources, and also called _control plane operations,_ for example:

- security operations (for example, the `AttachRolePolicy` API call)
- new resources creation (for example, the `CreateDefaultVpc` API call)
- network configuration updates (for example, the `CreateSubnet` API call)
- logging configuration changes (for example, the `CreateTrail` API call)

Also, it includes non-API calls such as logins to the AWS Management Console, see [Non-API Events Captured by CloudTrail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-non-api-events.html).

Also, these events can be about _read_ or _write_ API calls. Read operations are operations that only request information about AWS resources (for example, `DescribeSubnets`), and Write - any modification requests (for example, `TerminateInstances`), see [Read and write events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html#read-write-events-mgmt).

#### Data events

See [Data events](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-data-events-with-cloudtrail.html#logging-data-events).

These are information about operations with or within a concrete AWS resource, also called _data plane operations_, for example:

- S3 buckets operations such as `GetObject`, `DeleteObject`, and `PutObject`
- AWS Lambda function calls
- Amazon DynamoDB operations such as PutItem, DeleteItem и UpdateItem

They are not logged by default and must be enabled when creating a trail.

#### Insights events

CloudTrail Insights will watch for anomalies and if will create a CloudTrail event if find any.

Such anomalies could be an unusual number of Delete API calls to an AWS S3 bucket or too high (or too low) calls like `AuthorizeSecurityGroupIngress`.

### CloudTrail trail

As mentioned above, a _trail_ can stored events over 90 days and can be integrated with CloudTrail and CloudWatch. Also, it can collect data from all regions, while CloudTrail Dashboard displays information only about the current region.

#### Create a CloudTrail trail

Go to the _Trails_, click on the _Create trail_ button:

![AWS: CloudTrail - обзор и интеграция с CloudWatch и Opsgenie](https://cdn-images-1.medium.com/max/1024/0*w3LOPLAGticySZ-Y.png)

Set its name, chose to _Create new S3 bucket_, specify its name, and for now disable its data encryption:

![AWS: CloudTrail - обзор и интеграция с CloudWatch и Opsgenie](https://cdn-images-1.medium.com/max/1024/0*ljIATqR4omc1SXIK.png)

To configure alerts, enable sending events to AWS CloudWatch Logs:

![AWS: CloudTrail - обзор и интеграция с CloudWatch и Opsgenie](https://cdn-images-1.medium.com/max/863/0*xrvPOC7QfUr4d_-S.png)

Leave _Management events_ and enable _Insights_.

For the _Management events_ enable both Read and Write, for the _Insights events_ enable the _API call rate_:

![](https://cdn-images-1.medium.com/max/1024/0*ixGPszc-Eq3pz19u.png)

Check data in CloudWatch Logs:

![](https://cdn-images-1.medium.com/max/1024/0*XzTlTed2lqDYbxAA.png)

Now, we can create CloudWatch Metrics from these events and configure an Alarm.

### CloudTrail and CloudWatch integration

Next, let’s create a new CLoudWatch metric based on CloudTrail events, and then will configure an Alarm that will send a notification to AWS Simple Notification Service (SNS), which in its turn will forward them to Opsgenie, and from there we will receive messages to Slack.

There is a question, which events we really need to watch. This can be googled by the “_cloudtrail security alerts_” query that will give two interesting materials — [Threat Hunting with CloudTrail and GuardDuty in Splunk](https://www.chrisfarris.com/post/reinforce-threat-hunting/), and [Key CloudTrail Events To Monitor for Security in AWS](https://www.gorillastack.com/blog/real-time-events/important-aws-cloudtrail-security-events-tracking/), or just check examples in the [reating CloudWatch Alarms with an AWS CloudFormation Template](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/use-cloudformation-template-to-create-cloudwatch-alarms.html).

For use, we will use the following:

- logins to the AWS Console outside of our office (as the office IP example here — 8.8.8.8): `{ ($.eventName = ConsoleLogin) && ($.sourceIPAddress != "8.8.8.8") }`
- root user logins to the AWS Console: `{ ($.eventName = ConsoleLogin) && ($.userIdentity.type = "Root") }`
- logins with errors to the AWS Console: `{ ($.eventName = ConsoleLogin) && ($.errorMessage = "Failed authentication") }`
- unauthorized operations: `{ ($.errorCode = "\*UnauthorizedOperation") || ($.errorCode = "AccessDenied\*") }`
- notifications on spinning up huge EC2 instances, for example, the biggest type used by our team is c5.4xlarge : `{ ($.eventName = RunInstances) && (($.requestParameters.instanceType = \*.8xlarge) || ($.requestParameters.instanceType = \*.4xlarge)) }`
- new IAM users creation: `{ $.eventName="CreateUser" }`

See [Filter and pattern syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html).

At first, let’s create a simple alert about any login to the AWS Console

To start, take a look how a login in the event will look like: log in to your account, wait 10–15 minutes, make a filter by the _Event name_ == `ConsoleLogin`:

![](https://cdn-images-1.medium.com/max/1024/0*56eejZ2_6AxEqam-.png)

#### Create an AWS CloudWatch custom metric

Go back to a log-group create above, on the _Metric filters_ tab click _Create metric filter_.

In the _Filter pattern_ use `{ ($.eventName = ConsoleLogin) }`:

![](https://cdn-images-1.medium.com/max/1024/0*40mK4owyliM1C8aj.png)

Fill its fields:

![](https://cdn-images-1.medium.com/max/899/0*2I9EA3s-gN6xMlnf.png)

Check the metrics on graphs:

![](https://cdn-images-1.medium.com/max/1024/0*rzP4dO1apOKtqKBs.png)

#### Opsgenie and AWS Simple Notifications Service

Go to the AWS SNS, create a new SNS topic with the _Standart_ type:

![](https://cdn-images-1.medium.com/max/916/0*y0UdHOy_QhgRXJtR.png)

In the Opsginie _Integration list_ find _Incoming Amazon SNS_:

![](https://cdn-images-1.medium.com/max/1024/0*QLddYa-jjMZ7p_cD.png)

Save the integration, save the API key:

![](https://cdn-images-1.medium.com/max/873/0*-rGEWiQ5ShgdKuPu.png)

Go back to the SNS, click on the _Create Subscription_, chose _HTTPS_, set the URL and API key provided by Opsgenie:

![](https://cdn-images-1.medium.com/max/973/0*HjdTNmgndjcupYjd.png)

Receive a notification that subscription was confirmed:

![](https://cdn-images-1.medium.com/max/1024/0*rGh3Xr36GzgyXjWT.png)

#### Create an AWS CloudWatch Alarm

Go to the _Alarms_, click on the _Create alarm_:

![](https://cdn-images-1.medium.com/max/1024/0*mXa_1qcF_cXcFx4L.png)

Chose a metric:

![](https://cdn-images-1.medium.com/max/1024/0*ZGSiegJv5f_YNtxk.png)

![](https://cdn-images-1.medium.com/max/1024/0*pb3kKf-WKE6-PQZd.png)

Describe the alert’s conditions: — more or equal 1 event (alert on each login):

![](https://cdn-images-1.medium.com/max/1024/0*75BmJXExqn29T81R.png)

Configure sending notifications to the SNS topic created above:

![](https://cdn-images-1.medium.com/max/1024/0*ofjhpooZnlV59adp.png)

Set the alert’s name, and save it:

![](https://cdn-images-1.medium.com/max/1024/0*ADUk2lxRvsBE_x5z.png)

In a couple of minutes check its status: instead of _Insufficient data_ it will become _ОК_ (or _In alarm_ if there are logins):

![](https://cdn-images-1.medium.com/max/1024/0*phNAZFtNkUp52-sd.png)

Log in to the AWS Console, and wait for 10–15 minutes for a CloudTrail event, and you’ll get your alarm triggered::

![](https://cdn-images-1.medium.com/max/1024/0*DuGyr4r7QYTPHjqy.png)

And an alert in Slack:

![](https://cdn-images-1.medium.com/max/768/0*rohMR5xd72P07-I9.png)

#### Opsgenie and custom fields from AWS SNS

What I’d like to do else, is to filter Slack’s message text to display only custom fields.

This can be done [Opsgenie string functions](https://support.atlassian.com/opsgenie/docs/string-processing-methods-in-opsgenie-integrations/).

Go to the _Incoming SNS_ integration, click _Advanced_:

![](https://cdn-images-1.medium.com/max/1024/0*bNjzz1Kka2gx0bxp.png)

In the _Description_ field add the `Message.extract()` call:

```
{{Message.extract(/AlarmDescription":"(.+)","AWSAccountId"/)}}

AWS account ID: {{Message.extract(/AWSAccountId":"(.+)","NewStateValue"/)}}

AWS region: {{Message.extract(/Region":"(.+)","AlarmArn"/)}}
```

![](https://cdn-images-1.medium.com/max/527/0*AfDmVP0wZzEfBePx.png)

Now, when Opsgenie will receive a message in JSON from AWS SNS it will take only three fields from there — `AlarmDescription`, `AWSAccountId`, and `Region`, and as result, we will see in Slack alarm's message in the next form:

![](https://cdn-images-1.medium.com/max/489/0*yJwIJo4_NCVwdOA0.png)

Done.

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/aws-cloudtrail-overview-and-integration-with-cloudwatch-and-opsgenie/)_._

* * *