Cross-Account log data sharing using Kinesis Data Firehose
Tags: aws, community, firehose
Published: 2023-02-05 00:35:53

## **Introduction :**

Nowadays, One of the most fundamental security measures when working on a multi-account AWS environment is the ability to consolidate, manage, and analyze logs coming from various AWS Services in multiple accounts and multiple AWS Regions in one single place.

---

## **Solution Overview :**

Based on the necessity to create a single **Dashboard** from where our **Security Team** can analyze all the logs coming from multiple **AWS Application Accounts**. We ended up with the following solution : 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nb7bejwzcoy33twt785s.png)

Using the above solution we were able to stream **CloudWatch logs** for a specific set of AWS Services (for example: **AWS WAF**) from multiple **Application Accounts** to a **Centralized Log Archive Account**.

---

## **Solution Setup :**

let's start by setting up the required resources in our **Centralized Log Account**, then we will move to the **Application Account**

### <u>**Centralized Log Account :**</u>

We are going to create the following resources :


1- **A Centralized Kinesis Firehose Stream & S3 Bucket :**

Using the **AWS Console**, create your **Kinesis Firehose Stream** by setting the Source as **Direct Put** and the Destination as **Amazon S3**.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3b2yeqq6cli6qea9d0cz.png)

You can choose an already existing Bucket as a destination otherwise you can create a new one.

For the rest you can keep the default values, otherwise you can follow the AWS Documentation for any further customizations : [Link](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html ) 

---

2- **IAM Role for CloudWatch Logs Destination :**

To enable the CloudWatch Logs Destination which We are going to create in the next step to send data to the Kinesis Firehose Stream We already setup in the previous step, We need to create a IAM Role "**CWLtoKinesisFirehoseRole**" with the following permissions: 

```
{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":["firehose:*"],
        "Resource":["arn:aws:firehose:region:LogAccountID:*"]
      }
    ]
}
```

With the following Trust Relationships to allow the CloudWatch service to assume it :

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
```

---

3-  **CloudWatch Logs Destination :**

The **CloudWatch Logs Destination** will work as the **Access Point** for your remote AWS Accounts to stream their logs to your **centralized Kinesis Firehose** in the **Log Account**.

The **CloudWatch Logs Destination** is a **regional** resource but can stream data to a Kinesis Firehose Stream in a **different region**, So you can create multiple CloudWatch Logs Destinations in different regions targeting your **Centralized Kinesis Firehose Stream** "_This is what we've done in our case_".

The necessity to create Multiple CloudWatch Logs Destinations in different regions is based on the regions from where you want to stream logs in your Application Accounts.

The Application Accounts should stream CloudWatch logs to the Centralized Log Account using the CloudWatch Logs Destination in the **same region**.

We are going to create it using the CLI as there is no way to do so using the AWS Console.

> CLI Command :
 
```
aws logs put-destination --destination-name "demoFirehoseCrossAccount" --target-arn "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT" --role-arn "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole"
```
**_target-arn_** and **_role-arn_** are refering to the **Kinesis Firehose stream** and the **IAM Role** we've created in the previous steps.

> Command Result :

```
		{
			"destination": {
				"destinationName": "demoFirehoseCrossAccount",
				"targetArn": "arn:aws:firehose:region:LogAccountID:deliverystream/DEMO-FIREHOSE-CROSSACCOUNT",
				"roleArn": "arn:aws:iam::LogAccountID:role/CWLtoKinesisFirehoseRole",
				"arn": "arn:aws:logs:region:LogAccountID:destination:demoFirehoseCrossAccount",
				"creationTime": 1675210769461
			}
		}
```

Save the command result somewhere as we are going to need the **Destination ARN** during resources creation in the **Application Account**.

---

4- **CloudWatch Logs Destination Policy :**

Each **CloudWatch Logs Destination** should have a **policy** attached to it.

Using the Destination Policy, you can decide which Remote Accounts can stream their logs through this **CloudWatch Logs Destination**.

Create a **policy.json** file with the following content, to allow remote Account/s to create **CloudWatch Subscription Filters** targeting the **CloudWatch Logs Destination** indicated in the **Resource** section of the Policy : 

```
{ 
		"Version" : "2012-10-17", 
		"Statement" : [ 
			{ 
				"Sid" : "", 
				"Effect" : "Allow",
				"Principal" : {
				   "AWS" : "ApplicationAccountID"
				},
				"Action" : "logs:PutSubscriptionFilter", 
				"Resource" : "arn:aws:logs:region:logAccountID:destination:demoFirehoseCrossAccount"
			} 
		] 
	}
```
**CloudWatch Subscription Filters** are what we are going to create in the **Application Accounts**

The **AWS Principal** should be the Account ID/IDs of the **Application Accounts** and **Resource** is the **ARN **of the **CloudWatch Logs Destination** created in the previous step.

> CLI Command :

```
aws logs put-destination-policy --destination-name "DEMO-FIREHOSE-CROSSACCOUNT" --access-policy "file://policy.json"
```
---

### <u>**Application Account :**</u>

In your Application Accounts all you need to create is a CloudWatch Kinesis Firehose **Subscription Filter** for the **CloudWatch Log Group** you want to stream to your **Centralized Log Account** :


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/djhwwq59eu18hpq4hpw6.png)

as a **Destination** choose **Cross-Account** and then insert the **CloudWatch Logs Destination ARN** for the one created in the Centralized Log Account : 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8x7ib3oi0t1a7561zd7r.png)

---

## **Finally :**


Now, We've all the required resources to stream **CloudWatch Logs** from multiple **Application Accounts** to a **Centralized Log Account**.

If you've the necessity to elaborate your data before storing it in S3, **Kinesis Data Firehose** can invoke a **Lambda** function to transform incoming source data and deliver the transformed data to destination [LINK](https://docs.aws.amazon.com/firehose/latest/dev/data-transformation.html).

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/30657zmosk3qwhkv8jig.png)

---

## **Helpful Links :**

- [Cross-account log data sharing using Kinesis Data Firehose](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CrossAccountSubscriptions-Firehose.html)
- [How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console](https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-logs-stream-to-kinesis/)