How to set up cross-account access to Amazon Kinesis Data Streams
Published: 2024-02-29 11:24:11

## Overview - Amazon Kinesis Data Streams and separated AWS accounts

[Amazon Kinesis Data Streams](https://aws.amazon.com/kinesis/data-streams/) (hereafter referred to as KDS) is a managed data processing service designed for the real-time collection of high traffic of data and facilitating its transfer to subsequent AWS services. It is particularly suited for handling streaming data, such as logs, where order matters, making it a commonly used service for IoT data collection. For example, it can be specified as the data export destination for [Amazon Monitron](https://aws.amazon.com/monitron/), which allows for predictive maintenance of industrial equipment through machine learning.

KDS serves as an intermediary between data producers and data consumers, employing this architecture for its operation.

![Typical Architecture using Amazon Kinesis Data Streams](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ndnh9jtivsaz91bwdhrv.png)

When fully leveraging AWS Cloud, it's not uncommon to operate separate AWS accounts for purposes of distinction and cost management. In such cases, there might be a need to process streaming data collected by KDS in another AWS account.

This blog introduces the procedure for sharing a data stream with another AWS account and referencing it from AWS Lambda (specifying the Lambda function as a trigger) using [the resource-based policy of Amazon Kinesis Data Streams](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html) ([update as of November 2023](https://aws.amazon.com/jp/about-aws/whats-new/2023/11/amazon-kinesis-data-streams-cross-account-access-aws-lambda/)). A data stream, in this context, is akin to a pipeline through which data flows.

## The Architecture and Setup

The architecture is as follows.

![The architecture for cross-account access](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/seudfjcmr2nmcs71yylq.png)

### Key Points for Setup

Here are the key points for setup:

* The execution role of the Lambda function on the data processing side(Account B) requires the `AWSLambdaKinesisExecutionRole` policy.
* For the KDS data stream on the data producer side(Account A), the resource-based policy should specify the IAM role ARN of the execution role of the Lambda function on the data processing side as the Principal, and set the Allow Actions to include `kinesis:DescribeStream`, `kinesis:DescribeStreamSummary`, `kinesis:GetRecords`, `kinesis:GetShardIterator` and `kinesis:ListShards`.

Especially regarding the second point on resource-based policies, it's important to note that specifying `kinesis:DescribeStream` might be missed when using the dialog in the management console. It needs to be manually added using the JSON editor (as of Feb. 2024, reported).

The following official documents might also be helpful:

* [Sharing your data stream with another account](https://docs.aws.amazon.com/streams/latest/dev/controlling-access.html#sharing-data-streams)
* [Sharing access with cross-account AWS Lambda functions](https://docs.aws.amazon.com/streams/latest/dev/resource-based-policy-examples.html#Resource-based-policy-examples-lambda)

### Steps

* The steps involve several cross-accounts: data producer side(Account A) → data processing side(Account B) → data producer side → data processing side. Make sure not to mix up the targets.
* Everything must be in the same region; sharing is not possible across different regions (e.g., if the data stream is in us-west-2 and the Lambda function is in ap-northeast-1). If you wish to send data to a different region, consider the architecture using Amazon EventBridge mentioned in the epilogue.

#### Step 1) On the data producer side - Account A

##### 1-1. Create a data stream in [Amazon Kinesis](https://console.aws.amazon.com/kinesis) (e.g., `kds-sharing-example1`)

See here for the creation method ([Step1: Create Data Stream](https://docs.aws.amazon.com/streams/latest/dev/tutorial-stock-data-kplkcl-create-stream.html)).

![Create Data Stream](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gevvlsv6lfzub2096ryl.png)

For testing or small data volumes, the "Provisioned" capacity mode with "1" provisioned shard is sufficient.

NOTE: that while this guide assumes the creation of a new data stream, existing data streams can also be repurposed.

#### Step 2) On the data processing side - Account B

##### 2-1. Create a Lambda function in [AWS Lambda](https://console.aws.amazon.com/lambda/home#/functions) (e.g., `kds-reader1`).

See here for the creation method ([Create a Lambda function with the console](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html#getting-started-create-function)).

![Create a Lambda function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g1w5xyu8lez6u5jx2eld.png)

The source code is as follows (Python 3). It simply emits the `event` to Amazon CloudWatch Logs, which is sufficient for operational verification. After modifying the code source as below, click "Deploy" to deploy.

```python
def lambda_handler(event, context):
    print(event)
    return True
```

NOTE: There's a template for testing the function with Amazon Kinesis Data Streams sample data, which can be used for testing.

![Test data for Lambda function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jsqyvp8qmnet3m6shi61.png)

##### 2-2. Attach the `AWSLambdaKinesisExecutionRole` policy to the execution role of `kds-reader1`.

After viewing the details of `kds-reader1`, go to "Cofiguration" > "Permissions" and click on the role name assigned to the execution role to view the role's settings (in the figure below, click on `kds-reader1-role-mp67l1v2`).

![Cofiguration > Permissions on Lambda Function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mofjpf8qmw0qw0b1ydyn.png)

Go to "Add Permission" > "Attach Policy" for the permission policy to display the list of policies to attach.

![Attach Policy on AWS IAM role](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n262g551rv7h6etgjxvy.png)

Select `AWSLambdaKinesisExecutionRole` from the list of "Other Permission Policies" and then click "Add Permission".

![AWSLambdaKinesisExecutionRole](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/drafoa1di02iqc15bx04.png)

Policy addition is complete.
Check for `AWSLambdaKinesisExecutionRole` is added to the list of allowed policies as shown below.

![list of allowed policies](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gleumsy050qi3e6ao3oa.png)

NOTE: We assume that the role for executing Lambda functions is an IAM role that is automatically created and appended when a Lambda function is created. Existing IAM roles can also be used.

##### 2-3. Noted the ARN of the IAM role.

This IAM role's ARN (in this step, `kds-reader1-role-q6zcv9kq`) will be used on next step.

#### Step 3) Back on the data producer side - Account A

##### 3-1. configure resource-based policy for `kds-sharing-example1` in [Amazon Kinesis](https://console.aws.amazon.com/kinesis)

After viewing the details of `kds-sharing-example1`, go to "Data stream sharing" > "Create Policy".

![Create Policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a4ekgpnn2kgj2qpou99w.png)

In Policy Details, select the Visual Editor, check "Data stream sharing throughput read Access," enter the ARN of the IAM role you wrote down earlier in "Specify Principal(s)," and click "Create Policy."

![Visual Editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x5znkdjqjq6uewvlkg16.png)

NOTE: The principal should be the ARN of the IAM role; specifying an ARN other than the IAM role's ARN (e.g., the ARN of a Lambda function or data stream) will result in an error and the policy cannot be created.

When the resource-based policy appears, click "Edit" to display the JSON editor. Here, add `"kinesis:DescribeStream",` to the list of Actions as shown below. Finally, click "Save Changes".

![Add a privilege](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kdxpeijqd35gric765lk.png)

NOTE: If the same privileges have already been added at the visual editor stage, the above editing process is not necessary.

Configuration of the resource-based policy is complete.
Check to that the ARN of the IAM role is attached to the Principal and the five permissions are attached to the Action, as shown below.

![Resource-based policy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qzlbpx642jq9hqg33cq6.png)

##### 3-2. Noted the ARN of `kds-sharing-example1`.

`kds-sharing-example1` (KDS Stream's) ARN will be used on next step.

#### Step 4) Back on the data processing side - Account B

4-1. set up a trigger for `kds-reader1` in [AWS Lambda](https://console.aws.amazon.com/lambda/home#/functions)

After viewing the details of `kds-reader1`, go to "Configuration" > "Triggers" > "Add trigger".

![Add a Trigger](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m9o76m16fiu0m14o50j9.png)

In the Trigger configuration, select Kinesis for "Select a source". Then set the ARN of `kds-sharing-example1` to "Kinesis stream" in the displayed contents. Leave the other items as they are and click "Add".

![Trigger Configuration](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/stbxi3twy3jdkbc4m9gz.png)

NOTE1: When the focus moves to the text box, the message "No item" is displayed. No problem, ignore.

NOTE2: If you get an API error when clicking "Add", check the following two things (1) Resource-based policy permissions on the data stream side(Account A). In particular, make sure that `kinesis:DescribeStream` is included. (2) Permissions for the Lambda function execution role. In particular, make sure that the `AWSLambdaKinesisExecutionRole` policy is attached.

Trigger configuration is complete.
You can see that Kinesis has been added to the trigger (input source) of kds-reader1 as shown below.

![Configuration Function](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ydqbwnjj95krj591i0yc.png)

### How to check

To check, send data to the data stream (kds-sharing-example1 in this example) on the data generator side(Account A) and check the Amazon CloudWatch Logs output on the data processor side(Account B).

[AWS CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html#how-to-get-started) on the data generator side(Account A) sends data to the data stream via AWS CLI.

```bash
aws --cli-binary-format raw-in-base64-out \
  kinesis put-record --stream-name kds-sharing-example1 \
  --partition-key DUMMY1 \
  --data '{"this_is": "test record"}'
```

If the result of the command execution is as follows, the data transmission has succeeded.

```
{
    "ShardId": "shardId-000000000000",
    "SequenceNumber": "49649718468451075013017298672854645152715037125279481858"
}
```

If the following log is confirmed in the log of kds-reader1 in CloudWatch Logs on the data processing side (Account B), the setting was successful.

```
{'Records': [{'kinesis': {'kinesisSchemaVersion': '1.0', 'partitionKey': 'DUMMY1', 'sequenceNumber': '49649683864473953433843496278632352517587667908684677122', 'data': 'eyJ0aGlzX2lzIjogInRlc3QgcmVjb3JkIn0=', 'approximateArrivalTimestamp': 1709100911.667}, 'eventSource': 'aws:kinesis', 'eventVersion': '1.0', 'eventID': 'shardId-000000000000:49649683864473953433843496278632352517587667908684677122', 'eventName': 'aws:kinesis:record', 'invokeIdentityArn': 'arn:aws:iam::888800008888:role/service-role/kds-reader1-role-q6zcv9kq', 'awsRegion': 'us-east-1', 'eventSourceARN': 'arn:aws:kinesis:us-east-1:999900009999:stream/kds-sharing-example1'}]}
```

## Epilogue - Architecture with Amazon EventBridge

In this article introduced sharing data streams using Amazon Kinesis Data Streams resource-based policies. This will allow, for example, the Amazon Monitron data introduced at the beginning of this article to be used by other AWS accounts, which will give you more flexibility in the operation of your AWS account.

Other possible architectures for using Amazon Kinesis Data Streams data streams with other AWS accounts include sending them through the Amazon EventBridge event bus.

![The architecture using Amazon EventBridge ](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i6h9koe8k6z4sv4r5d0c.png)

The advantage of this would be that it can be configured as a no-code and managed service. It can also be used across different regions, and although there is a fee for using Amazon EventBridge, the architecture is well worth it.

Here are some URLs to help you create this architecture.

* [Amazon Kinesis stream as a source](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-kinesis.html) ([Update information](https://aws.amazon.com/jp/about-aws/whats-new/2023/11/amazon-kinesis-data-streams-eventbridge-pipes-console/))
* [Sending and receiving Amazon EventBridge events between AWS accounts](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cross-account.html)
* [Sending and receiving Amazon EventBridge events between AWS Regions](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-cross-region.html)

I personally believe that the ability to create a configuration that matches the skills you possess and the type of operation you wish to achieve is the best part of building blocks.

Not only this configuration, but it would be a good idea to TRY a configuration that is appropriate for the time, along with new features that may come out in the future!

[EoT]