Setting EC2 application as SNS subscriber
Tags: aws, cloud, ec2, cloudskills
Published: 2022-05-15 11:25:42

## **Description:**

With this implementation we are trying to automate data load and consumption process using S3 trigger and EC2.


## **Brief explanation:**

- AWS Customer uploads invoice to S3 Bucket.
- S3 bucket has IAM Bucket policy to only allow specific customers (in this case it will be IP address) with putObject permission.
- Once the object is uploaded, SNS trigger will allow EC2 Instance application to read the object from S3 bucket. 

- Application in EC2 will process the invoice and generates CSV from it. Which is then uploaded to target S3 bucket.
- Same application writes original S3 file to document database Dynamo DB. 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lfk4xajhrjoi2vv6mnt7.png)

Below blog covers the part of configuring SNS subscription with application running in EC2.

##**EC2 Instance configuration**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ch1ysr15mzvt688oj8nq.png)

**Instance configuration-**
  Ubuntu 18.04
  T2-micro

This instance is configured as public instance, public IP – 107.22.136.3 is assigned to it. This will be needed for SNS subscription.

**Security Group:**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n6axfl4tu8bjs9rqtarz.png)
 

* As shown in snapshot above, 2 security groups are added.
    * Port 8080 is opened for SNS end point access to EC2 instance.
    * Port 22 is opened for personal IP address for developer access via SSH and SCP.

## **S3 Configuration**

Source S3 bucket is created which is configured to trigger based on put event. 

We can either keep this bucket open for public( ideally not recommended ) or block public access and interact with AWS Services or applications running on EC2. In our use case we used EC2 instance to upload object on this bucket.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n06670o1mwnc7hlk8cnx.png)

To make it private, blocked public access.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4xdhqfnmgwixfcdxdgeq.png)

**Enable Events** 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vk6xsdwaecfyt407p4ns.png)

In this case we added lifecycle policy to delete file in 24 hrs. As shown below expiration rule is set to expire/delete object after upload of 1 day.  


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ii3gczeijwgxxik8ftwp.png)

## **SNS Configuration**

Based on the above description, requirement is to integrate S3 trigger with SNS to automate file upload and processing. Below steps are followed to implement the same.

To add subscription to EC2, firsly we need to launch a service with-in ec2 and then configure SNS using http/https protocol.

This sample test uses python with django to enable service, but aws documentation also have other samples for reference

https://docs.aws.amazon.com/sns/latest/dg/sns-create-subscribe-endpoint-to-topic.html

Below snapshot shows service in ec2 running on port 8080.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eba8yjhytkqi00nsipak.png)

**SNS Topic creation and adding subscription**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/10wsaevpdta158jinx12.png)


**SNS Access policy -**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hlhplzv0cuain6fh1bxg.png)


SNS Access policy shows only S3 source bucket uploads are allowed to cause the event trigger. Its also restricted by source account and ARN.

**SNS Subscription -**  

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qkbvws5kdz7yvxpwzp4q.png)

Subscription is created using EC2 instance launched above as an endpoint. As shown in snapshot above endpoint URL is http://107.22.136.3:8080/sns. 

**Subscription Confirmation**

In previous section where we launched application in EC2 on port 8080, now receives the request and provides subscription confirmation link below as highlighted.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hx404r5hlz59xqbb6mlr.png)

On copying the link and pasting in webbrowser, receives below XML response.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f9dyruh7ampp79giai18.png)
 
**S3 Upload Event notification**

Once setting is done, uploaded any object on Source S3 bucket which triggers notification and json payload as shown below.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qfmw0bt3e03j9a4frq7j.png)
 
We can build rest of the process once notification is received by application. In our use case we will run data copy to another S3 bucket or DynamoDB.

Happy Learning!!!