Dynamic AWS S3 Notification Configuration using Terraform
Tags: terraform, aws, sns, s3
Published: 2022-03-07 09:08:46

---
title: Dynamic AWS S3 Notification Configuration using Terraform
published: true
date: 2021-06-08 16:22:09 UTC
tags: terraform,aws,sns,s3
canonical_url: https://itnext.io/dynamic-aws-s3-notification-configuration-using-terraform-43a74998e97a
---

It is the simple stuff that you expect to be, well, simple. Until they aren’t.

![](https://cdn-images-1.medium.com/max/1024/0*BWHk3RbzVKDTvGkY)<figcaption>Photo by <a href="https://unsplash.com/@jstrippa?utm_source=medium&amp;utm_medium=referral">James Harrison</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption>

So the situation is like this:

- I have an aws **s3 bucket**
- I have multiple **dynamically created consumers** for objectCreated event from this bucket
- **Each consumer subscribes to specific prefix on this bucket.** i.e. service 1 will listen to new files on srv1/ path only. service2 will listen to new files on srv2/ path only and so on.

Since one SNS cannot differentiate between s3 prefixes, I need to have multiple SNS topics — one for each service

Doing it manually — it is super easy, just create another pair of sns/sqs and I’m ready to go. But manual process belongs to the dark ages of 1995, I need to do it automatically using IaaC, Terraform in this case.

### Easy, let’s do it!

So there I went, knowing exactly what I need to do. Create a terraform file to manage the s3 bucket, sns topics and their policies, and attach each sns topic to the bucket with correct prefix (/srv1, /srv2, etc…)

Like any good programmer — I wanted to copy paste stuff just to get it going, so I googled “aws s3 sns notification terrafom” and clicked the first link (which super convinently was terraform documentations). This is the link:

> [https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3\_bucket\_notification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification)

A simple note on that page caught my eyes and broke my heart:

![](https://cdn-images-1.medium.com/max/1024/1*xhMknuPeY_oG3v6vHjUouQ.png)

What does that all mean??

### Not so easy…

well, it means that I can attach one aws\_s3\_bucket\_notification to the bucket. It also means, I have to know in advance all topics I’m going to have since I have to write them as part of the terraform code.

But I said before, I have a dynamic list of consumers, I can’t know all of them in advance, am I stuck?

### Creativity time

So after thinking about it for a while, I tried playing with the aws api, fetching configuration and updating it. So if I can do it using the API and get what I want, I surely can automate it!

It took few trials and errors and I wrote a python script which accepts few parameters — bucket name, sns arn, and the relevant prefix. The scripts get the current configuration from aws API, check if the subscription already exists and adds it if it doesn’t.

_In case someone needs the python file, I’d be happy to share it_

Now I just need to run the script as part of Terraform.

#### Terraform Null resource and local exec provisioner

> The null\_resource resource implements the standard resource lifecycle but takes no further action.

> The triggers argument allows specifying an arbitrary set of values that, when changed, will cause the resource to be replaced.

Basically null\_resource can do anything, it has no backing resource in the cloud, it has a trigger and a provisioner which runs once the trigger changed.

So I’ve added a null resource wherever I have an SNS topic, pass the bucket name, the sns arn and the prefix, and that’s it!

```
provider "null" {}

resource "null_resource" "s3_object_created_subscription" {

  triggers = {
     prefix = var.prefix
  }

  provisioner "local-exec" {
     command = "python ${var.bucket_name} ${var.sns_arn} ${var.prefix}"
  }
}
```

This little piece of code can run from many terraform state files and attach an sns to a bucket with whatever prefix and how many prefixes I need. Like any terraform resource it has state, so it will run only once, unless you change the prefix.

### Conclusion

I know that’s a hack. It might break since s3 API is eventually consistent. But it does not bother me since I don’t have a massive amount of parallel changes.

The reasoning behind it, vs using multiple buckets, is the bucket limit I tackled. Amazon currently allows only 1000 buckets per account. That’s actually a lot of buckets. But once you have multiple services, let’s say 30, each one requires at least 1 bucket and you have dynamic test environments, you can easily see how you reach this limit.

* * *