Infrastructure with Terraform – Tweeting from a lambda
Tags: aws, terraform, ecr, lambda
Published: 2022-02-14 11:37:51

---
title: Infrastructure with Terraform – Tweeting from a lambda
description: Let's create the necessary infrastructure for our lambda, including an ECR repo and CloudWatch events
tags: aws, terraform, ecr, lambda
published: true
cover_image: https://ik.imagekit.io/thatcsharpguy/posts/python-lambdas/cycles-part-5_FE8AgrAJ6.jpg
series: Tweeting from an AWS Lambda
--- 

Since we are working in AWS with a *lambda* we need to create infrastructure in there.

As a programmer I like to define everything in code, however infrastructure provisioning is something that until recently needed to be managed manually – either through a graphical interface or a CLI with limited scripting capabilities.

Over the years, tools have emerged that brought us closer to the dream of being able to create infrastructure just by defining it in code, tools such as Ansible, CloudFormation and Terraform allow us to do just that. And it is precisely the last one that I chose to create the necessary elements for this series of posts.

It is not my interest to explain to you how Terraform works (I don't even know properly myself, in this post I did the minimum for the *lambda* to work). The way I present this post is by describing the content of the *terraform/main.tf* file that will contain the infrastructure.

## Providers

Terraform interacts with remote systems (such as AWS) through plugins; these plugins are known as `providers`.

Each terraform module must specify the `providers` it needs via the block `required_providers`, each provider has a name, a location, and a version. For example, in the lambda example that I am going to post, I am using 2 providers:

- `aws`, which exists in `hashicorp/aws` any version adhering to `3.27.X` will work
- `null`, it is an special provider, I'll tell you more about it later.

```
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }

    null = {
      version = "~> 3.0.0"
    }
  }
  required_version = ">= 0.14.9"

  backend "s3" {
    bucket = "feregrino-terraform-states"
    key    = "lambda-cycles-final"
    region = "eu-west-1"
  }
}
```

### Backend configuration

Within the terraform configuration block you can also see that there is another block defined as `backend "s3"`, this block helps us specify where the state file will be located, in this file we keep the state of the infrastructure that we have created with terraform so far. As I discussed in the first post of the series, this file will exist in an *S3 bucket*, the specification of which we put in the `backend` block.

### Provider configuration

Some providers require extra configuration, for example, AWS requires us to configure things like the region we want to connect to, the profile and the credentials we are going to use. Although the recommendation is that you do not put passwords or secrets in code, for example, in the AWS configuration I have:

```
provider "aws" {
  profile = "default"
  region  = "eu-west-1"
}
```

## Data Sources

Terraform allows us to access data defined outside our configuration files, through `data` blocks, through these we can access information about the user who is executing commands in AWS, using `aws_caller_identity`:

```
data "aws_caller_identity" "current_identity" {}
```

## Local Values

I like to think of local values ​​as variables within each module, and we must define them within a `locals` block; `locals` can also take values ​​from other sources, such as variables or *data sources* to simplify access to them:

```
locals {
  account_id          = data.aws_caller_identity.current_identity.account_id
  prefix              = "lambda-cycles-final"
  ecr_repository_name = "${local.prefix}-image-repo"
  region              = "eu-west-1"
  ecr_image_tag       = "latest"
}
```

## AWS

### Secretos

Given the nature of the service I am trying to deploy, it is necessary to access the secrets stored in the AWS secret manager, these must be specified as *data sources*, with `data` blocks, in the case of secrets, it is necessary to access the secret with `aws_secretsmanager_secret` and then to the latest version of it with `aws_secretsmanager_secret_version`:

```
data "aws_secretsmanager_secret" "twitter_secrets" {
  arn = "arn:aws:secretsmanager:${local.region}:${local.account_id}:secret:lambda/cycles/twitter-2GMvKu"
}

data "aws_secretsmanager_secret_version" "current_twitter_secrets" {
  secret_id = data.aws_secretsmanager_secret.twitter_secrets.id
}
```

### ECR repository

As the *lambda* is going to be deployed using a docker container it is necessary to create a repository in ECR, we can use the `aws_ecr_repository` resource by specifying the repository name from one of the local variables:

```
resource "aws_ecr_repository" "lambda_image" {
  name                 = local.ecr_repository_name
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = false
  }
}
```

### Creating a Docker image

Once the repository is created, it is necessary to upload an image to it, however Terraform is used to define infrastructure, not to perform tasks such as building a docker image, much less uploading it. I am going to assume that for this step, before executing the Terraform I already have an image built with the name `lambda-cycles`, the only thing that would be missing then is uploading it to the ECR repository.

We can use a little *hack* to accomplish this with Terraform by using a *null* resource (`null_resource`) and a called provider `local-exec` that allows you to specify commands to be executed on the local computer:

```
resource "null_resource" "ecr_image" {
  triggers = {
    python_file_1 = filemd5("../app.py")
    python_file_2 = filemd5("../plot.py")
    python_file_3 = filemd5("../tweeter.py")
    python_file_4 = filemd5("../download.py")
    requirements  = filemd5("../requirements.txt")
    docker_file   = filemd5("../Dockerfile")
  }
  provisioner "local-exec" {
    command = <<EOF
           aws ecr get-login-password --region ${local.region} | docker login --username AWS --password-stdin ${local.account_id}.dkr.ecr.${local.region}.amazonaws.com
           docker tag lambda-cycles ${aws_ecr_repository.lambda_image.repository_url}:${local.ecr_image_tag}
           docker push ${aws_ecr_repository.lambda_image.repository_url}:${local.ecr_image_tag}
       EOF
  }
}
```

Did you notice the `triggers` block? this block will help us track changes to files that will determine if the *lambda* container has changed; with `filemd5` we get a hash of the specified files. This would mean that if we make any changes to the *.py* files the Docker image to be rebuilt and uploaded to the ECR repository.

### Image information

It is necessary to generate a *data source* (in the form of a `aws_ecr_image`) that specifies a dependency on the creation and publication of the image, we can do this thanks to `depends_on`:

```
data "aws_ecr_image" "lambda_image" {
  depends_on = [
    null_resource.ecr_image
  ]
  repository_name = local.ecr_repository_name
  image_tag       = local.ecr_image_tag
}
```

### *Policies* and *permissions*

Before creating the *lambda*, I have to take care of other administrative tasks, the first is to create a role the *lambda* can assume to be executed:

```
resource "aws_iam_role" "lambda" {
  name               = "${local.prefix}-lambda-role"
  assume_role_policy = <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Action": "sts:AssumeRole",
           "Principal": {
               "Service": "lambda.amazonaws.com"
           },
           "Effect": "Allow"
       }
   ]
}
 EOF
}
```

Now, since I want to monitor my *lambda*, and to know if any errors occurred during its execution, it is necessary to grant it permissions so that it can create logs in *CloudWatch*:

```
data "aws_iam_policy_document" "lambda" {
  statement {
    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents"
    ]
    effect    = "Allow"
    resources = ["*"]
    sid       = "CreateCloudWatchLogs"
  }
}

resource "aws_iam_policy" "lambda" {
  name   = "${local.prefix}-lambda-policy"
  path   = "/"
  policy = data.aws_iam_policy_document.lambda.json
}
```

## Lambda – at least

Now that I have almost everything in place, I can create the lambda via the `aws_lambda_function` resource, this is one of the more convoluted definitions in this tutorial, so I'll try to explain it a bit more in detail:

The first thing I do is add a dependency to my docker image build with `depends_on`, then I specify the name of the *lambda* and the role it should assume with `function_name` and `role`. I know in advance that this lambda can take a bit of time so I'll leave it `timeout` a bit high.

Once we create our image in ECR we must tell the lambda that the package_typeis an image, followed by the image_uriso that it knows where to find it.

Once we create our image in ECR we must tell the *lambda* that the `package_type` is an image, followed by the `image_uri` so that it knows where to find it.

Finally, since my *lambda* is going to send a Tweet, it is necessary to pass the necessary secrets to it. Again, in the interest of keeping everything as private as possible, we will have to define them as environment variables (instead of hardcoding them); I achieve this from the block `environment` and extracting the secrets from –yeah, it is repetitivw– the secrets previously stored in AWS:

```
resource "aws_lambda_function" "lambda" {
  depends_on = [
    null_resource.ecr_image
  ]
  function_name = "${local.prefix}-lambda"
  role          = aws_iam_role.lambda.arn
  timeout       = 300
  image_uri     = "${aws_ecr_repository.lambda_image.repository_url}@${data.aws_ecr_image.lambda_image.id}"
  package_type  = "Image"
  environment {
    variables = {
      API_KEY             = jsondecode(data.aws_secretsmanager_secret_version.current_twitter_secrets.secret_string)["API_KEY"]
      API_SECRET          = jsondecode(data.aws_secretsmanager_secret_version.current_twitter_secrets.secret_string)["API_SECRET"]
      ACCESS_TOKEN        = jsondecode(data.aws_secretsmanager_secret_version.current_twitter_secrets.secret_string)["ACCESS_TOKEN"]
      ACCESS_TOKEN_SECRET = jsondecode(data.aws_secretsmanager_secret_version.current_twitter_secrets.secret_string)["ACCESS_TOKEN_SECRET"]
    }
  }
}
```

### Running every X minutes

So far so good, if you run terraform up to this point we woill have created several resources: an ECR repository, a docker image, and a lambda. But the icing on the cake is missing, and that is that the point of turning the code into a lambda; I want to run it multiple times throughout the day, every so often.

To achieve this task, I can use a trigger with the AWS CloudWatch service, something that executes my lambda at time intervals defined by me, this is possible with Terraform as well.

The first thing is to define an event rule in CloudWatch:

```
resource "aws_cloudwatch_event_rule" "every_x_minutes" {
  name                = "${local.prefix}-event-rule-lambda"
  description         = "Fires every 20 minutes"
  schedule_expression = "cron(0/20 * * * ? *)"
}
```

This event needs a target, in this case it's my lambda:

```
resource "aws_cloudwatch_event_target" "trigger_every_x_minutes" {
  rule      = aws_cloudwatch_event_rule.every_x_minutes.name
  target_id = "lambda"
  arn       = aws_lambda_function.lambda.arn
}
```

And of course, like almost everything in AWS, we also need to grant it permissions so that the event can invoke the *lambda*:

```
resource "aws_lambda_permission" "allow_cloudwatch_to_call_lambda" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.lambda.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.every_x_minutes.arn
}
```

***et voilà !*** – we already have all the necessary ingredients to run and create our lambda using Terraform.

Remember, all of this content exists in the *terraform/main.tf* file within the repository we've been working on.

This is how [the repository looks like at this point](https://github.com/fferegrino/tweeting-cycles-lambda/tree/part-4-terraform).

Remember that you can find me on Twitter [at @feregri_no](https://twitter.com/feregri_no) to ask me about this post – if something is not so clear or you found a typo. The final code for this series is on [GitHub](https://github.com/fferegrino/tweeting-cycles-lambda) and the account tweeting the status of the bike network is [@CyclesLondon](https://twitter.com/CyclesLondon).