Background process in Lambda using SQS
Tags: lamda, sqs, go
Published: 2022-11-21 07:19:34

## AWS Lambda

`AWS Lambda` is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications and only pay for what you use.

## The Problem

We want some work to process at background (asynchronously) in Lambda even if the Lambda function returns. But when the lambda function returns, all the background services automatically stop.

## The Solution

So here we are going to use Amazon SQS for this purpose. We will send a message to a queue, and it will trigger another lambda function that will run in the background.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/665tdzsoy8pd2hyb1hl7.png)

### What is Amazon SQS?

Amazon `Simple Queue Service (SQS)` is a managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Using SQS, you can send, store, and receive messages asynchronously between software components at any volume, without losing messages or requiring other services to be available.

SQS offers two types of message queues. `Standard queues` offer maximum throughput, best-effort ordering, and at-least-once delivery. `SQS FIFO queues` are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

## Golang Example to Run Background Processes in Lambda Using SQS

### Prerequisites

- For building and deploying your functions, you’ll be using the `Serverless Framework`, which is the most widely used tool for the job. Assuming you have a recent version of [Node.js](https://nodejs.org/en/) installed, you can install the `Serverless CLI` with the following npm command

  ```console
  $ npm install -g serverless
  ```

  Once you have the Serverless CLI installed, you must configure it to use the AWS access keys of your account

  ```console
  $ serverless config credentials --provider aws --key <access key ID> --secret <secret access key>
  ```

  You can get the access key and secret key from the `My Security Credentials` option. Use `Create New Access Key` if you do not have one already

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/91vrhnamx37zffwrg32c.png)

- If you don’t have Go installed yet, you can either [download an installer](https://golang.org/dl/) from the official website or use your favorite package manager to install it

### Create a Messaging Queue

Let's create a queue. Go to Simple Queue Service (SQS) in your AWS account. You will see the following interface there


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wc4y3m1hgh1z2zo9jm0g.png)

Create queue from here and select type `FIFO` and let's name it `test.fifo` as we want to process exactly once message, in the exact order that they are sent. you can also use the `standard queue` according to your need. And we'll keep all other settings to default for now.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3qkw7gyhl9n92z7ut09j.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k9uh39de69v2vtux7hw4.png)

And now let's code.

### Sending Message on the SQS Queue

Now that you have everything you need, let’s install the [AWS SDK for Go](https://github.com/aws/aws-sdk-go) library.

```console
$ go get github.com/aws/aws-sdk-go
```

After this, we can proceed to write the code to send a message on our SQS queue to trigger the lambda function that will perform the background task.

```go
package handlers

import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/credentials"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/sqs"
  "os"

  "encoding/json"
  "log"
  "net/http"
)


func yourHttpHandlerFunction(w http.ResponseWrite, r *http.Request) {
  // tasks you need to perform before the background task
  ...
  
  // sending details to other lambda function to perform the background task
  message := SqsTriggerMessage{
    Message: "You can add different fields here according to your data requirements in the background task.",
  }

  err := sendMessage(message)
  if err != nil {
    log.Println(err)
	w.WriteHeader(http.StatusInternalServerError)
    return
  }

  // other tasks you need to perform
  ...
  
  // sending user an immediate response
  w.WriteHeader(http.StatusOK)
}

type SqsTriggerMessage struct {
  Message string
}

func sendMessage(data interface{}) error {
  b, err := json.Marshal(data)
  if err != nil {
    return err
  }

  svc := sqs.New(
    session.Must(
      session.NewSession(
        &aws.Config{
          Credentials: credentials.NewStaticCredentials(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
          Region:      aws.String(os.Getenv("AWS_REGION")),
        },
      ),
    ),
  )

  result, err := svc.GetQueueUrl(&sqs.GetQueueUrlInput{
    QueueName: aws.String("queue-name"),
  })
  if err != nil {
    return err
  }

  _, err = svc.SendMessage(&sqs.SendMessageInput{
    MessageBody:            aws.String(string(b)),
    QueueUrl:               result.QueueUrl,
    MessageGroupId:         aws.String("group-id"),
    MessageDeduplicationId: aws.String("deduplication-id"),
  })

  return err
}

```

This previous code can be broken into a few simple steps:

- Define a message struct that contains the data you need in the background task
- Marshal your struct into JSON and send that JSON as message to the SQS queue
  We can also send this through the message attributes, but I found this method more convenient to use
- Give the user a `200` response and return. Your lambda function will finish after this, but the message you sent on the SQS queue will trigger the other lambda function to perform the background task

### The Lambda Function to Handle SQS Event

Let's create another program to handle sqs events.

```go
package main

import (
  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"

  "encoding/json"
  "log"
)

func main() {
  lambda.Start(handleSqsRequest)
}

func handleSqsRequest(sqsEvent events.SQSEvent) error {
  for _, message := range sqsEvent.Records {
    var request SqsTriggerMessage
    err := json.Unmarshal([]byte(message.Body),&request)
    if err!=nil {
      log.Println(err)
      continue
    }
	// task you need to perform based on the data in the message
    log.Println(request.Message)
	...
  }

  return nil
}

type SqsTriggerMessage struct {
  Message string
}
```

This previous code can be broken into a few simple steps:

- We wrote a `handleSqsRequest` function that will receive the messages from queue and registered it in the main function using the `AWS Lambda for Go library`
- In the event handler function, we are unmarshalling the message body into our message struct, and printing the message on the console. You can send the details of the task you need to perform and use those to call appropriate function to do that task

Now we have 2 Lambda functions the first one sends the message to SQS queue, that needs to trigger our second lambda function to perform the background task

### Deployment

Our lambda functions are now ready, and we can proceed by deploying it with the Serverless Framework. Our application is deployed by the Serverless framework based on the `serverless.yml` configuration file.

> If you are not familiar with the `.yml` syntax, you can read this [serverless.yml guide](https://bref.sh/docs/environment/serverless-yml.html).

We must first create a `serverless.yml` file that defines what we are deploying.

```yaml
service: your-service-name
provider:
  name: aws
  runtime: go1.x
package:
  exclude:
    - ./**
  include:
    - ./bin/**
functions:
  main-program:
    handler: bin/main-program
    events:
      - http:
          path: /
          method: get
      ...
  sqs-handler:
    handler: bin/sqs-handler
    events:
      - sqs: <replace_this_with_your_sqs_queue_arn>
```

In the above file, we defined both our lambda functions.
- The first one runs the main program on the HTTP requests through `Amazon API Gateway`. In this program we need to perform the background task, so we will send a message on the SQS queue
- The second one runs our sqs handler on an SQS event in the queue which `arn` we provide here. You can get the `arn` from your aws account where we created it. `Amazon SQS > Queues > test.fifo`


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zrl02ky36effyby8f9q7.png)

Next up, we will build our code, and deploy it using the `serverless deploy` command.

```console
GOOS=linux GOARCH=amd64 go build -o bin/main-program .
GOOS=linux GOARCH=amd64 go build -o bin/sqs-handler ./sqs

serverless deploy
```

You can see the logs of all your lambda functions through the `CloudWatch`. To access the logs, go to `CloudWatch > Log groups` in your amazon account.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3hjcqme99fztwoz18eco.png)

You can see the logs from here. Let's check the `sqs-handler` logs. It must have printed the received message.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ppgk16mjjtwujn005lhz.png)


## References

- [Amazon SQS](https://aws.amazon.com/sqs/)
- [AWS Lambda and SQS](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html)
- [AWS Lambda Function Handler in Go](https://docs.aws.amazon.com/lambda/latest/dg/golang-handler.html)
- [Serverless Framework and SQS](https://aws.amazon.com/lambda/serverless-architectures-learn-more/)

Get the full source code from [this github repository](https://github.com/datumbrain/background-process-using-sqs).