AWS SQS with spring boot & Localstack with Junit Testing
Tags: aws, testing, java
Published: 2021-05-17 09:31:02

## Preface

Building microservices architecture often involves creating microservices communicating using a message bus or any means of loosely coupled means such as AWS Simple Queue Service, dearly called AWS SQS.

## What are we building
Complete Code: [spring-boot-localstack](https://github.com/sanket0612/spring-boot-localstack)
Here is step-by-step guide of setting up a simple spring-boot web application talking to AWS SQS using localstack to mock the AWS environment.

This includes bare minimum configurations required to create a web app communicating using SQS only.

## Basic Definitions:
1. [localstack](https://github.com/localstack/localstack): Simply a tool to mock AWS Cloud Provider in your local environment, to help develop cloud applications.
2. [Junit5](https://junit.org/junit5/): A testing framework for Java application based on Java8.
3. [awaitability](https://github.com/awaitility/awaitility): A tool to express expectations for asynchronous system in an easy and concise manner.
4. Docker: Run any process or application in a containerised manner.

## Pre-requisites:
1. Basic knowledge of Java and spring-boot.
2. Environment setup for running Docker e.g. docker for mac, or just a happy linux system
3. Can setup [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to play around with application. A command line utility to interact with AWS services.
4. A Familiar IDE.

## Setup Basic Project

1. Get on to the second-best website on internet. [Spring Initializer](https://start.spring.io/)
2. Create a spring project preferably with 2.3 Spring boot version and following dependencies
 * Spring web
 * Lombok (java utility to avoid writing boilerplate code)
 * AWS Simple Queue Service
3. Also add following dependencies externally in pom.xml

```
<dependency>
	<groupId>org.awaitility</groupId>
	<artifactId>awaitility</artifactId>
	<version>3.1.3</version>
	<scope>test</scope>
</dependency>
```
### Create simple event and data models to send and receive messages:
```
@Data
@AllArgsConstructor
@NoArgsConstructor
@Builder
public class SampleEvent {

  private String eventId;
  private String version;
  private String type;
  private ZonedDateTime eventTime;
  private EventData data;
}

@Data
@AllArgsConstructor
@NoArgsConstructor
@Builder
public class EventData {

  private String name;
  private int age;
  private String description;
  private EventType eventType;

  public enum EventType {
    CREATED, PROCESSED
  }
}
```
### Create a simple controller annotated with _@SqsListener_ to listen to a queue.
```
@SqsListener(value = "${cloud.aws.sqs.incoming-queue.url}", deletionPolicy = SqsMessageDeletionPolicy.ON_SUCCESS)
  private void consumeFromSQS(SampleEvent sampleEvent) {
    log.info("Receive message {}", sampleEvent);
    //do some processing
    sampleEvent.setEventTime(ZonedDateTime.now());
    sampleEvent.getData().setEventType(EventData.EventType.PROCESSED);
    amazonSQSAsync.sendMessage(outgoingQueueUrl, mapper.writeValueAsString(sampleEvent));
  }
```

### Spring property configurations:

setup application.yml with aws sqs properties, such as:
```
localstack:
  host: localhost

cloud:
  aws:
    credentials:
      access-key: some-access-key
      secret-key: some-secret-key
    sqs:
      incoming-queue:
        url: http://localhost:4576/queue/incoming-queue
        name: incoming-queue
      outgoing-queue:
        name: outgoing-queue
        url: http://localhost:4576/queue/outgoing-queue
    stack:
      auto: false
    region:
      static: eu-central-1

logging:
  level:
    com:
      amazonaws:
        util:
          EC2MetadataUtils: error
```
__Notes__:
a. Aws credentials can also be set up environmental variables or in a .aws/credentials file, (further read 😉)
b. define name and url of queues so application can listen to and write to queue. Localstack runs SQS service on port number: 4576
c. The below logging property is to avoid having multiple lines of error where application tries to connect to EC2 instance of localstack. (workaround 😏)

### AWS Local SQS configs:
Inside the java configuration for SQS we create some beans to allow our application to talk to SQS service provided by localstack. You could add a profile for each config when deploying app in production i.e. actual AWS environment.

```
@Bean
//endpoint config for connecting to localstack and not actual aws environment.
  public AwsClientBuilder.EndpointConfiguration endpointConfiguration(){
    return new AwsClientBuilder.EndpointConfiguration("http://localhost:4576", region);
  }

  @Bean
  @Primary
//This bean will be used for communicating to AWS SQS
  public AmazonSQSAsync amazonSQSAsync(final AwsClientBuilder.EndpointConfiguration endpointConfiguration){
    AmazonSQSAsync amazonSQSAsync = AmazonSQSAsyncClientBuilder
        .standard()
        .withEndpointConfiguration(endpointConfiguration)
        .withCredentials(new AWSStaticCredentialsProvider(
            new BasicAWSCredentials(awsAccesskey, awsSecretKey)
        ))
        .build();
    createQueues(amazonSQSAsync, "incoming-queue");
    createQueues(amazonSQSAsync, "outgoing-queue");
    return amazonSQSAsync;
  }
//create initial queue so our application can talk to it
  private void createQueues(final AmazonSQSAsync amazonSQSAsync,
                            final String queueName){
    amazonSQSAsync.createQueue(queueName);
    var queueUrl = amazonSQSAsync.getQueueUrl(queueName).getQueueUrl();
    amazonSQSAsync.purgeQueueAsync(new PurgeQueueRequest(queueUrl));
  }
```

We can use QueueMessagingTemplate for sending and receiving messages from AWS SQS 
```
 @Bean
  public QueueMessagingTemplate queueMessagingTemplate(AmazonSQSAsync amazonSQSAsync){
    return new QueueMessagingTemplate(amazonSQSAsync);
  }
```

_Also setup QueueMessageHandlerFactory so it can convert incoming messages from SQS as String to the actual object you want, in this case Simple Event, using objectmapper._ 
You can configure objectmapper separately. Add your custom deserialiser o such by registering  your module or add custom datetime conversion.

```
@Bean
  public QueueMessageHandlerFactory queueMessageHandlerFactory(MessageConverter messageConverter) {

    var factory = new QueueMessageHandlerFactory();
    factory.setArgumentResolvers(singletonList(new PayloadArgumentResolver(messageConverter)));
    return factory;
  }

  @Bean
  protected MessageConverter messageConverter(ObjectMapper objectMapper) {

    var converter = new MappingJackson2MessageConverter();
    converter.setObjectMapper(objectMapper);
    // Serialization support:
    converter.setSerializedPayloadClass(String.class);
    // Deserialization support: (suppress "contentType=application/json" header requirement)
    converter.setStrictContentTypeMatch(false);
    return converter;
  }
```

Finally add this docker compose yaml asking docker to create a localstack container:

```
version: '3.0'

services:
  localstack:
    image: localstack/localstack:0.10.7
    environment:
      - DEFAULT_REGION=eu-central-1
      - SERVICES=sqs
    ports:
      - "4576:4576"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
```

### Starting the application
1. start the localstack using: **docker-compose up** (in same directory as docker file)
2. run application using: **mvn spring-boot:run**
3. send a message to sqs using AWS CLI for e.g.:
```
aws --endpoint="http://localhost:4576" --region=eu-central-1 sqs send-message --queue-url http://localhost:4576/queue/incoming-queue --message-body '{
  "eventId": "some-event-id-1",
  "eventTime": "2016-09-03T16:35:13.273Z",
  "type": "some-type",
  "version": "1.0",
  "data": {
    "name": "Dev. to",
    "age": 20,
    "description": "User created",
    "eventType": "CREATED"
  }
}'
```
4. You should see log messages of event received and forwarded.
5. Also added a Junit Test which uses same configuration as for local run. 

__Thank you!!__