Infrastructure as Code on AWS using Go and Pulumi
Tags: iac, aws, pulumi, go
Published: 2021-11-09 12:06:47

---
title: Infrastructure as Code on AWS using Go and Pulumi
published: true
description: An introduction to Pulumi 
tags: iac, aws, pulumi, go, golang
//cover_image: https://direct_url_to_image.jpg
---
When we talk about Infrastructure as Code or IaC, the first tool that comes to mind is [Terraform](https://terraform.io). Terraform, created by HashiCorp, has become the standard for documentation and infrastructure management, but its declarative language, HCL (HashiCorp Configuration Language), has some limitations. The main limitation is not being a programming language but a configuration one.

Some alternatives have been emerging to fulfill these needs, such as:

- [AWS Cloud Development Kit](https://aws.amazon.com/pt/cdk/), Amazon's solution that allows us to use TypeScript, Python, and Java to program the infrastructure using the cloud provider's solutions;

- [Pulumi](https://www.pulumi.com), which allows us to use TypeScript, JavaScript, Python, Go, and C# to program infrastructures using solutions from AWS, Microsoft Azure, Google Cloud, and Kubernetes installations.
I will introduce Pulumi, using the Go language to create some infrastructure examples on AWS.

## Installation

To make use of Pulumi, we first need to install its command-line application. Following the [documentation](https://www.pulumi.com/docs/get-started/aws/begin/), I installed it on my macOS using the command:

    brew install pulumi

On the website, you can see how to install it on Windows and Linux.

## Configure AWS Account Access

Since I will use AWS in this example, the next necessary step is to configure the credentials. For that, I got my access key and secret from the AWS dashboard and set the required environment variables:

    export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
    export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>

## Creating the project

With the initial dependencies configured, we can now create the project:

    mkdir post-pulumi
    cd post-pulumi
    pulumi new aws-go

One of the creation steps requires setting up an account on the Pulumi website. For that, the command-line application opens the browser for this step to be completed. So I logged in with my Github account, completed the registration, returned to the terminal, and continued the project creation without any problems.

You can see the result of running the command can [at this link](https://app.warp.dev/block/ngaPDDNEPbf5eTuMgDHsqW). In addition, at the end of the process, it installs all the necessary dependencies for creating the project in Go.

## Files created

Looking at the directory contents, we can see that some configuration files and a `main.go` were created.

*Pulumi.yaml*

```yaml
name: post-pulumi
runtime: go
description: A minimal AWS Go Pulumi program

```

*Pulumi.dev.yaml*

```yaml
config:
  aws:region: us-east-1
```

*main.go*

```go
package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an AWS resource (S3 Bucket)
		bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
		if err != nil {
			return err
		}

		// Export the name of the bucket
		ctx.Export("bucketName", bucket.ID())
		return nil
	})
}
```

When running

    pulumi up

The bucket was created in S3, as the code indicates.

And the command:

    pulumi destroy

Destroy all the resources, in this case, the S3 bucket.

## First example - creating a static page in S3

Now let's do some more complex examples.

The first step is to create a static page, which we are going to deploy:

    mkdir static


Inside this directory, I created the file:

*static/index.html*
```html
<html>
    <body>
        <h1>Hello, Pulumi!</h1>
    </body>
</html>
```


I changed *main.go* to reflect the new structure:

```go
package main

import (
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Create an AWS resource (S3 Bucket)
		bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
			Website: s3.BucketWebsiteArgs{
				IndexDocument: pulumi.String("index.html"),
			},
		})
		if err != nil {
			return err
		}

		// Export the name of the bucket
		ctx.Export("bucketName", bucket.ID())

		_, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{
			Acl:         pulumi.String("public-read"),
			ContentType: pulumi.String("text/html"),
			Bucket:      bucket.ID(),
			Source:      pulumi.NewFileAsset("static/index.html"),
		})
		if err != nil {
			return err
		}
		ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
		return nil
	})
}

```


To update run:

    pulumi up

And confirm the change.

The code snippet:

```go
ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
```


Generate as output the address to access *index.html*:

```
Outputs:
  + bucketEndpoint: "http://my-bucket-357877e.s3-website-us-east-1.amazonaws.com"
```

The case above is a straightforward example, but it already demonstrates the power of the tool. So let's make things a little more complex and fun now.

## Second example - a site inside a container

Let's create a *Dockerfile* with a web server to host our static content:

*static/Dockerfile*

```Dockerfile
FROM golang

ADD . /go/src/foo

WORKDIR /go/src/foo
RUN go build -o /go/bin/main

ENTRYPOINT /go/bin/main

EXPOSE 80
```

Let's now create the *static/main.go* file, which will be our web server:

```go
package main

import (
	"log"
	"net/http"
)

func main() {
	r := http.NewServeMux()
	fileServer := http.FileServer(http.Dir("./"))
	r.Handle("/", http.StripPrefix("/", fileServer))
	s := &http.Server{
		Addr:    ":80",
		Handler: r,
	}
	log.Fatal(s.ListenAndServe())
}
```

Let's change *main.go* to include the infrastructure of an ECS cluster and everything else needed to run our container:

```go
package main

import (
	"encoding/base64"
	"fmt"
	"strings"

	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
	elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		// Read back the default VPC and public subnets, which we will use.
		t := true
		vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
		if err != nil {
			return err
		}
		subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
		if err != nil {
			return err
		}

		// Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
		webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
			VpcId: pulumi.String(vpc.Id),
			Egress: ec2.SecurityGroupEgressArray{
				ec2.SecurityGroupEgressArgs{
					Protocol:   pulumi.String("-1"),
					FromPort:   pulumi.Int(0),
					ToPort:     pulumi.Int(0),
					CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
				},
			},
			Ingress: ec2.SecurityGroupIngressArray{
				ec2.SecurityGroupIngressArgs{
					Protocol:   pulumi.String("tcp"),
					FromPort:   pulumi.Int(80),
					ToPort:     pulumi.Int(80),
					CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
				},
			},
		})
		if err != nil {
			return err
		}

		// Create an ECS cluster to run a container-based service.
		cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
		if err != nil {
			return err
		}

		// Create an IAM role that can be used by our service's task.
		taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
			AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
		})
		if err != nil {
			return err
		}
		_, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
			Role:      taskExecRole.Name,
			PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
		})
		if err != nil {
			return err
		}

		// Create a load balancer to listen for HTTP traffic on port 80.
		webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
			Subnets:        toPulumiStringArray(subnet.Ids),
			SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
		})
		if err != nil {
			return err
		}
		webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
			Port:       pulumi.Int(80),
			Protocol:   pulumi.String("HTTP"),
			TargetType: pulumi.String("ip"),
			VpcId:      pulumi.String(vpc.Id),
		})
		if err != nil {
			return err
		}
		webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
			LoadBalancerArn: webLb.Arn,
			Port:            pulumi.Int(80),
			DefaultActions: elb.ListenerDefaultActionArray{
				elb.ListenerDefaultActionArgs{
					Type:           pulumi.String("forward"),
					TargetGroupArn: webTg.Arn,
				},
			},
		})
		if err != nil {
			return err
		}

		//create a new ECR repository
		repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
		if err != nil {
			return err
		}

		repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
			creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
				RegistryId: rid,
			})
			if err != nil {
				return nil, err
			}
			data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
			if err != nil {
				fmt.Println("error:", err)
				return nil, err
			}

			return strings.Split(string(data), ":"), nil
		}).(pulumi.StringArrayOutput)
		repoUser := repoCreds.Index(pulumi.Int(0))
		repoPass := repoCreds.Index(pulumi.Int(1))

		//build the image
		image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
			Build: docker.DockerBuildArgs{
				Context: pulumi.String("./static"),
			},
			ImageName: repo.RepositoryUrl,
			Registry: docker.ImageRegistryArgs{
				Server:   repo.RepositoryUrl,
				Username: repoUser,
				Password: repoPass,
			},
		})
		if err != nil {
			return err
		}

		containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
			fmtstr := `[{
				"name": "my-app",
				"image": %q,
				"portMappings": [{
					"containerPort": 80,
					"hostPort": 80,
					"protocol": "tcp"
				}]
			}]`
			return fmt.Sprintf(fmtstr, name), nil
		}).(pulumi.StringOutput)

		// Spin up a load balanced service running NGINX.
		appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
			Family:                  pulumi.String("fargate-task-definition"),
			Cpu:                     pulumi.String("256"),
			Memory:                  pulumi.String("512"),
			NetworkMode:             pulumi.String("awsvpc"),
			RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
			ExecutionRoleArn:        taskExecRole.Arn,
			ContainerDefinitions:    containerDef,
		})
		if err != nil {
			return err
		}
		_, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
			Cluster:        cluster.Arn,
			DesiredCount:   pulumi.Int(5),
			LaunchType:     pulumi.String("FARGATE"),
			TaskDefinition: appTask.Arn,
			NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
				AssignPublicIp: pulumi.Bool(true),
				Subnets:        toPulumiStringArray(subnet.Ids),
				SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
			},
			LoadBalancers: ecs.ServiceLoadBalancerArray{
				ecs.ServiceLoadBalancerArgs{
					TargetGroupArn: webTg.Arn,
					ContainerName:  pulumi.String("my-app"),
					ContainerPort:  pulumi.Int(80),
				},
			},
		}, pulumi.DependsOn([]pulumi.Resource{webListener}))
		if err != nil {
			return err
		}
		// Export the resulting web address.
		ctx.Export("url", webLb.DnsName)
		return nil
	})
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
	var res []pulumi.StringInput
	for _, s := range a {
		res = append(res, pulumi.String(s))
	}
	return pulumi.StringArray(res)
}

```

Complex? Yes, but this complexity is inherent to AWS features and not Pulumi. We would have similar complexity if we were using Terraform or CDK.

Before running our code, we need to download the new dependencies:

    go get github.com/pulumi/pulumi-docker
    go get github.com/pulumi/pulumi-docker/sdk/v3/go/docker


Now just run the command:

    pulumi up

The execution output will generate the URL of the *load balancer*, which we will use to access the contents of our container in execution.

## Reorganizing the code

Now we can start making use of the advantages of a complete programming language like Go. For example, we could use language features like functions, concurrency, conditionals, etc. In this example, we are going to organize our code better. For this, I created the *iac* directory and the *iac/fargate.go* file. After that, I moved most of the logic from `main.go` to the new file:

```go
package iac

import (
	"encoding/base64"
	"fmt"
	"strings"

	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
	elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
	"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
	"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func FargateRun(ctx *pulumi.Context) error {

	// Read back the default VPC and public subnets, which we will use.
	t := true
	vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
	if err != nil {
		return err
	}
	subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
	if err != nil {
		return err
	}

	// Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
	webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
		VpcId: pulumi.String(vpc.Id),
		Egress: ec2.SecurityGroupEgressArray{
			ec2.SecurityGroupEgressArgs{
				Protocol:   pulumi.String("-1"),
				FromPort:   pulumi.Int(0),
				ToPort:     pulumi.Int(0),
				CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
			},
		},
		Ingress: ec2.SecurityGroupIngressArray{
			ec2.SecurityGroupIngressArgs{
				Protocol:   pulumi.String("tcp"),
				FromPort:   pulumi.Int(80),
				ToPort:     pulumi.Int(80),
				CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
			},
		},
	})
	if err != nil {
		return err
	}

	// Create an ECS cluster to run a container-based service.
	cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
	if err != nil {
		return err
	}

	// Create an IAM role that can be used by our service's task.
	taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
		AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
	})
	if err != nil {
		return err
	}
	_, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
		Role:      taskExecRole.Name,
		PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
	})
	if err != nil {
		return err
	}

	// Create a load balancer to listen for HTTP traffic on port 80.
	webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
		Subnets:        toPulumiStringArray(subnet.Ids),
		SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
	})
	if err != nil {
		return err
	}
	webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
		Port:       pulumi.Int(80),
		Protocol:   pulumi.String("HTTP"),
		TargetType: pulumi.String("ip"),
		VpcId:      pulumi.String(vpc.Id),
	})
	if err != nil {
		return err
	}
	webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
		LoadBalancerArn: webLb.Arn,
		Port:            pulumi.Int(80),
		DefaultActions: elb.ListenerDefaultActionArray{
			elb.ListenerDefaultActionArgs{
				Type:           pulumi.String("forward"),
				TargetGroupArn: webTg.Arn,
			},
		},
	})
	if err != nil {
		return err
	}

	repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
	if err != nil {
		return err
	}

	repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
		creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
			RegistryId: rid,
		})
		if err != nil {
			return nil, err
		}
		data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
		if err != nil {
			fmt.Println("error:", err)
			return nil, err
		}

		return strings.Split(string(data), ":"), nil
	}).(pulumi.StringArrayOutput)
	repoUser := repoCreds.Index(pulumi.Int(0))
	repoPass := repoCreds.Index(pulumi.Int(1))

	image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
		Build: docker.DockerBuildArgs{
			Context: pulumi.String("./static"),
		},
		ImageName: repo.RepositoryUrl,
		Registry: docker.ImageRegistryArgs{
			Server:   repo.RepositoryUrl,
			Username: repoUser,
			Password: repoPass,
		},
	})
	if err != nil {
		return err
	}

	containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
		fmtstr := `[{
				"name": "my-app",
				"image": %q,
				"portMappings": [{
					"containerPort": 80,
					"hostPort": 80,
					"protocol": "tcp"
				}]
			}]`
		return fmt.Sprintf(fmtstr, name), nil
	}).(pulumi.StringOutput)

	// Spin up a load balanced service running NGINX.
	appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
		Family:                  pulumi.String("fargate-task-definition"),
		Cpu:                     pulumi.String("256"),
		Memory:                  pulumi.String("512"),
		NetworkMode:             pulumi.String("awsvpc"),
		RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
		ExecutionRoleArn:        taskExecRole.Arn,
		ContainerDefinitions:    containerDef,
	})
	if err != nil {
		return err
	}
	_, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
		Cluster:        cluster.Arn,
		DesiredCount:   pulumi.Int(5),
		LaunchType:     pulumi.String("FARGATE"),
		TaskDefinition: appTask.Arn,
		NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
			AssignPublicIp: pulumi.Bool(true),
			Subnets:        toPulumiStringArray(subnet.Ids),
			SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
		},
		LoadBalancers: ecs.ServiceLoadBalancerArray{
			ecs.ServiceLoadBalancerArgs{
				TargetGroupArn: webTg.Arn,
				ContainerName:  pulumi.String("my-app"),
				ContainerPort:  pulumi.Int(80),
			},
		},
	}, pulumi.DependsOn([]pulumi.Resource{webListener}))
	if err != nil {
		return err
	}
	// Export the resulting web address.
	ctx.Export("url", webLb.DnsName)
	return nil
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
	var res []pulumi.StringInput
	for _, s := range a {
		res = append(res, pulumi.String(s))
	}
	return pulumi.StringArray(res)
}

```

The next step was to configure the `iac` directory to be a Go language module:

	cd iac
	go mod init github.com/eminetto/post-pulumi/iac
	cd ..
	go mod edit -replace github.com/eminetto/post-pulumi/iac=./iac
	go mod tidy

Our `main.go` can now be simplified:

```go
package main

import (
	"github.com/eminetto/post-pulumi/iac"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
	pulumi.Run(func(ctx *pulumi.Context) error {
		return iac.FargateRun(ctx)
	})
}

```

That way, we can better manage the structure of the code that will handle AWS resources. We can reuse this code in other projects, use environment variables, write [tests](https://www.pulumi.com/docs/guides/testing/), or whatever else our imagination allows.

## Conclusion

Using a tool like Pulumi significantly increases the [range of options](https://www.pulumi.com/docs/) that we can use in building a project's infrastructure while maintaining readability, [code reuse](HTTPS ://www.pulumi.com/docs/guides/pulumi-packages/) and organization.