EC2 Charges — They are NAT a joke
Tags: aws, cloud, tutorial
Published: 2023-03-29 11:31:52

How often do you check your cloud provider bill and costs? Do you know what your biggest costs in the cloud are? Have you checked if all the costs are justified?

This story begins with me doing just that...

I was investigating a project that is running in AWS. One of the biggest costs is **Elastic Compute Cloud** (EC2), which accounts for nearly 70% of the total bill:

![AWS bill](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jhfiwvt3k0bkj723kfci.png)

---

However, when I was having a look at these `EC2` costs in more detail, there was an extra `EC2-Other` service which accounted for a hefty amount of the total `EC2` costs:

![EC2 bill](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8jzjeea68h6jhhd8pe4w.png)

---

## EC2-Other — sorry, what? 😐
After some research into this `EC2-Other` service, it sounds almost like it is an expected cost, and it was just part and parcel of using `EC2`:

> The EC2-Other category includes multiple service-related usage types, tracking costs associated [with] Amazon EBS volumes and snapshots, elastic IP addresses, NAT gateways, data transfer, and more.

– [EC2 Cost Explorer](https://aws.amazon.com/blogs/aws-cloud-financial-management/tips-and-tricks-for-exploring-your-data-in-aws-cost-explorer-part-2/)

So, for a moment I stopped looking into these costs and just assumed they were normal.

## EC2-Other — sorry, no. ✋
A few days had passed, and I was still investigating that same project. This time I was looking specifically at a monthly bill breakdown. This was when I noticed something odd:

![NAT Gateway charges](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z5msetq3m9hyhx9fujf9.png)

---

Did you spot it? 

**52,664 GB** (~53 TB) of data being processed through our **NAT gateway** — that is suspicious. We would expect to see some data being transferred through the NAT gateway but that amount is absurd.

My investigations began, and well it did not take long to find an answer.

I determined that every time a container in `EC2` was pulling an image from `ECR` (Elastic Container Registry) it was being transferred through our **NAT gateway**. Every file that was being transferred from `S3` to a container or vice versa, this was going through the gateway. Even our logs being sent from any container to `CloudWatch`, through the gateway…

Now the costs make a little more sense.

## EC2-Other — sorry, bye 👋
Do not panic — there is a solution — enter [VPC Endpoints](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/vpc-endpoints.html), also known as [AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html).

> You can use AWS PrivateLink to connect the resources in your VPC to services using private IP addresses, as if those services were hosted directly in your VPC.

– [AWS PrivateLink concepts — Amazon Virtual Private Cloud](https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html)

If you use your console for changes to infrastructure then follow this tutorial on how to set these endpoints up for S3, ECR and logs: [Create Private Links via Console](https://www.easydeploy.io/blog/how-to-create-private-link-for-ecr-to-ecs-containers-to-save-nat-gatewayec2-other-charges/) or also this one provided by AWS: [New VPC Endpoint for S3](https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/).

However, if you use `CDK` to deploy your infrastructure, like a hero, then here is how to set up your 3 new VPC endpoints. I am using Kotlin but if you use TypeScript or another language it shouldn’t be too hard to adjust.

## CDK (In Kotlin)
Here are your imports for this setup:

```
import software.amazon.awscdk.services.ec2.GatewayVpcEndpoint
import software.amazon.awscdk.services.ec2.GatewayVpcEndpointProps
import software.amazon.awscdk.services.ec2.GatewayVpcEndpointAwsService
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpoint
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpointProps
import software.amazon.awscdk.services.ec2.InterfaceVpcEndpointService
```
You’ll need to have your VPC available in this stack.

## Gateway Endpoint
Firstly here is your Gateway Endpoint for S3:

```
private val s3VpcEndpoint = GatewayVpcEndpoint(
  scope,
  "your-s3-endpoint",
  GatewayVpcEndpointProps.builder()
    .vpc(vpc)
    .service(GatewayVpcEndpointAwsService.S3)
    .build()
)
```
At the time of writing this, **gateway endpoints** were only available for S3 and DynamoDB — otherwise you must use an **Interface Endpoint**. Gateway endpoints for S3 are offered at no cost and the routes are managed through route tables.

## Interface Endpoint
Interface endpoints are priced at $0.01/per AZ/per hour. Cost depends on the Region, check [current pricing](https://aws.amazon.com/privatelink/pricing/). Data transferred through the interface endpoint is charged at $0.01/per GB (depending on Region).

For the Interface Endpoint for **ECR** you need to have your security groups available:

It is important to include the correct security groups here or else you could have issues with your containers not being able to pull from ECR.

```
private val ecrVpcEndpoint = InterfaceVpcEndpoint(
  scope,
  "your-ecr-endpoint",
  InterfaceVpcEndpointProps.builder()
.service(InterfaceVpcEndpointService("com.amazonaws.$yourRegion.ecr.dkr"))
    .vpc(vpc)
    .privateDnsEnabled(true)
    .securityGroups(
      listOf(exampleSecurityGroup)
    )
    .build()
)
```
Then for the Interface Endpoint for **Logs**:

```
private val logsVpcEndpoint = InterfaceVpcEndpoint(
  scope,
  "your-logs-endpoint",
  InterfaceVpcEndpointProps.builder()
    .service(InterfaceVpcEndpointService("com.amazonaws.$yourRegion.logs"))
    .vpc(vpc)
    .privateDnsEnabled(true)
    .securityGroups(
      listOf(exampleSecurityGroup)
    )
    .build()
)
```

Sadly with CDK you cannot add a name tag to these endpoints ([see issue here](https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/196)) so when you deploy you will see something that looks like this:

![Resulting endpoints](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z01mhavkgce5txa0n9m2.png)

---

So, I pushed these changes to our production environment halfway through the day 23rd June, but I think you can see that quite clearly here:

![Resulting costs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5rp00oapujyulpqmksyu.png)

---

So after leaving our system to run for a little while with these endpoints in place we started to see the `EC2-Other` costs drop significantly. For us, our costs dropped from over $2000 per month to just over $200!

I hope that you too can find a use for these VPC Endpoints to help lower your costs!

There is an increase, of course, in the costs for your `VPC` for these endpoints, but I am not too worried about the $1 increase there:

![Increased VPC costs](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v8dssbipkos6kbl6hxni.png)

---

Thanks for coming on this journey with me, and I hope this can help you save some precious dolla bills 💲

Emma.