EC2 real network bandwidth
Tags: devops, aws, terraform, performance
Published: 2024-05-31 13:49:12

EC2 instances does not always provide exact numbers on their network capacity. In some cases where we need to optimize our network traffic we might need to know the exact numbers for our own instances. 

## iperf & Terraform

One way to achieve this is using [`iperf3`](https://iperf.fr/). This tool uses a client-server model, which is ideal for testing network performance between two hosts. The server listens for incoming test requests, and the client initiates the test, making it easy to measure the capacity of the network link between them.

The next step involves setting up two EC2 instances. The first will act as a server, which will help us determine its actual network capacity. The second will act as a client. The client instance should have a higher network capacity than the server to ensure that the server's network link is fully utilized during the test.

To implement this, we need to create a pair of EC2 instances along with their corresponding resources, such as roles and security groups, in our AWS account. Doing this manually for every EC2 instance type we need to measure could be tedious, so we'll use [Terraform](https://www.terraform.io/) for this task.

Following are the important parts of terraform code to setup this test environment to measure network capacity of `m3.medium` instances.

``` HCL
resource "aws_instance" "iperf_server" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m3.medium"

  subnet_id       = data.aws_subnet.current.id
  security_groups = [aws_security_group.iperf.id]

  iam_instance_profile = aws_iam_instance_profile.iperf.name

  tags = {
    Name = "${var.name}-iperf-server"
  }

  key_name = aws_key_pair.iperf.key_name
  connection {
    host        = coalesce(self.public_ip, self.private_ip)
    type        = "ssh"
    user        = var.ssh_user
    private_key = tls_private_key.iperf.private_key_pem
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum -y update",
      "sudo yum -y install iperf3"
    ]
  }
}

resource "aws_instance" "iperf_client" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "m5.12xlarge"

  subnet_id       = data.aws_subnet.current.id
  security_groups = [aws_security_group.iperf.id]

  iam_instance_profile = aws_iam_instance_profile.iperf.name

  tags = {
    Name = "${var.name}-iperf-client"
  }

  key_name = aws_key_pair.iperf.key_name
  connection {
    host        = coalesce(self.public_ip, self.private_ip)
    type        = "ssh"
    user        = var.ssh_user
    private_key = tls_private_key.iperf.private_key_pem
  }

  provisioner "remote-exec" {
    inline = [
      "sudo yum -y update",
      "sudo yum -y install iperf3"
    ]
  }
}

resource "aws_security_group" "iperf" {

  name = "${var.name}-iperf-seg"
  description = "Allow inbound traffic for iperf"

  vpc_id = data.aws_vpc.current.id

  ingress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    self      = true
  }

  ingress {
    from_port   = 5001  # Assuming iperf server listens on port 5001
    to_port     = 5001
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "port 22"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.ssh_cidr_ingress_blocks
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    "Name" : "${var.name}-iperf-seg"
  }

}

output "server_private_ip" {
  value       = aws_instance.iperf_server.private_ip
}

output "server_public_ip" {
  value       = aws_instance.iperf_server.public_ip
}

output "client_public_ip" {
  value       = aws_instance.iperf_client.public_ip
}
```
## Conducting the test

Once we have created our test env, we just need to ssh into the server instance and start `iperf` in server mode

``` bash
$ iperf -s
```

Then we’ll ssh into the client and start the test attaching to the server using its private ip to reduce extra network overhead.

``` bash
$ iperf -c <server_private_ip>
```

If we plan to test multiple instance types, we can create a script to run the tests as follows.

``` bash
terraform_path=.

vars=$(terraform -chdir=$terraform_path output -json)
client_public_ip=$(echo $vars | jq -r '.client_public_ip.value')
server_public_ip=$(echo $vars | jq -r '.server_public_ip.value')
server_private_ip=$(echo $vars | jq -r '.server_private_ip.value')

keypair="$terraform_path/iperf-keypair.pem"

ssh -i $keypair \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    ec2-user@$server_public_ip "iperf3 -s -D"

ssh -i $keypair \
    -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
    ec2-user@$client_public_ip -t "iperf3 -c $server_private_ip -t 150 -i 60"
```

## Real figures

I ran this test for some instance types. The results are as follows:

| Instance Type | Bandwidth |
| --- | --- |
| m3.medium | 681 Mbits/sec |
| c4.large | 628 Mbits/sec |
| m4.large | 768 Mbits/sec |
| r4.large | 9.53 Gbits/sec |
| i3.large | 4.97 Gbits/sec |
| t2.nano | 1.04 Gbits/sec |
| t3.medium | 4.52 Gbits/sec |

Remember that burstable instances, such as the `t2` and `t3` families, incorporate the concept of network credits. These credits permit them to "burst" their network performance beyond the baseline level for short periods, as needed. Therefore, the figures in the table are not directly comparable between burstable and non-burstable instance types.

To establish the baseline network performance of burstable instances, we need to conduct the test for a duration sufficient to use up the accumulated network credits.