AWS: VPC Flow Logs — an overview and example with CloudWatch Logs Insights
Tags: monitoring, aws, security, todayilearned
Published: 2022-07-23 07:19:11

---
title: AWS: VPC Flow Logs — an overview and example with CloudWatch Logs Insights
published: true
date: 2022-07-19 06:01:26 UTC
tags: monitoring, aws, security, todayilearned
canonical_url: https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/
---

### AWS: VPC Flow Logs — an overview and example with CloudWatch Logs Insights

![](https://cdn-images-1.medium.com/max/1024/1*RotXy7XKKz_-U59wSQ57sA.jpeg)

AWS VPC Flow Logs allow you to log traffic information between network interfaces in a VPC. Further, these logs can be stored in AWS S3 or sent to AWS CloudWatch Logs, while enabling traffic logging does not affect the performance of the network interface in any way.

Let’s briefly review the basic concepts, and available settings, and set up Flow Logs for VPC with data transfer for analysis to CloduWatch Logs.

- [VPC Flow Logs overview](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_Flow_Logs_overview)
- [VPC Flow Logs use cases](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_Flow_Logs_use_cases)
- [Flow Log record — fields](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#Flow_Log_record_-_fields)
- [VPC Flow Logs Limitations](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_Flow_Logs_Limitations)
- [Create VPC Flow Log](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#Create_VPC_Flow_Log)
- [CloudWatch Logs Log Group](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#CloudWatch_Logs_Log_Group)
- [IAM Policy and IAM Role](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#IAM_Policy_and_IAM_Role)
- [VPC — enabling Flow Logs](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_-_enabling_Flow_Logs)
- [CloudWatch Logs Insights](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#CloudWatch_Logs_Insights)
- [VPC Flow Log — Custom format](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_Flow_Log_-_Custom_format)
- [Flow Log Custom format, and CloudWatch Logs Insights](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#Flow_Log_Custom_format_and_CloudWatch_Logs_Insights)
- [Logs Insights examples](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#Logs_Insights_examples)
- [Useful links](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#Useful_links)
- [VPC Flow Logs](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#VPC_Flow_Logs)
- [CloudWatch Logs](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/#CloudWatch_Logs)

### VPC Flow Logs overview

Logs can be enabled for an entire VPC, for a subnet, or for an external surface. If enabled for the entire VPC — logging will be enabled for all interfaces of the VPC.

Services for which you can use Flow Logs:

- Elastic Load Balancing
- Amazon RDS
- Amazon ElastiCache
- Amazon Redshift
- Amazon WorkSpaces
- NAT gateways
- Transit gateways

The data is recorded as an [_flow log records_](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-log-records), and uses a record with predefined fields.

#### VPC Flow Logs use cases

What can be tracked with Flow logs?

- SecuirtyGroup/Network Access List rules — blocked requests will be marked as REJECTED
- what we implement logs for us — to get a picture of the traffic between VPCs and services in order to understand who consumes the most traffic, where and how much cross-AZ traffic, and so on
- monitoring remote logins to the system — monitor ports 22 (SSH), 3389 (RDP)
- port scan tracking

#### Flow Log record — fields

Each entry in the log is data about the IP traffic received during the aggregation interval and is a line with fields separated by spaces, where each field contains information about the data transfer, for example — Source IP, Destination IP, and a protocol.

By default, the following format is used:

```
${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
```

See the _Available fields_ table in the [documentation](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-log-records), everything in the Version 2 column is included in the default format.

When creating Flow Logs, we can use the _default_ format, or create our own — we will consider it below:

![](https://cdn-images-1.medium.com/max/606/0*hC-oV732eFONHPt6.png)

#### VPC Flow Logs Limitations

- cannot be used with EC2-Classic instances
- you can’t create logs for VPC peerings if they lead to a different account’s VPC
- after creating a log — you can not change its configuration or format of records

Also, keep in mind that:

- records to Amazon DNS are not logged but are written if your own DNS is used
- traffic to and from address 169.254.169.254 to get [EC2 instance metadata](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) is not logged
- traffic between the EC2 network interface and AWS Network Load Balancer interface is not logged

See all limitations in [Flow log limitations](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-logs-limitations).

### Create VPC Flow Log

To create a Flow Log, we need to specify:

- resource(s) whose logs we will write — VPC, subnet, or a specific network interface
- the type of traffic that we are logging (accepted traffic, rejected traffic, or all traffic)
- and where we will write the data — to the S3 bucket, or to the CloudWatch Logs

For now, let’s see what happens with CloudWatch Logs, and next time we’ll try to visualize in [Kibana](https://rtfm.co.ua/en/elastic-stack-an-overview-and-elk-installation-on-ubuntu-20-04/).

#### CloudWatch Logs Log Group

Create a Log Group:

![](https://cdn-images-1.medium.com/max/1024/0*_afGw33UrI810gIf.png)

![](https://cdn-images-1.medium.com/max/831/0*sMHA891o7CG8cDbf.png)

#### IAM Policy and IAM Role

In order for the Flow Logs service to be able to write to our CloudWatch, we need to configure its access rights.

Go to the AWS IAM, and create an IAM Policy and an IAM Role.

Start with an IAM Policy:

![](https://cdn-images-1.medium.com/max/1024/0*AoZlzgPrBhMliWVS.png)

Add the rules:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
```

Save:

![](https://cdn-images-1.medium.com/max/1024/0*cpzfqQ-tQp_ft8WJ.png)

Now, create an IAM Role.

Go to the IAM Roles, create a new one, type of EC2:

![](https://cdn-images-1.medium.com/max/1024/0*zdI--87N6R2Jb1MT.png)

Find the policy, we’ve created above, and attach it:

![](https://cdn-images-1.medium.com/max/1024/0*tnhs_uVHHgO0X5oj.png)

Set its name, save:

![](https://cdn-images-1.medium.com/max/1024/0*9iRy_1tckaqsXyru.png)

Go to the _Role Trust relationship_ (see [AWS: IAM AssumeRole — описание, примеры](https://rtfm.co.ua/aws-iam-assumerole-opisanie-primery/)), edit it — for the `Service` field set the _vpc-flow-logs.amazonaws.com_:

![](https://cdn-images-1.medium.com/max/1024/0*Do_SzfKG_CNqYCrs.jpg)

Set:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
```

Save:

![](https://cdn-images-1.medium.com/max/1024/0*TgEUopwUvbKDvwkA.png)

#### VPC — enabling Flow Logs

And finally, go to a VPC to enable Logs — click on the _Flow Logs > Create_:

![](https://cdn-images-1.medium.com/max/1024/0*jqeQMWjJnjUXULNQ.png)

Set its name, _Filter_, _Interval_:

![](https://cdn-images-1.medium.com/max/822/0*wqE2-wdXrJ6tGTsB.png)

In the _Destination_ chose CloudWatch Logs, specify the Log Group and IAM Role:

![](https://cdn-images-1.medium.com/max/823/0*PlJU6iXy99FmYoQz.png)

Format — leave _Default_.

Check the _Status_:

![](https://cdn-images-1.medium.com/max/770/0*khtH05gbWreuO-pE.png)

And in a couple of minutes we’ll see our data:

![](https://cdn-images-1.medium.com/max/1024/0*L3mF_phktLlssBUq.png)

In the Log Group you can find a first stream named by an Elastic Network Interface, from where the data is taken:

![](https://cdn-images-1.medium.com/max/1024/0*BIZWGG2dCXGfuBrK.jpg)

### CloudWatch Logs Insights

Let’s take a quick look at the CloudWatch Logs Insights.

Click on the _Queries_ to get some hints:

![](https://cdn-images-1.medium.com/max/1024/0*b60gFpwIahu6e1BH.png)

For example, to find top-15 of hosts, that served most of the packets:

![](https://cdn-images-1.medium.com/max/1024/0*u0LUaE7_0SO0fez8.png)

Or by volume of data transferred:

```
stats sum(bytes) as BytesSent by srcAddr, dstAddr
| sort BytesSent desc
```

![](https://cdn-images-1.medium.com/max/1024/0*Lfqi5ArL_EgT7NTt.png)

Okay, that’s good. But what about other forms?

For example, I’d like to see a requests direction (egress/ingress), and a value of the `pkt-dstaddr` field.

### VPC Flow Log — Custom format

See more on the [Flow log record examples](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html).

For now, we can set the following format:

```
region vpc-id az-id subnet-id instance-id interface-id flow-direction srcaddr dstaddr srcport dstport pkt-srcaddr pkt-dstaddr pkt-src-aws-service pkt-dst-aws-service traffic-path packets bytes action
```

In the CloudWatch Logs create a new Log group, call it _bttrm-eks-dev-1–21-vpc-fl-custom_, don’t forget about retention:

![](https://cdn-images-1.medium.com/max/1024/0*_3cCYugJP1vSn4uy.png)

Go back to the VPC, create a new Flow Log, and call it _bttrm-eks-dev-1–21-vpc-fl-custom_:

![](https://cdn-images-1.medium.com/max/694/0*ZBbB9YMACnLocJU1.png)

Chose the _Custom Format_ and fields, which we’d like to see. In doing so, take into account that the order of fields you’ll specify here will be used to order records in the log.

I.e. if the first field will be “_region_” — then in the final log it also will be set as the first field:

![](https://cdn-images-1.medium.com/max/733/0*E0LaLcr3jfe9vPSG.png)

And the result:

```
${region} ${vpc-id} ${az-id} ${subnet-id} ${instance-id} ${interface-id} ${flow-direction} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${pkt-srcaddr} ${pkt-dstaddr} ${pkt-src-aws-service} ${pkt-dst-aws-service} ${traffic-path} ${packets} ${bytes} ${action}
```

![](https://cdn-images-1.medium.com/max/733/0*rqqh1yrWPLTOTGeM.png)

#### Flow Log Custom format, and CloudWatch Logs Insights

But if we will go to the CloudWatch Logs Insights now, and will try any query used before, we will not get the fields, that we’ve set

![](https://cdn-images-1.medium.com/max/1024/0*2iHdSQzkWgN5ryX3.png)

So, we can see the data, but how can we split it into the fields?

In our project, I don’t think we will use CloudWatch Logs a lot, most likely data will be sent to an S3 bucket, and then to the (logz.io), therefore, I will not go deep here in detail, but let’s see the principle of operation — it will come in handy later for working with ELK.

CloudWatch Logs by default creates several meta fields that we can use in requests:

- `@message`: the "raw" data - the whole message in the text
- `@timestamp`: the time of the event
- `@logStream`: a sLog stream name

For the Custom format to see the fields, we need to use the [`parse`](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html) and pass the `@message` to it, so it will parse the content by the fields we specified:

```
parse @message "* * * * * * * * * * * * * * * * * * *" 
| as region, vpc_id, az_id, subnet_id, instance_id, interface_id, 
| flow_direction, srcaddr, dstaddr, srcport, dstport, 
| pkt_srcaddr, pkt_dstaddr, pkt_src_aws_service, pkt_dst_aws_service, 
| traffic_path, packets, bytes, action 
| sort start desc
```

Here, the number of asterisks “`*`" in the `@message` must be the same, as many fields we set - `${vpc-id}`, etc.

Also, the names of the fields must contain no dashes. I.e. the `${vpc-id}` name we need to eat as `vpc_id` (or `vpcID` - as you like).

Check:

![](https://cdn-images-1.medium.com/max/1024/0*on_P5C2smVMOPl_x.png)

Wow! Now, we got all our fields.

Beside of the parse, we can use `filter`, `display`, `stats`. See all in the [CloudWatch Logs Insights query syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html).

#### Logs Insights examples

And let’s try to make a couple of queries, for example  —  to get all requests that were blocked by a SecuirtyGroup/Network Access List — they’ll be marked as _REJECTED._

Let’s take our previous query:

```
parse @message "* * * * * * * * * * * * * * * * * * * * * *" 
| as start, end, region, vpc_id, az_id, subnet_id, instance_id, interface_id, 
| flow_direction, srcaddr, dstaddr, srcport, dstport, protocol, 
| pkt_srcaddr, pkt_dstaddr, pkt_src_aws_service, pkt_dst_aws_service, 
| traffic_path, packets, bytes, action
```

And add there:

- `filter action="REJECT"`
- `stats count(action) as redjects by srcaddr`
- `sort redjects desc`

here:

- filter by an action applied to the packet — select all _REJECTED_
- count number of events by the _action_ filed, selecting by an IP address of the source, and display it in the _redjects _column
- and sort by the _redjects_

So, the full query now will be:

```
parse @message "* * * * * * * * * * * * * * * * * * *" 
| as region, vpc_id, az_id, subnet_id, instance_id, interface_id, 
| flow_direction, srcaddr, dstaddr, srcport, dstport, 
| pkt_srcaddr, pkt_dstaddr, pkt_src_aws_service, pkt_dst_aws_service, 
| traffic_path, packets, bytes, action 
| filter action="REJECT" 
| stats count(action) as redjects by srcaddr 
| sort redjects desc
```

And its result:

![](https://cdn-images-1.medium.com/max/469/0*xMLOycFZ9Yg91X3q.png)

We also can use negative filters and combine the with the and/or operators.

For example, to remove from the output all IPs starting with the _162.142.125_  —  add a filter `filter (srcaddr not like "162.142.125.")`:

```
...
| filter action="REJECT"
| filter (srcaddr not like "162.142.125.")
| stats count(action) as redjects by srcaddr
| sort redjects desc
```

See the [Sample queries](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax-examples.html).

And add a filter to select only incoming requests — `flow_direction` == _ingress_:

```
...
| filter action="REJECT"
| filter (srcaddr not like "162.142.125.") and (flow_direction like "ingress")
| stats count(action) as redjects by flow_direction, srcaddr, dstaddr, pkt_srcaddr, pkt_dstaddr
| sort redjects desc
```

![](https://cdn-images-1.medium.com/max/926/0*T34IRcT0Bm2X60J_.png)

Now, we got the top of rejected requests, when a SecurityGroup or VPC Network Access List rule worked.

And let’s check what is the IP in the dstaddr - who was the final destination?

Go to the E_C2 > Network Interfaces_, find by the Private IP:

![](https://cdn-images-1.medium.com/max/796/0*-IIM7o5BX2_c1iYt.png)

Find the “_Elastic IP address owner_”:

![](https://cdn-images-1.medium.com/max/204/0*Xdn4Qug-QHZAiq2c.png)

OK, it’s one of the Load Balancers.

If an IP can’t be found in the AWS, it could be a [Kubernetes Endpoint](https://rtfm.co.ua/en/kubernetes-what-is-endpoints/), check it with the:

kubectl get endpoints — all-namespaces | grep 10.1.55.140

dev-ios-check-translation-ns ios-check-translation-backend-svc 10.1.55.140:3000 58d

dev-ios-check-translation-ns ios-check-translation-frontend-svc 10.1.55.140:80 58d

Actually, that’s all.

### Useful links

#### VPC Flow Logs

- [Collect Amazon VPC flow logs with Elastic Agent](https://docs.elastic.co/en/integrations/aws/vpcflow) — поля и прочее для Кибаны.
- [Flow log record examples](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html)

#### CloudWatch Logs

- [AWS Security Logging Fundamentals — VPC Flow Logs](https://panther.com/cyber-explained/aws-security-logging-vpc-flow-logs/) — пример Cloudwatch Logs с Athena
- [Analyze Query And Visualize AWS Cloudwatch Logs Using Logs Insight](https://dheeraj3choudhary.com/analyze-query-and-visualize-aws-cloudwatch-logs-using-logs-insight) — прмиеры запросов в Cloudwatch Logs
- [How to Visualize and Refine Your Network’s Security by Adding Security Group IDs to Your VPC Flow Logs](https://aws.amazon.com/blogs/security/how-to-visualize-and-refine-your-networks-security-by-adding-security-group-ids-to-your-vpc-flow-logs/)
- [Visualizing log data in graphs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Visualizing-Log-Data.html)
- [Analyze CloudWatch Logs like a pro](https://marbot.io/blog/analyze-cloudwatch-logs-like-a-pro.html)
- [Filter, Parse & Group Log Data in AWS CloudWatch Logs Insights](https://harishkm.in/2020/06/25/filter-parse-group-log-data-in-aws-cloudwatch-logs-insights/)
- [How can I analyze custom VPC Flow Logs using CloudWatch Logs Insights?](https://aws.amazon.com/ru/premiumsupport/knowledge-center/cloudwatch-vpc-flow-logs/) — ещё примеры с логами и выборками в Инсайтах
- [How can I use CloudWatch Logs Insights queries with my VPC flow log?](https://aws.amazon.com/ru/premiumsupport/knowledge-center/vpc-flow-logs-and-cloudwatch-logs-insights/)
- [CloudWatch Logs Insights query syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
- [Triaging Networking Issues Using AWS CloudWatch Logs Insights](https://www.cloudreach.com/en/technical-blog/networking-issues-cloudwatch-logs-insights/)

_Originally published at_ [_RTFM: Linux, DevOps, and system administration_](https://rtfm.co.ua/en/aws-vpc-flow-logs-an-overview-and-example-with-cloudwatch-logs-insights/)_._

* * *