Setup an ingress rate limiter with envoy and istio
Tags: servicemesh, istio, envoy, ratelimit
Published: 2021-08-22 08:26:55

With the removal of the mixer component in istio 1.5, the configuration of rate limiting changed. Since istio > 1.5, rate limiting is done with an EnvoyFilter that applies to your proxies. Therefore it makes sense to have a closer look at the istio data plane to gain a better understanding about envoy.

![istio architecture](https://istio.io/latest/docs/ops/deployment/architecture/arch.svg)
*Image from https://istio.io/latest/docs/concepts/what-is-istio/*

As you can see in the above image, the service mesh dataplane consists of envoy proxies sitting in front of each of your services. An Ingressgateway or Egressgateway is also just an envoy proxy at the edge. Because all traffic in the mesh goes through these proxies, you gain observability of your network traffic as well as control by configuring the proxies to your needs. To better understand the possibilites of a service mesh, we should take a deeper look into the envoy proxy itself.

## What is envoy?

In the [envoy docs](https://www.envoyproxy.io/docs/envoy/v1.17.0/intro/what_is_envoy) we read:

> Envoy is an L7 proxy and communication bus designed for large modern service oriented architectures. The project was born out of the belief that:
>
> "The network should be transparent to applications. When network and application problems do occur it should be easy to determine the source of the problem."

Since envoy runs as a sidecar in your pods, you can update the proxy configuration on the fly without touching the different services itself.

Every Proxy has a built in filter chain. In this chain there are pluggable filters which provide the traffic management capabilites. The filters perform different tasks like buffering, rate limiting and, last but not least routing. You can think of the filter chain like a middleware in your webserver.

To avoid confusion you should have a look at the [envoy terminology](https://www.envoyproxy.io/docs/envoy/v1.17.0/intro/arch_overview/intro/terminology#terminology).
We cited the most important terms for this article from the envoy docs.

>**Host**: An entity capable of network communication (application on a mobile phone, server, etc.). In this documentation a host is a logical network application. A physical piece of hardware could possibly have multiple hosts running on it as long as each of them can be independently addressed.
>
>**Downstream**: A downstream host connects to Envoy, sends requests, and receives responses.
>
>**Upstream**: An upstream host receives connections and requests from Envoy and returns responses.
>
>**Listener**: A listener is a named network location (e.g., port, unix domain socket, etc.) that can be connected to by downstream clients. Envoy exposes one or more listeners that downstream hosts connect to.
>
>**Cluster**: A cluster is a group of logically similar upstream hosts that Envoy connects to. Envoy discovers the members of a cluster via service discovery. It optionally determines the health of cluster members via active health checking. The cluster member that Envoy routes a request to is determined by the load balancing policy.

Now, that we have a glimpse of the istio data plane. Let's solve an actual problem with it.

## Rate limit a service
Envoy has [local (non-distributed)](https://www.envoyproxy.io/docs/envoy/v1.17.0/intro/arch_overview/other_features/local_rate_limiting) and global rate limiting capabilities. This article will focus on the [global rate limiting architecture](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/other_features/global_rate_limiting#arch-overview-global-rate-limit).

To use global rate limiting you need an external rate limiter service that keeps track of the domains that have to be rate limited. Luckily envoy provides a redis based [ratelimit service](https://github.com/envoyproxy/ratelimit). In the following sections you will learn how to configure the redis based ratelimit service and how to rate limit specific routes.

![envoy filter chain](https://i.imgur.com/UBs89Jd.png)
*Image created with draw.io by Natalia Sattler and Moritz Rieger*

### Deploy the ratelimit service

For the deployment of the ratelimit service you can use the [ratelimitservice.yaml](https://github.com/istio/istio/blob/master/samples/ratelimit/rate-limit-service.yaml) as a starting point. If you already have a redis instance in your kubernetes cluster, then feel free to use it for the rate limiter service as well and remove redis related parts. 
Adjust the ConfigMap `ratelimit-config` with your rate limiting rules and specify the `REDIS_URL` in the `ratelimit` Deployment. Now you can deploy the ratelimit service to your kubernetes cluster.

### Rate limiting rules

You should check the ratelimit service [documentation](https://github.com/envoyproxy/ratelimit#configuration) for details on how to configure the rate limiting rules. For now you can just use the example configuration from below and adjust it to your needs.

*Example configuration:* 
```yaml=
apiVersion: v1
kind: ConfigMap
metadata:
  name: ratelimit-config
data:
  config.yaml: |
    domain: foo-domain
    descriptors:
      - key: BAR
        value: "/foo"
        rate_limit:
          unit: minute
          requests_per_unit: 1
      - key: BAR
        rate_limit:
          unit: minute
          requests_per_unit: 100
```

From the rate limit docs
>**Domain**: A domain is a container for a set of rate limits. All domains known to the Ratelimit service must be globally unique.

>**Descriptor**: A descriptor is a list of key/value pairs owned by a domain that the Ratelimit service uses to select the correct rate limit to use when limiting.

In the example we use `foo-domain` to group our rate limiting rules:
1. all descriptors with key `BAR` and value `/foo` will have a rate limit of 1 request per minute
2. all other values of the descriptor with key `BAR` will have a rate limit of 100 requests per minute.
3. all other keys are not rate limited

The rate limit service determines whether a request should be limited based on the descriptors provided in the check call. If the call is made with `("BAR","/foo")`, rule number one of the rate limiter will be used, if it is called with `("BAR","/foobar")`, the second rule kicks in. Because there is no special handling for the `/foobar` value. If another request with `("BAR","/test")` comes in, the rule number two will be used as well. Note, that the rate limit is not shared with `("BAR","/foobar")`. Each of the requests can be queried 100 times per minute. The implicit rule number three gets applied if the rate limit service is invoked with something like `("FOO","/foo")`. For an unspecified key, no ratelimiting is applied at all.

One important thing to mention is that the value of the descriptor, though it looks like a part of an URI, does not have to denote the requested URI. It is merely a value of the descriptor and could be anything. In the example it is specified this way because of the envoy filter we configure later on.

When the configuration of the rate limiter is done and the service is beamed into your kubernetes cluster, it is time to ensure that all incoming requests will be guarded by this service.

### Introduce the ratelimit service to Envoy

At first, register the ratelimit service as an envoy cluster. This is done via an envoy filter patch which is applied at the cluster level. The config patch specifies the new envoy cluster with a name of your choice and the according endpoints of the rate limiter service (line 11-36).

This cluster can then be referenced from the ratelimit http filter so that the filter knows where to send the rate limit queries.

This is exactly what the second path does. We insert the rate limit http filter into to the filter chain of the ingress gateway proxy. In line 55-62 we reference to the previously created cluster and rate limiter domain in the filter definition.

*Example*
```yaml=
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: CLUSTER
    match:
      proxy:
        proxyVersion: ^1\.15.*
      cluster:
        # kubernetes dns of your ratelimit service
        service: ratelimit.default.svc.cluster.local
    patch:
      operation: ADD
      value:
        name: rate_limit_cluster
        type: STRICT_DNS
        connect_timeout: 10s
        lb_policy: ROUND_ROBIN
        http2_protocol_options: {}
        load_assignment:
          # arbitrary  name
          cluster_name: rate_limit_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    # kubernetes dns of your ratelimit service
                    address: ratelimit.default.svc.cluster.local
                    port_value: 8081
  - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        proxy:
          proxyVersion: ^1\.15.*
        listener:
          filterChain:
            filter:
              name: 'envoy.http_connection_manager'
              subFilter:
                name: 'envoy.router'
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ratelimit
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            # arbirary domain, ensure it matches with the domain used in the ratelimit service config
            domain: foo-domain
            failure_mode_deny: true
            rate_limit_service:
              grpc_service:
                envoy_grpc:
                  # must match load_assignment.cluster_name from the patch to the CLUSTER above
                  cluster_name: rate_limit_cluster
                timeout: 10s
              transport_api_version: V3
```

### Use rate limit filter in an ingress gateway

So far we have configured the rate limit filter and the cluster. Now it is time to use this filter in our ingress gateway to rate limit incoming requests.
In the folowing yaml we attach the rate limit filter to the ingress gateway virtual host filter chain and specify the descriptor to be used with rate limiter service queries.

*Example*

```yaml=
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      proxy:
        proxyVersion: ^1\.15.*
      context: GATEWAY
      routeConfiguration:
        # Should be in the namespace/name format. Use this field in conjunction with the portNumber and portName to accurately select the Envoy route configuration for a specific HTTPS server within a gateway config object.
        gateway: istio-system/istio-gateway
        portNumber: 443
        portName: https
    patch:
      operation: MERGE
      value:
        rate_limits:
        - actions:
          # This action results in the following descriptor ("BAR","/foo") where "/foo" is the requested path.
          # :path is resolved to the actual requested path at runtime and used as the descriptor value
          - request_headers:
              header_name: ':path'
              descriptor_key: 'BAR'
```

### Test envoy configuration

Test yor configuration by just issuing the following request to your ingress gateway two times in a row `curl -i https://<your host>/foo`. The second request should result in an 429 HTTP error.

If not, troubleshoot your configuration.

#### 1. Check if the rate limiter cluster is registered

Verify that your rate limiter cluster is properly registered
`istioctl proxy-config cluster <your-istio-ingressgateway-pod>.istio-system -o json`

This command should produce something like
```json=
[
  {
    "name": "rate_limit_cluster",
    "type": "STRICT_DNS",
    "connectTimeout": "10s",
    "loadAssignment": {
      "clusterName": "rate_limit_cluster",
      "endpoints": [
        {
          "lbEndpoints": [
            {
              "endpoint": {
                "address": {
                  "socketAddress": {
                    "address": "ratelimit.default.svc.cluster.local",
                    "portValue": 8081
                  }
                }
              }
            }
          ]
        }
      ]
    },
    "http2ProtocolOptions": {}
  },
  ...
]
```

If not, use the `istioctl proxy-status` command to check the status of the cluster.

`istioctl proxy-status <your-istio-ingressgateway-pod>.istio-system`

#### 2. Check if the rate limiter filter is attached to the http filter chain

Verify that the rate limiter filter is attached to the http filter chain
`istioctl proxy-config listener <your-istio-ingressgateway-pod>.istio-system -o json`

This command should produce something like
```json=
[
  {
    "name": "0.0.0.0_8443",
    "address": {
      "socketAddress": {
        "address": "0.0.0.0",
        "portValue": 8443
      }
    },
    "filterChains": [
      {
        "filterChainMatch": {
          "serverNames": [
            "www.example.com"
          ]
        },
        "filters": [
          {
            "name": "envoy.filters.network.http_connection_manager",
            "typedConfig": {
              "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
              "statPrefix": "outbound_0.0.0.0_8443",
              "rds": {
                "configSource": {
                  "ads": {},
                  "resourceApiVersion": "V3"
                },
                "routeConfigName": "https.443.https.istio-gateway.istio-system"
              },
              "httpFilters": [
                ...
                                {
                  "name": "envoy.filters.http.ratelimit",
                  "typedConfig": {
                    "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit",
                    "domain": "foo-domain",
                    "failureModeDeny": true,
                    "rateLimitService": {
                      "grpcService": {
                        "envoyGrpc": {
                          "clusterName": "rate_limit_cluster"
                        },
                        "timeout": "10s"
                      },
                      "transportApiVersion": "V3"
                    }
                  }
                },
                ...
```

If not, check the problems with the same command as above.

#### 3. Check if the rate limiter action configuration is applied to your ingress gateway virtual host configuration

Verify the route configuration with the following command
`istioctl proxy-config route <your-istio-ingressgateway-pod>.istio-system -o json` 

This command should produce something like
```json=
[
  {
    "name": "https.443.https.istio-gateway.istio-system",
    "virtualHosts": [
      {
        "name": "www.example.com:443",
        "domains": [
          "www.example.com",
          "www.example.com:*"
        ],
        "routes": [...],
        "rateLimits": [
          {
            "actions": [
              {
                "requestHeaders": {
                  "headerName": ":path",
                  "descriptorKey": "PATH"
                }
              }
            ]
          }
        ],
...
```
If not, check the problems with the same command as above.

## Be aware of pitfalls!

### Query parameters make a path unique

After you have applied all the configurations from above and verified that ratelimiting works. You may expect to get same rate limiting rules applied to the request `https://<your host>/foo?param=value` as to `https://<your host>/foo`. But this is not the case. The applied rule will be the second one (with 100 req/min).

![ohh really](https://media2.giphy.com/media/oKdjMdWXl9ys8/giphy.gif)
*Image from giphy.com*

This is because the [pseudo-header field `:path`](https://tools.ietf.org/html/rfc7540), which we used for the descriptor value includes the path and all query parts of the target URI. The ratelimiter service on its part does not detect any special configuration for the descriptor `("BAR", "/foo?param=value")` and therfore use the default of key `"BAR"`.

To make the search params work as expected with the rate limiter we have to cut the off the value. This can be done with just another filter [envoy.filters.http.header_to_metadata](https://www.envoyproxy.io/docs/envoy/latest/api-v2/config/filter/http/header_to_metadata/v2/header_to_metadata.proto).

To use this filter you have to change your filter definitions:

*filter-ratelimit*
```yaml=
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: CLUSTER
    match:
      proxy:
        proxyVersion: ^1\.15.*
      cluster:
        # kubernetes dns of your ratelimit service
        service: ratelimit.default.svc.cluster.local
    patch:
      operation: ADD
      value:
        name: rate_limit_cluster
        type: STRICT_DNS
        connect_timeout: 10s
        lb_policy: ROUND_ROBIN
        http2_protocol_options: {}
        load_assignment:
          # arbitrary  name
          cluster_name: rate_limit_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    # kubernetes dns of your ratelimit service
                    address: ratelimit.default.svc.cluster.local
                    port_value: 8081
- applyTo: HTTP_FILTER
    match:
      proxy:
        proxyVersion: ^1\.15.*
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: 'envoy.http_connection_manager'
            subFilter:
              name: 'envoy.router'
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.header_to_metadata
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.header_to_metadata.v3.Config
          request_rules:
          - header: ':path'
            on_header_present:
              # use an arbitary name for the namespace
              # will be used later to extract descriptor value
              metadata_namespace: example
              # use an arbitary key for the metadata
              # will be used later to extract descriptor value
              key: uri
              regex_value_rewrite:
                pattern:
                  # regex matcher
                  google_re2: {}
                  # truncates parameters from path
                  regex: '^(\/[\/\d\w-]+)\??.*$'
                substitution: '\1'
- applyTo: HTTP_FILTER
    match:
      proxy:
        proxyVersion: ^1\.15.*
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: 'envoy.http_connection_manager'
            subFilter:
              name: 'envoy.filters.http.header_to_metadata'
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.ratelimit
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
          # ensure the domain matches with the domain used in the ratelimit service config
          domain: foo-domain
          failure_mode_deny: true
          rate_limit_service:
            grpc_service:
              envoy_grpc:
                # must match load_assignment.cluster_name from the patch to the CLUSTER above
                cluster_name: rate_limit_cluster
              timeout: 10s
            transport_api_version: V3
```

This adds a new filter, which is registered just before the rate limiter filter and which rewrites the `:path` header and adds it to the metadata. To use the metadata for the descriptor value update the following envoy filter definition.

*filter-ratelimit-svc*
```yaml=
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: filter-ratelimit-svc
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      context: GATEWAY
      routeConfiguration:
        # Should be in the namespace/name format. Use this field in conjunction with the portNumber and portName to accurately select the Envoy route configuration for a specific HTTPS server within a gateway config object.
        gateway: istio-system/istio-gateway
        portNumber: 443
        portName: https
    patch:
      operation: MERGE
      value:
        rate_limits:
        - actions:
          - dynamic_metadata:
              descriptor_key: BAR
              metadata_key:
                key: example
                path:
                - key: uri
```

When you now call your service with `curl -i https://<your host>/foo?param=value` followed by `curl -i https://<your host>/foo`. You should receive 429 on the second call which verifies that both URIs are limited by the same rule.

### Envoy filter syntax change

When you introduce EnvoyFilter into your istio service mesh configuration, you should pay special attention to Envoy Filters, even on minor upgrades. 

> As [EnvoyFilter is a break glass API without backwards compatibility guarantees](https://istio.io/latest/news/releases/1.7.x/announcing-1.7/upgrade-notes/#envoyfilter-syntax-change), we recommend users explicitly bind EnvoyFilters to specific versions and appropriately test them prior to upgrading.

### Envoy version ≠ istio version

Be aware which version of envoy is deployed in your istio service mesh. The Version of istio and envoy is not in sync.
You can determine your version of envoy with the following command:
`kubectl exec -it <PODNAME-WITH-ENVOY-SIDECAR> -c istio-proxy -n istio-system -- pilot-agent request GET server_info`

```json
{
 "version": "dc78069b10cc94fa07bb974b7101dd1b42e2e7bf/1.15.1-dev/Clean/RELEASE/BoringSSL",`
...
}
```

<span>Title image by <a href="https://unsplash.com/@punttim?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Tim Gouw</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></span>

{% user nsattler %}
{% user moritzrieger %}