Add Custom Headers to Outgoing request from applications deployed on Kubernetes
Published: 2024-04-26 15:24:11



## Introduction
Kubernetes in todays world is the defacto platform for deploying containerized applications including microservices because of the various benefits this platform offers including automated deployment, scaling security, resiliency, load balancing and self healing along with many others. 
This article will explain how we can automate the insertion of custom headers in outgoing requests from the applications that are deployed in Kubernetes with Istio Service Mesh.


## Communication between Pods using Kubernetes Service
Each Pod deployed gets its onw unique internal IP address in the cluster. When applications running inside pods wants to connect to other pods, they can connect using the podname or pod ipaddress. 
In case there are multiple replicas of an application running leading to multiple pods then kubernetes service is used to reach the destination. 
Below diagram shows the pod to pod communication through kubernetes service. 

![Pod communication inside Kubernetes with K8S Service](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bx34i7iiplkw9xour0i3.png)

## Istio Service Mesh 
Service Mesh is an infrastructure layer that handles communication and networking between services in an application. It handles traffic, monitors performance and enforces different policies. 
Some of the benefits of using service mesh includes Service Discovery, Traffic control, Observability, Security and Compliance
[Istio](https://istio.io/latest/) is an open source service mesh which offers all of the above mentioned features of a typical service mesh. You can read about it more on this link [Istio / Architecture](https://istio.io/latest/docs/reference/config/networking/envoy-filter/)
Istio uses an extended version of the Envoy proxy. Envoy is a high-performance proxy developed in C++ to mediate all inbound and outbound traffic for all services in the service mesh. Envoy proxies are the only Istio components that interact with data plane traffic. Istio deploys Envoy proxies as side car containers to workloads logically augmenting the services with many built in features of Envoy.

Below diagram shows the communication across applications when Istio Service Mesh is used.


![Pod communication inside Kubernetes with Istio](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2l5j7pliy2nuif1gly8p.png)

## Insert Custom Headers to Outgoing Requests
Envoy proxy intercepts all the incoming and outgoing requests  within the pod to/from all the containers that are running in the pod. This configuration can be overridden to implement various functionalities like logging, auditing, authentication, modifying requests etc. Envoy filters are additively applied to the proxy configuration. We can add multiple filters with different functionalities.
Here we will see how can we extend this functionality to add custom headers to outgoing requests from the pods.

We need to create the EnvoyFilter kubernetes Istio CRD and override the function which is invoked when a request is intercepted by the proxy.
The following code enables Envoys Lua filter for all outbound HTTP calls on service port 8080 of the myapplication service pod  with labels "app: myapplication". The Lua filter adds a custom header key and value to the request handle headers object.

```
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: customheaders-filter
spec:
  workloadSelector:
    labels:
      app: myapplication
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
	    portNumber: 8080
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |

              function envoy_on_request(request_handle)
                request_handle:headers():add( "my-custom-header", "my-custom-header-value )
              end

```


We can also extend this function to add headers only if the destination host is expected one. 

```
  function envoy_on_request(request_handle)
	local destinationHost = request_handle:headers():get(":authority")
	if destinationHost ~= nil and destinationHost ~= '' then
	  request_handle:logInfo(" found authority  :"..destinationHost)
	else
	  request_handle:logInfo(" no authority found")
	  destinationHost = ''
	end

	request_handle:logInfo("destination host -> "..destinationHost) 
	request_handle:logInfo("configured host -> {{ $customHeadersByHost.host }}")

	if destinationHost:find({{ $customHeadersByHost.host | quote }}, 0, true) ~= nil then
	  {{- range $header := $customHeadersByHost.headers }}
	  request_handle:headers():add({{ $header.name | quote }} , {{ $header.value | quote }} )
	  {{- end }}
	  request_handle:logInfo(" added headers for host - {{ $customHeadersByHost.host }}")
	else
	  request_handle:logInfo(" destination does not match configured host ")
	end

  end

```

Inorder to view the envoy-filter proxy logs from the pods we need to add the below annotations to the Pod definition if logging is restricted at istio system level.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  sidecar.istio.io/logLevel: info
spec:
  ..
  ..


```

To view the logs that are added in the on request function above, we can view them using kubectl command

```
kubectl logs -n mynamespace myappliation-pod -c istio-proxy
```