Service Mesh Showdown: Consul vs Istio
Tags: kubernetes, hashicorp, consul, istio
Published: 2020-06-26 15:23:29

As we look to move to a microservice environment at $job, several requirements such as mutual TLS between services, distributed tracing, and the lockdown of traffic came about. A service mesh provides a solution for securing and monitoring traffic as it flows through the cluster. After discussing with several folks who have implemented or use service meshes, it came down to using Consul Connect or installing Istio.

## Consul Connect
An internal team uses consul for their testing environment, so going in there was a level of expertise within the organization. Consul would plug right into our current build workflow, as it utilizes Helm to deploy. Once deployed, the envoy sidecar will exist within a pod to proxy traffic between pods.

#### Deployment
For the helm chart I used the following value overrides:
```yml
global:
  name: consul
  datacenter: dc1
  # Enable all services to run on TLS
  tls:
    enabled: true

# Sync services from k8s to consul (more complete picture)
syncCatalog:
  enabled: true

# Default Inject Sidecar proxy into all containers
connectInject:
  enabled: true
  default: true
  centralConfig:
    enabled: true
client:
  enabled: true

# Highly available stack that enables telemetry
server:
  replicas: 3
  bootstrapExpect: 3
  storageClass: nfs-client
  extraConfig: | 
   {
    "telemetry": {
        "prometheus_retention_time": "24h",
        "disable_hostname": true
     }
   }

affinity: |
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: {{ template "consul.name" . }}
            release: "{{ .Release.Name }}"
            component: server
      topologyKey: kubernetes.io/hostname
```


#### Test Application
For a test app, I utilized Consul [counting](https://hub.docker.com/r/hashicorp/counting-service) + [dashboard](https://hub.docker.com/r/hashicorp/dashboard-service) service. The general notion is the dashboard app communicates with the counting app to get a number to display. To allow envoy to proxy to other services, annotations are utilized on the pod. For this app, by adding the below annotation, it will proxy all `localhost:9001` calls over mutual TLS: 

` "consul.hashicorp.com/connect-service-upstreams": "counting:9001"`

#### Analyzing Traffic Rules
Consul utilizes [intentions](https://www.consul.io/docs/connect/intentions) to define routing rules. One of the tests I did was disabling traffic from the dashboard to the counting service. When annotated this traffic became denied, and the resulting display occurred.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/89xo624uhntgkgjzzg4l.png)

However, one thing I discovered was this restriction could be bypassed by changing the URL from `locahost:9001` to `counting:9001`, which would utilize kubedns for service discovery, and then route the traffic.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/ghs572mgdgtv6816kve5.png)


#### Distributed Tracing
After reading this [blog post](https://www.hashicorp.com/blog/enabling-distributed-tracing-with-hashicorp-consul/), I wanted to implement distributed tracing into the environment. The Hashicorp [example](https://github.com/hashicorp/consul-demo-tracing/tree/master/jaeger), made it seem extremely simple to implement, however, I ran into several hurdles:
1. [Proxy escaping characters](https://github.com/hashicorp/consul-helm/issues/269)
2. Helm chart doesn't natively support, so had to deploy [zipkin](https://github.com/fabric8io/kubernetes-zipkin)
3. Found [a quote from logz.io](https://logz.io/blog/zipkin-vs-jaeger/), that made me pivot away from zipkin to jaeger:

```If Kubernetes is running in production, then adopt Jaeger. If there’s no existing container infrastructure than Zipkin, it makes for a better fit because there are fewer moving pieces. ```

## Istio
Initially going with Istio, the only thing I've ever heard about it is summed in [Ivan Pedrazas's Tweet](https://t.co/OGP8ZaKC3Y):

```"Daddy, tell me a horror story"
"SSL with Istio and Kubernetes"
"Is it as bad as the NFS monster one?"
"Oh no, nothing is worse than the NFS monster"```

However, I wanted to keep an open mind and had a discussion with IBM's [JJ Asghar](https://www.linkedin.com/in/jjasghar/) and a mentor [Drew Mullen](https://www.linkedin.com/in/drew-mullen). Through discussion, I learned how the horror stories of Istio have vastly been improved recently, with a [simplified control plane](https://istio.io/latest/blog/2020/istiod/). This was enough to start exploring how to incorporate it within our environment and perform the same analysis as the Consul Connect test.

#### Deployment
As of 1.5 Istio [deprecated](https://istio.io/latest/blog/2020/istiod/#the-cost-of-complexity) the use of Helm in replacement for their own CLI tool, which culminates many individual charts. This will call for some rework of our existing pipeline, but the tool itself is fairly simple. Utilizing profiles, you can determine which components you want to be installed.


|                        | default | demo | minimal | remote |
|---------------------   |---------|------|---------|--------|
|<b>Core components</b>  |         |      |         |        |
|istio-egressgateway     |         | X    |         |        |
|istio-ingressgateway    | X       | X    |         |        |
|istiod                  | X       | X    | X       |        |
|                        |         |     |       |        |
|<b>Addons</b>           |         |      |         |        |
|grafana                 |         | X    |         |        |
|istio-tracing           |         | X    |         |        |
|kiali                   |         | X    |         |        |
|prometheus              | X       | X    |         | X      |

For this use case, I wanted the minimalize footprint, but still wanted to add some additional components. Luckily, `istioctl` allows for that, with the default profile. Since we have a few overrides, well create a config file:

```yml
# install.yml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    tracing:
      enabled: true
    kiali:
      enabled: true
 ```

``` istioctl install --set profile=default -f install.yml```

Once Installed creating a namespace with the label `istio-injection: enabled`, or through the pod annotation: `sidecar.istio.io/inject: "true"` will ensure all pods get the envoy sidecar.

#### Analyzing Traffic Rules
With the envoy proxy sidecar enabled, mTLS is enabled for free. However, there are additional Custom Resource Definitions (CRDs) that you can define to enforce your environment is secure. 

###### Mutual TLS
To enforce mutual TLS utilizing the CRD's PeerAuthentication and DestinationRule should be defined. The below configuration requires all communication to in a given $NAMESPACE to use mTLS, while enforcing the `app: dashboard-counter` use it.

```yml
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "require-mtls-dashboard"
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: dashboard-counter
  mtls:
    # Several modes available (STRICT, PERMISSIVE, UNSET)
    mode: STRICT
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "default"
spec:
  host: "*.$NAMESPACE.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
```

Kiali provides a nice visualization of this all, whereas you see the entire communication from the unmarked ingress to the counter service going over TLS.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/xb69ugscnvw1uxjtkhvd.png)

###### Authorization
If you analyze various sources for best practices, one of the recommended to implement is to have a default deny-all. This practice is performed through [Authorization Policies](https://istio.io/latest/docs/tasks/security/authorization/):

```yml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: deny-all
 namespace: $NAMESPACE
spec:
  {}
```

Once a deny all is in place, all requests will receive an `RBAC: access denied` notification. Then you can define another CRD to allow actions for communication.
```yml
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-method-get
spec:
  selector:
    matchLabels:
      app: dashboard-counter
  action: ALLOW
  rules:
    - {}
```

#### Distributed Tracing
With Istio, distributed tracing comes as a configuration flag. By using the default with tracing, Jaeger is installed into the cluster, and you can visualize with Kiali.
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/uga1i9kosdww1f2i4k2f.png)


## Summary
After analyzing Istio vs Consul, a lot of features I was looking for seemed to come out of the box with Istio. The benefits of using CRDs vs API calls also weighed heavily since that another auth system is not in play. With Consul, although it was nice to plugin with Helm, the bypass of intentions with service discovery was ultimately the negator.