Istio Canary Deploys
Tags: istio, kubernetes, canary, testing
Published: 2020-08-10 17:47:00

Performing a canary release, rather it be for A/B testing or controlled rollout, is a common practice within software deployments. Istio enables canary routing, through a plethora of [HTTP Match Requests](https://istio.io/latest/docs/reference/config/networking/virtual-service/#HTTPMatchRequest), but for my particular use case, I wanted to route to the canary release based on a particular request header. Internally, this enables us to dark deploy, run a/b tests, and enable developers to test features before cutting our production traffic over.

### Prerequisites:
- Istio installed namespace

### Architecture:
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/s6g2rlozkxfdp8d7zpca.png)

For this POC, I built a sample app which simply returns “test-a” if it's using the “a” image tag, or “test-b” if using the “b” tag. When a user makes an HTTP request call, if it has the header `version: canary`, it will route to the canary release, otherwise, it will route to the production release. All source code is linked below.

### Setup
With [istio installed namespace](https://istio.io/latest/docs/setup/getting-started/#install), we need to deploy two deployments of our application, which is fronted by a service. For simplicity, I’ve redacted the config to only the essential pieces:

```yaml
# Production Release
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-canary-testing
  name: app-production
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-canary-testing
  template:
    metadata:
      labels:
        app: app-canary-testing
        version: production
    spec:
      containers:
      - image: danquack/canary-testing:a
        name: canary-testing
        ports:
        - containerPort: 9000
          name: http
          protocol: TCP
```

```yaml
# Canary Release
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-canary-testing
  name: app-canary
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-canary-testing
  template:
    metadata:
      labels:
        app: app-canary-testing
        version: canary
    spec:
      containers:
      - image: danquack/canary-testing:b
        name: canary-testing
        ports:
        - containerPort: 9000
          name: http
          protocol: TCP
```

A couple of the key differences between these two deployments is the image tag (prod: a, canary: b) and the pod version label, which we will utilize later.

With the deployment defined, next is setting up the service:
```yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app: app-canary-testing
  name: app-canary-testing
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: app-canary-testing
```

### Istio Configuration
To configure routing we are going to configure three resources, which will define ingress, subsets (groups driven from labels), and routing rules.

#### Ingress
Configure a Istio Gateway to route traffic for our hostname app.local
```yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: canary-testing
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - app.local
    port:
      name: http
      number: 80
      protocol: HTTP
```

### Destination Rule
Destination rules are designed to define traffic policies for load balancing or group individual versions of services. With subsets, we can define individual groups driven from the label we set on our pod. This config states for our service hostname, group the pods based on the label `version`:
```yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: canary-testing
spec:
  host: app-canary-testing
  subsets:
  - labels:
      version: canary
    name: canary
  - labels:
      version: production
    name: production
```

#### Virtual Service
Now that all the core pieces are in place, we can configure the joining definition, which will allow us to route the traffic in the desired manner. When the traffic comes through the ingress gateway, istio will check our virtual service, to determine where to send that packet. In this setup, because the desired response from a requestor who sends an HTTP  request with the header `version: canary` is to route to the canary pod, we will configure the matching header, and then route to the service with the matching subset created from the above Destination Rule.
```yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: canary-testing
spec:
  gateways:
  - canary-testing
  hosts:
  - app.local
  http:
  - match:
    - headers:
        version:
          exact: canary
    name: canary-route
    route:
    - destination:
        host: app-canary-testing
        subset: canary
  - name: Production Route
    route:
    - destination:
        host: app-canary-testing
        subset: production
```

### Testing
To verify our deploy is working, first, grab the ingress host:
```bash
$ kubectl -n istio-system get service istio-ingressgateway
```

Then well try a sample curl call to our production endpoint:
```bash
$ curl http://$INGRESS_HOST -H "Host: app.local"
test-a
```

Then append the http header, `version: canary`, which will verify your canary release
```bash
$ curl http://$INGRESS_HOST -H "Host: app.local" -H "version: canary"
test-b
```

All source code included can be found on Github:
{% github danquack/canary-testing no-readme %}