How do you Integrate Emissary Ingress with OPA
Tags: opa, emissaryingres, security, api
Published: 2022-04-28 08:24:04

API gateways play a vital role while exposing microservices. They are an additional hop in the network that the incoming request must go through in order to communicate with the services. An API gateway does routing, composition, protocol translation, and user policy enforcement after it receives a request from client and then reverse proxies it to the appropriate underlying API. As the API gateways are capable of doing the above-mentioned tasks, they can be also configured to send the incoming client requests to an external third-party authorization (authz) server. The fate of the incoming request then depends upon the response from this external authz server to the gateway. This is exactly where Open Policy Agent (OPA) comes into the picture.

There are many open source Kubernetes native API gateways out there like Contour, Kong Gateway, Traefik, Gloo, etc.. In this article, we will be exploring the Emissary Ingress.

Let's dive deep and start understanding more bit about [Emissary Ingress](https://github.com/emissary-ingress/emissary).

## What's Emissary Ingress?

Emissary Ingress was earlier known as Ambassador API gateway, it is an open source Kubernetes native API gateway and is currently a [CNCF Incubation Project](https://www.cncf.io/projects/emissary-ingress/). Like many other Kubernetes gateways, Emissary has also been built to work with Envoy Proxy. It is deployed as complete stateless architecture and supports multiple plugins such as traditional SSO authentication protocols (e.g.: OAuth, OpenID Connect), rate limiting, logging, and tracing service. Emissary utilizes its [ExtAuth protocol](https://www.getambassador.io/docs/emissary/latest/topics/running/services/ext_authz/) in [AuthService](https://www.getambassador.io/docs/emissary/latest/topics/running/services/auth-service/) resource to configure the authentication and authorization for incoming requests. `ExtAuth` supports two protocols: gRPC and plain HTTP. For gRPC interface, the external service must implement Envoy's [external_auth.proto](https://github.com/emissary-ingress/emissary/blob/master/api/envoy/service/auth/v2/external_auth.proto).


## OPA

Open Policy Agent is a well-known general-purpose policy engine and has emerged as a policy enforcer across the stacks be it API gateways, service meshes, Kubernetes, microservice, CICD, or IAC. OPA decouples decision making from policy enforcement such that whenever your software needs to make a decision regarding the incoming requests, it queries OPA. [OPA-Envoy](https://github.com/open-policy-agent/opa-envoy-plugin) extends OPA with a gRPC server that implements the Envoy [External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_authz_filter), thus making itself compatible to be as an external authz server to Emissary.  

## Integrating Emissary Ingress with OPA

![Emissary OPA](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a6qxlbqfc6jjahyavq0v.png) 

The above figure shows highlevel architecture of Emissary and OPA integration. When incoming request from client reaches Emissary, it sends an authorization request to OPA which contains input JSON. OPA evaluates this JSON against the Rego policies provided to it and responds to Emissary, if this result JSON from OPA has `allow` as `true` then only the client request is further routed to API or else the request is denied by Emissary and never reaches the API. Now, we will be installing Emissary Ingress and integrate it with OPA for external authorization. 

### Getting Started
First, we will be needing to start a Minikube cluster. If you don't have Minikube, you can install it from [here](https://minikube.sigs.k8s.io/docs/start/).

```shell
minikube start
```
Install the Emissary Ingress to the minikube through Helm. 
```shell
# Add the Repo:
helm repo add datawire https://app.getambassador.io
helm repo update
 
# Create Namespace and Install:
kubectl create namespace emissary && \
kubectl apply -f https://app.getambassador.io/yaml/emissary/2.2.2/emissary-crds.yaml
kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system
helm install emissary-ingress --namespace emissary datawire/emissary-ingress && \
kubectl -n emissary wait --for condition=available --timeout=90s deploy -lapp.kubernetes.io/instance=emissary-ingress
```

Or go to [Emissary Ingress Documentation](https://www.getambassador.io/docs/emissary/latest/tutorials/getting-started/) to install it through Kubernetes YAMLs.

### Configuring the routing for demo application  

Different gateways have their own set of configurations for exposing a service. In Emissary, we need to configure the routing through Mappings and Listeners. 

Mapping resource simply tells Emissary which service to redirect the incoming request to. It is highly configurable like `Ingress`. You can learn more about Mapping resource on [Introduction to the Mapping resource](https://www.getambassador.io/docs/emissary/latest/topics/using/intro-mappings/) page. 
We will create a simple `Mapping` resource which will redirect all the incoming requests to our demo application's service that is `demo-svc`.

```shell
cat <<EOF | kubectl apply -f -
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: demo-app-mapping  
spec:
  hostname: "*"
  prefix: /
  service: demo-svc
EOF
```

The Listener resource instructs Emissary where to listen on the network for the incoming request. Here we will create a listener to listen on port `8080` and `HTTP` protocol and associates with hosts in `All` namespace . For detailed info visit [Listener Docs](https://www.getambassador.io/docs/emissary/latest/topics/running/listener/).

```shell
cat <<EOF | kubectl apply -f -
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: demo-app-listener-8080
  namespace: emissary
spec:
  port: 8080
  protocol: HTTP
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL
EOF
```      

### Install the Demo Application

Install a simple echo server as a demo application.

```shell
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      containers:
      - name: http-svc
        image: gcr.io/google_containers/echoserver:1.8
        ports:
        - containerPort: 8080
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
  name: demo-svc
  labels:
    app: demo-app
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: demo-app
EOF
```   

Communicate with the demo app at different paths.

```shell
minikube service emissary-ingress -n emissary
```

Note: The above exposing method may not work for macOS users. They can use busybox and configure it to hit the emissary local endpoint instead.

Copy the private URL with target port 80. The URL must be IP `192.168.49.2` followed by a NodePort like `http://192.168.49.2:30329`. Export the NodePort value to `$NODEPORT` environment variable and curl to that at paths as follows:

```shell
curl http://192.168.49.2:$NODEPORT/public
```
and 
```shell
curl http://192.168.49.2:$NODEPORT/secured
```

OPA has not yet been added to the setup and the above curl requests are directly sent to API without any policy enforcement. 

### How to Install and Configure OPA?

OPA will be reading the policies fed to it via a configmap. Create the following configmap which contains a policy that allows all incoming requests only through `GET` method.

```shell
cat <<EOF | kubectl apply -n emissary -f  -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-policy
data: 
  policy.rego: |-
    package envoy.authz

    default allow = false

    allow {
       input.attributes.request.http.method == "GET" 
    }
EOF
```   

OPA can be configured as an external authorization server via deploying it as an independent deployment or as a sidecar to the `emissary-ingress`. Here we will add it as a sidecar. Save the following YAML as `opa-patch.yaml`.

```yaml
spec:
  template:
    spec:
      containers:
      - name: opa
        image: openpolicyagent/opa:latest-envoy
        ports:
        - containerPort: 9191
        args:
          - "run"
          - "--server"
          - "--addr=0.0.0.0:8181"
          - "--set=plugins.envoy_ext_authz_grpc.addr=0.0.0.0:9191"
          - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
          - "--set=decision_logs.console=true"
          - "--ignore=.*"
          - "/policy/policy.rego"
        volumeMounts:
          - mountPath: /policy
            name: demo-policy
            readOnly: true
      volumes:
      - name: demo-policy
        configMap:
          name: demo-policy
```   

patch the emissary-ingress deployment and wait for the all the  `emissary-ingress` pods to restart.

```shell
kubectl patch deployment emissary-ingress -n emissary --patch-file opa-patch.yaml
```

Wait until all the emissary-ingress pods come to `Running` state with OPA sidecar.

Create the following `AuthService`. AuthService is a resource which configures Emissary to communicate with an external service for Authn and Authz of incoming request. We are configuring it communicate with OPA on localhost since OPA is deployed as a sidecar. 

```shell
cat <<EOF | kubectl apply -f -
apiVersion: getambassador.io/v3alpha1
kind: AuthService
metadata:
  name: opa-ext-authservice
  namespace: emissary
  labels:
    product: aes
    app: opa-ext-auth
spec:
  proto: grpc
  auth_service: localhost:9191
  timeout_ms: 5000
  tls: "false"
  allow_request_body: true
  protocol_version: v2
  include_body:
    max_bytes: 8192
    allow_partial: true
  status_on_error:
    code: 504
  failure_mode_allow: false
EOF
```  

Try doing curl now, since the policy accepts requests coming through `GET` method and there are no restrictions on path, both the request will get a 200 OK response.

```shell
curl -i http://192.168.49.2:$NODEPORT/public
curl -i http://192.168.49.2:$NODEPORT/private
```
Now lets edit the policy to accept incoming requests at path `/public` only and request to any other path will be denied.

```shell
cat <<EOF | kubectl apply -n emissary -f  -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-policy
data: 
  policy.rego: |-
    package envoy.authz

    default allow = false

    allow {
       input.attributes.request.http.method == "GET"
       input.attributes.request.http.path == "/public" 
    }
EOF
```  

Now restart the emissary ingress deployment for policy changes to take effect. 

```shell
kubectl rollout restart deployment emissary-ingress -n emissary
```

Wait until all the emissary-ingress pods come to `Running` state after restart.

Now do a curl request at path `/public`, it will be accepted but at path `/private` it will be denied by OPA with a 403 response and hence the request will not reach the demo API.

```shell
curl -i http://192.168.49.2:$NODEPORT/public
curl -i http://192.168.49.2:$NODEPORT/private
```
## Conclusion

The decision-making about the incoming request from the client to exposed API can be decoupled to OPA as an external authorization server in the Emissary Ingress setup. OPA can be added as a plug-and-play policy enforcer to Emissary and any other gateways supporting the Envoy [External Authorization API](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_authz_filter). 

We hope you found this post informative and engaging. Connect with us over [Twitter](https://twitter.com/infracloudio) and [Linkedin](https://www.linkedin.com/company/infracloudio/) and start a conversation.

## References and further reading

 - [API Gateways and Service Mesh](https://www.cncf.io/blog/2020/03/06/the-difference-between-api-gateways-and-service-mesh/)
 - [API Gateways- Nginx](https://www.nginx.com/learn/api-gateway/)   
 - [OPA](https://www.openpolicyagent.org/docs/latest/)
 - [OPA Envoy-plugin](https://github.com/open-policy-agent/opa-envoy-plugin)
 - [Emissary Ingress Github](https://github.com/emissary-ingress/emissary)
 - [Emissary ExtAuth protocol](https://www.getambassador.io/docs/emissary/latest/topics/running/services/ext_authz/)
 - [AuthService](https://www.getambassador.io/docs/emissary/latest/topics/running/services/auth-service/)