osm-edge: Using access control policies to access services with the service mesh
Tags: servicemesh, accesscontrol, osmedge, pipyproxy
Published: 2022-11-08 06:46:19


Deploying a service mesh in a complex brownfield environment is a lengthy and gradual process requiring upfront planning, or there may exist use cases where you have a specific set of services that either aren't yet ready for migration or for some reason can not be migrated to service mesh.

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the [osm-edge](https://flomesh.io/osm-edge) service mesh.

> [osm-edge](https://github.com/flomesh-io/osm-edge/) forked from [Open Service Mesh](https://github.com/openservicemesh/osm) is a lightweight, extensible, cloud-native, SMI-compatible service mesh built purposely for Edge computing. osm-edge uses lightweight programmable proxy [Pipy](https://flomesh.io/pipy) as a sidecar proxy.

![Accessing services within service mesh](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c9jtgh3t12dmfoadxdcr.png)

osm-edge offers two ways to allow accessing services within the service mesh:

* via Ingress
  * [FSM](https://github.com/flomesh-io/fsm) Ingress controller
  * Nginx Ingress controller
* Access Control
  * Service
  * IPRange


The first method to access the services in the service mesh is via Ingress controller, and treat the services outside the mesh as the services inside the cluster. The advantage of this approach is that the setup is simple and straightforward and the disadvantages are also apparent, as you cannot achieve fine-grained access control, and all services outside the mesh can access services within the mesh.

This article will focus on the second approach, which allows support for fine-grained access control on who can access services within the service mesh. This feature is newly added and available in release 1.2.0 [osm-edge v1.2.0](https://github.com/flomesh-io/osm-edge/releases/tag/v1.2.0).

Access Control can be configured via two resource types: Service and IP range. In terms of data transmission, it supports plaintext transmission and mTLS-encrypted traffic.

Let's get started with a demo.

## Demo environment preparation

### Kubernetes cluster

Using the minimalist Kubernetes distribution [k8e](https://getk8e.com):

```shell
curl -sfL https://getk8e.com/install.sh | K8E_TOKEN=k8e-mesh INSTALL_K8E_EXEC="server --cluster-init --write-kubeconfig-mode 644 --write-kubeconfig ~/.kube/config" sh -
```

### osm-edge CLI

```shell
system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.2.0
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
./${system}-${arch}/osm version
cp ./${system}-${arch}/osm /usr/local/bin/
```

### Install osm-edge

Execute the following commands to install the related components of osm-edge.

```shell
export osm_namespace=osm-system
export osm_mesh_name=osm

osm install \
     --mesh-name "$osm_mesh_name" \
     --osm-namespace "$osm_namespace" \
     --set=osm.image.pullPolicy=Always
```

Check to make sure all pods are up and running properly.

### Deploy the sample application

```shell
#Mock target service
kubectl create namespace httpbin
osm namespace add httpbin
kubectl apply -n httpbin -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/httpbin/httpbin.yaml

#Mock external service
kubectl create namespace curl
kubectl apply -n curl -f https://raw.githubusercontent.com/flomesh-io/osm-edge-docs/main/manifests/samples/curl/curl.yaml

#Wait for the dependent POD to start normally
kubectl wait --for=condition=ready pod -n httpbin -l app=httpbin --timeout=180s
kubectl wait --for=condition=ready pod -n curl -l app=curl --timeout=180s
```

## Demo

At this point, we send a request from service  `curl` to target service `httpbin` by executing the following command.

```shell
kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
command terminated with exit code 56
```

The access fails because by default the services outside the mesh cannot access the services inside the mesh and we need to apply an access control policy.

Before applying the policy, you need to enable the access control feature, which is disabled by default.

```shell
kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessControlPolicy":true}}}' --type= merge
```

### Plaintext transfer

Data can be transferred in plaintext or with two-way TLS encryption. Plaintext transfer is relatively simple, so let's demonstrate the plaintext transfer scenario first.

#### Service-based access control

First, create a `Service` for service `curl`.

```shell
kubectl apply -n curl -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: curl
  labels:
    app: curl
    service: curl
spec:
  ports:
    - name: http
      port: 80
  selector:
    app: curl
EOF
```

Next, create an access control policy with the source `Service` `curl` and the target service `httpbin`.

```shell
kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: curl
    name: curl
EOF
```
Execute the command again to send the authentication request, and you can see that this time an `HTTP 200` response is received.

```shell
kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 08:47:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive
```

Before continuing with the rest of the demonstration, execute the command `kubectl delete accesscontrol httpbin -n httpbin` to delete the policy you just created.

#### IP range-based access control, plaintext transfer

First, get the pod IP address of the service `curl`.

```shell
curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"
```

Access control using IP ranges is simple, just set the access source type to `IPRange` and configure the IP address you just obtained.


```shell
kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
EOF
```

Execute the command again to test if the control policy is in effect.

```shell
kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -sI http://httpbin.httpbin:14001/get
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 09:20:57 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
connection: keep-alive
```

Remember to execute `kubectl delete accesscontrol httpbin -n httpbin` to clean up the policy.

The previous ones we used were plaintext transfers, next we look at encrypted transfers.

### Encrypted transfers

The default access policy certificate feature is off, turn it on by executing the following command.

```shell
kubectl patch meshconfig osm-mesh-config -n "$osm_namespace" -p '{"spec":{"featureFlags":{"enableAccessCertPolicy":true}}}' --type=merge
```

Create `AccessCert` for the access source to assign a certificate for data encryption. The controller will store the certificate information in `Secret` `curl-mtls-secret` under the namespace `curl`, and here also assign SAN `curl.curl.cluster.local` for the access source.

```shell
kubectl apply -f - <<EOF
kind: AccessCert
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: curl-mtls-cert
  namespace: httpbin
spec:
  subjectAltNames:
  - curl.curl.cluster.local
  secret:
    name: curl-mtls-secret
    namespace: curl
EOF
```

Redeploy `curl` and mount the system-assigned `Secret` to the pod.

```shell
kubectl apply -n curl -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: curl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      labels:
        app: curl
    spec:
      serviceAccountName: curl
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - image: curlimages/curl
        imagePullPolicy: IfNotPresent
        name: curl
        command: ["sleep", "365d"]
        volumeMounts:
        - name: curl-mtls-secret
          mountPath: "/certs"
          readOnly: true
      volumes:
        - name: curl-mtls-secret
          secret:
            secretName: curl-mtls-secret
EOF
```

#### Service-based Access Control

The next step is to create an access control policy that uses encrypted transport. When configuring the target service, enable client certificate checking by specifying `tls.skipClientCertValidation = false`. For the access source, in addition to the `Service` type of access source, the SAN `curl.curl.cluster.local` should be specified via `AuthenticatedPrincipal`.

```shell
kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: Service
    namespace: curl
    name: curl
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF
```

Test whether the access policy is effective. When sending the request, we have to specify the CA certificate, key and certificate to be used for the `curl` command of the access source, which will be responded to by `HTTP 200` normally.

```shell
kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:44:05 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
```

#### IP Range-Based Access Control

After service-based access control, IP range-based control is simple. You just need to specify the type of access source as `IPRange` and specify the IP address of the access source. As the application `curl` is redeployed, its IP address needs to be retrieved (perhaps you have discovered the drawbacks of IP range-based access control).

```shell
curl_pod_ip="$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items[0].status.podIP}')"
```

```shell
kubectl apply -f - <<EOF
kind: AccessControl
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
    tls:
      skipClientCertValidation: false
  sources:
  - kind: IPRange
    name: ${curl_pod_ip}/32
  - kind: AuthenticatedPrincipal
    name: curl.curl.cluster.local
EOF
```

Send the request again for testing.

```shell
kubectl exec "$(kubectl get pod -n curl -l app=curl -o jsonpath='{.items..metadata.name}')" -n curl -- curl -ksI https://httpbin.httpbin:14001/get --cacert /certs/ca.crt --key /certs/tls.key --cert /certs/tls.crt
HTTP/2 200
server: gunicorn/19.9.0
date: Mon, 07 Nov 2022 10:58:55 GMT
content-type: application/json
content-length: 267
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-69dc7d545c-qphrh
```

Bingo! The access is successful, indicating that our policy has taken effect

## Summary

This blog post will talk about the approaches which can be used to enable services outside of the service mesh to communicate with services within the osm-edge service mesh. There may exist use cases where you have a specific set of services that aren't yet ready for migration to service mesh or for some reason can't be migrated yet.

The access control approaches presented in this article are suitable for solving such problems, and allow you to choose the appropriate type of access source and whether to transfer data in plaintext or encrypted, depending on your needs. The control is more granular than using a unified Ingress for access.