Using FSM Ingress controller with osm-edge service mesh
Tags: ingress, servicemesh, pipy, kubernetes
Published: 2022-08-18 15:18:11

## Background

 The Kubernetes Ingress API is designed with a separation of concerns, where the Ingress implementation provides an entry feature infrastructure managed by operations staff; it also allows application owners to control the routing of requests to the backend through rules.

The [osm-edge](https://github.com/flomesh-io/osm-edge) supports multiple Ingress implementations to manage ingress traffic and provides the `IngressBackend` API to configure back-end services to receive access from trusted ingress points. This article focuses on the integration of osm-edge with [FSM](https://github.com/flomesh-io/fsm) to manage ingress traffic.

![FSM](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s49fww6fvf1r8rkiueoc.png)

### Introduction to FSM

[FSM](https://github.com/flomesh-io/fsm) is another open-source product from [Flomesh](https://flomesh.io) for Kubernetes north-south traffic, gateway api contoller, and multi-cluster management. FSM uses [Pipy](https://flomesh.io), a programmable proxy at its core, and provides an Ingress controller, *Gateway API* controller, load balancer, cross-cluster service registration discovery, and more.


## Integration with FSM

FSM is already integrated inside osm-edge and can be enabled during osm-edge installation; it can also be installed independently via helm for existing osm-edge mesh.

### Prerequisites

* Kubernetes cluster, version 1.19.0 and higher
* Helm 3 CLI for standalone installation of FSM

Download the osm-edge CLI at

```shell
system=$(uname -s | tr [:upper:] [:lower:])
arch=$(dpkg --print-architecture)
release=v1.1.1
curl -L https://github.com/flomesh-io/osm-edge/releases/download/${release}/osm-edge-${release}-${system}-${arch}.tar.gz | tar -vxzf -
. /${system}-${arch}/osm version
cp . /${system}-${arch}/osm /usr/local/bin/
```

### Integrated installation

```shell
export osm_namespace=osm-system 
export osm_mesh_name=osm 

osm install --set fsm.enabled=true \
    --mesh-name "$osm_mesh_name" \
    --osm-namespace "$osm_namespace" 
```

### Standalone installation

If osm-edge is installed without FSM enabled, you can use a standalone installation to install it.

```shell
helm repo add fsm https://charts.flomesh.io

export fsm_namespace=osm-system 
helm install fsm fsm/fsm --namespace "$fsm_namespace" --create-namespace
```

Verify that all pods are up and running properly.

```shell
kubectl get pods -n osm-system
NAME READY STATUS RESTARTS AGE
repo-8756f76fb-2f78g 1/1 Running 0 5m53s
manager-866585bbd5-pbg7q 1/1 Running 0 5m53s
osm-bootstrap-7c6689ff57-47ksk 1/1 Running 0 5m53s
osm-controller-57888cfc7c-tnqxl 2/2 Running 0 5m52s
osm-injector-5f77898899-45f65 1/1 Running 0 5m53s
bootstrap-fd5894bcc-nr7hf 1/1 Running 0 5m53s
cluster-connector-local-68c7584c8b-qf7xm 1/1 Running 0 2m43s
ingress-pipy-6fb8c8b794-pgthl 1/1 Running 0 5m53s
```

### Configuration

In order to authorize clients by restricting access to backend traffic, we will configure `IngressBackend` so that only ingress traffic from the `ingress-pipy-controller` endpoint can be routed to the backend service. In order to discover the `ingress-pipy-controller` endpoint, we need the `osm-edge controller` and the corresponding namespace to monitor it. However, to ensure that the FSM functions properly, it cannot be injected with sidecar.

```shell
kubectl label namespace "$osm_namespace" openservicemesh.io/monitored-by="$osm_mesh_name"
```

Save the external IP address and port of the entry gateway, which will be used later to test access to the backend application.

```shell
export ingress_host="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}') "
export ingress_port="$(kubectl -n "$osm_namespace" get service ingress-pipy-controller -o jsonpath='{.spec.ports[? (@.name=="http")].port}')"
echo $ingress_host:$ingress_port
```

## Deploying the sample service

The next step is to deploy the sample `httpbin` service. 

```shell
## Create namespace kubectl create ns httpbin
kubectl create ns httpbin

# Add the namespace to the grid
osm namespace add httpbin

# Deploy the application
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 14001
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      serviceAccountName: httpbin
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        command: ["gunicorn", "-b", "0.0.0.0:14001", "httpbin:app", "-k", "gevent"]
        ports:
        - containerPort: 14001
EOF
```

Verify that the pod and service are created and the application is running successfully.

```shell
kubectl get pods,services -n httpbin
NAME READY STATUS RESTARTS AGE
pod/httpbin-54cc8cf5d-7vclc 2/2 Running 0 14s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/httpbin ClusterIP 10.43.105.166 <none> 14001/TCP 14s
```

## Configure entry rules

Next we want to access the deployed `httpbin` service from outside the cluster and need to provide the ingress configuration rules.

```shell
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  namespace: httpbin
  annotations:
    pipy.ingress.kubernetes.io/rewrite-target-from: /httpbin
    pipy.ingress.kubernetes.io/rewrite-target-to: /httpbin
spec:
  ingressClassName: pipy
  rules:
  - host: httpbin.org
    http:
      paths:
      - path: /httpbin
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 14001
EOF
```

Accessing the `httpbin` service using the IP address and port recorded above will result in the following `502 Bad Gateway` error response. This is because we have not set the ingress of the FSM as a trusted portal.

```shell
curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 502 Bad Gateway
content-length: 0
connection: keep-alive
```

Execute the following command to set the FSM ingress as a trusted entry.

```shell
kubectl apply -f - <<EOF
kind: IngressBackend
apiVersion: policy.openservicemesh.io/v1alpha1
metadata:
  name: httpbin
  namespace: httpbin
spec:
  backends:
  - name: httpbin
    port:
      number: 14001 # targetPort of httpbin service
      protocol: http
  sources:
  - kind: Service
    namespace: "$osm_namespace"
    name: ingress-pipy-controller
EOF
```

Try requesting the `httpbin` service again, and you will be able to access it successfully.

```shell
curl -sI http://"$ingress_host":"$ingress_port"/httpbin/get -H "Host: httpbin.org"
HTTP/1.1 200 OK
server: gunicorn/19.9.0
date: Thu, 18 Aug 2022 05:18:50 GMT
content-type: application/json
content-length: 241
access-control-allow-origin: *
access-control-allow-credentials: true
osm-stats-namespace: httpbin
osm-stats-kind: Deployment
osm-stats-name: httpbin
osm-stats-pod: httpbin-54cc8cf5d-7vclc
connection: keep-alive
```

## Summary

FSM Ingress exposes the application access points within the Kubernetes cluster for easy management of portal traffic. osm-edge's IngressBackend API provides an additional line of defense against accidental data leakage by exposing services within the mesh to the public.