Using Flux, a GitOps Tool, Using Istio, a Service Mesh, with Amazon Elastic Kubernetes Service (EKS) - Part 3
Tags: aws, eks, servicemesh, istio
Published: 2023-11-10 22:20:31

This is the third and final part of the series on Using Istio, a Service Mesh, with Amazon Elastic Kubernetes Service (EKS). In the last two articles, we discussed various topics. These include what Service Mesh is, what Istio is, what technologies we used, the prerequisites and architecture overview, reviewed the Terraform code used to deploy the AWS resources and successfully deployed the Terraform Code to AWS. We also installed the the Istio CLI tool, istioctl, ran the "configure.sh" script to configure the Flux Repository, and installed Flux to the Amazon EKS cluster.  Finally, we reviewed the Addons and Applications managed by Flux, discussed the Istio Components and Addons used by Istio, and reviewed how Istio works with Applications and Microservices.

In this final and third article, we will cover the following topics.

- Access the Applications managed by Flux
- Demonstrate how Istio works
- Review the Istio Addons
- Clean up apps and infrastructure

## Access Applications managed by Flux

Let's access the Applications managed by Flux. Retrieve the ***public_base_domain_name*** specified in the "locals.tf" when you ran Terraform earlier.

### Access Bookinfo App

For the Bookinfo App, you can access the Bookinfo URL "https://bookinfo.public_base_domain_name/productpage" in your favorite web browser.

![Bookinfo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yn4e8pb4utqrtvvrsuok.png)
![Bookinfo](./images/bookinfo.png)

### Access Podinfo App

For the Podinfo App, you can access the Podinfo URL "https://podinfo.public_base_domain_name" in your favorite web browser.

![Podinfo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cb91kzawqyygfat0gwfj.png)

## Access Istio Addons

Let's access the Istio Addons before we demonstrate how Istio works. Retrieve the ***public_base_domain_name*** specified in the "locals.tf" when you ran Terraform earlier.

### Access Kiali Dashboard

To access the Kiali Dashboard, follow these steps.

1. Access the Kiali URL "https://kiali.public_base_domain_name" in your favorite web browser.
  
    ![Kiali Login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xprtz414jjtdvqo7h58a.png)    

2. Generate and retrieve **token** by running the following command

    ```bash
    kubectl -n istio-system create token kiali-service-account
    ```
     
    ![Kiali Token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9m92dijhkq87vzr4nivq.png)

3. Copy and paste the generated token from above to the Kiali Login page.
    
    ![Kiali Login with Token](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xfyq89kfjvziev6440p2.png)

4. Log into the Kiali Dashboard.

    ![Kiali Overview Dashboard after login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3pja6j7nsuy0nwhd1qwm.png)

### Access Grafana Dashboard

To access the Grafana Dashboard, follow these steps.

1. Access the Kiali URL "https://grafana.public_base_domain_name" in your favorite web browser.

    ![Grafana Login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/otg6lm9dnyl1n1ley10l.png)

2. Log into Grafana with the following username and password.

   - Username: admin
   - Password: Grafana&Git0ps

    ![Grafana After Login](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/g1wx6k0tg0e65280iamz.png)

## Demonstrate how Istio works

Now that we know how to access the Applications let's demonstrate how Istio works. We will send several requests to the Bookinfo and Podinfo applications by running curl commands from the terminals or shells. We will do this to simulate enough traffic for Istio and the Istio Addons to capture the network traffic so the dashboards in Kiali and Grafana can display correctly.

## Populate Data for Kiali and Istio Dashboards in Grafana

1. Retrieve the ***public_base_domain_name*** specified in the "locals.tf" when you ran Terraform earlier.

2. Open two separate terminals or shells.

   **NOTE**: Run these commands as often as you want to populate the data in Kiali and Grafana.

3. Run the following command in the first terminal or shell to send traffic to the Bookinfo application. Replace <public_base_domain_name> with the actual domain name.

    ```bash
    for i in {1..720}; do curl -s -o /dev/null "https://bookinfo.<public_base_domain_name>/productpage" ; done
    ```

4. In the second terminal or shell, run the following command to send traffic to the Podinfo application. Replace <public_base_domain_name> with the actual domain name.
 
    ```bash
    for i in {1..720}; do curl -s -o /dev/null "https://podinfo.<public_base_domain_name>" ; done
    ```

It will take 4 to 5 minutes for the above commands to finish.

## Access and Review Data in Kiali Dashboard

While the commands above run, re-access the Kiali Dashboard and review the data.

1. Let's change the time range to the last 10 minutes by clicking in the upper right corner of the web page. Click on "Last 1m" to "Last 10m"..

    ![Kiali change to Last 10m](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qhksc94s17lytc9182f3.png)

2. Click on "Graph" on the left side of the Kiali Dashboard.

    ![Kiali change to Graph](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/irhco7u1z08ucvuap8nz.png)

3. Click on "Select Namespaces" and choose "bookinfo".

    ![Kiali Graph Select NS](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x666t0nmqd4fvtzoyoyk.png)

    ![Kiali Graph Select Bookinfo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/x33bwh36kpqd1cf0g7uo.png)

4. Click anywhere on the Kiali Dashboard, and you should see something similar.

    ![Kiali Graph Bookinfo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v98dpgmeku5sgv269qlp.png)

5. Click on "Display" and select "Response Time" and "Security".

    ![Kiali Graph Bookinfo Change Display](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wgufm3i4ryhqh2ti8299.png)

6. Click anywhere on the Kiali Dashboard again, and you should see the response times and a little green lock on each connection.

    ![Kiali Graph Bookinfo Display Security](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/byjp3ar7zbugu55y2dpt.png)

7. Click on "productpage v1" on the Kiali Graph Dashboard, and you should see something like this. It will display information on the right-hand side of the Dashboard, such as Traffic and Traces.

    ![Kiali Graph Bookinfo Productpage V1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9ij3xpqwnwpwbcpc9rwx.png)

8. Let's add the podinfo namespace to the Kiali Graph Dashboard. Click Namespace again and then podinfo. After selecting, you should see something like this.

    ![Kiali Graph Bookinfo Podinfo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/aq30lfblibxz9m5h6gpp.png)

9. Feel free to explore other parts of the Kiali Dashboard. The Kiali Dashboard pulls its data from Prometheus, Grafana, and Jaeger.

## Access Istio Dashboards in Grafana

Let's re-access the Grafana Dashboards for Istio and review the data. For details on what each Istio Dashboard does, click [here](https://istio.io/latest/docs/tasks/observability/metrics/using-istio-dashboard/).

**NOTE**: If the data is not showing in the Grafana Dashboards for Istio, re-run the commands from above to populate data for Bookinfo and Podinfo again.

1. Change to the Istio Dashboards in Grafana by clicking on Home in the upper left of the Grafana home page

    ![Grafana Home Button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wqcngnzlzgesw06zvged.png)

2. Click on Dashboards

    ![Grafana Dashboards Button](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ufaaveyzbqzq2o14z0xs.png)

3. Hover over Istio and then click "Go to folder".

    ![Grafana Dashboards Hover](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cobueazell43lw0srfh9.png)

4. Istio Dashboards should now appear.

    ![Istio Dashboards Folder](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s27bhmvok6zl9onn3foj.png)

5. Click on Istio Mesh Dashboard In Grafana. This dashboard provides a global view of all services and applications used in the Service Mesh.

    ![Istio Mesh Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j1vpcwwuaqbjavx8i86h.png)

6. Let's navigate to the Istio Service Dashboard by clicking on Istio in the upper left-right next to "Home > Dashboards > Istio" and then click "Istio Service Dashboard".


    ![Istio Mesh Dashboard Istio Folder](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wcp97h4q5njc63xyaiwf.png)

    ![Istio Folder Istio Service Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/955fq32y1gmc6ue6w7w4.png)

    ![Istio Service Dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wbzld1hoicefwqx96wvk.png)

7. Click on service on the Istio Service Dashboard and change to "productpage.bookinfo.svc.cluster.local" service and then expand General, Client Workloads, and Service Workloads sections.

    ![Istio Service Dashboard Product Page](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/yrffhqubgnu24gw90jnu.png)

8. Review the General, Client Workloads, and Service Workloads sections. In the Client Workloads and Service Workloads sections, you can see that mutual TLS (mTLS) is enabled and working correctly.

    ![Istio Service Dashboard Client Workloads](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/msw4d66rmipe8zjbb8ul.png)

9. Feel free to explore other parts of the Istio Dashboards in Grafana.

## Clean Up

### Clean up Applications managed by Flux from Kubernetes

1. Suspend Applications managed by Flux

    ```bash
    flux suspend source git flux-system
    flux suspend kustomization apps flux-system \
      monitoring-controllers monitoring-configs
    ```

2. Delete Applications managed by Flux

    ```bash
    kubectl delete -f ./k8s/apps/base/bookinfo/config.yaml
    kubectl delete -f ./k8s/apps/base/bookinfo/cert_request.yaml
    kubectl delete -f ./k8s/apps/base/podinfo/config.yaml
    kubectl delete -f ./k8s/apps/base/podinfo/cert_request.yaml
    kubectl delete -f ./k8s/apps/base/bookinfo/release.yaml
    flux delete helmrelease -s podinfo
    flux delete helmrelease -s -n monitoring loki-stack
    flux delete helmrelease -s -n monitoring kube-prometheus-stack
    ```

3. Wait 1 to 5 minutes for Applications to be removed from Kubernetes

4. Delete Application sources managed by Flux

    ```bash
    flux delete source helm -s podinfo
    flux delete source helm -s -n monitoring grafana-charts
    flux delete source helm -s -n monitoring prometheus-community
    ```

5. Verify Applications are removed

    ```bash
    kubectl -n bookinfo get all
    kubectl -n istio-ingress get ingresses bookinfo-ingress
    kubectl -n podinfo get all
    kubectl -n istio-ingress get ingresses podinfo-ingress
    kubectl -n monitoring get all
    ```

### Clean up Infrastructure Applications managed by Flux from Kubernetes

1. Suspend Infrastructure Applications managed by Flux

    ```bash
    flux suspend kustomization infra-apps
    ```

2. Delete Infrastructure Applications managed by Flux

    ```bash
    kubectl delete -f ./k8s/infrastructure/apps/kiali/config.yaml --force=true --grace-period=0
    kubectl delete -f ./k8s/infrastructure/apps/kiali/cert_request.yaml
    kubectl patch kiali kiali -n istio-system -p '{"metadata":{"finalizers": []}}' --type=merge
    kubectl delete kiali --all --all-namespaces
    kubectl delete jaegers.jaegertracing.io -n observability jaeger
    flux delete helmrelease -s kiali
    flux delete helmrelease -s jaeger-operator
    kubectl delete crd kialis.kiali.io
    ```

3. Wait 1 to 5 minutes for Kubernetes Addons to be removed from Kubernetes

4. Delete Application sources managed by Flux

    ```bash
    flux delete source helm -s jaegertracing
    flux delete source helm -s kiali
   ```

5. Verify Kubernetes Addons were removed successfully

    ```bash
    kubectl -n kiali-operator get all
    kubectl -n observability get all
    kubectl -n istio-ingress get ingresses kiali-ingress
    ```

6. If any resources are not deleted, manually delete them.

### Clean up Kubernetes Addons managed by Flux from Kubernetes

1. Suspend Kubernetes Addons managed by Flux

    ```bash
    flux suspend kustomization infra-configs infra-controllers
    ```

2. Delete Kubernetes Addons managed by Flux

    ```bash
    flux delete helmrelease -s aws-load-balancer-controller
    flux delete helmrelease -s cert-manager
    flux delete helmrelease -s cluster-autoscaler
    flux delete helmrelease -s external-dns
    flux delete helmrelease -s istio-base
    flux delete helmrelease -s istiod
    flux delete helmrelease -s istio-ingressgateway
    flux delete helmrelease -s metrics-server
    ```

3. Wait 1 to 5 minutes for Kubernetes Addons to be removed from Kubernetes

4. Delete Application sources managed by Flux

    ```bash
    flux delete source helm -s cert-manager
    flux delete source helm -s cluster-autoscaler
    flux delete source helm -s eks-charts
    flux delete source helm -s external-dns
    flux delete source helm -s istio
    flux delete source helm -s jetstack
    flux delete source helm -s metrics-server
    ```

5. Verify Kubernetes Addons were removed successfully

    ```bash
    kubectl -n kube-system get all -l app.kubernetes.io/name=external-dns
    kubectl -n kube-system get all -l app.kubernetes.io/name=aws-load-balancer-controller
    kubectl -n kube-system get all -l app.kubernetes.io/name=aws-cluster-autoscaler
    kubectl -n cert-manager get all
    kubectl -n istio-ingress get ingresses kiali-ingress
    kubectl -n istio-ingress get all
    kubectl -n istio-system get all
    kubectl get ingressclasses -l app.kubernetes.io/name=aws-load-balancer-controller
    ```

6. If any resources are not deleted, manually delete them.

## Uninstall Flux from Kubernetes

1. Uninstall Flux

    ```bash
    flux uninstall -s
    ```

2. Verify Flux was removed successfully

    ```bash
    kubectl get all -n flux-system
    ```

### Clean up Terraform

1. Navigate to the root of the directory of the GitHub repo and run the following commands

    ```bash
    cd terraform

    terraform destroy
    ```

2. Check Terraform destroy results

    ![Terraform Destroy](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2fmyjofk1d51340ulceq.png)

## Conclusion

In conclusion, this guide provided a comprehensive overview of utilizing Istio, a service mesh, and Amazon EKS. A service mesh is an essential infrastructure component that enhances communication and operational capabilities within a microservices architecture. Istio, as a leading service mesh implementation, provides a robust set of features for traffic management, security, and observability without heavy lifting from developers. Istio works seamlessly with Amazon EKS, as EKS supports Kubernetes natively, and Istio is designed to work with Kubernetes-managed services. Users can leverage Istio within EKS to manage microservices traffic, enforce security policies, and observe communication patterns between services, all while taking advantage of the managed Kubernetes environment that EKS provides. This combination allows for a powerful, scalable, and secure system for managing microservices at scale.