Implementing resilient applications with API Gateway (Health Check)
Tags: api, microservices, apisix, apigateway
Published: 2022-08-13 10:35:00

## API Health Check 🩺

We know that **API services** fail due to any number of reasons, such as networks issues, connection (failed to open a connection to a data source like a [SQL Server database](https://www.microsoft.com/en-us/sql-server/sql-server-downloads)) and API performance issues, failure to authenticate to dependencies, outage of a shared dependency, crash because of critical bugs, memory leaks and many more😬. In such scenarios, our services should be resilient enough to deal with predictable failures when they happen before they become more significant headaches🤯 than they need to be. Dealing with single server failures is more effortless, as the number of applications grows, you face new challenges in monitoring the health of each microservice to understand how your microservices-based application behaves when any of them becomes unavailable.

As a part of **API monitoring** the best practices, **health check**❤️ allows immediate-real-time information about the state of your APIs, containers, and microservices. An [API health check](https://microservices.io/patterns/observability/health-check-api.html) is able to check anything that may interrupt the API from serving incoming requests😎.

For example, when you are using orchestrators like [Kubernetes](https://kubernetes.io/) and [Service Fabric](https://azure.microsoft.com/en-us/services/service-fabric/) periodically perform health checks and determine that a `service/container` is unhealthy, it stops routing requests to that instance. It also usually creates a new instance of that container.

![Health check API Gateway APISIX](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3rapdefjg7nzssmikxbx.png) 

## API Health Check with an API Gateway

The simplest and standardized way to validate the status of a service is to define a new [health check](https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/monitor-app-health) endpoint like `/health` or `/status` as a separate [REST service](https://en.wikipedia.org/wiki/Representational_state_transfer) implemented within a microservice component. [ASP.NET Core offers](https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-6.0) Health Checks Middleware and libraries for reporting the health of app components. 

👉🏼 **Health checks** are exposed by a microservice as HTTP/HTTPS endpoints.

Then enable a health check managing functionality at the **API Gateway** level. [API Gateway](https://apisix.apache.org/docs/apisix/terminology/api-gateway/) acts as an **orchestrator** that can use this status report to decide how to manage the traffic, load balance to a healthy node, fail-fast due to some cascading failures or simply alerts you when it notices something goes wrong. API Gateway also ensures that routing and other network-level components work together successfully to deliver a request to the API process. It helps you to detect in the early stage and fix issues for your running application much more easily.

## Apache APISIX Upstream Health Check 

[Apache APISIX](https://apisix.apache.org/) API Gateway supports almost all modern resiliency patterns like the _timeouts, fallbacks, retry, and circuit breaker_ I discussed in another blog post **[Implementing resilient applications with API Gateway (Circuit breaker)](https://dev.to/apisix/implementing-resilient-applications-with-api-gateway-circuit-breaker-ggk)**, which can also be applied to your Microservices APIs. Apache APISIX [Health check mechanism](https://apisix.apache.org/docs/apisix/health-check/) performs health checks on each target upstream service regularly, marking it as healthy or unhealthy based on whether they are responsive or not. You can also retrieve anytime the health status of specific nodes by using the [Control API](https://apisix.apache.org/docs/apisix/control-api/)-health check endpoint.

## Active and Passive Health Checks

APISIX provides two types of health checks:

- ➕ _Active checks_ - APISIX periodically requests a health check API endpoint exposed by a target (backend service) to identify if an upstream service is available to serve the request.

- ➖_Passive checks_, APISIX acts as a proxy for operations that might fail. The proxy monitors the number of recent failures that have occurred and uses this information to decide whether to allow the operation to proceed, or simply return an exception directly. The passive health check is also known as a [Circuit breaker](https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/implement-circuit-breaker-pattern). 

## How Upstream Active Health Check works❓

💁In this blog post, we are focusing on mainly **the active health** check and how to use it, I covered the passive health check approach in more depth [here](https://medium.com/apache-apisix/implementing-resilient-applications-with-api-gateway-circuit-breaker-dc9b3096992e) by using of [API Breaker Plugin](https://apisix.apache.org/docs/apisix/plugins/api-breaker/).

You can enable and manage the **active health check** through the [upstream configuration](https://apisix.apache.org/docs/apisix/terminology/upstream/) where health check configuration settings are defined.
To enable active health checks for the upstream, you need to specify the configuration items such as _counters, statuses or interval_ (how often API health status is determined in seconds), and here a short breakdown of different scenarios that might happen:

🟢 If the upstream returns a healthy HTTP status code, it will increment the `upstream.checks.active.healthy.successes `counter for the target node. If the success counter reaches the limit specified in the configuration, then the upstream node will be marked as _healthy_. 
 
🔴 If it fails to connect, it will increment the `upstream.checks.active.unhealthy.http_failures` counter for the target node. If this counter reaches the limit specified in the configuration, then the upstream node will be marked as _unhealthy_ respectively.  

There are other properties like `timeouts`, and `tcp_failures`, you can discover more about them in the [documentation](https://apisix.apache.org/docs/apisix/health-check/). 

## Apache APISIX Upstream Health Check demo

🙋🏼 Once we have the little background on how [Apache APISIX](https://apisix.apache.org/) handles health checking for its upstream services, let's jump into the demo of implementing the health check orchestration mechanism for our existing sample project [ASP.NET Core WEB API](https://github.com/Boburmirzo/apisix-dotnet-docker) with a single GET endpoint (retrieves all products list).

### Prerequisites

☝️ If you followed the previous blog post about [Manage .NET Microservices APIs with Apache APISIX API Gateway](https://dev.to/apisix/manage-net-microservices-apis-with-apache-apisix-api-gateway-2cbk), make sure you have read it and completed steps (To run APISIX, etcd, and ASP.NET WEB API) before continuing with a demo session.

### Register health check feature in the backend

To begin, we need to use the [HealthChecks](https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-6.0) feature in our back-end Product ASP.NET microservice. With the help of [built-in APIs](https://www.nuget.org/packages/Microsoft.Extensions.Diagnostics.HealthChecks) in .NET, we can configure the health check service easily. 


The following example creates a health check endpoint at `/api/health` path. We need to just add this code to `Configure` method in `Startup.cs` file

``` c#
app.UseHealthChecks("/api/health");
```

Next, we need to add simple logic to generate random health statuses each time you request the endpoint gives different result by indicating the health as _Healthy, Degraded, or Unhealthy._ Of course, you can add your custom logic to define the health status properly.

``` c#
services.AddHealthChecks()
    .AddCheck("random-health-status", () =>
    {
        Random rnd = new();
        for (int j = 0; j < 100; j++)
        {
           var rndNum = rnd.Next();
            if (rndNum % 2 == 0)
            {
                return HealthCheckResult.Unhealthy();
            }
            else {
                return HealthCheckResult.Healthy();
            } 
        }
        return HealthCheckResult.Degraded();
    });

```

With the above changes, now we added the health check functionality to our API. If you use `curl` cmd to make an HTTP request to the health check endpoint or navigate to the following URL on your browser, you will see the health endpoint is responding. 

![Apache APISIX HealthCheck](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wb5rinuvbum2m87tldax.png)

To make the example close to reality, you can also add one more instance of Product microservice. Simply, define another service in `docker-compose.yaml` file running in different ports like below:

``` yaml
...
  productapi1:
    image: productapi
    build:
      context: ./productapi
      dockerfile: Dockerfile
    ports:
      - "5555:80"    
    networks:
      apisix:
  productapi2:
    image: productapi
    build:
      context: ./productapi
      dockerfile: Dockerfile
    ports:
      - "5556:80"    
    networks:
      apisix:
...
 
```
 
### Create an Upstream with Health Check configuration

To use the health check feature effectively and automate monitoring in the API Gateway, you need to first create a new [Upstream](https://apisix.apache.org/docs/apisix/terminology/upstream/), configure its properties and define a new [Route](https://apisix.apache.org/docs/apisix/terminology/route/) with the upstream health check capability.

The following upstream configuration example monitors the health check of two nodes `productapi1` and `productapi2` by triggering every 2 seconds `/api/health` endpoint of each, if any of them is not available(unhealthy), it forwards the requests to a healthy node or report service unavailable error:

``` bash
curl "http://127.0.0.1:9080/apisix/admin/upstreams/1" -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" -X PUT -d '
{
 "nodes": {
     "productapi1:80": 1,
	 "productapi2:80": 1
 },
 "type": "roundrobin",
 "checks": {
     "active": {
		 "type": "http",
         "http_path": "/api/health",
         "healthy": {
             "interval": 2,
             "successes": 1
         },
         "unhealthy": {
             "interval": 1,
             "http_failures": 2
         }
     }
   }
}'
```

☝️ You can use `the upstream.checks.active.type` field to specify whether to perform HTTP or HTTPS probes.

### Create a Route for the upstream

Next, we will configure new route so that APISIX can forward the request to the corresponding upstream service we created in the previous step:

``` bash
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "name": "Route for health check",
    "methods": ["GET"], 
    "uri": "/api/products", 
    "plugins": {},
	"upstream_id": "1"
}'
```

> Note that we can also achieve the same configuration results above with the CLI as with the [Apache APISIX Dashboard](https://apisix.apache.org/docs/dashboard/USER_GUIDE/). You can learn more about the usage of the dashboard in the [Getting started with Apache APISIX Dashboard](https://youtu.be/-9-HZKK2ccI) video tutorial. 

### How to validate Health Check🙎  

We can easily test how Apache APISIX monitors health checks of targets by stopping `productapi1` or `productapi1` container from docker 🤪 which causes the `503 Service Temporarily Unavailable` error and when one target is unhealthy, you can see the requests are forwarded to another healthy node. The following curl cmd to simply trigger the `/api/product` endpoint:

``` curl
curl http://localhost:9080/api/products -i
```

If you look closer into the logs of both `productapi` service instances, you can also notice the endpoint receives from APISIX many times requests to check health status.

``` text
apisix-dotnet-docker-productapi1-1       | fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
apisix-dotnet-docker-productapi1-1       |       Health check random-health-status with status Unhealthy completed after 0.0081ms with message '(null)'
```
 

## What's next ➡️

In this blog post, we learned the usage of **Apache APISIX API Gateway** to enable health monitoring for microservices APIs. There are many aspects of using **API health check endpoints** for API load balancing, reporting metrics, and testing APIs dependencies, such as databases and external service endpoints, to confirm availability. APISIX can be also the central point for all types ( logging, monitoring, and tracing) of API Observability and further derive useful data into other monitoring, analytics, and visualization solutions like Prometheus, Grafana, or Elasticsearch. Learn more about [API Observability with Apache APISIX Plugins](https://medium.com/apache-apisix/api-observability-with-apache-apisix-plugins-ce32cb3311a9).

## Related resources

➔ [Implementing resilient applications with API Gateway (Circuit breaker)](https://dev.to/apisix/implementing-resilient-applications-with-api-gateway-circuit-breaker-ggk).

➔ [Manage .NET Microservices APIs with Apache APISIX API Gateway](https://medium.com/@ApacheAPISIX/manage-net-microservices-apis-with-apache-apisix-api-gateway-13e85853876f).

➔ [API Gateway Caching for ASP.NET Core WEB API](https://medium.com/apache-apisix/api-gateway-caching-for-asp-net-core-web-api-e7ce01ecb9ed)

## Recommended content 💁

➔ Watch Video Tutorial [Getting Started with Apache APISIX](https://youtu.be/dUOjJkb61so).

➔ Watch Video Tutorial [Manage .NET Microservice API with Apache APISIX API Gateway](https://youtu.be/nU8-uQkAHA4).

➔ Read the blog post [Overview of Apache APISIX API Gateway Plugins](https://dev.to/apisix/overview-of-apache-apisix-api-gateway-plugins-2m8o).

➔ Read the blog post [Run Apache APISIX on Microsoft Azure Container Instance](https://dev.to/apisix/run-apache-apisix-on-microsoft-azure-container-instance-1gdk).

➔ Read the blog post [API Security with OIDC by using Apache APISIX and Microsoft Azure AD](https://dev.to/apisix/api-security-with-oidc-by-using-apache-apisix-and-microsoft-azure-ad-50h3).

➔ Read the blog post [API Observability with Apache APISIX Plugins](https://dev.to/apisix/apis-observability-with-apache-apisix-plugins-1bnm).


## Community⤵️

🙋 [Join the Apache APISIX Community](https://apisix.apache.org/docs/general/join/) 
🐦 [Follow us on Twitter](https://twitter.com/ApacheAPISIX)
📝 [Find us on Slack](https://join.slack.com/t/the-asf/shared_invite/zt-vlfbf7ch-HkbNHiU_uDlcH_RvaHv9gQ)
📧 [Mail to us](dev@apisix.apache.org) with your questions.