How to monitor your requests between multiple applications... 🤔
Published: 2023-10-16 11:20:53

# TL;DR

Imagine that you are sending an HTTP request from one app; this one sends it to another app that sends it to another app. **What if something happens along the way?**

You need to mark all the requests to pinpoint where the problem occurred.

In this tutorial, you'll learn how to use OpenTelemetry to monitor your applications with insights into performance, tracing, and metrics.

**You will learn:**

- How to monitor our application using OpenTelemetry.
- How to view logs and ship traces.
- How to view Metrics and filter the traces.
- How to automate the monitoring of applications.



![Logged](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/83qk71hrarm24iis4477.gif)

<hr />

## Odigos - Open-source Distributed Tracing

**Monitor all your apps simultaneously without writing a single line of code!**
Simplify OpenTelemetry complexity with the only platform that can generate distributed tracing across all your applications.

We are really just starting out.
Can you help us with a star? Plz? 😽

https://github.com/keyval-dev/odigos

[![GiveStar](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k6l36g0qcds52ww1do96.png)](https://github.com/keyval-dev/odigos)


* * *

Let's set it up🔥
------------------------------------

Here, I'll walk you through installing the package dependencies required for this project.

Create your project folder for the web application as done below.
 
```bash
mkdir monitor-app
cd monitor-app
mkdir server
```

### [](#setting-up-the-nodejs-server)Setting up the Node.js server 🤖

Navigate into the server folder and create a `package.json` file.

```bash
cd server && npm init -y
```    

Install Express.  

```bash
npm install express
```

[ExpressJS](https://expressjs.com) is a back-end web application framework for building RESTful APIs with Node.js.

Create an `index.js` file - the entry point to our web server.  

```bash
// 👇server/index.js
touch index.js
```   

Set up a Node.js server using Express. The code snippet below returns a JSON object with one key when we do a `get` request to `/static-data` or `/fetch-data` endpoint in our browser.
 
```javascript
 // 👇server/index.js
const express = require("express");
const app = express();
const PORT = 8000;

app.use(express.urlencoded({ extended: true }));
app.use(express.json());

app.get("/static-data", (req, res) => {
  res.json({
    msg: "Hello world!",
  });
});

app.get("/fetch-data", (req, res) => {
  // Simulate a real-world database call
  setTimeout(() => {
    res.json({
      msg: "Hello world!",
    });
  }, 2000);
});

app.listen(PORT, () => {
  console.log(`The server is listening on port: ${PORT}`);
});
```

> **💡TIP OF THE DAY**  
> Since the node version [v18.11.0](https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V18.md#18.11.0) we no longer need **nodemon** to watch for changes and restart the server, it is now built into Node with `--watch` flag.

Configure `node --watch` by adding the `start` & `trace-start` command to the list of scripts in the `package.json` file.  

```json
// 👇server/package.json
"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1",
  "start": "node --watch index.js",
  "trace-start": "node -r ./tracing.js --watch index.js"
},
```

*   `start`: The code snippet starts the web server.
*   `trace-start`: This code snippet starts the web server while preloading the `tracing.js` file. We will be using this command for demonstration.

* * *

### [](#manually-setting-up-opentelemetry-and-signoz)Manually Setting up OpenTelemetry and SigNoz ⚒️

[OpenTelemetry](https://opentelemetry.io/) is used to instrument, generate, collect, and export telemetry data. [SigNoz](https://signoz.io/) is an open-source APM. It helps developers monitor their applications & troubleshoot problems, an open-source alternative to other APMs available. It is a one-stop solution to get access to metrics, logs, and traces of an application.

We'll start by installing the required OpenTelemetry packages.

> ⚠️ If you are running this on Windows, run each of these commands individually.  

```javascript
npm install --save @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http
```

> For **SigNoz** installation, follow the installation steps shown [here](https://signoz.io/docs/install/docker/).

Instantiate tracing by creating a `tracing.js` file and using the below code.

```javascript
    // 👇server/tracing.js
"use strict";
const process = require("process");
const opentelemetry = require("@opentelemetry/sdk-node");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-http");
const { Resource } = require("@opentelemetry/resources");
const {
  SemanticResourceAttributes,
} = require("@opentelemetry/semantic-conventions");

const exporterOptions = {
  url: "http://localhost:4318/v1/traces", // default URL for sending tracing data.
};

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
  traceExporter,
  instrumentations: [getNodeAutoInstrumentations()],
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "Monitor-Node-App",
  }),
});

// initialize the SDK and register with the OpenTelemetry API
sdk.start();

// shut down the SDK on process exit
process.on("SIGTERM", () => {
  sdk
      .shutdown()
      .then(() => console.log("Tracing terminated!"))
      .catch((error) => console.log("Error terminating tracing.", error))
      .finally(() => process.exit(0));
});
```

We are monitoring our Node application using the Node SDK `@opentelemetry/sdk-node` for tracing and metrics, `@opentelemetry/auto-instrumentations-node` for easy setup of instrumentations, and `@opentelemetry/exporter-trace-otlp-http` for exporting trace data to OTLP-compatible receivers via HTTP and JSON.

We are done with the basic instrumentation of our demo Node application with OpenTelemetry and SigNoz 🥂. Now, run the `trace-start` command.
 
```bash
npm run trace-start
```

Visit the SigNoz UI `http://localhost:3301`. We should be presented with something like this after we log in.

[![SigNoz Default UI](https://res.cloudinary.com/practicaldev/image/fetch/s--dURJPqvv--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/povsb2fu2o91b9vnk0pr.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--dURJPqvv--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/povsb2fu2o91b9vnk0pr.png)

Don't worry about all this data; it is pre-populated by SigNoz. Note that we do not see our `Monitor-Node-App` in the list of services. This is because we have not made requests to our server.

For SigNoz to show the traces, we need to generate some load on our server. For this, we can refresh endpoints multiple times on our browser or run the following CLI command on our endpoints `/static-data` or `/fetch-data` to automatically do that for us.

**For Linux Users**

> ℹ️ Make sure to have `curl` installed.  

```shell
for i in {1..20}; do curl http://localhost:8000/fetch-data; done
```

**For Windows Users**
  
```shell
1..20 | ForEach-Object { Invoke-RestMethod -Uri 'http://localhost:8000/fetch-data' }
```  

Now, on visiting the SigNoz UI once again we should see our service in the list of services.

[![SigNoz UI](https://res.cloudinary.com/practicaldev/image/fetch/s--Y5epQymO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7v2ngcoljg6hesyk7ay8.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--Y5epQymO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7v2ngcoljg6hesyk7ay8.png)

In this trace, we can exactly see how much time each part of the entire request took. In our case not many, since the request is fairly simple.

![Walkthrough of SigNoz](https://res.cloudinary.com/practicaldev/image/fetch/s--SVb4S3KZ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_66%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/04r9njmy6nhibvf74by5.gif)

For proper visualization, we will take a look at the request from one of the demo applications already provided with SigNoz.

[![SigNoz request Explain](https://res.cloudinary.com/practicaldev/image/fetch/s--0TnWMHz9--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jwv52ab669987cf2010y.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--0TnWMHz9--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jwv52ab669987cf2010y.png)

As we can clearly see, the first request which basically executes SQL Query goes through multiple steps, and, at last, it finally executes the SQL query. We can also see the time taken for each span in the request.

SigNoz's Traces dashboard offers ready-made visualizations for our tracing data. It includes strong filters to analyze the spans based on our selected criteria and chosen time frame.

[![SigNoz Dashboard](https://res.cloudinary.com/practicaldev/image/fetch/s--McleVa5N--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mnnz52nrpizob32xgin8.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--McleVa5N--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mnnz52nrpizob32xgin8.png)

* * *

### [](#visualize-metrics-from-hostmetrics)Visualize Metrics from Hostmetrics 👀

The most common way to accept metrics in SigNoz is through the Prometheus receiver.

For simplicity, I will demonstrate how to Visualize default metrics provided by Hostmetrics that is by default enabled on SigNoz installation.

All the list of available metrics can be found [here](https://signoz.io/docs/userguide/send-metrics/#metrics-from-hostmetrics-receiver).

Under the **Dashboards** section in the SigNoz UI, create a new **Time Series** panel.

[![SigNoz metric dashboard](https://res.cloudinary.com/practicaldev/image/fetch/s---y48FmWI--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xicj026at6x6lg1opr6q.png)](https://res.cloudinary.com/practicaldev/image/fetch/s---y48FmWI--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xicj026at6x6lg1opr6q.png)

In the PromQL query input, please enter the following: For example, let's use the metric `system_cpu_load_average_5m`, as the name suggests it gives the 5-minute load average of the CPU. Feel free to adjust the metric to your preference.

[![Metric Graph](https://res.cloudinary.com/practicaldev/image/fetch/s--P7YRBYxJ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j36cnp3jyr7usfeop4a4.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--P7YRBYxJ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/j36cnp3jyr7usfeop4a4.png)

* * *

[](#automatically-connect-with-odigos)Automatically connect with Odigos 🤝
--------------------------------------------------------------------------

[Odigos](https://odigos.io/) is an open-source observability control plane that enables organizations to create and maintain their observability pipeline.

Remember all the hassle we went through when [manually](#manually-setting-up-opentelemetry-and-signoz) setting up? With Odigos, there's no need for that. It automatically instruments our application, eliminating the need to set up OpenTelemetry or anything on our own. Odigos handles it all for us. 🤯

> Instructions on how to install **Odigos** can be found [here](https://docs.odigos.io/installation).

We need to install SigNoz on Kubernetes so that OpenTelemetry can send the collected data to it.

SigNoz can be installed on Kubernetes easily using Helm:  

```shell
helm repo add signoz https://charts.signoz.io
kubectl create ns platform
helm --namespace platform install my-release signoz/signoz
```

To port forward SigNoz UI on our local machine, run the following.

```shell
export POD_NAME=$(kubectl get pods --namespace platform -l "app.kubernetes.io/name=signoz,app.kuber
    netes.io/instance=my-release,app.kubernetes.io/component=frontend" -o jsonpath="{.items[0].metadata.name}")
    
kubectl --namespace platform port-forward $POD_NAME 3301:3301
```

Now, you can access the SigNoz UI on `http://localhost:3301`

For the demonstration, we will use another e-commerce application from Google to get a bigger picture of distributed tracing. The repo does not contain any instrumentation or monitoring setup.

Create a clone of this repository [microservices-demo](https://github.com/keyval-dev/microservices-demo)  

```bash
git clone https://github.com/keyval-dev/microservices-demo.git
```

Create a new local Kubernetes cluster, by running the following command. We will need it when setting up Odigos.

> ℹ️ For Windows Users, make sure to have Docker Desktop running.  

```bash
minikube start
```

> Instructions on how to install **minikube** can be found [here](https://minikube.sigs.k8s.io/docs/start/).

Deploy the cloned application with the following command

```bash
kubectl apply -f https://raw.githubusercontent.com/keyval-dev/microservices-demo/master/release/kubernetes-manifests.yaml
```

Start the Odigos UI with the following command.
 
```bash
./odigos ui
```

Visit the Odigos UI on `localhost:3000`. Select all the microservices in the namespace as shown below.

[![Odigos UI](https://res.cloudinary.com/practicaldev/image/fetch/s--4dwc5YIr--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/edq4qcopcm408wqgxrho.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--4dwc5YIr--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/edq4qcopcm408wqgxrho.png)

After selecting all the applications we will be asked for:

*   **Destination Name**: A name for our destination.
*   **Host Endpoint**: Use the host endpoint for SigNoz.

After doing this, we should now be prompted with the following UI.

[![Odigos UI](https://res.cloudinary.com/practicaldev/image/fetch/s--nPq37bOX--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cowug72v6xzxayw2oi3i.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--nPq37bOX--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cowug72v6xzxayw2oi3i.png)

Congratulations🎉! The Odigos setup has been successful. Now, it should automatically send traces, metrics, and logs to our observability backend.

Now, to see that in practice, we need to first set up the external IP for our `frontend-external` LoadBalancer service.

Running the `kubectl get services` command should show the following. Notice the `<pending>` IP for the `external-frontend` service.

[![Kubernetes Services](https://res.cloudinary.com/practicaldev/image/fetch/s--qebdJ9TO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7khodfb93czrl360a8mk.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--qebdJ9TO--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7khodfb93czrl360a8mk.png)

By default, it is not possible to access `LoadBalancer` service on the browser. For this, we need to set up `minikube tunnel`. For more info on this command visit [here](https://minikube.sigs.k8s.io/docs/commands/tunnel/).  

```bash
minikube tunnel
kubectl get services
```

Now, we can see the IP and port assigned to the `external-frontend` service.

[![Kubernetes Services Tunnel Image](https://res.cloudinary.com/practicaldev/image/fetch/s--k19sdDJ5--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lmstfmq5cu418n9hbh11.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--k19sdDJ5--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lmstfmq5cu418n9hbh11.png)

Visit `http://127.0.0.1:8081` on the browser to access the application.

[![Google e-commerce application](https://res.cloudinary.com/practicaldev/image/fetch/s--qIwqSoqV--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/osqhlyq3pahhwb195ps0.png)](https://res.cloudinary.com/practicaldev/image/fetch/s--qIwqSoqV--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/osqhlyq3pahhwb195ps0.png)

Run the SigNoz service and execute the following command.  

```bash
kubectl port-forward -n tracing svc/<service_name> <port_to_use>:<port>
```

Now, we should be able to view all the traces of our application as we did previously.

* * *

Wrap Up! 📝
-----------------------

So far, we've learned how to closely monitor our app's performance with traces and Hostmetrics, first manually using OpenTelemetry and SigNoz, and then automating the process with Odigos.

> Note: We are not limited to using SigNoz; we can use any observability database out there. Just for the sake of this article, I have chosen to use SigNoz.

It is always a good idea to have a monitoring system for any production-level application. This will help us debug errors if any arise in the future. Most importantly, it will help us identify any performance bottlenecks in our application, resulting in enhanced efficiency and user experience.

The source code for this tutorial is available here:

https://github.com/keyval-dev/blog/tree/main/instrumenting-node-app

Thank you for reading! 🎉