Launching support for ClickHouse as storage backend for SigNoz
Tags: devops, database, monitoring, observability
Published: 2021-07-23 13:04:54

## What is SigNoz?
SigNoz is an open-source alternative to DataDog, New Relic etc. It is a full-stack application monitoring and observability platform, which can be used to track both metrics and traces. Link to our GitHub repo 👇

{% github signoz/signoz %}

## Launching support for ClickHouse
In our latest release v0.3.2 , we launched support for ClickHouse as storage backend for SigNoz. ClickHouse is a column-oriented database management system (DBMS) for online analytical processing of queries (OLAP). Now when you install SigNoz, you will have the option to choose between ClickHouse or Kafka + Druid as a database system.
![You can install SigNoz through with commands at your terminal. When you install SigNoz, you will get the option to choose between ClickHouse or Kafka+Druid as a storage backend](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4b1u2ay7hmo4mrmz39tb.png)<figcaption>Users can choose between ClickHouse or Kafka + Druid for their storage system of choice while installing SigNoz
</figcaption>

In this article, let's dig deeper into why we decided to introduce support for Clickhouse as a database storage system and how our users can benefit from it.

## Community demands for ClickHouse
As an open-source software, we build what our community wants. We listened to [demands](https://github.com/SigNoz/signoz/issues/22) from our community for supporting ClickHouse as a storage backend.

![Picture of the github issue a user raised for supporting ClickHouse](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2y29wbpmspj8kynkjxvv.png)<figcaption>Community request for supporting Clickhouse as a storage backend for SigNoz</figcaption>

As a storage backend, ClickHouse is less resource-intensive than using Kafka + Druid. It makes getting started with SigNoz much quicker, and users can now try out SigNoz on their local machine easily as the entire setup takes __less than 1.5GB of memory space.__

## Why Clickhouse for SigNoz?
### Improvements in installation time
One of the key focus area of our engineering team is to make trying out SigNoz as simple as possible. Extending support for ClickHouse as a database is part of one such initiative.

As we are self-hosted and open source, you can try SigNoz out in a matter of a few minutes to understand how it can help you with performance monitoring.

You can get started with SigNoz with just three commands at your terminal:

```
$ git clone https://github.com/SigNoz/signoz.git
$ cd signoz/deploy/
$ ./install.sh
```
You can then proceed with either ClickHouse or Kafka+Druid installation set up. On a macOS 2017 model with 8 GB RAM, following were the installation times in seconds for the two set ups:

- ClickHouse set up: 93.99 seconds
- Kafka + Druid set up: 269.25 seconds

So in just over a minute, you will be able to access our dashboard with the Clickhouse set up on your local host.

![SigNoz dashboard](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/01sspm4ldf5oiqtidyjj.png)<figcaption>Track metrics & use traces to monitor app performance with SigNoz</figcaption>

### Improvements in memory usage
Let's analyse the performance of ClickHouse and Kafka+Druid set ups with `docker stats` command. The `docker stats` command displays a live data stream with CPU, memory usage, memory limit, block I/O, and network IO metrics for all the running containers.

![Memory usage with ClickHouse installation with docker stats command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6hacjb1wgk5py3503lpv.png)<figcaption>Memory usage stats snapshot with ClickHouse installation</figcaption>

Total memory usage with ClickHouse set up: 340.391 MiB

![Memory usage with kafka+druid installation with docker stats command](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3d9ogc702du4gnxc4dzu.png)<figcaption>Memory usage stats snapshot with kafka + Druid installation</figcaption>

Total memory usage with Kafka + Druid set up: 2,896.82 MiB

As we can see, the ClickHouse set up uses about __8.5x less memory__ than the Kafka + Druid set up for getting started with SigNoz. It makes trying out SigNoz much easier for developers even on their local machine.

## SigNoz architecture with ClickHouse
![SigNoz architecture with ClickHouse](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/s1lmyo7noo90bbi8331p.png)<figcaption>SigNoz Architecture with ClickHouse as storage backend</figcaption>

With ClickHouse as the storage backend, OpenTelemetry collector directly writes to ClickHouse. The query service makes queries to ClickHouse to fetch relevant data points and display it on the frontend UI.

We will also be soon bringing support for long term storage from ClickHouse to S3.

## Upcoming features in the ClickHouse set up
We will soon be enabling custom metrics in SigNoz with the ClickHouse  storage backend. Metrics represent the health of your system over time and are a crucial component of observability. SigNoz uses OpenTelemetry for instrumentation and you can measure metrics like p99, p50 latency, etc.

And with custom metrics you will be able to define more specific metrics to gauge the health of your system.

As an open-source monitoring tool, we want to provide our users flexibility in setting up monitoring for their services. Monitoring data is often critical and private to a company, and as such, users should have flexibility in choosing a database that their teams feel most comfortable with. Providing support for different storage backends is an effort in this direction.

SigNoz is also self-hosted, so you don't need to send your critical data to SaaS vendors outside your cloud. Visit our GitHub repo, and try out SigNoz in a matter of minutes.

We are constantly looking out for community feedback and will be pleased to hear from you.
{% github signoz/signoz %}