Revisiting PromCon 2018 Panel On Prometheus Long-Term Storage
Tags: prometheus, monitoring, storage, cloudnative
Published: 2018-11-20 11:06:26

---
title: Revisiting PromCon 2018 Panel On Prometheus Long-Term Storage
published: true
description: Based on a 2018 PromCon panel discussion, learn about options to retain metrics in Prometheus for longer time periods
tags: prometheus,monitoring,storage,cloud native
cover_image: https://thepracticaldev.s3.amazonaws.com/i/md06ijaw5lvneol6e5bg.jpg
canonical_url: https://dev.to/mhausenblas/2018-prometheus-long-term-storage
---

While Prometheus itself does not offer a clustered storage solution to store data across multiple machines out of the box, there are a number of so called Long-Term Storage (LTS) options available. In this article we do a high-level review of some the LTS options that were the topic of a [PromCon 2018 panel](https://youtu.be/VvJx0WTiGcA?t=23774) I had the pleasure to witness in person, back in August of this year: 

{% youtube 3pTG_N8yGSU %}

Note that below I won't provide a recommendation _which_ LTS solution you should pick, since it very much depends on your specific requirements and preferences.

Prometheus [stores](https://prometheus.io/docs/prometheus/latest/storage/) data on local storage, which limits the data you can query or otherwise process to the most recent days or weeks, depending on how much space you have available, locally. If you, however, have the need to retain data for longer time periods, for example for long-term capacity planning, analyzing usage trends, or for regulatory reasons in a certain vertical—think: financial domain or health care—then you likely benefit from a LTS solution. Let's have a look at the offerings discussed in the PromCon panel (in alphabetical order):

## Cortex

[Cortex](https://github.com/weaveworks/cortex) provides horizontally scalable, multi-tenant, long term storage for Prometheus metrics when used as a remote write destination, and a horizontally scalable, Prometheus-compatible query API.

See also:

* [Cortex: open-source, horizontally-scalable, distributed Prometheus](https://www.youtube.com/watch?v=Xi4jq2IUbLs), 06/2017
* Project Frankenstein: Multitenant, Scale-Out Prometheus: [video](https://youtu.be/3Tb4Wc0kfCM) | [slides](http://www.slideshare.net/weaveworks/project-frankenstein-a-multitenant-horizontally-scalable-prometheus-as-a-service), 09/2016

## InfluxDB

[InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) is a time series database designed to handle high write and query loads and is meant to be used as a backing store for any use case involving large amounts of timestamped data, including DevOps monitoring, application metrics, IoT sensor data, and real-time analytics. InfluxDB supports the [Prometheus remote read and write API](https://docs.influxdata.com/influxdb/v1.6/supported_protocols/prometheus).

See also:

* [InfluxDB Now Supports Prometheus](https://www.influxdata.com/blog/influxdb-now-supports-prometheus-remote-read-write-natively/), 09/2017
* [InfluxDB + Prometheus](https://speakerdeck.com/pauldix/influxdb-plus-prometheus) presentation, 08/2017 

## M3

[M3](http://m3db.github.io/m3/) is a metrics platform, originally developed at Uber, that is built on M3DB, a distributed time     series database and it provides an [integration of M3DB with Prometheus](http://m3db.github.io/m3/integrations/prometheus/).

See also:

* [M3: Uber’s Open Source, Large-scale Metrics Platform for Prometheus](https://eng.uber.com/m3/), 08/2018
* [Hacker News thread](https://news.ycombinator.com/item?id=17707842)

## Thanos

[Thanos](https://github.com/improbable-eng/thanos) is a set of components that can be composed into a highly available metric system with unlimited storage capacity. It can be added seamlessly on top of existing Prometheus deployments and leverages the Prometheus 2.0 storage format to cost-efficiently store historical metric data in any object storage while retaining fast query latencies. Additionally, it provides a global query view across all Prometheus installations and can merge data from Prometheus HA pairs on the fly.

See also:

* [The new Prometheus storage engine](https://www.youtube.com/watch?v=6P-RmOWWA4U), 06/2018
* [Introducing Thanos: Prometheus at scale](https://improbable.io/games/blog/thanos-prometheus-at-scale), 05/2018

---- 

So to sum up: while Prometheus itself does not support long-term retention of the time series data of interest, there are a number of solutions you can choose from to keep the metrics around for as long as needed. Hope this quick review gives you an idea of some of the available options and can serve as the basis for your own research, should you find yourself in the a situation to have to select one. I wish you successful monitoring and please do share your findings and/or hands-on experiences with above discussed or other LTS solutions not covered here.

> Cover image kudos to [jesse orrico via Unsplash](https://unsplash.com/photos/h6xNSDlgciU).