Reference Data Stack for Data-Driven Startups
Published: 2022-03-03 21:15:20

At [Monosi](https://monosi.dev), we recently set up our internal data stack to fully enable data collection and metrics tracking. I figured a post like this may be useful for earlier stage companies that are in the process of setting up their own pipelines as they start tracking metrics and becoming more data driven.

This post is a high level overview of the technologies we are using and why we chose them. If you’re interested in a more technical overview or tutorial with actual Terraform files to spin up a replica stack, comment below and I can write up a more detailed post.

## Technologies Used

Monosi is an [open source data observability platform](https://monosi.dev), therefore our stack is composed mostly of open source technologies. Our current stack looks like:

![Monosi Stack Overview](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gzv99rbkbh5rugzalvq8.png)
 
Technologies include [Snowplow](https://snowplowanalytics.com/), [Airbyte](https://airbyte.com/), [PostgreSQL](https://www.postgresql.org/), [dbt](https://www.getdbt.com/), [Snowflake](https://www.snowflake.com/), [Metabase](https://www.metabase.com/), and [Monosi](https://monosi.dev) itself. All of it is hosted on AWS running in a VPC. 

## Data Collection / Extraction / Ingestion

There are two ways we collect data and ingest it into our pipeline.

First, we have data coming in from various third party tools such as Google Analytics, MailChimp, Github, Slack, etc. For extraction we use [Airbyte](https://airbyte.com/). 

![Extraction with Airbyte](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c8vs43exz1dfwdirr82z.png)

We chose Airbyte because of their open source self-hosted offering and their connector flexibility. There are other open source tools in the market like [StitchData](https://www.stitchdata.com/) or [Meltano](https://meltano.com/) but have limitations as discussed in [this post](https://airbyte.com/blog/airbyte-vs-singer-why-airbyte-is-not-built-on-top-of-singer) by the Airbyte team. If you don’t want to self host extraction, consider [Fivetran](https://fivetran.com/).

![Snowplow event collection](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jig570ngvqeif1diyvu2.png)

We also have telemetry set up on our Monosi product which is collected through [Snowplow](https://snowplowanalytics.com/),. As with Airbyte, we chose Snowplow because of its open source offering and because of their scalable event ingestion framework. There are other open source options to consider including [Jitsu](https://jitsu.com/) and [RudderStack](https://www.rudderstack.com/) or closed source options like [Segment](https://segment.com/). Since we started building our product with just a CLI offering, we didn’t need a full CDP solution so we chose Snowplow.

## Data Storage

To store our collected data, we use a Snowflake data warehouse for all final data collection and a PostgreSQL database as an event destination. 

![Airbyte to Snowflake](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ckf1adpz9f3z747x1ayt.png)

All data from third party tools extracted with Airbyte is piped into Snowflake. We chose Snowflake because it’s easy to get started with, has a simple UX, and we use it internally for testing. Snowflake offers a free trial to quickly get started with and provides an intuitive UI. Since Monosi supports a Snowflake integration, it also makes development and testing easier.

![Snowplow to PostgreSQL](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qheay3n915rq2k1spz20.png)

All events collected with Snowplow are piped into a PostgreSQL database. This is separate because we wanted an individual database to act as storage for product event collection and it was quick to setup with Snowplow.

![Postgres to Snowflake and dbt](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/40xaz99xb42cik8x6n7v.png)

The setup also enables us to sync all of our event data to the data warehouse [with Airbyte](https://airbyte.com/recipes/postgresql-database-to-snowflake) and perform necessary transformations on the data with [dbt cloud](https://www.getdbt.com/). 

## Data Visualization / Analysis

![Snowflake to Metabase](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fc7gfc20atq14c1v119f.png)

To analyze the data stored in Snowflake and Postgres, we use [Metabase](https://metabase.com). We chose Metabase because of it’s open source offering and easy to use interface. Other open source tools like [Lightdash](https://www.lightdash.com/) and [Superset](https://superset.apache.org/) exist which we may add to the stack as our data team grows. 

## Data Reliability

![Snowflake and Monosi](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sp7o5rl1oe72insiabk2.png)

As a data driven company, we need to make sure that our data is flowing reliably from extraction to visualization. It is important to have our metrics up to date and accurately defined at all times. Hence, when a certain part of the data stack stops working or changes (e.g. event ingestion, data extraction, schema updates) we need to be aware of what happened and why. 

This is where our own tool, [Monosi](https://monosi.dev), comes in. To ensure data reliability within our stack, we’ve deployed our own internal instance of Monosi. With Monosi, we get alerted if anomalies occur in our data and can perform root cause analysis.

## Other Tooling

There are other tools that we will have to adopt in the future but haven’t yet due to lack of necessity. Specifically, one category that is popular in modern data stacks is Reverse ETL ([Hightouch](https://hightouch.io/), [Census](https://www.getcensus.com/), or [Grouparoo](https://www.grouparoo.com/)). We currently don’t have a usecase for piping data back into 3rd party tools but it will definitely come up in the future.

## Wrapping up

Thanks for reading the overview of what our startup data stack looks like! If you have any questions about the setup or would like to see a more in depth post with actual Terraform files that handle both service creation and networking, leave a comment below!