Quick Start Guide to Amundsen Demo 🚀
Tags: beginners, tutorial, datascience, opensource
Published: 2023-05-09 15:56:55

## {% details Outlines %}
- [Introduction 🌟](#introduction)
  - [How does it work? 🛠️](#how-does-it-work)
    - [Discover trusted data 📊](#discover-trusted-data)
    - [See automated and curated metadata 📚](#see-automated-and-curated-metadata)
    - [Share context with coworkers 👥](#share-context-with-coworkers)
    - [Learn from others 👩‍🎓](#learn-from-others)
- [Installation ⚙️](#installation)
- [Miscellaneous 🧩](#miscellaneous)
- [The End 🏁](#the-end)
{% enddetails %}

## Introduction 🌟

[Amundsen](https://www.amundsen.io/) is **an advanced data discovery and metadata engine** designed to **boost the productivity** of **data analysts, data scientists, and engineers** when interacting with data.

![data discovery fact](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ganh0rulxa7putowyxi.png)

It achieves this by indexing data resources (tables, dashboards, streams, etc.) and powering a page-rank style search based on usage patterns (e.g., highly queried tables show up earlier than less queried tables).

[![landing page of amundsen in demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7hlik859f2u4a2jt2ckx.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7hlik859f2u4a2jt2ckx.png)

> In simple terms, it's like Google search for data. The project is named after Norwegian explorer Roald Amundsen, the first person to discover the South Pole.

[![architecture of amundsen](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1pgozhp7l45qbf3omr8l.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1pgozhp7l45qbf3omr8l.png)

Amundsen is hosted by the LF AI & Data Foundation. It includes **three microservices, one data ingestion library, and one common library**:

| **Amundsen Libraries** | **Description** |
| --- | --- |
| **amundsenfrontendlibrary** | A Flask application with a React frontend. |
| **amundsensearchlibrary** | A search service that leverages Elasticsearch for search capabilities. |
| **amundsenmetadatalibrary** | A metadata service that leverages Neo4j or Apache Atlas as the persistent layer to provide various metadata. |
| **amundsendatabuilder** | A data ingestion library for building metadata graph and search index. |
| **amundsencommon** | A common library that holds shared codes among Amundsen's microservices. |
| **amundsengremlin** | A library that holds code used for converting model objects into vertices and edges in gremlin, used for loading data into an AWS Neptune backend. |
| **amundsenrds** | Contains ORM models to support relational database as metadata backend store in Amundsen. |

Check out their [GitHub](https://github.com/amundsen-io/amundsen) for more information.

### How does it work? 🛠️

#### Discover trusted data 📊

![search result](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5zo1x9qjijlf1oyq1q1l.png)

Search for data within your organization with a simple text search. A PageRank-inspired search algorithm recommends results based on names, descriptions, tags, and querying/viewing activity on the table/dashboard.

#### See automated and curated metadata 📚

![metadata example](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1jc1l6lpv5q5r5sd4vsu.png)

Build trust in data using automated and curated metadata — descriptions of tables and columns, other frequent users, when the table was last updated, statistics, a preview of the data if permitted, etc. Easily triage by linking the ETL job and code that generated the data.

#### Share context with coworkers 👥

![data resources](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jlrl4drggg0eiynx5vmi.png)

Update tables and columns with descriptions, reduce unnecessary back and forth about which table to use and what a column contains.

#### Learn from others 👩‍🎓

![user details](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/mtwnptbg7ijs9385xslw.png)

See what data fellow co-workers frequently use, own, or have bookmarked. Learn what the most common queries for a table look like by seeing dashboards built on a given table.

Check out [Amundsen's website](https://www.amundsen.io/) and their [documentation](https://www.amundsen.io/amundsen/) for more information.

## Installation ⚙️

Ensure you have at least 3GB of disk space available to Docker. You'll need to install `docker` and `docker-compose`. 

Follow these guides to install Docker based on your operating system:
- [Linux](https://docs.docker.com/engine/install/)
- [macOS](https://docs.docker.com/desktop/install/mac-install/)
- [Windows](https://docs.docker.com/desktop/install/windows-install/)

And here's a guide on how to install `docker-compose` for [all systems](https://docs.docker.com/compose/install/).

You can check your current Docker version with this command:
```
docker -v
```

And to check your `docker-compose` version, use this command:
```
docker-compose -v
```

We'll be using WSL2 for this guide, and we'll start by cloning this [repo](https://github.com/amundsen-io/amundsen) and its submodules:

```
git clone --recursive https://github.com/amundsen-io/amundsen.git
```

![git clone](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cphhb0cftpqd8uwiitv8.png)

Next, enter the cloned directory:

```
cd amundsen
```

If this is your first time, make sure you've allocated the necessary memory. The minimum needed for all the containers to run with the loaded sample data is 3GB.

![amundsen: docker configuration setting](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5pd4uh6q4j6a7lzf5oew.png)

If you're using WSL2, you can check your allocation through `.wslconfig`. Follow this [guide](https://learn.microsoft.com/en-us/windows/wsl/wsl-config#configuration-setting-for-wslconfig) to set your `.wslconfig` for WSL2. 

As an example, here's the `.wslconfig` I use:
```
[wsl2]
memory=6GB
processors=2
swap=3GB
```
If you've made changes to the configuration, restart your PC so they can take effect. If no changes were necessary, proceed to the next step.

For this demo, we'll be using Neo4j Backend. Run the following command:

```
docker-compose -f docker-amundsen.yml up
```

![docker compose up](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eg8dul36siunnxbh5zhb.png)

**In a separate terminal window**, change your directory to `databuilder` to ingest the provided sample data into Neo4j:

```
cd databuilder
```

Install the dependencies in a virtual environment. For this, we'll be using `pyenv`, a tool for managing multiple Python versions, and its plugin `pyenv-virtualenv` for managing multiple virtual environments. If you don't have these installed, check out these guides on how to install [pyenv](https://realpython.com/intro-to-pyenv/#installing-pyenv) and [creating a virtual environment with pyenv](https://realpython.com/intro-to-pyenv/#creating-virtual-environments). 

Check your pyenv versions:

```
pyenv versions
```

Activate the environment in the current directory. In my case, my virtual environment is called `amundsen_demo`:

```
pyenv local amundsen_demo
```

![pyenv local](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jig2vydtr2rz6qd81xi9.png)

Finally, upgrade the version of pip, the package installer for Python:

```
pip3 install --upgrade pip
```

Next, install the Python packages listed in the `requirements.txt` file. This file contains a list of dependencies required by a Python project:

```
pip3 install -r requirements.txt
```

Then, install the Amundsen Data builder package using pip:

```
python3 setup.py install
```

![python setup](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/4aez24k1bddqngc95u3x.png)

We'll then load data into Neo4j and Elasticsearch databases without using an Airflow DAG (Directed Acyclic Graph) with this script:

```
python3 example/scripts/sample_data_loader.py
```

The script consists of several jobs:

| **Amundsen Jobs** | **Description** |
| --- | --- |
| *run_csv_job* | Reads table data from a CSV file, writes the data to another local directory as a CSV file, and then publishes the data to Neo4j, a graph database management system. |
| *run_table_column_job* | Similar to run_csv_job, but processes a CSV file containing column data instead. |
| *create_last_updated_job* | Creates a job that gets the current time, converts it into a predefined data model, and publishes it to Neo4j. |
| *create_es_publisher_sample_job* | Creates a job that extracts data from Neo4j and publishes it to Elasticsearch, a search and analytics engine. |

The script imports necessary modules, sets up configuration, and uses various extractors, loaders, and publishers from the Amundsen Databuilder library to perform the tasks mentioned above.

![sample data loader](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pla56mxbttztgi0fl2sa.png)

Now you can view the UI at [http://localhost:5000](http://localhost:5000) and try searching for `test`. You should get some results.

[![test view UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpuynxx8rxwa6kwd8pnb.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/bpuynxx8rxwa6kwd8pnb.png)

You can also perform an **exact-match** search for a table entity. For instance, search for `test_table1` in the table field of the filter, and it'll return the matching records.

To verify the dummy data has been ingested into Neo4j, visit [http://localhost:7474/browser/](http://localhost:7474/browser/)

[![neo4j browser auth](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9q5nwsbf5kmui0nt7go9.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9q5nwsbf5kmui0nt7go9.png)

and run this in the query box:

```
MATCH (n:Table) RETURN n LIMIT 25
```

[![query results](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a03df7mdfdza0ydni7su.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a03df7mdfdza0ydni7su.png)

You can verify the data has been loaded into the `metadataservice` by visiting:

- [test_table1](http://localhost:5000/table_detail/gold/hive/test_schema/test_table1)

[![test table1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jxig9jgkawunchy4ozzh.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jxig9jgkawunchy4ozzh.png)

- [test_table2](http://localhost:5000/table_detail/gold/dynamo/test_schema/test_table2)

[![test table2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/72466kqjv76hn9znlvml.png)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/72466kqjv76hn9znlvml.png)

Finally, don't forget to stop your running multicontainer app after you've finished using it:

```
docker-compose -f docker-amundsen.yml down
```

![docker compose down](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nprt1c3i5hqwigeeb4z0.png)

## Miscellaneous 🧩
- [Installation Guide](https://www.amundsen.io/amundsen/installation/)
- [My Quick Start Amundsen Demo 2023 [No Sound No Commentary]](https://youtu.be/G-5woTwTtAs)
{% embed https://youtu.be/G-5woTwTtAs %}
- [Amundsen: A Data Discovery Platform From Lyft](https://www.youtube.com/watch?v=EOCYw0yf63k)
{% embed https://www.youtube.com/watch?v=EOCYw0yf63k %}
- [Slides: Amundsen: A Data Discovery Platform From Lyft](https://www.datacouncil.ai/hubfs/DataEngConf/Data%20Council/Slides%20SF%2019/Amundsen%20-%20A%20Data%20Discovery%20Platform%20from%20Lyft.pdf?hsLang=en)

## The End 🏁
That concludes our quick start guide to setting up and running a demo of Amundsen. I hope you found this post helpful and informative. If you have any questions or if there's something you'd like to know more about, feel free to drop a comment below or reach out to me directly. 

Remember, there's no limit to what you can achieve with the right tools and a little bit of know-how. Keep exploring, keep learning, and as always, keep pushing the boundaries of what's possible with data.

If you want to stay updated with my latest posts and activities, or if you just want to connect, follow me on Beacons:

{% cta https://beacons.ai/ranggakd %} Take a look 👀 {% endcta %}

Happy coding, everyone! 🚀