Running The Feast Feature Store With Dragonfly
Tags: redis, database, architecture
Published: 2023-08-08 16:09:00

## A brief introduction to Feast

[Feast](https://github.com/feast-dev/feast) stands as an exceptional open-source feature store, revolutionizing the efficient management and uninterrupted serving of machine learning (ML) features for real-time applications.
At its core, Feast offers a sophisticated interface for storing, discovering, and accessing features—the individual measurable properties or characteristics of data essential for ML modeling.
Operating on a distributed architecture, Feast harmoniously integrates several pivotal components, including the Feast Registry, Stream Processor, Batch Materialization Engine, and Stores.

Facilitating both offline and online storage paradigms, Feast enables comprehensive time-series analysis by harnessing historical feature values stored in data sources.
For real-time responsiveness, Feast leverages online stores, ensuring features are served at remarkably low latencies.
With a simple `feast materialize` command, feature values flow from data sources to the online store, empowering your ML applications with unparalleled agility and performance.

One of the online stores supported by Feast is Redis. In this blog post, we will explore how to use Dragonfly as a drop-in replacement for Redis as an online store for Feast.

## In-memory online stores

One of the most critical factors determining the success of a feature store is its ability to serve features at low latency.
High latency can severely impact model performance and the user experience, leading to delayed predictions and suboptimal outcomes.

In-memory data stores offer a substantial advantage in terms of low-latency feature serving.
By storing data directly in memory, they eliminate the need for disk I/O operations, which are often a bottleneck in retrieving data from traditional storage systems.
With data residing in RAM, in-memory data stores rapidly retrieve and serve features, resulting in near-instantaneous response times.

The significance of low-latency feature serving provided by in-memory data stores cannot be overstated.
It empowers machine learning models to access necessary features swiftly, allowing for more efficient and timely predictions or inferences.
Whether it's delivering personalized recommendations, making instant decisions, or powering real-time analytics, in-memory data stores ensure quick access to features, contributing to improved model performance and elevated user satisfaction.

## Meet Dragonfly

Dragonfly, a cutting-edge in-memory data store, distinguishes itself with novel algorithms and data structures integrated into a multi-threaded, share-nothing architecture.

Its hardware efficiency allows Dragonfly to thrive across diverse machine configurations—running a single node on an 8GB machine or vertically scaling to colossal 1TB machines with 64 cores.
This versatility not only delivers substantial infrastructure cost savings but also streamlines architectural complexity.

Another true marvel lies in Dragonfly's exceptional API compatibility, effortlessly serving as a drop-in replacement for Redis not only in Feast but also in many other scenarios.
As of version 1.6.2, Dragonfly boasts an impressive implementation of over 200 Redis commands, covering the vast majority of use cases, including support for the Hash data structure relied upon by Feast for storing feature values.

With Dragonfly's impressive blend of compatibility, efficiency, and comprehensive features, we embark on an exploration of its immense benefits to Feast as an online feature store, unlocking new dimensions of performance and scalability.

## Running Feast with Dragonfly: a hands-on guide

In this section, we will go through some hands-on steps to demonstrate how to run Feast with Dragonfly seamlessly.
This tutorial is highly inspired by the [official Feast documentation](https://docs.feast.dev/getting-started/quickstart), with the emphasis on integrating Dragonfly as an online store for Feast.

### 1. Prerequisites

Make sure `Python` and `pip` are installed on your platform. Then, we can install the Feast SDK and CLI:

```bash
pip install feast
```

In order to use Dragonfly as the online store, we will need to install the `redis` extra:

```bash
pip install 'feast[redis]'
```

### 2. Create a feature repository

To start, we can use the `feast` CLI to bootstrap a new feature repository.

```bash
feast init feast_dragonfly
cd feast_dragonfly/feature_repo
```

A [Feast feature repository](https://docs.feast.dev/reference/feature-repository) consists of

- A collection of Python files containing feature declarations.
- A `feature_store.yaml` file containing infrastructural configuration.
- A `.feastignore` file containing paths in the feature repository to ignore.

We are interested in the `feature_store.yaml` file since it contains the configuration of infrastructure, such as the online store, for Feast.
Update the `feature_store.yaml` file with the following content:

```yaml
project: feast_dragonfly
registry: data/registry.db
provider: local
online_store:
  type: redis
  connection_string: "localhost:6379"
```

### 3. Start Dragonfly

There are several options available to [get Dragonfly up and running quickly](https://www.dragonflydb.io/docs/getting-started), we will be using Docker for this tutorial:

```bash
docker run --network=host --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly
```

Integrating Dragonfly as an online store in Feast is astonishingly straightforward without a hitch.
Instead of running a Redis instance locally, we effortlessly initiated Dragonfly.
In the provided `feature_store.yaml` configuration file, we directed Feast to utilize the online store, pointing to `localhost:6379`, which aligns with Dragonfly's default running configuration.

The beauty of this transition lies in the ease of implementation—it doesn't require a single change in Feast's core configuration.
We merely directed Feast to use Dragonfly, and that's all it took! From this point onward, the tutorial continues as a typical Feast guide, empowering you to explore its vast capabilities.

### 4. Register feature definitions and deploy the feature store

Within the same `feast_dragonfly/feature_repo` directory, use the following command:

```bash
feast apply
```

The `apply` command scans Python files in the current directory (`example_repo.py` in this case) for feature view and entity definitions, registers the objects, and deploys the infrastructure.
We should see the following output upon success:

```text
....
Created entity driver
Created feature view driver_hourly_stats_fresh
Created feature view driver_hourly_stats
Created on demand feature view transformed_conv_rate
Created on demand feature view transformed_conv_rate_fresh
Created feature service driver_activity_v1
Created feature service driver_activity_v3
Created feature service driver_activity_v2
```

### 5. Generate training data

Save the code below as `generate_training_data.py`:

```python
from datetime import datetime
import pandas as pd

from feast import FeatureStore

# Note: see https://docs.feast.dev/getting-started/concepts/feature-retrieval for 
# more details on how to retrieve for all entities in the offline store instead
entity_df = pd.DataFrame.from_dict(
    {
        # entity's join key -> entity values
        "driver_id": [1001, 1002, 1003],
        # "event_timestamp" (reserved key) -> timestamps
        "event_timestamp": [
            datetime(2021, 4, 12, 10, 59, 42),
            datetime(2021, 4, 12, 8, 12, 10),
            datetime(2021, 4, 12, 16, 40, 26),
        ],
        # (optional) label name -> label values. Feast does not process these
        "label_driver_reported_satisfaction": [1, 5, 3],
        # values we're using for an on-demand transformation
        "val_to_add": [1, 2, 3],
        "val_to_add_2": [10, 20, 30],
    }
)

store = FeatureStore(repo_path=".")

training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
        "transformed_conv_rate:conv_rate_plus_val1",
        "transformed_conv_rate:conv_rate_plus_val2",
    ],
).to_df()

print("----- Feature schema -----\n")
print(training_df.info())

print()
print("----- Example features -----\n")
print(training_df.head())
```


To generate training data, run:

```bash
python generate_training_data.py
```

### 6. Ingest batch features into the online store

Next, we serialize the latest values of features since the beginning of time to prepare for serving:

```bash
feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")
```

When feature data is stored using Dragonfly as the online store, Feast utilizes the Hash data structure to store a two-level map.
The first level of the map contains the Feast project name and entity key. The entity key is composed of entity names and values.
The second level key (i.e., `field` of the Hash) contains the feature table name and the feature name, and the Hash `value` contains the feature value.
Feel free to connect to the local Dragonfly instance using `redis-cli` if you want to learn more about how Feast stores features in Dragonfly.

### 7. Fetching feature vectors for inference

At inference time, we need to quickly read the latest feature values for different drivers (which otherwise might have existed only in batch sources) from the online feature store using `get_online_features()`.
Save the script below as `fetch_feature_vectors.py`:

```python
from pprint import pprint
from feast import FeatureStore

store = FeatureStore(repo_path=".")

feature_vector = store.get_online_features(
    features=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
    entity_rows=[
        # {join_key: entity_value}
        {"driver_id": 1004},
        {"driver_id": 1005},
    ],
).to_dict()

pprint(feature_vector)
```

To fetch feature vectors, run:

```bash
python fetch_feature_vectors.py
```

We should see output similar to the following:

```python
{
    'acc_rate': [0.1056235060095787, 0.7656288146972656],
    'avg_daily_trips': [521, 45],
    'conv_rate': [0.24400927126407623, 0.48361605405807495],
    'driver_id': [1004, 1005]
}
```

## Conclusion

In this blog post, we witnessed the effortless integration of Dragonfly as an online store for Feast.
By simply directing Feast to use Dragonfly, we embraced a modern in-memory data store capable of swift feature serving.

Our primary focus in the tutorial above was on the Redis command compatibility of Dragonfly.
However, Dragonfly's capabilities go beyond mere protocol and command compatibility.
Whether deployed on small instances or powering large-scale machines, Dragonfly's versatility shines, offering unmatched hardware efficiency that reduces infrastructure costs and complexity.
To delve deeper into Dragonfly's hardware efficiency, please explore [this dedicated blog post](scaling-performance-redis-vs-dragonfly) on scaling and performance comparisons.

Our [command reference](https://www.dragonflydb.io/docs/category/command-reference) and [documentation](https://www.dragonflydb.io/docs) provide an extensive resource to gain a comprehensive understanding of Dragonfly's full spectrum of capabilities.

At our core, we are dedicated to fostering a thriving open-source community.
As passionate supporters of cutting-edge technology, our team is actively working with more open-source projects to ensure streamlined integration with Dragonfly.
By extending our reach and engaging in collaborative efforts, we aim to empower diverse ecosystems with the unmatched efficiency of Dragonfly.
Stay tuned for more Dragonfly integrations by subscribing to our newsletter and joining our social media channels!