Testing Redesigned Data Pipelines with Split
Tags: code, experimentation, testinginproduction
Published: 2022-08-16 17:23:32

---
title: Testing Redesigned Data Pipelines with Split
published: true
date: 2022-07-13 23:40:21 UTC
tags: Code,experimentation,testinginproduction
canonical_url: https://www.split.io/blog/testing-pipelines-split/
---

A data pipeline is typically used to extract, transform, and load data from the source platform to the destination system. This process usually involves aggregating data from multiple sources, transforming, validating, and eventually loading it into the target system. Every now and then, a data pipeline requires a redesign. This could be due to constantly evolving customer requirements, system performance metrics (including cost, time, etc.), and more. 

Split’s existing data pipeline for [AWS S3 Inbound Integration](https://www.split.io/product/integrations/amazon-s3/) sends [feature flag event data](https://help.split.io/hc/en-us/articles/360020585772-Events) through S3 buckets. The S3 Inbound Spark job reads customers’ files from S3 buckets and updates Split’s Event table. Then, [status report files](https://help.split.io/hc/en-us/articles/360053674072-Amazon-S3#about-the-status-file) are generated and populated in the same S3 buckets. 

As the customer demand for existing S3 Inbound Integration increased, so did the cost of scaling S3 Inbound Spark Jobs. Split investigated methods to reduce costs and also researched ways to enhance customer experience by reporting additional status file information. Based on the investigation, Split designed a new pipeline using its feature flagging capabilities. Split’s feature flags helped to perform A/B testing functionality to validate cost and load feature tests in production. Thus enabling controlled production rollouts without impacting the customer experience.

## Existing S3 Inbound Integration Pipeline

![](https://www.split.io/wp-content/uploads/2022/07/Testing-Redesigned-Pipelines-01-1024x694.png?width=1024)
_Fig 1. Existing Split S3 Inbound Integration Pipeline_

As depicted above, data in Split’s existing S3 Inbound Integration pipeline is ingested from the customer buckets, sending file notifications to an _AWS SNS_ topic. Then, this _SNS_ topic invokes Split’s _Events_ service’s import endpoint to verify requests against existing integrations. As a result, Split asynchronously sends metadata of the file through the _AWS SQS_ queue

Split’s S3 Inbound Job runs on a cadence of every 10 seconds to determine if any new messages are present in the _AWS SQS_ queue. Upon receipt of new messages, the S3 Inbound Job processes files to extract and store data within Split’s _Events_ _Table_. Upon completion, S3 Inbound Job generates a consumed files status report and stores it in the configured customers’ _S3_ bucket. 

Split’s existing S3 Inbound Integration pipeline was built using a single _Spark Batch Job_. As a result, all of the ingested data files to the existing pipeline were consumed sequentially. Needless to say, this causes bottlenecks for the S3 Inbound Integration. In order to increase throughput, we either vertically scale these instances or horizontally add more of them. The total cost of the existing S3 Inbound Job increases by more than 50% in terms of Databricks Units (DBU). 

Additionally, the status report generated for the consumed files lacked valuable information that customers desired. For example, the total number of records in each file, error codes associated with them, and more. 

## New S3 Inbound Integration Pipeline

![](https://www.split.io/wp-content/uploads/2022/07/Testing-Redesigned-Pipelines-02-1024x554.png?width=1024)
_Fig 2. New Split S3 Inbound Integration Pipeline_

Similar to that of the existing pipeline, data is ingested in the new pipeline from the customers’ S3 buckets. This data then dispatches file notifications to an _AWS SNS_ topic. Then, the _Events_ _Service_ verifies import requests against existing integrations and asynchronously sends file metadata through an _AWS SQS_ queue

However, now the new pipeline’s _Events_ _Service_, as portrayed in _Fig. 2,_ forwards messages to a new _AWS SQS_ queue. A _Copy File Job_ now reads these messages and places them into an internal Split _S3_ bucket. Next, a _Copy Data Job_ uses [Databricks’ Autoloader](https://docs.databricks.com/spark/latest/structured-streaming/auto-loader.html) to efficiently load them into a Bronze table. Then, a _Data Validation Job_ determines whether the data is valid or not against the _MongoDB Integration_ _Table_. 

Regardless of the validation result, this job places the validated data into the _Silver_ _Table_. Afterward, a _Data Ingestion Job_ performs data replication between the _Silver Table_ and an _Events_ _Table_. A _Status Report Job_ reads the _Silver Table_ to generate the _Status Report_ into the customer’s S3 bucket. 

Split’s new S3 Inbound Integration pipeline is designed with Spark streaming architecture. This new design change resulted in the parallel consumption of the ingested data files and reduced the resource utilization costs. The new pipeline’s responsibilities were split (pun intended) using the[medallion architecture](https://databricks.com/glossary/medallion-architecture) to separate out tasks. These tasks include ingestion, schema validation, data validation, and error feedback so customers can self-correct for further error handling. 

## Testing the Pipelines

Just building this new Inbound Integration pipeline was not enough. It was not only vital that Split reduce costs, but also ensure its scalability and reliability. For the latter, Split used feature flagging to run A/B testing between the existing and the new pipelines as seen in _Fig. 3_.  

![](https://www.split.io/wp-content/uploads/2022/07/Testing-Redesigned-Pipelines-03-1024x413.png?width=1024)
_Fig 3. Split feature flag approach to A/B testing_

Typically, in a data pipeline, one would control data flow, say, through multiple Kafka consumer groups. As in _Fig. 4_, Kafka _Consumer 1_ dispatches data to the existing pipeline. Simultaneously, Kafka _Consumer 2_ pushes the same data to the new pipeline.

![](https://www.split.io/wp-content/uploads/2022/07/Testing-Redesigned-Pipelines-04-1024x413.png?width=1024)
_Fig 4. Example of Kafka’s approach to A/B testing_

Split used a feature flag (_Fig. 3_) approach to control data flow into two _AWS SQS_ queues. The first _SQS_ queue sends data to the existing pipeline while the second sends it to the new pipeline. This allows both pipelines to seamlessly co-exist in the production environment for A/B testing. 

Additionally, Split’s feature flag strategy allows to fine-tune the amount of data ingested through Split’s [Allocate Traffic](https://help.split.io/hc/en-us/articles/360020525572-Allocate-traffic) feature. It can also seamlessly onboard customers onto the new pipeline with Split’s [Set Targeting Rules](https://help.split.io/hc/en-us/articles/360020791591-Target-customers). Furthermore, Split’s [Default Treatment](https://help.split.io/hc/en-us/articles/360020527672-Set-the-default-treatment) assists in quicker rollbacks to the existing pipeline with a push of a single button. Whether it is on-demand or in the event of unforeseen production pipeline issues, Split has you covered.

## Scalability and Cost Comparison

The new S3 Inbound Integration pipeline has six new _Spark Jobs_ than that of the existing pipeline. These new jobs ensure that the new pipeline scales faster and better than the existing one. For example, picture a testing scenario where there are more than 50 GigaBytes of simulated data plus normal production data. This data can be ingested within 5-minute intervals, the new pipeline was able to process both data within 15 minutes. The pipeline is also horizontally auto-scaling into fewer instances. In comparison, the old pipeline would need 3 times as many instances and 10 hours to process the same data. 

As for total cost measured in Databricks Units (DBU), the existing and the new S3 Inbound Integration pipeline significantly vary. This is depicted in _Fig. 5_, where the blue and yellow bars illustrate the total cost of the existing and the new pipelines, respectively. For any month, the S3 Inbound Integration pipeline’s resource consumption is at least 80% lower than that of the existing pipeline. 

![](https://lh6.googleusercontent.com/RI2WFmqK7_KdVr-1G_4dUC-hGP9YrgRwoXoz1No7GtxBf60MUv6AClb_V8Cq7-UiWq4xmPppkcYH7cLEq-jWDm8qTdM2BPqkUpfE3B03HMbXuu1aO2DSMt1RiOz_DzEnWu5BrfD5sBIrcremCw)
_Fig 5. The total cost of existing and new Split S3 Inbound Integration pipeline in Databricks Units (DBU)_

## Conclusion 

Split’s data pipeline for S3 Inbound Integration sends feature flag event data through _AWS_ S3 buckets, generating status report files. The scaling costs of this pipeline are directly proportional to the increase in customer demands. 

In conjunction with its feature flag strategy, Split designed its new S3 Inbound Integration pipeline to save on total operational costs. It can also enhance status file reports, perform scalability testing, and perform controlled rollouts in production environments. Additionally, Split’s feature flags allow for a faster and seamless rollback to existing pipelines, if needed. New S3 Inbound Integration pipeline scales better, requiring fewer and smaller instances, thereby saving 80% of the total operational cost.

With controlled rollouts, Split has successfully migrated a few customers to the new S3 Inbound Integration. It’s been active for more than one quarter in production environments with zero customer impact, providing enhanced status reports in near real-time.