Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models (2025)

Abstract: To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to welcome community contribution. Additionally, we provide a concise, two-minute video demonstration of our system, illustrating its capabilities and implementation.

## introduction the success of large language models (llms) is widely attributed to the scale of the data @xcite , otherwise known as the 'scaling law' @xcite where llm performance directly correlates with data size. consequently, there has been an exponential growth in the need for massive data to further fuel llm development. such increase in demand leads to more complex data processing pipelines, as even simple operations need to be optimized for data processing at enormous scales. to handle such data workloads efficiently and effectively, distributed systems and techniques such as spark @xcite and slurm @xcite have become crucial. unfortunately, the existing open-source data processing tools based on distributed systems @xcite @xcite either lack easy customization support or a wide variety of operations such † corresponding author 1 https://github.com/upstageai/ dataverse 2 https://www.youtube.com/watch?@xmath0, decontamination @xcite , bias mitigation @xcite , and toxicity reduction @xcite . this forces researchers to undergo a steep learning curve or cobble together tools from various sources, hindering efficiency and user experience. in response to these limitations, we present dataverse, a unified open-source etl (extract, transform, load) pipeline with a user-friendly design that enables easy customization. inspired by the transformers library @xcite , dataverse is built with a design principle of minimizing complex inheritance structures. such design choice allows for easy addition of custom data operations. specifically, the etl pipeline in dataverse is defined by block-based interface, which enables intuitive customization of etl pipelines by simply adding, removing, or reshuffling blocks. further, dataverse natively supports a wide range operations needed to cover diverse data processing usecases. moreover, the data processing workloads can be distributed among multiple nodes with spark by simply setting the necessary configurations. further, user-friendly debugging features via jupyter notebooks are included for fast build-test of custom etl pipelines. in addition, dataverse supports multi-source data ingestion from on-premise storage, cloud platforms, and even web scraping. this feature empowers users to easily transform raw data from various sources. driven by the aforementioned features, we posit that dataverse will be a useful tool for effortlessly building custom etl pipelines at scale for fast llm development. ## why dataverse? in the era of llms, data scales exponentially @xcite , necessitating an efficient and scalable solution @xcite . not only that, the fast pace of the llm literature comes with the need to support a wide range of data operations such as toxicity and bias removal @xcite , personally identifiable information (pii) obfuscation @xcite , and data quality filterings @xcite . thus, on top of utilizing distributed systems, llm-aware data processing also requires natively supporting a wide variety of operations and easy addition of custom data operations. while there are many existing data processing libraries such as proposed @xcite @xcite , none of them are not yet the whole package of being easy to customize and supporting a wide variety data operations. to address this gap, we introduce dataverse with a user-friendly design in mind, allowing users to utilize (custom) data processing tools and distributed system via simply setting the blocks and configurations. in the following sections, we detail the comparison between dataverse and other existing open-source frameworks for llm-aware data processing. ## dataverse dataverse is an open-source library for building etl pipeline for llms with user-friendly design at its core. the overview of the dataverse library is shown in figure 1 . ## conclusion to address the surging needs of data processing at massive scales, owing to the rise in popularity of llms, we propose dataverse, an open-source library for etl pipelines designed with scalability and future growth in mind. dataverse is userfriendly designed with a block-based interface, allowing users to easily add custom functions for data processing while also natively supporting a wide array of commonly used data operations. furthermore, dataverse offers scalable solutions through its seamless integration with spark and aws emr, allowing users to use dataverse to handle data workloads of varying sizes. we envision dataverse becoming a central hub for llm data processing, facilitating collaboration, knowledge exchange, and ultimately accelerating advancements in the field.