Unitxt: Flexible, Shareable and Reusable Data Preparation and Evaluation for Generative AI (2024)

Abstract: In the dynamic landscape of generative NLP, traditional text processing pipelines limit research flexibility and reproducibility, as they are tailored to specific dataset, task, and model combinations. The escalating complexity, involving system prompts, model-specific formats, instructions, and more, calls for a shift to a structured, modular, and customizable solution.Addressing this need, we present Unitxt, an innovative library for customizable textual data preparation and evaluation tailored to generative language models. Unitxt natively integrates with common libraries like HuggingFace and LM-eval-harness and deconstructs processing flows into modular components, enabling easy customization and sharing between practitioners. These components encompass model-specific formats, task prompts, and many other comprehensive dataset processing definitions. The Unitxt Catalog centralizes these components, fostering collaboration and exploration in modern textual data workflows. Beyond being a tool, Unitxt is a community-driven platform, empowering users to build, share, and advance their pipelines collaboratively. Join the Unitxt community at https://github.com/IBM/unitxt

## introduction textual data processing has always been at the heart of nlp, but in the current landscape it has taken on new roles. a prominent one comes from llms' role as general interfaces, that receive an example, but also the task they should perform, general system instruction and other specifications, all in natural language. thus, the inputs -or promptsthat a model receives now consist of many components, that can be combined in different ways: task instructions @xcite , in-context demonstrations @xcite , system prompts and more. at the same time, for text generation models, model outputs are themselves rich textual data, and thus can be processed and evaluated with a range of different approaches and paradigms. therefore, textual data processing for llms is growing increasingly complex. it incorporates a large number of non-trivial design choices and parameters, which pose new challenges for maintaining flexibility and reproducibility in llm research. broadly, research in computer science, and in particular within nlp, thrives on that combination of flexibility and reproducibility. on the one hand, it should be simple to try new ideas: to compare different approaches, choose parameters, and easily switch out one workflow or architecture with another. on the other hand, the results of these explorations must be shared in such a way that others are able to -and crucially, are likely toreproduce and try them. to enable the above, code reuse, a well-defined api and ease of use are pivotal, ensuring reproducibility and applicability in practice. how such traits allow for widespread adoption is epitomized by the hugging face transformers library @xcite . today, a modest set of hyperparameters is sufficient to reproduce a training or inference workflow. this has had an undeniable and dramatic impact on the ability to make progress in the field. such is not the case, however, for textual data pipelines. unfortunately, data-preparation for llms has no standards, processing model inputs or outputs of the same data often comes with rewriting the code, leading to mismatches in reported values @xcite , unanswerable examples and hidden bugs @xcite and general time waste. crucially, the additional components beyond traditional processing, such as in-context demonstrations, have no canonical api. this prevents fair comparisons between different studies, discourages exploring combinations, hinders integrating a particular approach (say, a new type of system prompt) into an existing nlp system, and prevents major 207 scale-ups in terms of datasets, tasks and metrics. to address these gaps, we introduce a new collaborative framework for unified textual data processing named unitxt. this new python library supports multilingual textual data processing through flexible pipelines called recipes. a recipe (see §4.1 and examples in §3) is a sequence of textual data processing operators, including, among others, operators that load data, pre-process it, handle the preparation of different parts of a prompt, or evaluate model predictions (see figure 1 ). aiming for reuse, unitxt ships with a catalog containing a wide variety of pre-defined recipes for various tasks. these are all based on a diverse set of built-in operators that are also shared in the catalog. having a centralized location for these components, where anyone can add new ingredients (such as recipes or operators), or share existing ones, fosters collaboration, transparency and reproducibility. as fitting a recipe, the modularity of unitxt enables mixing and matching of ingredients to create new recipes. this ability to mix and match ingredients enables unitxt to support 100k+ recipe configurations, allowing users to experiment with a large set of such recipes by to obtain multiple configurations of tasks, datasets and new formatting (see §3 for example). changing libraries is always a nuisance; therefore, unitxt is designed to seamlessly integrate with preexisting code, offering a hassle-free experience without even needing a pip install. for instance, unitxt can load huggingface datasets and produce outputs that adhere to the same format, allowing it to integrate seamlessly with other parts of your codebase ( §4.4.1). demonstrating this, incorporating unitxt, with all its tasks, datasets, templates and metrics into lm-eval-harness @xcite required only 30 lines of code, while preserving the current api and ensuring a smooth transition and compatibility with existing workflows (app. a). unitxt, an open-source library, is under active development by ibm and the community. the code and documentation are available on github at: https://github.com/ibm/unitxt , the ui, at https://bit.ly/unitxt-explore while the demo video is at https://bit.ly/ unitxt-video . ## use cases unitxt for evaluation: the increasing capabilities of llms require evaluation frameworks that test models over an unprecedented number of datasets, tasks and configurations @xcite @xcite . unitxt can serve as the backbone of such evaluation efforts, by supporting easy changes across multiple important axes, including tasks, languages, prompt structure (e.g. instructions, verbalizations, etc.), augmentation robustness and more. moreover, with the unitxt catalog, different distinct projects can share their full evaluation pipelines, making their data-preparation and evaluation metrics reproducible. unitxt for training: modern llm training frameworks have extensive data requirements to attain state-of-the-art performance. multiple datasets across diverse domains and languages need to be leveraged to impart broad capabilities; various prompt formulations and verbalizations are necessary to enable instruction-following, where verbalizations are the final text form. however, combining heterogeneous data sources and textual representations poses significant engineering challenges. without a common underlying framework, data augmentation, multitask learning and few-shot tuning become prohibitively complex. this is where unitxt steps in, as an indispensable data backend. unitxt enables seamless fusion of diverse datasets. moreover, the standardized format also facilitates changes to the datasets, dynamic prompt generation, data augmentations and model-specific format, to name just a few. by handling the data wrangling complexity, unitxt empowers researchers to focus on creating performant, robust and safe llms. for both evaluation and training, unitxt has already been adopted as a core utility for llms in ibm by multiple teams working on various nlp tasks, including classification, extraction, summarization, generation, question answering, code, biases and more. in total, the open source catalog contains more than 100k possible pipeline configurations. ## unitxt: library tour to introduce unitxt, we begin with a tour of the library, and specifically, with the creation of a recipe. a recipe contains all the data-processing and metric configurations needed, including the data, task, standardize dataset fields verbalize generate with model de-verbalize standardize metrics inputs task template format load data calculate metric resources … format for model unitxt.evaluate() unitxt.load_dataset() operators: … rename field split train/val add noise labels sample demo add system prompt finalize format augment spaces template and formatting (see details in §4). here we define a recipe that loads the sts-b dataset for a sentence similarity task: @xmath0. stsb , # dataset info card @xmath1. text_similarity , @xmath2. helpful , @xmath3. user_agent , @xmath4 with a recipe, a concrete dataset can be loaded: importantly, every data instance in a dataset loaded with a unitxt recipe contains a fully prepared source text, which can be directly passed as input to the model. for example, here is such source text for one sentence-similarity data instance, integrated with three formatting decisions, a "helpful model" system-prompt, a user-agent response schema and one demonstration: [ user ]: for the following texts rank the similarity between 1 to 5. text 1: " i love ice cream " text 2: " i like ice cream " [ agent ]: 4.8 [ user ]: text 1: " i hate pizza " text 2: " i like pizza " [ agent ]: loading a dataset with a unitxt recipe also adds a metric-ready target text (created from the original target) to each data instance. to evaluate the model's textual predictions, we call: @xmath5. evaluate ( dataset , @xmath6, ) the evaluation results are a dictionary of task defined metric names and the values computed for them. ## design in this section we outline the design of unitxt. unitxt processes data by applying a modular sequence of operators, which are segmented into 5 key ingredients ( §4.2) color-coded as in fig. 1 : ■ resources, ■ task, ■ template, ■ format and ■ extensions. these ingredients are then used to build the data preparation ( §4.4.1) and evaluation ( §4.4.2) pipelines. ## unitxt ui: explore & preview the objective of the user interface is to guide users through the essential steps of recipe creation, illustrated with pertinent examples. additionally, it allows for catalog exploration. the ui complements the experience with the option to execute the examples on some pre-set model (e.g., flan-t5base), get the predictions and associated scores. the interaction entry point is the tasks. upon clicking, the tasks taxonomy is presented, and the users have the option to choose the applicable task type. selecting a task results in showing only the relevant datasets and templates. once the user selects a dataset, and a template, and presses "generate prompts" a random example enhanced with the template is loaded. if the user wants to augment the input with system prompt, or response-schema those will be instantly added when opted for. as in-context learning evaluations are supported, the user can select the preferred number of shots. once satisfied with the example, the user has the option to proceed with executing it on a model, wherein the predictions and corresponding scores will be displayed for this specific example. further, going to the code tab, the user can copy the associated code into a notebook and run. users have the option to explore various examples, enhancing their comprehension and confidence in the chosen configuration. ## related work standardized data processing for evaluation and training has been a longstanding need in the nlp community and has been repeatedly addressed in the past. datasets @xcite and evaluate foot_0 are community-driven libraries, providing a standardized interface to diverse corpora and metrics, as well as supporting many data processing operations. these packages, however, fall short of providing a standardized, shareable and reproducible framework to cast the raw data into textual prompts and cast them back from text to a metric digestible format. the lack of such a framework hinders reproducibility, as often slight variations in ad-hoc text processing code may yield significantly different evaluation scores. moreover, it also prevents users from easily scaling up their experiment, as each task and dataset often requires specific code for processing and evaluation. unitxt builds on top of these frameworks, harnessing them as resources ( §4.1) to produce a full data-preparation and evaluation framework. while several existing frameworks have contributed to data pipeline management workflows, a common drawback, for those we are aware of, is the absence of a well-defined and flexible modularity in their design, such as the ability to define specific components for system prompts, task instructions and model-specific formats. this absence of clearly defined components makes it challenging to share and customize such pipelines effectively, across different datasets and tasks. like unitxt, tasksource @xcite supplies tools for consistent preprocessing over different datasets, simplifying their usage. however, it is primarily designed for discriminative tasks, uses fixed formats and lacks a modular design that enables sharing, mixing and matching, and overall flexibility in processing steps. promptsource @xcite focuses on making and sharing natural language prompts but doesn't handle other types of data processing. each prompt is tied to just one dataset, making it hard to reuse and share. furthermore, prompts aren't split into system, instruction, and format parts, limiting options for flexibility and reuse. seqio @xcite , offers task-based pipelines encompassing pre-processing, post-processing, and handling metrics. however, a structured breakdown of these processing steps is absent, limiting the creation of shareable catalogs within the community. in this framework, each process is a generic function and specialized steps are missing, like those designed for system prompts. a different branch of solutions are language model evaluation frameworks such as opencompass (contributors, 2023), helm @xcite and lm-eval-harness @xcite also implement their own standardized data processing pipelines in order to obtain verbalized prompts for lms. these, however, are highly coupled with the inference engine and cannot be used as standalone data-processing pipelines or integrated into other code bases. ## conclusion in this paper, we have introduced unitxt, an opensource python library aimed at unifying textual data processing pipelines for large language models. unitxt provides a modular, flexible framework that enables mixing and matching of various pipeline components like loaders, templates, formats and metrics. unitxt key capabilities are, standardization, flexibility, collaboration and scale. unitxt has already been successfully deployed for large language model evaluation and training within ibm. as the library matures through opensource community involvement, we hope its adoption will grow to push the frontiers of textual data processing for llms. we believe unitxt has the potential to significantly impact research and development of large language models by unifying textual data processing. through its emphasis on flexibility, reproducibility and collaboration, unitxt can help drive progress towards more capable, safer and trustworthy llms.