Jellyfish: Instruction-Tuning Local Large Language Models for Data Preprocessing (2024)

Abstract: This paper explores the utilization of LLMs for data preprocessing (DP), a crucial step in the data mining pipeline that transforms raw data into a clean format. We instruction-tune local LLMs as universal DP task solvers that operate on a local, single, and low-priced GPU, ensuring data security and enabling further customization. We select a collection of datasets across four representative DP tasks and construct instruction data using data configuration, knowledge injection, and reasoning data distillation techniques tailored to DP. By tuning Mistral-7B, Llama 3-8B, and OpenOrca-Platypus2-13B, our models, Jellyfish-7B/8B/13B, deliver competitiveness compared to GPT-3.5/4 models and strong generalizability to unseen tasks while barely compromising the base models’ abilities in NLP tasks. Meanwhile, Jellyfish offers enhanced reasoning capabilities compared to GPT-3.5. Our models are available at: https://huggingface.co/NECOUDBFM/JellyfishOur instruction dataset is available at: https://huggingface.co/datasets/NECOUDBFM/Jellyfish-Instruct

## introduction data preprocessing (dp) is a critical step in the data mining pipeline that involves transforming raw data into a manageable and processable format ready for use. over the past decades, significant strides have been made in various dp tasks. until 2021, most efforts were concentrated on one or two specific tasks such as error detection (ed) @xcite , data imputation (di) @xcite haochen zhang and yuyang dong are co-first authors who contributed equally to this work. chuan xiao is the corresponding author. our models are available at: https://huggingface.co/ necoudbfm/jellyfish . our instruction dataset is available at: https://huggingface.co/datasets/necoudbfm/ jellyfish-instruct . abedjan, 2020; @xcite , schema matching (sm) @xcite , and entity matching (em) @xcite . a key challenge in developing generic solutions to dp is that these tasks differ in nature: they deal with errors, anomalies, matches, etc. and require different actions such as detection, repairing, and alignment. with the advent of llms, researchers have found a key to address this challenge, spurring the development of generic solutions for a wider array of dp tasks. the advantages of llms over non-llm dp methods reside in their natural language generation ability, internal knowledge, reasoning ability, generalizability, and adaptability through few- @xcite or zero-shot @xcite prompting, thereby reducing the cost of human labor (e.g., hyperparameters tuning) and yielding better intepretability. despite these strengths existing llm-based dp solutions @xcite @xcite , with reliance on gpt apis, have raised concerns about data breaches, as evidenced by openai's confirmed data breach @xcite . another limitation is the difficulty in domain specification @xcite . when dealing with data from highly specialized domains, training the llms used in these solutions can be costly (e.g., gpt-3.5) and even unavailable due to frozen parameters (e.g., gpt-4), posing difficulty in customizing the model. in response to these challenges, we propose to construct instruction data and tune llms for various dp tasks. the tuned model, namely jellyfish, distinguish itself with several key features: (1) jellyfish is a universal dp task solver tuned to the following tasks: ed and di for data cleaning, and sm and em for data integration. (2) varying from 7b to 13b, jellyfish can operate on a local, single, and low-priced gpu, ensuring data security and allowing further tuning. (3) capable of understanding natural language, jellyfish allows users to manually craft instructions for dp tasks (or simply use our prompts in this paper) and apply prompt engineering techniques to tailor it to specific tasks and datasets. (4) unlike many existing methods that rely heavily on handcrafted knowledge during inference @xcite , jellyfish features domain knowledge in its instruction tuning and enables optional knowledge injection during inference. whereas instruction tuning of llms has been largely used for unstructured text @xcite , the construction of jellyfish is non-trivial in the sense that (1) it tunes for structured data, (2) it finds a good data configuration for various dp tasks, and (3) it specifies domain knowledge that can be applied to unseen datasets. besides, it is expected that the model's performance in nlp tasks can be preserved for generalizability and further customization. to the best of our knowledge, this is the first study that investigates instruction tuning for dp with llms as universal solutions. as depicted in figure 1 , jellyfish is constructed by carefully selecting data from several public datasets widely used for dp evaluation, considering their impacts on the overall performance. by instance serialization, raw data is serialized into instruction tuning prompts. by knowledge injection, task-and dataset-specific knowledge -particularly domain knowledge that can be extended to unseen datasets -is infused to the prompts. moreover, we resort to mixtral-8x7b-instruct-v0.1 to generate reasoning data. as such, jellyfish distills mixtral's knowledge in reasoning dp results. our evaluation focuses on tuning a set of prevalent open llms, including mistral-7b-instruct-v0.2 (as jellyfish-7b), llama 3-8b (as jellyfish-8b), and openorca-platypus2-13b (as jellyfish-13b). the results show that our instruction data applies to all these base models, substantially improving the dp performance. compared to two categories of baseline methods, (1) non-llm methods -typically solutions based on machine learning (ml) or pre-trained language models (plms) -and (2) llm methods -typically gpt series methods, jellyfish-13b consistently outperforms non-llm methods on its seen datasets, and its effectiveness on unseen datasets even surpasses non-llm methods on their respective seen datasets. meanwhile, jellyfish-7b/8b also exhibit competitiveness, especially on di and em tasks. for unseen tasks, jellyfish models also deliver strong performance, rivaling gpt-3.5/4 models and showcasing generalizability to a wider range of dp tasks beyond the four tasks used for tuning. our evaluation reveals the impacts of data configuration and the use of reasoning data in building jellyfish, and discovers that jellyfish barely compromises the base model's nlp performance. furthermore, experiments demonstrate the advantage of jellyfish's interpretation over gpt-3.5 in reasoning capabilities as well as the effectiveness of knowledge injection. ## preliminaries in data mining, dp is a crucial step that deals with noise, missing values, inconsistencies, and heterogeneity in data. major dp procedures include data cleaning, data integration, data transformation, and data reduction @xcite . in this initial exploration of llms for dp, we concentrate on tabular data, one of the most common data types. our data model operates on relational tables specified by schemas. we assume all attributes are either numerical (including binary) or textual (including categorical) values. diverging from the traditional definition that presents the entire dataset and finds or fixes all the errors (or matches, etc.) within, we define the problem by handling one record (or a pair, depending on the task) at a time, so the prompt can be easily written and its length is within llms' token limitation. next, we outline the dp tasks involved in this study: these four tasks form the most critical part of dp @xcite and are extensively discussed in the context of data mining @xcite . we use them for instruction tuning. besides, we consider two unseen tasks: (1) column type annotation (cta): given a table with no header, our task is to infer the type of each column from a set of predefined types (e.g., name, time, location). (2) attribute value extraction (ave): given a text description of an entity and a set of predefined attributes, the task is to extract attribute values from the text description. we term each input object an instance, i.e., a record for ed and di, a pair of attributes for sm, a pair of records for em, a table or a column for cta, and a text description for ave. ## inference with jellyfish for inference, the prompt is same as the instruction data shown in figure 2 . users can craft datasetspecific knowledge into the prompt, such as the domain knowledge (e.g., constraints) outlined in the previous section. such user-specified knowledge is optional. feature engineering. users can optionally select a subset of features to improve performance. for instance, for em in the beer dataset, name and factory are more relevant features, while style and abv are less relevant. hence users may choose to use only name and factory as attributes. such feature engineering can be also implemented in the prompt as specific knowledge, e.g., you should only consider name and factory and ignore other attributes. prompt engineering. prompt engineering @xcite is the process of structuring text to enhance the model performance. we incorporate few-shot prompting @xcite , which conditions the jellyfish models to learn from a small selection of examples drawn from the dataset. the prompts for few-shot examples are reported in appendix f. ## related works the dp tasks targeted in this paper have been extensively studied. while traditional methods mainly depend on hand-crafted rules @xcite @xcite , advanced approaches employ ml techniques. (1) for ed, holodetect @xcite utilizes few-shot learning with a noisy channel model, and raha @xcite resorts to a series of ml pipelines such as feature engineering. (2) for di, notable approaches are based on vae @xcite , gan @xcite , attention mechanisms @xcite , and plms @xcite . (3) for sm, learning-to-rank @xcite , deep similarity matrix @xcite , and attention-based approach have been developed @xcite . (4) for em, prevalent approaches employ deep learning models for blocking @xcite or pairwise matching @xcite , as well as plms for both procedures @xcite . moreover, a plm solution @xcite is available for both sm and em. (5) for cta, prevalent approaches are mainly based on table representation learning @xcite @xcite . recently, chatgpt has been utilized @xcite . (6) for ave, early approaches employ lstm @xcite . with the advent of plms, many solutions resort to tuning bert @xcite @xcite . a recent work @xcite ) considered tuning gpt-3.5 and prompting gpt-4. the above methods are non-llm solutions prior to the llm era. many of them are based on plms that only apply to one or two dp tasks and need fine-tuning on the target dataset. recent progress developed generic solutions to dp based on frozen llms such as gpt-3 @xcite , gpt-3.5, and gpt-4 @xcite . fine-tuning gpt-3.5 and chatgpt for a variety of table-related tasks has also been investigated @xcite . besides the six tasks investigated in this paper, other dp tasks include data repairing @xcite @xcite , data fusion @xcite , and data transformation @xcite . we will investigate them in our future work.