D ecorate LM : Data Engineering through Corpus Rating, Tagging, and Editing with Language Models (2024)

Abstract: The performance of Large Language Models (LLMs) is substantially influenced by the pretraining corpus, which consists of vast quantities of unsupervised data processed by the models. Despite its critical role in model performance, ensuring the quality of this data is challenging due to its sheer volume and the absence of sample-level quality annotations and enhancements. In this paper, we introduce DecorateLM, a data engineering method designed to refine the pretraining corpus through data rating, tagging and editing. Specifically, DecorateLM rates texts against quality criteria, tags texts with hierarchical labels, and edits texts into a more formalized format. Due to the massive size of the pretraining corpus, adopting an LLM for decorating the entire corpus is less efficient. Therefore, to balance performance with efficiency, we curate a meticulously annotated training corpus for DecorateLM using a large language model and distill data engineering expertise into a compact 1.2 billion parameter small language model (SLM). We then apply DecorateLM to enhance 100 billion tokens of the training corpus, selecting 45 billion tokens that exemplify high quality and diversity for the further training of another 1.2 billion parameter LLM. Our results demonstrate that employing such high-quality data can significantly boost model performance, showcasing a powerful approach to enhance the quality of the pretraining corpus.

## introduction the advent of large language models (llms) has ushered in transformative changes across various domains of artificial intelligence @xcite , from natural language processing to complex task execution @xcite . the backbone of these models' effectiveness lies in their training processes, specifically in the quality and composition of their pre-training corpora @xcite . traditionally, llms are pre-trained on vast datasets composed of billions of tokens harvested from diverse text sources. data quality is of vital importance for training llm @xcite . however, acquiring highquality data is a formidable challenge due to the sheer volume and unstructured nature of it. the reliance on large-scale unsupervised data leads to the inclusion of numerous low-quality texts within the training data. this infusion of poorquality data can adversely affect the models' learning processes, resulting in performance deficiencies and limitations in their applicability. however, the existing methods for curating and enhancing the quality of such datasets are often inadequate. they typically lack the capacity to scale to the size required while maintaining or improving data quality, primarily due to the absence of fine-grained annotations and the impracticality of manual oversight. addressing these challenges requires innovative approaches that can scale with the data requirements of llms while ensuring enhancements in data quality. this paper introduces decoratelm, a comprehensive data engineering methodology designed to refine the pretraining corpus through a systematic "decorating" process. the term "decorating" in this context refers to a series of processes aimed at enriching the data with additional metadata, improving its structure, and ensuring its relevance and quality. decoratelm employs a three-phase strategy to accomplish these goals. the first phase, rating, involves evaluating texts against a predefined set of quality criteria. these criteria are designed to assess the educational value, expertise, fact and trivia, reasoning level, scarcity, structural format, story-likeness and subjectivity of texts. the second ## related work in recent years, the quality and selection of data for training language models receive considerable attention. researchers propose various methodologies to assess, select, and improve high-quality data, with the goal of enhancing both the performance and efficiency of models @xcite @xcite . data annotation and rating. qurating, deita, and alpagasus are employed for data annotation, each utilizing distinct methodologies to enhance training via refined rating scores @xcite @xcite . phi-1 and mods use gpt-4 and deberta to improve educational data and precise data selection, accelerating learning and fine-tuning @xcite . ## experiments in this section, we conduct data experiments to demonstrate the effectiveness of decorated corpus. ## conclusion in this paper, we present decoratelm, a data engineering method designed to refine the pretraining corpus through data rating, tagging and editing. decoratelm employs a dual-training strategy, wherein two student models with 1.2 b parameters are trained: one designed for rating and tagging, and the other focused on editing. our experiments show that introducing rating and editing in data corpus significantly enhances data quality by improving the overall performance of slm on various existing benchmarks. furthermore, our empirical study verifies that the implemented tagging strategy achieves a more balanced distribution of categories within the training dataset. this equilibrium in categorization enables a more thorough comprehension of slm proficiency across diverse domains. these encouraging results underscore the importance of training data quality in fully exploiting the capabilities of large language models, thereby suggesting several compelling avenues for future research.