L ong W anjuan: Towards Systematic Measurement for Long Text Quality (2024)

Abstract: The quality of training data is crucial for enhancing the long-text capabilities of foundation models. Despite existing efforts to refine data quality through heuristic rules and evaluations based on data diversity and difficulty, there’s a lack of systematic approaches specifically tailored for assessing long texts. Addressing this gap, our work systematically measures the quality of long texts by evaluating three fundamental linguistic dimensions: coherence, cohesion, and complexity. Drawing inspiration from the aforementioned three dimensions, we introduce a suite of metrics designed to evaluate the quality of long texts, encompassing both statistical and pre-trained language model-based ones. Leveraging these metrics, we present LongWanjuan, a bilingual dataset specifically tailored to enhance the training of language models for long-text tasks with over 160B tokens. In LongWanjuan, we categorize long texts into holistic, aggregated, and chaotic types, enabling a detailed analysis of long-text quality. Furthermore, we devise a data mixture recipe that strategically balances different types of long texts within LongWanjuan, leading to significant improvements in model performance on long-text tasks.

## introduction effectively processing long texts is a crucial capability of language models and has recently become a focal point of research @xcite @xcite . tasks such as long document summarization @xcite , long document question answering @xcite , repository-level code tasks @xcite , and retrieval-augmentation generation @xcite often involve handling thousands or even tens of thousands of tokens. the quality of data is vital for the long-text capabilities of foundation models @xcite figure 1 : the three dimensions for measuring the quality of long texts: coherence, cohesion and complexity. @xcite . there have been efforts made to improve data quality. some approaches employ heuristic rules, such as deduplication and the removal of overly short data entries @xcite . additionally, some other approaches consider data diversity and perplexity based on pre-trained language models @xcite . however, these filtering rules are designed for general training data and do not take into account the unique characteristics of long texts. to systematically assess the quality of long texts, we adhere to linguistic fundamentals and evaluate them through three dimensions: coherence @xcite , cohesion @xcite , and complexity @xcite , as illustrated in figure 1 . these three dimensions have often been used to evaluate and analyze text quality @xcite . coherence measures the overall consistency and clarity of the text as a whole @xcite @xcite . cohesion gauges the strength of connections between sentences or sections of the text @xcite @xcite . complexity assesses the linguistic sophistication within the text @xcite @xcite . given that long texts typically contain more extensive content, they necessitate elevated levels of these characteristics to effectively convey information and engage in discussion. drawing from these three fundamental dimensions, we propose a set of metrics to quantitatively analyze the quality of long texts. these metrics encompass both statistical and pre-trained model-based approaches, offering strong interpretability. further details on these metrics can be found in section 3. based on the characteristics across these three dimensions, we categorize the long texts in pretraining dataset into three types: holistic long texts, encompassing complete works such as books, academic papers, reports, novels, and interviews; aggregated long texts, consisting of short texts related by topic or fragmented texts like extensive lists or tables; and chaotic long texts, characterized by nonsensical content such as garbled data. drawing upon these classifications, we manually annotated a validation set of 200 samples from slimpajama @xcite and wanjuan @xcite to validate the correlation between our proposed metrics and human judgments. our quantitative metrics effectively differentiate between the three categories of long texts. building on these analysis and metrics, we create a bilingual long-text dataset with category labels, named longwanjuan, containing over 160b tokens. with longwanjuan, we propose a data mixture recipe to mitigate the imbalance between holistic long texts and aggregated long texts within the dataset. specifically, by removing chaotic long texts and upsampling aggregated long texts, we continue to train internlm2-7b @xcite with an additional 5b tokens, thereby achieving state-ofthe-art performance for long texts on models of the 7b parameter scale. the effectiveness and generalizability of this recipe are analyzed in section 5.4. in summary, our contributions are as follows: ## method long texts, characterized by their extended contexts and abundant information, pose distinct challenges in maintaining textual integrity and quality. we systematically measure the quality of long texts through three dimensions: coherence, cohesion, and complexity. each dimension is accompanied by corresponding quantitative metrics, allowing for an effective measurement of long text quality. ## conclusion we try to systematically analyze the quality of long texts from three linguistic dimensions: coherence, cohesion, and complexity. inspired by these dimensions, we develop a series of metrics based on statistics and pre-trained models to quantitatively assess the quality of long texts. utilizing slimpajama and wanjuan, we construct the longwanjuan dataset and categorize texts into three types: holistic, aggregated, and chaotic texts, according to our proposed metrics. we introduce a data mixture recipe based on the longwanjuan dataset to address the issue of the imbalance between holistic long texts and aggregated long texts, achieving state-of-the-art performance on the longbench benchmark. our experimental analysis further validates the effectiveness of the proposed recipe.