Leveraging Web-Crawled Data for High-Quality Fine-Tuning (2024)

Abstract: Most large language models are fine-tuned using either expensive human-annotated data or GPT-4 generated data which cannot guarantee performance in certain domains. We argue that although the web-crawled data often has formatting errors causing semantic inaccuracies, it can still serve as a valuable source for high-quality supervised fine-tuning in specific domains without relying on advanced models like GPT-4. To this end, we create a paired training dataset automatically by aligning web-crawled data with a smaller set of high-quality data. By training a language model on this dataset, we can convert web data with irregular formats into high-quality ones. Our experiments show that training with the model-transformed data yields better results, surpassing training with only high-quality data by an average score of 9.4% in Chinese math problems. Additionally, our 7B model outperforms several open-source models larger than 32B and surpasses well-known closed-source models such as GPT-3.5, highlighting the efficacy of our approach. We have released our code at https://github.com/zhouj8553/Web_to_SFT.

## introduction large language models (llms) have attracted much attention over the past years and high-quality data has been a crucial factor in achieving excellent performance. currently, two primary methodologies are employed for data acquisition. the first approach involves leveraging @xcite or other llms for distillation, such as alpaca @xcite , orca @xcite , and wizardlm @xcite , to enhance the capabilities of smaller models. the second approach @xcite @xcite annotates or selects data manually to further enhance model performance, emphasizing the significance of data quality over data quantity. how-ever, in certain domains like mathematics, even the state-of-the-art model gpt-4 fails to achieve outstanding performance @xcite @xcite . meanwhile, obtaining a large volume of human-annotated data within a short timeframe is not only challenging but also costly. conversely, web-crawled data tends to have a larger volume despite being prone to noise and formatting errors. leveraging processed webcrawled data for training can significantly alleviate the challenges associated with data collection in specific domains. we focus on mathematical reasoning, which requires a deep understanding of mathematical concepts and proficient reasoning abilities. previous studies @xcite have demonstrated the benefits of enhancing datasets with synthetic data. typically, these studies @xcite ) make full use of the excellent performance of gpt-4 on english mathematical datasets to generate simulated data for distillation to smaller models. in contrast, we explore the potential to acquire high-quality data without depending on additional powerful llms such as gpt-4, which doesn't perform well enough in chinese. we consider the ability to enhance performance without external models as crucial. this is because, in the event of becoming the top model in the field, it is vital to promptly leverage existing data for performance improvement. we identified two significant advantages of webcrawled data: it (1) has a large volume and (2) contains most of the necessary information to solve specific problems, despite its poor formatting. drawing on the intuition that rewriting data is comparatively simpler than performing intricate reasoning tasks for llms, we propose a method to augment the dataset by converting web-crawled data into high-quality ones. our approach begins by automatically aligning low-quality webcrawled data with high-quality seed data to gen-@xmath0, high-@xmath1. we subsequently utilize these pairs to fine-tune an llm, developing a model specifically designed to transform low-quality web-crawled data into highquality data. our experiments demonstrate that this approach significantly improves data quality and boosts model performance, surpassing traditional rule-based methods. the key contributions of our work are as follows: ## discussions relationship with rag the widely discussed rag @xcite @xcite technology is conducted during the inference period. providing references to the model and allowing the model to refer to these references in generating answers, helps the model reduce "hallucinations", especially for knowledge-intensive tasks. our method can be seen as rag during the training process. distilling the model's unknown knowledge into the training data can further enhance the model's capabilities. the injection of knowledge can also positively impact the model's generalization in related domains. ## conclusion we observed that in mathematical problems, format errors in the web-crawled data not only cause confusion in the output format but also result in semantic inaccuracies. building on this insight, we propose a simple and efficient method that leverages the abundant information in web-crawled data and the strong understanding capabilities of llms. our method enables the transformation of web-crawled data into high-quality ones without additional language models such as gpt-4. experiments demonstrate the superiority of our method. in the future, it is worth exploring how to extend this method to enhance data quality in various other scenarios.