Investigating Web Corpus Filtering Methods for Language Model Development in J apanese (2024)

Abstract: The development of large language models (LLMs) is becoming increasingly significant, and there is a demand for high-quality, large-scale corpora for their pretraining.The quality of a web corpus is especially essential to improve the performance of LLMs because it accounts for a large proportion of the whole corpus. However, filtering methods for Web corpora have yet to be established.In this paper, we present empirical studies to reveal which filtering methods are indeed effective and analyze why they are.We build classifiers and language models in Japanese that can process large amounts of corpora rapidly enough for pretraining LLMs in limited computational resources. By evaluating these filtering methods based on a Web corpus quality evaluation benchmark, we reveal that the most accurate method is the N-gram language model. Indeed, we empirically present that strong filtering methods can rather lead to lesser performance in downstream tasks.We also report that the proportion of some specific topics in the processed documents decreases significantly during the filtering process.

## introduction the quality of the pretraining data significantly impacts the performance of large language models (llms) @xcite . the training data mostly comprise web documents, and therefore it is essential to remove low-quality documents or paragraphs from them efficiently. however, no quality filtering method has been established for these documents. rule-based filtering can quickly remove documents with many unnecessary alphabets, symbols, and specific repetitive sentences. however, they have no comprehensive understanding of documents to be removed and can overdo or overlook * currently affiliated with hakuhodo technologies certain types of documents. on the other hand, learning-based filtering methods can remove some low-quality documents that seem to maintain a level of quality and therefore bypass the rule-based filters. however, it is not verified yet which filtering methods are better and what types of documents are removed by such filters. in empirical experiments, we examine learningbased filtering methods to remove low-quality documents in a web corpus. we focus on japanese corpora in this paper because there has been little study on methods for filtering japanese corpora for developing llms, while many models have been developed in recent years, especially for the japanese language. we test the perplexity of a language model and a relatively fast classifier to process a massive volume of corpora. the experimental results show that the perplexity filtering method based on an n-gram language model is the best. we also pretrain bert @xcite on a japanese web corpus filtered by the n-gram language model and evaluated it on japanese general language understanding evaluation, jglue @xcite . the results show that massively strong filtering results in performance deterioration. furthermore, topic analysis on the web corpus shows that the proportion of specific topics decreases during the filtering process. ## investigation method in this study, we ignore any bias caused by a filtering process and focus only on the quality of a corpus. furthermore, we use learning-based methods in our investigation in the hope that they can deal with documents that cannot be removed by rulebased methods alone. in addition, since we need to process a large corpus, we use a fast classification method. therefore, we examine learning-based methods using a classifier and a language model. ## experiments 4.1 experimental settings to evaluate filtering performance, we use the web corpus quality evaluation benchmark (wcqeb) foot_8 , which was created by llm-jp. this dataset consists of 500 japanese mc4 @xcite documents, each of which is manually labeled "accepted", "harmful", or "low quality". the evaluation metrics are accuracy, precision, recall, detection-power, f-score, and roc-auc for accepted harmful low quality number 235 20 245 the binary classification of whether a benchmark document is of high quality (positive example) or low quality (negative example). detection-power is the percentage of low-quality documents that are correctly classified as low-quality. roc-auc is calculated only for classifiers for which prediction probability is available. we use recall and detection-power as our main metrics for method comparison. the method with the highest recall and detection-power is the one that can remove the most low-quality documents without missing the most high-quality documents. for the implementation of the single / multilayer perception, we use the python library scikit-learn 10 . to train the single / multilayer perceptron and fast-text, we use 5,000 documents each from the bc-cwj (high-quality documents) and the japanese web corpus (low-quality documents) for a total of 10,000 documents. we evaluate the classification results for the benchmark documents. note that the single-layer perceptron does not provide outputs as probabilities, and thus probability calibration is performed using scikit-learn. to train the language model, we use only sentences ending with a punctuation mark "。" out of the entire bccwj corpus. a threshold is set based on the perplexity distribution during inference, and the benchmark documents are classified and evaluated around the threshold. the transformer-based language model is trained with gpt-neox @xcite with 19m parameters. the n-gram language model is trained using kenlm @xcite , and the 2 to 5-gram language models are compared, with 1-gram as the unit of word segmentation of the morphological analyzer, mecab 11 . as a preliminary experiment, we also tested the character-based models, but we finally adopted the mecab segmentation units, which were more accurate. we also measured the inference speed of each method, quantifying the time required to classify 10,000 documents in japanese mc4. measurements were taken three times, and the average was calculated. the experiments were conducted using five cores of an intel xeon gold 6148 processor. ## additional experiments we conduct additional experiments for detailed performance evaluation through bert pre-training with a filtered web corpus and fine-tuning with jglue, japanese general language understanding evaluation. we also analyze how the topics of the web corpus are affected by the filtering intensity. furthermore, we show the relevance of url domain filtering to one based on the n-gram language model. ## conclusion in this study, we used machine learning-based methods for quality filtering of a japanese web corpus and compared their performance on a quality evaluation benchmark. the experimental results showed that the classification method using the perplexity by an n-gram language model had the highest accuracy. however, too much filtering led to performance degradation in downstream tasks. in the future, we plan to evaluate downstream tasks with larger models and consider filtering at more fine-grained units such as paragraphs.