Separating the Wheat from the Chaff with BREAD : An open-source benchmark and metrics to detect redundancy in text (2023)

Abstract: Data quality is a problem that perpetually resurfaces throughout the field of NLP, regardless of task, domain, or architecture, and remains especially severe for lower-resource languages. A typical and insidious issue, affecting both training data and model output, is data that is repetitive and dominated by linguistically uninteresting boilerplate, such as price catalogs or computer-generated log files. Though this problem permeates many web-scraped corpora, there has yet to be a benchmark to test against, or a systematic study to find simple metrics that generalize across languages and agree with human judgements of data quality. In the present work, we create and release BREAD, a human-labeled benchmark on repetitive boilerplate vs. plausible linguistic content, spanning 360 languages. We release several baseline CRED (Character REDundancy) scores along with it, and evaluate their effectiveness on BREAD. We hope that the community will use this resource to develop better filtering methods, and that our reference implementations of CRED scores can become standard corpus evaluation tools, driving the development of cleaner language modeling corpora, especially in low-resource languages.

## introduction in this paper, we introduce a benchmark and propose a suite of metrics to help identify a common facet of low-quality data: repetitive boilerplate that is not reflective of natural linguistic content. large language corpora scraped from the internet are becoming invaluable tools as self-supervised language modeling has gained prominence as a driving force of advancements in nlp @xcite @xcite . in the case of many low-resource languages, noisy in-language data often makes up a significant proportion of any scraped corpus @xcite . very often, this noise is in the form of repetitive boilerplate: uninteresting data without linguistic diversity, such as a long list of similar products from an e-commerce website. automatically reducing repetitive boilerplate in low-resource language corpora remains an important problem to extend nlp to the thousands of languages currently underserved by language technology. to address the problem of redundant boilerplate, our contributions in the current work are two-fold: the difference between a paragraph of natural text and a long, repetitive list does not depend on the source language or the particular thing that is repeating. therefore, it is possible to build language-agnostic metrics that ignore textual features entirely, and operate purely on the tokenfrequency distribution. using this intuition, we explore three ngram-based metrics: type-token ratio (ttr), measuring the percentage of unique ngrams; ngram-moment, measuring the peakiness of the frequency distribution; and ngram-zipfianness, measuring the distance from the expected frequency distribution of natural language. our objective is to detect redundant language within one document. this is different from a commonly studied problem in data quality management, where redundancy refers to a dataset containing many redundant copies of similar natural documents. we open-source the bread benchmark and the cred metrics, making it a replicable resource for the community. ## related work in the field of data-quality management, quality scores are used for measurement and improvement, and often incorporated into an iterative process @xcite . for nlp, there are many existing works highlighting the importance of cleaning data for training neural models @xcite @xcite . many denoising approaches rely on classifiers @xcite @xcite or cross-entropy distance between models @xcite @xcite , an approach often applied to data weighting and curriculum training @xcite @xcite . there are neural diversity metrics, like @xcite , which uses the cosine distance between task2vec embeddings @xcite , and has been applied to measure llm output diversity @xcite . although data noise has always been a recognized problem, it has become a more pressing issue in recent years, as models have become more and more expressive, therefore also more capable of memorizing noise. statistical machine translation models were more robust to data noise and tended only to benefit from bigger data @xcite (with a few exceptions, like @xcite ), and works on data filtering were usually focused on improving training efficiency (for instance, @xcite ). despite their generally higher performance, neural models tended to be much more sensitive to data noise @xcite , possibly as a result of being able to memorize statistical outliers @xcite . even early versions of paracrawl damaged mt performance @xcite , and the winners of the yearly wmt campaign tend to rely heavily on data filtering (junczys-dowmunt, 2018a; @xcite @xcite . as a result, there have been several data filtering shared tasks in wmt @xcite @xcite , and open-sourcing of various iterations of data cleaner bicleaner @xcite @xcite , which use a variety of approaches, including bilingual dictionaries, random forests, and neural models. while neural metrics or complex ensembles like bicleaner are often effective, they 1) are harder to interpret; 2) may filter on artifacts like domain, rather than quality; 3) will tend only to work for languages they have explicitly been trained on; and 4) cannot be replicated between works unless a public implementation is released. for this reason, the baseline metrics released with bread are simple, interpretable, surface-level metrics, that work independent of language and domain. a token-based metric to measure the diversity and redundancy of token ngrams between documents in a corpus (rather than within segments of one document) is self-bleu @xcite , which is based on the widely used bleu score @xcite . on a more granular level of character ngrams, the chrf @xcite and chrf++ @xcite metrics measure similarity between documents, correlating better with human judgement than token-level metrics like bleu, especially for low-resource and highly-inflecting languages @xcite @xcite . we follow this intuition and use character-ngram metrics. the frequency moment score defined in the present work is similar to segment-level chrf applied with itself as its own reference. perhaps the most similar approaches to those in the present work come from a separate field, namely detecting redundancy and diversity in relational or tabular data @xcite . @xcite and @xcite define interpretable minimality scores to measure redundancy at a schema-level for tabular data, based on cluster density, which is equivalent to the ttr in the present work. ## bread: dataset and annotation we release bread (boilerplate and redundancy evaluation on assorted documents), an expertannotated dataset spanning 360 languages, to tune and benchmark methods for filtering repetitive boilerplate. bread consists of randomly-chosen documents from the multilingual, common-crawlbased madlad-400 dataset @xcite , which are then annotated by expert nlppractitioner annotators. our annotation schema consists of two highconfidence classes and two low-confidence classes. the high-confidence classes are 1) rep, repetitive boilerplate (@xmath0, and 2) ok, natural text (@xmath1. to keep the examples in rep and ok high-confidence, we also use two low-confidence codes: boil, for documents that are clearly nonlinguistic boilerplate or noise, but are not necessarily repeating (@xmath2; and unk for where the annotator was not sure (@xmath3. documents labeled as unk were discarded. see appendix bread is split into a tune and a test set, each with 1000 documents. we propose two benchmarks, scored with f1 on the following binary prediction problems: ## methods we explore three well-studied, straightforward metrics based on ngram frequency distributions and evaluate their effectiveness in the domain of measuring repetitive boilerplate. we explore both character ngrams and token ngrams, as well as combinations of the two. as with bleu @xcite , we consider using multiple n-gram lengths at once, and combining these scores by averaging them. by construction, all metrics assign a higher score to noisier text. the input to all our metrics is the smoothed frequency distribution of ngrams within a document. distributions of ngrams tend to be noisier for shorter texts, so we apply laplace smoothing with parameter λ, and clip the distribution with ϵ-thresholding (keeping only ngrams with probability over some ϵ @xcite ). let f (i) n be the raw frequency of the ith most common n-gram. we define our smoothed frequency distribution as: this said, the authors would like to foreshadow that this distribution clipping and smoothing end up not being very important parameters for wellperforming metrics, so the reader may safely ignore this and imagine that the metrics are a function of raw frequency. the metrics we explored are as follows: ttr: as an intuitive and well-known baseline metric, we use the token-type ratio (ttr) @xcite , which is the fraction of unique tokens in a document (types) over the total number of tokens. we use n-grams as tokens. frequency moment score: the second score we consider is the generalized moment of the frequency probability distribution, the sum of all frequencies when applying a with a nonlinearity g(x). for a character n-gram with length n, the momet score is defined as: the nonlinearity g(x) is a parameter we can vary to best fit out benchmark. intuitively, setting any superlinear g(x), this metric measures redundancy, or peakiness, of the ngram counts, as the score is larger when there is more weight in the head of the distribution. when g(x) = x k , the score corresponds to the kth moment of the distribution; when g(x) = -x log(x), it corresponds to the entropy. zipfianness: human languages have a largely consistent word distribution: across languages, the empirical frequency of how often different words appear follows a zipfian, or power-law distribution, where the word in frequency rank r has frequency roughly proportional to 1 r @xcite . for example, in english the most common word "the" occurs around double the rate of the second most common word "of". to test whether a document is distributed like natural text, we can check whether its ngram distribution matches the empirical ngram distribution of a human language, which we estimate as a function of the n-gram length with a slight modification of the classic 1 r value (details in appendix d). 2 therefore, we define the zipfianness score as follows: where fi is the estimated frequency of the ith most common token, and d(x, y) is a distance met-ric. for d(x, y), we consider |x -y| 2 , log(|x -y|), log 2 (|x -y|) and jsd(x, y). we initially also considered kl divergence (in both directions) and absolute distance, but they proved less effective. ## baseline metric results on bread as shown in fig. 2 , all metrics have fairly good correlation with human judgement, even when they are trained on the out-of-domain split of bread (the off-diagonal entries). for detecting repetition alone (bread-repeat; top row), both the moment score and the zipfianness score performed about 5% better on both tuning and test sets than ttr. when detecting both noise and boilerplate (bread-noisy), the difference in scores is more pronounced, with zipfianness outperforming ttr by 9% on the test split. the moment score, which like ttr is only able to detect redundancy but not other types of noise, barely outperforms ttr. it is worth noting that for questions of data noise, there is a large difference between apparently close scores, if they are both close to 100. @xcite note ( §massive class imbalances: 99% accuracy is not enough) that if a language identification model has a precision of 99.0, using it to generate a dataset for a typical low-resource language will yeild a dataset with just under a tenth of a percent of sentences in the target language. increasing this precision to 99.9%, though under 1% better in additive terms, is a 10x improvement in dataset precision. keeping this in mind, we see that although we have a ways to go with better data quality scores, the improvement in noise detection from 78 f1 to 85 f1 is quite substantial! for a qualitative understanding of what scores on bread look like, one can refer to figure 3 , which shows the moment score as a function of length, along with the decision boundary. details of the best hyperparameters per ngram length are given in appendix ## open-sourcing we open-source reference implementations of these metrics. following the example of sacrebleu @xcite , each score has a unique signature re-porting all relevant hyperparameters, so it is fully reproducible. in order to suit different levels of noise and different preferences of precision versus recall, we release versions of each classifier that have been tuned for f1 on a balanced version of bread, as well as a version that has been tuned on the p4 score @xcite with bread upweighted so it is 75% clean data. ## conclusion data quality is an evergreen problem, and as nlp is widening to a growing set of low-resource languages, where noise is a more severe problem, the need for more interpretable metrics to asses noise becomes especially prominent. recent approaches to highly multilingual technologies like nmt and langid have reported severe noise issues for lowresource languages @xcite , and many publicly available datasets with low-resource languages in fact contain no inlanguage content @xcite . nonetheless, there was heretofore no public benchmark for boilerplate and noise detection.the present work introduces bread, a multilingual, expert-annotated benchmark for detecting noise. it also investigates several interpretable, language-agnostic baseline metrics based on character ngram frequency distributions, as well as their scores on the public dataset madlad-400. finally, it open-sources reference implementations of several language-agnostic metrics for scoring and classifying data.