Quality Does Matter: A Detailed Look at the Quality and Utility of Web-Mined Parallel Corpora (2024)

Abstract: We conducted a detailed analysis on the quality of web-mined corpora for two low-resource languages (making three language pairs, English-Sinhala, English-Tamil and Sinhala-Tamil). We ranked each corpus according to a similarity measure and carried out an intrinsic and extrinsic evaluation on different portions of this ranked corpus. We show that there are significant quality differences between different portions of web-mined corpora and that the quality varies across languages and datasets. We also show that, for some web-mined datasets, Neural Machine Translation (NMT) models trained with their highest-ranked 25k portion can be on par with human-curated datasets.

## introduction despite the advances in nmt research, the availability of parallel corpora is still a deciding factor of nmt model performance. this puts low-resource languages at a clear disadvantage @xcite . even the use of pre-trained language models (plms) is not quite enough to overcome the impact of data scarcity @xcite . publicly available web-mined parallel corpora (bitext) such as ccmatrix @xcite , ccalign @xcite , wikimatrix @xcite , nllb @xcite , and paracrawl @xcite bring a glimmer of hope against this data scarcity problem. compared to human-curated datasets, these are larger in quantity and contain data for hundreds of languages, including several low-resource languages. there are further initiatives to mine bitext for yet more languages as well @xcite . however, @xcite analysed a sample of 100 sentence pairs from some of these corpora and showed that these web-mined corpora have serious quality issues, especially for lowresource languages. @xcite noticed a drop in nmt results when a model was trained using a random 100k sample of ccalign. @xcite injected different noise types found in web-mined corpora (by analysing a random sample) into a clean parallel corpus and showed that it has a debilitating impact on nmt performance. these findings paint a grim picture of the utility of web-mined corpora. however, they all considered a random sample of these corpora to determine their quality. this implicitly assumes that the quality is consistent throughout the corpus. in this research, we show that analysing a random sample of such large web-mined corpora can be misleading. we selected parallel corpora for two low-resource languages sinhala and tamil, which made three language pairs pairs: english-sinhala (en-si), english-tamil (en-ta) and @xcite . instead of quality checking a very small random sample of a web-mined corpus as done by @xcite , we ranked the sentence pairs by means of a similarity measure and extracted top 25k, bottom 25k and a random 25k portions of each corpus. we improved the error taxonomy of @xcite and carried out a human (intrinsic) evaluation on a random sample of 250 from each of these portions. our results show that there are significant quality differences between the three portions, and the quality of the top 25k portion is much better than the other portions. we also noted major variations of quality across web-mined corpora belonging to different language pairs. we then carried out an extrinsic evaluation. we separately trained nmt systems by using these top, bottom, as well as the random 25k samples of the corpora and tested them with two different evaluation sets. these results also showed that nmt models trained with the top 25k portion are significantly better. nmt models trained with the full version of some of these corpora were even lagging behind models trained with their top 25k portion. the nmt model trained with the top 25k portion of the en-si and en-ta parts of the nllb corpus performed even better than a model trained with a human-curated corpus. we then fixed the translation issues in the top 25k of the nllb corpus using human translators. the time taken to clean the corpus was slightly less than the time taken to translate the corpus from scratch. although an nmt model trained with this cleaned corpus outperformed the uncleaned corpus, the resultant meagre gains cannot be justified when considering the time and money spent on the translators. in summary, our results caution the researchers not to haphazardly use the web-mined corpora with just random sampling. simply ranking a web-mined corpus first and then using only the high-quality portion would result in better accuracy in much less training time. we also hope other researchers (especially those working on lowresource languages) would carry out similar analyses for datasets of their languages. this will help future researchers make informed decisions when selecting web-mined corpora for nmt research. ## related work web-mined parallel corpora are gathered from any available website without guarantees about quality. @xcite ; @xcite pointed out that nmt systems built with such webmined corpora have performance issues. the common way to determine the quality of a parallel corpus is by analysing the performance of a machine translation system trained with that corpus @xcite @xcite . however, this does not indicate the types of noise in the corpus. human evaluation of the quality of parallel sentences (let them be web-mined, machine-generated, or human-generated) requires some criteria for the evaluators to make a judgement. @xcite introduced the direct assessment criteria, where each sentence pair is ranked on a 0-100 scale. however, such a numerical scale does not shed light on the different types of noise in web-mined corpora. @xcite analysed a webmined corpus and introduced the first categorisation of noise. the categories are: misaligned sentences, mis-ordered words, wrong language, untranslated sentences, and short segments. @xcite extended this categorisation with three new classes: raw crawled data, over/under-translation, and synthetic translation. in contrast to the above categorisations, @xcite 's taxonomy has labels for both correct and erroneous sentence pairs: 1.) correct translation -natural sentence, 2.) correct translation but boilerplate or low quality, 3.) correct translationshort, 4.) incorrect translation but both correct languages, 5.) source or target wrong language but both still linguistic content, and 6.) not a language. @xcite conducted a human evaluation using their taxonomy for three web-mined corpora (ccaligned, paracrawl v7.1, wikimatrix) and covered data from both high and low resource languages. @xcite used that taxonomy to evaluate english-catalan corpora. ## languages we selected three language pairs: english-sinhala (en-si), english-tamil (en-ta) and sinhala-tamil (si-ta). tamil @xcite and sinhala (si) are large institutional languages @xcite . however, considering their data availability, @xcite categorised tamil as a mid-resource language and sinhala as an extremely low-resource language. in the more recent language categorization by @xcite , sinhala has moved one class up, and the position of tamil is unchanged. sinhala, in particular, is contained only in the island nation of sri lanka, and has only seen slow progress in language technologies @xcite . but, being a multilingual country, translation systems are of utmost importance to sri lanka. this is particularly true for si-ta, as most government documents are first prepared in sinhala and then translated to tamil and english @xcite . ## web-mined parallel corpora ## quality estimation by humans as mentioned earlier, @xcite carried out the first human evaluation on the quality of web-mined corpora. although they reported results for a large number of languages including low-resource languages, their discussion was mainly centred around the language-wise aggregated results. thus they only used randomly selected 100 sentences from each language-specific corpus. @xcite carried out a similar study for catalan-english, but they also considered only 100 samples from a corpus. in contrast, we carried out a more detailed analysis of web-mined corpora belonging to the three language pairs by first ordering each parallel corpus according to the quality of the sentence pair. our hypothesis is that the quality of a web-mined corpus is not consistent across a dataset, thus analysing a random portion of the corpus would not give a clear picture of the quality of the corpus. ours herold et al. (2022)'s nl wl wrong language (src|trg) un untranslated (src|trg) x misaligned sentences cs short segments (max. length) cb misordered words (src|trg), raw crawled data, over-/under translation, synthetic translations cn cc participants: fifteen translators were employed to conduct the human evaluation across the three language pairs. evaluator selection and training details are in appendix b. ## human evaluation results each sentence pair was evaluated by three evaluators. the average agreement (measured in pearson correlation) per language pair is as follows: en-si 0.40, en-ta 0.55 and si-ta 0.57 (detailed results are in ## qualitative analysis of corpora in addition to the human evaluation discussed in the previous section, we also carried out a manual inspection of the top 25k portion of each corpus. ## impact of embedding technique we used laser-3 for ranking sentence pairs. the other commonly used measurements are labse and xlm-r @xcite . mbert is another option, however, sinhala is not included in this model. in fact, @xcite compared laser-1, labse, and xlm-r embedding performance for sentence alignment and reported that labse is superior to the other two. to determine whether the embedding technique has a noticeable impact, we ranked the ccmatrix en-si corpus using labse and xlm-r. then we selected the top, bottom, and random splits from this corpus and trained vanilla transformer-based nmt models. figure 6 shows the comparison (full result in ## conclusion we presented a fine-grained evaluation of the quality of web-mined corpora for three low-resource language pairs. we showed that the quality of such corpora significantly varies across different portions. our findings also indicate that simply using the highest quality portion of a web-mined corpus yields nmt results that may be on par with humancurated corpora in some instances. however, we are wary of further cleaning this top portion in hopes of better results, as the result gains do not justify the required human effort. project artefacts are released and the details are shared in the project github 9 . for our analysis, we considered the web-mined corpora without any pre-processing. if they were pre-processed (say) to remove duplicates, short phrases, or text in the wrong language, the performance of the embedding techniques may vary. we plan to investigate this in future. we also plan to expand this analysis to other low-resource languages.