Human evaluation of web-crawled parallel corpora for machine translation (2022)

Abstract: Quality assessment has been an ongoing activity of the series of ParaCrawl efforts to crawl massive amounts of parallel data from multilingual websites for 29 languages. The goal of ParaCrawl is to get parallel data that is good for machine translation. To prove so, both, automatic (extrinsic) and human (intrinsic and extrinsic) evaluation tasks have been included as part of the quality assessment activity of the project. We sum up the various methods followed to address these evaluation tasks for the web-crawled corpora produced and their results. We review their advantages and disadvantages for the final goal of the ParaCrawl project and the related ongoing project MaCoCu.

## introduction machine translation and particularly neural machine translation is a data hungry process. data, ideally in the form of parallel texts, is many times scarce for many languages, poorly varied for others or very low quality. multilingual websites are a great source of parallel data to complement these poor data scenarios, enabling the use and usefulness of machine translation for many use cases. but the web is wild and automatic harvesting of parallel data is not exempt of errors. web-crawled parallel content, usually noisy, can be then filtered for quality. the final parallel sentences that make it to a web-crawled parallel corpus will have gone through a complex pipeline before they are compiled and released in the form of a parallel corpus. once produced, how good are these parallel sentences? how good is the corpus as a whole? what kind of errors does it contain? are these errors problematic for building machine translation? what type of evaluation process can help us to identify action points to improve the production pipeline? these are the questions that we were trying to answer when designing the tasks that would be carried out as part of the quality assessment activity in the paracrawl project. @xcite provides a full description of the project, methods to gather corpora and a description of released corpora and their usefulness to create machine translation systems. paracrawl goal was the release of the largest collection of parallel corpora harvested from multilingual websites to advance machine translation. initially targeting 23 co-official european languages paired with english, the final version contains also norwegian nynorsk, norwegian bokmål and icelandic paired with english and 3 corpora for co-official languages in spain paired with spanish. version 9 accounts for 1.457 million unique sentence pairs across 29 language pairs. foot_0 additionally, 17 corpora for other language combinations have been released as bonus corpora. in the following sections, we review related work and focus on the human evaluation methods. we also report about extrinsic automatic evaluation experiments through machine translation. we try to analyse how human and automatic evaluation methods relate and discuss their usefulness to to answer our questions. ## related work besides paracrawl, there have been a number of past and recent efforts to compile parallel corpora from web-crawled content. among the recent ones, we find, for example, wikimatrix @xcite , ccaligned @xcite or os-car @xcite . many of these parallel corpora are usually evaluated through machine translation @xcite where automatic filtering of corpora and its impact on machine translation quality has gained interest in the last years @xcite @xcite . some other recent work like @xcite has, in contrast, put the focus on human evaluation and recommend techniques to evaluate and improve multilingual corpora to avoid low-quality data releases. ## human evaluation human evaluation of the corpora in paracrawl was done in 3 different ways depending on the version of the corpus: a) based on error annotation of parallel sentences, b) based on post-editing (pe) of the output of mt systems trained with the crawled parallel corpora and c) based on manual searches over the parallel sentences using a concordancer. we detail each of these methods in the following subsections. ## automatic evaluation automatic evaluation was done mainly by the addition of paracrawl data to wmt data from the translation shared task @xcite as an ongoing experiment carried out since the first version of the corpus released in januany 2018 up to the final version until present dated from september 2021. mt evaluation based on sub samples of paracrawl and the addition to europarl @xcite was also explored for an early version but was abandoned by lack of resources and time. ## conclusions and future work we have presented in this paper a summary of the tasks carried out as part o the quality assessment activities of the paracrawl project to evaluate the production of web-crawled parallel corpora for machine translation. we have extensively described and discussed how we implemented different human evaluation tasks based on error annotation, post-editing and searches over the corpora and their results. we have also briefly reported about the extrinsic evaluation through machine translation conducted in parallel with human evaluation. besides describing the methods and experiments, we have discussed their usefulness to meet the goals of the paracrawl project and their limitations. the advantages and disadvantages of these methods are now being discussed in macocu, 10 a similar effort for which quality assessment activities are being planned not only for bilingual corpora but also for monolingual ones. for human evaluation, annotation is probably going to be focused on single issues tasks rather that multiple and hierarchic ones. searches and post-editing are under discussion as well as the suitability for other tasks like direct assessment, ranking and fluency, this last maybe suitable also for monolingual corpora. for extrinsic automatic evaluation, more balanced corpora sizes or not only concatenation of data but also fine tuning is being considered. monolingual training corpus cs-en en-cs de-en en-de fi-en en-fi lv-en en-lv ro-en en-ro