Findings of the WMT 2023 Shared Task on Parallel Data Curation (2023)

Abstract: Building upon prior WMT shared tasks in document alignment and sentence filtering, we posed the open-ended shared task of finding the best subset of possible training data from a collection of Estonian-Lithuanian web data. Participants could focus on any portion of the end-to-end data curation pipeline, including alignment and filtering. We evaluated results based on downstream machine translation quality. We release processed Common Crawl data, along with various intermediate states from a strong baseline system, which we believe will enable future research on this topic.

## introduction a machine translation (mt) system is only as good as the data it is trained on. however, the academic research community often overlooks the details of this task, using pre-curated corpora. to promote research in this area, this shared task 1 focuses on finding pairs of sentences or documents that are translations of each other based on a collection of web crawled data. mt models are trained by the organizers on the data found by participants, and performance is then judged using automatic metrics. this shared task builds on prior shared tasks on document alignment @xcite and sentence filtering @xcite @xcite . however, this task is intentionally open-ended, and designed to allow participants to improve on various different parts of the data curation pipeline. we chose the estonian-lithuanian language pair for several reasons. the amount of data we extracted in that language pair was enough to train a reasonable mt model, while being small enough that the task was still accessible to academic participants with limited hardware resources. we avoided english, as many toolkits are developed/optimized 1 http://www2.statmt.org/wmt23/data-task.html on english data, and results on english may not generalize well. and finally, we avoided languages which were closely related, as this could favor methods which do not generalize well. to lower the barrier to entry and allow participants to focus their research and compute resources, we release intermediate stages of a strong baseline data curation system. we encourage future work to build upon resources provided in this shared task. this paper gives an overview of the task, presents its results, and provides some analysis. ## related work parallel data has been required for training machine translation systems ever since the field transitioned to statistical machine translation @xcite . to train that first statistical system, brown et al. aligned english-french sentences from the proceedings of the canadian parliament, often referred to as hansards, using a very simple system to segment each side into sentences and then align them using only sentence length @xcite . the field of parallel data curation has come a long way since then, with modern methods extracting billions of sentence pairs in hundreds of languages, as opposed to the few million enabled by hansards. currently, there are two main approaches to parallel data curation: (1) document and sentence alignment, and (2) comparable corpora methods. ## shared task definition this shared task presented the open-ended problem of finding the best possible subset of aligned sentence pairs from unaligned documents sourced from the internet. participants were evaluated on downstream machine translation system performance. parallel data curation from web can be computationally demanding due to the sheer scale of webcrawled data. for this reason, in addition to our documents, we also released pre-computed intermediate steps from a baseline, so participants can choose to focus on one aspect of the task (e.g. sentence filtering.) for this shared task, the organizers provided: participants were allowed to use only pre-trained models and datasets publicly released with a research-friendly license on or before may 1, 2023. ## evaluation we evaluated submissions by using the curated data to train machine translation systems. for preprocessing, we split sentences into subwords by applying byte-pair encoding (bpe) @xcite using 32,000 merge operations. the bpe vocabulary is learned jointly for the source and target language. we apply a minimum vocabulary frequency of 100 per language. we use sockeye @xcite to train transformer @xcite translation models with 512 hidden units, 8 attention heads, 6 layers and feed-forward layers of size 2048. for training we use an effective batch size of 400k target tokens. we use 4096 target tokens per gpu, and gradient accumulation to obtain 400k target tokens regardless of the number of gpus. we use the adam optimizer (kingma and ba, 2014) with β 1 = 0.9 and β 1 = 0.98, an initial learning rate of 0.06325, a linear warmup for 4000 updates and an inverse square root learning rate decay. checkpoints are written every 500 updates and training is stopped once validation perplexity does not improve for 12 checkpoints. the checkpoint with the lowest validation perplexity is used as the final checkpoint. all systems are trained on nodes with 8 v100 gpus. we use bleu @xcite and chrf @xcite as quality metrics. evaluation metrics are computed using sacrebleu @xcite . we considered data from four domains for evaluation: emea, foot_3 eubookshop, foot_4 europarl, foot_5 jrc-acquis, 7 and euconst. 8 all data is released by opus @xcite . from each domain, we created a dev, test, and held-out-test set. we use up to 10,000 lines for each. if less data is available, it is split between the three sets. we also kept euconst as a held-out domain. ## systems we report the results of four different systems: the baseline, two participant systems, and a contrastive system. ## results we see that the contrastive margin score system outperforms the naive top-1 cosine baseline. this confirms the finding of @xcite that margin scoring outperforms raw cosine similarity. the contrastive margin score system also outperforms the participant submissions that directly build and improve upon the naive top-1 cosine baseline. data filtering and alignment tend to be complimentary, so the filtering methods proposed by the participants would likely improve upon the contrastive margin score system if they were applied on top of it. ## conclusion while data curation is the first step in the training of any mt (or machine learning) model, this tends to be a less-published-upon topic in academic research. in this shared task, we have released the processed webcrawled data, and a baseline system with intermediate outputs. we hope this task lowers the barrier of entry and allow participants to focus on any aspect of the data curation pipeline (document alignment, sentence alignment, filtering, etc.) we have trained and evaluated mt systems on the datasets curated by participating teams. we have presented results for two participant submissions, in addition to two more systems built by the shared task organizers. we hope this work serves as a building block for future research on this topic.