Findings of the C o C o4 MT 2023 Shared Task on Corpus Construction for Machine Translation (2023)

Abstract: This paper provides an overview of the first shared task on choosing beneficial instances for machine translation, conducted as part of the CoCo4MT 2023 Workshop at MTSummit. This shared task was motivated by the need to make the data annotation process for machine translation more efficient, particularly for low-resource languages for which collecting human translations may be difficult or expensive. The task involved developing methods for selecting the most beneficial instances for training a machine translation system without access to an existing parallel dataset in the target language, such that the best selected instances can then be manually translated. Two teams participated in the shared task, namely the Williams team and the AST team. Submissions were evaluated by training a machine translation model on each submission’s chosen instances, and comparing their performance with the chRF++ score. The system that ranked first is by the Williams team, that finds representative instances by clustering the training data.

## introduction it is a well-known fact that machine translation (mt) systems, especially those that use deep learning, require massive amounts of data. some of the types of resources available are monolingual, multilingual, translation memories, and lexicons. those types of resources are gener-ally created for formal purposes such as parliamentary proceedings @xcite , particularly when the data is parallel. the quality and abundance of such resources for niche or rare domains such as medicine or science is limited, meaning that focused annotation efforts are required when an mt system for such domains needs to be developed. additionally, corpora for lowresource languages, languages with less digital resources available, tends to be less abundant and of lower quality. while mt systems developed using unsupervised methods and monolingual corpora have been effective to a great extent @xcite , parallel data is still crucial, particularly in the case of low-resource languages, as shown by @xcite . however, collection or annotation of parallel data is constrained by access to bilingual translators, who may be rare or highly expensive. therefore, making the data annotation process cost effective by ensuring that the translated instances are of high quality and will lead to high-performing mt systems when used for training. for maximum value, it is desirable to have access to this information before a dataset in the target language is actually constructed. that is, if the annotation budget only permits a limited number of sentences to be translated, but there is a large number of source language sentences, it is optimal to choose sentences for human translation that are expected to be highly beneficial. towards making the annotation process more efficient, in this shared task, we solicit methods for the identification of such beneficial instances effectively without requiring training data in the target language foot_0 . we provide multi-way parallel data from several high-resource languages such as english and german, which can be used to identify instances that are helpful for model training, such as by observing training dynamics @xcite . participants are required to submit the english sentences corresponding to instances chosen by their algorithms as the most beneficial. notably, this task does not necessarily require training mt models -simple heuristics that can indicate the quality of an instance can also be submitted. the performance of all submissions, including the baselines, are evaluated by training an mt model (specifically, mbart) on the selected instances. we use the chrf++ metric @xcite to compare all systems. the shared task officially began on may 19, 2023 with the release of all training data. baselines were then added on june 6, 2023 foot_1 . interested participants were asked to officially register for the shared task through a google forms submission, on which four teams registered. the participation phase concluded on july 21, 2023, until which date submissions could be made by sending text files with the chosen instances to the official coco4mt 2023 email address. of the four teams that registered, only two teams made a submission before the conclusion of the shared task. both teams described their methods and shared an open-source implementation through a system description paper. ## data all data used for model training, evaluation and instance selection is sourced from the johns hopkins university bible corpus @xcite . this is a multi-way parallel corpus containing verses from the christian bible translated into more than 1600 languages. languages: as outlined above, we provide data for a set of "high-resource" languages, which are intended to be used for developing systems to select beneficial instances. for this purpose, we choose the languages english, german, indonesian, and korean. we also provide data in the "low-resource" languages of gujarati, french and burmese for participants to evaluate the performance of their methods. this setting can be considered to be a simulated low-resource setting, since for the purpose of this dataset, all languages are multi-way parallel. finally, we evaluate all submissions on the surprise languages of vietnamese, lithuanian and kazakh, not revealed to the participants until the conclusion of the shared task. the data is in the form of source-target translation pairs, with the source language always being english. size and splits: the multi-way parallel section of the corpus for our languages of interest consists of 34831 sentences. from this, we create training, validation and test sets of sizes 22204, 3919, and 8708 respectively by randomly sampling the original data. ## evaluation submission format: participants were asked to submit indices of the top 20% of the training data (or 4440 sentences), corresponding to the best instances chosen by their systems. we then extract the source and target language sentences corresponding to the indices to prepare data files for mt models. model: we evaluate submissions by finetuning the mbart model @xcite on the chosen instances. mbart is a multilingual denoising autoencoder trained on data from 25 languages, extracted from common crawl @xcite . we use the mbart-large-cc25 checkpoint from facebook, which contains all 10 of our languages of interest in its pretraining data. training: we use the implementation and the default hyperparameters of mbart-large from the huggingface hub @xcite . all data is tokenized with the corresponding mbartcc-25 sentence-piece tokenizer, and any empty lines on the source side are filtered out prior to training. we train each model for 20 epochs on a single nvidia v100 gpu, and use earlystopping based on validation set performance. we train five random runs of each model, and report the averaged score across all runs. baselines: we develop two baselines for comparing the submissions to, namely random and max. the random baseline randomly samples 20% of all instances from the training set. the max baseline sorts the english sentences in descending order by number of tokens, and selects the top 20%. metrics: all systems are evaluated with the chrf++ score @xcite , computed using the sacrebleu toolkit @xcite . ## submissions two teams participated in the coco4mt 2023 shared task, under the team names williams and ast. we describe their submissions below, and further details can be found in the system description papers attached to the proceedings. ## results we first report the performance of all models on all languages in we highlight the fact that all three non-random methods described here are model-agnostic methods, that can identify instances with just access to parallel data and sentence embedding methods. the simple heuristic of choosing the longest sentences holds up well in comparison to more nuanced methods, even outperforming the others for lithuanian. we leave it to future work to explore more advanced heuristics as well as develop model-specific methods to choose beneficial instances for machine translation. ## conclusion in this overview paper, we presented the results of the first coco4mt 2023 shared task. the goal of the task was to discover methods to improve cost-efficiency of the machine translation annotation process, by identifying beneficial instances even without an existing parallel dataset. participants were given access to data from four languages from the jhu bible corpus to develop their algorithms, and three more languages to evaluate their systems. the task received two submissions, which were evaluated on three surprise or test languages. the winner of the shared task is the submission by team williams, which clusters all training set instances, and selects representative examples while minimizing redundancies. we hope that the findings of this task will spur more research on improving annotation efficiency, particularly for low-resource languages.