The AST Submission for the C o C o4 MT 2023 Shared Task on Corpus Construction for Low-Resource Machine Translation (2023)

Abstract: We describe the AST submission for the CoCo4MT 2023 shared task. The aim of the task is to identify the best candidates for translation in a source data set with the aim to use the translated parallel data for fine-tuning the mBART-50 model. We experiment with three methods: scoring sentences based on n-gram coverage, using LaBSE to estimate semantic similarity and identify misalignments and mistranslations by comparing machine translated source sentences to corresponding manually translated segments in high-resource languages. We find that we obtain the best results by combining these three methods, using LaBSE and machine translation for filtering, and one of our n-gram scoring approaches for ordering sentences.

## introduction reliable parallel corpora are key to developing useful machine translation (mt) systems. parallel corpora are commonly compiled by aligning corresponding documents in two or more languages on the sentence level, with the aim of pairing semantically equivalent segments in the languages. this can work well when source texts and translations of these texts are available. in cases where they are not available or not abundant, mining comparable corpora or web-scraped texts can be used to produce useful parallel pairs or to augment available parallel data. for some very low-resource (lr) language pairs or domains, neither parallel documents nor comparable corpora may be available, and in order to build an mt system that generates useful translation, creating a minimum set of training data by manually translating them may be necessary. in scenarios where corpus creation for machine translation is carried out by translating sentences from a source language into a low-resource target language or in a specialized domain, it is important to use the annotation budget efficiently in order to come up with sentences that are likely to produce high-quality translations. in the coco4mt 2023 shared task, participants are to come up with ways to identify the best examples to translate from a high-resource (hr) source language, without any existing data in the target language. translations of the source data are provided into a number of hr-languages and these translations can be used to help identify the best candidates for translation into the low-resource target language. a parallel corpus of 22,204 lines in english, german, indonesian and korean are provided. participants can select up to 20% of these lines, with the corresponding pairs used to fine-tune an mbart model @xcite , namely the mbart-50 @xcite . the winner of the shared task is the team whose instances result in the highest scoring model as measured by chrf++ @xcite . parallel corpora compiled by aligning corresponding documents in two or more languages on the sentence level are available for a large number of language pairs, e.g. in the opus collection @xcite . @xcite collect a combination of available parallel corpora and web-scraped material in a number of indian languages. @xcite collect webscraped data, predominantly in the languages of the european union and @xcite aim to collect corpora for under-resourced european languages. for under-resourced languages unsupervised methods exploiting monolingual corpora have been applied @xcite , but have been found to need abundant monolingual data in similar domains, for the performance not to deteriorate @xcite . @xcite present a system for choosing source language instances to annotate for mt. they find cross-lingual commonalities in instances that are useful for mt training and use these to identify instances for training a new language pair. ## system description in selecting the best subset of segments using the hr-language pairs we consider sentence length, data diversity, misalignments and semantic equivalence. our approaches are described in section 3.2. before carrying out our experiments we did some preprocessing of the training, validation and test data sets. ## results and discussion we fine-tuned the mbart-50 model on the different datasets and selected the set of lines that obtained the highest bleu score @xcite for @xmath0. our highest-scoring method uses the chrf++ evaluation of translated sentences as well as a labse threshold score to filter the datasets and remove lines that we deem more likely to be detrimental than others. we then order the remaining lines from highest to lowest scores obtained by running our greedy scoring algorithm and select the top 20% of the original number of lines, resulting in training sets of 4,440 lines for each language pair. this list of lines was submitted as our optimal list of instances for corpus construction. ## conclusion we have described our approach to selecting 20% of the lines in a source language training set, for pairing with translations in other languages in order to obtain the optimal training data set from what is available. we find that our approach, as measured by bleu to evaluate hrlanguage translation models, increases translation quality on the language that is not as well represented in the multilingual language model being fine-tuned. this may indicate that using only a low amount of data to fine-tune a model such as mbart-50 is more effective when the language is not well represented in the training data for the model.