W illiams College’s Submission for the C oco4 MT 2023 Shared Task (2023)

Abstract: Professional translation is expensive. As a consequence, when developing a translation system in the absence of a pre-existing parallel corpus, it is important to strategically choose sentences to have professionally translated for the training corpus. In our contribution to the Coco4MT 2023 Shared Task, we explore how sentence embeddings can be leveraged to choose an impactful set of sentences to translate. Based on six language pairs of the JHU Bible corpus, we demonstrate that a technique based on SimCSE embeddings outperforms a competitive suite of baselines.

## introduction it has become increasingly possible to train decent translation models with small, highquality parallel corpora. for instance, @xcite recently showed that just 6000 professionally-translated sentences made a big impact on the quality of trained translation models for 39 low-resource languages. this raises a research question. suppose we want to develop a model that translates between a high-resource language and a low-resource language (or possibly a specialized domain of a moderately-resourced language). how should one select a "seed" dataset to have professionally translated from the high-resource language into the low-resource language? this is the subject of the coco4mt 2023 shared task. for our contribution to this shared task, we focus on model-agnostic approaches. in contrast to approaches that leverage model uncertainty to select sentences to professionally translate @xcite , model-agnostic approaches @xcite select a training corpus based exclusively on the distribution and content of sentences in the high-resource language. while ignoring model uncertainty may result in lower-quality data selection for a particular model, it has the potential advantage of broader applicability, as the data selection is not tied to a specific model architecture. ## task description define a translation model t as a function that maps source-language documents to targetlanguage documents. each translation pair (x, t(x)) (where x is a source-language document) is assumed to have a non-negative real-valued quality q(x, t(x)). given a distribution p x over source-language documents, the quality of translation model t is the expected translation quality: a parallel corpus is a set of pairs (x, y), where y is a target-language translation of sourcelanguage document x. a trainer τ takes a parallel corpus as its input, and outputs (possibly nondeterministically) a translation model t. the coco4mt 2023 shared task is about optimizing parallel corpus construction for training translation models. we have access to a monolingual corpus of documents @xmath0, ..., x n } in a high-resource language, assumed to be drawn i.i.d. from document distribution p x . we also have access to a cost function c : @xmath1. for monolingual corpus x and a subset i ⊆ {1, ..., n} of selected document ids, let z x,@xmath2, y i ) | @xmath3.e. y i is the professional translation of document x i ) from commissioning translations for the documents associated with the selected ids. the cost of building parallel corpus z x,i is therefore: the goal of the task is to construct a parallel corpus z x,i that produces a translation model of maximal expected quality, subject to the constraint that the construction cost c(z x,i ) is less than a specified budget b. in other words, we want to compute: the official coco4mt 2023 shared task uses a uniform cost function, i.e. c(x i ) = 1 for all x @xmath4. in other words, all documents have the same translation cost. the budget @xmath5.2 * |x| is 20% of the documents in monolingual corpus x. we will also experiment with a token-based translation cost, i.e. where c(x i ) equals the number of tokens in document x i foot_0 . the corresponding budget b will be 20% of the tokens in corpus x. ## our approach inspired by the delfy algorithm, we investigated other ways to balance representativeness with redundancy. specifically, we focused on using document embeddings to select the documents to translate. document embeddings seek to cluster documents based on lexical and semantic similarity. for instance, if we apply the popular document embedding model simcse @xcite to the jhu bible corpus @xcite , we obtain clusters like the ones shown in figure 1 . for instance, the most similar sentence to my haters have hunted me like a bird without cause is i hate them with a perfect hatred; they have become my enemies. the intuition behind our approach was to balance representativeness and redundancy by choosing representatives from each cluster of embeddings. we embedded each document using simcse @xcite , then determined the nearest neighbor (based on cosine similarity) of each document. in other words, we compiled the following set of document pairs: based on this ranking, we selected documents from the high-resource corpus until the budget was exhausted. when using a uniform translation cost (i.e. when all documents had the same translation cost), ties were broken based on sentence length foot_1 . when using a token-based translation cost, ties were broken randomly. we experimented with alternative approaches to selecting cluster representatives (e.g. centroids of k-nearest neighbor clustering), but these all underperformed the simple method described above. for each language pair, we show the difference between the bleu score of a model trained on documents selected by a given technique, versus a corresponding model trained on a randomly selected corpus. for each experiment, the budget is 20% of the high-resource training corpus (counted in terms of tokens and documents, respectively). ## experiments we used the experimental setup provided by the coco4lm 2023 shared task. the training corpus consisted of roughly 22k english sentences from the jhu bible corpus @xcite , along with translations in six other languages: german (de), indonesian (id), korean (ko), french (fr), gujarati (gu), burmese (my). additionally, there was a development corpus of 3919 sentences and a test corpus of 8708 sentences (all from the biblical domain). since none of the evaluated techniques used the non-english data for training foot_2 , we evaluated on all six en-x language pairs (en-fr, en-gu, en-my, en-de, en-id, en-ko). for each corpus selection technique, we fine-tuned the mbart-50 model @xcite on the documents foot_3 identified by that strategy and evaluated using bleu @xcite . we used the implementation and default parameters of mbart-50 provided by huggingface. results are shown in figure 2 . our sentence embedding approach performed consistently across the two translation costs and six language pairs (with one exception: en-fr translation for the token-based translation cost). surprisingly, the delfy algorithm consistently underperformed the random baseline, despite successes in other studies @xcite . perhaps this was due to the narrow domain (biblical) or the limited size of the training corpus. ## conclusion, limitations, and future work sentence embeddings show potential as an instrument for selecting a good proxy dataset for a translation domain. however, it is important to acknowledge the limitations of this pilot study.