Evaluating Sentence Alignment Methods in a Low-Resource Setting: An E nglish- Y orù B á Study Case (2023)

Abstract: Parallel corpora are still crucial to train effective Machine Translation systems. This is even more true for low-resource language pairs, for which Neural Machine Translation has been shown to be less robust to domain mismatch and noise. Due to time and resource constraints, parallel corpora are mostly created with sentence alignment methods which automatically infer alignments. Recent work focused on state-of-the-art pre-trained sentence embeddings-based methods which are available only for a tiny fraction of the world’s languages. In this paper, we evaluate the performance of four widely used algorithms on the low-resource English-Yorùbá language pair against a multidomain benchmark parallel corpus on two experiments involving 1-to-1 alignments with and without reordering. We find that, at least for this language pair, earlier and simpler methods are more suited to the task, all the while not requiring additional data or resources. We also report that the methods we evaluated perform differently across distinct domains, thus indicating that some approach may be better for a specific domain or textual structure.

## introduction parallel corpora are vital training data for machine translation (mt) systems, especially for low-resource languages where data is scarce @xcite . while unsupervised methods trained only on monolingual data have been proposed for neural mt, they are still sensitive to noise and domain mismatch @xcite , and are outperformed by supervised and semi-supervised systems trained on relatively small parallel corpora @xcite . collecting and curating data for the creation of a parallel corpus manually is a costly and time consuming task that requires expertise in the languages involved. it is even more difficult for low to no-resource languages, for which the number of speakers and research may be lower. thus, today parallel corpora are mostly created by employing automatic methods for sentence alignment. sentence alignment is the task of taking parallel documents split into sentences and finding a bipartite graph which matches minimal groups of sentences that are translation of each other @xcite . in other words, to find target sentences with the same meaning to that of the source segments in multilingual texts @xcite . several approaches have been proposed, from simple length-based algorithms @xcite , to more complex methods employing multilingual sentence embeddings @xcite . our work evaluates four commonly used sentence alignment methods using menyo20k @xcite , a high-quality benchmark english-yorùbá 1 parallel corpus, as reference. we experiment with 1-to-1 alignments with and without reordering. our results show that, at least for this language pair, earlier, simpler systems may be more suited, as they perform better and do not require other data than the documents to be aligned, allowing them to be employed even when no other text or knowledge of the languages is available. moreover, we leverage the domain annotation of the menyo20k corpus to observe that the alignment methods in our evaluation perform differently across various domains. this indicates that some approach may be better suited to a specific domain or textual structure. after this introduction, we report on some recent work aimed at evaluating and improving sentence alignment for low-resource language pairs in section 2. then, in section 3 we describe briefly the alignment methods in our evaluation, which methodology is outlined in section 4. section 5 reports the results of our experiments and section 6 draws some conclusions. ## related work some recent work deals with sentence alignment in a low-resource setting, focusing on evaluating and improving modern sentence embedding-based methods. tien et al. (2021) finds that vecaling has several limitations: it performs poorly in aligning sentences which are located far apart in source and target documents, or it may align sentences which are not translations of one another, but have a high similarity score. moreover, the error can be propagated to from one pair to another, since the sentence that should be in one alignment is moved to a further one. they propose a new method that overcomes these limitations by firstly aligning paragraphs and generating candidate sentence pairs only among the aligned paragraph's sentences. they work with the vietnamese-lao low-resource language pair by translating the lao documents to vietnamese, on which laser has been trained. then they find the sentence pairs with cosine similarity weighted for the ratio of text length and then retrieve the target sentence in the original language. they report significant improvements in precision and recall over vecaling on their test set. chimoto and bassett (2022) experiment with laser and labse @xcite to extract bitext for two unseen low-resource african languages, luhya and swahili. they find that both pre-trained models perform poorly at zero-shot alignment on luhya. they thus fine-tune the embeddings on a small set of parallel luhya sentences and report significant gains, with the accuracy of labse increasing from 22% to 53.3%. this is further improved to over 85% by restricting the datastet to sentence embedding pairs with cosine similarity above 0.7. @xcite evaluates the effectiveness of pre-trained language models such as laser, xlm-r @xcite , and labse on document and sentence alignment in the context of the low-resource languages of sinhala, tamil, and english. they introduce a weighting mechanism based on small-scale bilingual lexicons to improve the semantic similarity measure used by the methods they evaluate, thus improving the resulting alignments. contrarily to our work, they find that the multilingual sentence embeddings-based methods significantly outperform the hunalign baseline on their test language pairs. this discrepancy should be investigated in future work. ## baselines the works introduced in section 2 deal mostly with sentence embeddings-based methods. 2 while these methods were shown to be effective and able to generalize to unseen languages in some instances @xcite , this did not hold in other low-resource test cases, for which further work on both the system and the model was needed to reach a satisfactory performance @xcite . moreover, they are still not free from issues in handling sentences which are found far apart in the documents and employ non-optimal scoring functions, such as raw sentence embedding cosine similarity @xcite . lastly, they still require resource-heavy pre-trained models which are available only for a tiny fraction of the world's languages. thus, we also include earlier methods in our evaluation. the earliest widely documented statistical-based methods were explored by @xcite on the assumption that the length of a sentence is highly correlated with the length of its translation. moreover, they concluded that there is a stable ratio between the sentence lengths in any two language. their method assigns a probabilistic score to each correspondence of sentences, based on the scaled difference of lengths, in number of characters, of the two sentences and the variance of this difference. the score is used in a dynamic programming framework to find the maximum likelihood alignment of sentences. they worked on a trilingual corpus of economic reports issued by the union bank of switzerland (ubs) in english, french, and german, and a bilingual sample of 90 million words from proceedings of the parliament of canada in english and french. @xcite describe hunalign, a hybrid method, combining a dictionary with a lengthbased approach. hunalign starts by producing a crude word to word translation for each word token in the dictionary according to the token with the highest frequency in the target corpus. the pseudo target language is then compared to the actual target text on a sentence to sentence basis with a similarity score based on the number of shared words, which is the heaviest component of the scoring, and the sentence length in characters based on the ratio of longer to shorter. once the similarity matrix is obtained for the relevant pairs, dynamic programming is used to find the optimal alignment with penalties for skipping and coalescing sentences. the algorithm works even in the absence of a dictionary in which case the texts are first aligned with the source text as the crude translation of itself and then a simple dictionary can be bootstrapped by collecting source-target token pairs with an association score higher than 0.5. they mainly experiment on hungarian, but cite also romanian and slovenian, motivated by the need to build parallel corpora for "medium density languages". @xcite presents bleualign, an automatic alignment method based on mt. they propose to use automatically translated text and a measure of the quality of this translation, in this case bleu @xcite , as a similarity score to find reliable alignments to be used as anchor points. @xcite details an iterative approach for bleualign. they build a rough alignment using the gale-church algorithm and then train a first mt system on these aligned data. they then use the generated translations to compute the sentence level bleu score and employ it as a measure of alignment. they work on a corpus of french and german text obtained by ocr from the yearbooks of the swiss alpine club between 1864-1982. they claim the system to be more resilient to noise and fairly language independent, despite depending heavily on the provided translation, and thus on a mt system with reasonable performance for their language pair. this is problematic in resource-poor conditions due to the need for enough data to train the mt system and it is computationally more demanding due to the need of an automatic translation. @xcite presents vecalign. they propose a sentence alignment scoring function based on the similarity of bilingual sentence embeddings, which has been shown to be effective in related tasks such as filtering non-parallel sentences and locating parallel sentences in comparable corpora. moreover, blocks of sentences can be represented as the average of their embeddings, which does not depend on the number of sentences being compared, thus reducing the computational complexity. they use the laser multilingual sentence embeddings @xcite and compute similarity as cosine similarity, normalized with randomly selected embeddings to avoid hubness @xcite , i.e. the tendency of some vectors ("hubs") to appear in the top neighbour lists of many items. to align the text, they start by creating an approximate sentence alignment using the average embeddings of adjacent sentences. then they refine this alignment with the original sentence vectors, limiting the search in a small window around the approximate alignment. they claim state-of-the-art results on the bleualign dataset and on bible test sets @xcite . in this low-resource setting, they work on arabic, turkish, somali, afrikaans, tagalog, and norwegian. all these languages but norwegian appear in the training data for laser, albeit in different sizes. they consider verse-alignments as their gold-standard, for which they report an average improvement of 28 verse-level f 1 score on hunalign in bootstrap mode. as we will show in section 5, this improvement in performance does not hold in our experiments on english-yorùbá. ## methodology the objective of our work is to evaluate the widespread sentence alignment methods briefly described in section 3 in a low-resource setting. to achieve this, we carry out two experiments on menyo20k @xcite , a high quality english-yorùbá multidomain parallel corpus. overall the dataset contains 20.100 sentences gathered from various domains such as news articles, ted talks, movie and radio transcripts, science and technology text, yorùbá proverbs, books, and short articles curated from the web. monolingual text crawled from the web were professionally translated and verified by native speakers. we thus assume the corpus as a a gold-standard for our experiments. for our purposes we concatenate the train, dev, and test splits in which the corpus is divided into one text file containing 1-to-1 alignments. the second experiment, shuffled-order, is similar to natural-order, with the addition of reordering: we artificially shuffle the target side by randomly scrambling the sentences in a window of 3. more precisely, we start from sentences at lines 1 to 3 and we randomly shuffle them in this group; we then move on to sentences at lines 4 to 6, and scramble them as well. we continue in this manner until the end of the document is reached. this is done to avoid creating unrealistic data, since it is not usual for sentences that should be aligned to be very far apart in the translation of same text. we then proceed as in natural-order, by applying the alignment methods and evaluating their outputs against the gold standard. whenever possible, we used the implementations available online foot_2 with the configuration that required the least amount of pre-existing resources or further work, such as fine-tuning. for the gale-church method we employed the implementation provided with bleualign. we use laser @xcite to compute the sentence embeddings for vecalign. while the encoder for yorùbá was provided in the library as part of the laser3 extension (nllb team, 2022), we had to train our own sentencepiece (kudo, 2018) model. 5 hunalign was run without a precompiled dictionary. since no end-to-end iterative implementation of bleualign was found, we applied the method without a reference translation. we also attempted to train a nmt model only on the menyo20k corpus aligned with the gale-church algorithm, as the bleualign authors suggest in their second paper. for this, we trained a standard transformer @xcite using fairseq @xcite with the following parameters: vocabulary size 2000, adam optimizer, dropout 0.1, label smoothing 0.1, max tokens 4096, and optimizing for bleu. after 60 epochs, however, the model failed to reach more than 5 bleu in both translation directions, with its output hallucinated and noisy, and was thus deemed not useful to further alignment steps with bleualign. ## results the upper rows of the table report the results for natural-order, the simple 1-to-1 alignment without reordering. the gale-church baseline perform best in 8 out of 12 domains, with the percentage of correct alignments between 82.95% for the radio domain, and the 100% of udhr. it scores 99.96% on menyo-all. bleualign is the least performing method in 9 out of 12 domains, sharing its only 100% on udhr with all the other methods. it fares particularly badly for the literary domain, getting just 37.87% of alignments correctly for book and 56.07% on proverbs. on menyo-all, it returns 79.16% of correct align- it is apparent that the structured nature of the universal human rights declaration generally favours alignment. conversely, the more fluid nature of proverbs may hamper methods such as hunalign, which rely on lexical information for alignment. this domain, however, seems to be better handled using just length information. the lower half of ## conclusions in this paper we presented an evaluation of four commonly used sentence alignment methods when applied to a low-resource language pair, such as english-yorùbá. while working well for high resource languages and domains, more recent sentence embeddingbased alignment methods do not perform similarly for a low-resource pair such as the one in our study. earlier methods, based on sentence length statis-tics and bootstrapped dictionaries, returned better alignments on the menyo20k corpus. all of these methods, however, do not seem suitable when sentence reordering is involved. some methods appear to perform better for specific domains, as shown by the difference in scores for the literary domain, such as with the book and proverbs splits where text is less structured and translations may not be literal. conversely, all methods return perfect alignments on the highly structured text of the udhr. even without these results, one may argue that simpler methods, which do not require a huge amount of resources, both in term of computation and data, and are mostly language-independent, are better suited to the low-resource setting. bleualign assumes the use of machine translated data, and thus a mt system, which has to be trained to satisfactory quality. this is usually not possible in a lowand no resource settings. vecalign requires multilingual sentence embeddings, in our case laser, which in turn need language specific encoders and a sentencepiece model. in turn, these components need further data than simply the documents to be aligned.