O ne A ligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval (2022)

Abstract: Aligning parallel sentences in multilingual corpora is essential to curating data for downstream applications such as Machine Translation. In this work, we present OneAligner, an alignment model specially designed for sentence retrieval tasks. This model is able to train on only one language pair and transfers, in a cross-lingual fashion, to low-resource language pairs with negligible degradation in performance. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result on the Tateoba dataset, outperforming an equally-sized previous model by 8.0 points in accuracy while using less than 0.6% of their parallel data. When finetuned on a single rich-resource language pair, be it English-centered or not, our model is able to match the performance of the ones finetuned on all language pairs under the same data budget with less than 2.0 points decrease in accuracy. Furthermore, with the same setup, scaling up the number of rich-resource language pairs monotonically improves the performance, reaching a minimum of 0.4 points discrepancy in accuracy, making it less mandatory to collect any low-resource parallel data. Finally, we conclude through empirical results and analyses that the performance of the sentence alignment task depends mostly on the monolingual and parallel data size, up to a certain size threshold, rather than on what language pairs are used for training or evaluation.

## introduction cross-lingual sentence retrieval aims at aligning parallel sentence pairs that are translations of each other from unlabeled multilingual documents. such mined data can be used in multiple downstream applications such as machine translation and cross-lingual word sense disambiguation @xcite @xcite . even under a half-automated setting with human-in-the-loop, a faithful aligner can help narrow down the candidate pool so that humans do not need to deal with an enormous search space such as cross-lingual web-document pairs @xcite or the entire internet. a retrieval model has also been used to filter existing parallel corpora to improve their quality @xcite or to perform quality estimation @xcite where the reference translations are not available. for sentence retrieval tasks, a majority of recent work is either completely unsupervised @xcite @xcite or leverages all parallel data available @xcite , sometimes to the extent of 879 language pairs @xcite . the unsupervised approach has the benefit of not collecting any parallel data; yet it usually achieves relatively low accuracies on standard benchmark datasets such as tatoeba @xcite , which evaluates on 36 language pairs including multiple low-resource ones. the supervised approach, on the other hand, assumes data access to a plethora of low-resource language pairs, which by definition is difficult to acquire and to ensure their quality. this all-or-nothing choice between the unsupervised and supervised approaches leaves a significant gap on whether zero-shot crosslingual transfer works for such tasks. our work aims to shed light on a recipe of how to distribute the efforts for cross-lingual parallel data collection: (1) how much monolingual data is enough for each language? (2) how many finetuning language pairs are enough? (3) is it necessary to collect low-resource language pairs? (4) to what extent does the amount of parallel data matter? (5) should these language pairs be centered around english? to have a strong enough model to perform analyses that address the above questions, we propose onealigner, 1 a classifier that is able to align crosslingual sentences by training on parallel examples of only one language pair. onealigner is built on top of xlm-roberta (xlm-r) @xcite with its architecture tailored to the alignment task: the model leverages a supervised version of bert-score @xcite to compute semantic similarity and builds a normalization layer into its architecture to counteract the popular sentence effect, where some sentences in the source language tend to have a high similarity score with any sentence in the target language. though not our main contribution, these additions lead to the state-of-the-art accuracy 94.9 foot_1 on the tatoeba dataset when trained on all language pairs from opus-100 @xcite , outperforming models that are trained with 180 times more parallel examples @xcite by 8.0 points. when trained on any single rich-resource language pair, this model is able to match the performance of a model (within a 2.0 gap in accuracy) trained on all language pairs under the same data budget. to further close the already-narrow gap between using one language pair and all pairs while adhering to the rich-resource-only constraint, we scale up the number of language pairs with the top-k rich-resource ones, reaching a 94.0 accuracy on tatoeba, only 0.4 off as compared to training on all language pairs under the same data budget. we also explore either training or evaluating on language pairs that are not centered around english. we find that whether to train on an englishcentered language pair and whether the training pair overlaps with the evaluation pair do not influence model performance -the model will perform similarly as long as two conditions are met: (1) the amount of parallel data size crosses a certain threshold; and (2) the pretraining monolingual data that corresponds to the evaluation languages also surpasses a size threshold. ## model 2.1 base model to align sentences in different languages, it is beneficial to start with a model that has already learned cross-lingual representations to some extent. our onealigner thus builds on top of xlm-r @xcite , a transformer-based model @xcite pre-trained on the monolingual cc-100 dataset @xcite covering 100 languages. this model obtained state-of-the-art per-formance on cross-lingual classification, sequence labeling, and question answering. ## conclusion we present onealigner, an alignment model tailored to sentence retrieval tasks. we show that this model transfers well under a cross-lingual setting even when trained on a single language pair. through experiments and analyses, our work helps uncover what factors influence sentence alignment performance and identifies monolingual data size, parallel data size, and the number of richresource language pairs as the top priorities to which one should distribute their data collection efforts. though having covered a broad range of languages and settings, this work still leaves many unexplored territories: (1) how do we deal with languages not present in the pretraining phase given that the vocabulary is not constructed toward them? (2) why is the cross-lingual transfer successful in the first place? what has the model learned during finetuning? (3) does onealigner generalize to other retrieval tasks other than cross-lingual sentence alignment? we leave these as future work.