Meeting the Needs of Low-Resource Languages: The Value of Automatic Alignments via Pretrained Models (2023)

Abstract: Large multilingual models have inspired a new class of word alignment methods, which work well for the model’s pretraining languages. However, the languages most in need of automatic alignment are low-resource and, thus, not typically included in the pretraining data. In this work, we ask: How do modern aligners perform on unseen languages, and are they better than traditional methods? We contribute gold-standard alignments for Bribri–Spanish, Guarani–Spanish, Quechua–Spanish, and Shipibo-Konibo–Spanish. With these, we evaluate state-of-the-art aligners with and without model adaptation to the target language. Finally, we also evaluate the resulting alignments extrinsically through two downstream tasks: named entity recognition and part-of-speech tagging. We find that although transformer-based methods generally outperform traditional models, the two classes of approach remain competitive with each other.

## introduction word alignment is a valuable tool for extending the coverage of natural language processing (nlp) applications to low-resource languages through, e.g., statistical machine translation (smt; @xcite or annotation projection @xcite @xcite . the traditional approach for generating alignments has been with statistical methods such as giza++ @xcite and fastalign @xcite , which provide strong alignment quality while remaining quick and lightweight to run. recently, new methods have been proposed which extract alignments from massive pretrained multilingual models, and outperform these longstanding methods @xcite . our code and data can be found at https://github. com/abteen/alignment . cia × × × × × × × × × × × × nisqam película nisqata uraykachirqa hinaspam naciones unidasman paqarintanta aparurqa . la cia descargó la película y la llevó a las naciones unidas al día siguiente . ## related work alignment word alignment is a long studied task, with origins in the ibm models for statistical machine translation @xcite , which are the basis of giza++ @xcite and fastalign @xcite . as these approaches can only generate one-to-many alignments, models are trained in both forward and reverse directions (reversing the role of source and target), and final alignments are created via symmetrization heuristics @xcite ; other approaches explicitly symmetrize during training @xcite . 1 while these models rely on only position and word identity information, subword information can be integrated without requiring costly inference @xcite , leading to better parameter estimation for rare words. alignments can also be extracted from neural translation models @xcite . word alignment also enables annotation projection @xcite which can offer strong performance, particularly for low-resource languages @xcite @xcite @xcite . multilingual transformer models pretrained multilingual models @xcite @xcite have become the de facto standard approach for cross-lingual transfer. in general, these models are an extension of their monolingual variants, created by including data from many languages in their pretraining. they rely on a subword vocabulary @xcite which jointly spans all of the pretraining languages. models are pretrained using a masked language modeling (mlm) objective and a translation language modeling (tlm; conneau and lample, 2019) objective that uses parallel sentences. outside of continued pretraining @xcite , models can be adapted using adapters @xcite or through vocabulary adaptation @xcite . word alignment methods which depend on these models have also been proposed @xcite ; we focus on awesome align @xcite because it outperforms other unsupervised methods. ## experiment 2: extrinsic evaluation we further compare aligner performance extrinsically by evaluating downstream task performance when using a projected training set. we consider two tasks: ner and pos tagging. ## analysis as data for low-resource languages often varies considerably in both amount and length, we consider two additional analysis experiments which control for these factors. we focus solely on quechua, as it has the most parallel data available. results are presented in figure 2 with numerical results in tables c.4 and c.5. ## conclusion in this work, we have investigated the performance of modern word aligners versus classical approaches for languages unseen to pretrained models. while classical methods remain competitive, the lowest aer on average is achieved by modern neural approaches. however, using these models comes with a larger computational cost. therefore, the trade-off between training requirements and overall performance must be considered. if access to computing resources is limited or training time is a factor, classical approaches remain a viable approach which should not be discounted.