Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similarity (2024)

Abstract: Fine-tuning and testing a multilingual large language model is a challenge for low-resource languages (LRLs) since it is an expensive process. While previous studies have predicted the performance of natural language processing (NLP) tasks using machine learning methods, they primarily focus on high-resource languages, overlooking LRLs and shifts across domains. Focusing on LRLs, we investigate three factors (the size of the fine-tuning corpus, domain similarity between fine-tuning and testing corpora, and language similarity between source and target languages), which can potentially impact the model performance by using classical regression models. Our results indicate that domain similarity has the most important impact on predicting the performance of Machine Translation models.

## introduction fine-tuning large language models for natural language processing (nlp) tasks across varying languages, tasks, and domains is a resource-intensive and environmentally harmful process. @xcite . this challenge is especially magnified for low-resource languages (lrls). however, knowing how well a language model performs on a particular language can be useful information, such as improving the accuracy of quality estimation (qe) models @xcite . therefore, there is a need to estimate the performance of these models for lrls without conducting time-consuming and computationally expensive model pre-training and fine-tuning. existing approaches for predicting the performance of models for nlp tasks have shown promise using linear regression and gradientboosting trees @xcite @xcite . these studies have considered data size, typological features, and language similarity as factors contributing to the model performance. however, most of these studies are conducted for high-resource languages (hrls) (e.g., romance and germanic families) thus limiting their applicability to lrls. furthermore, performance drops in nlp tasks have been observed due to domain shift @xcite . however, this factor is not explicitly considered in the existing works that predict the performance of language models. based on the aforementioned limitations in the literature, we considered three factors for the machine translation (mt) performance prediction for lrls using classical regression models. these factors are the size of the fine-tuning corpus, the domain similarity between fine-tuning and testing corpora, and the language similarity between source and target languages. then, we tested the statistical reliability of these regression models and evaluated them based on their prediction accuracy. we selected those with relatively high accuracy for each factor and explored how data partitioning (described in § 2) affects the quality of fit using these preferred models. additionally, we analyzed the importance of the factors by ranking them based on their correlation with the mt performance, their weights in multifactor regression models, and their importance in multifactor models using the random forest regressor. our contributions are as follows: 1) we developed a statistically rigorous method for performance prediction that can be repeated on any combination of lrls, nlp tasks, and llms; 2) we specifically evaluated the impact of various factors on the performance of mt models; 3) we provided domain-specific and language-specific interpretations based on the performance of the regression models. our data is collected from experiments of a prior study @xcite on fine-tuning and testing different corpora and target languages using the multilingual large language model mbart ( language model and evaluation metric mbart is a pre-trained multilingual sequence-tosequence model that is built based on the encoderdecoder transformer architecture @xcite . @xcite has shown that mbart outperforms mt5, another multilingual large language model, especially on lrls. @xcite also suggested the use of spbleu as the evaluation metric for lrls because it is a sentence-level metric that is more robust to the lack of reference translations than corpus-level metrics like bleu. although the size has been found to impact model loss rather than performance, @xcite has demonstrated a negative linear relationship between performance and model loss. languages we covered five south asian languages that are all considered low-resource other than hindi (hi) @xcite , ( corpora we had two fine-tuning corpora for each language. the first fine-tuning corpus is either an administrative (government; si,ta) or a news (pmindia; hi, gu, ka) corpus. the second finetuning corpus is a religious (bible) corpus. due to limited availability, we scrapped the bible corpus for si from a different website foot_1 . for testing corpora, on top of the administrative/ news corpus and religious corpus, we also had an open-domain corpus (flores). also due to limited availability, we used a slightly different corpus, flores-v1 instead of flores-101 for si. for complete details of the corpora, see appendix a.1). we define the experiments where the fine-tuning and testing corpora are from the same domain as in-domain experiments, and out-domain otherwise. to ensure that mt systems perform consistently across corpora of varying sizes, we extracted fixed-size finetuning sets from each corpus as in ## factors and featurization of factors we consider three potential factors that impact the performance score of the mt models: 1) the size of fine-tuning corpus, 2) the domain similarity between fine-tuning and testing corpora, and 3) the language similarity between source and target language. we represent these factors as feature variable(s) used as predictor(s) in the regression models described in the next section. these predictors are: ϕ @xmath0; ϕ @xmath1; ϕ @xmath2. ## methodology in this section, we outline our methodology for modeling and evaluating spbleu predictions using factors mentioned previously, including the exploration of different regression models and their statistical reliability. we also examine the importance of individual features through correlation and feature importance analyses. ## results in this section, we discuss the performance of our regression models based on their rmse in k-fold cross-validation ( ## discussion in this study, we revealed that domain similarity plays an important role in mt. in other words, it significantly affects the performance of mt models. all three feature rankings in § 5.3, as depicted in additionally, the sri lanka constitution states that "sinhala shall be the language of administration and be used for the maintenance of public records and the transaction of all business" for most regions (sri lanka const. art. xxii, § 1). tamil, also an official language of sri lanka, would instead be translated. this difference in language choice could also explain why sinhala outperforms tamil in government-related in-domain documents and why domain similarity is such a powerful predictor in these cases. furthermore, we have detected heteroscedasticity in various models. for jsd, the data points will be heteroscedastic due to the inherent high domain divergence, resulting in experiments with very low spbleu. in contrast, low domain divergence is highly variable, as other factors, such as language and fine-tuning set size, can impact the mt performance. the observation that jsd does not guarantee good model performance in single-factor regression motivates us to consider alternative techniques. the alternative techniques should be more robust or include additional variables to capture variations during low-jsd predictions. additionally, we observed from the boxplots of residuals that residuals are skewed towards low spbleu. ## conclusion in our research, we conducted a comprehensive analysis focusing on three key factors (the size of the fine-tuning corpus, domain similarity between the fine-tuning and testing corpora, and the linguistic similarity between the source and target languages) affecting performance prediction of the mt for five south asian languages. we find that domain similarity exerts the most significant influence on performance, surpassing even the impact of fine-tuning the corpus size. additionally, the background of the corpora and language being translated emerged as a crucial factor in predicting performance and stability. lastly, we verify that our approach to ascertain predictive factors for lrls' performance is statistically rigorous. this approach enables performance prediction without the need for fine-tuning and testing resource-intensive and costly language models, ultimately fostering greater accessibility and equity for lrls.