Low-Resource Machine Translation through the Lens of Personalized Federated Learning (2024)

Abstract: We present a new approach called MeritOpt based on the Personalized Federated Learning algorithm MeritFed that can be applied to Natural Language Tasks with heterogeneous data. We evaluate it on the Low-Resource Machine Translation task, using the datasets of South East Asian and Finno-Ugric languages. In addition to its effectiveness, MeritOpt is also highly interpretable, as it can be applied to track the impact of each language used for training. Our analysis reveals that target dataset size affects weight distribution across auxiliary languages, that unrelated languages do not interfere with the training, and auxiliary optimizer parameters have minimal impact. Our approach is easy to apply with a few lines of code, and we provide scripts for reproducing the experiments (https://github.com/VityaVitalich/MeritOpt).

## introduction while 7,000+ languages are currently in use worldwide, most existing natural language processing (nlp) tasks and large language models (llms) cover at most 500 of them @xcite @xcite . many languages still possess low amount of resources, and a lot of nlp tasks for such languages remain unsolved. these facts indicate the difficulty and non-triviality of using llms that typically require large amounts of data. a popular direction of approaching low-resource languages (lrls) is machine translation: automatic translation between most of these low-resource languages to high-resource ones is more economically and socially motivated than developing language-specific systems @xcite . to solve the tasks for lrls, a lot of studies employ the related languages or languages originating from the same geographical and historical background @xcite 1 https://github.com/vityavitalich/meritopt 2024; @xcite . despite the positive effect, it usually requires empirical knowledge, and many guesses and trials of different approaches when choosing the best combination of languages used, the most suitable amount of data, and the best learning strategy @xcite . new approach. to address these issues, we present our approach called meritopt to train llms for the target language while multiple datasets for different languages are available. the key idea behind our method is inspired by @xcite , who focus on a specific (personalized) federated learning formulation @xcite . the method from @xcite is a special case of our method. we emphasize that the idea is borrowed from the federated learning (fl) field, however, no fl itself is applied in the paper. fl focuses on the specific setting of distributed training, when there exist multiple clients with their own (and private) data. in the scenario of @xcite , versions of the federated averaging algorithm are natural choices for solving the problem since collecting raw data from the clients is prohibited due to privacy constraints. in contrast, we do not have clients or distributed systems for the problem we are considering. we focus on exploring the underlying algorithmic techniques in application to heterogeneous datasets rather than the distributed training. we are not restricted by any privacy constraints since the datasets we consider are open. that is, our work is not an fl paper. our approach is also more robust than the existing baselines as it adjusts the impact of each language (aggregation weights) during training without any explicit inductive bias towards language relatedness. in particular, our strategy benefits from the updates from the "important" languages and tolerates the updates from the "not important" ones. this setup is extremely beneficial for the interpretability of the training process. in this study, we primarily focus on low-resource languages. however, our approach can be applied to any similar task (not necessarily in nlp). the main requirement is to possess multiple heterogeneous input datasets, while the goal is to train the model suitable for some target data distribution. therefore, we apply the algorithm to the machine translation task using two datasets: the subset from the large-scale multilingual machine translation shared task (small track #2) @xcite and the subset of sami languages from the multilingual benchmark for finno-ugric languages @xcite . to test the method effectively within our compute budget, we focus our study on scenarios with one target language and the remaining languages as auxiliary languages. our approach can be further applied to the datasets with several target languages and several translation directions. two research questions are addressed in this paper: (i) "can meritopt improve the results of the multilingual or single language baselines using aggregation weights?" and (ii) "how do the target language weights and the weights of related and non-related languages change across training?". the contributions of the paper are as follows: ## related work in this section, we discuss the existing methods for low-resource language nlp tasks, especially for low-resource machine translation (lrmt) @xcite , and also give a brief overview of the existing methods in personalized federated learning, and methods to estimate the impact of auxiliary data. regarding similar approaches, the paper of @xcite also assigns the non-uniform weights for different languages. however, we do not compute any gradient similarity metrics and approximately solve an auxiliary problem to find the aggregation weights. ## experiments in this section, we apply the methodology to learn low-resource languages with the help of related languages. we also discuss the data used, the baselines, and the evaluation metrics. ## results and discussion tables 1 and 2 show that meritopt is indeed helpful during training: our approach achieves better performance for most setups and languages. for javanese and tagalog languages (small and medium) and for sami languages of comparable sizes (south, scolt, and inari). ## conclusion in this paper, we implement the meritopt algorithm from the personalised federated learning to the low-resource machine translation task. we show that it can achieve better results than traditional approaches and requires 2 to 10 times fewer gradient steps than baselines (e.g., 8k vs. 85k, 12k vs. 23k). meritopt also allows us to observe the weight distribution between the target and related languages: javanese benefits more from the indonesian language, while for tagalog, the most important languages are indonesian and malay. different weights for different languages also prevent the model from overfitting: after learning the target language dataset, its weights are dropped down while other weights start growing. another takeaway is about the target dataset size: the bigger the dataset is, the more the algorithm keeps relying on it rather than on the auxiliary languages. this might result in worse model performance and "distract" the model from convergence.