Pretrained Models for Multilingual Federated Learning (2022)

Abstract: Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.

## introduction federated learning (fl) is a machine learning technique that trains a model across multiple distributed clients holding local data samples, without ever storing client data in a central location @xcite . these techniques are appealing for those who wish to learn from data in a privacy-preserving way, without ever transmitting the data off of a client device. fl becomes essential when data is especially sensitive, as is the case at hospitals, legal firms, financial institutions, or in countries that enact legislation concerning data privacy (such as the eu's gdpr or the us's hipaa). fl has been applied to problems in natural language processing (nlp) since its inception, particularly in use of the language modeling task @xcite @xcite @xcite en fr ru zh model en fr ru zh en fr ru zh en fr ru zh zh zh zh zh model ru ru en en en en client 1 client 2 client 3 client 4 en x4 fr x4 ru x4 zh x4 federated learning with iid data federated learning with non-iid data centralized (standard) learning client 1 client 2 client 3 client 4 model figure 1: a depiction of different learning strategies with federated learning (fl) and multilingual data, with 4 clients and 16 instances from en, fr, ru, and zh in this toy example. black lines indicate gradient flow. centralized learning is the standard training method (no privacy), fl with iid data partitions the data into iid data subsets for each client, while fl with non-iid data has the languages separated across clients. and singh, 2020). another large area of fl research is focused on analyzing performance when the data is non identically independently distributed (non-iid). in these cases, many works have shown that fl performance is sub-par with respect to centralized learning methods @xcite @xcite . despite the large amount of research in fl for nlp, how different languages impact the fl training process has yet to be explored @xcite . furthermore, multilingual fl provides an interesting and natural setting to explore non-iid data, of which different languages are an obvious example. in this work, we explore multilingual federated learning across three multilingual language tasks and different stages of model pretraining. our results show that fine-tuning pretrained models with fl can perform similarly to pretrained models finetuned with the standard centralized method (the no privacy setting), despite having completely non-iid language partitioned data. this finding shows that pretrained models provide an effective way for practitioners (and consumers) of multilingual data to gain the privacy benefits of fl at little or no cost to the final task performance. ## background and related work the term federated learning was first proposed in @xcite , who applied the federatedaveraging algorithm to the tasks of language modeling and image classification. since then, much of the theoretical and applied work in fl (e.g. chen et al. (2019b); wu et al. (2020 ) among many others) has considered language modeling as a key task or benchmark. concurrent with the growing interest in federated learning, nlp has rapidly shifted toward the use of pretrained language models (plms) (e.g., @xcite @xcite ). these plms are used for both the core task of next word prediction and as a starting point for learning other downstream nlp tasks. this pretrain-and-fine-tune paradigm has since become ubiquitous in modern nlp and has inspired a large and active area of research in model pretraining. multilingual versions of these pretrained models have since been developed and are often used with transfer learning techniques to increase performance for tasks where data is limited (e.g. mbert from @xcite . the intersection of distributed learning from private data partitions and plms is still a nascent area. several works have explored more efficient methods of federated communication with the purpose of enabling these larger nlp models for production situations @xcite . our work is orthogonal to these (and could be combined in future work), as we explore the effects of multilingual data on plm fl, rather than creating methods to enable their use. other papers focus on the gap between federated learning performance and centralized performance, evaluating on a wide variety of english nlp tasks @xcite @xcite . although they focus on differential privacy (dp) rather than fl, @xcite find that direct plm training is difficult with standard dp methods, but that finetuning plms on english data is possible with private learning techniques. we differ from all these works by studying private learning, specifically fl, for plms in the novel multilingual setting. ## conclusion in this work we provided the first analysis of multilingual language data on federated learning algorithms. we found that fine-tuning a pretrained model with fl methods can yield similar performance to centralized learning, even when clients are partitioned by language (non-iid fl). however, models trained from random initializations still show a large gap between centralized and federated learning. our results suggest that learning on private partitioned data is possible without having to incur a large performance penalty. we hope that these results will aid practitioners in using fl (and also downstream consumers) and inspire the broader community to consider multilingual data in future federated learning research for natural language processing. random initializations, running for twice as many epochs. resulting models appeared to converge by loss but had extremely low bleu scores. thus, we only include pretrained results in