Efficient Test Time Adapter Ensembling for Low-resource Language Varieties (2021)

Abstract: Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models. Specialized language and task adapters have recently been proposed to facilitate cross-lingual transfer of multilingual pretrained models (Pfeiffer et al., 2020b). However, this approach requires training a separate language adapter for every language one wishes to support, which can be impractical for languages with limited data. An intuitive solution is to use a related language adapter for the new language variety, but we observe that this solution can lead to sub-optimal performance. In this paper, we aim to improve the robustness of language adapters to uncovered languages without training new adapters. We find that ensembling multiple existing language adapters makes the fine-tuned model significantly more robust to other language varieties not included in these adapters. Building upon this observation, we propose Entropy Minimized Ensemble of Adapters (EMEA), a method that optimizes the ensemble weights of the pretrained language adapters for each test sentence by minimizing the entropy of its predictions. Experiments on three diverse groups of language varieties show that our method leads to significant improvements on both named entity recognition and part-of-speech tagging across all languages.

## introduction massively multilingual pretrained models @xcite @xcite combined with cross-lingual transfer now define the state of the art on a variety of nlp tasks @xcite . within this paradigm, multilingual pretrained models are fine-tuned on annotated data of a task in a high-resource language, and transferred to other languages. several recent works propose parameter-efficient fine-tuning methods that insert small adapter modules between the layers of pretrained models @xcite . in this line of work, the pretrained model is usually frozen while only the adapters are fine-tuned for a downstream task, which is conducive to both improving the model's learning ability and compactness with respect to storage on disk or in memory. the adapters can be applied to the cross-lingual transfer setting by training separate language and task adapters @xcite . specifically, @xcite propose to perform zero-shot transfer by first training language-level adapters on monolingual data in different languages and then a task adapter on annotated data in the source language. one drawback of this framework is that a separate language adapter is required for each target language, which is problematic in cases where the data to train these adapters cannot be easily obtained, such as for languages with diverse regional or demographic variations. in fact, certain language varieties are not included in the standard language identification tools, which makes it challenging to reliably obtain even unlabeled data @xcite @xcite . to give just one example, the nordic languages and dialects form a dialect continuum where the total number of language varieties is difficult to estimate, and language varieties constantly emerge in culturally and linguistically diverse areas (svendsen @xcite . although highly related, these language varieties have many systematic differences, which need to be addressed by nlp systems that equitably serve all speakers @xcite . one potential mitigation strategy is directly using an adapter trained on another similar language variety, but we find this sub-optimal in experiments ( § 4). instead, we propose two methods to combine existing language adapters to adapt the model to new language varieties at test time without any training data. first, we find that simply ensembling multiple related language adapters can significantly improve the fine-tuned model, compared with using individual language adapters. second, we propose entropy minimized ensemble of adapters (emea; fig. 1 ), which adapts the ensemble weight of the language adapters for each test instance by minimizing the ensembled model's prediction uncertainty. our experiments show that emea further improves over vanilla ensembling for three groups of uncovered language varieties on both the named entity recognition and part-of-speech tagging tasks. ## adapters for cross-lingual transfer to facilitate our discussion, we briefly summarize the mad-x framework @xcite for zero-shot cross-lingual transfer and identify its shortcomings. the goal of mad-x is to fine-tune a multilingual pretrained model m to m downstream tasks t 1 , t 2 , ..., t m , each of which could be in n languages l 1 , l 2 , ..., l n . to this end, mad-x relies on language and task adapters, which are lightweight functions inserted in the transformer layers in m-usually a feed-forward down-projection followed by an up-projection. specifically, let h be the output of an intermediate layer in m, then l j (h) is the transformation that projects h into the embedding space for language l j , and t i (l j (h)) is the transformation that projects l j (h) into the embedding space for task t i . mad-x trains the adapters t i (•) and l j (•) in two steps. first, for each language l j , its adapter l j is inserted into m to replace the output of each layer h with l j (h). the resulting model, which we denote as l j • m, is trained on unlabeled data in l j using an unsupervised objective such as masked language modeling (mlm; @xcite . second, for each task t i , its adapter t i is inserted on top of a src language adapter l src . the resulting model t i • l src • m is trained on the downstream task t i in language l src . after these two steps, t i • l j • m can be used to perform zero-shot cross-lingual transfer for any task t i and language l j . shortcomings this approach requires a separate adapter for each language one wishes to support. the online database adapterhub foot_0 aims to improve the efficiency and reuse of trained language and task adapters @xcite but currently supports only about 50 languages, and hence most languages are not covered. more importantly, as mentioned in the introduction, certain languages have diverse regional varieties and difficulty of reliably obtaining data for them makes adapter-based approaches especially brittle in these cases. in the following § 3, we propose strategies to improve the robustness of language adapters to uncovered languages without training new adapters. ## generalizing language adapters to related languages we consider the setting where we have a multilingual pretrained model m as well as the pretrained task adapters t 1 , t 2 , ..., t m and language adapters l 1 , l 2 , ..., l n . we want to use m and the existing adapters to support a new language l new , which is not in {l 1 , l 2 , ..., l n } on a given task t without training a new adapter for l new . related language adapters one potential solution is to find the most related language l @xmath0, l 2 , ..., l n } and then use t • l rel • m to do inference in l new . however, this has two disadvantages. first, the task adapter t is only trained in the setting of t • l src • m, so it might not generalize well to the test time setting of t • l rel • m (as shown in § 4.1). second, while the pretrained model m may be relatively robust against distribution shifts @xcite , the specialized language adapters might make the model brittle to language variations because they are trained for specific languages. our experiments in § 4.1 show that this solution indeed leads to poor performance. adapter ensembling as a first solution to this problem, we propose an extremely simple strategy of averaging the transformed outputs of multiple language adapters. specifically, we use both the source language adapter l src and adapters from related languages with similar linguistic properties to the new language. let r be the set of the source and related language adapters. to do inference on a task t for the new language l new , we transform the output h of each layer in m with the language adapters as l avg (h) = 1 r r @xmath1. entropy minimized ensemble of adapters while ensembling is a simple and effective strategy to combine multiple potentially beneficial language adapters, the equal weighing of all language adapters could be sub-optimal for l new ; different language varieties, or even sentences, could benefit from a different weighting of the pretrained language adapters. to further improve adapter ensembling, we generalize l avg (h) into a learnable weighted average: the intuition behind our method is that a good adapter weight α for a test input x should make the model more confident in its prediction for x, that is, it should lead to lower model entropy over the input @xcite . specifically for structured prediction tasks, we want to classify each word x w in a test input x with w words into one of the possible c classes. we consider the entropy: h(x; α) = -w @xmath2; α) log p (c|x w ; α), where p (c|x w ; α) is the prediction of the model t • l wavg (h) • m. since p (c|x w ; α) is a function of the ensemble weights α, we can calculate the gradient of α as g @xmath3; α). to minimize the entropy loss, we can simply do gradient descent steps on each α i using the corresponding gradient g i by α @xmath4, where γ is the learning rate. we can then use the updated α to calculate the final prediction for x. in § 4, we find that a single step of gradient update already leads to better performance than simple ensembling. we can additionally perform multiple steps of gradient descent to obtain a better α at the cost of lower inference speed. alg. 1 shows the pseudo code of our method foot_1 . ## experiments data we focus on zero-shot cross-lingual transfer with english as the source language. we algorithm 1: training with emea input :uniform weights α 0 , weighted adapter output; lwavg(h, α 0 ); test data x; number of update steps t output :prediction ŷ 1 for t in 0, 1, ..., t-1 do calculate entropy ## discussion and conclusion language and dialect cannot be simply categorized into monolithic entities. thus a truly intelligent nlp system should be able to recognize and adapt to personalized language varieties after it is trained and deployed. however, the standard system evaluation is built on the assumption that an nlp model is fixed once it is trained. in this paper, we focus on a specific case of this general problem-we find that specialized language adapters might not be robust to unseen language variations, and that utilization of multiple existing pretrained language adapters alleviates this issue. we hope our findings can inspire future work on models that are robust and adaptive to language variations. we identify two limitations of this paper, which we leave to future work. first, there are limited datasets and benchmarks that evaluate nlp models' ability to generalize to unseen dialect variations. therefore, we only test our method on ner and pos tagging tasks because they have the best language coverage. it is an important future direction to construct high-quality datasets that consider language and dialect variations. second, our method has slower inference speed due to test time computation. future work can aim to reduce the cost by algorithmic or hardware innovations.