UD apter: Language Adaptation for Truly U niversal D ependency Parsing (2020)

Abstract: Recent advances in multilingual dependency parsing have brought the idea of a truly universal parser closer to reality. However, cross-language interference and restrained model capacity remain major obstacles. To address this, we propose a novel multilingual task adaptation approach based on contextual parameter generation and adapter modules. This approach enables to learn adapters via language embeddings while sharing model parameters across languages. It also allows for an easy but effective integration of existing linguistic typology features into the parsing network. The resulting parser, UDapter, outperforms strong monolingual and multilingual baselines on the majority of both high-resource and low-resource (zero-shot) languages, showing the success of the proposed adaptation approach. Our in-depth analyses show that soft parameter sharing via typological features is key to this success.

## introduction monolingual training of a dependency parser has been successful when relatively large treebanks are available @xcite . however, for many languages, treebanks are either too small or unavailable. therefore, multilingual models leveraging universal dependency annotations @xcite have drawn serious attention @xcite @xcite . multilingual approaches learn generalizations across languages and share information between them, making it possible to parse a target language without supervision in that language. moreover, multilingual models can be faster to train and easier to maintain than a large set of monolingual models. however, scaling a multilingual model over a high number of languages can lead to sub-optimal results, especially if the training languages are typologically diverse. often, multilingual neural models have been found to outperform their monolingual counterparts on low-and zero-resource languages due to positive transfer effects, but underperform for high-resource languages @xcite @xcite , a problem also known as "the curse of multilinguality". generally speaking, a multilingual model without language-specific supervision is likely to suffer from over-generalization and perform poorly on high-resource languages due to limited capacity compared to the monolingual baselines, as verified by our experiments on parsing. in this paper, we strike a good balance between maximum sharing and language-specific capacity in multilingual dependency parsing. inspired by recently introduced parameter sharing techniques @xcite , we propose a new multilingual parser, udapter, that learns to modify its language-specific parameters including the adapter modules, as a function of language embeddings. this allows the model to share parameters across languages, ensuring generalization and transfer ability, but also enables language-specific parameterization in a single multilingual model. furthermore, we propose not to learn language embeddings from scratch, but to leverage a mix of linguistically curated and predicted typological features as obtained from the uriel language typology database @xcite which supports 3718 languages including all languages represented in ud. while the importance of typological features for cross-lingual parsing is known for both non-neural @xcite @xcite and neural approaches @xcite , we are the first to use them effectively as direct input to a neural parser, without manual selection, over a large number of languages in the context of zero-shot parsing where gold pos labels are not given at test time. in our model, typological features are crucial, leading to a substantial las increase on zero-shot languages and no loss on high-resource languages when compared to the language embeddings learned from scratch. we train and test our model on the 13 syntactically diverse high-resource languages that were used by @xcite , and also evaluate it on 30 genuinely low-resource languages. results show that udapter significantly outperforms stateof-the-art monolingual @xcite and multilingual @xcite parsers on most high-resource languages and achieves overall promising improvements on zero-shot languages. contributions we conduct several experiments on a large set of languages and perform thorough analyses of our model. accordingly, we make the following contributions: 1) we apply the idea of adapter tuning @xcite to the task of universal dependency parsing. 2) we combine adapters with the idea of contextual parameter generation @xcite , leading to a novel language adaptation approach with state-of-the art ud parsing results. 3) we provide a simple but effective method for conditioning the language adaptation on existing typological language features, which we show is crucial for zero-shot performance. ## previous work this section presents the background of our approach. multilingual neural networks early models in multilingual neural machine translation (nmt) designed dedicated architectures @xcite whilst subsequent models, from @xcite onward, added a simple language identifier to the models with the same architecture as their monolingual counterparts. more recently, multilingual nmt models have focused on maximizing transfer accuracy for low-resource language pairs, while preserving high-resource language accuracy @xcite @xcite , known as the (positive) transfer -(negative) interference trade-off. another line of work builds massively multilingual pre-trained language mod-els to produce contextual representation to be used in downstream tasks @xcite . as the leading model, multilingual bert (mbert) foot_1 @xcite which is a deep self-attention network, was trained without language-specific signals on the 104 languages with the largest wikipedias. it uses a shared vocabulary of 110k wordpieces @xcite , and has been shown to facilitate cross-lingual transfer in several applications @xcite . concurrently to our work, @xcite have proposed to combine language and task adapters, small bottleneck layers @xcite , to address the capacity issue which limits multilingual pre-trained models for cross-lingual transfer. ## proposed model in this section, we present our truly universal dependency parser, udapter. udapter consists of a biaffine attention layer stacked on top of the pretrained transformer encoder (mbert). this is similar to @xcite , except that our mbert layers are interleaved with special adapter layers inspired by @xcite . while mbert weights are frozen, biaffine attention and adapter layer weights are generated by a contextual parameter generator @xcite that takes a language embedding as input and is updated while training on the treebanks. note that the proposed adaptation approach is not restricted to dependency parsing and is in principle applicable to a range of multilingual nlp tasks. we will now describe the components of our model. ## experiments data and training details for our training languages, we follow @xcite , who selected from ud 2.3 (nivre et al., 2018) 13 treebanks "from different language families, with different morphological complexity, scripts, character set sizes, training sizes, domains, and with good annotation quality" (see codes in for the encoder, we use bert-multilingualcased together with its wordpiece tokenizer. since dependency annotations are between words, we pass the bert output corresponding to the first wordpiece per word to the biaffine parser. we apply the same hyper-parameter settings as @xcite . additionally, we use 256 and 32 for adapter size and language embedding size respectively. in our approach, pre-trained bert weights are frozen, and only adapters and biaffine attention are trained, thus we use the same learning rate for the whole network by applying an inverse square root learning rate decay with linear warmup @xcite . appendix a.1 gives the hyper-parameter details. baselines we compare udapter to the current state of the art in ud parsing: [1] uuparser+bert @xcite , a graph-based blstm parser @xcite using mbert embeddings as additional features. to enable a direct comparison, we also re-train udify on our set of 13 high-resource languages both monolingually (one treebank at a time; monoudify) and multilingually (on the concatenation of languages; multi-udify). finally, we evaluate two variants of our model: 1) adapter-only has only task-specific adapter modules and no languagespecific adaptation, i.e. no contextual parameter generator; and 2) udapter-proxy is trained without typology features: a separate language embedding is learnt from scratch for each in-training language, and for low-resource languages we use one from the same language family, if available, as proxy representation. importantly, all baselines are either trained for a single language, or multilingually without any language-specific adaptation. by comparing udapter to these parsers, we highlight its unique character that enables language specific parameterization by typological features within a multilingual framework for both supervised and zero-shot learning setup. ## analysis in this section, we further analyse udapter to understand its impact on different languages, and the importance of its various components. ## conclusion we have presented udapter, a multilingual dependency parsing model that learns to adapt languagespecific parameters on the basis of adapter modules @xcite and the contextual parameter generation (cpg) method @xcite which is in principle applicable to a range of multilingual nlp tasks. while adapters provide a more general tasklevel adaptation, cpg enables language-specific adaptation, defined as a function of language embeddings projected from linguistically curated typological features. in this way, the model retains high per-language performance in the training data and achieves better zero-shot transfer. udapter, trained on a concatenation of typologically diverse languages @xcite , outperforms strong monolingual and multilingual baselines on the majority of both high-resource and low-resource (zero-shot) languages, which reflects its strong balance between per-language capacity and maximum sharing. finally, the analyses we performed on the underlying characteristics of our model show that typological features are crucial for zero-shot languages.