Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks (2021)

Abstract: This paper studies the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks. Prior research has found that only a few attention heads are important in each mono-lingual Natural Language Processing (NLP) task and pruning the remaining heads leads to comparable or improved performance of the model. However, the impact of pruning attention heads is not yet clear in cross-lingual and multi-lingual tasks. Through extensive experiments, we show that (1) pruning a number of attention heads in a multi-lingual Transformer-based model has, in general, positive effects on its performance in cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned can be ranked using gradients and identified with a few trial experiments. Our experiments focus on sequence labeling tasks, with potential applicability on other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and XLM-R, on three tasks across 9 languages each. We also discuss the validity of our findings and their extensibility to truly resource-scarce languages and other task settings.

## introduction prior research on mono-lingual transformer-based @xcite models reveals that a subset of their attention heads makes key contributions to each task, and the models perform comparably well @xcite or even better @xcite with the remaining heads pruned 1 . while multi-lingual transformer-based models, e.g. mbert @xcite and xlm-r @xcite , are widely applied in cross-lingual and multi-lingual nlp tasks foot_0 @xcite @xcite , no attempt has been made to extend the findings on the aforementioned mono-lingual research to this context. in this paper, we explore the roles of attention heads in cross-lingual and multi-lingual tasks for two reasons. first, better understanding and interpretability of transformerbased models leads to efficient model designs and parameter tuning. second, head-pruning makes transformer-based models more applicable to truly resource-scarce languages if it does not negatively affect model performance significantly. the biggest challenge we face when studying the roles of attention heads in cross-lingual and multi-lingual tasks is locating the heads to prune. existing research has shown that each attention head is specialized to extract a collection of linguistic features, e.g., the middle layers of bert mainly extract syntactic features @xcite and the fourth head on the fifth layer of bert greatly contributes to the coreference resolution task @xcite . thus, we hypothesize that important feature extractors for a task should be shared across languages and the remaining heads can be pruned. we evaluate two approaches used to rank attention heads, the first of which is layer-wise relevance propagation @xcite ). @xcite interpreted the adaptation of lrp in transformerbased models on machine translation. motivated by @xcite and @xcite , we design a second ranking method based on gradients since the gradients on each attention head reflect its contribution to the predictions. we study the effects of pruning attention heads on three sequence labeling tasks, namely part-ofspeech tagging (pos), named entity recognition (ner), and slot filling (sf). we focus on sequence labeling tasks since they are more difficult to annotate than document-or sentence-level classification datasets and require more treatment in crosslingual and multi-lingual research. we choose pos and ner datasets in 9 languages, where english (en), chinese (zh), and arabic (ar) are candidate source languages. the multiatis++ corpus @xcite is used in the sf evaluations with en as the source language. we do not include syntactic chunking and semantic role labeling tasks due to lack of availability of manually written and annotated corpora. in these experiments, we rank attention heads based only on the source language(s) to ensure the extensibility of the learned knowledge to cross-lingual tasks and resource-poor languages. in our preliminary experiments comparing the gradient-based method and lrp, the average f1 score improvements on ner with mbert are 0.69 (cross-lingual) and 0.24 (multi-lingual) for lrp and 0.81 (cross-lingual) and 0.31 (multi-lingual) for the gradient-based method, though both methods rank attention heads similarly. thus we choose the gradient-based method to rank attention heads in all our experiments. our evaluations confirm that only a subset of attention heads in each transformer-based model makes key contributions to each cross-lingual or multi-lingual task and that these heads are shared across languages. performance of models generally drop when the highest-ranked or randomly selected heads are pruned, validating the head rankings generated by our gradient-based method. we also observe performance improvements on tasks with multiple source languages by pruning attention heads. our findings potentially apply to truly resource-scarce languages since we show that the models perform better with attention heads pruned when fewer training instances are available in the target languages. the contributions of this paper are three-fold: ## datasets we use human-written and manually annotated datasets in experiments to avoid noise from machine translation and automatic label projection. we choose pos and ner datasets in 9 languages, namely en, zh, ar, hebrew (he), japanese (ja), persian (fa), german (de), dutch (nl), and urdu (ur). as for ner, we use nl, en, and de datasets from @xcite @xcite . additionally, we use the people's daily dataset foot_2 , iob2corpus foot_3 , aqmar @xcite , armanperosnercorpus @xcite , mk-pucit @xcite , and a news-based ner dataset @xcite for the languages cn, ja, ar, fa, ur, and he, respectively. since the ner datasets are individually constructed in each language, their label sets do not fully agree. as there are four ne types (per, org, loc, misc) in the three sourcelanguage datasets, we merge other ne types into the misc class to allow cross-lingual evaluations. we evaluate sf models on multiatis++ with en as the source language and spanish (es), portuguese (pt), de, french (fr), zh, ja, hindi (hi), and turkish (tr) as target languages. there are 71 slot types in the tr dataset, 75 in the hi dataset, and 84 in the other datasets. we do not use the intent labels in our evaluations since we study only sequence labeling tasks. thus our results are not directly comparable with @xcite . ## methodology here, we introduce the gradient-based method we use in the experiments to rank the attention heads. feng et al. (2018) claim that gradients measure the importance of features to predictions. since each head functions similarly as a standalone feature extractor in a transformer-based model, we use gradients to approximate the importance of the feature set extracted by each head and rank the heads accordingly. @xcite determine importance of heads with accumulated gradients at each head in a training epoch. different from their approach, we fine-tune the model on the training set and rank the heads using gradients on the development set to ensure that the head importance rankings are not significantly correlated with the training instances in one source language. specifically, our method generates head rankings for each language in three steps: (1) we fine-tune a transformer-based model on a mono-lingual task for three epochs. we show spearman's rank correlation coefficients (spearman's ρ) between head rankings of each language pair generated by our method on pos, ner, and sf in figure 1 . the highestranked heads largely overlap in all three tasks, while the rankings of unimportant heads vary more in mbert than xlm-r. ## experiments and analysis and xlm-r. specifically, we use the pre-trained bert-base-multilingual-cased and xlm-roberta-base models for their comparable model sizes. the models are fine-tuned for 3 epochs with a learning rate of 5e-5 in all the experiments. we use the official dataset splits and load training instances with sequential data samplers, so the reported evaluation scores are robust to randomness. ## discussions in this section, we perform case studies to confirm the validity of our head ranking method. we also illustrate the extensibility of the knowledge we learn from the main experiments to a wider range of settings, e.g. when the training dataset is limited in size or constructed over multiple source languages. ## conclusion and future work this paper studied the contributions of attention heads in transformer-based models. past research has shown that in mono-lingual tasks, pruning a large number of attention heads can achieve comparable or higher performance than the full models. however, we were the first to extend these findings to cross-lingual and multi-lingual sequence labeling tasks. using a gradient-based method, we identified the heads to prune and showed that pruning attention heads generally has positive effects on mbert and xlm-r performances. additional case studies empirically demonstrated the validity of our findings and showed further extensibility of them to a wider range of task settings. in addition to better understanding of transformerbased models under cross-and multi-lingual settings, our findings can be applied to existing models to achieve better performance with reduced training data and resource consumption. future work could include improving model interpretability in other cross-lingual and multi-lingual tasks, e.g. xnli @xcite and other passage-level classification tasks.