A Closer Look at Transformer Attention for Multilingual Translation (2023)

Abstract: Transformers are the predominant model for machine translation. Recent works also showed that a single Transformer model can be trained to learn translation for multiple different language pairs, achieving promising results. In this work, we investigate how the multilingual Transformer model pays attention for translating different language pairs. We first performed automatic pruning to eliminate a large number of noisy heads and then analyzed the functions and behaviors of the remaining heads in both self-attention and cross-attention. We find that different language pairs, in spite of having different syntax and word orders, tended to share the same heads for the same functions, such as syntax heads and reordering heads. However, the different characteristics of different language pairs clearly caused interference in function heads and affected head accuracies. Additionally, we reveal an interesting behavior of the Transformer cross-attention: the deep-layer cross-attention heads work in a clear cooperative way to learn different options for word reordering, which can be caused by the nature of translation tasks having multiple different gold translations in the target language for the same source sentence.

## introduction for traditional statistical machine translation, such as phrase-based translation @xcite , the translation process is very clear: source phrases are translated into target phrases according to translation rules and then target phrases are reordered to ensure the fluency of the target sentence. however, in state-of-the-art neural translation models @xcite @xcite , how the model learns to translate is substantially less obvious. the behavior of the transformer model @xcite remains particularly hazy, as it contains many different selfand cross-attention heads in different layers. a number of existing studies conducted analyses of functions and behaviors of attention heads in transformer translation models. @xcite found that the transformer attention is noisy, as most of the transformer heads can be pruned away without significant loss in translation quality. they also identified three important functions of self-attention in the transformer encoder, such as heads focusing on syntax. @xcite demonstrated that the crossattention of the transformer model frequently attends to uninformative source words to balance the contribution of source and target context for predicting the next word. @xcite showed that some cross-attention heads learn alignment for the current target word, achieving higher accuracies than cross-attention heads that learn alignment for the next target word. however, these methods only analyzed attention in bilingual models, not for multilingual transformer models. multilingual translation, i.e., training a single transformer to learn translation for multiple different language pairs, has received much attention in recent years and obtained promising results @xcite @xcite . a number of studies investigated how a multilingual transformer learns to translate different language pairs. several of these @xcite @xcite learned language-dependent weight masks to identify language-dependent subnetworks. @xcite trained the multilingual transformer to learn language-specific layers and improved translation quality. @xcite and @xcite assessed how different language pairs share important heads in multilingual transformer models. however, prior work has not yet studied the specific functions and behaviors of different attention heads in multilingual transformer models. in this paper, we investigate functions and behaviors 2 of both self-attention and cross-attention for multilingual translation. we find that different language pairs with different syntax and different word orders tend to share the same heads for the same functions (such as syntax heads and reordering heads), but the different characteristics of different language pairs can clearly cause interference in function heads and affect head accuracies compared to bilingual models. we further obtain an interesting finding about how the transformer learns word reordering: different cross-attention heads in deep layers work in a cooperative way to learn different options for reordering. this may result from the fact that there are multiple different gold translations (reorderings) in the target language for the same source sentence 3 . ## related work there are a number of studies on analyzing layer representations of different transformer layers. voita et al. (2019a) other prior work analyzed transformer attention to better understand a particular aspect of the translation process. @xcite analyzed transformer attention for negation translation and showed that negation is often rephrased during training, which can make it more difficult for the model to learn a reliable link between 2 following @xcite 's work, we use a weightbased method for analyzing attention head behaviors. it is also possible to use a norm-based method @xcite , which may provide a more detailed interpretation of the inner workings of transformers compared to weight-based methods in some cases. 3 in the training data of translation models, it is rather rare that the same source sentence has multiple different translated target sentences, but it is very common that the same source phrase has multiple different translated target phrases. therefore, translation models are able to learn to translate a source sentence into different target sentences. there is also prior work that studied representation sharing in multilingual translation. @xcite proposed a multiway, multilingual model with language-specific encoders and decoders and showed result quality improvements over models trained on only one language pair. several authors @xcite @xcite considered language-dependent gating and adaptation for layer representations. @xcite proposed parallel encoder and decoder layers with language-dependent weighted layer aggregation. @xcite presented a universal representer to replace both encoder and decoder models to enable parameter sharing between encoder and decoder and they made the representer sensitive for specific languages using language-sensitive embedding, attention, and discriminator. zhu et al. (2020) incorporated a language-aware interlingua into the encoder-decoder architecture, which enables the model to learn a language-independent representation from the semantic spaces of different languages, while still allowing for languagespecific specialization of a particular language pair. additionally, @xcite showed that controlling the proportion of each language pair in the training data can balance the amount of interference between languages in multilingual models. @xcite developed a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference with a novel efficient training recipe. @xcite investigated how to utilize intra-distillation to learn more language-specific parameters and then showed the importance of these language-specific parameters. however, these methods did not investigate the head functions in multilingual models. ## conclusion this paper analyzes attention head functions and behaviors in multilingual transformer translation models. we find that different language pairs, in spite of having different syntax and word orders, tend to share the same heads for the same functions, such as syntax heads and reordering heads. however, the different characteristics of different language pairs clearly cause interference in function heads and affect head accuracies, which can be an important disadvantage of multilingual models compared to bilingual models. additionally, we reveal an interesting behavior of the transformer cross-attention: the deep-layer crossattention heads work in a cooperative way to learn different options for word reordering, which can be caused by the nature of translation tasks having multiple different gold translations (reorderings) in the target language for one source sentence.