Cross-Attention is All You Need: A dapting Pretrained T ransformers for Machine Translation (2021)

Abstract: We study the power of cross-attention in the Transformer architecture within the context of transfer learning for machine translation, and extend the findings of studies into cross-attention when training from scratch. We conduct a series of experiments through fine-tuning a translation model on data where either the source or target language has changed. These experiments reveal that fine-tuning only the cross-attention parameters is nearly as effective as fine-tuning all parameters (i.e., the entire translation model). We provide insights into why this is the case and observe that limiting fine-tuning in this manner yields cross-lingually aligned embeddings. The implications of this finding for researchers and practitioners include a mitigation of catastrophic forgetting, the potential for zero-shot translation, and the ability to extend machine translation models to several new language pairs with reduced parameter storage overhead.

## introduction the transformer @xcite has become the de facto architecture to use across tasks with sequential data. it has been dominantly used for natural language tasks, and has more recently also pushed the state-of-the-art on vision tasks @xcite . in particular, transfer learning from large pretrained transformer-based language models has been widely adopted to train new models: adapting models such as bert @xcite and @xcite for encoder-only tasks and models such as bart @xcite and mbart @xcite for encoder-decoder tasks like machine translation (mt). this transfer learning is predominantly performed in the form of fine-tuning: using the values of several hundred million parameters from the pretrained model to initialize a model and start training from there. fine-tuning pretrained models often involves updating all parameters of the model without making a distinction between them based on their importance. however, copious recent studies have looked into the relative cruciality of multi-headed self-and cross-attention layers when training an mt model from scratch @xcite @xcite . cross-attention (also known as encoder-decoder attention) layers are more important than self-attention layers in the sense that they result in more degradation in quality when pruned, and hence, are more sensitive to pruning @xcite . also, crossattention cannot be replaced with hard-coded counterparts (e.g., an input-independent gaussian distribution) without significantly hurting the performance, while self-attention can @xcite . with the ubiquity of fine-tuning as a training tool, we find a similar investigation focused on transfer learning missing. in this work, we inspect cross-attention and its importance and capabilities through the lens of transfer learning for mt. at a high level, we look at training a model for a new language pair by transferring from a pretrained mt model built on a different language pair. given that, our study frames and addresses three questions: 1) how powerful is cross-attention alone in terms of adapting to the new language pair while other modules are frozen? 2) how crucial are the cross-attention layers pretrained values with regard to successful adaptation to the new task? and 3) are there any qualitative differences in the learned representations when cross-attention is the only module that gets updated? to answer these questions, we compare multiple strategies of fine-tuning towards a new language pair from a pretrained translation model that shares one language with the new pair. these are depicted in figure 1 : a) ignoring the pretrained parameters and training entirely from randomly initialized parameters (i.e. 'from scratch') b) finetuning all parameters except the embeddings for the language in common, foot_1 (i.e. 'regular' fine-tuning, our upper bound), c) fine-tuning solely the cross-attention layers and new embeddings, and d) fine-tuning only the new embeddings. here, new embeddings refer to randomly initialized embeddings corresponding to the vocabulary of the new language. in figures 1a-1d , we assume the new language pair has a new source language and not a new target language; figure 1e shows an example of target-side transfer. in the experiments that follow we will always train new, randomly initialized embeddings for the vocabulary of the newly introduced language. generally, all other parameters are imported from a previously built translation model and, depending on the experiment, some will remain unchanged and others will be adjusted during training. our experiments and analyses show that finetuning the cross-attention layers while keeping the encoder and decoder fixed results in mt quality that is close to what can be obtained when finetuning all parameters ( §4). evidence also suggests that fine-tuning the previously trained crossattention values is in fact important-if we start with randomly initialized cross-attention parameter values instead of the pretrained ones, we see a quality drop. furthermore, intrinsic analysis of the embeddings learned under the two scenarios reveals that full fine-tuning exhibits different behavior from cross-attention-only fine-tuning. when the encoder and decoder bodies are not fine-tuned, we show that the new language's newly-learned embeddings align with the corresponding embeddings in the pretrained model. that is, when we transfer from fr-en to ro-en for instance, the resulting romanian embeddings are aligned with the french embeddings. however, we do not observe the same effect when fine-tuning the entire body. in §5 we see how such aligned embeddings can be useful. we specifically show they can be used to alleviate forgetting and perform zero-shot translation. finally, from a practical standpoint, our strategy of fine-tuning only cross-attention is also a more lightweight fine-tuning approach @xcite that reduces the storage overhead for extending models to new language pairs: by fine-tuning a subset of parameters, we only need to keep a copy of those instead of a whole-model's worth of values for the new pair. we quantify this by reporting the fraction of parameters that is needed in our case relative to having to store a full new model for each adapted task. our contributions are: 1) we empirically show the competitive performance of exclusively finetuning the cross-attention layers when contrasted with fine-tuning the entire transformer body; 2) we show that when fine-tuning only the cross-attention layers, the new embeddings get aligned with the respective embeddings in the pretrained model. the same effect does not hold when fine-tuning the entire transformer body; 3) we demonstrate effective application of this aligning artifact in mitigating catastrophic forgetting @xcite and zero-shot translation. ## experimental setup in this section, we describe our experiments and the data and model that we use to materialize the analysis outlined in §2.2. ## results and analysis our preliminary empirical results consist of five experiments for each of the child language pairs based on methods described in §3.1: scratch, {src,tgt}, {src,tgt}+body, {src,tgt}+xattn, and {src,tgt}+randxattn. our core results, which rely on transferring from the fr-en parent under each setting, are reported in ## via nearest neighbor retrieval we use the bilingual dictionaries released as a resource in the muse @xcite repository. 8 for instance, to compare the german embeddings from each of the src+body and src+xattn de-en models to the french embeddings learned in the parent model, we use the de-fr dictionary. we filter our learned embeddings (which are, in general, of subwords) to be compatible with the muse vocabulary. of the 8,000 german subwords in the vocabulary, 2,025 are found in muse. for each of these, we find the closest french embedding by cosine similarity; if the resulting (german, french) pair is in muse, we consider this a match. via this method, we find the accuracy of the bilingual lexicon induction through the embeddings of src+xattn model is 55%. however, the accuracy through the embeddings of src+body is much lower at 19.7%. due to only considering the exact matches against the gold dictionary, this is a very strict evaluation. we also manually look at a sample of 40 words from the german set and check for the correctness of retrieved pairs for those using an automatic translator: while src+xattn scores in the range of 80%, src+body scores in the range of 30%. details of this manual inspection are provided in overall, these results confirm that embeddings learned under {src,tgt}+xattn effectively get aligned with corresponding parent embeddings. however, this is not the case with embeddings learned under {src,tgt}+body. this suggests such effect is not the default pattern in translation models, but rather an artifact of the freezing choices made in {src,tgt}+xattn. ## utilities of aligned embeddings we saw how fine-tuning only cross-attention results in cross-lingual embeddings with respect to parent embeddings. that is how cross-attention is able to use the baked-in knowledge in the encoder and decoder without any further updates to them. in this section, we discuss two areas where this can be turned to our advantage: mitigating forgetting and performing zero-shot translation. ## related work studying cross-attention. several recent works consider the importance of self-and cross-attention heads in the transformer architecture @xcite @xcite . the consensus among these works is that crossattention heads are relatively more important than self-attention heads when it comes to introducing restrictions in terms of pruning and hard-coding. ## conclusion we look at how powerful cross-attention can be under constrained transfer learning setups. we empirically show that cross-attention can single-handedly result in comparable performance with fine-tuning the entire transformer body, and it is through no magic: it relies on translation knowledge in the pretrained values to do so and has new embeddings align with corresponding parent language embeddings. we furthermore show that such aligned embeddings can be used towards catastrophic forgetting mitigation and zero-shot transfer. we hope this investigative study encourages more analyses in the same spirit towards more insights into the inner workings of different modules and how they can be put to good use.