On the differences between BERT and MT encoder spaces and how to address them in translation tasks (2021)

Abstract: Various studies show that pretrained language models such as BERT cannot straightforwardly replace encoders in neural machine translation despite their enormous success in other tasks. This is even more astonishing considering the similarities between the architectures. This paper sheds some light on the embedding spaces they create, using average cosine similarity, contextuality metrics and measures for representational similarity for comparison, revealing that BERT and NMT encoder representations look significantly different from one another. In order to address this issue, we propose a supervised transformation from one into the other using explicit alignment and fine-tuning. Our results demonstrate the need for such a transformation to improve the applicability of BERT in MT.

## introduction contextualized token representations produced by pretrained language models (lms), in particular bert @xcite , have ushered in a new era, allowing the separation of unsupervised pretraining of powerful representation spaces, from the supervised training of task-specific, comparatively shallow classifiers on top of these representations. bert-based models have consistently shown stateof-the-art performance in a variety of tasks, which is largely attributed to the rich information captured by the representations. these capabilities and its transformer-based architecture suggest that bert could improve neural machine translation (nmt) as well. however, as shown by @xcite , although useful, information encoded by bert is not sufficient by itself for successful mt. the reason for this is still an open question. some of the most widely accepted hypotheses to date argue that either there is a fundamental discrepancy between the masked language modeling training objective of bert compared to the generative, left-to-right nature of the mt objective @xcite ; or that catastrophic forgetting @xcite takes place when learning the mt objective on top of the pretrained lm @xcite . the latter could be caused by the large size of the training data typically used in mt,and by the high capacity decoder network used in mt because to fit the high-capacity model well on massive data requires a huge number of training steps. however, since on the one hand, the left-to-right constraint in mt is potentially more relevant for the decoders than the typically bidirectional encoder that has access to the entire input sequence, and on the other hand, bert and other pre-trained lms have been successfully used for other complex problems with large training data and high capacity classifiers @xcite @xcite , it is reasonable to assume there may be further reasons for these discrepancies. we take a complementary stance and analyze the differences between the representation spaces produced by bert and those produced by the mt objective. we therefore attempt to align these spaces, and investigate whether such an explicit alignment would reshape the bert representation space to enable its use as an nmt encoder. to the best of our knowledge, this is the first study to investigate the intrinsic differences of pretrained lm and mt spaces, as well as the first attempt to explicitly align them. for reproducing our experiments, we make our code available at https://github.com/helsinki-nlp/geometry 2 methodology ## comparing the embedding spaces. we compare the representation spaces produced by bert and the encoder of a transformer trained on the mt task. bert is composed of 12 layers, plus an initial input embedding layer, with a dimension of 768. the mt system we apply consists of an input embedding layer followed by 6 transformer layers with a hidden dimension of 512. we use the pretrained bert-base-uncased model, as well as the pretrained english-german translation model opus-mt-en-de, both from the hug-gingface library @xcite . following @xcite , we extract embeddings using data from the semeval semantic textual similarity tasks from 2012 to 2016 @xcite . average similarity between random tokens. figure 1 presents the layer-wise cosine similarity (top) and the euclidean distance (bottom) distributions of randomly sampled words.the behavior of bert in figure 1 (top) is consistent with the findings of @xcite . the level of anisotropy of the embedding representations throughout layers of bert increases towards higher layers, with the exception of a slight drop at the last layer (l12), considering the average cosine similarity of the rep-resentations as a proxy measure of anisotropy. further, we notice figure 1 (bottom) that bert embeddings follow an inverted u-shape. this, together with the avgsim trend, means that the embedding space starts by stretching out and becoming narrower, later on to spread out shorter embeddings in layer 12, in line with @xcite . the mt-based representations, however, look significantly different. the cosine-based avgsim follows an almost u-like trend: it starts from a relatively high level at layer 0, then immediately drops and stays low throughout the middle layers, before a sudden increase at the final layer (l6). in particular: behavior (1) might be caused by the shared sourcetarget vocabularies and the embedding layer in the mt model in the encoder and the decoder being shared. such shared processing can result in a seeming inflation of the cosine similarity of randomly selected vectors, which actually belong to two different language spaces. to test for this hypothesis, we check the average euclidean distance between randomly selected tokens in figure 1 -bottom. interestingly, we do not observe considerable high levels of closeness between random words in layer 0, and the distribution is widespread. that is, the embeddings are organized in a narrow cone but have a wide range of lengths. this behaviour might arise from the system needing to represent both languages in the same space, and the interplay between training the embeddings layer at the target side while needing to keep source embeddings apart enough -future work is necessary to confirm this. motivated by these findings, we emphasize that using both metrics and observing how they interact allows for a more complete understanding of the representation spaces. 1 finding (2) is more relevant to our main question of the differences between the geometries of 1 cosine similarity does not take into account the magnitude of the vectors at play, making it susceptible to the existence a large value in one of the entries of a high-dimensional vector, while euclidean distance is hard to interpret in highdimensional spaces and it is unbounded from above. bert and mt. both metrics show a more gradual increase in the closeness of random tokens in the bert over layers, as compared to an abrupt increase in the mt space. therefore, we can deduce that the mt model can keep random representations successfully apart for all but the uppermost of the layers. we hypothesize that this monotonously increasing levels of closeness of random token embeddings in bert may be contributing to its suboptimal machine translation performance. to verify this hypothesis, in section 4 we present results on mt performance after alignment and in section 4.1 we show how the alignment method changes the embeddings distributions. similarity between tokens of the same form. selfsim will be high in less contextualized models, because such models use similar representations for each occurrence of the same token. highly contextualized models will have lower selfsim since every occurrence of the word will have a different representation. comparing the two spaces (figure 2 ), we again observe different trends. selfsim steadily drops for bert except for the last layer, showing an increase in the contextuality of the representations. for the mt model, on the other hand, we observe a steep drop at layer 6, indicating a sudden increase in contextuality here. all in all, bert gradually increases contextualization whereas the mt encoder tends to model individual lexical concepts in most layers before adding a strong contextual influence in the last one. once again, we see a different behavior in layer 0 of the mt model, which is characterized by low selfsim in the embedding layer. this a direct result of the high avgsim value at the embeddings layer (due to the shared vocabulary space) which is the anisotropy correction factor for selfsim. we deduce that anisotropy-corrected selfsim cannot straightforwardly be interpreted as a measure of contextuality in the embeddings layer of mt models with a shared source-target vocabulary. for comparison, we, therefore, also present the uncorrected selfsim (raw) value (dashed line) for this layer, which confirms this reasoning. similarity between tokens within the same sentence. we check the average similarity between tokens in the same sentence (intrasim). figure 2 reveals different behavior between the two models. in particular, we see a smooth increase over the layers for both models until the penultimate layer, pointing to an increasing level of in-sentence contextualization, as shown by the embeddings of the words in the same sentence gradually coming together. however, the behavior at the final layer is different between the two models. we observe an increase in intrasim for the bert model at the last layer, in contrast to the drop at the last layer of the mt model. in other words, the mt model is suddenly discriminating between the words in the sentence at layer 6, just before passing information to the decoder. we hypothesize that it may be useful for the mt decoder to have access to representations that are less contextualized at a source sentence level, since it still needs to add semantic information for decoding into the target language. notice that selfsim and intrasim decrease for final layer of the mt model. that is, similarity of word forms in different contexts is decreasing greatly and similarity of words to the mean sentence vector is (to a smaller degree) also decreasing. this might be an indication of the different constraints mt models have on contextualization. for example, the model may have a tendency to pay strong attention to syntactic and positional information, instead of focusing on shared semantics of the sentence. ## aligning the representation spaces to address the discrepancies observed in the bert and the mt encoder embedding spaces, we use the transformations from section 2.2. we use five different setups ( data. we use data from the english-german sections of the @xcite , europarl @xcite , extracted using opustools @xcite and the development tarball from the wmt2019 news translation shared task @xcite in the proportions indicated in ## related work analysis of contextualized representations. while there has been huge efforts to analyze word representations, most of it has been conducted using probing tasks @xcite @xcite . similarly, @xcite study the effects of fine-tuning bert representations on a specific set of probing tasks and analyse the change in the contextual representations using similarity analysis. @xcite quantitatively studied static word representations produced with skip-gram with negative sampling. their work was extended by @xcite for contextualized embeddings, in which they use word level measures of contextuality to contrast elmo @xcite , gpt-2 @xcite and bert @xcite . @xcite present a comparison of contextualized representations trained with different objectives, using cca and and mutual information to study information flow across networks. they conclude that although mt-produced representations do get refined with context, the change in those is not as extreme as for masked lm-produced representations (bert-like), which is in line with our observations of higher selfsim and lower intrasim (i.e. not ultra-contextualized embeddings) for mt and aligned models as compared to bert. pretrained lms in nmt. @xcite present a systematic comparison of methods to integrate bert into nmt models, including using bert at the embedding level or for initializing an encoder. @xcite propose a bert-fused mt system that uses additional attention modules between the outputs of bert and the encoder and decoder of the transformer, increasing the model parameters by the number of parameters the chosen bert flavour has. @xcite proposes a similar strategy, though using bert outputs only in the encoder, and a three-fold training technique. @xcite propose a simple yet effective two-stage optimization technique that first freezes bert, and then fine-tunes over the full model parameters set. we argue that this is similar to the align and fine-tune approach we propose for incorporating bert into mt. finally, a number of studies leverage pretraining techniques. mass @xcite is partly inspired by bert, but it is pretrained in nmt and is tailored to match the way prediction is done in nmt (leftto-right). @xcite enhance transformerbased mt systems performance by using a bart pretraining technique @xcite in a multilingual fashion to initialize an nmt system. alignment. numerous methods have been proposed for aligning (contextualized) word representations @xcite . @xcite learn an optimal linear transformation between embedding spaces. @xcite propose a similar approach using the centroids of the instances of the same word in different contexts. our work is closer to @xcite , which use a resource-efficient algorithm that takes into account the contextuality of embeddings. ## conclusion this paper provides an analysis of the intrinsic differences between bert and machine translation encoders. we compare the representation spaces of both models and pinpoint discrepancies between them. we show that this mismatch can be remedied through an alignment strategy, which successfully reshapes bert into an effective mt encoder. we also study the effects that the alignment methods have on the geometry of the embeddings spaces.