The Impact of Positional Encodings on Multilingual Compression (2021)

Abstract: In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings. Several modifications have been proposed over the sinusoidal positional encodings used in the original transformer architecture; these include, for instance, separating position encodings and token embeddings, or directly modifying attention weights based on the distance between word pairs. We first show that surprisingly, while these modifications tend to improve monolingual language models, none of them result in better multilingual language models. We then answer why that is: sinusoidal encodings were explicitly designed to facilitate compositionality by allowing linear projections over arbitrary time steps. Higher variances in multilingual training distributions requires higher compression, in which case, compositionality becomes indispensable. Learned absolute positional encodings (e.g., in mBERT) tend to approximate sinusoidal embeddings in multilingual settings, but more complex positional encoding architectures lack the inductive bias to effectively learn cross-lingual alignment. In other words, while sinusoidal positional encodings were designed for monolingual applications, they are particularly useful in multilingual language models.

## introduction multiple recent papers have attempted to pinpoint precisely what components of multilingual language models enable cross-lingual transfer. @xcite show that although wordpiece overlap tends to improve cross-lingual transfer performance, even languages with different scripts (and no shared subwords) may enable zero-shot transfer. @xcite report similar results on a wider range of tasks. @xcite show our starting point is @xcite , who claim that a) multilingual compression is caused by forced parameter sharing across languages, and that b) positional encodings play a significant role in the creation of a multilingual space, even in the absence of shared subwords and shared special tokens, like delimiters. contributions we build on @xcite and demonstrate, through a series of experiments on synthetic and real data, that the choice of positional encoding mechanism has a significant effect on cross-lingual model performance: while many positional encodings have been proposed in monolingual settings as improvements over sinusoidal or absolute positional encodings, originally proposed in @xcite and @xcite , including untied positional encodings (tupe; @xcite ) and relative positional encodings @xcite , none of these better facilitate cross-lingual compression or sharing. in fact, multilingual language models trained with untied or relative positional encodings exhibit much worse cross-lingual perfor-mance. we show that this is because sinusoidal embeddings facilitate compositionality, which we argue is particularly important for cross-lingual compression. we present a method for quantifying the compositionality of positional encodings, and find additional evidence for this hypothesis in word-position correlations and ablation studies. we are, to the best of our knowledge, the first to show this asymmetry between monolingual and multilingual language model training. our experiments rely on the protocols in @xcite , but in addition to simple experiments with their bible data, we also replicate all our experiments on wikipedia data. rather than relying on deterministic perturbations of data, as in @xcite and @xcite , we make novel use of galactic dependencies @xcite in our experiments. based on our experiments, we recommend caution when adopting methods developed for monolingual language models when training multilingual models, as well as that future work on positional encoding mechanisms also provides evaluations in multilingual settings. ## positional encodings positional encodings have been a mainstay of non-autoregressive transformer-based models right since @xcite first proposed the transformer architecture. the motivation being that given that transformers foot_0 are order-invariant (as opposed recurrent or convolutional networks), there must be some injection of word order into the encoder. rather than using conventional "embeddings", @xcite use fixed sinusoidal position encodings, where each dimension characterises a sinusoidal waveform of a fixed frequency. specifically, each encoding p is given as: several alternatives to sinusoidal encodings have been proposed since @xcite . most multilingual models tend to use bert-style @xcite learnt absolute positional encodings, where a unique vector is learned and assigned to each position; these vectors are then added to word representations before being passed to the self-attention mechanism. as an alternative to such position representations, where every position is represented by a unique vector, relative positional encodings have been proposed @xcite . rather than assigning representations to tokens based on their position, relative positional encoding involves assigning representations to position-position pairs; typically, these encodings are calculated separately and added to the attention matrix. we evaluate both the encodings proposed in @xcite and the encodings proposed in @xcite in our experiments below. @xcite propose eliminating positionposition correlations, and using separate parameters for word and position representations; @xcite propose using dependency trees instead of raw sequential positions. @xcite recommend eliminating the addition operation in bert-style representations; they argue that wordposition correlations are effectively nil, and that the addition introduces unnecessary noise. we evaluate two untied positional encodings proposed in @xcite . tupe modifies absolute representations by a) untying word-position correlations; b) using a separate set of parameters for positional attention and c) untying [cls] tokens from positions. we refer to recent surveys @xcite for a more detailed treatment of position encoding methods. we provide a summary of our methods in the above positional encodings have been introduced in the context of monolingual pretrained language models, and there has been only a limited amount of work addressing the effect of positional encodings on multilingual models. @xcite find that positional information tends to hurt machine translation, as the encoder learns a word-order bias towards the source languages. 2 artetxe et al. (2020) find that language-specific positional representations help in an adapter-based training scenario. @xcite attempt to account for structural differences between languages by using bracketing transduction grammar trees to reorder position labels (and find that it helps). @xcite find that models that are relatively agnostic to word-order tend to perform better in cross-lingual settings; they hypothesise that large multilingual encoders, being trained on languages with drastic differences in word orders, tend to have order-agnostic positional encodings, and thus discourage fine-tuning positional encodings downstream. contemporaneous with this work, @xcite show that positional information is important for monolingual models even given unnatural, randomly shuffled word ordering. dufter and schütze (2021) present a set of experiments training smaller language models on bilingual corpora, consisting of the same corpus in english and "fake-english", which is english with a shifted bpe vocabulary. they evaluate retrieval and translation scores at different layers; gold alignments are easy to derive given that the corpora are effectively parallel corpora, and that the vocabularies for both halves are effectively the same. as we build on these experiments, we adopt slightly simplified notation, and denote vocabulary-shifted corpora with square brackets, eg. @xcite . to induce structural diversity in our corpora, therefore, we reorder our corpora using galactic dependencies (gd) models @xcite . briefly, gd models sample ordering statistics based on dependency relations for the dependants of verbs and/or nouns from some superstrate language xx; when applied to sentences these during fine-tuning helps cross-lingual zero-shot generalization. in some substrate language (in the context of our experiments, en), the models reorder dependants of verb and/or noun nodes to match the ordering statistics of the substrate language they were trained on. we opt to reorder both nominal and verbal arguments, and follow the authors in denoting the sampling operation with a ∼, giving us for eg. en∼xx for an english language corpus, with dependent order statistics adapted from some language xx. note that gd reordering only works for projective sentences, and rather than retain un-reordered non-projective sentences, we exclude them from all our corpora. en so there were fourteen generations from abraham to david . en∼ar . there were so generations fourteen from abraham to david en∼de there were so from abraham to david fourteen generations . en∼eu there were so david to abraham from generations fourteen . en∼fi abraham from david to fourteen generations there were so . en∼fr fourteen generations from abraham to david were there so . en∼hi there so david to abraham from fourteen generations were . en∼sv there were so generations from abraham to david fourteen . this approach, while simple and useful, does have several limitations. predominantly, because our reordering is fundamentally syntactic/structural, our fake languages still maintain both the morphology of the source language (english in our case), and the same vocabulary distribution. thus, although scrambling ought to affect context and neighbourhoods, an english token and its corresponding fake token have exactly the same unigram distribution. training our model of choice is an underparameterised bert, as in @xcite . we train multiple such underparameterised bert models, each with a different encoding mechanism from section 2, on two bilingual corpora: en + [en] -a bilingual corpus comprised of english, and a fake vocab-shifted english. en + [en∼xx] -a bilingual corpus comprised of english, and a fake english that has had its constituents reordered to match the distribution of some language xx. we reorder our english starting point according to seven different faux-languages (just "languages" for brevity): arabic, german, basque, finnish, french, hindi and swedish. note that given that our starting point was english, there was no way for us to control for morphological differences; as such, languages with freer word order (like basque) are likelier to make our english corpora ambiguous. we use two corpora in this work: the first is the bible splits from dufter and schütze (2021), with the english easy-to-read bible as the training split, and the kjv bible as validation. the second corpus uses the english wikipedia as the training split, and common crawl as validation. we present corpus statistics in as in the original paper @xcite . for sinusoidal representations, while @xcite multiply token embeddings by √ d to avoid drowning them out with the [-1, 1] sinusoidal encoding range, we find that our default embedding size is too small for this to have an effect, and instead scale up token embeddings by 2 √ d before adding positional encodings. for all parameterised encoding models except tupe (relative), we use a maximum of @xmath1; the concrete transformers implementation of the relative methods means that this gives us 1023 total offsets. 4 for tupe (relative), we use a maximum of @xmath2, divided into 32 bins with logarithmically increasing bin sizes; this is taken from the original implementation in @xcite . ## evaluation we adopt dufter and schütze's (2021) evaluation pipeline, evaluating each of our models at layers 0 and 8; we also describe a multilingual score, which is defined as the average accuracy for the retrieval and translation tasks, at layers 0 and 8. we also measure perplexity, both on the monolingual first half of the corpus, and on both halves combined. note that true perplexities for masked language models are intractable @xcite . we use a trivial approximation and calculate perplexity based on the prediction loss for each masked token; note that while these suffice for comparison purposes, they are not true perplexities and should not be taken as such outside the context of these experiments. we present our results (averaged out over fauxlanguages) in figure 1 , with full results in appendix c. as expected, the more recent positional encodings are superior to sinusoidal or absolute positional encodings in the monolingual setting; but somewhat surprisingly, sinusoidal and absolute positional encodings are clearly outperforming the more recent approaches in the multilingual setting. we also note that the gap in multilingual performance only grows larger when a different word order is imposed on the target language; see the bottom row of figure 1 . interestingly, switching to structurally different l2s can sometimes reduce the language modelling perplexity of the l1: this could be due to regularisation induced by structural differences. ## analyses in an attempt to explain the significantly improved cross-lingual performance of absolute positional encodings, we tried to examine precisely what sort of encoding was being learnt. part of the original motivation behind sinusoidal encodings was that they would allow for compositionality; for any fixed offset k, there exists a linear transformation from p pos to p pos+k , making it easier to learn to attend to relative offsets; the proof of this is in appendix a. 5 we examined our absolute positional encodings to see whether or not they were being induced to learn some specific function. represents a specific dimension of the encoding vectors generated for positions 0 to 31. interestingly, it appears that absolute representations converge to waveforms that represent sinusoids somewhat, while neither of the untied experiments do so (cf. appendix b). we hypothesize that absolute representations converge to waveforms because of increased pressure for compositionality, being trained on structurally different languages. to test this, we quantify the extent to which the absolute, relative and untied encodings are compositional in the sense that there is a linear transformation from p pos to p pos+k for different k. to this end, we use procrustes analysis @xcite to learn a linear transformation for each k, based on the representations of p pos and p pos+k . specifically, we apply orthogonal procrustes analyses (schönemann, 1966), which avoid scaling and translation. first, we minimise arg min t ||p pos -tp pos+k || 2 . next, we apply t to a different randomly selected pos , i.e. we calculate @xmath3. the higher the final loss l, the less our encodings facilitate compositionality. in order to make learning t simpler, rather than selecting representations for single positions pos and pos , we select chunks of arbitrary size c, and stack their positions into a matrix. note that for sinusoidal representations, the loss is close to zero regardless of span. the losses are plotted over a range of offsets for both absolute representations and for tupe(a), in figure 5 ; we include a control model trained on a monolingual corpus. losses are averaged over 125 runs per offset, with random values of pos, pos and c. while both forms of representation appear to be similar (and relatively non-sinusoidal) when trained on the monolingual corpus, introducing bilingualism leads to a clear difference between the two: absolute positional representations tend to be a lot closer to sinusoidal representations than untied ones do. note, also, that this gap is clear-est for the (simpler) en + [en] experiment -this is unsurprising, as en + [en] is still perceived as bilingual due to the shifted vocabulary. the structural similarity between the two, however, makes it easier to build compositional representations by relying on offsets, as the model only needs to learn to represent one language, structurally speaking. we observe a similar gap when comparing pretrained bert models: bert-base-multilingual-cased exhibits more sinusoidal representations over a range of offsets, when compared to bert-base-cased, although the gap is narrower than with our toy models. ## discussion the main contribution of our work is practical, namely showing that findings about positional encodings in the context of monolingual language models do not apply straightforwardly to multilingual language models. in answering why sinusoidal embeddings are superior to more recent alternatives in the multilingual setting, we also found the compositionality of positional encodings to be predictive of multilingual compression in such models. while relative positional encodings seem designed for compositionality, they prevent efficient alignment of multilingual vocabularies. @xcite show that word order matters little for monolingual language model pretraining, and that pretrained language models seem to rely mostly on higher-order word co-occurrence statistics. our work shows that this finding does not generalize to pretraining multilingual language models. in the multilingual setting, word order clearly matters, as also shown in previous work @xcite , and compositional positional encodings seem to facilitate effective multilingual compression. this aligns with the observation that syntactic reordering à la @xcite is in some cases an effective way to encourage compositional cross-lingual representations. in general, our results illustrate how methods developed for monolingual language models should not be blindly adopted when training multilingual models, which potentially require different architectures. conversely, we would encourage future work on new positional encoding mechanisms for non-autoregressive models to also evaluate these mechanisms in multilingual settings. ## conclusion through a series of synthetic and real experiments with training multilingual language models, we showed that a) sinusoidal positional encodings perform better in multilingual settings than more recent alternatives (that have been shown to perform better in monolingual settings); b) this is likely because of an increased pressure for compositionality. we devised a method for quantifying the compositionality of positional encodings, and strengthened our results by also considering word-position correlations and ablation studies.