R ETHINKING EMBEDDING COUPLING IN PRE - TRAINED LANGUAGE MODELS
Hyung Won Chung
Google Research hwchung Tsai Johnson melvinp@ Sebastian Ruder
DeepMind ruder
A BSTRACT
We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models.
We show that decoupled embeddings provide increased modeling ﬂexibility, allowing us to signiﬁcantly improve the efﬁciency of parameter allocation in the input embedding of multilingual models.
By reallocating the input embedding parameters in the
Transformer layers, we achieve dramatically better performance on standard natural language understanding tasks with the same number of parameters during ﬁne-tuning.
We also show that allocating additional capacity to the output embedding provides beneﬁts to the model that persist through the ﬁne-tuning stage even though the output embedding is discarded after pre-training.
Our analysis shows that larger output embeddings prevent the model’s last layers from overspecializing to the pre-training task and encourage Transformer representations to be more general and more transferable to other tasks and languages.
Harnessing these ﬁndings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the number of parameters at the ﬁne-tuning stage.
I NTRODUCTION
The performance of models in natural language processing (NLP) has dramatically improved in recent years, mainly driven by advances in transfer learning from large amounts of unlabeled data (Howard & Ruder, 2018; Devlin et al., 2019).
The most successful paradigm consists of pre-training a large Transformer (Vaswani et al., 2017) model with a self-supervised loss and ﬁne-tuning it on data of a downstream task (Ruder et al., 2019).
Despite its empirical success, inefﬁciencies have been observed related to the training duration (Liu et al., 2019b), pre-training objective (Clark et al., 2020b), and training data (Conneau et al., 2020a), among others.
In this paper, we reconsider a modeling assumption that may have a similarly pervasive practical impact: the coupling of input and output embeddings 1 in state-of-the-art pre-trained language models.
State-of-the-art pre-trained language models (Devlin et al., 2019;
Liu et al., 2019b) and their multilingual counterparts (Devlin et al., 2019; Conneau et al., 2020a) have inherited the practice of embedding coupling from their language model predecessors (Press & Wolf, 2017; Inan et al., 2017).
However, in contrast to their language model counterparts, embedding coupling in encoder-only pre-trained models such as Devlin et al.
(2019) is only useful during pre-training since output embeddings are generally discarded after ﬁne-tuning.
In addition, given the willingness of researchers to exchange additional compute during pre-training for improved downstream performance (Raffel [Table] [Table] et al., 2020; Brown et al., 2020) and the fact that pre-trained models are often used for inference millions of times (Wolf et al., 2019), pre-training-speciﬁc parameter savings are less important overall.
On the other hand, tying input and output embeddings constrains the model to use the same dimensionality for both embeddings.
This restriction limits the researcher’s ﬂexibility in parameterizing the model and can lead to allocating too much capacity to the input embeddings, which may be wasteful.
This is a problem particularly for multilingual models, which require large vocabularies with high-dimensional embeddings that make up between 47–71% of the entire parameter budget (Table 1)
[Table]
suggesting an inefﬁcient parameter allocation
[Table]
In this paper
[Table]
we systematically study the impact of embedding coupling on state-of-the-art pretrained language models, focusing on multilingual models.
First, we observe that while na ¨ ıvely decoupling the input and output embedding parameters does not consistently improve downstream evaluation metrics, decoupling their shapes comes with a host of beneﬁts.
In particular, it allows us to independently modify the input and output embedding dimensions.
We show that the input embedding dimension can be safely reduced without affecting downstream performance.
Since the output embedding is discarded after pre-training, we can increase its dimension, which improves ﬁne-tuning accuracy and outperforms other capacity expansion strategies.
By reinvesting saved parameters to the width and depth of the Transformer layers, we furthermore achieve signiﬁcantly improved performance over a strong mBERT (Devlin et al., 2019) baseline on multilingual tasks from the XTREME benchmark (Hu et al., 2020).
Finally, we combine our techniques in a Rebalanced mBERT (RemBERT) model that outperforms XLM-R (Conneau et al., 2020a), the state-of-the-art cross-lingual model while having been pre-trained on 3 .
5 × fewer tokens and 10 more languages.
We thoroughly investigate reasons for the beneﬁts of embedding decoupling.
We observe that an increased output embedding size enables a model to improve on the pre-training task, which correlates with downstream performance.
We also ﬁnd that it leads to Transformers that are more transferable across tasks and languages—particularly for the upper-most layers.
Overall, larger output embeddings prevent the model’s last layers from over-specializing to the pre-training task (Zhang et al., 2020; Tamkin et al., 2020), which enables training of more general Transformer models.
R ELATED WORK
Embedding coupling Sharing input and output embeddings in neural language models was proposed to improve perplexity and motivated based on embedding similarity (Press & Wolf, 2017) as well as by theoretically showing that the output probability space can be constrained to a subspace governed by the embedding matrix for a restricted case (Inan et al., 2017).
Embedding coupling is also common in neural machine translation models where it reduces model complexity (Firat et al., 2016) and saves memory (Johnson et al., 2017), in recent state-of-the-art language models (Melis et al., 2020), as well as all pre-trained models we are aware of (Devlin et al., 2019; Liu et al., 2019b).
Transferability of representations Representations of large pre-trained models in computer vision and NLP have been observed to transition from general to task-speciﬁc from the ﬁrst to the last layer (Yosinski et al., 2014;
Howard & Ruder, 2018; Liu et al., 2019a).
In Transformer models, the last few layers have been shown to become specialized to the MLM task and—as a result—less transferable (Zhang et al., 2020;
Tamkin et al., 2020).
Multilingual models Recent multilingual models are pre-trained on data covering aroundlanguages using a subword vocabulary shared across all languages (Devlin et al., 2019; Pires et al., 2019; Conneau et al., 2020a).
In order to achieve reasonable performance for most languages, these models need to allocate sufﬁcient capacity for each language, known as the curse of multilinguality (Conneau et al., 2020a;
Pfeiffer et al., 2020).
As a result, such multilingual models have large vocabularies with large embedding sizes to ensure that tokens in all languages are adequately represented.
Efﬁcient models Most work on more efﬁcient pre-trained models focuses on pruning or distillation (Hinton et al., 2015).
Pruning approaches remove parts of the model, typically attention heads (Michel et al., 2019; Voita et al., 2019) while distillation approaches distill a large pre-trained model into a smaller one (Sun et al., 2020).
Distillation can be seen as an alternative form of allocating pre-training capacity via a large teacher model.
However, distilling a pre-trained model is expensive (Sanh et al., 2019) and requires overcoming architecture differences and balancing training data and loss terms (Mukherjee & Awadallah, 2020).
Our proposed methods are simpler and complementary to distillation as they can improve the pre-training of compact student models (Turc et al., 2019).
E XPERIMENTAL METHODOLOGY
Efﬁciency of models has been measured along different dimensions, from the number of ﬂoating point operations (Schwartz et al., 2019) to their runtime (Zhou et al., 2020).
We follow previous work (Sun et al., 2020) and compare models in terms of their number of parameters during ﬁnetuning (see Appendix A.1 for further justiﬁcation of this setting).
For completeness, we generally report the number of pre-training (PT) and ﬁne-tuning (FT) parameters.
Baseline Our baseline has the same architecture as multilingual BERT (mBERT; Devlin et al., 2019).
It consists of 12 Transformer layers with a hidden size H of 768.
Input and output embeddings are coupled and have the same dimensionality E as the hidden size, i.e. E out =
E in = H .
The total number of parameters during pre-training and ﬁne-tuning is 177M (see Appendix A.2 for further details).
We train variants of this model that differ in certain hyper-parameters but otherwise are trained under the same conditions to ensure a fair comparison.
Tasks For our experiments, we employ tasks from the XTREME benchmark (Hu et al., 2020) that require ﬁne-tuning, including the XNLI (Conneau et al., 2018), NER (Pan et al., 2017), PAWS-X
(Yang et al., 2019), XQuAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020), and TyDiQA-GoldP (Clark et al., 2020a) datasets.
We provide details for them in Appendix A.4.
We average results across three ﬁne-tuning runs and evaluate on the dev sets unless otherwise stated.
E MBEDDING DECOUPLING REVISITED
Na¨ıve decoupling Embeddings make up a large fraction of the parameter budget in state-of-theart multilingual models (see Table 1).
We now study the effect of embedding decoupling on such models.
In Table 2, we show the impact of decoupling the input and output embeddings in our baseline model ( § 3) with coupled embeddings.
Na¨ıvely decoupling the output embedding matrix slightly improves the performance as evidenced by a 0.4 increase on average.
However, the gain is not uniformly observed in all tasks.
Overall, these results suggest that decoupling the embedding matrices na¨ıvely while keeping the dimensionality ﬁxed does not greatly affect the performance of the model.
What is more important, however, is that decoupling the input and output embeddings decouples the shapes , endowing signiﬁcant modeling ﬂexibility, which we investigate in the following.
Input vs output embeddings Decoupling input and output embeddings allows us to ﬂexibly change the dimensionality of both matrices and to determine which one is more important for good transfer performance of the model.
To this end, we compare the performance of a model with
[Table] [Table] E in = 768 , E out = 128 to that of a model with E in = 128 , E out = 768 3 (the remaining hyperparameters are the same as the baseline in § 3).
During ﬁne-tuning, the latter model has 43% fewer parameters.
We show the results in Table 3.
Surprisingly, the model pre-trained with a larger output embedding size is competitive with the comparison method on average despite having 77M fewer parameters during ﬁne-tuning.
Reducing the input embedding dimension saves a signiﬁcant number of parameters at a noticeably smaller cost to accuracy than reducing the output embedding size.
In light of this, the parameter allocation of multilingual models (see Table 1) seems particularly inefﬁcient.
For a multilingual model with coupled embeddings, reducing the input embedding dimension to save parameters as proposed by Lan et al.
(2020) is very detrimental to performance (see Appendix A.5 for details).
The results in this section indicate that the output embedding plays an important role in the transferability of pre-trained representations.
For multilingual models in particular, a small input embedding dimension frees up a signiﬁcant number of parameters at a small cost to performance.
In the next section, we study how to improve the performance of a model by resizing embeddings and layers.
E MBEDDING AND LAYER RESIZING FOR MORE EFFICIENT FINE - TUNING
Increasing the output embedding size In § 4, we observed that reducing E out hurts performance on the ﬁne-tuning tasks, suggesting E out is important for transferability.
Motivated by this result, we study the opposite scenario, i.e., whether increasing E out beyond H improves the performance.
We experiment with an output embedding size E out in the range { 128, 768, 3072 } while keeping the input embedding size E in = 128 and all other parts of the model the same as described in §( H = 768 , 12 layers, etc).
We show the results in Table 4.
In all of the tasks we consider, increasing E out monotonically improves the performance.
The improvement is particularly impressive for the more complex question answering datasets.
It is important to note that during ﬁne-tuning, all three models have the exact same sizes for E in and H .
The only difference among them is the output embedding, which is discarded after pre-training.
These results show that the effect of additional capacity during pre-training persists through the ﬁne-tuning stage even if the added capacity is discarded after pre-training.
We perform an extensive analysis on this behavior in § 6.
We show results with an English BERT Base model in Appendix A.6, which show the same trend.
[Table] [Table] Adding capacity via layers We investigate alternative ways of adding capacity during pre-training such as increasing the number of layers and discarding them after pre-training.
For a fair comparison with the E out = 768 model, we add 11 additional layers (total of 23) and drop the 11 upper layers after pre-training.
This setting ensures that both models have the same pre-training and ﬁne-tuning parameters.
We show the results in Table 5.
The model with additional layers performs poorly on the question answering tasks, likely because the top layers contain useful semantic information (Tenney et al., 2019).
In addition to higher performance, increasing E out relies only a more expensive dense matrix multiplication, which is highly optimized on typical accelerators and can be scaled up more easily with model parallelism (Shazeer et al., 2018) because of small additional communication cost.
We thus focus on increasing E out to expand pre-training capacity and leave an exploration of alternative strategies to future work.
Reinvesting input embedding parameters Reducing E in from 768 to 128 reduces the number of parameters from 177M to 100M. We redistribute these 77M parameters for the model with E out =
768 to add capacity where it might be more useful by increasing the width or depth of the model.
Speciﬁcally, we 1) increase the hidden dimension H of the Transformer layers from 768 to 1024and 2) increase the number of Transformer layers ( L ) from 12 to 23 at the same H to obtain models with similar number of parameters during ﬁne-tuning.
Table 6 shows the results for these two strategies.
Reinvesting the input embedding parameters in both H and L improves performance on all tasks while increasing the number of Transformer layers
L results in the best performance, with an average improvement of 3.9 over the baseline model with coupled embeddings and the same number of ﬁne-tuning parameters overall.
A rebalanced mBERT We ﬁnally combine and scale up our techniques to design a rebalanced mBERT model that outperforms the current state-of-the-art unsupervised model,
XLM-R
(Conneau et al., 2020a).
As the performance of Transformer-based models strongly depends on their number of parameters (Raffel et al., 2020), we propose a Rebalanced mBERT (RemBERT) model that matches
XLM-R’s number of ﬁne-tuning parameters (559M) while using a reduced embedding size, resized layers, and more effective capacity during pre-training.
The model has a vocabulary size of 250k,
E in = 256 , E out = 1536 , and 32 layers with 1152 dimensions and 18 attention heads per layer and was trained on data covering 110 languages.
We provide further details in Appendix A.7.
We compare RemBERT to XLM-R and the best-performing models on the XTREME leaderboard in Table 7 (see Appendix A.8 for the per-task results).
The models in the ﬁrst three rows use [Table] [Table] additional task or translation data for ﬁne-tuning, which signiﬁcantly boosts performance (Hu et al., 2020).
XLM-R and RemBERT are the only two models that are ﬁne-tuned using only the English training data of the corresponding task.
XLM-R was trained with a batch size of 2 13 sequences each with 2 9 tokens and 1.5M steps (total of 6.3T tokens).
In comparison, RemBERT is trained with 2 11 sequences of 2 9 tokens for 1.76M steps (1.8T tokens).
Even though it was trained with 3 . 5 × fewer tokens and has 10 more languages competiting for the model capacity, RemBERT outperforms
XLM-R on all tasks we considered.
This strong result suggests that our proposed methods are also effective at scale.
We will release the pre-trained model checkpoint and the source code for
RemBERT in order to promote reproducibility and share the pre-training cost with other researchers.
O N THE IMPORTANCE OF THE OUTPUT EMBEDDING SIZE
We carefully design a set of experiments to analyze the impact of an increased output embedding size on various parts of the model.
We study the nature of the decoupled input and output representations ( § 6.1) and the transferability of the Transformer layers with regard to task-speciﬁc ( § 6.2) and language-speciﬁc knowledge ( § 6.3).
6. N ATURE OF INPUT AND OUTPUT EMBEDDING REPRESENTATIONS
We ﬁrst investigate to what extent the representations of decoupled input and output embeddings differ based on word embedding association tests (Caliskan et al., 2017).
Similar to Press & Wolf (2017), for a given pair of words, we evaluate the correlation between human similarity judgements of the strength of the relationship and the dot product of the word embeddings.
We evaluate on MEN (Bruni et al., 2014), MTurk771 (Halawi et al., 2012),
Rare-Word (Luong et al., 2013), SimLex (Hill et al., 2015), and Verb-143 (Baker et al., 2014).
As our model uses subwords, we average the token representations for words with multiple subwords.
We show the results in Table 8.
In the ﬁrst two rows, we can observe that the input embedding of the decoupled model performs similarly to the embeddings of the coupled model while the output [Table] embeddings have lower scores.
7 We note that higher scores are not necessarily desirable as they only measure how well the embedding captures semantic similarity at the lexical level.
Focusing on the difference in scores, we can observe that the input embedding learns representations that capture semantic similarity in contrast to the decoupled output embedding.
At the same time, the decoupled model achieves higher performance in masked language modeling.
The last three rows of Table 8 show that as E out increases, the difference in the input and output embedding increases as well.
With additional capacity, the output embedding progressively learns representations that differ more signiﬁcantly from the input embedding.
We also observe that the
MLM accuracy increases with E out .
Collectively, the results in Table 8 suggest that with increased capacity, the output embeddings learn representations that are worse at capturing traditional semantic similarity (which is purely restricted to the lexical level) while being more specialized to the MLM task (which requires more contextual representations).
Decoupling embeddings thus give the model the ﬂexibility to avoid encoding relationships in its output embeddings that may not be useful for its pre-training task.
As pre-training performance correlates well with downstream performance (Devlin et al., 2019), forcing output embeddings to encode lexical information can hurt the latter.
6. C ROSS - TASK T RANSFERABILITY OF T RANSFORMER LAYER REPRESENTATIONS
We investigate to what extent more capacity in the output embeddings during pre-training reduces the MLM-speciﬁc burden on the Transformer layers and hence prevents them from over-specializing to the MLM task.
Dropping the last few layers We ﬁrst study the impact of an increased output embedding size on the transferability of the last few layers.
Previous work (Zhang et al., 2020; Tamkin et al., 2020) randomly reinitialized the last few layers to investigate their transferability.
However, those parameters are still present during ﬁne-tuning.
We propose a more aggressive pruning scheme where we completely remove the last few layers.
This setting demonstrates more drastically whether a model’s upper layers are over-specialized to the pre-training task by assessing whether performance can be improved with millions fewer parameters.
We show the performance of models with 8–12 remaining layers (removing up to 4 of the last layers) for different output embedding sizes E out on XNLI in Figure 1.
For both E out = 128 and E out = 768 , removing the last layer improves performance.
In other words, the model performs better even with 7.1M fewer parameters.
With E out = 128 , the performance remains similar when removing the last few layers, which suggests that the last few layers are not critical for transferability.
As we increase E out , the last layers become more transferable.
With E out = 768 , removing more than one layer results in a sharp reduction in performance.
Finally when E out = 3072 , every layer is [Figure] [Figure] [Figure]
useful and removing any layer worsens the performance.
This analysis demonstrates that increasing
E out improves the transferability of the representations learned by the last few Transformer layers.
Probing analysis We further study whether an increased output embedding size improves the general natural language processing ability of the Transformer.
We employ the probing analysis of Tenney et al.
(2019) and the mix probing strategy where a 2-layer dense network is trained on top of a weighted combination of the 12 Transformer layers.
We evaluate performance with regard to core NLP concepts including part-of-speech tagging (POS), constituents (Consts.), dependencies (Deps.), entities, semantic role labeling (SRL), coreference (Coref.), semantic proto-roles (SPR), and relations (Rel.).
For a thorough description of the task setup, see Tenney et al.
We show the results of the probing analysis in Table 9.
As we increase E out , the model improves across all tasks, even though the number of parameters is the same.
This demonstrates that increasing
E out enables the Transformer layers to learn more general representations.
6. C ROSS - LINGUAL TRANSFERABILITY OF T RANSFORMER LAYER REPRESENTATIONS
So far, our analyses were not specialized to multilingual models.
Unlike monolingual models, multilingual models have another dimension of transferability: cross-lingual transfer, the ability to transfer knowledge from one language to another.
Previous work (Pires et al., 2019; Artetxe et al., 2020) has found that MLM on multilingual data encourages cross-lingual alignment of representations without explicit cross-lingual supervision.
While it has been shown that multilingual models learn useful cross-lingual representations, overspecialization to the pre-training task may result in higher layers being less cross-lingual and focusing on language-speciﬁc phenomena necessary for predicting the next word in a given language.
To investigate to what extent this is the case and whether increasing E out improves cross-lingual alignment, we evaluate the model’s nearest neighbour translation accuracy (Pires et al., 2019) on
English-to-German translation (see Appendix A.9 for a description of the method).
We show the nearest neighbor translation accuracy for each layer in Figure 2.
As E out increases, we observe that a) the Transformer layers become more language-agnostic as evidenced by higher accuracy and b) the language-agnostic representation is maintained to a higher layer as indicated by a ﬂatter slope from layer 7 to 11.
In all cases, the last layer is less language-agnostic than the previous one.
The sharp drop in performance after layer 8 at E out = 128 is in line with previous results on cross-lingual retrieval (Pires et al., 2019;
Hu et al., 2020) and is partially mitigated by an increased
E out .
In sum, not only does a larger output embedding size improve cross-task transferability but it also helps with cross-lingual alignment and thereby cross-lingual transfer on downstream tasks.
C ONCLUSION
We have assessed the impact of embedding coupling in pre-trained language models.
We have identiﬁed the main beneﬁt of decoupled embeddings to be the ﬂexibility endowed by decoupling their shapes.
We showed that input embeddings can be safely reduced and that larger output embeddings and reinvesting saved parameters lead to performance improvements.
Our rebalanced multilingual
BERT (RemBERT) outperforms XLM-R with the same number of ﬁne-tuning parameters while having been trained on 3 .
5 × fewer tokens.
Overall, we found that larger output embeddings lead to more transferable and more general representations, particularly in a Transformer’s upper layers.
A A PPENDIX A. E FFICIENCY COMPARISON BASED ON PARAMETER COUNT DURING FINE - TUNING
We compare the efﬁciency of models based on their number of parameters.
We believe this to be a reasonable proxy for a model’s efﬁciency as the performance of Transformer-based language models has been shown to improve monotonically with the number of parameters (Kaplan et al., 2020; Raffel et al., 2020;
Lepikhin et al., 2020; Brown et al., 2020; Shoeybi et al., 2019; Aharoni et al., 2019)
As the number of parameters during pre-training and ﬁne-tuning may differ we compare models based on their number of parameters during the ﬁne-tuning stage (without the task-speciﬁc head).
We argue that this is the most practically relevant number as a model is generally pre-trained only once but may be ﬁne-tuned or used for inference millions of times. [Table] [Table]
A. B ASELINE MODEL DETAILS
Our baseline model has the same architecture as multilingual BERT (mBERT; Devlin et al., 2019).
It consists of 12 Transformer layers with a hidden size H of 768 and 12 attention heads withdimensions each.
Input and output embeddings are coupled and have the same dimensionality E as the hidden size, i.e. E out =
E in = H .
The total number of parameters during pre-training and ﬁne-tuning is 177M. We do not use dropout following the recommendation from Lan et al.
We use the SentencePiece tokenizer (Kudo & Richardson, 2018) and a shared vocabulary of 120k subwords.
The model is trained on Wikipedia dumps in 104 languages following Devlin et al.
(2019) using masked language modeling (MLM).
We choose this baseline as its behavior has been thoroughly studied (K et al., 2020; Conneau et al., 2020b; Pires et al., 2019; Wu & Dredze, 2019).
A. T RAINING DETAILS
For all pre-training except for the large scale RemBERT, we trained using 64 Google Cloud TPUs.
We trained over 26B tokens of Wikipedia data.
All ﬁne-tuning experiments were run on 8 Cloud
TPUs.
For all ﬁne-tuning experiments other than RemBERT, we use batch size of 32.
We sweep over the learning rate values speciﬁed in Table 10.
We used the SentencePiece tokenizer trained with unigram language modeling
A. XTREME TASKS
For our experiments, we employ tasks from the XTREME benchmark (Hu et al., 2020).
We show statistics for them in Table 11.
XTREME includes the following datasets: The Cross-lingual Natural Language Inference (XNLI; Conneau et al., 2018) corpus, the Cross-lingual Paraphrase Adversaries from Word Scrambling (PAWS-X; Yang et al., 2019) dataset, part-of-speech (POS) tagging data from the Universal Dependencies v2.5 (Nivre et al., 2018) treebanks, the Wikiann (Pan et al., 2017) dataset for named entity recognition (NER), the Cross-lingual Question Answering Dataset (XQuAD; Artetxe et al., 2020), the Multilingual Question Answering (MLQA; Lewis et al., 2020) dataset, the gold passage version of the Typologically Diverse Question Answering (TyDiQA; Clark et al., 2020a) dataset, data from the third shared task of the workshop on Building and Using Parallel
Corpora (BUCC; Zweigenbaum et al., 2018), and the Tatoeba dataset (Artetxe & Schwenk, 2019).
We refer the reader to Hu et al.
(2020) for more details.
We average results across three ﬁne-tuning runs and evaluate on the dev sets unless otherwise stated. [Table] [Table]
A. C OMPARISON TO L AN ET AL . (2020)
Crucially, our ﬁnding differs from the dimensionality reduction in ALBERT (Lan et al., 2020).
While they show that smaller embeddings can be used, their input and output embeddings are coupled and use a much smaller vocabulary (30k vs 120k).
In contrast, we ﬁnd that simultaneously decreasing both the input and output embedding size drastically reduces the performance of multilingual models.
In Table [Table] we show the impact of their factorized embedding parameterization on a monolingual and a multilingual model.
While the English model suffers a smaller (0.8%) drop in accuracy, the multilingual model’s performance drops by 2.6%.
Direct application of a factorized embedding parameterization (Lan et al., 2020) is thus not viable for multilingual models.
A. E NGLISH MONOLINGUAL RESULTS
So far, we have focused on multilingual models as the number of saved parameters when reducing the input embedding size is largest for them.
We now apply the same techniques to the English 12-layer BERT Base with a 30k vocabulary (Devlin et al., 2019).
Speciﬁcally, we decouple the embeddings, reduce E in to 128, and increase the output embedding size or the number of layers during pre-training.
We show the performance on MNLI (Williams et al., 2018) and SQuAD (Rajpurkar et al., 2016) in Table 13.
By adding more capacity during pre-training, performance monotonically increases similar to the multilingual models.
Interestingly, pruning a 24-layer model to 12 layers reduces performance, presumably because some upper layers still contain useful information.
A. R EM BERT DETAILS
We design a Rebalanced mBERT (RemBERT) to leverage capacity more effectively during pretraining.
The model has 995M parameters during pre-training and 575M parameters during ﬁnetuning.
We pre-train on large unlabeled text using both Wikipedia and Common Crawl data, covering 110 languages.
The details of hyperparameters and architecture are shown in Table 14.
For each language l , we deﬁne the empirical distribution as
[Equation]
[Table]
[Table]
where n l is the number of sentences in l ’s pre-training corpus.
Following Devlin et al.
[Table] we use an exponentially smoothed distribution, i [Table] e., we exponentiaate p l by α = 0 .
5 and renormalize to obtain the sampling distribution.
Hyperparameters and pre-training details are summarized in Table 14.
Hyperparameters used for the leaderboard submission are shown in Table 15.
A. XTREME TASK RESULTS
We show the detailed results for RemBERT and the comparison per task on the XTREME leaderboard in Table 16.
Compared to Table
[Table]
which shows the average across task categories, this table shows the average across tasks.
A. N EAREST - NEIGHBOR TRANSLATION COMPUTATION
For an English-to-German translation, we sample M = 5000 pairs of sentences from WMT (Bojar et al., 2016).
For each sentence in each language, we obtain a representation v ( l ) LANG at each layer l by averaging the activations of all tokens (except the [CLS] and [SEP] tokens) at that layer.
We then compute a translation vector from English to German by averaging the difference between the vectors of each sentence pair across all pairs:
For each English sentence v ( l ) EN i , we can now translate it with this vector: v ( l ) EN i + ¯ v ( l ) EN → DE .
We locate the closest German sentence vector based on (cid:96) 2 distance and measure how often the nearest neighbour is the correct pair.
[Table]