It’s not G reek to m BERT : Inducing Word-Level Translations from Multilingual BERT (2020)

Abstract: Recent works have demonstrated that multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages. We study the word-level translation information embedded in mBERT and present two simple methods that expose remarkable translation capabilities with no fine-tuning. The results suggest that most of this information is encoded in a non-linear way, while some of it can also be recovered with purely linear tools. As part of our analysis, we test the hypothesis that mBERT learns representations which contain both a language-encoding component and an abstract, cross-lingual component, and explicitly identify an empirical language-identity subspace within mBERT representations.

## introduction multilingual-bert (mbert) is a version of bert @xcite , trained on the concatenation of wikipedia in 104 different languages. recent works show that it excels in zero-shot transfer between languages, for a variety of tasks @xcite , despite being trained with no parallel supervision. previous work has mainly focused on what is needed for zero-shot transfer to work well @xcite @xcite , and on characterizing the representations of mbert @xcite . however, we still lack a proper understanding of this model. in this work we study (1) how much word-level translation information is recoverable by mbert; and (2) how this information is stored. we focus on the representations of the last layer, and on the embedding matrix that is shared between the input and output layers -which are together responsible for token prediction. for our first goal, we start by presenting a simple and strong method to extract word-level translation information. our method is based on explicit querying of mbert: given a source word and a target language, we feed mbert with a template such as "the word 'source' in language is: @xcite ." where language is the target language, and source is an english word to translate. getting the correct translation as the prediction of the masked token exposes mbert's ability to provide word-level translation. this template-based method is surprisingly successful, especially considering the fact that no parallel supervision was provided to the model while training, and that word translation is not part of the training objective. this raises the possibility of easy disentanglement between language identity and lexical semantics in mbert representations. we test this hypothesis by trying to explicitly disentangle languageidentity from lexical semantics under linearity assumptions. we propose a method for disentangling a language-encoding component and a languageneutral component from both the embedding representations and word-in-context representations. furthermore, we learn the emperical "langauge subspace" in mbert, which is a linear subspace that is spanned by all directions that are linearly correlative to the language identity. we demonstrate that the representations are well-separated by language on that subspace. we leverage these insights and empirical results to show that it is possible to perform analogiesbased translation by taking advantage of this disentanglement: we can alter the language-encoding component, while keeping the lexical component intact. we compare between the template-based method and the analogies-based method and discuss their similarities and differences, as well as their limitations. the two methods together show that mbert acquired, to a large degree, the ability to perform word-level translation, despite the fact that it is not trained on any parallel data explicitly. the results suggest that most of the information is stored in a non-linear way, but with some linearly-recoverable components. our contribution in this work is two-fold: (a) we present two simple methods for word-level translation using mbert, that require no training or finetuning of the model, which demonstrate that mbert stores parallel information in different languages; (b) we show that mbert representations are composed of language-encoding and languageneutral components and present a method for extracting those components. our code is available at https://github.com/gonenhila/mbert . beyond focusing on zero-shot transfer abilities, an additional line of work studies the representations of mbert and the information it stores. using hierarchical clustering based on the cca similarity scores between languages, @xcite are able to construct a tree structure that faithfully describes relations between languages. @xcite learn a linear syntax-subspace in mbert, and point out to syntactic regulartieis in the representations that transfer across languages. in the recent work of @xcite , the authors define the notion of contextual word alignment. they design a fine-tuning loss for improving alignments and show that they are able to improve zero-shot transfer after this alignment-based fine-tuning. one main difference from our work is that they fine-tune the model according to their new definition of contextual alignment, while we analyze and use the information already stored in the model. one of the closest works to ours is that of @xcite , where they assume that mbert's representations have a language-neutral component, and a language-specific component. they remove the language specific component by subtracting the centroid of the language from the representations, and make an attempt to prove the assumption by using probing tasks on the original vs. new representations. they show that the new representations are more language-neutral to some extent, but lack experiments that show a complementary component. while those works demonstrate that mbert representations in different languages can be aligned successfully with appropriate supervision, we propose an explicit decomposition of the representations to language-encoding and language-neutral components, and also demonstrate that implicit word-level translations can be easily distilled from the model when exposed to the proper stimuli. ## word-level translation using pre-defined templates we study the extent to which it is possible to extract word-level translation directly from mbert. ## dissecting mbert representations in the previous section, we saw that mbert contains abundant word-level translation knowledge. how is this knowledge represented? we turn to analyze both the representations of words in context and those of the output embeddings. it has been assumed in previous work that the representations are composed of a languageencoding component and a language-neutral component @xcite . in what follows, we explicitly try to find such a decomposition: we decompose @xmath0, where v lang and v lex are orthogonal vectors, v lang is the part in the representation that is indicative of language identity, and v lex maintains lexical information, but is in- splitting the representations into components is done using inlp @xcite , an algorithm for removing information from vector representations. ## analogies-based translation in the previous section we established the assumption that mbert representations are composed of a language-neutral and a language-encoding components. in this section, we present another mechanism for word-translation with mbert, which is based on manipulating the language-encoding component of the representation, in a similar way to how analogies in word embeddings work @xcite . this new method has a clear mechanism behind it, and it serves as an additional validation for our assumption about the two independent components. the idea is simple: we create a single vector representation for each language, as explained below. then, in order to change the embedding of a word in language source to language target, we simply subtract from it the vector representation of language source and add the vector representation of language target. finally, in order to get the translation of the source word into the target language, we multiply the resulting representation by the output embedding matrix to get the closest words to it out of the full vocabulary. below is a detailed explanation of the implementation. ## discussion the template-based method we presented is nonlinear and puts a high lower bound on the amount of parallel information found in mbert, with surprisingly good results on word-level translation. the analogies-based method also gets impressive results, but to a lesser extent than the template-based one. in addition, the resulting representations in the analogies-based method are much less structured. these together suggest that most of the parallel information is not linearly decodable from mbert. the reasoning behind the analogies-based method is very clear: under linearity assumption we explicitly characterize and compute the decomposition to language-encoding and languageneutral components, and derive a word-level translation method based on this decomposition. the mechanism behind the template-based method and the source of its success, however, are much harder to understand and interpret. while it is possible that some parallel data, in one form or another, is present in the training corpora, this is still an implicit signal: there is no explicit supervision for the learning of translation. the fact that mlm training is sufficient -at least to some degree -to induce learning of the algorithmic function of translation (without further supervised finetuning) is nontrivial. we believe that the success of this method is far from being obvious. we leave further investigation of the sources of this success to future work. ## conclusion we aim to shed light on a basic question regarding multilingual bert: how much word-level translation information does it embed and what are the ways to extract it? answering this question can help understand the empirical findings on its impressive transfer ability across languages. we show that the knowledge needed for wordlevel translation is implicitly encoded in the model, and is easy to extract with simple methods, without fine-tuning. this information is likely stored in a non-linear way. however, some parts of this representations can be recovered linearly: we identify an empirical language-identity subspace in mbert, and show that under linearity assumptions, the representations in different languages are easily separable in that subspace; neutralizing the language-identity subspace encourages the model to perform word predictions which are less sensitive to language-identity, but are nonetheless semantically-meaningful. we argue that the results of those interventions support the hypothesis on the existence of identifiable language components in mbert. for language-identity prediction. as expected, the nullspace does not encode language identity: vmeasure drops to 11.5% and 11.4% in the embeddings layer and in the last layer, respectively.