Finding Universal Grammatical Relations in Multilingual BERT (2020)

Abstract: Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks’ internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.

## introduction past work @xcite has found that masked language models such as bert @xcite ) learn a surprising amount of linguistic structure, despite a lack of direct linguistic supervision. recently, large multilingual masked language models such as multilingual bert (mbert) and xlm @xcite have shown strong cross-lingual performance on tasks like xnli @xcite and dependency parsing @xcite . much previous analysis has been motivated by a desire to explain why bert-like models perform so well on downstream applications in the monolingual setting, which begs the question: what properties of these models make them so crosslingually effective? figure 1 : t-sne visualization of head-dependent dependency pairs belonging to selected dependencies in english and french, projected into a syntactic subspace of multilingual bert, as learned on english syntax trees. colors correspond to gold ud dependency type labels. although neither mbert nor our probe was ever trained on ud dependency labels, english and french dependencies exhibit cross-lingual clustering that largely agrees with ud dependency labels. in this paper, we examine the extent to which multilingual bert learns a cross-lingual representation of syntactic structure. we extend probing methodology, in which a simple supervised model is used to predict linguistic properties from a model's representations. in a key departure from past work, we not only evaluate a probe's performance (on recreating dependency tree structure), but also use the probe as a window into understanding aspects of the representation that the probe was not trained on (i.e. dependency labels; figure 1 ). in particular, we use the structural probing method of @xcite , which probes for syntactic trees by finding a linear transformation under which two words' distance in their dependency parse is approximated by the squared distance between their model representation vectors under a linear transformation. after evaluating whether such transformations recover syntactic tree distances across languages in mbert, we turn to analyzing the transformed vector representations themselves. we interpret the linear transformation of the structural probe as defining a syntactic subspace (figure 2 ), which intuitively may focus on syntactic aspects of the mbert representations. since the subspace is optimized to recreate syntactic tree distances, it has no supervision about edge labels (such as adjectival modifier or noun subject). this allows us to unsupervisedly analyze how representations of head-dependent pairs in syntactic trees cluster and qualitatively discuss how these clusters relate to linguistic notions of grammatical relations. we make the following contributions: our analysis sheds light on the cross-lingual properties of multilingual bert, through both zeroshot cross-lingual structural probe experiments and novel unsupervised dependency label discovery experiments which treat the probe's syntactic subspace as an object of study. we find evidence that mbert induces universal grammatical relations without any explicit supervision, which largely agree with the dependency labels of universal dependencies. foot_0 ## methodology we present a brief overview of @xcite 's structural probe, closely following their derivation. the method represents each dependency tree t as a distance metric where the distance between two words d t (w i , w j ) is the number of edges in the path between them in t . it attempts to find a single linear transformation of the model's word representation vector space under which squared distance recreates tree distance in any sentence. formally, let h ℓ 1:n be a sequence of n representations produced by a model from a sequence of n words w ℓ 1:n composing sentence ℓ. given a matrix @xmath0, we define a squared distance metric d b as the squared l 2 distance after transformation by b: we optimize to find a b that recreates the tree distance d t ℓ between all pairs of words (w ℓ i , w ℓ j ) in all sentences s ℓ in the training set of a parsed corpus. specifically, we optimize by gradient descent: for more details, see @xcite . departing from prior work, we view the probetransformed word vectors bh themselves-not just the distances between them-as objects of study. the rows of b are a basis that defines a subspace of r m , which we call the syntactic subspace, and may focus only on parts of the original bert representations. a vector bh corresponds to a point in that space; the value of each dimension equals the dot product of h with one of the basis vectors. foot_1 ## related work cross-lingual embedding alignment @xcite find that independently trained monolingual word embedding spaces in elmo are isometric under rotation. similarly, @xcite and @xcite geometrically align contextualized word embeddings trained independently. @xcite find that cross-lingual transfer in mbert is possible even without shared vocabulary tokens, which they attribute to this isometricity. in concurrent work, @xcite demonstrate that mbert embeddings of similar words in similar sentences across languages are approximately aligned already, suggesting that mbert also aligns semantics across languages. k et al. (2020) demonstrate that strong cross-lingual transfer is possible without any word piece overlap at all. ## discussion language models trained on large amounts of text have been shown to develop surprising emergent properties; of particular interest is the emergence of non-trivial, easily accessible linguistic properties seemingly far removed from the training objective. for example, it would be a reasonable strategy for mbert to share little representation space between languages, effectively learning a private model for each language and avoiding destructive interference. instead, our transfer experiments provide evidence that at a syntactic level, mbert shares portions of its representation space between languages. perhaps more surprisingly, we find evidence for fine-grained, cross-lingual syntactic distinctions in these representations. even though our method for identifying these distinctions lacks dependency label supervision, we still identify that mbert has a cross-linguistic clustering of grammatical relations that qualitatively overlaps considerably with the universal dependencies formalism. the uuas metric we note that the uuas metric alone is insufficient for evaluating the accuracy of the structural probe. while the probe is opti-mized to directly recreate parse distances, (that is, limitations our methods are unable to tease apart, for all pairs of languages, whether transfer performance is caused by subword overlap @xcite or by a more fundamental sharing of parameters, though we do note that language pairs with minimal subword overlap do exhibit nonzero transfer, both in our experiments and in others @xcite . moreover, while we quantitatively evaluate cross-lingual transfer in recovering dependency distances, we only conduct a qualitative study in the unsupervised emergence of dependency labels via t-sne. future work could extend this analysis to include quantitative results on the extent of agreement with ud. we acknowledge as well issues in interpreting t-sne plots @xcite , and include multiple plots with various hyperparameter settings to hedge against this confounder in figure 11 . future work should explore other multilingual models like xlm and xlm-roberta @xcite and attempt to come to an understanding of the extent to which the properties we've discovered have causal implications for the decisions made by the model, a claim our methods cannot support.