Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes (2021)

Abstract: State-of-the-art contextual embeddings are obtained from large language models available only for a few languages. For others, we need to learn representations using a multilingual model. There is an ongoing debate on whether multilingual embeddings can be aligned in a space shared across many languages. The novel Orthogonal Structural Probe (Limisiewicz and Mareček, 2021) allows us to answer this question for specific linguistic features and learn a projection based only on mono-lingual annotated datasets. We evaluate syntactic (UD) and lexical (WordNet) structural information encoded inmBERT’s contextual representations for nine diverse languages. We observe that for languages closely related to English, no transformation is needed. The evaluated information is encoded in a shared cross-lingual embedding space. For other languages, it is beneficial to apply orthogonal transformation learned separately for each language. We successfully apply our findings to zero-shot and few-shot cross-lingual parsing.

## introduction the representation learned by language models has been successfully applied in various nlp tasks. multilingual pre-training allows utilizing the representation for various languages, including lowresource ones. there is an open discussion about to what extent contextual embeddings are similar across languages @xcite @xcite . the motivation for our work is to answer: q1 is linguistic information uniformly encoded in the representations of various languages? and if this assumption does not hold: q2 is it possible to learn orthogonal transformation to align the embeddings? ## related work probing it is a method of evaluating linguistic information encoded in pre-trained nlp models. usually, a simple classifier for the probing task is trained on the frozen model's representation @xcite @xcite . the work of @xcite introduced structural probes that linearly transform contextual embeddings to approximate the topology of dependency trees. @xcite proposed new structural tasks and introduced orthogonal constraint allowing to decompose projected embeddings into parts correlated with specific linguistic features. @xcite probed different languages to examine what type of syntactic dependency annotation is captured in an lm. hall maudslay et al. (2020) modify the loss function, improving syntactic probes' ability to parse. ## method the structural probe @xcite is a gradient optimized linear projection of the contextual word representations produced by a pretrained neural model (e.g. @xcite , @xcite ). in a distance probe, the euclidean distance between projected word vectors approximates the distance between words in a dependency tree: b is the linear transformation matrix and h i , h j are the vector representations of words at positions i and j. another type of a probe is a depth probe, where the token's depth in a dependency tree is approximated by the euclidean norm of a projected word vector: orthogonal structural probes @xcite proposed decomposing matrix b and then gradient optimizing a vector and orthogonal matrix. the new formulation of an orthogonal distance probe is foot_0 : this procedure allowed optimizing a separate scaling vector d for a specific objective, allowing probing for multiple linguistic tasks simultaneously. in this work, an individual orthogonal transformation v is trained for each language, facilitating multi-language probing. this approach assumes that the representations are isomorphic across languages; we examine this claim in our experiments. our implementation is available at github: https://github.com/tom556/ orthogonaltransformerprobing . ## experiments we examine vector representations obtained from multilingual cased bert @xcite . ## conclusions we propose an effective way to multilingually probe for syntactic dependency (ud) and lexical hypernymy (wordnet). our algorithm learns probes for multiple tasks and multiple languages jointly. the formulation of orthogonal structural probe allows learning cross-lingual transformation based on mono-lingual supervision. our comparative evaluation indicates that the evaluated information is similarly distributed in the mbert's representations for languages typologically similar to english: spanish, french, and finnish. we show that aligning the embeddings with orthogonal transformation improves the results for other examined languages, suggesting that the representations are isomorphic. we show that the probe can be utilized in zero-and few-shot parsing. the method achieves better uas results for chinese, slovene, basque, and arabic in a zero-shot setting than previous approaches, which use a more complex biaffine parser. limitations in our choice of languages, we wanted to ensure diversity. nevertheless, four of the analyzed languages belong to an indo-european family that could facilitate finding shared encoding subspace for those languages.