Probing Multilingual BERT for Genetic and Typological Signals (2020)

Abstract: We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations.

## introduction cross-lingual text representations have become extremely popular in nlp, since they promise universal text processing in multiple human languages with labeled training data only in a single one. they go back at least to the work of @xcite , and have seen an exploding number of contributions in recent years. recent cross-lingual models provide representations for about 100 languages and vary in their training objectives. in offline learning, cross-lingual representations are obtained by projecting independently trained monolingual representations into a shared representational space using bilingual lexical resources @xcite . in joint learning @xcite , the cross-lingual representations are learned directly, for example as a byproduct of large-scale machine translation @xcite . as parallel data is scarce for less frequent language pairs, the multilingual bert model (mbert) simply trains the bert architecture @xcite on multilingual input from wikipedia. the cross-lingual signal is thus only learned implicitly because mbert uses the same representational space independent of the input language. this naive approach yields surprisingly high scores for cross-lingual downstream tasks, but the transfer does not work equally well for all languages. @xcite show that the performance differences between languages are gradual and that the representational similarity between languages seem to correlate with typological features. these relationships between languages remain opaque in cross-lingual representations and pose a challenge for the evaluation of their adequacy. evaluations in down-stream tasks are an unreliable approximation because they can often be solved without accounting for deep linguistic knowledge or for interdependencies between subgroups of languages @xcite . while more language-agnostic representations can be beneficial to improve the average performance in task-oriented settings and to smooth the performance differences between high-and low-resource languages @xcite , linguists are more interested in the representational differences between languages. the field of computational historical linguistics, for example, examines subtle semantic and syntactic cues to infer phylogenetic relations between languages @xcite . important aspects are the diachronic stability of word meaning @xcite and the analysis of structural properties for inferring deep language relationships @xcite . traditionally, these phenomena have been approximated using hand-selected word lists and typological databases. common ancestors for languages are typically inferred based on cases of shared word meaning and surface form overlap and it can be assumed that these core properties are also captured in large-scale cross-lingual representations to a certain extent. for example, @xcite find that phylogenetic relations between languages can be reconstructed from cross-lingual representations if the training objective optimizes monolingual semantic constraints for each language separately as in the multilingual muse model @xcite . muse is restricted to only 29 frequent languages, however. while mbert is a powerful cross-lingual model covering an order of magnitude more languages (104), a better understanding of the type of signal captured in its representations is needed to assess its applicability as a testbed for cross-lingual or historical linguistic hypotheses. our analysis quantifies the representational similarity across languages in mbert and disentangles it along genetic, geographic, and structural factors. in general, the urge to improve the interpretability of internal neural representations has become a major research field in recent years. whereas dense representations of images can be projected back to pixels to facilitate visual inspection, interpreting the linguistic information captured in dense representation of languages is more complex @xcite . diagnostic classifiers @xcite , representational stability analysis @xcite and indirect visualization techniques @xcite are only a few examples for newly developed probing techniques. they are used to examine whether the representations capture part-of-speech information @xcite , syntactic agreement @xcite , speech features @xcite , and cognitive cues @xcite . however, the majority of these interpretability studies focus solely on english. @xcite perform a contrastive analysis of the syntactic interpretability of english and polish representations and @xcite probe representations in three lower-resource languages. cross-lingual interpretability research for multiple languages focuses on the ability to transfer representational knowledge across languages for zero-shot semantics @xcite and for syntactic phenomena @xcite . in this work, we contribute to the nascent field of typological and comparative linguistic interpretability of language representations at scale @xcite and analyze representations for more than 100 languages. ## related work representational distance between two languages refers to the (averaged) differences between model representations for selected concepts in the two languages. interpretability analyses attempt to disentangle the typological factors that influence the representational distance. in this work, we distinguish between phylogenetic, geographic and structural factors. two languages are considered to be phylogenetically close if they descend from a common ancestor language. geographically close languages are languages which are primarily spoken in regions with a small physical distance on earth. structural similarity between languages refers to shared syntactic and morphological features. for many languages, the three categories overlap, but they are not necessarily linked. for example, spanish and basque are geographically close, but structurally and phylogenetically quite distant. previous approaches differ in the type of cross-lingual representations, the number of languages, and the methodology for determining representational distance and for interpreting the typological signal. @xcite obtain representations for 102 language pairs (english ↔ language x) using neural machine translation and then visualize the representations using dimensionality reduction. they explore the visualization qualitatively and find clusters which resemble language families. however, when zooming into the clusters, it becomes evident that a mixture of genetic and geographic factors contributes to the representational distance. for instance, dravidian and indo-aryan languages overlap completely. @xcite induce bilingual vector spaces for 21 europarl languages and quantify representational distance between languages by averaging over the pairwise similarity between word representations. they find that the differences can be better explained by geographic than by phylogenetic factors. conversely, @xcite analyze english translations of sentences in 17 europarl languages and find that syntactic traces of the native language of the translator can best be explained by language genetics. @xcite use the same dataset and train language representations on the linguistic structure of the sentences. they find that the representational distance between languages can be better explained by structural similarity (obtained from dependency trees) than by language genetics. pretrained cross-lingual models are optimized for tasks such as bilingual lexicon induction and machine translation. even if linguistic information is not explicitly provided during training, recent interpretability research indicates that phylogenetic properties are encoded in the resulting representations. @xcite obtain representations for swadesh word lists from the muse model @xcite which jointly optimizes monolingual and crosslingual semantic constraints. they find that hiearchical clustering over the representational distance between languages yields phylogenetically plausible language trees. interestingly, they cannot trace the phylogenetic signal in representations from the sentence-based laser model @xcite which is trained to learn language-neutral representations for machine translation. @xcite analyze representations from mbert and find that clustering over averaged representations for the 104 languages yields phylogenetically plausible language groups. they argue that mbert is not language-neutral and that semantic phenomena are not modeled properly across languages. in our analysis, we further quantify the representational distance and disentangle it along phylogenetic, geographic, and structural factors. bjerva and augenstein (2018) train cross-lingual representations in an unsupervised way for different linguistic levels including phonology, morphology, and syntax. they use them to infer missing typological features for more than 800 languages. @xcite also infer missing features in typological databases from cross-lingual representations. inferring such missing features can be considered a form of probing. indeed, in contemporaneous work, @xcite predict typological properties from representations of four different recent state-of-the-art cross-lingual encoders using probing classifiers. we do not use probing classifiers in our work because the choice of classifier and the size of its training data may affect the probing outcomes @xcite . ## methodology in the following, we first briefly describe the two cross-lingual embedding spaces analyzed in this work, mbert and fasttext ( §3.1). then, we detail how we compute distances between languages using representations from these spaces ( §3.2) and concept lists developed in historical linguistics ( §3.3). once we have language distances, we infer trees from distance matrices and compare these trees to gold standard phylogenetic trees ( §3.4) to evaluate how strong a historical linguistic tree signal is contained in our ## experiments in the following, we detail a series of probing experiments with both mbert and fasttext. to extract representations from mbert, we feed a single word v k (i) from a language i corresponding to concept k into mbert and extract the corresponding representations v (r) k (i) in all layers @xmath0, . . . , 12. we are fully aware that using mbert in a context-independent way ignores main benefits of the model. we do so in order to leverage concept lists at large scale for a majority of the 100 languages available in mbert. otherwise, we would have to experiment with sentence-aligned data, which is available only for much smaller subsets (< 30) of our languages. nevertheless, we believe that a good contextual model should also be equipped with good context-independent token representations. ## concluding remarks we applied a series of tests for probing mbert for typological signals. while the language trees inferred from mbert representations are sometimes close to the reference trees, they may confound multiple factors. a more-fined grained investigation of t-sne plots followed by matrix regression analyses suggests that representational distances correlate most with phylogenetic and geographical distances between languages. further, the rankings from cross-lingual stability scores correlate significantly with meaning lists for items supposed to be resistant to cross-temporal lexical replacement. our results contribute to the recent discourses on interpretability and introspection of black-box nlp representations @xcite @xcite . in our case, we asked how mbert perceives of the similarity of two languages and related this to phylogenetic, geographic and structural factors. in future work, we aim to use our inferred similarities to predict transfer behavior in downstream tasks between specific language pairs. finally, we strongly caution against using our conclusions as support for hypotheses relating semantics and language phylogeny (e.g., the sapir-whorf hypothesis). our results for bilingual lexicon induction indicate that mbert representations are only mildly semantic cross-lingually which corroborates similar findings in related work.