A multilabel approach to morphosyntactic probing (2021)

Abstract: We propose using a multilabel probing task to assess the morphosyntactic representations of multilingual word embeddings. This tweak on canonical probing makes it easy to explore morphosyntactic representations, both holistically and at the level of individual features (e.g., gender, number, case), and leads more naturally to the study of how language models handle co-occurring features (e.g., agreement phenomena). We demonstrate this task with multilingual BERT (Devlin et al., 2018), training probes for seven typologically diverse languages: Afrikaans, Croatian, Finnish, Hebrew, Korean, Spanish, and Turkish. Through this simple but robust paradigm, we verify that multilingual BERT renders many morphosyntactic features simultaneously extractable. We further evaluate the probes on six held-out languages: Arabic, Chinese, Marathi, Slovenian, Tagalog, and Yoruba. This zero-shot style of probing has the added benefit of revealing which cross-linguistic properties a language model recognizes as being shared by multiple languages.

## introduction morphologically rich languages present unique challenges to natural language processing. these languages typically exhibit complex agreement patterns and their high diversity of inflected forms can lead to sparse examples of vocabulary words in training data, even in large corpora @xcite . it is therefore worthwhile to explore how neural language models (lms), which serve as the foundation of many state-of-the-art systems, handle the morphological complexity of diverse languages. morphosyntactic features of natural languages bear meaningful information that is useful for downstream tasks, such as machine translation, question answering, and language generation. adding morphological supervision through multitask training regimes @xcite or morphologically-informed tokenization @xcite can improve the quality of multilingual language models. nonetheless, recent work has shown that lms trained without explicit morphological supervision can still produce useful representations that capture morphosyntactic phenomena (e.g., @xcite @xcite . to further these investigations, we propose using a multilabel probing task to assess the morphosyntactic representations of multilingual word embeddings. this work is premised on the intuition that, if a simple model (a "probe") can easily extract linguistic properties from embeddings, this indicates that the lm has learned to encode those features in some fashion @xcite @xcite . we show how a multilabel paradigm can shed light on the morphosyntactic representations of lms, both holistically and at the level of individual features. our contributions are threefold: first, we introduce an efficient probing paradigm for analyzing multiple morphosyntactic features, which we demonstrate with multilingual bert @xcite and seven typologically diverse languages: afrikaans, croatian, finnish, hebrew, korean, spanish, and turkish. second, we evaluate the probes on six "held-out" languages-arabic, chinese, marathi, slovenian, tagalog, and yorùbá-showing how this paradigm can be used in a zero-shot manner to illuminate the properties that multilingual bert represents similarly crosslinguistically. third, we release our code and multilabel probe predictions to guide future probing efforts and to serve as the foundation for future in-depth feature-level analyses. 1 this paper is structured as follows: section 2 reviews related work, motivating §3, which introduces multilabel morphosyntactic probing. section 4 then outlines the data and models we use to probe multilingual bert. in §5, we demonstrate the probing paradigm in a set of monolingual experiments, training and evaluating separate probes for the seven languages, and provide an example feature-level analysis of hebrew determiners. in §6, we delve into whether multilingual probes yield comparable insights to the monolingual probes. then, in a set of crosslingual experiments, §7 evaluates how the monolingual and multilingual probes handle the six held-out languages. finally, §8 discusses our findings and concludes the paper. ## related work numerous studies in recent years have sought to study the linguistic properties captured by neural language models (e.g., @xcite @xcite @xcite @xcite @xcite @xcite . in the morphology domain, the linspector suite by @xcite probes 24 languages via 15 linguistic tasks, including multiple tasks to identify morphological features. in a similar vein, edmiston (2020) uses several morphological prediction tasks to inspect embeddings from five monolingual transformer-based language models, focusing exclusively on indo-european languages. the probing paradigm proposed in this paper builds on these works, but consolidates morphosyntactic feature prediction under a single task that leads more naturally to the study of feature co-occurrence. recent probing work has also sought to curtail how much probes memorize about linguistic tasks to ensure that they reflect information available in their input embeddings-probes should be extractive rather than learnèd themselves. efforts to minimize memorization have included reducing the training data to probes @xcite and limiting probe complexity, such as through dropout (e.g., @xcite and the use of simpler architectures (e.g., a linear layer instead of a multilayer perceptron, as in @xcite @xcite . to guide the design and interpretation of probes, @xcite propose supplementing diagnostic tasks with control tasks, where a probe is trained to predict random outputs within the same output space as the diagnostic task, given the same embeddings. if the probe performs well on the control task, they caution that it has the capacity to memorize the linguistic features under consideration; conversely, if the probe does well on the diagnostic task but poorly on the control task, then it is a reliable diagnostic of linguistic representations in the embeddings (though see @xcite for interesting discussions). hewitt and liang operationalize this comparison as selectivity, the difference in performance between the diagnostic and control tasks. the greater the selectivity, the more the probe "expresses" the information encoded in its input. in this paper, we design a control task to complement multilabel probing. ## multilabel morphosyntactic probing we propose using multilabel morphosyntactic tagging to assess the morphosyntactic representations of neural lms. in this diagnostic task, we hold contextualized word embeddings constant, then train linear classifiers on top of them (cf. @xcite to perform morphosyntactic tagging. in its objective, morphosyntactic tagging resembles the second sigmorphon 2019 shared task, which called for labeling words in a sentence with their morphosyntactic descriptions @xcite . it is easy to imagine doing morphosyntactic tagging in a traditional multiclass fashion, where we train separate probes to identify different features, such as part of speech (pos), gender, or number (cf. şahin et al., 2020; edmiston, 2020). however, this style of probing is more likely to prompt narrow analyses that consider morphological properties in isolation. alternatively, we could train a single probe to extract complex labels like def.sing.masc.noun and 3rd.plur.masc.past.verb. thus, each word would have a single correct label and a final softmax layer would output the probability of each class being the correct one. however, a drawback to this approach is that, depending on the number of properties we would like to identify, this can result in a combinatoric nightmare , with few training examples per class. to overcome these limitations, we frame morphosyntactic tagging as a word-level multilabel task, allowing for a token to receive multiple feature labels (e.g., both @xmath0. such a paradigm allows us to encode features with multiple or ambiguous values (e.g., @xmath1,masc; a.k.a. multi-valued features) and enables a closer inspection of learnt agreement and feature co-occurrence patterns. figure 1 illustrates hypothetical gold vectors for two hebrew pronouns that differ only in number. ## experimental setup we demonstrate multilabel morphosyntactic probing with multilingual bert (henceforth, mbert; @xcite , using morphologically annotated corpora from universal dependencies (ud; @xcite ). @xcite @xcite . with the exception of the korean data, all of the corpora came pre-split into training, validation, and test sets. we performed an 80-10-10 split on the 1,000-sentence korean pud corpus. to throttle the probes' training data (cf. zhang and bowman, 2018), we reduced the other training sets to 800 sentences as well. next, in a set of multilingual experiments, we trained probes on a shuffled combination of the training sentences from the monolingual probes. however, we excluded the korean dataset from this analysis, due to the lack of documentation on its construction. finally, in a set of crosslingual transfer experiments, we evaluated the monolingual and multilingual probes on six held-out languages: arabic (padt; cf. @xcite @xcite , chinese (pud; cf. zeman et al., 2017), marathi (ufal; cf. ravishankar, 2017), slovenian (sst; cf. @xcite , tagalog (trg), and yorùbá (ytb; cf. ishola and zeman, 2020). to clarify, mbert was pre-trained on these languages; we consider them "held-out" in that we never train probes to extract linguistic properties from these corpora (i.e., the experiments are zeroshot). the datasets for the monolingual, multilingual, and crosslingual experiments are summarized in appendix a. all of the probes were trained to extract multiple features, such as pos, number, gender, case, and tense, as well as language-specific features, such as finnish infinitive forms. (it is due to the inclusion of parts of speech that we refer to the task as "morphosyntactic tagging".) since the languages vary in their linguistic properties, we used different label sets for each language and a semi-aggregated set for the multilingual probes. across our experiments, we extracted 166 different feature labels in total, as listed in appendix b. the ud corpora include decompositions of multiword tokens and separate annotations for their respective components. to keep the input to the probes faithful to naturalistic text, we embed the multiword tokens themselves, but aggregate the feature labels from their components (e.g., the hebrew multiword token 952% hasefer 'the book' is marked as both a determiner and noun). ## monolingual experiments in a set of monolingual experiments, we trained and evaluated individual diagnostic probes on afrikaans, croatian, finnish, hebrew, korean, spanish, and turkish, given representations from the even-numbered mbert layers. their microaveraged f 1 scores are conveyed in figure 2 , along with their results on the analogous control tasks. ## multilingual experiments we have used monolingual probes to assess the linguistic representations from mbert on a language- overt determiner 0.93 0.56 0.70 covert determiner 0.69 0.40 0.50 by-language basis. however, can we replace the individual monolingual probes with a single multilingual probe and derive comparable insights? to address this question, we trained multilingual probes on a shuffled combination of the training sets for afrikaans, croatian, finnish, hebrew, spanish, and turkish. the multilingual probes extracted an aggregated subset of the features captured by the monolingual probes. we then assessed the multilingual probes' performance on each language independently. overall, the multilingual probes exhibited slight dips in performance, but better selectivity, compared to their monolingual counterparts (figure 3 ). these trends occurred despite all of the multilingual models converging before they reached epoch 50. ## crosslingual experiments our probing paradigm can also be used to study which morphosyntactic features are encoded similarly cross-linguistically: if a monolingual probe can successfully extract a feature label given a heldout language, this suggests that the lm has come to recognize that property as being shared by the two languages. in this section, we evaluate the monolingual and multilingual probes on ud test sets for arabic, chinese, marathi, slovenian, tagalog, and yorùbá. these experiments are akin to prior work on zeroshot crosslingual transfer @xcite @xcite , though we differ in that we never fine-tune mbert. focusing once more on mbert-6, this section examines a small subset of labels, presented in figure 4 . however, see appendix f for the global f 1 scores across the held-out languages and full feature-level results from mbert-6. ## discussion and conclusion emerging studies on interpretability have highlighted a wealth of linguistic information that can be extracted from neural language models. contributing to this effort, we propose using a multilabel probing task to analyze the morphosyntactic representations of multilingual word embeddings. we demonstrate this probing paradigm with mbert @xcite . in a set of monolingual experiments ( §5), we trained individual probes for afrikaans, croatian, finnish, hebrew, korean, spanish, and turkish. we found that mbert-6 holds the most morphosyntactic information (cf. @xcite , with the probes obtaining microaveraged f 1 scores between 0.83 and 0.97. in a small case study of hebrew determiners ( §5.3), we illustrated an analysis that implicates multiple features (i.e., lexical category, pronominal type, number, and gender). crucially, traditional single-label efforts would require training multiple models to arrive at such an analysis and, in general, run the risk of overlooking relevant features. (we also suspect that training multiple one-off probes is less computationally efficient than training a single multilabel probe, though we leave this comparison for future work.) next, in a set of multilingual experiments ( §6), we saw that the multilingual probes marginally underperformed their monolingual counterparts, while largely upholding the same trends and exhibiting better selectivity. we attributed this con-trast in performance to the monolingual probes relying more on memorization, given a simpler task ( §6.2). these findings indicate that the multilingual probes may be more "expressive" diagnostics of linguistic representations (cf. @xcite . however, since our goal is to probe embeddings rather than to perform state-of-the-art morphosyntactic tagging, the monolingual and multilingual probes offer the same insights to the extent that they exhibit comparable trends and lend themselves to the same generalizations. in a set of crosslingual experiments, we further evaluated the monolingual and multilingual probes on data from six "held-out" languages: arabic, chinese, marathi, slovenian, tagalog, and yorùbá ( §7). we showed that applying the probes accordingly can help illuminate which linguistic properties a lm recognizes as being shared by multiple languages and what factors might lead a lm not to encode cohesive representations of a particular cross-linguistic feature. namely, we conjectured that cross-linguistic variation in the distribution of nominative morphology led mbert to form decentralized representations of nominative case; in turn, this made it more challenging for the probes to extract nominative case in the held-out languages. in sum, multilabel probe predictions can be used to perform holistic analyses of a language model's ability to encode systems of morphology, as well as more fine-grained analyses of individual features, agreement phenomena, and how shared properties are represented cross-linguistically. we release the predictions from our probes to support more detailed analyses of mbert's facility for morphosyntax; these predictions can also be used to focus future contributions by identifying which mbert layers to target for more complex probing of specific features. in addition, we encourage future efforts to probe different multilingual language models using the multilabel paradigm and to examine how these models might vary in their morphosyntactic representations (cf. @xcite . finally, future research should explore how global and feature-level morphosyntactic probe performance corresponds to the performance of downstream systems, especially amongst morphologically rich languages. helpful feedback and discussions. we also thank the uw research computing club for supporting our research through their cloud credit program.