Intrinsic Probing through Dimension Selection (2020)

Abstract: Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted. To enable intrinsic probing, we propose a novel framework based on a decomposable multivariate Gaussian probe that allows us to determine whether the linguistic information in word embeddings is dispersed or focal. We then probe fastText and BERT for various morphosyntactic attributes across 36 languages. We find that most attributes are reliably encoded by only a few neurons, with fastText concentrating its linguistic structure more than BERT.

## introduction natural language processing (nlp) is enamored of contextual word representations-and for good reason! contextual word-embedders, e.g. bert @xcite and @xcite , have bolstered nlp model performance on myriad tasks, such as syntactic parsing @xcite , coreference resolution @xcite , morphological tagging @xcite ) and text generation @xcite . given the large empirical gains observed when they are employed, it is all but certain that word representations derived from neural networks encode some continuous analogue of linguistic structures. 1 code and data are available at https://github. com/rycolab/intrinsic-probing . exactly what these representations encode about linguistic structure, however, remains little understood. researchers have studied this question by attributing function to specific network cells with visualization methods @xcite and by probing @xcite , which seeks to extract structure from the representations. recent work has probed various representations for correlates of morphological @xcite , syntactic @xcite @xcite , and semantic @xcite structure. most current probing efforts focus on what we term extrinsic probing, where the goal is to determine whether the posited linguistic structure is predictable from the learned representation. generally, extrinsic probing works argue for the presence of linguistic structure by showing that it is extractable from the representations using a machine learning model. in contrast, we focus on intrinsic probing-whose goals are a proper superset of the goals of extrinsic probing. in intrinsic probing, one seeks to determine not only whether a signature of linguistic structure can be found, but also how it is encoded in the representations. in short, we aim to discover which particular "neurons" (a.k.a. dimensions) in the representations correlate with a given linguistic structure.intrinsic probing also has ancillary benefits that extrinsic probing lacks; it can facilitate manual analyses of representations and potentially yield a nuanced view about the information encoded by them. the technical portion of our paper focuses on developing a novel framework for intrinsic probing: we scan sets of dimensions, or neurons, in a word vector representation which activate when they correlate with target linguistic properties. we show that when intrinsically probing high-dimensional representations, the present probing paradigm is insufficient ( §2). current probes are too slow to be used under our framework, which invariably leads to low-resolution scans that can only look at one or a few neurons at a time.instead, we introduce decomposable probes, which can be trained once on the whole representation and henceforth be used to scan any selection of neurons. to that end, we describe one such probe that leverages the multivariate gaussian distribution's inherent decomposability, and evaluate its performance on a large-scale, multi-lingual, morphosyntactic probing task ( §3). we experiment on 36 languages 2 from the universal dependencies treebanks @xcite . we find that all the morphosyntactic features we considered are encoded by a relatively small selection of neurons. in some cases, very few neurons are needed; for instance, for multilingual bert english representations, we see that, with two neurons, we can largely separate past and present tense in fig. 1 . in this, our work is closest to @xcite , except that we extend the investigation beyond individual neurons-a move which is only made tractable by decomposable probing. we also provide analyses on morphological features beyond number and tense. across all languages, 35 out of 768 neurons on average suffice to reach a reasonable amount of encoded information, and adding more yields diminishing returns (see fig. 2 ). interestingly, in our head-to-head comparison of bert and fasttext, we find that fasttext almost always encodes information about morphosyntactic 2 see app. f for a list. ## probing through dimension selection the goal of intrinsic probing is to reveal how "knowledge" of a target linguistic property is structured within a neural network-derived representation. if said property can be predicted from the representations, we expect that this is because the neural network encodes this property @xcite .we can then determine whether a probe requires a large subset or a small subset of dimensions to predict the target property reliably. 3 particularly small subsets could be used to manually analyze a network and its decision process, and potentially reveal something about how specific neural architectures learn to encode linguistic information. to formally describe our framework, we first define the necessary notation. we consider the probing of a word representation @xmath0. in this work, our goal is find a subset of dimensions c ⊆ @xmath1, . . . , d} such that the corresponding subvector of h c contains only the dimensions that are necessary to predict the target morphosyntactic property we are probing for. for all possible subsets of dimensions c ⊆ d, and some random variable π that ranges over p property values {π 1 , . . . , π p }, we consider a general probabilistic probe: p θ c (π = π | h c ); note that the model is conditioned on h c , not on h. our goal is to select a subset of dimensions using the log-likelihood of held-out data. we term this type of probing dimension selection. one can express dimension selection as the following combinatorial optimization problem: greed is not enough. a natural first approach to approximate a solution to eq. ( 1 ) is a greedy algorithm @xcite ). such an algorithm chooses the dimension that results in the largest increase to the objective at every iteration. however, some probes, such as neural network probes, need to be trained with a gradient-based method for many epochs. in such a case, even a greedy approximation is prohibitively expensive. for example, to select the first dimension, we train d probes and take the best. to select the second dimension, we train d -1 probes and take the best. this requires training o(dk) networks! in the case of bert, we have @xmath2. training on the order of 38400 neural models to probe for just one morphosyntactic property is generally not practical. what we require is a decomposable probe, which can be trained once on all dimensions and then be used to evaluate the log-likelihood of any subset of dimensions in constant or near-constant time. to the best of our knowledge, no probes in the literature exhibit this property; the primary technical contribution of the paper is the development of such a probe in §3. other selection criteria. our exposition above uses the log-likelihood of held-out data as a selection criterion for a subset of dimensions; however, any function that scores a subset of dimensions is suitable. for example, much of the current probing literature relies on accuracy to evaluate probes @xcite , and two recent papers motivate a probabilistic evaluation with information theory @xcite . one could select based on accuracy, mutual information, or anything else within our framework. in fact, recent work in intrinsic probing by @xcite could be recast into our framework if we chose a dimension selection criterion based on the magnitude of the weights of a linear probe. however, we suspect that a performance-based dimension selection criterion (e.g., log-likelihood) should be more robust given that a weight-based approach is sensitive to feature collinearity, variance and regularization. as we mentioned before, performance-based selection requires a probe to be decomposable, and to the best of our knowledge, this is not the case for the the linear probe of @xcite . ## a decomposable probe for morphosyntactic properties using the framework introduced above, our goal is to probe for morphosyntactic properties in word representations. we first describe the multivariate gaussian distribution as it is responsible for our probe's decomposability ( §3.1), and provide some more notation ( §3.2). we then describe our model ( §3.3) and a bayesian formulation ( §3.4). ## probing metrics in this section, we describe the metrics that we compute. we track both accuracy ( §4.1) and mutual information ( §4.2). ## experimental setup in this section we outline our experimental setup. ## results and discussion overall, our results strongly suggest that morphosyntactic information tends to be highly focal (concentrated in a small set of dimensions) in fast-text, whereas in bert it is more dispersed. averaging across all languages and attributes (fig. 2 ), fasttext has on average 0.306 lbnmi at two dimensions, which is around twice as much as bert at the same dimensionality. however, the difference between the two becomes progressively smaller, reducing to 0.053 at 50 dimensions. a similar trend holds for lba ( §4.1), with an even smaller difference at higher dimensions. on the whole, roughly 10 dimensions are required to encode any morphosyntactic attribute we probed fast-text for, compared to around 35 dimensions for bert. the pattern above holds across attributes (fig. 3 ), and languages (fig. 4 ). there is little improvement in fasttext performance when adding more than 10 dimensions and, in some cases, two fasttext dimensions can explain half of the information achieved when selecting 50. in contrast, while bert also displays highly informative dimensions, a substantial increase in lbnmi can be obtained by going from 2 selected dimensions to 10 and 50. among languages, the only exceptions to this are the indic languages, where bert concentrates more morphological information than fasttext already at 2 dimensions. interestingly, when looking at attributes, our results suggest that fasttext encodes most attributes better than bert (when considering the 50 most informative dimensions), except animacy, gender and number. these findings also hold for lba, where we additionally find little to no gain when comparing lba after 50 dimensions to accuracy on the full vector. visualizing the most informative dimensions for bert and fasttext may give some intuition for how this trend manifests. fig. 5 shows a scatter plot of the two most informative dimensions selected by our probe for english tense in fasttext and bert. we observed similar patterns for other morphosyntactic attributes. both embeddings have dimensions that induce some separability in english tense, but this is more pronounced in fasttext than bert. we cannot clearly plot more than two dimensions at a time, but based on the trend depicted in fig. 2 , we can intuit that bert makes up for at least part of the gap by inducing more separability as dimensions are added. ## related work there has been a growing interest in understanding what information is in nlp models' internal representations. studies vary widely, from detailed analyses of particular scenarios and linguistic phenomena @xcite @xcite @xcite @xcite to extensive investigations across a wealth of tasks @xcite @xcite . a plethora of methods have been designed and applied (e.g. @xcite @xcite to answer this question. probing @xcite ; figure 5 : scatter graph of two most informative fast-text (above) and bert (below) dimensions for english present and past tense. contours belong to our probe. @xcite is one prominent method, which consists of using a lightly parameterized model to predict linguistic phenomena from intermediate representations, albeit recent work has raised concerns on how model parameterization and evaluation metrics may affect the effectiveness of this approach @xcite @xcite . most work in intrinsic probing has focused in the identification of individual neurons that are important for a task @xcite @xcite . similarly, @xcite and @xcite use probing to analyze bert's attention heads, finding some interpretable heads that attend to positional and syntactic features. however, there has also been some work investigating collections of neurons. for example, @xcite observe that different training objectives can affect how focal an intermediate representation is. recently, dalvi et al. (2019) use the magnitude of the weights learned by a linear probe as a proxy for dimension informativeness, -0.4 -0.2 0 0.2 0.4 0.6 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 dimension 211 dimension 80 figure 6: two fasttext dimensions that are informative for portuguese number, but which do not appear jointly gaussian. this dimensions pair was not favored by our gaussian probing model; we found it by modeling p(h | v) with a gaussian-cauchy mixture model. and find dispersion varies depending on linguistic category. @xcite use unsupervised methods to find neurons that are correlated across various models, quantify said correlation, and upon manual analysis find interpretable neurons. in concurrent work in computer vision, @xcite identify units whose local, peak activations correlate with features in an image (e.g., material, door presence), show that ablation of these units has a disproportionately big impact on the classification of their respective features, and can be manually controlled, with interpretable effects. most similar to our analysis is lin-spector ( şahin et al., 2020), a suite of probing tasks that includes probing for morphosyntax. our work differs in two respects. firstly, whereas linspector focuses on extrinsic probing, we probe intrinsically. secondly, the scope of our morphosyntactic study is more typologically diverse (36 vs. 5 languages), albeit they consider more varieties of word representations, such as glove @xcite and elmo (peters et al., 2018)-but not bert. ## conclusion in this paper, we introduce an alternative framework for intrinsic probing, which we term dimension selection. the idea is to use probe performance on different subsets of dimensions as a gauge for how much information about a linguistic property different subsets of dimensions jointly encode. we show that current probes are unsuitable for intrinsic probing through dimension selection as they are not inherently decomposable, which is required to make the procedure computationally tractable. therefore, we present a decomposable probe which is based on the gaussian distribution, and evaluate its effectiveness by probing bert and fasttext for morphosyntax across 36 languages. overall, we find that fasttext is more focal than bert, requiring fewer dimensions to capture most of the information pertaining to a morphosyntactic property. future work. future work will be separated into two strands. the first will focus on how to better model the distribution of embeddings given a morphosyntactic attribute; as mentioned above, this should yield a better probe overall. the second strand of work pertains to a deeper analysis of our results, and expansion to other probing tasks. tal linzen, emmanuel dupoux, and yoav goldberg. 2016. assessing the ability of lstms to learn syntax-sensitive dependencies. transactions of the association for computational linguistics, 4:521-535. nelson f. liu, matt gardner, yonatan belinkov, matthew e. peters, and noah a. smith. 2019. linguistic knowledge and transferability of contextual representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1073-1094, minneapolis, minnesota. association for computational linguistics. rowan hall maudslay, josef valvoda, tiago pimentel, adina williams, and ryan cotterell. 2020. a tale of a probe and a parser. in proceedings of the 58th annual meeting of the association for computational linguistics. association for computational linguistics. florence, italy. association for computational linguistics. kelly zhang and samuel bowman. 2018. language modeling teaches you more than translation does: lessons learned through auxiliary syntactic task analysis. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 359-361, brussels, belgium. association for computational linguistics.