Linguistic Dependencies and Statistical Dependence (2021)

Abstract: Are pairs of words that tend to occur together also likely to stand in a linguistic dependency? This empirical question is motivated by a long history of literature in cognitive science, psycholinguistics, and NLP. In this work we contribute an extensive analysis of the relationship between linguistic dependencies and statistical dependence between words. Improving on previous work, we introduce the use of large pretrained language models to compute contextualized estimates of the pointwise mutual information between words (CPMI). For multiple models and languages, we extract dependency trees which maximize CPMI, and compare to gold standard linguistic dependencies. Overall, we find that CPMI dependencies achieve an unlabelled undirected attachment score of at most \approx 0.5 . While far above chance, and consistently above a non-contextualized PMI baseline, this score is generally comparable to a simple baseline formed by connecting adjacent words. We analyze which kinds of linguistic dependencies are best captured in CPMI dependencies, and also find marked differences between the estimates of the large pretrained language models, illustrating how their different training schemes affect the type of dependencies they capture.

## introduction a fundamental aspect of natural language structure is the set of dependency relations which hold between pairs of words in a sentence. such dependencies indicate how the sentence is to be interpreted and mediate other aspects of its structure, such as agreement. consider the sentence: several ravens flew out of their nests to confront the invading mongoose. in this example, there is a dependency between the verb flew and its subject ravens, capturing the role this subject plays in the flying event, and how it controls number agreement. all modern linguistic theories recognize the centrality it is im p o s s ib le t o k n o w w h e t h e r t h a t t h e o r y is r e a li s t ic . it is cpmi im p o s s ib le t o k n o w w h e t h e r t h a t t h e o r y is r e a li s t ic . it is impossible to know whether that theory is realistic . ## background pointwise mutual information @xcite is commonly used as a measure of the strength of statistical dependence between two words. formally, pmi is a symmetric function of the probabilities of the outcomes x, y of two random variables x, y , which quantifies the amount of information about one outcome that is gained by learning the other: in our case, the observations are two words in a sentence (drawn from discrete random variables indexed by position in the sentence, ranging over the vocabulary). pmi has been used in computational linguistic studies as a measure of how words inform each other's probabilities since @xcite . 2 much earlier work on unsupervised dependency parsing (e.g., @xcite @xcite @xcite used techniques involving maximizing estimates of total pointwise mutual information between heads and dependents, or maximizing the conditional probability of dependents given heads (these two objectives can be shown to be equivalent under certain assumptions; see §c). while such pmiinduced dependencies proved useful for certain tasks (such as identifying the correct modifier for a word among a selection of possible choices; de paiva alves, 1996), purely pmi-based dependency parsers did not perform well at the general task of recovering linguistic structures overall (see discussion in @xcite . the recent advent of pretrained contextualized lms (such as bert, xlnet; @xcite provides an opportunity to revisit the relationship between pmi-induced dependencies and linguistic dependencies. these networks are pretrained on very large amounts of natural language text using masked language mod-that theory is realistic . ## contextualized pmi dependencies linguistic dependencies are highly sensitive to context. for example, consider the following two sentences: i see that the crows retreated, and the mongoose pursued by crows retreated. in the first there is a dependency between retreated and crows, and in the second there is not. however, pmi between two words in a sentence is strictly independent of the other words in that sentence. here we define contextualized pointwise mutual information (cpmi) as the conditional pmi given context, which we estimate using pretrained contextualized lms. a contextualized lm m provides an estimate for the probability of words given context, which we use to define cpmi m between two words w i and w j in a sentence w as ## evaluating cpmi dependencies in this section, we analyze the degree to which cpmi-inferred dependencies from pretrained lms resemble linguistic dependencies. ## delexicalized pos-cpmi dependencies in this second experiment we estimate cpmidependencies over part-of-speech (pos) tags, rather than words. in the unsupervised dependency parsing literature there is an ample history of approaches making use of gold pos tags (see e.g., @xcite @xcite . additionally, a traditional objection to the idea of deducing dependency structures directly from cooccurrence statistics, beyond data sparsity issues, is the possibility that "actual lexical items are too semantically charged to represent workable units of syntactic structure" (as phrased by @xcite . that is, perhaps words' patterns of co-occurrence contain simply too much information about factors irrelevant to dependency parsing, so as to drown out the information that would be useful for recovering dependency structure. according to this line of thinking, we might expect linguistic dependency structure to be better related to the statistical dependencies between the categories of words, rather than lexical items themselves. thus a version of cpmi calculated over pos tags would be predicted to achieve higher accuracy than the cpmi calculated that theory is realistic . that theory is realistic . ## analysis in this section we outline main takeaways from a more detailed examination of the results from § §4-5, including additional analysis in §a.4. no relation label has high uuas in figure 5 , recall accuracy is plotted against gold dependency arc label. 9 when examining all lengths of dependency together (left) recall accuracy would seem to be correlated with mean arc length. but, filtering out all the gold arcs of length 1 (49% of arcs), we see that there is not a strong overall effect of arclength on mean accuracy for @xmath0. for most dependency labels, cpmi accuracy from each of the models is above the random baseline, but at or below to the connect-adjacent baseline. exceptions to this trend include dependency labels dobj (direct object), xcomp (which connects a verb or adjective to the root of its clausal complement). for wordpairs in these relations, cpmi estimates (xlnet in particular) achieve higher accuracy than the baselines. however, even in these cases, cpmi dependencies do not perform at a level that could be considered successful for an unsupervised parser. this is contrary to what would be expected if cpmi-dependencies were in a strong correspondence with linguistic dependencies, even if this only held for certain types of linguistic dependency. when considering arcs of @xmath1, there is no dependency arc label which has uuas above 0.5 from any of the models. more complete results including the other models not shown in figure 5 are given in uuas is not correlated with lm performance figure 6 shows per-sentence uuas plotted against log pseudo-perplexity (ppl) for bert and xlnet models (results are similar for other models; see §a.4.3, figure 9 ). these results show that correspondence between cpmi-dependencies and linguistic dependencies isn't higher on sentences on which the models are more confident. we also examined the accuracy of cpmi dependencies during training of bert (base uncased) from scratch. figure 11 (in appendix) shows the average perplexity of this model at checkpoints during training, along with average uuas of induced cpmi structures. uuas reaches its highest value before perplexity plateaus. fitting a linear regression, the coefficient of determination r 2 is very close to 0 for all models (here bert and xlnet are shown; other models are in figure 9 ) we should also stress that, throughout this paper, uuas is not a measure of lm quality. rather, it simply measures how well patterns of statistical dependence captured by the lm align with linguistic dependencies. better alignment may not be related to better language modelling. dependencies differ between lms dependency structures extracted from the different pretrained lms show roughly similar overall uuas, though the models agree with each other on only 25-48% of edges. they agree with the noncontextualized word embedding model word2vec at just slightly lower rates (21-27%), while agreeing with the linear baseline at higher rates (34-57%). see §a.4.1 and for these details. in particular, cpmi dependencies from all the models connect adjacent words more often than the gold dependencies do, but this effect is much more pronounced for bert models than for xlm, and xlnet models (figure 7 ). a possible reason for this difference lies in the way these models are trained. xlnet is trained to predict words according to randomly sampled chain rule decompositions, enforcing a bias to be able to predict words in any order, including longer dependencies. xl-net's probability estimates for words may therefore be sensitive to a larger set of words, rather than mostly the adjacent ones. whereas bert, trained with a less constrained masked lm objective, has probability estimates that are evidently more sensitive to adjacent words. ## related work probing pretrained embeddings in the past few years, a substantial amount of literature has emerged on probing pretrained language models (in the sense of e.g. @xcite , wherein a presumably weak network (a probe) is trained to extract linguistic information (in particular, dependency information, in e.g. @xcite from pretrained embeddings. extracting cpmidependencies differs from training a dependency probe in that it is entirely unsupervised, and is motivated by a specific hypothesis-about the relationship linguistic dependencies have with statistical dependence. nonparametric probing a number of other recent works have taken an unsupervised approach to investigating syntactic structure encoded by pretrained lms, largely focusing on self-attention weights (e.g. @xcite @xcite . very recently, zhang and hashimoto (2021, concurrent with this paper) examined conditional dependencies implied by masked language modelling using a nonparametric method similar to our cpmi, using bert to estimate conditional pmi (and conditional mi) between words. they extract maximum spanning trees, and report uuas on wsj dependency data. their results are similar to those reported here: namely, scores are much higher than a chance baseline, but close to a connect-adjacent baseline. while their numerical results are similar, their interpretation differs somewhat. given our analysis, we find less reason for optimism about the prospects of unsupervised dependency parsing directly from probability estimates by pretrained lms. ## discussion in this paper we explored the connection between linguistic dependency and statistical dependence. we contribute a method to use modern pretrained language models to compute cpmi, a contextdependent estimate of pmi, and infer maximum cpmi dependency trees over sentences. we find that these trees correlate with linguistic dependencies better than trees extracted from a noncontextual pmi estimate trained on similar 10 we thank an anonymous reviewer for alerting us to this work.