Syntactic Substitutability as Unsupervised Dependency Syntax (2023)

Abstract: Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. In this work, we explore the hypothesis that syntactic dependencies can be represented in language model attention distributions and propose a new method to induce these structures theory-agnostically. Instead of modeling syntactic relations as defined by annotation schemata, we model a more general property implicit in the definition of dependency relations, syntactic substitutability. This property captures the fact that words at either end of a dependency can be substituted with words from the same category. Substitutions can be used to generate a set of syntactically invariant sentences whose representations are then used for parsing. We show that increasing the number of substitutions used improves parsing accuracy on natural data. On long-distance subject-verb agreement constructions, our method achieves 79.5% recall compared to 8.9% using a previous method. Our method also provides improvements when transferred to a different parsing setup, demonstrating that it generalizes.

## introduction in recent years, large pretrained language models (llms), like bert @xcite , have led to impressive performance gains across many natural language processing tasks. this has led to a line of work attempting to explain how natural language understanding might occur within these models and what sorts of linguistic properties are captured. going one step further, we explore the hypothesis that syntactic dependencies can be extracted from llms without additionally learned parameters or supervision. previous work on syntax has tested (1) whether language models exhibit syntactically dependent work done while the first author was at mcgill university. the code and data can be found at https://github.com/mcgillnlp/syntactic-substitutability . figure 1 : syntactic relations represent intersubstitutability: the nominal subject 'kids' can be substituted with another noun 'girls' without affecting syntactic wellformedness. swapping the verb 'run' with 'play' and the preposition 'in' with 'to' are other examples for this sentence. substitutions define a set of sentences that can be used to model this property during parsing. instantiating the latter approach, @xcite train a probe in order to project model representations of words into a new vector space where a maximum spanning tree algorithm (mst) can be applied to induce the desired syntactic parse. however, it is not clear whether such a method relies solely on the information already present in the model, or whether the trained probe is contributing model-external knowledge @xcite . a less ambiguous approach is instead to use model-internal distributions directly as input to the tree induction algorithm without additional training. in this vein, previous work has made use of attention distributions from transformer-based lms (e.g. @xcite . this parallels observations made by @xcite that certain attention heads in bert correspond to dependency relations. however, given the large amount of information present in llms, nothing constrains the extracted parses to be syntactic parses when representations are used directly. in this paper, we propose a novel method to distill syntactic information by modelling a general property of syntactic relations which is independent of any specific formalism. this property, syntactic substitutability, captures the intuition that syntactic structures define categories of intersubstitutable words -illustrated in figure 1 . we make use of this notion by enumerating syntactically invariant sentences which can then be exploited together to induce their shared syntactic structure. our primary goal is to investigate whether modeling syntactic substitutability better extracts syntactic information from attention mechanisms, resulting in more accurately induced parses. inducing structures without relying on specific annotation schemata also allows us to better understand how the syntactic relations represented in a model might be similar to existing theoretical proposals. we demonstrate that our method, syntactic substitutability as unsupervised dependency syntax (ssud) leads to improvements in dependency parsing accuracy. as more substitutions are used, parsing scores increase. we also quantitatively show that the induced parses align more with an annotation schema where function words are treated as heads (experiment 1). when tested on long-distance subject-verb agreement constructions, ssud achieves an increase in recall @xmath0. we also demonstrate how our method can be transferred to and improve different parsing algorithms, showing that ssud generalizes effectively (experiment 3). ## related work our work is related to a long tradition of unsupervised syntactic parsing, for example, the generative dmv model @xcite and the compound probabilistic context free grammar model @xcite for constituency parsing. motivations for this stem from the idea that syntactic parses can be induced by computing an mst over scores between words in a sentence that represent how likely it is for two words to be in a syntactic dependency or how likely a span corresponds to a syntactic constituent. we seek to work with scores directly derivable from llms, following previous proposals which have used attention distributions of transformerbased models @xcite to calculate scores. examples in dependency parsing include raganato and tiedemann (2018) who use neural machine translation models, and htut et al. (2019) and limisiewicz et al. (2020) who use bert. for constituency parsing, kim et al. ( 2020 ) propose a method based on the syntactic similarity of word pairs, calculated as a function of their attention distributions. the use of attention distributions for parsing tasks is supported by the observation made by @xcite that certain attention heads correspond to syntactic dependencies in bert. however, they also observe that attention heads do not only capture syntactic information, but also other relationships like coreference. our method proposes syntactic substitutability to address this issue, as motivated in the next section. previous work from @xcite proposes an algorithm to select syntactically relevant heads, but we contend that this is maximally effective if the distributions within a single head also only capture syntactic information. we return this idea in experiment 3 and investigate the effect of adding ssud to this algorithm. other complementary methods use bert's contextualized representations to perform parsing. for example, @xcite propose a method which calculates scores based on the 'impact' that masking a word in the input has on the representations of the other words in the sentence, and @xcite train a probe with supervision to project vectors into a 'syntactic' space. another approach is using bert's masked language modeling objective to compute scores for syntactic parsing. work in this vein include @xcite and @xcite , motivated by a hypothesis stemming from @xcite that syntactic dependencies correspond to a statistical measure of mutual information. lastly, while the non-parametric use of substitutability for syntactic parsing has not been previously proposed, parallels can be drawn to work in language model interpretability. @xcite show that bert systematically learns to use word-order information to syntactically distinguish subjects and objects even when the respective nouns are swapped. this is a special case of substitution: the syntactic structure of these sentences is invariant despite their non-prototypical meaning. we take these results to mean that bert has a knowledge of syntax that is robust to substitution, and as a result, substitutability may be an effective constraint. ## syntactic substitutability and dependency relations in this section, we propose a method to model the formalism-neutral objective of substitutability within the induction of syntactic structures. this notion is often explicitly included in the definition of syntactic grammars, see for example @xcite and @xcite . intuitively, intersubstitutable words form syntactic categories which syntactic relations operate on. ## datasets and models as in previous work (e.g. hoover et al., 2021), we assess our method using two gold-standard en-glish dependency parsing datasets: (1) the sentence @xmath1; 389 sentences), and (2) the english section of the parallel universal dependencies dataset annotated with universal dependencies @xcite ) (en-pud; 1000 sentences). additionally, we assess our parses with surface-syntactic universal dependencies annotations @xcite ; see §5.4). we use section 21 of the penn treebank as a validation set. we also test our method on a more difficult, long-distance subject-verb agreement dataset from marvin and linzen (2018) (see experiment 2). the language model investigated here is bert @xcite , a transformer-based language model. specifically, we focus on the bert-base-uncased model, which has 12 layers with 12 self-attention heads each (110m parameters). to test generalization with respect to model size, we also use bert-large-uncased (336m parameters) in experiment 1. ## conclusion the results across the three experiments show that there is merit to modeling the property of syntactic substitutability when inducing syntactic depen-dency structures. indeed attention distributions do capture a surprising amount of syntactic information, despite never being trained to do so. with substitutability as a constraint, we can better make use of this information for unsupervised parsing and better understand the extent to which this property is learned by attention mechanisms. our results also suggest that previous probing results on attention mechanisms using single sentences may underestimate the amount of syntactic information present. we show that ssud effectively transfers to new parsing setups and to different datasets. a potential next step is to use this method cross-linguistically in order to better understand the representation of different languages within multilingual models. in addition to uses for nlp and computational linguistics, these interpretable, model-intrinsic structures may provide a new source of data for work in theoretical linguistics and syntactic annotation, and as further confirmation of the emergence of syntaxlike structures in neural network language models.