Probing for Incremental Parse States in Autoregressive Language Models (2022)

Abstract: Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax. This work evaluates the extent to which this behavior arises as a result of a learned ability to maintain implicit representations of incremental syntactic structures. We extend work in syntactic probing to the incremental setting and present several probes for extracting incomplete syntactic structure (operationalized through parse states from a stack-based parser) from autoregressive language models. We find that our probes can be used to predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior. This suggests implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.

## introduction the behavior of large-scale autoregressive neural language models (alms) appears to demonstrate impressive command of syntax @xcite @xcite @xcite . to what extent can we attribute this behavior to a model's maintaining and updating representations of incremental syntactic structures? interpretability work on bidirectional masked language models suggests that neural models of language may learn to encode syntactic structure through the geometry of their word embedding space. for example, @xcite demonstrate that the dependency parse tree of a sentence can be decoded by finding the minimum spanning tree on pairwise syntactic distances regressed from linearly transformed contextualized word embeddings. unlike the models considered in @xcite , alms are unidirectional, and the decision problem of incrementally deriving global syntactic structures has several nuances not present in the bidirectional setting. sentence prefixes (e.g. "i watched her duck ...") can be ambiguous in their intended meaning in ways that are disambiguated by their suffixes (e.g. "quack" vs. "under the table"). thus, incremental processors must maintain a belief state of parses that can be flexibly updated on the basis of future input. insofar as language in the real world (e.g., speech, text, sign) is processed sequentially, incremental disambiguation of structure and meaning is a task humans solve in everyday cognition, seemingly effortlessly @xcite @xcite . alms have been shown to recapitulate crucial features of human sentence processing @xcite and their representations moreover have been shown to align with language processing in the brain better than their bidirectional counterparts @xcite . we hypothesize that alms learn and maintain correlates of incremental syntactic structures which play an important role in mediating model behavior. we investigate this hypothesis through the lens of counterfactual probing, i.e., by learning classifiers over hidden states of pretrained alms to predict linguistic properties, and then using the probes to intervene on model representations. we present a suite of probing architectures for decoding belief states of incremental structures from pretrained transformer alms given the model's hidden state, each of which embodies a different hypothesis for how incremental parse states might be encoded. we validate (and adjudicate between) our probes by using them to predict and control model behavior. our results suggest that alms, through pretraining at scale, learn stack-like representations of syntax and use them in interpretable and controllable ways. ## motivating study prior work has developed methods for extracting syntactic parses from neural language models given global context. in particular, @xcite learn a distance function parameterized by a linear transformation (b) of the word embedding space of bert, in an initial study, we run hewitt and manning (2019)'s structural probe on the hidden states of gpt2 @xcite , a unidirectional alm, and find that gpt2's hidden states recover gold tree structures almost as well as similarly-sized bidirectional models (figure 1 ). for example gpt2-xl matches bert-base in correlation with syntactic distances (dspr.; .85 for both models). we further observe that this probe can recover the correct structures even on sentences with ambiguous prefixes, such as the 'np/z' ambiguity shown in figure 2 . in this ambiguity, the shared prefix ("even though the band left the party") is consistent with at least two interpretations: one in which the underlined span ("the party") is a direct object noun phrase (np) of the verb "left", and one in which the underlined content is the subject of a new noun phrase and "left" has a zero (z) complement. this is a well-studied phenomenon in psycholinguistics, and humans make rapid and accurate inferences in this setting @xcite . it is not at all obvious how a strictly left-to-right model, whose representations of the words in context are static (i.e., not affected by future words), can maintain a prefix representation that are dynamic enough to encode global syntactic distances with high accuracy. we visualize the evolution of pairwise syntactic distances for the two sentences in figure 2 along with the minimum spanning trees in figure 1 (right). we observe that parses probed from sentence prefixes up until the disambiguators are consistent with the np-complement parse (i.e., where "party" is attached to "left"), which is in line with previous work that analyzes gpt2's behavior on the np/z ambiguity @xcite . after observing "went," gpt2 emits a representation whose distance is interleaved between "left" and "party" (i.e., δ b (h party , h went ) < δ b (h party , h left )), thus altering the minimum spanning tree to be consistent with the zero-complement parse. this analysis provides an initial hypothesis for how incremental parsing arises in alms from the perspective of syntactic distance-alms emit word representations that maintain syntactic uncertainty that can be exploited by later emissions, effectively editing inferred dependencies and bypassing the need to re-position past emissions. however, this analysis also reveals several limitations that preclude it as a viable model of incremental parsing in alms: (1) this method assumes a spanning-tree and therefore cannot represent parses with open nodes, and (2) gpt2's behavior is consistent with a probabilistic parallel parser as it seems to entertain both possible parses until seeing the disambiguating words, but such a pure distancebased probe is not inherently probabilistic. 1 in the following section, we develop several parameterizations of a probabilistic incremental parser that can represent incomplete syntactic structures. ## an incremental-parse probe for autoregressive language models we argue that the stack representation of shiftreduce parsers provides a natural way to represent incomplete tree structures. in this paper, we work with the generative arc-standard dependency formalism @xcite , which maintains a stack of generated subtrees @xmath0, s 2 , s 3 , ...] (where the root of subtree s i is a word), and makes one of the following actions at each time step: left-arc: pop the top two nodes, create a new subtree by adding an arc s 1 → s 2 , and push the new subtree onto the stack, right-arc: pop the top two nodes, create a new subtree by adding an arc s 2 → s 1 , and push the new subtree onto the stack, gen: generate the next token. generation starts with only root (an embedding learned independently for each probe) on the stack, @xmath1, and terminates when only root is left on the stack and the period token has been generated. note that an action sequence a fully specifies a projective dependency structure. 2 hence, letting @xmath2, we use @xmath3.e., incomplete) tree structure state after emitting w t . 3 we design several probes that place distributions over action trajectories -p @xmath4. ## experiments we evaluate each of our probes on their parsing performance over the ptb test split as well as their ability to predict and control model behavior. ## related work our work is related to the growing literature quantifying the syntactic sensitivity of neural models of language at behavioral @xcite @xcite and representational @xcite similar to our work but in semantics, @xcite use probes to interpret text-encoders as maintaining an evolving semantic information state while interacting with a text-based game. relatedly, our work is among others that attempt to control neural models of language for counterfactual analysis or controllable ai more generally @xcite @xcite @xcite . lastly, we expect our work to be of consequence to computational psycholinguistics, where neural language models and incremental parsers have been historically prominent as candidate cognitive models @xcite @xcite @xcite . this work draws representational parallels between these two models. ## conclusion motivated by recent work showing human-like incremental parsing behaviour in alms, this work extends structural probing to the incremental setting. we find that autoregressive language models can perform on par with bidirectional models in terms of global parsing metrics despite the fact that an alm's representation cannot condition on fu-ture words. we present several hypotheses of the representation of incremental structure in alms, instantiate them in probe architectures, and evaluate their explanatory power across a wide range of experiment types: parsing performance, predicting model behavior, and counterfactual analyses. cumulatively, map performs the best, which is significant because this architecture incorporates strong constraints based on linguistic theory (i.e., stack-based representations) while still being relative agnostic (compared to the geometric probe) to the details of how parse probabilities might be encoded. this suggests that despite a lack of explicit feedback, language models not only rediscover linguistic structure through pretraining but learn to make 'inferences' over this structure in real-time comprehension.