Garden Path Traversal in GPT -2 (2022)

Abstract: In recent years, large-scale transformer decoders such as the GPT-x family of models have become increasingly popular. Studies examining the behavior of these models tend to focus only on the output of the language modeling head and avoid analysis of the internal states of the transformer decoder. In this study, we present a collection of methods to analyze the hidden states of GPT-2 and use the model’s navigation of garden path sentences as a case study. To enable this, we compile the largest currently available dataset of garden path sentences. We show that Manhattan distances and cosine similarities provide more reliable insights compared to established surprisal methods that analyze next-token probabilities computed by a language modeling head. Using these methods, we find that negating tokens have minimal impacts on the model’s representations for unambiguous forms of sentences with ambiguity solely over what the object of a verb is, but have a more substantial impact of representations for unambiguous sentences whose ambiguity would stem from the voice of a verb. Further, we find that analyzing the decoder model’s hidden states reveals periods of ambiguity that might conclude in a garden path effect but happen not to, whereas surprisal analyses routinely miss this detail.

## introduction openai's release of gpt-3 marked a major step in the field of massive language models, whose ability to generate news articles indistinguishable from those written by humans provides a salient example of the many social and political implications of these models @xcite @xcite . within 2 years of bert's release, over 150 studies have investigated bert's structure, exploring how its internal representations enable powerful and flexible language comprehension @xcite figure 1 : hidden state relations (top: cosine similarity, middle: manhattan distance, bottom: surprisal difference) between negated and non-negated forms of garden path and unambiguous sentences. the ambiguous verb "walked" primes the effect later in the sentence, while the unambiguous "taken" avoids it. the verb "lit" introduces a similar ambiguity, which hidden state metrics show but surprisal misses because the next word "by" does not trigger a garden path effect. @xcite @xcite . a few such studies include a decoder model in the set of models examined, but do not specifically design their analyses around this type of architecture @xcite . meanwhile, studies exploring gpt models alone tend to focus on properties of text generated from its language modeling head, and do not analyze the internal representations of the model in depth @xcite . the few studies that explore the hidden states of gpt-2 suggest an under-utilization of its massive latent space as representations are dominated by the presence of rogue dimensions @xcite @xcite . as massive decoder models become more ubiquitous and powerful, it will become ever more important to understand the internal processes by which they generate content so they can be streamlined and improved upon. in this paper, we use garden path traversal as a case study to demonstrate the value of directly analyzing properties of the embedding space in transformer decoder models. a garden path sentence is one where the parse that a reader expects at some point within the sentence is proven incorrect by the end of the sentence. we choose to explore this syntactic effect specifically because we believe the intuitive reaction a human has when reading such sentences provides a helpful frame for analyzing the behavior of a neural language model experiencing the same effect. we expect that by looking at the hidden states from which next word likelihoods are computed, we can observe the same patterns that surprisal analysis reveals, while uncovering more nuanced trends that surprisal misses because it depends on the joint distribution of the hidden state and the next word. moreover, we expect that manhattan distances will exhibit less variance in the effect of negating a given sentence type than either surprisals or cosine similarities after a zero-mean transformation, because manhattan distances are resilient to extreme values in a single dimension @xcite . on the other hand, the next word likelihoods used to compute surprisal tend to depend heavily on these dimensions, while the zero-mean translation required to create meaningful angular differences around the origin mean that a few extreme dimensions expose cosine similarities to similarly high variances @xcite . by analyzing how gpt-2 sequentially embeds tokens in space, we are able to identify how gpt-2 internally handles different garden path effects. specifically, we show that gpt-2 recognizes potential but unrealized garden path effects using metrics that examine the model's hidden states, whereas surprisal analysis fails to reveal this finding. we argue that analysis of a decoder model's hidden states enables more robust analysis than can be done using the next word likelihoods alone, which themselves are distilled from these hidden states. the contributions of this study are as follows: ## results & discussion our analysis reveals several properties of gpt-2's experience of the garden path effect. across all sentence types, manhattan distances and cosine similarities show that the model reacts more heavily to negation of garden path sentences than it does to these sentences' unambiguous counterparts, as is reflected by surprisal analyses done here and in previous studies @xcite . although our surprisal baselines mirror the trends seen in manhattan distance, using manhattan distances provides more consistent results compared to surprisal analysis. our results demonstrating exceedingly high variance in the surprisal analysis is in line with the findings of @xcite , who use surprisal to show that gpt-2 performs especially poorly and inconsistently on garden path effects. on the other hand, the high-level trends we expected to see are present across all metrics, with negation causing a less pronounced difference in unambiguous and blocked sentences than it does in garden path sentences. whereas figure 2 shows manhattan distances to have relatively low variance compared to the other metrics we examine, cosine similarity and surprisal suffer from very high variances within each sentence form. we believe that this is due to manhattan distance's resistance to gpt-2's rogue dimensions, which dominate cosine similarities after the zero-mean transformation be-cause even relatively minor differences in a few of these dimensions will have a much more substantial effect on the angular difference than more major differences in many smaller dimensions would @xcite . figure 1 illustrates how these high level trends appear within a given sentence. here, the verb "tried" triggers the garden path effect, which is directly preceded by a spike in manhattan distance and a dip in cosine similarity between the garden path and negated sentence forms as the model anticipates different continuations: in the garden path sentence the model is likely to predict some sort of punctuation or conjunction to end the clause, while in the negated form the model expects a verb to complete the clause that the ambiguous verb is subordinate to. an interesting property of the specific example in figure 1 , "the captive walked into the dark cave lit by torches tried to escape but failed." is that the verb "lit" also triggers a momentary garden path effect; the sentence could, for instance, simply continue, "the captive walked into the dark cave lit the torches." in the first case, the verb "lit" is a reduced relative of "that was lit", which refers to the cave, whereas in the second case the captive is the subject of "lit", which is the main verb of the sentence and thus would trigger a garden path effect. of course, the immediate next word "by" eliminates this possibility, which a human reader notices quickly enough that they do not experience the effect, but the decoder's causal attention mask allows it no such foresight. this possibility is thus worth considering because it helps explain why there is a spike in manhattan distance and a dip in cosine similarity at the preceding word, "cave". we believe the relative shallowness of the dip in cosine similarity before "lit" is due to the possible mv/rr ambiguity of the word, since even in the garden path case where punctuation can be expected, a verb such as "lit" can preempt an adjectival clause as it does here ("lit by torches"). this puts it in a curious superposition between introducing and triggering a garden path effect, at least until the next word "by" resolves this ambiguity. notably, in the garden path form neither hidden state metric returns to its baseline value until after the verb "lit", because the model expects this verb leads a subordinate clause whereas in the negated form it considers both possibilities. figure 5 : hidden state relations (top: cosine similarity, middle: manhattan distance, bottom: surprisal difference) between negated and non-negated forms of garden path and unambiguous sentences. the ambiguous verb "handed" primes the effect later in the sentence, while the unambiguous "given" avoids it. like in figure 1 , all metrics catch the garden path effect at the verb "grumbled", but only cosine similarity and manhattan distance anticipate the possible effect at "littered" on the other hand, the verb "tried" is not ambiguous in this way. the clause it might preempt, such as "tried for murder", would be improperly placed and awkward. thus, the model's hidden states prior to the verb "tried" in the garden path and negated sentences are nearly orthogonal to each other, whereas they bear more similarity right before the verb "lit". although surprisal spikes at the verb "tried" as well, the surprisal trajectory does not reflect the possible effect at "lit", illustrating the inadequacy of using this metric alone. this ambiguity, however, is only reflected in manhattan distance and cosine similarity, and demonstrates how internal metrics can help us understand relationships between the model's syntactic states that surprisal analysis alone would miss. we argue that surprisal misses this effect because it depends entirely on the word that triggers a garden path effect rather than the state of the decoder prior to the trigger, which more holistically encodes the ambiguity that creates the environment where a garden path effect can occur. in this case, it seems that the prior likelihood of various constructions after 'lit' leads gpt-2 to suspect it leads a reduced relative clause (i.e. 'that was lit') and is not the sentence's main verb, but our distance metrics between the model's hidden states prior to this potential trigger show that gtp-2 nonetheless registers the possibility of a garden path effect. however, more work is needed to explore how exactly each of these metrics measures this abstract concept, and how sensitive they are to other syntactic and semantic effects. we highlight two other examples of this phenomenon. in figure 3 , the preceding clause "the horse raced past the barn filled [...]" could be completed either with the actual continuation "with tractors [...]", or with a direct object for "filled", for instance: "the horse raced past the barn filled her trainer with admiration". in figure 5 , the clause "the lawyer handed the report littered [...]" could likewise either continue as shown, "with profanity [...]", or could instead be the main verb of a shorter sentence: "the lawyer handed the report littered as he walked across the street". whereas surprisal overlooks the temporary ambiguities in these examples because it relies on the next word to trigger the garden path effect, hidden state metrics reveal them because they can directly measure the ambiguity itself. our analysis revealed a few unexpected results. most prominent among these is the extent to which the addition of the negating token ("that") to unambiguous np/s sentences leaves the hidden representation of the sentence unchanged. across all metrics, the negated and garden path forms of np/s sentences are closest together, showing that except in cases where it resolves a clear ambiguity, the negating token in these sentences contributes very little to the model's internal representation. the blocked form of np/z sentences shows a similar indifference to the negating token (in this case, the addition of a comma), which curiously does not extend to the unambiguous form. ## conclusion this paper presents a suite of methods to analyze the internal representations of transformer decoder language models such as gpt-2, taking advantage of a richer reflection of the model's internal process than can be ascertained from the output of the language modeling head alone. we use manhattan distance and cosine similarity between the hidden states of gpt-2 to show that the model is affected by garden path effects in ways that are predictable based on human readers' difficulty with these sentences. although conventional surprisal analysis mirrors these effects in many cases, it exhibits higher variance than manhattan distances, and misses certain nuances. on this basis, we argue that linguists should look to a decoder model's hidden state for a more complete picture of syntactic state than surprisal alone can report. our belief is that manhattan distance should be the preferred metric for this type of analysis, but we invite researchers to introduce and explore new metrics to challenge this hypothesis. we hope that these early insights will help inspire deeper exploration of the hidden states of decoder-only language models. possible directions for future work could more closely examine how information is transformed across different decoder layers within gpt-2, and might explore causes for differences between manhattan distance and cosine similarity trajectories. the methods introduced in this study can also be used to explore decoder models' handling of arbitrary syntactic phenomena beyond garden path effects, such as verb subordination or negative polarity item licensing.