The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation (2021)

Abstract: Temporary syntactic ambiguities arise when the beginning of a sentence is compatible with multiple syntactic analyses. We inspect to which extent neural language models (LMs) exhibit uncertainty over such analyses when processing temporarily ambiguous inputs, and how that uncertainty is modulated by disambiguating cues. We probe the LM’s expectations by generating from it: we use stochastic decoding to derive a set of sentence completions, and estimate the probability that the LM assigns to each interpretation based on the distribution of parses across completions. Unlike scoring-based methods for targeted syntactic evaluation, this technique makes it possible to explore completions that are not hypothesized in advance by the researcher. We apply this method to study the behavior of two LMs (GPT2 and an LSTM) on three types of temporary ambiguity, using materials from human sentence processing experiments. We find that LMs can track multiple analyses simultaneously; the degree of uncertainty varies across constructions and contexts. As a response to disambiguating cues, the LMs often select the correct interpretation, but occasional errors point to potential areas of improvement

## introduction during sentence processing, humans incrementally derive analyses of linguistic expressions. at an initial stage, uncertainty regarding the analysis may be present if no contextual cues have been provided to allow for a unique interpretation; this results in a temporary ambiguity @xcite . for instance, the initial sentence portion of (1) is ambiguous as to the syntactic function of band, which can be a direct object, as in (1a), or an embedded subject as in (1b): like humans, autoregressive neural language models (henceforth, lms) need to deal with temporary ambiguities, as they process text incrementally. in this work, we probe the degree of syntactic uncertainty that lms maintain when processing temporary ambiguities, using generation (sampling) from those models as an analysis tool. lms are trained to output contextual probabilities of words in a next-word prediction task. in spite of this generic objective, these models have been shown to track syntactic information to a remarkable extent @xcite and build context-sensitive internal representations, potentially resolving ambiguities in the input (e.g., @xcite . the behavior of lms on temporary syntactic ambiguities was previously investigated through the lens of word surprisal (e.g., @xcite , providing evidence for incremental syntactic processing in lms. at the same time, the extent that a lm expects each interpretation of an ambiguous input, and therefore its degree of syntactic uncertainty, has not been quantified. in this paper, we generate text from a lm, as a window into the lm's processing of an unfolding sentence. as (1) shows, the completion of a temporarily ambiguous fragment clarifies the intended interpretation. we can use the lm's output probabilities to complete an input -a prompt -and infer how it was initially interpreted. we generate a set of completions by drawing multiple samples from the lm's output distribution. the proportion of completions that are consistent with a certain parse of the prompt is taken to indicate the degree that that parse is expected by the lm. we consider three types of temporary ambiguities in english; for each type, we derive prompts from sentences drawn from psycholinguistic experiments @xcite . we compare the lm's uncertainty on ambiguous prompts as well as unambiguous prompts that vary in the number and location of disambiguat-ing cues. we infer the interpretation of a generated sentence from the labels predicted by a syntactic parser. the lms we analyse are the lstm model released by @xcite and the transformer gpt2 @xcite . 1 we find that in the presence of temporary ambiguity lms can track multiple interpretations in parallel, displaying syntactic uncertainty. the degree of bias towards one analysis varies across ambiguity types and across specific sentences within each type. disambiguating cues often appropriately reduce the lms' syntactic uncertainty in favor of the correct analysis. at the same time, we also identify evidence of disambiguation issues in the resolution of np/z and noun/verb ambiguities. we complement our analyses with a study on the effect of different decoding strategies on syntactic uncertainty, and a comparison of our method to scoring-based analysis. ## related work temporary ambiguities have been studied extensively in psycholinguistics, as window into human incremental parsing: in case of ambiguities, is only one or a subset of the possible parses considered @xcite , or are all parses tracked in parallel, weighted by probability @xcite ? after disambiguation, how is the analysis revised @xcite , and do traces of initial misinterpretations linger @xcite ? we consider analogous questions, focusing on lms instead of humans. several studies have examined the syntactic abilities of lms, through targeted evaluations on specific syntactic phenomena (e.g., @xcite , or by analysing the degree to which syntactic information can be decoded from their internal representations (e.g., @xcite . these studies show that lms track syntax to a large extent, even when not explicitly trained to do so. some previous studies have investigated the behavior of lms on temporary ambiguities focusing on the garden-path effect @xcite , where a high cognitive cost at disambiguation is taken to signal a preference for the alternative analysis. on the one hand, lms' next-word probabilities can be used to model these effects @xcite . on the other, one can test whether lms themselves exhibit garden-path effects, looking at their surprisal at the disambiguation of the sentence @xcite . this was taken to indicate that, as the sentence unfolds, the lm maintains a representation of its syntactic state, akin to an incremental parser. as we do in the present work, futrell et al. (2018) generated completions from language models, but that study focused on lms' awareness of obligatory syntactic events (specifically in the case of relative clause completions). finally, van schijndel and linzen (2021) analyzed the syntactic predictions of lms after temporary ambiguous sentence portions, but limited their analysis to the part-of-speech probabilities for the single word that is expected to follow the ambiguous portion. although it is limited to one word, this approach is related to ours as the expectations of the lm are analyzed classifying its predictions based on syntactic information. ## temporary ambiguities this section describes the types of temporary ambiguity and the materials we use in our study (the full list of materials can be found in appendix d). for each type of ambiguity, we refer to the ambiguous word whose syntactic role determines the analysis of the sentence as the locus of the ambiguity. ## methods language models we evaluate two english language models: the lstm model from @xcite , which was trained on a 80mtokens wikipedia corpus, with 2 hidden layers of 650 units; and the transformer-based gpt2 (small; @xcite , which was trained on the 40gb webtext corpus, with 12 hidden layers of 768 units. 3 both lms are unidirectional: their predictions solely depend on the previous context. in the lstm, this is achieved through recurrent connections, while in gpt2 through masked selfattention. given previous evaluations, more fluent generation can be expected from gpt2, which surpasses the lstm in both number of parameters and size of training corpus. generation starting from a prompt, we generate a completion through stochastic decoding, sampling words from the lm's output distribution. the lm processes the prompt as input and outputs a probability distribution over the next token (1); we sample a word from this distribution sampled (2) and use this word as the next input token. the process is repeated to generate the next tokens. to obtain a sentence completion, we generate a fixed number of tokens from the prompt and crop the text to sentence boundaries identified using the spacy sentencizer. 4 more details are provided in appendix a. in our main experiments, we do not apply techniques that modify the lm's output distribution before sampling (e.g., nucleus sampling; @xcite . in section 8 we analyze how such decoding strategies affect syntactic uncertainty. ## the np/s ambiguity classification to classify a completion as a case of an np or an s analysis, we inspect the dependency label of the locus of ambiguity (direct @xmath0; @xmath1. in some of the completions, the locus of ambiguity forms part of a complex np (e.g., a modifier of another noun, as in the contract clauses): in this case, we use the dependency labels of the following words. to reduce noise from parser errors, we define a heuristic that corrects the most typical type of misclassification (np instead of s). 5 if these rules do not identify the interpre- ## the np/z ambiguity classification as in the case of the np/s ambiguity, the np and z interpretations can be distinguished based on the syntactic role (direct object or subject) of the locus of ambiguity. we therefore employ the same set of rules we used for np/s. the rule correcting cases where a subject is labeled as direct object turns out to be crucial for this classification, as the parser is prone to errors on np/z sentences. a total of 0.6% of of gpt2 completions and 1.4% of lstm completions cannot be identified as either np or z, and are thus discarded from analysis. the agreement between the automatic and manual annotations is high (cohen's @xmath2.86; accuracy of automatic classification with respect to manual: .95); the few divergences are cases where the annotator used the label unclear, which is not available to the parser. ## the noun/verb ambiguity classification to classify the generated sentences, we use the pos label predicted by the parser for the locus of ambiguity. when run on the sentence pairs that the prompts are derived from, the classification sometimes fails. to minimize noise, we discard items where the tagger does not correctly interpret at least one of the sentences associated with that ambiguity (4 in total)-we reason that if the parser makes errors on the original sentences, this is likely to occur also on the sentences generated by the lms from prompts taken from those sentences. this leaves us with 21 prompts for each subtype. the agreement between the automatic labels and the annotator's ones is high (cohen's @xmath3.83; accuracy of automatic classification with respect to manual: .91). differences occur due to tagging errors and sentences annotated as unclear by the linguist. results p (verb) values for each prompt type are shown in figure 3 , while examples of completions for ambiguous prompts -i.e., no cue -the dispersion of values is very high for both models. though there is on average a preference for the noun reading (mean p (verb) ≈ .4), the probability assigned to this interpretation varies across items. this indicates that the lms' initial syntactic preferences on this temporary ambiguity are highly dependent on its instance. for post-locus cue prompts, p (verb) adapts to the disambiguating cues, approaching 0 for the noun reading and 1 for the verb reading. in the latter case, we find dispersion of values, due to some completions labeled as np. a qualitative inspection shows that this occurs due to tagger errors or when a noun reading is licensed in spite of the post-locus cue (e.g., "some metal rings loudly beat into our ears."). overall, we do not find evidence of disambiguation issues on this type of prompt (e.g., (2-3) in by contrast, pre-locus cue prompts, especially in the verb subtype, pose more challenges to the lms. p (verb) values follow the expected trends -decreasing for the noun cases, and increasing for the verb cases -but exhibit variation. in some verb cases, we do not even find a preference for the verb reading (i.e., p (verb) < .5). pre-locus cue prompts are disambiguated by the number of the determiner: these results suggest that the lms are not fully responsive this cue, especially when it points to a verb reading. the lstm shows greater dispersion than gpt2, indicating greater disambiguation difficulty. a qualitative analysis confirms these observations: besides a portion of tagger errors, we find several completions that persist in the incorrect interpretation, violating number agreement (e.g., ( 4b ) and ( 5b ) in in general, we find that classification errors occur more often with the noun/verb ambiguity than with the previously analysed ones. as errors introduce noise, our quantitative estimates should be considered approximate. however, as mentioned earlier, the trends they point to are reliable as they are all confirmed by qualitative analyses of the data. ## effect of decoding strategy in our previous experiments, we generated completions by sampling from the lm's output distribution. we compare this approach to other decoding strategies, focusing on np/s no cue prompts. stochastic decoding variants of stochastic decoding modify the lm output distribution before sampling words from it. restricting or biasing the sampling process to high probability words can improve the quality of the generated text @xcite . in nucleus sampling, the lm distribution is truncated to top-probability words with cumulative probability p (e.g., @xmath4.9). another technique modifies the distribution by dividing the output scores by a parameter ttemperaturebefore softmax is applied: if @xmath5, 1), the distribution is skewed towards high probability words. we inspect how these decoding strategies affect the diversity of interpretations of the completions of an ambiguous prompt. for different combination of values of p and t, we generate set of completions from prompts and report the average p (s) ( maximization-based decoding is the preference for an analysis observed sampling multiple completions reflected by the analysis of the topprobability completion? we use beam search as our decoding strategy, returning the completion ranking highest in probability (beam @xmath7. as in this case we consider only one completion per prompt, p (s) for beam search in ## comparison to surprisal-based analysis previous work has probed the syntactic state of a lm in a temporarily ambiguous sentence by measuring surprisal (negative log probability) at the disambiguation point @xcite : if surprisal is higher than in the unambiguous sentence, we can infer that lm initially preferred the alternative, ultimately incorrect analysis. focusing on np/s sentence pairs, we compare this approach to our method to extract probabilities of analyses. we calculate (1) the difference in word surprisal of the disambiguating word that follows the locus of ambiguity (i.e., the post-locus cue) between the ambiguous and unambiguous sentences of a pair; and (2) the estimated p (s) values on no cue prompts. for both lms, we find a strong negative correlation between the difference in surprisal and p (s) (gpt2: spearman's ρ = -.70; lstm: ρ = -.81; both @xmath8.05): as p (s), as estimated through generation, increases, the lm is less surprised that the s analysis is introduced on the temporarily ambiguous sentence than on its unambiguous equivalent. the fact that the two methods -surprisal and generation-based -are aligned in their estimates corroborates the findings derived from either approach. both methods can be considered alternatives to probe the syntactic state of a lm, and one or the other may be favored depending on the study. surprisal estimates can be easily extracted from a lm, making them a more straightforward tool to apply. however, to analyse the expectations of a lm on an ambiguous input, the surprisal method requires a comparison to its unambiguous counterpart, which is not needed with our method. this allowed us to study a lm's behavior also on unambiguous inputs, providing insights about the sensitivity to disambiguating cues. moreover, generating text exemplifies the lm's expectations over the sentence, which can reveal phenomena that may not be clear on the basis of surprisal estimates alone. an example are the ungrammatical blended continuations of np/z prompts (e.g., "*as the couple danced the tango began, the paparazzi swooned."). ## discussion and conclusions in this work, we have probed the syntactic uncertainty of lms by generating sentence completions from the lms. our results contribute to research on the syntactic processing of lms, quantifying the extent that one analysis of the input is implicitly entertained by the model. we find that when processing temporary syntactic ambiguities, lms typically exhibit uncertainty about the analysis of the input; that is, they simultaneously consider multiple analyses to be viable. in line with previous analyses of lms on gardenpath sentences, on np/s and np/z sentences we detected a general preference for the np analysis (e.g., van schijndel and linzen 2018). but, while for np/z this preference was near-absolute and consistent across cases, on np/s and noun/verb ambiguities the lms' behavior varied across specific instantiations of the ambiguity. a promising direction for future work would be to determine whether this inter-item variation mirrors human expectations @xcite and/or corpus statistics (for noun/verb, e.g., how often suit is used as noun vs. verb). a plausible hypothesis is that a lm acquires default preferences for analyses from regularities in the training data. when disambiguating cues are given as part of the input, the lms tend to display the correct behavior: we observe the appropriate shifts in uncertainty in favor of the disambiguated parse, providing further evidence of their context-sensitivity. at the same time, certain issues arise for np/z and noun/verb ambiguities, suggesting that there is room for improvement in the lms' responsiveness and adaptation to disambiguating cues. on np/z ambiguities, some generated completions exhibit confusion over the sentence structure. this behavior calls to mind lingering effects of initial misinterpretations found in humans @xcite . on noun/verb ambiguities, the lms sometimes failed to use the number of a preceding determiner as cue for the correct parse. this may be due to difficulties in tracking number agreement in these constructions (in contrast to results by @xcite and subsequent research on subject-verb agreement), or in generally overriding a default preference for the incorrect analysis. using generation proved to be an informative tool to inquire a lm's uncertainty over an unfolding sentence, and could be used also to inquire more types of ambiguities (e.g., semantic). yet, there are some challenges to our proposed methodology. first, relying on an automatic classification of sentences can introduce noise: ambiguities can be difficult for nlp systems even when explicitly trained to analyse expressions @xcite . second, we could not automatically detect ungrammatical completions or with an unclear analysis (the parser always returns an output), whereas it may be useful to be identify these cases. while these issues did not prevent us from inferring the main trends in the lms behavior, all confirmed by qualitative inspections of the data, we look forward to future work that will attempt to overcome the aforementioned limitations.