When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality (2024)

Abstract: Incremental models that process sentences one token at a time will sometimes encounter points where more than one interpretation is possible. Causal models are forced to output one interpretation and continue, whereas models that can revise may edit their previous output as the ambiguity is resolved. In this work, we look at how restart-incremental Transformers build and update internal states, in an effort to shed light on what processes cause revisions not viable in autoregressive models. We propose an interpretable way to analyse the incremental states, showing that their sequential structure encodes information on the garden path effect and its resolution. Our method brings insights on various bidirectional encoders for contextualised meaning representation and dependency parsing, contributing to show their advantage over causal models when it comes to revisions.

## introduction this is the honey... even if we stopped mid-sentence here, you would likely have created a partial interpretation of this prefix considering the honey as (the beginning of) a noun phrase. it could have many continuations, e.g. that skunks like, or produced by stingless bees. but what if the next token is another noun, as bee? a semantic parser would have to revise its previous hypothesis to accommodate the fact that honey has become a modifier of bee. bidirectional nlp models (i.e. those that encode linguistic input using both its left and right context) have transformed the field of computational linguistics. but that has come at the cost of cognitive plausibility in various aspects, in particular establishing a disregard for language's temporal structure. while humans process language one increment at a time @xcite @xcite , bilstms (graves and a causal decoder is forced to output only one postag for the token can at this point and cannot change it anymore, even if its internal state encodes the local ambiguity. in contrast, a restart-incremental model can perform revisions and would thus be able to recover if the selected label turned out to be incorrect (as in left). @xcite and transformers @xcite have mechanisms that rely on access to the complete input sequence, making them unsuitable as off-the-shelf components for incremental applications. although their unidirectional counterparts, lstms and autoregressive (aka causal) transformers, are equipped with the possibility of incremental decoding (making predictions relying only on left context), their token representations are static and thus not updated as incoming tokens arrive @xcite , because the underlying encoding is unidirectional. this places them at the disadvantage of not being able to revise, which is a desired property to recover from mistakes or local ambiguities @xcite , as shown in figure 1 . the restart incremental (ri) paradigm foot_0 @xcite circumvents this issue by adding an interface upon any model, making it work incrementally by processing prefixes from scratch whenever a new input increment arrives @xcite . even though it is computationally costly, ri has been effectively studied in simultaneous mt @xcite , dialogue systems @xcite , disfluency detection @xcite and nlu pipelines @xcite . besides, it has the advantage of incorporating recent increments into past predictions, revising hypotheses when deemed necessary. a recent line of work has investigated restartincrementality for sequence labelling by profiling bidirectional encoders @xcite and modelling and evaluating revision policies @xcite @xcite . still, evaluations so far have been performed in a black box fashion: only the relations between input tokens and output labels have been considered. the output labels as top positions of the softmax operation are only one product of many intermediate computations that have not been examined. therefore, we still do not know why ri models revise when they revise. what happens internally in the model's mechanisms to encode the need to edit previous outputs? how can (static) causal representations be affected under ri bidirectionality? can we predict whether the model will recover from a wrong interpretation? to answer such questions, we need a shift to glass box interpretability methods that can shed light on the dynamics of updates in the internal states which lead to output revisions. linguistically motivated analyses are also required to examine the behaviour of bidirectional models with respect to specific phenomena known to cause reinterpretations, like garden path constructions. thus, our present contributions to make ri models more transparent and explainable are (a) a formalisation of ri sequential processors as transition systems that create structured step-by-step constructions not present in causal models; (b) a proposal of interpretability methods for the internal mechanisms of ri models; and (c) an analysis of the strategies employed by various models on stimuli containing local ambiguities, for which a well-defined motivation for reanalysis is known. ## related work alternatives to restart incrementality monotonic decoding is advantageous to avoid output instability but comes with the downside of not recovering if there is a genuine reason to revise. some attempts to overcome this issue are adapting backpropagation to update the internal representation of the output @xcite or gradient-based methods to update the cached internal representations @xcite without changing the model's parameters. although this alleviates the strict monotonicity, it requires adapting the model's implementation. ri, on the other hand, is available to anyone in possession of any model, as it only requires re-running it as is each time. via recomputations, bidirectional models innately incorporate new increments into its states and revise if needed @xcite . for causal models, revisions can occur if one resorts to beam search methods (see e.g. @xcite ), traversing the sequence of static states generated at each token to make predictions at each time step. however, this technique does not reside within the model's own mechanisms. besides, if the correct sequence falls out of the chosen beam size at some point, the needed revision signal will be lost. performing the search requires enough memory and additional processing time, which may become an issue for longer sentences. benefits of bidirectionality several works have studied the effects of bidirectionality in lms. @xcite show that bidirectional attention is beneficial for some tasks like infilling and fine-tuning (but detrimental for others like next token prediction), while bidirectional context is beneficial when used in conjunction with bidirectional attention. @xcite propose using repetitions to allow that early tokens incorporate later tokens in their representations in llms. @xcite also show advantages of applying layer-wise removal of the causal mask during llm fine-tuning. interpretability various methods have been proposed to analyse how neural networks encode linguistic information @xcite , with transformers front and centre, e.g. in how information flows in its self-attention @xcite and how its predictions are refined or affected by previous tokens layer after layer @xcite @xcite . voluminous bertology works exist @xcite ), in particular investigating its attention mechanisms, what it means for an embedding to be contextualised instead of static and what layers encode what type of linguistic structure. another angle is to look at how linguistic information is dynamically learnt throughout the training regime of lms @xcite . the realm of incremental processing has also been active. @xcite propose methods to interpret how rnns incrementally encode, integrate and retain information. the incremental aptitude of neural networks to encode syntactic information has also been examined with the aid of psycholinguistic experimental methodologies @xcite . cognitive motivation an evident test bed are stimuli containing local ambiguities to evaluate if models exhibit garden path effects. in psycholinguistics, a traditional debate holds between the early-commitment with eventual reanalysis vs. beam search approaches, where multiple hypotheses are kept in parallel, known as two and one-stage accounts, respectively @xcite . in this sense, the ri paradigm can be viewed as a two-stage account performing multiple forward-reanalyses @xcite . transformers and ambiguities recent findings point to models encoding multiple hypotheses in face of local ambiguities. @xcite probe to what extent autoregressive lms encode multiple syntactic analyses by assessing the probability assigned to each interpretation as they generate continuations of a prefix, finding that multiple interpretations are followed in parallel. methods relying on surprisal theory show that the magnitude of the garden path effect in autoregressive and rnn lms underestimate human behaviour @xcite . other works study bert-based models using differences in surprisal and attention between unambiguous and locally ambiguous sentences @xcite . other analyses other glass box analyses have been put forward. irwin et al. (2023) use probing tasks with garden paths to study how the bert family assigns semantic roles in qa tasks. jurayj et al. (2022) define vector similarity methods as an improvement over surprisal to shed light on how gpt-2 traverses garden paths, finding periods of ambiguity in the hidden states that do not always surface. @xcite study how meaning is built word by word by gpt-2 looking at the connection between the size of the updates in its output activations and the n400 psychometric in humans. ## formalisation a restart-incremental model is constructed upon an underlying (non-incremental) model, making it perform a sequence of re-computations as the input is processed increment by increment @xcite . we propose a general formalisation of this procedure, detail its structures for sequential processing and discuss their interpretation for bidirectional models. restart-incrementality let m : @xmath0. generally speaking, restart-incrementality is an interface i around m , such that i(m ) can be fed individual tokens w i , which-until the interface is reset-are taken as continuations of the sequence of tokens fed so far. hence, at time step t, i(m ) will be provided with the token w t . internally, the interface assembles the prefix w 1 , . . . , w t (denoted as w t below), which it feeds to m , to compute the corresponding sequences s @xmath1, . . . , s t t ) and o @xmath2, . . . , o t t ). 3 this way, from the perspective of m , it is always a sequence of tokens that is being processed from scratch, independently from any prior calls of m , while from the perspective of i(m ), with each call a single token is added. while for the further downstream processing, only the output revisions (that is, the elements where o t+1 differs from o t ) may be of interest, 4 for the purpose of our analysis the sequences s 1 , s 2 , . . . , s n corresponding to each prefix are kept in memory by i(m ). i(m ) can hence be regarded as a transition system where states are updated based on the action of recomputing with the integration of a new input increment. ## method we now introduce our method and motivate its interpretations. once we have extracted states as a ri model processes a sentence, we can analyse the dynamics of this process, i.e. how do these vectors evolve over time. the 3d structures (b) and (c) from figure 2 can be converted into 2d as in (a) if we apply a metric over the features dimension, summarising it into one value (e.g. shannon's entropy or divergence over distributions and similarity or distance metrics for embeddings). with such scores, we can then examine whether the behaviour of their variation aligns with output edits. if we look at columns of the triangular structures, we can analyse a sequence of states assigned to one token step by step. if we consider the rows, we see the effect that the recently added token had on the current prefix. this analysis can in principle be applied to any sentence. however, to shed light on how restart-incremental models edit the output and how they engage in reinterpretation, it is useful to analyse what happens when the linguistic input is known to contain local ambiguities. that way, we have a genuine motivation to expect revisions and can study how states change at key positions. let v i jk be the value assigned for feature k of token j at time step i in a sequence with n tokens. we can compare the differences between v i jk and four relevant positions with clear interpretations: ## analysis overall, we are interested in how future tokens affect states of past tokens and how such changes surface as output revisions. our main hypothesis is that ri models are led down the garden path when they first encounter a local ambiguity without further right context but, as the disambiguating region is integrated, states are updated to absorb the new interpretation. we provide insights into updates that may be allowing them to recover. scope we apply our method to two ri scenarios. firstly, we look at the construction of meaning representations in bidirectional lms ( §5.2). we assume that the representations of a token encode its meaning in the available context, tracking how temporarily ambiguous unambiguous unambiguous without added token figure 3 : we realign tokens, after computing the states, by removing the states of additional token(s) from the triangular structure. that way we can directly compare the states of a locally ambiguous sentence with its unambiguous counterpart. ## discussion & conclusion our method has shown that a ri interface applied to bidirectional encoders yields models that build sequences of states with a rich dynamics of updates of past representations. this grants these models with revisability, a property that is desirable in incremental systems @xcite but not present in unidirectional models. while causal models must create a representation based only on left context and stick to their first commitment, ri bidirectional models can profit from incorporating right context and revisit its previous decisions. our analysis has empirically revealed that the ri models we studied seem to run into the downsides of parsing, i.e. they are led down the garden path, but their initial internal representations are updated once the disambiguating region is processed, more than in the baselines. in other words, it seems that they make early commitments but then revise accordingly, as in two-stage approaches to language processing. in the analysis of contextualised embeddings, the effect is less pronounced on lower layers, but more prominent from middle to upper layers. this relates to the findings of @xcite who suggest that upper layers (that are more semantic) can be used to disambiguate decisions in lower layers (that are more syntactic). this is also in line with works showing that middle to upper layers are most informative for some tasks @xcite . treating the triangular structures as internal states of a transition system constructed from graphbased dependency parsers allows us to uncover how relations between tokens change as an effect of future tokens. by using the divergence computed with various reference time steps as a measure, we show that attention distributions may evolve differently throughout the course of processing and how they can become similar again through the disambiguation process. the shift in distributions also indicates edits in dependency arcs and labels, and more importantly, enlightens us more about why ri models revise when they do. while decoder-only llms have grown in popularity the past few years, we still see room for improvement, particularly regarding how their static representations can be updated (like ri models) as more tokens are processed. this can benefit tasks that require non-monotonic reasoning @xcite , regardless of whether it involves nlg or nlu. natural language is also inherently ambiguous, although its spectrum of ambiguity depends on its use @xcite which necessitates this processing mode. moreover, modern llms still struggle with this aspect of language understanding @xcite . applying a ri interface may result in a nonmonotonic output. however, there are certain cases where monotonicity is preferred. for instance, to reduce flicker in simultaneous machine translation models @xcite or when a stable and immediate output is always needed for further processing in an incremental pipeline. in both cases, the lookahead strategy @xcite can be used for disambiguation by considering to-kens within the lookahead window. alternative methods like beam search can also be employed, depending on how various properties such as monotonicity, timeliness, and decisiveness @xcite are prioritised in the use case. one of transformers' properties that does not align with human language processing, is how words are processed in parallel, even in autoregressive settings. ideally, models of human language should also face similar input and linguistic challenges as humans do @xcite . we show that when facing such challenge in the form of garden-path sentences, transformers with a ri interface still exhibit the ability to disambiguate using forward-reanalysis @xcite . future research can explore how the restart activity can serve as an indicator of updates that should not be immediately integrated into the output, or model a controller that is able to detect variations that lead to revisions and decide whether to delay outputs until a level of certainty has been reached. additionally, it would be interesting to see how a ri interface can be used to investigate the capability of nlp models to deal with linguistic ambiguities in other languages than english, e.g. the interpretation of relative clauses in dutch @xcite .