How Language Models Prioritize Contextual Grammatical Cues? (2024)

Abstract: Transformer-based language models have shown an excellent ability to effectively capture and utilize contextual information. Although various analysis techniques have been used to quantify and trace the contribution of single contextual cues to a target task such as subject-verb agreement or coreference resolution, scenarios in which multiple relevant cues are available in the context remain underexplored.In this paper, we investigate how language models handle gender agreement when multiple gender cue words are present, each capable of independently disambiguating a target gender pronoun. We analyze two widely used Transformer-based models: BERT, an encoder-based, and GPT-2, a decoder-based model.Our analysis employs two complementary approaches: context mixing analysis, which tracks information flow within the model, and a variant of activation patching, which measures the impact of cues on the model’s prediction. We find that BERT tends to prioritize the first cue in the context to form both the target word representations and the model’s prediction, while GPT-2 relies more on the final cue. Our findings reveal striking differences in how encoder-based and decoder-based models prioritize and use contextual information for their predictions.

## introduction pre-training language models on large data using the transformer @xcite architecture has led to remarkable advancements in natural language processing. a key advantage of this neural network topology is its ability to retrieve information from any part of the input, thus, constructing rich, contextualized representations. this capability allows the model to effectively deal with longrange dependencies @xcite and enables in-context learning phenomena, where the model can be adapted to solve downstream tasks using additional input context @xcite @xcite . grammatical dependencies, such as subject-verb agreement @xcite and coreference resolution @xcite , have been extensively used as well-defined tasks to study the contextual abilities of pre-trained language models @xcite @xcite @xcite . these tasks often require the model to capture and exploit the syntactic relationship between word pairs in the sentence; for example in the case of coreference resolution, the model needs to disambiguate a pronoun with respect to the subject as its single reference point in the context. despite a rich literature on this, the scenario where multiple grammatical cues are present within the context remains underexplored. in this paper, we use coreference resolution as our case study and analyze model behavior in cases where the context contains multiple sources of information that are relevant for the target task (which we refer to as 'cue' words), aiming to identify which contextual cues the model prioritizes when disambiguating target pronouns. consider the following example, in which the last pronoun as a target word that the model is asked to generate is marked in bold and all possible cues to disambiguate it ('she' versus 'he') are underlined: mary loves playing the piano. she practices every day, and her music teacher says she is very talented. specifically, we investigate how the model benefits from various cue words when generating the last target pronoun in the output. to this end, we make use of the biography corpus as it naturally contains numerous referential expressions that refer to the same individual. using two complementary analytical approaches, we analyze bert @xcite and gpt-2 @xcite , two models with different architectures and training objectives, across contexts with various numbers of cues, revealing a notable distinction between the behavior of encoder-based and decoder-based language models. firstly, we use value zeroing @xcite as a context-mixing method to track the flow of information from cue words to the representation of the target word at each layer of the model. we find that decoder-based models tend to incorporate the final cue words in the context to form the contextualized representation of the target word. in contrast, encoder-based models rely on the first cue words. secondly, we employ a variant of activation patching @xcite @xcite , a recently popular mechanistic interpretability method @xcite , to measure the impact of each cue word on the model's confidence in generating the target word. while context-mixing methods quantify information mixing in the representation space, the second approach focuses on the language model head to determine whether the encoded information is actually used for prediction. our empirical results show that the predictions of the two analysis methods are consistent with each other, implying that the cues that contribute more to the representation of the target word also play a crucial role in the model's final decision. specifically, our main finding indicates that, in contexts with multiple grammatical cues, encoder-based models tend to prioritize the earlier cues, while decoderbased models rely on the later cue words when disambiguating the target pronoun. foot_0 2 related work many analytical studies have been conducted to examine the grammatical capabilities of pre-trained language models, often by probing their layerwise representations for tasks such as part-of-speech tagging @xcite , dependency parsing @xcite , subject-verb agreement @xcite , and coreference resolution @xcite . these tasks have also been leveraged in another line of research, particularly as a case study for evaluating attribution methods @xcite @xcite , as they provide a clear ground truth for assessing the plausibility of attribution scores. for example, when predicting a pronoun, an appropriate attribution method is expected to highlight the subject of the sentence. while these studies focus only on cases with a single plausible cue in the context (e.g., subject) to disambiguate the target word (e.g., pronoun), our work investigates model behavior when multiple sources of information (cue words) exist in the context. for this purpose, we leverage two state-of-theart analysis methods from two active lines of interpretability research: one that aims at measuring token-to-token interactions in the model known as context mixing, while the other focuses on reverse engineering the model's decision and decompose it to understandable components, known as mechanistic interpretability. context mixing. this line of work focuses on tracking information flow in the model, providing a map score that quantifies token-to-token interactions at each layer. this can be achieved using a group of analytical approaches known as 'context-mixing' methods. although self-attention weights are often seen as a straightforward measure of context mixing in transformers, numerous studies have shown that relying solely on raw attention can be misleading @xcite . they often focus on meaningless and frequently occurring tokens in the input, such as punctuation marks and special separator tokens in models trained on text @xcite , or background pixels in vision transformers @xcite . 2 hence, several methods have been developed to broaden the scope of analysis and incorporate other model components into the computation of the context-mixing @xcite @xcite @xcite . ## experimental setup in this section, we describe the data and models that we use to set up our experiments. ## which cue does the model rely on to form a target representation? transformers perform well at retrieving information from any part of the context to build contextualized representations. our first step is to trace the flow of information within the model to understand how different contextual cues shape the representation of target words. ## conclusion in this paper, we examined how language models handle gender agreement when multiple valid gender cue words are present in the context. we carried out extensive experiments using two state-ofthe-art and complementary analytical approaches on two prominent language models with different model architectures: bert and gpt-2. our results suggest that encoder-based and decoder-based models behave differently in prioritizing contextual cues. more specifically, we observed that bert mainly relies on the earlier cues in the context, while gpt-2 mostly uses the later ones. these findings can be explored and leveraged in future to enhance model efficiency (by excluding redundant cues from the computations), update the models' beliefs (by intervening with the most crucial cues they rely on), or improve the way we interact with them through prompting (by considering the impact different cues may have in various positions within a given context).