Sorting through the noise: Testing robustness of information processing in pre-trained language models (2021)

Abstract: Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this paper we tackle a component of this question by examining robustness of models’ ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models’ use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.

## introduction in recent years, pre-trained language models (lms) have taken nlp by storm. as we work to interpret the impressive performance of these models, a persistent question is the extent to which lms are doing something like language "understanding". do these models robustly extract the nuances of information conveyed in text, or is their strong performance driven by more superficial mechanisms? in this paper we focus on examining a particular aspect of "understanding" in pre-trained lms: how robustly these models process, retain, and apply new facts presented in their inputs. we assume that a fundamental aspect of language "understanding" will be the capacity to represent and accumulate information from the meaning of the input text. so if we input "sebastian lives in france, and rowan lives in indonesia.", we would expect a model that understands language to form (or update) representations for these imaginary entities sebastian and rowan, such that those representations contain the information that sebastian lives in france, and that rowan lives in indonesia. while recent work has studied models as knowledge bases, testing their ability to reproduce facts encountered during training, in this paper we are asking an importantly different question-not about what models memorize during training, but about how sophisticated they are in processing and representing information from new input text after training. to test lms' ability to process and retain information from context, we design cloze tasks that incorporate a piece of critical information in context, and then prompt the model to complete a statement related to that information. to test the robustness of the processes informing these predictions, we introduce distracting but irrelevant content in the contexts, and test whether the models maintain correct predictions in the face of these distractions. additionally, to explore further the nature of mechanisms informing model predictions, we systematically vary the nature of the distracting contentmanipulating how many distractor words we use, how semantically related they are to critical words, and their relative positions in the sentence. we apply these tests to a range of recent pretrained lms and examine the impacts of our manipulations on model performance. the results indicate clearly that distracting content in the context is effective in undermining model predictions, and variation of distractor types suggests that models are particularly sensitive to influences of semantic similarity and relative word position. overall, the results support the conclusion that predictions in pre-trained lms are driven in large part by super-ficial contextual cues, rather than by robust representations of relevant facts from context. we make all data and code available for further testing. foot_0 ## related work prior work has tested lms as knowledge bases using cloze-style probes @xcite . as a starting point we rely on models' ability to display this type of knowledge, but our question differs importantly from that work: we are not asking whether models can recall facts about the real world from training-rather, we are trying to gauge the extent to which models form robust representations of new information presented in input after training. somewhat more similar to ours is work like @xcite , which explores the consistency of models' generation of facts in the face of rephrasing of prompts. the basic intuition behind this work-that lms' ability to make intelligent-looking predictions can be sensitive to the particulars of the context-is one that we also use as we ask more specific questions about models' processing of information in their input. a good deal of prior work has focused on testing for linguistic knowledge in language models @xcite . much of this work has prioritized testing syntax in pre-trained lms via agreement tests @xcite . others expand to broader sets of syntactic phenomena @xcite @xcite and semantic/pragmatic phenomena @xcite . other work has studied syntactic and semantic information in contextualized embeddings from these models @xcite @xcite . we take one step up from examination of these abstract linguistic capacities, with a focused examination of models' ability to use such linguistic scaffolding to process and retain new information described in text. our use of attractors to test model robustness takes inspiration from use of attractors within syntactic testing contexts @xcite ), but we focus on semantic relationships in defining attractors, and use the attractors to investigate different aspects of models' processing. some scattered work has explored more semantic types of attractors for testing lms-in particular, there is work looking at whether presence of certain context words will prime corresponding targets in context. such work has experimented with contextual factors like distance between prime and target @xcite , as well as contextual constraint @xcite . we build on this existing work with a more systematic exploration of impacts of different types of attractors, and with a more targeted goal of testing models' robustness in processing new facts from context. in focusing on models' ability to extract, retain, and deploy information conveyed in text, our work also relates to tasks in reading comprehension question answering @xcite @xcite @xcite . some such work, like the babi dataset @xcite and cbt @xcite , use insertion of additional material to make the tasks generally more difficulta tactic that also parallels the related method of adversarial testing @xcite @xcite . there are important similarities in the questions and strategies of these prior works and ours, but we differ in focusing specifically on information processing in lms, rather than performance of models supervised for a particular downstream task. unlike those works, our goal is to shed light on robustness of language "understanding", and nature of prediction mechanisms, that arise as a result of lm-based pre-training. ## methods we design our tests in the form of cloze tasks, so as to test the pre-trained lms in their most natural setting, without interference from fine-tuning. we start from a simple base context, in which the model is given a background fact about an imaginary entity, and then is asked to complete a related statement about the entity. for instance: sebastian lives in france. the capital of sebastian's country is @xcite we will refer to "france" here as the critical background word, and the correct completion "paris" as the target word. for all of our test items, we establish a baseline competence in our tested models, such that all models successfully prefer the correct target completion over a set of closely related completions (to be outlined shortly) within this simple base context. in this way, we establish that the models have the relevant "world knowl- base context zero attractor sebastian lives in france. the capital of sebastian's country is ____ multiple-entity attractor setting b-type attractors sebastian lives in france, rowan lives in indonesia, and daniel lives in chile. the capital of sebastian's country is ____ t-type attractors sebastian lives in france, rowan lives in jakarta, and daniel lives in santiago. the capital of sebastian's country is ____ unrelated attractors sebastian lives in france, rowan drives a car, and daniel writes poetry. the capital of sebastian's country is ____ single-entity attractor setting b-type attractors sebastian lives in france, and has visited indonesia and chile. the capital of sebastian's country is ____ t-type attractors sebastian lives in france, and has visited jakarta and santiago. the capital of sebastian's country is ____ unrelated attractors sebastian lives in france, drives a car, and writes poetry. the capital of sebastian's country is ____ of course, if a model is able to predict "paris" in this base example, this could be attributable to a number of causes. on one hand, it could be taken as evidence that the model was able to store a representation of sebastian as a resident of france, and then when queried about a related statement, the model was able to make use of that stored information to generate a correct prediction. alternatively, there may be more superficial explanations for the model's success in this completion: for instance, the model may simply be reacting to the fact that "france" was recently mentioned, and now the prompt is asking for a capital. what if "indonesia" had also been mentioned? would the model still recognize that "paris" is the correct completion? to tease apart these classes of explanation, we introduce distracting content in the sentences, and test how this content impacts models' outputs. following the number agreement literature @xcite , we refer to these inserted items as attractors. in a system that robustly represents and retains the critical background information from context, attractor content should not prevent the model from continuing to prefer the correct target completion. if the attractor content does change the models' preferences, then we can infer that more superficial predictive mechanisms are likely at play. ## results we begin by examining model prediction accuracy when attractors are semantically related to the critical background word or the target word. we define accuracy as percentage of instances in which models assign higher probability to the correct target than to any alternatives in the corresponding semantic set. this serves as the most direct test of the anticipated potential impact of our semantically related attractors, as it directly assesses whether presence of an irrelevant word (e.g., "indonesia"), which is semantically related to the critical background or target, and which invites a competing prediction ("jakarta"), will cause models to prefer that competing prediction over the correct target. figure 1 breaks down accuracy by number of attractors, for multiple-and single-entity settings. for the sake of space, we merge b-type and t-type attractors, which show largely similar patterns. 3 we see in figure 1 that addition of just a single semantically related attractor has clear impact on model performance, with models preferring the correct completion substantially less often than when no such attractors are present. roberta large shows the strongest resistance to this effect, but still shows clear disruption from the first attractor. as we insert additional attractors, for both conditions we see that rather than further hindering performance, for a couple of models accuracy actually improves. while we are not certain what drives this pattern, we speculate that a possible cause could be that models may learn to pay less attention to content that takes the form of lists. this is consistent with the fact that improvement with more attractors is mitigated in the multiple-entity setting, when at-3 see appendix figures 6-7 for b-type and t-type results. ## varying position of information in this section we explore the effects of further varying entity and attractor position, to better understand how the models utilize positional cues. ## discussion the experiments above were designed to do two things. the first purpose was to test whether pre-trained lms show evidence of robustly processing and storing new facts from input context. to test this, we inserted irrelevant, distracting content in addition to critical context information, to see whether model predictions would be affected. the results of these experiments show clearly that model predictions are indeed impacted by this irrelevant content, rather than remaining consistent on the basis of the critical contextual information. the second goal of these experiments was to explore the specific types of cues affecting model predictions, by varying the nature of distracting content. through these manipulations, we find influences of both semantic similarity and word position as superficial cues informing model predictions. in particular, we find that insertion of a single semantically related attractor has a dramatic effect on predictions, with the first such attractor reaching roughly ceiling impact. unrelated attractors also influence predictions, with more gradual, additive impacts. effects of semantic similarity interact additionally with effects of relative word position, with semantically related attractors showing much reduced influence when occurring prior to the key entity mention. these trends suggest that models rely on heuristics involving semantic similarity to critical words, and relative position of entity and descriptor words, for determining which elements of context are relevant for a prediction. while it should not come as a shock that language models use superficial contextual cues for prediction, findings of this kind must serve as reality checks as we consider the capacity of these models for language "understanding". we have made the simple assumption that "understanding" will involve robustly representing and retaining information from prior context, and generating predictions accordingly. the results presented here suggest that this criterion is not met. additionally, the results of these experiments take steps toward understanding the precise strategies that models do use in generating predictions, beginning to sketch out a picture in which models rely substantially on coarse-grained semantic similarity and word position cues to identify relevant contextual words. we do note that of the models tested, roberta large frequently distinguishes itself as least susceptible to our attractors-though it shows disruption all the same. since this model is pretrained on a larger dataset than other models tested here, size of pre-training data is a likely contributor to the model's superior performance. additionally, we find that within a given training regime, larger models mostly perform more robustly than smaller models, suggesting that model size is also a contributor in interaction with factors like training data size. we leave for future work the investigation of whether the comparatively strong performance of roberta large reflects truly more robust representations in that model-or use of superficial cues not yet targeted in the present work. these findings also have implications for studying mechanistic connections between pre-trained language models and language processing in humans. studies of human sentence processing have shown comparable susceptibility to interference from irrelevant context elements, depending on semantic and syntactic properties @xcite @xcite . systematic comparison of interference effects in humans and in language models stands to shed light on mechanistic similarities and differences in the ways that these language processing systems handle information from prior context. ## conclusion we have presented results manipulating inputs of pre-trained lms, to test the ability of such models to represent and retain information conveyed by input text. our results show that though models may appear to handle information correctly in simple settings, these correct predictions are easily broken by insertion of distracting material in the context. systematic manipulation of the distracting content further indicates key roles for semantic similarity and relative word position in models' selection of relevant contextual cues for prediction. overall, the results suggest that lm predictions are driven more by coarse-grained superficial cues than by extraction of robust meaning information from context. the results serve as a reality check for considerations of the extent to which lms "understand" their input, and lay groundwork to understand the mechanisms that do drive predictions in these models.