Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggers (2023)

Abstract: Recent applications of LLMs in Machine Reading Comprehension (MRC) systems have shown impressive results, but the use of shortcuts, mechanisms triggered by features spuriously correlated to the true label, has emerged as a potential threat to their reliability. We analyze the problem from two angles: LLMs as editors, guided to edit text to mislead LLMs; and LLMs as readers, who answer questions based on the edited text. We introduce a framework that guides an editor to add potential shortcuts-triggers to samples. Using GPT4 as the editor, we find it can successfully edit trigger shortcut in samples that fool LLMs. Analysing LLMs as readers, we observe that even capable LLMs can be deceived using shortcut knowledge. Strikingly, we discover that GPT4 can be deceived by its own edits (15% drop in F1). Our findings highlight inherent vulnerabilities of LLMs to shortcut manipulations. We publish ShortcutQA, a curated dataset generated by our framework for future research.

## introduction we consider the task of machine reading comprehension (mrc), also known as text-grounded question answering (qa) @xcite , where a model is given a text passage and a question, and has to answer the question based on the text, either by marking a span over the text or generating a short string. recently, llms @xcite such as gpt-instruct (gpt3.5) @xcite , @xcite emerge as strong models for performing the mrc task. the demonstrated text-grounded qa abilities of llms prompted the incorporation of llms within a search-engine setups in which a retrieval model retrieves documents, and the llm answers by extracting answers from these documents, in websites such as google.com and bing.com. however, previous mrc models are known to often answer by relying on shortcuts (also called shallow heuristics) rather than on full understanding of the text @xcite . does this tendency transfer also to llms? to examine the role of shortcuts in-depth, it is necessary to edit samples to activate these shortcuts in the models. previous attempts at this mainly involved simple edits such as word replacements @xcite @xcite , or more intricate edits designed for specific cases, bound by the structure of the original text @xcite . given that larger models exhibit greater resilience and are less inclined towards simple shortcuts compared to their smaller counterparts @xcite , a faithful investigation of shortcut usage in llms necessitates the application of more complex shortcut triggers. this paper sets out to explore two principal questions: first, whether an llm can be used to study shortcut usage in other llms, and second, whether a given llm is robust to the adversarial edits done by itself. we look into the interaction of llms and shortcuts by using an llm that functions as an editor, altering text to add or exclude shortcut triggers to mislead a different llm. our framework (see figure 1 ) uses a strong model to edit samples and is guided by the output of a weaker model. we evaluate the resulted edited samples (after manual verification that their semantics remain the same) on both the model that edited them and other, weaker models. in our experiments we found that gpt4 foot_0 is a reliable and effective editor, editing samples to mislead less proficient llms, like gpt3.5 foot_1 , resulting in a 30% drop in f1 score. interestingly, gpt4 is being misled too by some of the samples it made to mislead gpt3.5, reflected in a 15% decrease in f1 score. we release shortcutqa -a curated dataset of samples generated by our framework. our findings also highlight a potential attack vector that could be exploited for malicious intents (we discuss its implications in ethical statement). ## shortcuts we define here the term shortcuts (sometimes called shallow heuristics or clever hans features) as we use this term throughout the paper. given a model m , an mrc sample composed of a text t and question q, we define an intervention f j that edits t to add or exclude a property j, we say that m is misled by j if the following conditions hold: here a returns the gold label (right answer) for its input. equation 1 stands for the basic assumption that the model answers correctly in the first place. equation 2 express that the edit did not affect the right answer. equation 3 express that the model changed its prediction in the context of the semantics-preserving edit. ## methodology we propose a framework that adversarially edits samples to mislead a specific model (target model). the framework achieves this by adding or excluding shortcut triggers guided by the confidence levels of the target model. in our experiments, the editor was gpt4, and the target model was gpt3.5. our code and dataset will be available on github foot_2 . ## shortcutqa we run the procedure described in section 3 on 300 text/questions pairs from squad @xcite and 300 from newsqa @xcite with gpt4 as the edit model and textdavinci-003 as the guide model. we then manually verified that the edits did not change the semantics of the text w.r.t the original answer (discarding samples that failed this verification). this left us with 247 edited squad samples and 243 newsqa samples, a total of 490 verified samples which we use in our evaluations. the analysis of shortcutqa in ## evaluating models on shortcutqa in ## discussion our findings highlight the ability of large language models (llms), specifically gpt4, to exploit known shortcuts to mislead less proficient models, illuminating a new dimension of inter-model interactions. interestingly, we find that gpt4 is susceptible to be misled by the same adversarial manipulations it created, suggesting intrinsic vulnerabilities and pushing the boundary of our understanding of llms' resilience to shortcuts. our results underlines the importance of further investigations into llms robustness, resilience, and potential susceptibilities to failure. we release the dataset we used for the evaluation, shortcutqa, which we see as a valuable resource for stress-testing and learning the vulnerabilities of llms in the future.