It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension (2024)

Abstract: Natural language processing has seen rapid progress over the past decade. Due to the speed of developments, some practices get established without proper evaluation. Considering one such case and focusing on reading comprehension, we ask our first research question: 1) How does the order of inputs – i.e., question and context – affect model performance? Additionally, given recent advancements in input emphasis, we ask a second research question: 2) Does emphasizing either the question, the context, or both enhance performance? Experimenting with 9 large language models across 3 datasets, we find that presenting the context before the question improves model performance, with an accuracy increase of up to 31%. Furthermore, emphasizing the context yields superior results compared to question emphasis, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer. Experimenting with both prompt-based and attention-based emphasis methods, we additionally find that the best method is surprisingly simple: it only requires concatenating a few tokens to the input and results in an ac- curacy improvement of up to 36%, allowing smaller models to outperform their significantly larger counterparts.

## introduction for the task of reading comprehension (rc), models receive two kinds of inputs: 1) a context, e.g., a wikipedia article, and 2) a question that should be answered according to the context @xcite . while early efforts to address this task usually involve models that encode each of these separately @xcite @xcite @xcite , more recently, large language models (llms) receive a concatenation of the two inputs @xcite @xcite ; baek setting/ emphasis question: <@xmath0: <@xmath1: <@xmath2: <@xmath3: where is the world's largest ice sheet located today. context: the antarctic ice sheet is the largest single mass of ice on earth [...] context: the antarctic ice sheet is the largest single mass of ice on earth [...]. question: where is the world's largest ice sheet located today. context question: where is the world's largest ice sheet located today. context: the antarctic ice sheet is the largest single mass of ice on earth [...] context: the antarctic ice sheet is the largest single mass of ice on earth [...]. question: where is the world's largest ice sheet located today. question+ context question: where is the world's largest ice sheet located today. context: the antarctic ice sheet is the largest single mass of ice on earth [...] context: the antarctic ice sheet is the largest single mass of ice on earth [...]. question: where is the world's largest ice sheet located today. figure 1 : example from the natural questions dataset in which we show the different settings we experiment with: question or context first in the input prompt, and the different substring emphasis (in bold). <@xmath4; <@xmath5. @xcite @xcite . surprisingly, there is no current standard of what the ordering of such input components should be. for example, @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite provide the question first in each prompt, while @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite provide the context first. moreover, there is no current standard of how to present the two input components in general. for example, considering the question and context @xmath6, respectively, @xcite add the special tokens "question:" and "context:" before the question and context, while @xcite use "<@xmath7:** <@xmath8, @xcite use " @xcite : <@xmath9: <@xmath10, @xcite use "<@xmath11, and others such as @xcite @xcite , employ their own methods. while at first sight this might not seem impor-tant, many works have shown that lms can be extremely susceptible to slight variations in the input sequence @xcite @xcite . furthermore, recent research has found that different presentations of inputs can help emphasize them and improve models' ability to follow instructions @xcite . based on these observations, we ask the following research questions (rqs): 1) how does the order of inputs -i.e., question and context -affect model performance? 2) does emphasizing either the question, the context, or both enhance performance? a summary of these questions can be seen in figure 1 . we evaluate 9 llms on 3 datasets and find the following: 1) the ordering of the question and context is crucial, and improves model performance with an accuracy increase of up to 31%. 2) both prompt-based and attention-based emphasis methods are capable of strongly improving models' performance, where emphasizing the context yields superior results compared to emphasizing the question, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer. 3) the best emphasis method is surprisingly simple: it only requires a simple concatenation of a few tokens to the input and results in an accuracy improvement of up to 36%, allowing smaller models to outperform their significantly larger counterparts. ## related work reading comprehension reading comprehension involves the task of understanding a given context, such as a wikipedia passage and answering questions based on that context @xcite . to that end, researchers develop models capable of comprehending written text and extracting relevant information to accurately respond to queries @xcite @xcite . traditional approaches often encode the context and question separately @xcite @xcite @xcite , while more recent advancements leverage llms that concatenate both inputs into a single string @xcite @xcite @xcite @xcite . these models need to possess a deep understanding of the provided context to generate accurate responses to a wide range of questions, and many have shown that they do. achieving high performance in reading comprehension tasks requires not only effective encoding of textual information, but also sophisticated reasoning and inference abilities to derive answers from the context accurately @xcite . therefore, ongoing research on reading comprehension focuses on improving model architectures @xcite @xcite , training strategies @xcite , and evaluation metrics @xcite to enhance the comprehension and reasoning capabilities of these systems. here, we address the gap in research focused on how the inputs themselves can impact performance. prompt engineering a related area -prompt engineering @xcite focuses on modifying the input prompt to improve the performance of lms without altering their underlying architecture or training regime. and while lms require a deep understanding of the provided context to generate accurate responses, recent studies have demonstrated that large performance enhancements can be achieved through prompt engineering alone @xcite @xcite . this approach involves various techniques such as adding different input strings @xcite , providing step-by-step instructions @xcite , or incorporating additional contextual information into the prompt @xcite . by carefully crafting the input prompt, researchers aim to guide the model towards relevant information and improve its ability to comprehend and generate coherent responses. emphasis methods it is important to note that researchers often do not have the ability to precisely guide the model using prompt engineering, and much of prompt development is based on intuition. that is, researchers often have to try many different prompts manually or automatically until they find those that increase performance, and often just for their specific models @xcite @xcite . in comparison, recent work on input emphasis, including attention steering (as) and marked prompting (mp) @xcite , have shown great success in improv-ing models' ability to follow instructions. these methods aim to guide the focus of models towards various segments of the input sequence, by either adding tokens to the sequence or rescaling attention weights for relevant tokens. as is closely related to work that avoids modifying models' architectures, or training regime, however, it takes a more direct approach by modifying parts of the input directly by rescaling the attention values of specific heads corresponding to specific tokens. interpretability emphasis methods, such as as, are also related to model interpretability, which is concerned with understanding the contributions of different model components, and in particular, head attribution @xcite . for example, meng et al. (2023); geva et al. (2021); kobayashi et al. (2023) show that different knowledge from the training data is found within the feedforward layers, while others show that attention heads have similar patterns @xcite . ## models we experiment with 9 different llms. llama-2-7b and llama-2-13b llama-2-7b and llama-2-13b @xcite are llms which contain 7 and 13 billion parameters, respectively, and are trained on 2 trillion tokens. we use these models as they perform well on the reading comprehension task @xcite and recent work shows that their performance can be improved using emphasis methods @xcite . falcon-7b and falcon-7b instruct these two models contain 7 billion parameters each, and are trained on 1.5 trillion tokens @xcite . we opt for these models because they are newer and have demonstrated significant success across various tasks. additionally, falcon-7b instruct comes with an instruct version, enabling us to compare the performance of both variations. mpt-7b and mpt-7b instruct these are two llms with 7 billion parameters, trained on 1 trillion tokens @xcite . chosen for their recent development and proven versatility. gpt-j-6b gpt-j-6b @xcite ) contains 6 billion parameters and is trained on the pile dataset @xcite . we use this model as in addition to the fact that it has been shown to perform well on question answering tasks @xcite , it is also often compared again our largest model -llama-2 @xcite and recent work shows that its performance can be improved using emphasis methods @xcite . gpt-2-xl gpt-2-xl @xcite a llm with 1.5 billion parameters and is trained on webtext @xcite . while much smaller than current state-of-the-art models, such as chatgpt (openai, 2023a) or gpt 4 (ope-nai, 2023b), we experiment with it as many lowresource settings require usage of smaller models. gpt-2-large our last model, gpt-2-large @xcite , contains 774 million parameters and, similar to gpt-2-xl, is trained on webtext. we use it for similar reasons as those we described in the gpt-2-xl section. ## conclusion focusing on reading comprehension, we evaluate 1) how the order of the question and context affects model performance; and 2) whether emphasizing either the question, the context, or both enhances performance. experimenting with 9 llms across multiple datasets, we find that presenting the context before the question improves model performance, with an accuracy increase of up to 31%. furthermore, emphasizing the context yields superior results compared to emphasizing the question, and in general, emphasizing parts of the input is particularly effective for addressing questions that models lack the parametric knowledge to answer.