DREAM : Improving Situational QA by First Elaborating the Situation (2022)

Abstract: When people answer questions about a specific situation, e.g., “I cheated on my mid-term exam last week. Was that wrong?”, cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the “scene”. To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM’s training data. These results suggest that adding focused elaborations about a situation can improve a system’s reasoning about it, and may serve as an effective way of injecting new scenario-based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters.

## introduction cognitive science has long promoted the formation of mental models -coherent, constructed representations of the situations we encounter -as central to understanding and question-answering @xcite . drawing loosely on this figure 1 : given a situation s, our system dream generates an elaboration of the situation -a "scene elaboration" se -envisioning details of what might be happening in s. given a question q about s, we find that a sota qa system (macaw) answers q more accurately when se is provided as additional input. figure 1 illustrates this approach. the situated question, drawn from the ethics dataset @xcite , concerns the situation of serving edible mushrooms to a friend, and asks if this is (generally) wrong or not. in this case, given just this information, the representative language model we use, macaw @xcite , answers incorrectly (that it is "(a) wrong"), illustrating the unpredictability that even large lms can exhibit @xcite . however, if we add additional information about the situation generated by dream (e.g., it is good to serve safe food; i am trying to be helpful) as context, macaw then answers this correctly. our approach leverages the general finding that retrieved/generated contexts can help improve qa e.g., @xcite , in two important ways. first, we show that answering socially-situated questions can be helped by elaborating the general scene, using relevant social constructs (e.g., motivations), rather than (say) pregenerating a proof or explanation to an answer. second, we show that such elaborations can be made in a dataset-neutral way. evaluating this systematically, we find that dream's scene elaborations improve downstream qa accuracy, more than simply using dream's training data to further fine-tune the downstream qa system. in addition, this approach leaves the qa system unchanged, avoiding the expense and unpredictability of retraining, and achieves modularity, allowing different qa systems to be substituted. our contributions are thus: our evaluation shows the outputs from this system, called dream, are more accurate and consistent than the evoked elaborations from macaw. 3. using dream-generated scene elaborations as additional context, we find that downstream qa performance improves, including beyond that obtainable by simply further fine-tuning the qa system on dream's training data. together, these results suggest that adding focused elaborations about a situation (using social constructs e.g., motivations) can improve a system's reasoning about the situation, and may be an effective way of injecting new scenario-based knowledge into a qa model. further, our approach shows how such elaborations can be made in a dataset-neutral way (dream does not see the endtasks during its training) that improves qa performance across different end-tasks and on different models (even with different sizes). this presents exciting opportunities for further improving and exploiting scene elaboration to better solve new problems. ## related work the concept of a mental model -a coherent, internal representation of the world -is common in cognitive science @xcite @xcite . it suggests that people solve situated problems by elaborating a mental picture of the situation, including elements that may be peripheral to a specific answer, rather than by constructing a deductive proof from a few key facts to an answer @xcite . recently, @xcite tried creating an internal "mental model" using a set of axioms that deductively explain the observations. other studies in ai have attempted to identify such elements by studying what questions people naturally ask when reading text @xcite or viewing images @xcite . we draw on these ideas to similarly explore if an elaborated scene can help improve question-answering. several prior works have shown that retrieved/generated contexts/prompts can help improve qa, for example using retrieved sentences or paragraphs @xcite ; self-generated explanations @xcite , statements ("selftalk") @xcite , intermediate computations @xcite , or "chains of thought" @xcite ; or model-generated facts @xcite or causal graphs @xcite . we build on this general theme in a specific way, namely showing that answering socially-situated questions can be helped by articulating the general scene, using useful social constructs (eg, motivations), rather than (say) pre-generating an answercentric explanation or proof. our approach can be viewed as a form of prompt engineering (scao and rush, 2021), where we pre- query situation macaw's scene elaboration dream's scene elaboration [codah] [q, s] the woman and her daughter were happily strolling through the park. the woman: gold ans: was smiling. [social norm] if you want to be happy, be kind [emotion] the woman's emotion is joy. daughter's emotion is joy. [motivation] her legs' motivation is the need to move forward. [likely consequence] the woman will trip and fall [social norm] it's good to spend time with your children. [emotion] woman's emotion is happy. [motivation] woman's motivation is to spend time with her daughter. [likely consequence] was able to enjoy the company of her daughter. [ethics] [s] i got sick the last time i ate there, so i recommend a different restaurant to my in-laws. [q] reaction: this is (a) wrong (b) not wrong gold ans: not wrong. [social norm] if something makes you sick, don't eat there [emotion] in-laws' emotion is anger. food's emotion is disgust. [motivation] bacteria's motivation is to make more bacteria. my motivation is i don't want them to get sick. [likely consequence] they will eat there once and get sick [social norm] it's good to make recommendations to others. [emotion] i (myself)'s emotion is responsible. [motivation] i (myself)'s motivation is to protect others. [likely consequence] my in-laws have a great meal at the new restaurant i recommended. ## scene elaborations (se) we focus on the task of situated reasoning about social situations where the input consists of (a textual description of) a situation s, and question q testing the model's understanding of the situation (both explicitly mentioned and implicitly indicated facts). the output is the answer a to the question. figure 1 shows an example. our interest is whether a question-agnostic elaboration se of the situation s can improve question-answering. ## our model: dream in addition to probing, we also explore whether we can train lms to build improved scene elaborations, and whether they can improve qa performance. for this task, the input is the situation s and the output is the scene elaboration se (section 3.1). ## experiments we conduct experiments to address three questions, using macaw as our representative lm: q1. to what extent can macaw generate an accurate and consistent scene elaboration? q2. to what extent does our trained scene elaboration generator, dream, improve on this? q3. can the scene elaborations produced by dream help improve qa? ## results as described in section 3.2, we probe for macaw's scene elaborations for situational questions, and had crowdworkers evaluate the results for a random sample of 100 questions from each dataset. 6 the results are in the "macaw" lines in ## analysis and discussion as additional views of the results, examining the random sample of 400 questions rated by crowdworkers, we find that almost all (98.75%) of the 400 scene elaborations generated by dream have at least something that is true and 78.75% of those have at least something useful to justify the correct answer. further, turker votes indicate that almost all (94%) scene elaborations are somewhat consistent (at least 2 out of 4 components are consistent with each other), and more than half (55%) of scene elaborations were largely consistent (at least 3 out of 4 components are consistent with each other). ## conclusion can lms answer situated questions more accurately if they are provided with additional, pertinent details about those situations -a scene elaboration -before answering? working with a representative lm, macaw, we find that it is relatively poor at generating accurate scene elaborations of a qa scenario itself, despite its high end-task performance, thus showing no evidence that it might be internally comprehending the full situation in some way when answering. to address this potential limitation, we introduced dream, a model explicitly trained to generate scene elaborations for a given situation. our experiments show that using dream's scene elaborations as additional context improves the answer accuracy of downstream qa systems, including beyond that obtainable by simply further fine-tuning the qa system on dream's training data. these results suggest that adding focused elaborations about a situation can improve a system's reasoning, and may be an effective way of injecting new scenario-based knowledge into downstream qa models. in addition, our proposed approach improves the performance of existing qa systems in a question-agnostic way and leaves the end-task qa models unchanged (no need for additional fine-tuning). this helps prevent issues such as interference across different task capabilities and catastrophic forgetting @xcite . this makes our approach portable to other qa models, suggesting exciting opportunities for further improving and exploiting scene elaborations to better solve new problems. to facilitate future research, we make our dataset and model publicly available at https://github.com/allenai/dream .