CURIE : An Iterative Querying Approach for Reasoning About Situations (2022)

Abstract: Predicting the effects of unexpected situations is an important reasoning task, e.g., would cloudy skies help or hinder plant growth? Given a context, the goal of such situational reasoning is to elicit the consequences of a new situation (st) that arises in that context. We propose CURIE, a method to iteratively build a graph of relevant consequences explicitly in a structured situational graph (st graph) using natural language queries over a finetuned language model. Across multiple domains, CURIE generates st graphs that humans find relevant and meaningful in eliciting the consequences of a new situation (75% of the graphs were judged correct by humans). We present a case study of a situation reasoning end task (WIQA-QA), where simply augmenting their input with st graphs improves accuracy by 3 points. We show that these improvements mainly come from a hard subset of the data, that requires background knowledge and multi-hop reasoning.

## introduction a long-standing challenge in reasoning is to model the consequences of an unseen situation in a context. in the real world unexpected situations are common. machines capable of situational reasoning are crucial because they are expected to gracefully handle such unexpected situations. for example, when eating leftover food, would it be more safer from virus if we microwave the food? -answering this requires understanding the complex events virus contamination and effect of heat on virus. much of this information remains implicit (by grice's maxim of quantity @xcite ), thus requiring inference. recently, nlp literature has shown renewed interest in situational reasoning with applications in qualitative reasoning @xcite ; rq2: situational graphs improve situational reasoning qa when appended to the question context. @xcite , physical commonsense reasoning @xcite , and defeasible inference @xcite . these tasks take as input a context providing background information, a situation (st), and an ending, and predict the reachability from st to that ending. however, these systems have three limitations: (i) systems trained on these tasks are often domain specific, (ii) these tasks do not require a supporting structure that elicits the dynamics of the reasoning process, and (iii) these tasks are addressed as a classification problem restricting to a closed vocabulary setting. to address these limitations, we propose curiea system to iteratively query pretrained language models to generate an explicit structured graph of consequences, that we call a situational reasoning graph (st-graph). the task is illustrated in figure 1 : given some context and situation st (short phrase), our system generates a st-graph based on the contextual knowledge. curie supports the following kinds of reasoning: as shown in figure 1 , our approach to this task is to iteratively compile the answers to questions 1 and 2 to construct the st-graph where imminent/eventual capture multihop reasoning questions. compared to a free-form text output obtained from an out-of-the-box sequence-to-sequence model, our approach gives more control and flexibility over the graph generation process, including arbitrarily reasoning for any particular node in the graph. the generated st-graphs are of high quality as judged by humans for correctness. in addition to human evaluation, we also show that a downstream task that requires reasoning about situations can compose natural language queries to construct a st-reasoning graph via curie. the resulting st-graph can be simply augmented to their input to achieve performance gains, specifically on the subset of hard questions that require background knowledge and multihop reasoning. in summary, this paper addresses the following research questions: rq1: given a context and a situation, how can we generate a situational reasoning (st) graph? to answer rq1, we present curie, the first domain-agnostic situational reasoning system that takes as input a context and a situation st and iteratively generates a situational reasoning graph ( §2). our system is effective at situational reasoning across three datasets as validated by human evaluation and automated metrics. rq2: can the st-graphs generated by curie improve performance of a downstream task? to answer rq2, we show that st graphs generated by curie improve a st-reasoning task (wiqa-qa) by 3 points on accuracy by simply augmenting their input with our generated situational graphs, especially for a hard subset that requires background knowledge and multi-hop reasoning ( §4). ## curie for situational reasoning curie provides both a general framework for situational reasoning and a method for constructing streasoning graphs from pretrained language models. ## rq1: establishing baselines for st-graph generation this section reports on the quality of the generated st reasoning graphs and establishes strong baseline scores for st-graph generation. we use the datasets described in section §2.2 for our experiments. for language model m. we find that context is essential for performance (w/o t ). we provide these baseline scores as a reference for future research. ## rq2: curie for downstream tasks in this section, we describe the approach for augmenting st graphs for downstream reasoning tasks. we first identify the choice of tasks (st-tasks) for domain adaptive pretraining @xcite and obtain curie language model m (based on gpt-2). the downstream task then provides input context, st and (relation, type) tuples of interest, and obtains the st-graphs (see algorithm 1) from curie. we describe one such instantiation in §4.1. ## conclusion we present curie, a situational reasoning that: (i) is effective at generating st-reasoning graphs, validated by automated metrics and human evaluations, (ii) improves performance on two downstream tasks by simply augmenting their input with the generated st graphs. further, our framework supports recursively querying for any node in the st-graph. our future work is to design models that seek consistency, and study recursive st-reasoning as a bridge between dialog and reasoning.