Chain of Condition: Construct, Verify and Solve Conditions for Conditional Question Answering (2024)

Abstract: Conditional question answering (CQA) is an important task that aims to find probable answers and identify missing conditions. Existing approaches struggle with CQA due to two challenges: (1) precisely identifying necessary conditions and the logical relationship, and (2) verifying conditions to detect any that are missing. In this paper, we propose a novel prompting approach, Chain of condition, by first identifying all conditions and constructing their logical relationships explicitly according to the document, then verifying whether these conditions are satisfied, finally solving the logical expression to indicate any missing conditions and generating the answer accordingly. Experiments on two CQA benchmark datasets show our chain of condition outperforms existing prompting baselines, establishing a new state of the art. Furthermore, with only a few examples, our method can facilitate GPT-3.5-Turbo or GPT-4 to outperform all existing supervised models.

## introduction conditional question answering (cqa) aims to answer questions where the information provided by the user may not be sufficient, and any missing information should be requested from the user to determine the answer @xcite @xcite @xcite . cqa is a challenging and promising task, which has been gaining increasing attention recently @xcite @xcite @xcite . figure 1 shows an example. the user asks for the amount of benefit she would receive, but according to the policy, the applicant must not claim for other benefits and has an unemployment certificate as prerequisite. these ## preliminary we investigate the conditional question answering (cqa) task, where all missing condition should be requested from the user to determine the answer. formally, the task's input consists of the user's question q and scenario s providing some background information, and a reference document d describing the policy being questioned. the an-swer should be inferred from the document. unlike other qa tasks, the document in the cqa task contains many conditions @xmath0; c 2 ; ...; c n } ⊂ d that must be satisfied by the user. the complete output includes the answer along with any unmentioned conditions @xmath1, c (u) ), where ## methodology we introduce chain of condition, a novel approach to guide large language models (llms) for conditional question answering tasks. our chain of condition pipeline is similar to the human process for dealing with such problems, thus can demonstrate a more coherent and interpretable solving path for users to read, as illustrated in figure 2 . chain of condition includes three steps: condition identification and logical construction, condition verification, expression resolution and an-swer generation. our framework decompose the original cqa task into smaller sub-tasks, allowing the model to solve them sequentially. first, we identify all conditions and recognize their logical relationships according to the document, forming a logical expression of conditions. secondly, we verify each condition's fulfillment in the user's description. finally, we take the verification results to solve the logical expression and identify all missing conditions through python interpreter, and generate the answer by taking the condition solutions into account. a prompt example for each step is shown in appendix g. ## conclusion in this work, we propose chain of condition, a novel prompting approach for conditional question answering. our approach prompts llms to identify conditions, organize them into logical expressions and introduces a python interpreter for resolution, effectively improving precision and enhancing interpretability. experiments on all benchmark datasets show that chain of condition outperforms existing prompting baselines. additionally, our method surpasses supervised baselines when utilizing a strong backbone model, demonstrating the promising future of prompting llms for cqa and paving the way for future research directions.