PACIFIC : Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance (2022)

Abstract: To facilitate conversational question answering (CQA) over hybrid contexts in finance, we present a new dataset, named PACIFIC. Compared with existing CQA datasets, PACIFIC exhibits three key features: (i) proactivity, (ii) numerical reasoning, and (iii) hybrid context of tables and text. A new task is defined accordingly to study Proactive Conversational Question Answering (PCQA), which combines clarification question generation and CQA. In addition, we propose a novel method, namely UniPCQA, to adapt a hybrid format of input and output content in PCQA into the Seq2Seq problem, including the reformulation of the numerical reasoning process as code generation. UniPCQA performs multi-task learning over all sub-tasks in PCQA and incorporates a simple ensemble strategy to alleviate the error propagation issue in the multi-task learning by cross-validating top- k sampled Seq2Seq outputs. We benchmark the PACIFIC dataset with extensive baselines and provide comprehensive evaluations on each sub-task of PCQA.

## introduction financial question answering (qa) systems aim to answer user's instant queries by selecting appropriate information from financial documents, which often contain a hybrid of tabular and textual content, and performing complex quantitative analysis. existing studies on financial qa @xcite @xcite mainly focus on building single-turn qa systems to passively respond to user queries. however, in real-world information-seeking applications @xcite , the system is expected to (i) answer highly context-dependent questions in a multi-turn conversation, and (ii) proactively assist users in performing complicated information seeks. in an interactive setting, users tend to ask followup or co-referencing questions @xcite without repeating previous information, and provide a succinct or brief query that may be ambiguous or lack the necessary content. especially in financial qa, the user queries often contain multiple constraints from different aspects for the concerned objective, as the examples shown in fig. 1 . even just missing one constraint may cause ambiguity. therefore, a proactive conversational system that can help clarify the ambiguity is of great importance in financial qa. to this end, this paper introduces a new dataset to promote research into proactive conversational question answering in finance, named pacific. pacific is constructed by using the qa pairs in an expert-annotated financial qa dataset, tat-qa @xcite , as guidance to build conversation sessions with consecutive topics. as shown in fig. 1 , we rewrite the original selfcontained questions into conversational questions with anaphora (co-referencing among different turns) and ellipsis (omitting repeated words in the follow-up questions), as well as construct ambiguous questions that require clarification. accordingly, we define a new task, named proactive conversational question answering (pcqa), which combines the problems of clarification question generation (cqg) @xcite and conversational question answering (cqa) @xcite . pcqa consists of three sub-tasks: (i) given the user's query, the system first identifies whether the question is ambiguous (i.e., clarification need prediction). (ii) if so, the system will proactively ask a clarifying question to clarify the uncertainty (i.e., cqg). (iii) if not, it will directly answer the question (i.e., cqa). compared with existing datasets listed in ## method we introduce the unipcqa model, which unifies all sub-tasks in pcqa as the seq2seq problem and performs multi-task learning among them. clarification need prediction clarification question generation conversational question answering dialogue history c t document d y t clari. question r t answer r t input response r t output ## experiments we first evaluate methods on two widely-studied tasks in conversational information seeking, including (i) clarification question generation (cqg) and (ii) conversational question answering (cqa). then we benchmark the overall performance of proactive conversational question answering (pcqa) on pacific. ## conclusions in this paper, we present a new dataset, pacific, for proactive conversational question answering over a hybrid context of tables and text. accordingly, we define the problem of proactive conversational question answering that combines clarification question generation and conversational question answering. in addition, we reformulate the numerical reasoning process as code generation and recast all sub-tasks in pcqa into a seq2seq problem solved by a unified model, unipcqa. extensive experiments show that the pacific dataset is very challenging and demonstrate the need to build models that can handle hybrid input and output formats as well as diverse numerical reasoning.