HPE : Answering Complex Questions over Text by Hybrid Question Parsing and Execution (2023)

Abstract: The dominant paradigm of textual question answering systems is based on end-to-end neural networks, which excels at answering natural language questions but falls short on complex ones. This stands in contrast to the broad adaptation of semantic parsing approaches over structured data sources (e.g., relational database, knowledge graphs), that convert natural language questions to logical forms and execute them with query engines. Towards combining the strengths of neural and symbolic methods, we propose a framework of question parsing and execution on textual QA. It comprises two central pillars: (1) We parse the question of varying complexity into an intermediate representation, named H-expression, which is composed of simple questions as the primitives and symbolic operations representing the relationships among them; (2) To execute the resulting H-expressions, we design a hybrid executor, which integrates the deterministic rules to translate the symbolic operations with a drop-in neural reader network to answer each decomposed simple question. Hence, the proposed framework can be viewed as a top-down question parsing followed by a bottom-up answer backtracking. The resulting H-expressions closely guide the execution process, offering higher precision besides better interpretability while still preserving the advantages of the neural readers for resolving its primitive elements. Our extensive experiments on MuSiQue, 2WikiQA, HotpotQA, and NQ show that the proposed parsing and hybrid execution framework outperforms existing approaches in supervised, few-shot, and zero-shot settings, while also effectively exposing its underlying reasoning process.

## introduction end-to-end neural models that transductively learn to map questions to their answers have been the dominating paradigm for textual question answering @xcite ow1 our code is available at https://github.com/ salesforce/hpe question: when was the last time duane courtney's team beat the winner of the 1894-95 fa cup? ing to their flexibility and solid performance. however, they often suffer from a lack of interpretability and generalizability to more complex scenarios. symbolic reasoning, on the other hand, relies on producing intermediate explicit representations such as logical forms or programs, which can then be executed against a structured knowledge base (e.g., relational database, knowledge graph, etc.) to answer questions @xcite @xcite . these methods naturally offer better interpretability and precision thanks to the intermediate symbolic representations and their deterministic executions. however, they might be limited in expressing a broad range of natural questions in the wild depending on the semantic coverage of the underlying symbolic language and its grammar. neural module networks @xcite @xcite have been proposed to combine neural and symbolic modality together. however, they require a symbolic language and a corresponding model that only covers limited scenarios in the specific task or domain. to apply this approach on new tasks or domains, new languages and neural modules need to be introduced. therefore, designing a generalizable framework that uses a high-coverage symbolic expression and a flexible neural network that can be versatilely used in various scenarios becomes our goal. in this work, we propose a hybrid question parser and execution framework, named hpe, for textual question answering, which combines neural and symbolic reasoning. we introduce h-expression as an explicit representation of the original question, which contains primitives and operations @xcite . as shown in figure 1 , we consider the single-hop questions as primitives (leaves) and use symbolic operations (internal nodes) to connect them hierarchically. we introduce a semantic parser based on a seq2seq framework to parse a complex question into its corresponding h-expression in a top-down manner. to execute an h-expression, we design a hybrid executor (h-executor), which utilizes a neural model (reader) to answer each single-hop question, and then uses deterministic rules to hierarchically compose a final answer from the singlehop answers in a bottom-up fashion. notably, hexpression facilitates modularity in that the reader is replacable and its training can be done globally with the massive single-hop qa data. our contributions can be summarized as follows: although our framework shares some similarities with this line of studies, there is a crucial difference in that we keep both symbolic and neural representations coincide, whereas they use a neural model to replace the non-differentiable symbolic representation to be able to train end-to-end. furthermore, our main contribution is the use of hexpressions and the answer backtracking procedure for qa and not the question decomposition. ## approach we formulate textual question answering as the task of answering a question q given the textual evidence provided by a set of passages. we assume access to a dataset of tuples {(q i , a i , p i )} n @xmath0, where a i is a text string that defines the correct answer to question q i with p i being the passage set. question: when was the last time duane courtney's team beat the winner of the 1894-95 fa cup? h-parser h-expression: join ( qa( when was the last time ans#2 beat ans#1? ) , union ( qa( what is member of sports team of duan courtney ), qa( who is winner of 1894-95 fa cup ) ) ) join qa( when was the last time ans#2 beat ans#1? ) union qa( what is member of sports team of duan courtney? ) qa( who is winner of 1894-95 fa cup? ) q1 q2 q3 figure 2 : an overview of parsing with h-parser, which involves translating the input question into an hexpression, and subsequently reshaping it into a treestructure, facilitating the determination of the node's execution sequence. in this work, we cast the task as question parsing with hybrid execution. for a given question q i , a question parser (h-parser) is tasked to generate the corresponding h-expression l i (sec. 3.1). this along with the supporting passage set p i are then given to the execution model (h-executor) to generate an answer (sec. 3.2). the h-executor uses a reader for simple question answering and executes the symbolic operations based on answers in a bottom-up manner to get the final prediction. ## experiments we conduct experiments on two multi-hop multipassage textual qa datasets, musique and 2wikimulti-hopqa, which contain complex questions and the corresponding decomposed simple questions for the supervised setting. we also test models' generalization on the few shot setting using 5-20% of the training data. in real scenarios, neither the decomposed questions nor the complexity of questions is known. therefore, we also investigate our models under the zero-shot setting on both complex (hotpotqa) and simple (nq) qa datasets. in the end, we carry out a case study to show the interpretability of our framework. ## conclusion we propose hpe for answering complex questions over text, which combines the strengths of neural network approaches and symbolic approaches. we parse the question into h-expressions followed by the hybrid execution to get the final answer. our extensive empirical results demonstrate that hpe has a strong performance on various datasets under supervised, few-shot, and zero-shot settings. moreover, our model has a strong interpretability exposing its underlying reasoning process, which facilitates understanding and possibly fixing its errors. by replacing our text reader with kb or table based neural network, our framework can be extended to solve kb and table qa.