Question-Answer Sentence Graph for Joint Modeling Answer Selection (2023)

Abstract: This research studies graph-based approaches for Answer Sentence Selection (AS2), an essential component for retrieval-based Question Answering (QA) systems. During offline learning, our model constructs a small-scale relevant training graph per question in an unsupervised manner, and integrates with Graph Neural Networks. Graph nodes are question sentence to answer sentence pairs. We train and integrate state-of-the-art (SOTA) models for computing scores between question-question, question-answer, and answer-answer pairs, and use thresholding on relevance scores for creating graph edges. Online inference is then performed to solve the AS2 task on unseen queries. Experiments on two well-known academic benchmarks and a real-world dataset show that our approach consistently outperforms SOTA QA baseline models.

## introduction automated question answering (qa) research has received renewed attention thanks to diffusion of virtual assistants. for example, google home, siri, and alexa provide general information inquiry services, while many others serve customer requests in very different application domains. two main tasks have been widely studied: (i) answer sentence selection (as2), which, given a question and set of answer-sentence candidates, consists of selecting sentences (e.g., retrieved by a search engine) that correctly answer the question; and (ii) machine reading (mr), e.g., @xcite , which, given a question and reference text, involves finding an exact text span that answers the question. as2 models can more efficiently target large text databases (as they originated from the trec-qa track @xcite ) and there is evidence that they are currently used in personal assistants, e.g., alexa @xcite . the approach above was significantly improved by the answer support-based reranker (asr/masr) @xcite , which jointly models answer candidates of each question. essentially, the authors showed that answer candidates bring additional information for determining if a target answer t is correct, and proposed an ad-hoc joint model. asr/masr uses tanda as fundamental building blocks for its model, but improves on tanda's score via a component with aa similarity relations. our model further improves this via a qq similarity component integrated with a graph neural network (gnn) to capture more interrelation dependency. a way to go beyond asr/masr's approach is to consider other questions similar to the target one along with their answers for deciding over t. for example, the selection ability of the as2 model would improve if it could learn the additional context. the latter can be provided by other similar question/answer pairs. for example, in the lower part of the correctness characterization provided by the similar questions above can hardly be learned by the model from individual (q, c) pairs, as the characterization does not hold in general. it can be applied relatively to the list of candidates, {q, c 1 , c 2 , c 3 }. therefore, these patterns should be learned from comparing similar questions together with their list of candidates. in this paper, we propose to jointly model questions with their list of candidates for as2. this enables the usage of the information from other questions, similar to the given question, together with their answers. for this purpose, we design a graph-based qa model relying on the assumption that, given question q and a corresponding answer a, there exist other questions, q i and answers a i semantically similar to q, and a, respectively. these similar questions and answers can provide evidence to decide the correctness of a. to model the complex interactions among these different sentences, we use graph neural networks (gnn) @xcite applied to graphs which utilize interaction among question and answer pairs. our main innovation regards graph construction: (i) we propose to build a different small graph for each target question, such that we improve efficiency, and effectiveness as we use specific relevant questions; (ii) as we target answer selection, which traditionally is modeled as classification of question and answer pairs, (q, a), we associate graph nodes with them. we use training data to assign pairs with positive and negative labels. (iii) we form edges between pairs using models that automatically score different types of relations: question-question (qq), question-answer (qa), and answer-answer (aa), and then we apply thresholds to reduce the number of active edges. we test our models over three datasets, wik-iqa, trec-qa, and wqa, where the latter is an internal dataset built with de-identified customer questions. our gnn for qa improves the best pointwise model for as2, i.e., tanda, over all datasets (up to 7 absolute points on trec-qa corresponding to 75% of error reduction in accuracy). it also establishes the new sota among joint models, improving asr by 4 and 2 absolute points on wikiqa and trec-qa, respectively. ## methodology in this section, we first present an overview of our qa graph model, followed by a discussion of our graph construction procedure. then we describe our gnn model design, and lastly detail our training and inference framework. ## training and inference figure 2 summarizes eqag-gnn's pipeline. ## experiments here, we compare eqag-gnn against sota qa models. then, we show an ablation study for eqag-gnn for the best hyperparameters. finally, we discuss a case study with error analysis. ## case study we provide a case study and error analysis comparing performance of eqag with tanda. ta-q: how is jameson irish whiskey made ? c 1 : jameson is similar in its adherence to the single distillery principle to the single malt tradition but jameson blends column still spirit with single pot still whiskey a combination of malted barley with unmalted or " green " barley distilled in a pot still. ## conclusions to our knowledge, our model is the first graphbased approach for jointly modeling sentence-level semantics of question-answer pairs for as2 as an offline processing application, such as those required by community qa, forums, etc. this is different from previous methods using graphs, e.g., multihop or graph-based qa, which mainly model semantics via entities. our approach builds query-specific small-scale training graphs for offline learning, through (q, a) pairs as nodes, and edges encoding relations between members of pairs to capture both supporting question-question, and answer-answer dependencies. further, we demonstrate that our approach achieves significant performance gains over existing sota models on as2 for metrics of p@1, map, and mrr.