M u GER 2: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid Question Answering (2022)

Abstract: Hybrid question answering (HQA) aims to answer questions over heterogeneous data, including tables and passages linked to table cells. The heterogeneous data can provide different granularity evidence to HQA models, e.t., column, row, cell, and link. Conventional HQA models usually retrieve coarse- or fine-grained evidence to reason the answer. Through comparison, we find that coarse-grained evidence is easier to retrieve but contributes less to the reasoner, while fine-grained evidence is the opposite. To preserve the advantage and eliminate the disadvantage of different granularity evidence, we propose MuGER2, a Multi-Granularity Evidence Retrieval and Reasoning approach. In evidence retrieval, a unified retriever is designed to learn the multi-granularity evidence from the heterogeneous data. In answer reasoning, an evidence selector is proposed to navigate the fine-grained evidence for the answer reader based on the learned multi-granularity evidence. Experiment results on the HybridQA dataset show that MuGER2 significantly boosts the HQA performance. Further ablation analysis verifies the effectiveness of both the retrieval and reasoning designs.

## introduction traditional knowledge-based question answering systems derive answers to questions based on homogeneous knowledge, including knowledge graphs @xcite , tables @xcite , or passages @xcite and achieve remarkable performances. however, they neglect a more general scenario requiring reasoning over heterogeneous data to answer a question. to investigate this challenge, @xcite figure 1 : two example questions of hqa task. existing hqa models usually consist of two components, a retriever to learn evidence and a reasoner to leverage the evidence to derive the answer. intuitively, the heterogeneous data shown in figure 1 can provide different granularity evidence to the hqa models, such as coarse-grained column and row and fine-grained cell and link. previous models always retrieve coarse or fine-grained evidence and directly use a span-based reading comprehension model to reason the answer. for example, @xcite choose a coarse-grained region as the evidence, e.g., a table row. on the contrary, @xcite and @xcite focus on the fine-grained units, table cell and link. intuitively, compared with fine-grained evidence, coarse-grained evidence is easier to be accurately retrieved. however, it contributes less to the reasoner's performance since it contains more noisy information. the fine-grained evidence is just the opposite for the retriever and the reasoner. to prove this argument, we conduct experiments on the development set of hybridqa to analyze the retriever and the reasoner performance foot_0 with different granularity evidence. the results of evidence retrieval and answer reasoning are evaluated with the retrieval recall (r@1) and the f1 score, respectively. as shown in figure 2 , the coarse-grained evidence achieves better retrieval recalls but lower f1 scores than the fine-grained evidence. with the granularity changing from coarse to fine, the f1 score increases while the r@1 declines. the results are consistent with our hypothesis. to preserve the advantage and eliminate the disadvantage of different granularity evidence, we propose muger 2 , a multi-granularity evidence retrieval and reasoning approach for hqa. in the retrieval stage, a unified retriever is designed to learn the multi-granularity evidence from the heterogeneous data involving columns, rows, cells, and links. compared with existing methods, the multi-granularity evidence preserves the retrieval recall and provides more aspects of information to reason the answer. in the reasoning stage, to avoid the redundant information in the multi-granularity evidence that lowers the answer reader's performance, an evidence selector (e-sel) is designed to navigate the fine-grained evidence for the reader by fusing the learned multi-granularity evidence. together with our multi-granularity evidence retrieval and reasoning designs, our muger 2 preserves both the retrieval recall and the f1 score and further boosts the end-to-end hqa performance. we conduct extensive experiments on the hy-bridqa dataset to verify the effectiveness of our proposed muger 2 . experiment results show that muger 2 achieves remarkable improvements, which outperforms a publicly available strong baseline by 10.0% em and 12.9% f1 scores on the end-to-end hqa performance. ablation studies verify the effectiveness of the proposed multigranularity evidence retrieval and the evidence selector. moreover, results compared with different mono-granularity baselines prove the effectiveness of multi-granularity evidence for hqa. our contributions are as follows: (1) we analyze the limitation of hqa systems which solely rely on only one certain granularity evidence in both the retriever and reasoner and propose a multi-granularity evidence retrieval and reasoning approach, which boosts the end-to-end hqa results. (2) we propose a joint retrieval method that improves the evidence retrieval performance and an evidence selector to accurately navigate the fine-grained evidence from the multi-granularity information to preserve the reader's performance. (3) we conduct extensive experiments on the hybridqa dataset and show the effectiveness of muger 2 . ## case study through statistics on the development set of hy-bridqa, we find there are 135 questions whose answers are only recalled by our multi-granularity evidence, but failed to be recalled by the coarseand fine-grained evidence. an example of such a question is given in figure 5 . the table along with linked passages on the left is the given knowledge to answer the question. the answer facebook watch is contained in link1 of cell(2,0) where (2,0) represents (row2, column0). the table on the right lists the retrieved different granularity evidence. it is observed that the q : which social network video platform does the program broacast since 1992 on mtv by viacomcbs now air ？ a : facebook watch title years network postmodern mtv 1980s-1990s mtv profiles in rock 1982 mtv ... real world 1992-present mtv biorhythm 1998 mtv singled out 1995-1998 mtv postmodern mtv was a nightly program in the late 1980s and very early 1990s on mtv .... singled out is ...mtv revamped the series in late 2018 for its youtube channel .... row0 row1 row2 row3 column0 column1 column2 link1 link0 link2 cell(1,0) granularity retrieval reasoning column column0 mtv classic row row4 youtube cell cell(4,0) youtube link link2 youtube multi column0 , row2 , cell(3,0) , link1 facebook watch row4 the real world is .... that currently airs on facebook watch after airing on mtv from 1992-2017 .... 40% 57% multi-granularity evidence retrieval evidence selector reading comprehension figure 6: error of the three modules of muger 2 -large. although the column granularity based method retrieves the @xmath0, the reading comprehension model fails to derive the answer text accurately from the column. in contrast, our method retrieves evidence @xmath1, row, cell, link>. the evidence is correctly retrieved except for the cell since the question is an in-passage question. the result demonstrates that the multi-granularity evidence improves the retrieval recall which benefits from learning the interactive information. the evidence selector further help to navigate the fine-grained cell or link based on the multi-granularity evidence. ## error analysis to further analyze our method, we also make statistics on error cases in the prediction of muger 2large. the error statistics are based on the development set of hybridqa. as shown in figure 6 , these cases can be divided into three categories according to the causes. the first category is caused by evidence retrieval errors, which means the retrieved evidences overlook the answer. according to our statistics, multi-granularity evidence has an advantage in improving coverage of answers. therefore, the proportion of this category is very low, only 3%. the second category is caused by the evidence selector, which accounts for about 40% in the whole results. there are two reasons for the evidence selector to make the error. firstly, although multi-granularity evidence improves the coverage, it also introduces noise. as a result, it may also lead to a mis-navigation of the fine-grained cell or link. in our results, it accounts for about 33%. in addition, for those questions which are offered correct cells or links, they also may be answered incorrectly due to the wrong judgment of the answer type. it accounts for about 7% in our results. the last category is caused by the reading comprehension module. limited by the performance of the rc model, even if feed correct fine-grained evidence, the rc model also may produce the wrong answer. according to our analysis, this category has the maximum proportion, accounting for about 57% of the whole error cases. ## discussion more recently, two novel approaches, mate @xcite and mitqa @xcite , are proposed and achieve significant results, respectively 70.0 and 71.9 f1 scores on the hybridqa test set. specifically, mate mainly focuses on strengthening the ability to encode large tables with multi-view attention. while mitqa designs a multi-instance training method based on distant supervision to filter the noise from multiple answer spans. in contrast, our muger 2 puts more effort into integrating multi-granularity evidence for the hqa problem to enhance the reasoning ability. we think that the two novel approaches and our model pay attention to different perspectives for solving the hqa problems, and it's probably to combine them to achieve better performance. ## related work recently, many researchers attempt to tackle the question answering task using heterogeneous knowledge @xcite @xcite @xcite . however, traditional researches only combine the information of heterogeneous knowledge, but fail to answer questions that require reasoning over heterogeneous data. to fill in the gap, the hybrid question answering (hqa) task together with the hybridqa dataset is proposed by @xcite . hybridqa provides a wikitable @xcite along with its hyperlinked wikipedia passages @xcite as evidence for each question. based on the hybridqa, @xcite further propose the ott-qa dataset, which requires the system to retrieve relevant tables and text for the given questions. zhu et al. (2021) and and chen et al. (2021b) propose tat-qa and finqa requiring numerical reasoning over the heterogeneous data. existing works on hybridqa usually retrieve mono-granularity evidence from the heterogeneous data to derive the answer. hybrider @xcite proposes a two-phase pipeline framework to retrieve a table cell as the evidence and feeds its value and linked passages into an rc model to extract the answer. dochopper @xcite propose an end-to-end multi-hop retrieval method to directly retrieve a passage sentence or a cell value as the evidence. in addition, @xcite explore an unsupervised multi-hop qa model mqa-qg, which can generate human-like multi-hop training data from heterogeneous data resources. ## conclusion we propose muger 2 , a multi-granularity evidence retrieval and reasoning approach for hybrid question answering (hqa). in our method, a unified retriever is designed to learn multi-granularity evidence and an evidence selector is proposed to navigate fine-grained evidence for the reader. experiment results show that the muger 2 boosts the end-to-end hqa performance and outperforms the strong baselines on the hybridqa benchmark.