H ybrid QA : A Dataset of Multi-Hop Question Answering over Tabular and Textual Data (2020)

Abstract: Existing question answering datasets focus on dealing with homogeneous information, based either only on text or KB/Table information alone. However, as human knowledge is distributed over heterogeneous forms, using homogeneous information alone might lead to severe coverage problems. To fill in the gap, we present HybridQA, a new large-scale question-answering dataset that requires reasoning on heterogeneous information. Each question is aligned with a Wikipedia table and multiple free-form corpora linked with the entities in the table. The questions are designed to aggregate both tabular information and text information, i.e., lack of either form would render the question unanswerable. We test with three different models: 1) a table-only model. 2) text-only model. 3) a hybrid model that combines heterogeneous information to find the answer. The experimental results show that the EM scores obtained by two baselines are below 20%, while the hybrid model can achieve an EM over 40%. This gap suggests the necessity to aggregate heterogeneous information in HybridQA. However, the hybrid model’s score is still far behind human performance. Hence, HybridQA can serve as a challenging benchmark to study question answering with heterogeneous information.

## introduction question answering systems aim to answer any form of question of our interests, with evidence provided by either free-form text like wikipedia passages @xcite @xcite or structured data like freebase/wikidata @xcite @xcite and wikitables @xcite . both forms have their advantages, the free-form corpus has in general better coverage while structured data has better compositionality to handle complex multi-hop questions. due to the advantages of different representation forms, people like to combine them in real world applications. therefore, it is sometime not ideal to assume the question has answer in a passage. this paper aims to simulate a more realistic setting where the evidences are distributed into heterogeneous data, and the model requires to aggregate information from different forms for answering a question. there has been some pioneering work on building hybrid qa systems @xcite @xcite . these methods adopts kb-only datasets @xcite @xcite to simulate a hybrid setting by randomly masking kb triples and replace them with text corpus. experimental results have proved decent improvement, which shed lights on the potential of hybrid question answering systems to integrate heterogeneous information. though there already exist numerous valuable questions answering datasets as listed in to fill in the gap and construct a heterogeneous qa dataset, we present hybridqa, which is collected by crowdsourcing based on wikipedia tables. during annotation, each crowd worker is presented with a table along with its hyperlinked wikipedia passages to propose questions requiring multi-hop reasoning over both forms of informa- name year season flag bearer xxxi 2016 summer yan naing soe xxx 2012 summer zaw win thet xxix 2008 summer phone myint tayzar xxviii 2004 summer hla win u xxvii 2000 summer maung maung nge xx 1972 summer win maung the 2016 summer olympics officially known as the games of the xxxi olympiad (portuguese : jogos da xxxi olimpíada) and commonly known as rio 2016 , was an international multi-sport event …… yan naing soe ( born 31 january 1979 ) is a burmese judoka . he competed at the 2016 summer olympics in the men 's 100 kg event , …… he was the flag bearer for myanmar at the parade of nations . win maung ( born 12 may 1949 ) is a burmese footballer . he competed in the men 's tournament at the 1972 summer olympics … zaw win thet ( born 1 march 1991 in kyonpyaw , pathein district , ayeyarwady division , myanmar ) is a burmese runner who …… q: where does the burmesse jodoka participate in the olympic opening ceremony as a flag bearer? a: rio q: when does the oldest flag burmese bearer participate in the olympic ceremony? a: 1972 q: for the olympic event happening after 2014, what session does the flag bearer participate? a: parade of nations q: for the xxxi and xxx olympic event, which has an older flag bearer? a: xxxi q: which event does the does the xxxi olympic flag bearer participate in? a: men's 100 kg event hardness …… myint tayzar phone ( burmese : ြမင့ ် ေတဇာဖ+ န် း ) born july 2 , 1978 ) is a sprint canoer from myanmar who competed in the late 2000s . q: in which year did the judoka bearer participate in the olympic opening ceremony? a: 2016 figure 1: examples of annotated question answering pairs from wikipedia page 2 . underlined entities have hyperlinked passages, which are displayed in the boxes. the lower part shows the human-annotated question-answer pairs roughly categorized based on their hardness. dataset size #docs kb/ table multi-hop webquestions 5.8k no yes yes webqsp 4.7k no yes yes webqcomplex 34k no yes yes metaqa 400k no yes yes wikitableqa 22k no yes yes squad-v1 107k 1 no no drop 99k 1 no yes triviaqa 95k >1 no no hotpotqa 112k >1 no yes natural-qa 300k >1 no yes hybridqa 70k >>1 yes yes in our experiments, we implement three models, namely table-only model, passage-only, and a heterogeneous model hybrider, which combines both information forms to perform multi-hop reasoning. our experiments show that two homogeneous models only achieve em lower than 20%, while hybrider can achieve an em over 40%, which concludes the necessity to do multihop reasoning over heterogeneous information on hybridqa. as the hybrider is still far behind human performance, we believe that it would be a challenging next-problem for the community. ## dataset in this section, we describe how we crawl highquality tables with their associated passages, and then describe how we collect hybrid questions. the statistics of hybridqa is in table/passage collection to ease the annotation process, we apply the following rules during table crawling. 1) we need tables with rows between 5-20, columns between 3-6, which is appropriate for the crowd-workers to view. 2) we restrain the tables from having hyperlinked cells over 35% of its total cells, which provide an abundant amount of textual information. for each hyperlink in the table, we retrieve its wikipedia page and crop at most the first 12 sentences from its introduction session as the associated passage. 3) we apply some additional rules to avoid improper tables and finally collect 13,000 high-quality tables. question/answer collection we release 13k hits (human intelligence task) on the amazon mechanical turk platform, where each hit presents the crowd-worker with one crawled wikipedia table along with its hyperlinked passages. we require the worker to write six questions as well as their answers. the question annotation phase is not trivial, as we specifically need questions that rely on both tabular and textual information. in order to achieve that, we exemplify abundant examples in our amazon turker interface with detailed explanations to help crowd-workers to understand the essence of the "hybrid" question. the guidelines are described as follows: table reasoning step specifically includes (i) filter our table rows based on equal/greater/less, e.g. "for the xxxi olympic event", (ii)) superlative operation over a column, e.g. "the earliest olympic event", (iii) hop between two cells, e.g. "which event ... participate in ...", (iv) extract information from table, e.g. "in which year did the player ... ". based on the above criteria, we hire five csmajored graduate students as our "human expert" to decide the acceptance of a hit. the average completion time for one hit is 12 minutes, and payment is $2.3 u.s. dollars/hit. annotation de-biasing as has been suggested in previous papers @xcite @xcite , the existing benchmarks on multi-hop reasoning question answering have annotation biases, which makes designing multi-hop models unnecessary. we discuss different biases and our prevention as follows: in order to deal with such a bias, we use an algorithm to match the answer with linked passages to find their span and reject the hits, which have all the answers centered around the first few sentences. statistics after we harvest the human annotations from 13k hits (78k questions), we trace back the answers to its source (table or passage). then we apply several rules to further filter out lowquality annotations: 1) the answer cannot be found from either table or passage, 2) the answer is longer than 20 words, 3) using a tf-idf retriever can directly find the answer passage with high similarity without relying on tabular information. we filter the question-answer pairs based on the previous criteria and release the filtered version. as our goal is to solve multi-hop hybrid questions requiring a deeper understanding of heterogeneous information. we follow hotpotqa @xcite to construct a more challenging dev/test split in our benchmark. specifically, we use some statistical features like the "size of the table", "similarity between answer passage and question", "whether question directly mentions the field", etc. to roughly classify the question into two difficulty levels: simple (65%) and hard (35%). we construct our dev and test set by sampling half-half from the two categories. we match the answer span against all the cells and passages in the table and divide the answer source into three categories: ## data analysis in this section, we specifically analyze the different aspects of the dataset to provide the overall characteristics of the new dataset. ## model in this section, we propose three models we use to perform question answering on hybridqa. ## conclusion we present hybridqa, which is collected as the first hybrid question answering dataset over both tabular and textual data. we release the data to facilitate the current research on using heterogeneous information to answer real-world questions. we design hybrider as a strong baseline and offer interesting insights about the model. we believe hybridqa is an interesting yet challenging nextproblem for the community to solve.