Multi-Row, Multi-Span Distant Supervision For T able+ T ext Question Answering (2023)

Abstract: Question answering (QA) over tables and linked text, also called TextTableQA, has witnessed significant research in recent years, as tables are often found embedded in documents along with related text. HybridQA and OTT-QA are the two best-known TextTableQA datasets, with questions that are best answered by combining information from both table cells and linked text passages. A common challenge in both datasets, and TextTableQA in general, is that the training instances include just the question and answer, where the gold answer may match not only multiple table cells across table rows but also multiple text spans within the scope of a table row and its associated text. This leads to a noisy multi-instance training regime. We present MITQA, a transformer-based TextTableQA system that is explicitly designed to cope with distant supervision along both these axes, through a multi-instance loss objective, together with careful curriculum design. Our experiments show that the proposed multi-instance distant supervision approach helps MITQA get sate-of-the-art results beating the existing baselines for both HybridQA and OTT-QA, putting MITQA at the top of HybridQA leaderboard with best EM and F1 scores on a held out test set.

## introduction transformer-based question answering (qa) methods have evolved rapidly in recent years to handle open-domain, multi-hop reasoning over retrieved context paragraphs. many existing qa datasets and benchmarks measure performance over homogeneous data sources, such as text @xcite @xcite and more recently tables @xcite @xcite @xcite . even though * equal contribution † work done while at ibm research real-world documents often contain tables embedded in free form text, qa over such a hybrid corpus, i.e., a combination of tables and text -a.k.a. text-tableqa -remains relatively unexplored. as illustrated in figure 1 , even a relatively simple table from wikipedia often references several entities, definitions or descriptions from the table elements. a question may be best answered by matching some parts of it to table elements and other parts to linked text spans. existing transformer-based qa solutions need significant modifications to score such heterogeneous corpus units. a key challenge is to reduce the cognitive burden of supervision to (question, answer) pairs, without humans having to identify the specific table cell or text span where the answer was mentioned. in texttableqa, such 'distant' supervision is particularly challenging because it occurs along two distinct axes: (1) there could be multiple rows and associated passages that mention the answer string; and (2) even for a specific table row with linked passages, the same candidate answer may occur in multiple text spans. many of them may be spurious and detrimental to system training. in response, we present mitqa -a text-tableqa system specifically engineered to address the above challenges. mitqa defines each table row, together with linked passages, as the fundamental retrieval unit. to adapt to memory-hungry transformer networks, constrained by the number of input tokens they can efficiently process, mitqa uses a novel query-informed passage filter to prepare a contextual representation of each retrieval unit. mitqa then uses an early interaction (cross attention) transformer network to score retrieval units. while training mitqa, its most salient features are multi-instance loss functions and data engineering curricula to tackle distant supervision, along both the multi-row and multi-span axes. many of the above challenges are not faced by homogeneous text-only or table979,477 people in its 2018 metropolitan statistical area and 22,679,948 residents in its ... the bank of america tower is a 1,200 ft (365.8 m) skyscraper in the midtown area of manhattan in new york city. it is located at 1111 avenue of the americas (sixth avenue) between 42nd and ... comcat technology center aon center bank of america new york city chicago philadelphia question: what year was the tallest building in philadelphia open to the public? answer : 2018 figure 1: an instance of question answering over hybrid context of table and text (from hybridqa). gold answer in correct context is highlighted in blue and gold answer appearing in irrelevant context is highlighted in red. the context used to arrive at the answer in the correct passage is shaded in yellow. the relevant row to be retrieved is shaded green and irrelevant rows are shaded red. only qa systems. we report results on extensive experiments on two recent texttableqa challenge data sets, hybridqa and ott-qa, where our system outperforms baselines and is currently at the top of hybridqa 1 leaderboard. source code is available at https://github.com/ primeqa/primeqa . ## related work tableqa has gained much popularity in recent years, resulting in diverse approaches including semantic parsing-based @xcite @xcite @xcite and more recently bert-based @xcite systems for table encoding by, inter alia, herzig et al. (2020); yin et al. (2020); glass et al. (2021a). a more realistic application scenario is "text-tableqa" where tables are often embedded in documents and a natural language query needs to combine information from a table as well as its correlated textual context to find an answer. hybridqa @xcite pioneered a text-tableqa benchmark, with wikipedia tables linked to relevant free-form text passages (e.g., wikipedia entity definition pages). they curated questions which need information from both tables and text to answer correctly. they also proposed hybrider as the first system in texttableqa with an f1 score of 50%, leaving much scope for improvement. the ott-qa @xcite benchmark extended hybridqa to an open domain setting where a system needs to retrieve a relevant set of tables and passages first before trying to answer a question. moreover, the links from table and passage are not provided explicitly. to our knowledge, no existing texttableqa system @xcite @xcite attempts to handle the challenge of multiple candidate instances arising from distant supervision during system training, owing to multiple matching table rows and multiple matching spans within a row and its linked text. our experiments with hybridqa and ott-qa show that superior handling of multi-instance matches by mitqa improves qa accuracy. ## mitqa system architecture in this section we first describe the modules shown in figure 2 , that are shared for closed-domain (table and linked text provided, as in hybridqa) and open-domain (ott-qa) applications. after that we describe tableretriever and rowpassagelinker that are needed for open-domain scenarios. ## conclusion texttableqa requires reasoning over table cells and linked passage contents. weak supervision poses a challenge: the target answer might be mentioned in multiple row cells and/or as multiple spans in linked passages. we design a novel qa pipeline that uses multiple row and multiple answer based novel training strategies to identify correct rows first and then use the row cells for relevant passage lookup. we propose efficient strategies for filtering linked passages to retain the most relevant ones for the question, and a novel re-ranker to rank the answers obtained from different rows and their respective linked passages. our system, mitqa, performs better than recent systems on hybridqa and ott-qa benchmarks, with large improvements in f1 scores. we have also tried different combinations of our proposed strategies to substantiate the benefit from each of them separately. in future, we would like to explore the following directions: (1) answering complex numerical questions over hybrid context of table and text, (2) handling more complex table with structural hierarchies, and (3) enhancing mitqa to provide interpretable explanations for answers.