Mixed-modality Representation Learning and Pre-training for Joint Table-and-Text Retrieval in O pen QA (2022)

Abstract: Retrieving evidences from tabular and textual resources is essential for open-domain question answering (OpenQA), which provides more comprehensive information. However, training an effective dense table-text retriever is difficult due to the challenges of table-text discrepancy and data sparsity problem. To address the above challenges, we introduce an optimized OpenQA Table-Text Retriever (OTTeR) to jointly retrieve tabular and textual evidences. Firstly, we propose to enhance mixed-modality representation learning via two mechanisms: modality-enhanced representation and mixed-modality negative sampling strategy. Secondly, to alleviate data sparsity problem and enhance the general retrieval ability, we conduct retrieval-centric mixed-modality synthetic pre-training. Experimental results demonstrate that OTTeR substantially improves the performance of table-and-text retrieval on the OTT-QA dataset. Comprehensive analyses examine the effectiveness of all the proposed mechanisms. Besides, equipped with OTTeR, our OpenQA system achieves the state-of-the-art result on the downstream QA task, with 10.1% absolute improvement in terms of the exact match over the previous best system.

## introduction open-domain question answering @xcite @xcite aims to answer questions with evidence retrieved from a largescale corpus. the prevailing solution follows a twostage framework @xcite , where a retriever first retrieves relevant evidences and then a reader extracts answers from the evidences. existing openqa systems @xcite @xcite have demonstrated ## methodology we present otter, an openqa table-text retriever. we first introduce the basic dual-encoder architecture for dense retrieval ( § 3.1). we then describe three mechanisms to mitigate the tabletext discrepancy and data sparsity problems, i.e., modality-enhanced representation ( § 3.2), mixedmodality hard negative sampling ( § 3.3), and mixed-modality synthetic pre-training ( § 3.4). ## experiment settings in this section, we describe the experiment settings on the task of open-domain question answering over tables and text, and report the performance of our system on the table-and-text retrieval, and downstream question answering. ## experiments: table-and-text retrieval in this section, we evaluate the retrieval performance of our openqa table-text retriever (otter). we first compare otter with previous retrieval approaches on ott-qa. then we conduct extensive experiments to examine the effectiveness of the three proposed mechanisms. ## experiments: question answering in this section, we experiment to show how ot-ter affects the downstream qa performance. ## related works in openqa @xcite @xcite , the retriever is an essential component to identify relevant evi-dences for answer extraction. in contrast to sparse information retrieval methods @xcite @xcite , recent openqa systems tend to adopt dense retrieval approaches utilizing dense representations learned by pre-trained language models @xcite @xcite . these methods are powerful in capturing contextual semantics. the prevailing openqa datasets mainly take the unstructured passage as evidence, including natural questions @xcite , triviaqa @xcite , webquestions @xcite , curatedtrec @xcite and squad @xcite . there are also recent works studying openqa in the tabular domain @xcite @xcite . @xcite consider a more challenging setting that takes both tabular corpus and textual corpus as the knowledge sources, which is also the setting in this paper. our approach differs from existing methods mainly in two aspects: targeted evidence source and mixed-modality learning mechanisms. first of all, we retrieve mixed-modality evidence from both tabular and textual corpus, which is different from text-based retrievers @xcite @xcite and table-based retrievers @xcite @xcite . secondly, our proposed three mixed-modality learning mechanisms also differ from existing methods. as for mixed-modality representation, previous work @xcite ) mainly uses the single representation of the special token for ranking. our method incorporates single modal representation to enrich the mixed modal representation. as for mixed-modality negative sampling, instead of finding an entire negative evidence with either sparse or dense methods @xcite @xcite @xcite @xcite , we construct more challenging hard negative by only replacing partial single-modality information at once. as for mixed-modality synthetic pre-training, our pre-training strategy is different in the pre-training task, knowledge source and the method of synthesizing pseudo question. there are also works investigating joint pre-training over tables and text @xcite @xcite . however, these methods mainly take the table metadata as the source of text and do not consider the retrieval task. instead, we use linked passages as a more reliable knowledge source, and target on retrievalbased pre-training. there are some attempts on incorporating pre-training task to improve retrieval performance @xcite @xcite , which target on textual-domain retrieval or using templatebased method for query construction. differently, our approach focuses on a more challenging setting that retrieves evidence from tabular and textual corpus and adopts a generation-based query synthetic method. besides, @xcite explore to generate multi-hop questions for tables and text, but they focus on an unsupervised manner. ## conclusion in this paper, we propose an optimized dense retriever called otter, to retrieve joint table-text evidences for openqa. otter involves three novel mechanisms to address table-text discrepancy and data sparsity challenges, i.e., modality-enhanced representations, mixed-modality hard negative sampling, and mixed-modality synthetic pre-training. we experiment on ott-qa dataset and evaluate on two subtasks, including retrieval and qa. results show that otter significantly outperforms other retrieval methods by a large margin, which further leads to a substantial absolute performance gain of 10.1% em on the downstream qa. extensive experiments illustrate the effectiveness of all three mechanisms in improving retrieval and qa performance. further analyses also show the ability of otter in retrieving more relevant evidences from heterogeneous knowledge resources.