Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models (2021)

Abstract: Open-domain extractive question answering works well on textual data by first retrieving candidate texts and then extracting the answer from those candidates. However, some questions cannot be answered by text alone but require information stored in tables. In this paper, we present an approach for retrieving both texts and tables relevant to a question by jointly encoding texts, tables and questions into a single vector space. To this end, we create a new multi-modal dataset based on text and table datasets from related work and compare the retrieval performance of different encoding schemata. We find that dense vector embeddings of transformer models outperform sparse embeddings on four out of six evaluation datasets. Comparing different dense embedding models, tri-encoders with one encoder for each question, text and table increase retrieval performance compared to bi-encoders with one encoder for the question and one for both text and tables. We release the newly created multi-modal dataset to the community so that it can be used for training and evaluation.

## introduction finding the answer to a factual question in a large collection of documents is a tedious task that many people from a broad range of domains have to complete on a daily basis. in order to address this task with machine learning approaches, it has been formalized as open-domain extractive questionanswering (qa). more specifically, given a natural language question and a database of text documents as a knowledge base, open-domain extractive qa aims to extract a substring that answers the given question out of one of the documents. the standard approach for this task is a pipeline architecture consisting of two components: a retriever selecting a small subset of relevant documents from the database and a reader extracting granular answers out of each of these retrieved documents @xcite . in this paper, we focus on the retriever and present a transformerbased tri-encoder model as an implementation of this component. retrievers can also be implemented with bag-of-words retrieval methods, such as tf-idf or bm25, that transform all documents and the question to sparse vector representations. however, as these methods rely on a lexical overlap of the question and the documents, they fail to capture synonymy and other semantic relationships. this limitation motivates the use of dense vector representations and we compare dense retrieval models to a bm25 baseline in our experiments. a survey on term-based, early semantic, and neural semantic models for document retrieval as a first step before document reranking and down-stream tasks, such as question answering, has been published by @xcite . to date, most of the research centered around question-answering focuses on using free-form text as the single source for answering questions. however, valuable information can obviously be found in other modalities as well. for instance, a lot of information is stored in semi-structured tables; according to @xcite , more than 14.1 billion tables can be found on the world wide web. given that a user typically does not know in advance in which modality the answer to their question resides, a qa system capable of jointly handling text and tables is needed. one major challenge in building such a system is to represent texts and tables in a way that allows capturing semantic similarity and retrieving texts and tables that are semantically related to a given question. contributions the contributions of this paper can be summarized as follows: (1) we present biencoder and tri-encoder models that are capable of joint retrieval of tables and texts; (2) we create and release a multi-modal dataset for training and evaluating models on this task; 1 (3) we compare sparse retrieval models with dense retrieval models using bi-encoders and tri-encoders on our new multi-modal dataset and on five uni-modal datasets from related work. outline the remainder of this paper is structured as follows: section 2 summarizes existing methods for uni-modal retrieval of texts on the one hand and of tables on the other hand. further, it discusses the only two, recently published approaches for joint retrieval of tables and texts. section 3 briefly describes the existing uni-modal datasets and our new multi-modal dataset, which we use to train the retrieval models presented in section 4 and to evaluate these models in section 5. section 6 concludes the paper and gives directions for future work. ## related work to the best of our knowledge the only existing work that addresses joint retrieval of tables and texts is by @xcite , @xcite and @xcite . to address the challenge of limited context available for tables, @xcite fuse a table segment and text passages into one block if they mention the same named entities. each block is represented with a single dense embedding so that a table and relevant passages are jointly retrieved as a group if the embedding is similar to that of the question. this grouping makes sense because @xcite address the task of multihop qa, where information needs to be aggregated from multiple tables and texts to answer a question. in contrast to that, we address the slightly different task of single-hop qa, where either only one table or one text is needed to answer a question. therefore, our approach represents tables and texts with separate embeddings, which are in the same embedding space. the advantage here is that the model learns to estimate relevance on the more finegrained level of individual tables or texts and can decide whether a particular table or text is more relevant to the question. @xcite also address the task of multi-hop qa on texts and tables but retrieve them individually. they make use of the sparse retrieval method bm25 and a transformer-based reranker to generate a set of candidate texts and tables. with two separate bm25 indices for texts and tables, they retrieve a set of documents for each modality. in a second step, they apply a joint bert-based reranker to reduce the size of candidate texts and tables. @xcite create a multimodalqa dataset containing questions that require joint reasoning over tables, texts, and images. otherwise closely related to our approach are tabert by @xcite and tapas by @xcite , who use tables and texts for language model pre-training but do not consider joint retrieval of tables and texts. turl focuses on representation learning for tables but also does not consider the retrieval task @xcite . due to this limited amount of prior research, we discuss related work on the separate tasks of uni-modal text retrieval and table retrieval in the following. ## datasets the training and evaluation of models examined in this paper on the task of multi-modal retrieval makes use of five datasets from related work: nq @xcite , nq-tables @xcite , wikisql @xcite , a subset of wikisql, which we call wikisql ctx-independent , and ott-qa @xcite . this section briefly explains the characteristics of these datasets and of our newly created multi-modal retrieval dataset comprising tables and texts, which we call multimodalretrieval. nq natural questions (nq) @xcite is an open-domain qa dataset on wikipedia articles. it consists of questions, their answers and the text passages the answers reside in. the questions are natural because they consist of real user queries issued to the google search engine instead of questions posed by annotators after reading a text passage, which was done to create other popular qa datasets, such as the stanford question answering dataset (squad) @xcite . having natural questions does not only ensure that the questions correspond to information needs by real users but also makes the dataset open-domain, i.e., the questions are context-independent and can be understood without their accompanying text passage that contains the answer. for the purpose of retrieval, we utilize @xcite preprocessed variant of nq. nq-tables the answers to most of the questions inside the nq dataset can be found in plain unstructured text. however, a subset of the questions are answered by tables. as a consequence, @xcite construct nq-tables, a tablespecific qa dataset based on nq. to achieve this objective, they extract all the tables and all the questions whose answers reside inside a table. they come up with a dataset consisting of 9,594 questions in the training set, 1,068 questions in the development set, 966 questions in the test set and 169,898 tables in total. given that this dataset is a subset of nq, the questions share the characteristic of being context-independent. wikisql wikisql @xcite is a closed-domain qa dataset consisting of 24,241 ta-bles and 80,654 natural language questions together with their corresponding sql query. to build this dataset, @xcite generate a number of random sql queries for each table. these sql queries are then transformed into crude questions using templates. finally, amazon mechanical turk crowd workers paraphrase these crude questions into natural language questions, which are checked by two additional crowd workers. wikisql ctx-independent since wikisql is a closed-domain qa dataset, the majority of its questions is context-dependent, i.e., they do not provide enough context to be answered without their accompanying table and are therefore not suitable for training or evaluation on the retrieval task. one example for such insufficient questions inside the dataset is: "who is the player that wears number 42?", which cannot be answered without additional context given in the table, such as the name of a sports team and a year. as a consequence, to make use of wikisql, all questions that do not provide enough context for retrieval need to be filtered out. for automating this filtering, we labeled a subset of wikisql's questions with regard to whether they are either context-independent or under-specified resulting in 4,553 labels as training set and 612 labels as test set. these labels are then used to train a classifier that predicts whether a question provides enough context. we fine-tune a roberta-base @xcite language model achieving an accuracy of 0.8134 and a macro-averaged f1-score of 0.7748 on the test set. next, we apply this classifier to the whole wikisql dataset to filter out all the questions that are predicted as under-specified. ott-qa ott-qa @xcite is an open-domain multi-hop qa dataset of texts and tables from wikipedia built upon hybridqa @xcite , a closed-domain multi-hop qa dataset. multi-hop qa refers to the fact that most questions inside the dataset require a combination of different texts and/or tables instead of a single document in order to be answered. to generate an open-domain version of hybridqa, @xcite let crowd workers decontextualize all the questions. furthermore, they add additional question-answer pairs on newly crawled tables. since the published annotations contain only the gold tables but not the gold texts, we use ott-qa only for generating table retrieval training samples and evaluating the uni-modal retrieval of tables. ## approach our two approaches for the joint retrieval of tables and texts comprise bi-encoder and tri-encoder models based on uni-modal dense retrieval methods by @xcite and @xcite . in our first approach, the bi-encoder uses one language model to encode the questions and a second model to encode both the tables and the text passages. our second approach adds a third encoder, such that there is one separate encoder for questions, text passages and tables. in contrast to the bi-encoder where tables and texts are encoded by the same model, the tri-encoder routes tables to the table encoder model and text passages to the text encoder model. we train three different bi-encoders and three different tri-encoders, which differ in the underlying language models as specified in the first tri-encoder model uses bert-small instances for all of its three encoders. also for the triencoder approach, we analyze the impact of using tapas for at least one of the encoders. therefore, ## experiments we compare the presented dense retrieval models to the sparse retrieval method bm25 and evaluate them based on recall@k with @xmath0, 20, 100}. the search space to evaluate the models needs to consist of both texts and tables. for this purpose, 500,000 text passages are randomly sampled from @xcite 's preprocessed wikipedia passages making sure that the gold passages are among these passages. furthermore, besides the text passages, all the tables from wiki-sql, ott-qa, and nq-tables are used, resulting in 656,166 tables and therefore approximately 1.2 million documents in total. the models are evaluated on a random sample of 1,000 questions of each dataset's test split as listed in following @xcite , a document retrieved for a question originating from nq or nq-tables is considered a correct match if the document contains the answer string of the granular answer. given that the derivation of the granular answer for questions originating from wikisql and ott-qa might need further aggregation, such as summation or counting, and, therefore, the answer string does not need to be present in a relevant document, a retrieved document is only considered a correct match if it is the gold annotated table. this evaluation procedure might have the effect of incorrectly judging non-gold tables that contain the answer to a query as irrelevant. however, since we apply the same evaluation procedure for all models, the numbers should be comparable. the evaluation shows that bm25 outperforms all dense methods on both the full wikisql dataset and wikisql's context-independent questions. it outperforms the best dense retrieval model on this dataset, the tri-encoder consisting of three bert models, by 21.9 percentage points on all wikisql questions and 30.4 percentage points on contextindependent wikisql questions with regard to to better understand how the lexical overlap of questions and accompanying tables influences bm25's performance, we split the test sets of wiki-sql and wikisql ctx-independent into subsets with different ranges of lexical overlap. as can be observed in figure 1 , the recall of both bm25 and dense retrieval highly correlates with lexical overlap. furthermore, while bm25 outperforms dense retrieval for questions with high lexical overlap, it is the other way round for questions with low lexical overlap. on the sample of the nq-tables test set, all dense retrieval models outperform bm25, except for the tri-encoder that consists of bert instances as question and text passage encoder and a tapas instance as table encoder. the best performing model on this dataset is the tri-encoder consisting of three bert encoders. this model outperforms the sparse retrieval method bm25 by 29.8 percentage points with regard to recall@10. for the sampled questions of the ott-qa development set, four out of the six dense retrieval models outperform bm25. the best performing model is again the tri-encoder model that is composed of three bert-small encoders. this model outperforms bm25 by 33.6 percentage points with regard to recall@10. bm25 outperforms the biencoder consisting of a bert model as question encoder and a tapas model as text and table encoder as well as the tri-encoder consisting of two bert models serving as question encoder and text encoder, respectively, and a tapas model serving as table encoder. when it comes to the performance on the text modality, i.e., questions deriving from nq whose gold-label answer resides in a text passage, four out of the six dense retrieval models outperform bm25. for this case, the best performing model is not a tri-encoder but the bi-encoder comprising two bert models. this model outperforms bm25 by 16.8 percentage points with regard to recall@10. however, this bi-encoder exceeds the tri-encoder consisting of three bert encoders only slightly by one percentage point. the sparse retrieval method bm25 beats the bi-encoder consisting of a bert model as question encoder and a tapas model as text and table encoder as well as the tri-encoder consisting of two tapas models serving as question encoder and table encoder, respectively, and a bert model serving as text encoder. in summary, the best performance on the wiki-sql test set is achieved by the sparse retrieval method bm25. the tri-encoder consisting of three bert encoders shows the best performance on the remaining two tabular datasets, ott-qa and nq-tables. on the nq dataset, i.e., questions whose answers reside in the textual modality, the bi-encoder consisting of two bert encoders performs best but is almost on par with the tri-encoder consisting of three bert models. we can conclude that, under the limited experimental conditions, in particular, on the datasets used in our study, models involving tapas as question, text, and/or table encoder perform worse than models that rely only on bert language models. @xcite show that to be able to use tapas for the retrieval of tables, tapas needs to be additionally pre-trained on the table retrieval task. these pre-trained table retrieval models, which are used in the bi-encoders and tri-encoders that involve one or more tapas instances as encoder, are, however, pre-trained solely on the task of table retrieval and not text retrieval. given the fact that the plain tapas model cannot be adapted to retrieval from scratch but needs this special pre-training, it might be the case that, to use tapas efficiently for the retrieval of both texts and tables, it needs to be pre-trained in a multi-modal setting on the retrieval of both texts and tables. furthermore, batch size is significant for training retrieval models, as higher batch sizes make the training harder by adding more in-batch negatives. while the training of a bi-encoder does not allow a batch size higher than 38 and the training of a tri-encoder does not allow a batch size higher than 28 on a tesla v100 gpu with 16 gb of memory, @xcite make use of a batch size of 256 for training their tapas-based table retrieval models. accordingly, it might be the case that tapas is more unstable to train and requires, therefore, larger batch sizes. ## conclusion and future work this paper presented a transformer-based approach using bi-encoder and tri-encoder models for multimodal retrieval of tables and texts. with experi-ments on five datasets from related work and one newly created dataset, we show that the presented dense retrieval models outperform the sparse retrieval model bm25 if there is a low lexical overlap of questions and relevant tables and texts. more specifically, the tri-encoder architecture performs better on ott-qa and nq-tables, which represent the tabular modality, while the bi-encoder architecture performs slightly better on the nq dataset representing the textual modality. we observe that the best retrieval models are those that rely only on bert models as encoder and do not make use of tapas. from an application point of view, future work could integrate the presented retrieval models as one component in a multi-modal open-domain qa pipeline and evaluate it on a real-world use case. such a pipeline would facilitate information access immensely by combining valuable information from both sources rather than relying only on either texts or tables. another promising path for future work is to extend our approach to more modalities with transformer-based models for images, videos, or speech. these models could serve as encoders for documents of different modalities to jointly train an n-encoder architecture, where one encoder is tailored to the queries and the remaining n-1 encoders are tailored to each of the modalities that the user would like to search on. last but not least, the research community would surely benefit from the creation of more multi-modal datasets to improve training and evaluation of multi-modal retrieval models and we are only making a first step in this direction with creating and releasing a dataset of tables and texts.