Ad-hoc Document Retrieval using Weak-Supervision with BERT and GPT 2 (2020)

Abstract: We describe a weakly-supervised method for training deep learning models for the task of ad-hoc document retrieval. Our method is based on generative and discriminative models that are trained using weak-supervision just from the documents in the corpus. We present an end-to-end retrieval system that starts with traditional information retrieval methods, followed by two deep learning re-rankers. We evaluate our method on three different datasets: a COVID-19 related scientific literature dataset and two news datasets. We show that our method outperforms state-of-the-art methods; this without the need for the expensive process of manually labeling data.

## introduction the ad-hoc retrieval task has been extensively studied by the information retrieval (ir) community. traditional ir models evaluate ad-hoc queries against documents mainly on a syntactic (exact) word-matching basis @xcite . recent years advances in deep learning (dl) methods have lead to further improvement in ir tasks, and among others, in ad-hoc document retrieval @xcite . dl methods add a semantic dimension to ir methods. however, such methods usually require large amounts of labeled data for model training. in this work, we describe a novel weaklysupervised method for training dl methods for ad-hoc document retrieval. motivated by the recent work of @xcite on frequently asked questions (faq) retrieval, we assume that documents have at least three fields, namely title, abstract and content. such documents are actually quite common nowadays in the scientific and news domains. our main hypothesis is that: titles and ab- * * work done while affiliated with ibm. whenever a document is missing a title, we consider its first sentence as its augmented title. in a similar way, whenever a document is missing an abstract, we consider the first 512 words of its content as the abstract. the three fields are used for retrieving candidate documents. inspired by @xcite , the title and abstract fields are further used as a weaksupervision data source for training two independent bert @xcite models, that are then used to re-rank those candidates documents. the first model matches user queries to documents' abstracts. here we use the title-to-abstract associations to fine-tune a bert model to semantically match queries to abstracts. the second model matches user queries to titles. here our assumption is that by generating title paraphrases, we can train a model to match user queries to titles. to this end, we use gpt2 @xcite to generate title paraphrases, which are then utilized for fine-tuning the second bert model. while our work is closely related to @xcite , with the lack of human-curated questions (such as in faqs), we still need to resort to title paraphrases as (noisy) pseudo-questions and transfer @xcite 's method to the more general task of ad-hoc document retrieval. moreover, compared to faqs that are relatively short, the current task deals with documents that can be quite long. thus, in current paper we use three fields (title, abstract, content) and present a strong ir base line instead of only two fields and a simple ir baseline used in @xcite as a proof of concept, we evaluate our method on three benchmarks: trec-covid -a scientific literature dataset on covid-19 topics; and trec's newswire corpora: associated press (ap) and wall street journal (wsj). by combining the two weakly-supervised bert models with an existing strong ir baseline, we demonstrate that the former can help to elevate the performance of the latter. our approach further outperforms state-ofthe-art methods on these benchmarks. ## related work with the lack of training data, several weaklysupervised alternatives have been explored so far for the task at hand. @xcite and @xcite have utilized rankings produced by bm25 model as training samples. @xcite have used pseudo query-document pairs that already exhibit relevance (e.g., newswire headline-content pairs). @xcite have utilized wikipedia's internal linkage to define automated queried topics. @xcite have used anchor texts and their linked web pages as query-document pairs. our work is different from all those works as we train a model to generate title paraphrases that are used to enable query-to-title (question) matching and not only query-to-abstract (answer) matching. @xcite have proposed a zero-shot retrieval approach using synthetic query generation by training a generative model on a different community qa data. our work differs from @xcite in three main aspects. first, @xcite focuses on qa, where answers are very short, while we generate title paraphrases from full abstracts. second, we train a model to generate title paraphrases which are used to enable not only query-toabstract (answer) matching, but also query-to-title (question) matching. third, @xcite filters the input qa pairs that are used to train the generative model by taking only pairs that were voted by at least one-user on those community qa (cqa) sites. we do not have such voting so we use a smart filtering on the output data (namely on the generated title-paraphrases) as described in section 3.3. the work in @xcite suggests an efficient neural method for initial retrieval of candidates. their method uses a two-tower architecture which learns a different representation for passages and for queries. while their method can be used as an initial retrieval (instead of our ir method), the authors of @xcite still require an additional re-ranking step. thus it does not replace our two weakly-supervised bert re-ranking models. moreover, our two bert models learn a joint attention-based representation for pairs of (query, abstarct) and (query, title) while in @xcite they learn a separate representation for queries and passages. ## method inspired by @xcite , we consider the adhoc document retrieval problem as an instance of faq retrieval, where a document's title represents the question and its abstract the answer. our proposed retrieval approach allows to enhance existing state-of-the-art ad-hoc retrieval methods with weakly-supervised neural models that are completely trained from the documents collection itself without the need to supply manual relevance labels. following the common approach @xcite , these neural-models are utilized for re-ranking candidate documents retrieved by a given ir baseline. in what follows, the initial candidate documents retrieval uses pure ir similarities and relevance models (section 3.1). the re-ranking step exploits two independent weakly-supervised bert models, namely: bert-q-a (section 3.2) for matching queries to abstracts and bert-q-t (section 3.2) for matching queries to titles. the final re-ranking is obtained by combining the outcome of the baseline ir method and the two bert-based re-rankers using an unsupervised latefusion step (section 3.4). the components of our approach are described in the rest of this section. ## conclusions and future work we have cast a solution for faq retrieval to a solution for ad-hoc document retrieval, where titles and abstracts took the role of questions and answers in faqs. we have shown that, using the corpus itself, we could generate weakly-supervised title paraphrases for training a bert model that matches queries to titles. coupled with a second bert model that was trained to match queries to abstracts, we have experimentally shown on three different benchmarks that our proposed method outperformed state-of-the-art alternatives. as a future work, we plan to utilize automatic summarization for missing abstracts, instead of taking the first 512 content tokens.