Neural Ranking with Weak Supervision for Open-Domain Question Answering : A Survey (2023)

Abstract: Neural ranking (NR) has become a key component for open-domain question-answering in order to access external knowledge. However, training a good NR model requires substantial amounts of relevance annotations, which is very costly to scale. To address this, a growing body of research works have been proposed to reduce the annotation cost by training the NR model with weak supervision (WS) instead. These works differ in what resources they require and employ a diverse set of WS signals to train the model. Understanding such differences is crucial for choosing the right WS technique. To facilitate this understanding, we provide a structured overview of standard WS signals used for training a NR model. Based on their required resources, we divide them into three main categories: (1) only documents are needed; (2) documents and questions are needed; and (3) documents and question-answer pairs are needed. For every WS signal, we review its general idea and choices. Promising directions are outlined for future research.

## introduction open-domain question answering (odqa) aims to provide precise answers in response to the user's questions by drawing on a large collection of documents @xcite . the majority of modern odqa models follow the retrieve(-rerank)read architecture: 1) given a question, a set of relevant documents are selected from a large document collection, and 2) the reader model produces an answer given this selected set and the question @xcite @xcite . compared with parametric models without access to external knowledge, this architecture can better adapt to updated knowledge, offer easier interpretation and reduce hallucination @xcite @xcite . conventional methods use sparse retrievers (srs) such as tf-idf and bm-25 in the first stage resource weak-supervision signal documents ( §3) self contrastive learning ( §3.1) question generation ( §3.2) documents +questions ( §4) sparse retriever ( §4) pre-trained language model ( §4) supervised teacher model ( §4) documents +qa pairs ( §5) answer as document ( §5.1) answer-document mapping ( §5.2) latent-variable model ( §5.3) to match questions and documents via lexical overlap @xcite , a process that may overlook semantically relevant documents with low lexical overlap with the question. neural ranking (nr) models resolve this issue by encoding the questions and documents into dense vectors so that synonyms and paraphrases can be mapped to similar vectors through task-specific fine-tuning @xcite . however, training good nr models requires substantial amount of relevance annotations to perform competitively and nr models have been found to generalize poorly across domains @xcite . in practice, collecting question-document relevance annotations is time-consuming. for a given question, an extensive annotation effort may be required to find the relevant documents. repeating this annotation for every language and domain is not feasible @xcite . to reduce annotation costs, many techniques have been proposed to train nr models with weak supervision (ws) signals instead. this survey aims to provide a clear taxonomy to characterize these ws signals based on their required resources. there are three common resources that can be leveraged: (1) document set, which is a bare minimum for building a odqa system; (2) questions without ground-truth relevance or answer annotations; while there have been surveys that describe general neural information retrieval (ir) approaches @xcite @xcite @xcite , we focus specifically on the low-resource scenarios, which makes our contribution unique in this respect. the closest to our work is the beir benchmark for zero-shot cross-domain evaluation of ir models @xcite and its multiple related studies @xcite @xcite . nonetheless, these studies test specific algorithms but do not provide a holistic overview of how they are related. our survey can be useful in that: (1) future ws research can use it as a reference book to compare with similar techniques. (2) it can serve as a practical guide for choosing the best ws sginals to train nr models given different availability levels of resources. (3) as the retrieve(-rerank)-read paradigm is generic and has been increasingly popular across nlp tasks like machine translation @xcite and intent detection @xcite , it can have broader impact in many other applications. therefore, although this survey illustrates with the use case of odqa, the introduced techniques are intended to go beyond specific applications. in the following sections we first lay out the necessary background knowledge ( §2), then explain popular ws signals when different resources are available in sections 3 to 5. in conclusion, we highlight promising directions for future work ( §6). ## background neural ranking (nr) for odqa let q, d and a denote the question, document and answer set. given a question @xmath0, the nr model assigns a relevance score r(q, d) to each @xmath1. afterwards, a reader will estimate the score g(a|q, d topk ) to predict the final answer @xmath2. the nr model can be implemented using various architecture with increasing model complexity. for computational efficiency, normally a bi-encoder architecture @xcite is first applied to pre-select top candidates from the whole document set, then a more complex cross-interaction model is applied to provide more accurate relevance scores only for the preselected candidates @xcite . the training objective for the nr model r can be formalized as: neural ranking with weak supervision in the standard supervised setting we need relevance annotations for (q, d) → {+, -} to train r with eq 1. obtaining high-quality relevance annotations requires tremendous human labor and is expensive to scale to multiple domains @xcite . weak supervision (ws) is a widely-used approach to reduce such cost by leveraging supervision signals from e,g., heuristic rules, knowledge bases or external models @xcite . ws signals are cheap to obtain but might contain significant noise which will affect the nr performance. therefore, understanding their working mechanisms and pros and cons are important to obtain a good nr model. we group ws signals into 3 classes by the resources that they need: (1) documents: only document collection d is needed; ## resource: documents this section discusses two main techniques to produce ws signals requiring only the document set: method pseudo question perturbation d with added perturbation summary (pseudo) summary of d proximity nearby text of d cooccurrence text sharing cooccurred spans with d hyperlink text with a hyperlink to/from d ## resource: documents + questions this section includes ws signals that require additional access to a question set q. in practice, annotating question-document relations usually requires domain experts to read long documents and careful sampling strategies to ensure enough positive samples, while unlabeled questions are much easier to obtain either through real user-generated content or simulated traffic. therefore, it is common to have a predominance of unlabeled questions. the crucial point is to establish the missing relevance labels. suppose a ws method can provide the missing label ws(q, d) for a question-document pair (q, d), then we can use it to supervise the nr model by: there are three popular types models that can provide such ws signal here: (1) sparse retriever, (2) pre-trained language model and (3) supervised teacher model. ## resource: documents + qa pairs many domains have large numbers of already answered questions from customer services, technical support or web forums @xcite . these qa pairs can provide richer information than only unlabeled questions. however, most answers are based on personal knowledge, derived from experience, and do not include a reference to any external document. this prevents their direct use as training data for the nr model. this section introduces three standard methods that exploit qa pairs to provide ws signals despite this difficulty: (1) answer as document, (2) answer-document mapping and (3) latent-variable models. ## conclusions we review standard ws signals used for training nr models in odqa and provide a structured way of classifying them according to the required resource. for ws signal, we discuss different options and summarize the pros and cons. as a final wrap-up, we list promising directions that we believe worth exploring further: (1) how to select the most suitable technique for a given scenario? despite the wide range of applicable techniques, it is non-trivial to decide how to select the best one except for an empirical experimentation. (2) to which extent are these techniques complementary? existing work compares performance only between similar types of methods but not across the whole range of techniques and resources available. this makes it hard to decide whether different approaches could potentially complement each other and how they should be combined effectively. (3) do methods work across languages? the vast majority of current research is conducted on english datasets. even though all described methods in this survey have no explicit restrictions on languages they can be applied to, it is likely that their performance will vary across languages, especially for the methods relying on handcrafted heuristics.