Neural Retriever and Go Beyond: A Thesis Proposal (2022)

Abstract: Information Retriever (IR) aims to find the relevant documents (e.g. snippets, passages, and articles) to a given query at large scale. IR plays an important role in many tasks such as open domain question answering and dialogue systems, where external knowledge is needed. In the past, searching algorithms based on term matching have been widely used. Recently, neural-based algorithms (termed as neural retrievers) have gained more attention which can mitigate the limitations of traditional methods. Regardless of the success achieved by neural retrievers, they still face many challenges, e.g. suffering from a small amount of training data and failing to answer simple entity-centric questions. Furthermore, most of the existing neural retrievers are developed for pure-text query. This prevents them from handling multi-modality queries (i.e. the query is composed of textual description and images). This proposal has two goals. First, we introduce methods to address the abovementioned issues of neural retrievers from three angles, new model architectures, IR-oriented pretraining tasks, and generating large scale training data. Second, we identify the future research direction and propose potential corresponding solution.

## introduction the convenience and advance of internet not only speed up the spread of information and knowledge, but also the generation of new information. such phenomenon also boosts humans needs of knowledge and frequency of acquiring information, which makes information retrieval (ir) an important task in human life. ir aims to find relevant information from a large corpus to satisfy an information need. it also plays an important role in other tasks such as open domain question answering and open domain dialogue, where external knowledge are needed. not only that, ir can also assistant other systems to achieve a tough goal. by providing external knowledge, ir can help numerical reasoning systems to reach the correct answer @xcite , and ir can enrich or update the knowledge of large pretrained language models (prlms) @xcite . by filtering and selecting examples @xcite , ir can assist in-context learning (icl), a process allows large prlms do a new task instructed by prompts and few examples with few-shot tuning @xcite or without any fine-tuning @xcite . ir has a long history and the first automated information retrieval system can be traced back to the 1950s. in this work, we call information retrieval methods or systems as retrievers. traditional retrievers are mainly based on term-matching, i.e. searching for information that has an overlap with terms in the query. tf-idf and bm25 @xcite are two strong and efficient algorithms in this category. although these algorithms consider the importance and frequency of terms in query and document, they suffer from term-mismatch issues and lack of semantic understanding of the query and document @xcite . using neural models to represent the concatenation of query and passage is a promising way to achieve semantic matching @xcite . these methods are only applicable at small scale retrieval but not at large scale. recently, dual-encoder architecture retrievers based on large pretrained language models (prlms), such as bert @xcite have shown capability to do semantic matching and can be applicable at large scale @xcite @xcite . such neural retrievers (nr) involve two prlms which are used to compute the vector representation of queries and documents respectively. neural retriev- score score maxsim maxsim maxsim @xmath0. despite the success of neural retrievers, they still face many challenges. in the next section, we will present a brief overview of five types of retrievers and the efforts made toward building stronger retrievers. section 3 describes four limitations of current nrs and promising solutions. section 4 discusses three more research directions and potential solutions. we conclude the proposal in section 5. ## retrievers in general in general, the modern retrievers can be categorized in five classes (adapted from @xcite ). lexical retrievers such as bm25 are based on token-matching between two high-dimensional sparse vectors. the sparse vectors are represented based on the frequency of the terms in documents and thus does not require any annotated training data. regardless of the simplicity of the algorithms, such methods perform well on new domains @xcite . dual-encoder dense retrievers consists of two encoders where the query encoder and context encoder generate a single dense vector representation for query and context respectively. then the score can be computed by inner-dot product or cosine-similarity between the two representations @xcite @xcite . language models such as bert @xcite are preferred choices for encoders. sparse retrievers use sparse representations instead of dense representations for query and document @xcite @xcite . late-interaction retrievers different from dense retrievers who use sequence-level representations of query and document, they use token-level representations for the query and passage: a bag of multiple contextualized token embeddings @xcite . the late-interactions are aggregated with sum of the max-pooling query term and a dot-product across all passage terms. re-ranking retrievers include two stages, coarsesearch by efficient methods (e.g. bm25) and finesearch by cross-attentional re-ranking models. the re-ranking model takes input as the concatenation of the query and one candidate given by the first stage and produce a score based on the cross representation (e.g. the [cls] token), and such process is repeated for every candidate, and finally re-rank candidates based on the generated scores. without changing the architectures, different efforts have been made toward learning better representation of dense vectors and improving the efficiency in terms of training resources as well as short inference time. one way to improve the representation of dense vectors is to construct proper negative instances to train a neural retriever. inbatch negative training is a frequently used strategy to train dense retrievers, and the larger the batch size is, the better performance a dense retriever can achieve @xcite . using hard negative candidates is better than using random or simple in-batch negative samples, for example, @xcite mine negative candidates by bm25 and @xcite mine negative candidates from the entire corpus using an optimized dense retriever. @xcite selects the negative candidates from the same topic cluster, such a balanced topic aware sampling method allows the training with small batch size and still achieves high quality dense representation. colbert @xcite is proposed to improve the efficiency of the ranking model. since every token can be pre-indexed, it prevents inference time from getting representation of context. while colbert is faster than singlemodel, it is slower compared to dual-models, thus, it is not suitable for retrieval at large scale. on the other hand, @xcite shortens the inference time by using sparse representation for queries. @xcite integrates dense passage retriever and cross-attention ranker and use adversarial training to jointly both module. above methods are usually used to retrieve a document (e.g. a paragraph in wikipedia) which can potentially contain the answer to a query. some other retrievers directly retrieve the answer phrase (or entities) so that they can be directly used to answer questions without a reader @xcite @xcite . while such methods can reduce the latency, it also increases the memory to store potential phrases which will be much larger than the number of raw documents. on the other hand, @xcite use generative model to generate the entities which largely reduce the memory. ## research gaps and solutions in this section, we will describe multiple research gaps and the proposed methods introduced in @xcite . ## future work previous section describes multiple research problems for neural retrievers, while we provide some solutions, each problem can be further investigated. in the following, we identify more research directions and propose potential solutions. @xcite has shown bm25 with expended documents using generated questions is an efficient way to retrieve documents. such a method also showed good generalization across different domains @xcite . the template-based question generation proposed in this work has better domain adaptation than the previous question generation method. it is interesting to see how each module in the pipeline performs on new domain without further fine-tuning. for example, can the template selection model select good templates for passage from new domain; can the question generation model generate good questions given a new template? evaluating how our template-based question generation pipeline works when apply it to document expansion is an interesting future work. ## conclusion in this proposal, we focus on an important task: information retrieval. from word-matching retrievers to neural retrievers, many efforts have been made toward building stronger retrievers that can achieve high recall and precision. we summarize five types of modern retrievers and methods to address some existing issues. while the development in this field is exciting, retrievers still have a long journey to go. we hope this proposal can shed some light on building a more capable retriever in future.