Hyperlink-induced Pre-training for Passage Retrieval in Open-domain Question Answering (2022)

Abstract: To alleviate the data scarcity problem in training question answering systems, recent works propose additional intermediate pre-training for dense passage retrieval (DPR). However, there still remains a large discrepancy between the provided upstream signals and the downstream question-passage relevance, which leads to less improvement. To bridge this gap, we propose the HyperLink-induced Pre-training (HLP), a method to pre-train the dense retriever with the text relevance induced by hyperlink-based topology within Web documents. We demonstrate that the hyperlink-based structures of dual-link and co-mention can provide effective relevance signals for large-scale pre-training that better facilitate downstream passage retrieval. We investigate the effectiveness of our approach across a wide range of open-domain QA datasets under zero-shot, few-shot, multi-hop, and out-of-domain scenarios. The experiments show our HLP outperforms the BM25 by up to 7 points as well as other pre-training methods by more than 10 points in terms of top-20 retrieval accuracy under the zero-shot scenario. Furthermore, HLP significantly outperforms other pre-training methods under the other scenarios.

## introduction open-domain question answering (openqa) aims to answer factual open questions with a large external corpus of passages. current approaches to openqa usually adopt a two-stage retriever-reader paradigm @xcite to fetch the final answer span. the performance of openqa systems is largely bounded by the retriever as it determines the evidential documents for the reader to examine. traditional retrievers, such as tf-idf and bm25 @xcite , are considered incapable of adapting to sce-in 2011 he directed his first international feature film, romantic comedy "letters to santa (listy do m.)" who directs the romantic comedy "letters to santa"? letters to santa (polish: listy do m.), alternatively known as letters to st. nicholas, is a 2011 polish-language romantic comedy film, directed by the director mitja okorn. the action takes place during one single christmas eve, when a few adults find the loves of their lives. the film's plot refers to the 2003 romantic comedy "love actually", though events of the movie differ from the ones in the 2003 film. it was shot in warsaw, poland from 27 january to march 2011. the movie had 3 sequels, letters to santa 2 released in 2015, letters to santa 3 in 2017, and listy do m. 4 that was produced in 2020 and rescheduled to premiere after ## hyperlink-induced pre-training (hlp) in this section, we firstly discuss the background of openqa retrieval, then our methodology and training framework. ## experimental setup in this session, we discuss the pre-training corpus preparation, downstream datasets, the hyperparameter and the basic setup for our experiments. ## analysis 6.1 ablation study to better understand how different key factors affect the results, we conduct ablation experiments with results shown in hyperlink-based topologies our proposed dual-link (dl) and co-mention (cm) q-p pairs, provide evidence induced by different hyperlinkbased topologies. to examine their respective effectiveness, we pre-train retrievers on q-p pairs derived from each topology and their combinations. we present zero-shot retrieval results in ## conclusion this paper proposes hyperlink-induced pretraining (hlp), a pre-training method for openqa passage retrieval by leveraging the online textual relevance induced by hyperlink-based topology. our experiments show that hlp gains significant improvements across multiple qa datasets under different scenarios, consistently outperforming other pre-training methods. our method provides insights into openqa passage retrieval by analyzing the underlying bi-text relevance. future work involves addressing tasks like ms marco where the granularity of the information-seeking target is at the passage level.