D u R eader-Retrieval: A Large-scale C hinese Benchmark for Passage Retrieval from Web Search Engine (2022)

Abstract: In this paper, we present DuReader-retrieval, a large-scale Chinese dataset for passage retrieval. DuReader-retrieval contains more than 90K queries and over 8M unique passages from a commercial search engine. To alleviate the shortcomings of other datasets and ensure the quality of our benchmark, we (1) reduce the false negatives in development and test sets by manually annotating results pooled from multiple retrievers, and (2) remove the training queries that are semantically similar to the development and testing queries. Additionally, we provide two out-of-domain testing sets for cross-domain evaluation, as well as a set of human translated queries for for cross-lingual retrieval evaluation. The experiments demonstrate that DuReader-retrieval is challenging and a number of problems remain unsolved, such as the salient phrase mismatch and the syntactic mismatch between queries and paragraphs. These experiments also show that dense retrievers do not generalize well across domains, and cross-lingual retrieval is essentially challenging. DuReader-retrieval is publicly available at https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval.

## introduction passage retrieval requires systems to select candidate passages from a large passage collection. in recent years, pre-trained language models @xcite have been applied to retrieval problems, known as dense retrieval @xcite @xcite . the success of dense retrieval relies on the availability of high quality, large-scale, human-annotated corpora. a number of popular datasets are already available for english passage retrieval, including ms-marco (nguyen et al., † the work was done when the first author was doing internship at baidu. all queries are selected from real requests made by users at baidu search, and document passages are from the search results. similar to @xcite , we create the dureader retrieval from dureader @xcite , a chinese machine reading comprehension dataset, and obtain the human labels for paragraphs by distant supervision (see section 2.2). an example from dureader retrieval is shown in positive psg. 1 : 百度经验:jingyan.baidu.com花卉名称:太阳花播种时间:春、夏、秋均可播种为一年生肉质草本植物。株高10～15cm。花瓣颜色鲜艳,有白、深黄、红、紫等色。园艺品种很多,有单瓣、半重瓣、 重瓣之分。喜温暖、阳光充足而干燥的环境,极耐瘠薄,一般土壤均能适应,能自播繁衍。见阳光花开,早、晚、阴天闭合,故有太阳花、午时花之名。花期6～7月。太阳花种子非常细小。常采用 育苗盘播种,极轻微地覆些细粒蛭石,或仅在播种后略压实,以保证足够的湿润。发芽温度21～24℃,约7～10天出苗,幼苗极其细弱,因此如保持较高的温度,小苗生长很快,便能形成较为粗壮、肉质 的枝叶。这时小苗可以直接上盆,采用10厘米左右直径的盆,每盆种植2～5株,成活率高,生长迅速。 baidu experience: jingyan.baidu.com flower name: grandiflora sowing time: spring, summer, and autumn can be sown as an annual succulent herb. plant height is 10-15cm. the petals are bright in color, white, dark yellow, red, purple and other colors. there are many horticultural varieties, including single, semi-double and double petals. it likes a warm, sunny and dry environment, is extremely tolerant to barrenness, and can adapt to general soils and reproduce by itself. it is named grandiflora because it blooms when the sun is rising and closes in the morning, evening and cloudy days. it flowers from june to july. grandiflora's seeds are very small. the seedling trays are often used for sowing, very lightly covered with fine vermiculite, or only slightly compacted after sowing to ensure sufficient moisture. the germination temperature is 21~24 • c and the seedlings emerge in about 7~10 days. the seedlings are extremely thin. therefore, if the temperature is kept high, the seedlings will grow quickly, and thicker, fleshy branches and leaves can be formed. at this time, the seedlings can be directly put into pots, using pots with a diameter of about 10 cm, planting 2 to 5 plants per pot, with high survival rate and rapid growth. positive psg. 2 : 抹平容器中培养土平面,将剪来的太阳花嫩枝头插入竹筷戳成的洞中,深入培养土最多不超过2厘米。为使盆花尽快成形、丰满,一盆中可视花盆大小,只要能保持2厘米的间距,可扦插多株(到成苗 拥挤时,可分栽他盆)。接着浇足水即可。新扦插苗可遮阴,也可不遮阴,只要保持一定湿度,一般10天至15天即可成活,进入正常的养护。太阳花极少病虫害。平时保持一定湿度,半月施一次千分之 一的磷酸二氢钾,就能达到花大色艳、花开不断的目的。如果一盆中扦插多个品种,各色花齐开一盆,欣赏价值更高。每年霜降节气后(上海地区)将重瓣的太阳花移至室内照到阳光处。入冬后放 在玻璃窗内侧,让盆土偏干一点,就能安全越冬。次年清明后,可将花盆置于窗外,如遇寒流来袭,还需入窗内养护。 flatten the soil surface in the container, insert the cut branches of grandiflora into the hole made by the bamboo chopsticks, and deepen the soil for no more than 2 cm. to make the potted flowers take shape and fullness as soon as possible, multiple plants can be cut as long as the spacing of 2 cm can be maintained (when the seedlings are crowded, they can be planted in other pots). then pour plenty of water. the new cuttings can be shaded or not. as long as they maintain a certain humidity, they can survive 10 to 15 days and enter normal maintenance. grandiflora has very few pests and diseases. maintain a certain humidity at ordinary times, and apply one-thousandth of potassium dihydrogen phosphate once a half month to achieve the purpose of large flowers and continuous blooming. if there are multiple varieties of cuttings in one pot, the flowers of all colours will bloom in one pot, and the appreciation value will be higher. every year after the frost falls (shanghai area), the double-flowered grandiflora is moved indoors to shine in the sun. put it on the inside of the glass window after the winter, and let the potting soil dry a little to survive the winter safely. after the qingming festival in the following year, the flowerpots can be placed outside the window. in the construction of dureader retrieval , we try to alleviate the above issues and improve the quality of the development and testing sets in the following two ways (see section 2.3): moreover, inspired by @xcite , we provide two testing sets (see section 2.4) from the medical domain (cmedqa @xcite * and ccovid-news † ) as the separate testing sets for outof-domain evaluation. additionally, we also provide a set that contains human translated queries for cross-lingual retrieval evaluation (see section 2.5) @xcite . in this paper, we conduct extensive experiments. in our in-domain experiments, we find that there are many challenges to be addressed, such as salient phrase mismatches and syntactic mismatches (see section 3.5). it is also difficult for dense retrievers we summarize the characteristics of our dataset and our contributions as follows: our dataset comprises more than 90k queries and more than 8m unique passages from baidu search. ## dureader retrieval in this section, we introduce our dureader retrieval dataset (see dataset statistics in dataset lang #que. #psg. source of que. source of psg. psg. annotation ms-marco (nguyen et al., 2016) en 516k 8.8m user logs web doc. human triviaqa (joshi et al., 2017) en 95k 650k trivia web. wiki./web doc. dist. sup. natural questions (kwiatkowski et al., 2019) en 61k 21m user logs wiki doc. dist. sup. mmarco-chinese (bonifacio et al., 2021) cn 516k 8.8m user logs web doc. tranlation ccovid-news ‡ cn 4.9k 5k user question covid-19 news doc. human cmedqa-2.0 (zhang et al., 2018) cn 108k 203k user question medical forum question-answer pairs tiangong-pdr (wu et al., 2019) cn 70 11k user logs news doc. human sougou-qcl (zheng et al., 2018) cn 537k 9m click logs web data click signal dureader retrieval (our work) cn 97k 8.9m user logs web doc. dist. sup. + human ## conclusion this paper presents a large-scale chinese passage retrieval dataset to benchmark the retrieval systems. in order to ensure the quality of our dataset, we employ two strategies: 1) reducing the false negatives in development and testing sets using a pooling approach and human annotations, and 2) removing the training queries that are semantically similar to the development and testing queries. in addition, we provide two testing sets for out-of-domain evaluation, and a set for cross-lingual evaluation. we examine several retrieval baselines, including the traditional sparse retrieval system and the neural retrievers, and present the challenges and the limitations of our dataset. we hope this dataset can help facilitate the research of passage retrieval.