On Complementarity Objectives for Hybrid Retrieval (2023)

Abstract: Dense retrieval has shown promising results in various information retrieval tasks, and hybrid retrieval, combined with the strength of sparse retrieval, has also been actively studied. A key challenge in hybrid retrieval is to make sparse and dense complementary to each other. Existing models have focused on dense models to capture “residual” features neglected in the sparse models. Our key distinction is to show how this notion of residual complementarity is limited, and propose a new objective, denoted as RoC (Ratio of Complementarity), which captures a fuller notion of complementarity. We propose a two-level orthogonality designed to improve RoC, then show that the improved RoC of our model, in turn, improves the performance of hybrid retrieval. Our method outperforms all state-of-the-art methods on three representative IR benchmarks: MSMARCO-Passage, Natural Questions, and TREC Robust04, with statistical significance. Our finding is also consistent in various adversarial settings.

## introduction representing and matching queries and documents (or answers) is crucial for designing models for information retrieval (ir) and open-domain question answering (qa). existing approaches have been categorized into sparse and dense retrieval. classic sparse (or symbolic) retrieval such as bm25 @xcite , quantifies the lexical overlaps (or exact matches) between query q and document d, weighted by term frequency (tf) and inverse document frequency (idf). such computation can be efficiently localized to a few high-scoring q-d pairs with an inverted index, may fail to match pairs with term mismatches. for example, a text pair with identical intent-"facebook change password" and "fb modify passwd"-does not share any common word, so the pair cannot be matched by lexical retrieval. to overcome such mismatches, dense retrieval models, such as bert-based dpr @xcite or cocondenser @xcite , aim to support soft "semantic matching", by encoding queries and documents into lowdimensional embedding vectors. dense representation is trained so that "password" and "passwd" are located close in the space even though they have different lexical representations. these complementary advantages of each model have naturally motivated hybrid models @xcite @xcite , which we denote as bm25+dpr, extracting scores from both models and selecting documents with the highest linearly combined scores. to illustrate how we advance bm25+dpr baseline, figure 1(a) shows recall@10 of bm25+dpr on natural questions, where a yellow circle, represents questions answerable by bm25, or s, and a blue circle, represents those answerable by dpr, or d. desirably, two retrievers together should cover all questions in the universe u, but failure is 46.5%, which corresponds to u -d ∪ s. to improve, there are two directions: (1) enlarging |d| and (2) making it more complementary to s. figure 1(b ) plots clear @xcite , aiming to emphasize "residual" features neglected in sparse model, or, increase |d -s|. though |d -s| increased from 15.2% (figure 1a ) to 20.0% (figure 1b ), as intended, failure did not decrease significantly, from 46.5% (figure 1a ) to 41.8% (figure 1b ). we argue this decrease in failure cases, is confounded by enlarging |d| from 47.6 (figure 1a ) to 54% (figure 1b ), by comparing with a hypothetical scenario keeping d fixed, but reducing failure cases significantly from 41.8% to 14.1% when the intersection is reduced. roc is designed to be 1 when two models are disjoint (figure 1c ), and 0 when d is subsumed to s. roc has the following two advantages: with these advantages, we use our metric in ## proposed method we propose roc in section 3.1 and then discuss how s (section 3.2) and d (section 3.3) are implemented. section 3.4 discusses how d and s can be combined to optimize roc. ## experiment in this section, we describe experimental setting and formulate our research questions to guide our experiments. ## conclusion we study the problem of hybrid retrieval, where existing state-of-the-arts have pursued a partial notion of complementarity. in contrast, we propose roc, a metric that captures a fuller notion of the complementarity between sparse and dense models. we then propose a simple but effective twolevel orthogonality objective to enhance roc and verify that optimizing roc enhances both com-plementarity and retrieval, leads to outperforming state-of-the-arts in three representative ir benchmarks, msmarco-passage, natural questions, and trec robust04, and generalizing to adversarial settings.