Improving Embedding-based Large-scale Retrieval via Label Enhancement (2021)

Abstract: Current embedding-based large-scale retrieval models are trained with 0-1 hard label that indicates whether a query is relevant to a document, ignoring rich information of the relevance degree. This paper proposes to improve embedding-based retrieval from the perspective of better characterizing the query-document relevance degree by introducing label enhancement (LE) for the first time. To generate label distribution in the retrieval scenario, we design a novel and effective supervised LE method that incorporates prior knowledge from dynamic term weighting methods into contextual embeddings. Our method significantly outperforms four competitive existing retrieval models and its counterparts equipped with two alternative LE techniques by training models with the generated label distribution as auxiliary supervision information. The superiority can be easily observed on English and Chinese large-scale retrieval tasks under both standard and cold-start settings.

## introduction retrieval systems such as search engines have been a vital tool in helping people access the vast amount of information online. as shown in figure 1 , existing methods for large-scale retrieval will first utilize a less powerful but more efficient retrieval algorithm (retriever) to reduce the potential candidates, and then employ more powerful models (ranker) to re-rank the retrieved documents @xcite ). we will focus on improving retriever in this paper. with pre-trained word embeddings @xcite @xcite and language models (e.g., bert @xcite and roberta @xcite ) achieving great success in a wide variety of nlp tasks, researchers have begun to leverage bertstyle models to solve large-scale retrieval problems. these models consider the retrieval phase as a regression task trained with 0-1 hard labels, representing only two types of relevance degrees (relevant or irrelevant) between query-document pairs @xcite . the relevance degrees between queries and documents, however, can have much more possibilities. for example, we present a query and three actual results retrieved by the google search engine in figure 2 . though all three documents are relevant to the query, the relevance degrees can vary significantly if we assign a real-valued number indicating to what extent a query and a document relate. on the other hand, even if a query and a document are marked as irrelevant by the hard label, a weak relevance degree may exist between them. in such scenarios, label distribution @xcite , which involves the relevance degrees between queries and documents, is a more reasonable description of an instance. the observation inspires us to explore the label distribution to improve existing large-scale retriever models trained with hard labels. we can easily expect the following two novel le methods for retriever models. ## the proposed approach given a training set @xmath0, y i , l i )|1 ≤ @xmath1, the hard label l @xmath2, 1} denotes whether a query x i and document y i are relevant or not. our proposed le method can automatically generate label distributions d i for each query-document pair x i , y i , which is further introduced to assist retrieval tasks. the details are demonstrated in the following subsections. ## conclusion this paper first introduced label distribution to characterize the relevance degree between queries and documents in large-scale retrieval problems. then we designed a novel and effective label enhancement method that generates label distributions via fusing context-free tf-idf information and contextual bert representation. an improved retriever model was achieved easily by incorporating the generated label distributions as auxiliary supervision information. our method's superiority can be observed on four datasets of english and chinese.