Unsupervised Document Expansion for Information Retrieval with Stochastic Text Generation (2021)

Abstract: One of the challenges in information retrieval (IR) is the vocabulary mismatch problem, which happens when the terms between queries and documents are lexically different but semantically similar. While recent work has proposed to expand the queries or documents by enriching their representations with additional relevant terms to address this challenge, they usually require a large volume of query-document pairs to train an expansion model. In this paper, we propose an Unsupervised Document Expansion with Generation (UDEG) framework with a pre-trained language model, which generates diverse supplementary sentences for the original document without using labels on query-document pairs for training. For generating sentences, we further stochastically perturb their embeddings to generate more diverse sentences for document expansion. We validate our framework on two standard IR benchmark datasets. The results show that our framework significantly outperforms relevant expansion baselines for IR.

## introduction information retrieval (ir) is the task of retrieving the most relevant documents, including scientific ones @xcite , for a given query. ir systems have received considerable attention as they are not only required to search documents for information, but are also used as a core component in various downstream language understanding tasks such as open-domain question answering @xcite , fact verification @xcite and information extraction @xcite . as the simplest approach to ir tasks, classical term-based ranking models, such as bm25 @xcite and query likelihood (ql) mod-els @xcite , have been widely used. these term-based ranking models measure the lexical overlaps between query and document pairs using a sparse representation of words, to match the relevant documents for the given query. notwithstanding their simplicity, they achieve decent performances, even compared to the recent dense representation models @xcite , which require a large number of paired query-document samples. however, these termbased sparse models are intrinsically vulnerable to the vocabulary mismatch problem, which happens when a query and its relevant document are lexically divergent. thus, we should address the limitations of both sparse and dense models, about the vocabulary mismatch problem and the need for a large amount of training data, respectively. along this line, there are methods that expand queries and documents with their relevant terms. they include document expansion methods @xcite that introduce additional contextrelated terms to given documents and query expansion methods @xcite that augment given queries with additional terms. by doing so, we can explicitly generate lexically richer documents or queries. compared with query expansion, document expansion has two strengths. first, a document expansion model can generate much more relevant terms for the given document, since documents are generally much longer than queries. also, documents can be expanded during indexing time so that the responding process for the user's query is not delayed, in contrast to queries that must be expanded during retrieval time. thus, document expansion is more appropriate for a real-time system, together with making available more context-related words from the given information @xcite . in this work, we focus on document expansion, and propose to abstractly generate the key infor-q: why do some people take life so seriously? udeg original document generated text user's query ## related work information retrieval a two-stage pipeline is the most prominent approach for ir. this pipeline first retrieves query-relevant documents with their sparse representations, and then re-ranks them by using neural networks @xcite @xcite . in this two-stage pipeline, the overall performance is critically dependent on the first retrieval stage, since the failure of the retrieval stage would highly affect the second re-ranking stage. therefore, this bottleneck on the first stage has to be addressed for performance enhancement @xcite . bm25 and query likelihood (ql) are the most popular adhoc retrieval models for the first stage @xcite @xcite . more recently, instead of using sparse models, methods of using dense representations have been proposed @xcite @xcite , which can help alleviate the vocabulary mismatch problem through a dense representation space. however, recent work has revealed the limitations on their performance and efficiency @xcite @xcite . furthermore, these dense representation methods are based on supervised learning, where pairs of query and related-document are usually required to ensure reasonable performance. query / document expansion query and document expansions have been widely used in ir systems. in terms of query expansion, @xcite proposed pseudo relevance feedback (rm3), which is revisited in more recent work @xcite ) for its strength. there are also methods that expand queries using generation schemes @xcite . however, query expansion suffers from its intrinsic drawbacks, as queries need to be manipulated during the retrieval phase and have relatively less information than documents @xcite . thus, we take the alternative route: expanding documents. @xcite and @xcite proposed to expand documents with generated text using a supervised model trained on query-document pairs. in contrast, our framework generates document-related sentences regardless of the existence of the corresponding query. @xcite proposed to expand documents with sequence-to-sequence models which output keyphrases; however, their models have to be trained from scratch on a specific domain. ## method our goal is to expand the document for ir tasks by generating document-related text, which contains novel but semantically similar terms for the given document without using query-document pairs. in this section, we describe formal description of the ir task. ## experimental setups here, we describe datasets, models, evaluation metrics, and implementation details for experiments. ## results and discussion in this section, we show the overall performance of our udeg, and then analyze the results in detail. ## conclusion we presented a novel framework, which we refer to as unsupervised document expansion with generation (udeg), that generates diverse terms with stochastic perturbation over pre-trained language models, and efficiently enriches the document representation, without using any query infor-mation for training. remarkably, udeg employed in a retrieval system shows significant performance improvements on two standard benchmark datasets. also, a detailed analysis shows that an abstractive generation framework with stochastic perturbation positively contributes to the retrieval performance. not only synonymy, but also other problems of the ir system such as polysemy could be addressed using our udeg framework, to be left for the future work. we believe that the benefits of using diversely generated document-relevant sentences would allow further improvements on any ir system, targeting at scholarly and scientific information.