Unsupervised Dense Retrieval for Scientific Articles (2022)

Abstract: In this work, we build a dense retrieval based semantic search engine on scientific articles from Elsevier. The major challenge is that there is no labeled data for training and testing. We apply a state-of-the-art unsupervised dense retrieval model called Generative Pseudo Labeling that generates high-quality pseudo training labels. Furthermore, since the articles are unbalanced across different domains, we select passages from multiple domains to form balanced training data. For the evaluation, we create two test sets: one manually annotated and one automatically created from the meta-information of our data. We compare the semantic search engine with the currently deployed lexical search engine on the two test sets. The results of the experiment show that the semantic search engine trained with pseudo training labels can significantly improve search performance.

## introduction search engines are deeply integrated into elsevier's information services of its scientific literature data. an example is the one provided by sciencedirect foot_0 , providing researchers with search services on more than 19m full text articles. these search engines are currently based on lexical search models such as bm25. the deployment of such models is effortlessly simplified by using popular industrystandard libraries such as elasticsearch foot_1 . however, lexical search suffers from the lexical gap problem such as misspellings, synonyms, abbreviations, ambiguous words, and ignoring of word order @xcite . recently, dense retrieval (dr) models have proven to be highly effective in solving the lexical gap problem while still remain fast search speed @xcite . dr models map queries and passages to a common vector space and retrieve relevant passages results what is machine learning by searching for (approximate) nearest neighbors. dr has been well studied on laboratory data but still in the early stage for industry-level applications @xcite . dr is mainly applied in multi-modal search in industry where traditional lexical search is not possible, like text-image search @xcite or music search @xcite . it is of great interest to use state-of-the-art dr models to build semantic search engines for industry. such search engines can enable efficient access and search to scientific literature of elsevier and help researchers in their journey @xcite . our goal in this work is to develop a semantic search engine that needs no relevance-labeled data to train the dr model, thus allowing easy adaptation to new domains and easy deployment in industry. there are several challenges to be tackled. first, training a dr model requires sufficient labeled data such as ms-marco @xcite , whereas there is often no such data for specific domains or startups. in our case, we have a large collection of passages from scientific articles but no relevance label. furthermore, it is shown that dr models trained on one domain do not generalize to another @xcite . the passages in our corpus have a different word distribution compared to that in ms-marco. besides, the passages are also unbalanced regarding their domains (see section 4.3). therefore, using the models trained on ms-marco will not yield high retrieval performance. it is interesting to tackle the domain difference problem. finally, there is no test set to evaluate search performance and creating a goodquality test set is time-consuming and expensive. all these challenges hinder the application of dr models in industry setting. in this work, we trained a dr model using a state-of-the-art unsupervised dense retrieval model called gpl @xcite . it uses a pretrained query generator to generate queries from passages. the passage is considered as positive for the generated queries. negative passages for generated queries are retrieved using existing dense retrieval models trained on ms-marco. an existing cross-encoder model trained on ms-marco produces relevance scores of query-passage pairs as supervision signals to train the dr model. finally, we constructed two test sets by either manual annotation or automatic extraction of existing relevance information from the meta field of the corpus. the experimental results show that our best model can significantly improve the retrieval performance compared to lexical and semantic search baselines. the semantic search engine we have created for our product is shown in figure 1 . it is currently deployed and running in a beta test mode. ## conclusions & future work in this work, we build a semantic search engine on scientific articles. to tackle the challenge of no labeled data for both training and test, we apply a state-of-the-art unsupervised dense retrieval model named gpl. as the articles are unbalanced across different domains, we sample passages from multiple domains to form balanced training batches. we also created two test sets for the evaluation: one manually annotated and one automatically constructed from the meta information of our corpus. we compare the semantic search engine with the currently deployed lexical search engine on the test sets. both the qualitative and quantitative experiment results show that the semantic search engine can significantly improve the search performance. this results suggest that gpl is a robust and effective model for unsupervised dense retrieval. for the future work, we will train the query generator and the negative retriever on our data to generate a better quality of both positive and negative training example to improve the retrieval performance.