Multi-objective Representation Learning for Scientific Document Retrieval (2022)

Abstract: Existing dense retrieval models for scientific documents have been optimized for either retrieval by short queries, or for document similarity, but usually not for both. In this paper, we explore the space of combining multiple objectives to achieve a single representation model that presents a good balance between both modes of dense retrieval, combining the relevance judgements from MS MARCO with the citation similarity of SPECTER, and the self-supervised objective of independent cropping. We also consider the addition of training data from document co-citation in a sentence context and domain-specific synthetic data. We show that combining multiple objectives yields models that generalize well across different benchmark tasks, improving up to 73% over models trained on a single objective.

## introduction with the explosive growth of the volume of scientific publications, researchers increasingly rely on sophisticated discovery and recommendation tools to find relevant literature and related work @xcite . in particular, the development of neural information retrieval @xcite has led to a quest for dense document representations that capture the semantics of documents better than the previous generation of keyword-based retrieval methods. such representations are typically achieved by specializing pre-trained large language models for the retrieval task. in this paper, we focus on the case of the biencoder @xcite , where at indexing time documents are embedded as dense vector representations and stored in a fast approximate nearest neighbor system, and at retrieval time queries are encoded using the same model and similarity search is performed in the vector space. how can we best combine multiple data sets and task objectives to train a scientific document retriever that can be queried both using short queries and documents? although in practice a retrieval system could use multiple different document embeddings for multiple tasks, a single multi-purpose document representation offers great advantages in terms of storage space, computational resources, and operational efficiency. the main contributions of this paper are the following: ## results and discussion single vs. multi-objective training. we first study the impact of adding supervision signals from multiple sources compared to a single training objective. combining objectives. we study the impact of the four combining methods: in-batch mixing, the results are consistent across the 3 pairs of objectives. there is no clear preference between alternating batches and in-batch mixing for two objectives. the models trained with the latter perform best on scidocs and iclr2022 while the models trained with alternate batches perform best on beir. maybe the diverse domains of the beir data sets create negatives that are too easy to spot, while the domains of scidocs and iclr2022 are relatively similar making the negative harder and forcing the model to learn better representations. when using three objectives, alternating between batches performs well across the three evaluation metrics and seems like the best compromise. ## conclusion we explored multi-objective dense retrieval training as a way to optimize models for both short queries and document queries. we study three ways to combine objectives (in-batch mixing, alternating between batches, and finetuning). we find that using multiple objectives is a way to train dense retrievers that perform well for short and long query retrieval. considering the performances across a subset of beir, scidocs and iclr2022 our best models achieve an average relative improvement between 13.4 and 19.2% compared to the best single objective models. our work here focused on bi-encoders, and future work could explore whether multi-objective training is also beneficial for a cross-encoder architecture @xcite . furthermore, we find that pre-training a model using independent cropping and finetuning it on a target objective consistently improves the retrieval performance compared to only training on the target objective. we also introduced context-to-document, a new weakly supervised training objective using the citation context sentence as the query for the cited document. this signal outperforms the co-cited relevance signal and improves the model performance on short query retrieval. for future work, we would like to explore how to pre-filter citation contexts that do not contain useful information to identify a relevant document, thus removing potential noise from the training data. finally, we released a new document to document retrieval data set composed of iclr 2022 papers and 1,844 human relevance judgement. all our experiments were conducted using a minilm-l6 (22.7 million parameters) model which is more than 10 times smaller than bert @xcite (345 million parameters). future work should consider how multiobjective training scales to larger models. with this work, we hope to make a step in the direction of a multi-purpose document representation to reduce storage space, computational resources, and increase the operational efficiency of scientific retrieval systems.