A Self-supervised Joint Training Framework for Document Reranking (2022)

Abstract: Pretrained language models such as BERT have been successfully applied to a wide range of natural language processing tasks and also achieved impressive performance in document reranking tasks. Recent works indicate that further pretraining the language models on the task-specific datasets before fine-tuning helps improve reranking performance. However, the pre-training tasks like masked language model and next sentence prediction were based on the context of documents instead of encouraging the model to understand the content of queries in document reranking task. In this paper, we propose a new self-supervised joint training framework (SJTF) with a self-supervised method called Masked Query Prediction (MQP) to establish semantic relations between given queries and positive documents. The framework randomly masks a token of query and encodes the masked query paired with positive documents, and uses a linear layer as a decoder to predict the masked token. In addition, the MQP is used to jointly optimize the models with supervised ranking objective during fine-tuning stage without an extra further pre-training stage. Extensive experiments on the MS MARCO passage ranking and TREC Robust datasets show that models trained with our framework obtain significant improvements compared to original models.

## conclusion this paper proposes a self-supervised joint train ing framework sjtf to improve the representation learning for document ranking. by integrating with this training framework, models are able to better understand the content of queries and capture the semantic relation between queries and positive doc uments, which guides the models to identify the relevant documents among a large number of can didate documents. experimental results show that our proposed framework enhances the represen tation learning of reranking models and achieves better performance compared with baselines in the document reranking task.