D oc S plit: Simple Contrastive Pretraining for Large Document Embeddings (2023)

Abstract: Existing model pretraining methods only consider local information. For example, in the popular token masking strategy, the words closer to the masked token are more important for prediction than words far away. This results in pretrained models that generate high-quality sentence embeddings, but low-quality embeddings for large documents. We propose a new pretraining method called DocSplit which forces models to consider the entire global context of a large document. Our method uses a contrastive loss where the positive examples are randomly sampled sections of the input document, and negative examples are randomly sampled sections of unrelated documents. Like previous pretraining methods, DocSplit is fully unsupervised, easy to implement, and can be used to pretrain any model architecture. Our experiments show that DocSplit outperforms other pretraining methods for document classification, few shot learning, and information retrieval tasks.

## introduction generating high-quality text embeddings for documents is a long-standing open problem. most previous studies focus on either learning sentencelevel representations @xcite @xcite where training data usually contain short text or designing specific model structures for larger-range dependencies @xcite , but effective and efficient document representation learning methods are less explored. this paper presents docsplit, the first unsupervised pretraining method designed specifically for large documents. docsplit is simple to use, and it can be applied to any model architecture to improve document representations. docsplit uses contrastive learning, and our key contribution is a new method for generating positive samples for contrastive learning. figure 1 provides a graphical deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning. artificial neural networks were inspired by information processing and distributed communication nodes in biological systems. artificial neural networks have various differences from biological brains. specifically, artifical neural networks tend to be static and symbolic, whereas the biological brain of most living organisms is dynamic and analog. ## method we first review contrastive learning at a high level, then we describe our docsplit method. ## related work there are two categories of related work: models trained using contrastive learning, and models designed for large documents. ## experiments we perform a careful ablation study to isolate the effects of the docsplit pretraining method on downstream task performance. first, we pretrain separate models for each group of baseline models described in section 3 above. then, we perform downstream experiments on standard classification, few-shot learning, and document retrieval tasks. in all cases, our pretrained models significantly outperform prior work. ## conclusion docsplit is the first unsupervised pretraining method designed explicitly for large documents. docsplit can be used to pretrain any model architecture, and we provide code to do so at https://blinded_for_review the longformer-based docsplit long model provides sota performance on document classification, few-shot learning, and retrieval tasks.