Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning (2023)

Abstract: Learning quality document embeddings is a fundamental problem in natural language processing (NLP), information retrieval (IR), recommendation systems, and search engines. Despite recent advances in the development of transformer-based models that produce sentence embeddings with self-contrastive learning, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations. Therefore, we train Longfomer-based document encoders using a state-of-the-art unsupervised contrastive learning method (SimCSE). Further on, we complement the baseline method -siamese neural network- with additional convex neural networks based on functional Bregman divergence aiming to enhance the quality of the output document representations. We show that overall the combination of a self-contrastive siamese network and our proposed neural Bregman network outperforms the baselines in two linear classification settings on three long document topic classification tasks from the legal and biomedical domains.

## introduction the development of quality document encoders is of paramount importance for several nlp applications, such as long document classification tasks with biomedical @xcite , or legal @xcite documents, as well as information retrieval tasks @xcite @xcite . despite the recent advances in the development of transformer-based sentence encoders @xcite @xcite via unsupervised contrastive learning, little do we know about the potential of neural document-level encoders targeting the encoding of long documents (ks of words). training corpus average text length reimers and gurevych (2019) inter alia snli 22 mnli 113 ms marco 335 wikipedia 200 our work ecthr 1,613 mimic 1,621 scotus 5,853 the computational complexity of standard transformer-based models @xcite (plms) given the quadratic self-attention operations poses challenges in encoding long documents. to address this computational problem, researchers have introduced efficient sparse attention networks, such as longformer @xcite , bigbird @xcite , and hierarchical transformers @xcite . nonetheless, fine-tuning such models in downstream tasks is computationally expensive; hence we need to develop efficient document encoders that produce quality document representations that can be used for downstream tasks out-ofthe-box, i.e., without fully (end-to-end) fine-tuning the pre-trained encoder, if not at all. besides computational complexity, building good representation models for encoding long documents can be challenging due to document length. long documents contain more information than shorter documents, making it more difficult to capture all the relevant information in a fixed-size representation. in addition, long documents may have sections with different topics, which increases the complexity of encoding that usually leads to collapsing representations @xcite . more-over, long documents can be semantically incoherent, meaning that content may not be logically related or may contain irrelevant information. for these reasons, it is challenging to create a quality representation that captures the most important information in the document. to the best of our knowledge, we are the first to explore the application of self-contrastive learning for long documents ( self-supervised contrastive learning in nlp several self-contrastive methods have been proposed so far for nlp applications. to name a few: mirrorroberta @xcite , scd @xcite , micse @xcite , declutr @xcite , and simcse @xcite ) -described in section 3.2-, all create augmented versions (views) of the original sentences using varying dropout and comparing their similarity. the application of such methods is limited to short sentences and relevant downstream tasks, e.g., sentence similarity, while these methods do not use any additional component to maximize diversity in latent feature representations. ## results and discussion in classification performance: in the last line of in the other two datasets (ecthr and mimic), the performance of the linear models is still approx. 10-15 p.p. behind the ceilings in µ-f 1 . in ecthr, we find that self-contrastive learning improves performance in the first settings by 3 p.p. in µ-f 1 , while the additional divergence bregman loss does not really improve performance. this is not the in given the overall results, we conclude that building subnetwork ensembles on top of the document embeddings can be a useful technique for encoding long documents and can help avoid the problem of collapsing representations, where the model is unable to capture all the relevant information in the input. our approach has several advantages for long-document processing: efficiency considerations: in avoidance of collapsing representations: when processing long documents, there is a risk that the representation will collapse @xcite , meaning that the model will not be able to capture all the relevant information in the input. by mapping the document embedding from the base encoder into smaller sub-networks, the risk of collapsing representations is reduced, as the divergence loss attempts to reduce redundancy in the feature representation by minimizing the correlation. the results shown in ## conclusions and future work we proposed and examined self-supervised contrastive divergence learning for learning representation of long documents. our proposed method is composed of a self-contrastive learning framework followed by an ensemble of neural networks that are optimized by functional bregman divergence. our method showed improvement compared to the baselines on three long document topic classifications in the legal and biomedical domains, while the improvement is more vibrant in a few-shot learning setting. in future work, we would like to further investigate the impact of the bregman divergence loss in more classification datasets and other nlp tasks, e.g., document retrieval.