Length is a Curse and a Blessing for Document-level Semantics (2023)

Abstract: In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models. In this work, we question the length generalizability of CL-based models, i.e., their vulnerability towards length-induced semantic shift. We verify not only that length vulnerability is a significant yet overlooked research gap, but we can devise unsupervised CL methods solely depending on the semantic signal provided by document length. We first derive the theoretical foundations underlying length attacks, showing that elongating a document would intensify the high intra-document similarity that is already brought by CL. Moreover, we found that isotropy promised by CL is highly dependent on the length range of text exposed in training. Inspired by these findings, we introduce a simple yet universal document representation learning framework, **LA(SER) ^3 **: length-agnostic self-reference for semantically robust sentence representation learning, achieving state-of-the-art unsupervised performance on the standard information retrieval benchmark. [Our code is publicly available.](https://github.com/gowitheflow-1998/LA-SER-cubed)

## introduction in recent years, contrastive learning (cl) has become the go-to method to train representation encoder models @xcite @xcite . in the field of natural language processing (nlp), the effectiveness of the proposed unsupervised cl methods is typically evaluated on two suites of tasks, namely, semantic textual similarity (sts) @xcite and information retrieval (ir) (e.g., @xcite ). surprisingly, a large number of works only validate the usefulness of the learned representations on sts tasks, indicating a strong but widely-adopted assumption that methods optimal for sts could also provide natural transferability to retrieval tasks. similarity matrix text encoder original and elongated sentence what is nlp? what is nlp? ×20 what is nlp? ×50 what is nlp? ×100 { original and elongated sentence show svg enter latex show svg enter latex what is computer vision? what is computer vision? x20 what is computer vision? ×50 what is computer vision? ×100 due to the document length misalignment of these two types of tasks, the potential gap in models' capability to produce meaningful representation at different length ranges has been rarely explored @xcite . studies of document length appear to have been stranded in the era where methods are strongly term frequencybased, because of the explicit reflection of document length to sparse embeddings, with little attention given on dense encoders. length preference for dense retrieval models is observed by @xcite , who show that models trained with dot-product and cosine similarity exhibit different length preferences. however, this phenomenon has not been attributed to the distributional misalignment of length between training and inference domains/tasks, and it remains unknown what abilities of the model are enhanced and diminished when trained with a certain length range. in this work, we provide an extensive analysis of length generalizability of standard contrastive learning methods. our findings show that, with default contrastive learning, models' capability to encode document-level semantics largely comes from their coverage of length range in the training. we first depict through derivation the theoretical underpinnings of the models' vulnerability towards length attacks. through attacking the documents by the simple copy-and-concatenating elongation operation, we show that the vulnerability comes from the further intensified high intra-document similarity that is already pronounced after contrastive learning. this hinders a stable attention towards the semantic tokens in inference time. further, we show that, the uniformity/isotropy promised by contrastive learning is heavily lengthdependent. that is, models' encoded embeddings are only isotropic on the length range seen in the training, but remain anisotropic otherwise, hindering the same strong expressiveness of the embeddings in the unseen length range. in the quest to bridge these unideal properties, we propose a simple yet universal framework, la(ser) 3 : length-agnostic self-reference for semantically robust sentence representation learning. by providing the simple signal that "the elongated version of myself 1) should still mean myself, and thus 2) should not become more or less similar to my pairs", this framework could not only act as an unsupervised contrastive learning method itself by conducting self-referencing, but could also be combined with any contrastive learning-based text encoding training methods in a plug-and-play fashion, providing strong robustness to length attacks and enhanced encoding ability. we show that, our method not only improves contrastive text encoders' robustness to length attack without sacrificing their representational power, but also provides them with external semantic signals, leading to state-of-the-art unsupervised performance on the standard information retrieval benchmark. ## length-based vulnerability of contrastive text encoders length preference of text encoders has been observed in the context of information retrieval @xcite , showing that contrastive learning-based text encoders trained with dotproduct or cosine similarity display opposite length preferences. @xcite further devised "adversarial length attacks" to text encoders, demonstrating that this vulnerability can easily fool text encoders, making them perceive a higher similarity between a text pair by only copying one of them n times and concatenating to itself. in this section, we first formalize the problem of length attack, and then analyze the most important pattern (misaligned intra-document similarity) that gives rise to this vulnerability, and take an attention mechanism perspective to derive for the first time the reason why contrastive learning-based text encoders can be attacked. ## method: la(ser) 3 after examining the two fundamental reasons underlying the built-in vulnerability brought by standard contrastive learning, the formulation of our method emerges as an intuitive outcome. naturally, we explore the possibility of using only length as the semantic signal to conduct contrastive sentence representation learning, and propose la(ser) 3 : length-agnostic self-reference for semantically robust sentence representation learning. la(ser) 3 builds upon the semanticspreserved assumption that "the elongated version of myself 1) should still mean myself, and thus 2) should not become more or less similar to my pairs". la(ser) 3 leverages elongation augmentation during the unsupervised constrastive learning to improve 1) the robustness of in-document interaction pattern in inference time; 2) the isotropy of larger length range. we propose two versions of reference methods, for different format availability of sentences in target training sets. @xcite . ♠: unfair comparison. notably, infocse benefits from the pre-training of an auxiliary network, while the rest of the baselines and our method fully rely on unsupervised contrastive fine-tuning on the same training wiki setting as described in §4. note that with a batch size of 64, our method already outperforms all baselines to a large margin except infocse. since we train with a max sequence length of 256 (all baselines are either 32 or 64), we find that training with a larger batch size (128) further stabilizes our training, achieving state-of-the-art results. further, we achieve state-of-the-art with only a bert base . self-reference in la(ser) 3 self-ref setting, we take a sentence from the input as an anchor for each training input, and construct its positive pair by elongating the sentence to be m times longer. intra-reference la(ser) 3 intra-ref conducts intrareference within the document. the two components of a positive pair are constructed from different spans of the same document. since we are only to validate effectiveness of la(ser) 3 intra-ref , we implement this in the simple mutually-excluded span setting. in other words, the la(ser) 3 intra-ref variant takes a sentence (either the first or a random sentence) from the text as an anchor, uses the rest of the text in the input as its positive pair, and elongates the anchor sentence m times as the augmented anchor. for both versions, we use the standard infonce loss @xcite with in-batch negatives as the contrastive loss. ## experiments training datasets we conduct our experiments on two training dataset settings: 1) training wiki uses 1m sentences sampled from wikipedia, in line with previous works on contrastive sentence representation learning @xcite ; 2) training msmarco uses msmarco @xcite , which is equivalent to indomain-only setting of the beir information retrieval benchmark @xcite . ## ablation analysis in this section, we ablate two important configurations of la(ser) 3 . firstly, the usage of la(ser) 3 involves deciding which sentence in the document to use as the anchor ( § 5.1). secondly, how do we maximize the utility of self-referential elongation? is it more important for the model to know "me * @xmath0, or is it more important to cover a wider length range ( § 5.2)? ## conclusion in this work, we questioned the length generalizability of contrastive learning-based text encoders. we observed that, despite their seemingly strong representational power, this ability is strongly vulnerable to length-induced semantic shifts. we formalized length attack, demystified it, and defended against it with la(ser) 3 . we found that, teaching the models "my longer-@xmath1.