C on I s I : A Contrastive Framework with Inter-sentence Interaction for Self-supervised Sentence Representation (2022)

Abstract: “Learning sentence representation is a fundamental task in natural language processing and has been studied extensively. Recently, many works have obtained high-quality sentence representation based on contrastive learning from pre-trained models. However, these works suffer the inconsistency of input forms between the pre-training and fine-tuning stages. Also, they typically encode a sentence independently and lack feature interaction between sentences. To conquer these issues, we propose a novel Contrastive framework with Inter-sentence Interaction (ConIsI), which introduces a sentence-level objective to improve sentence representation based on contrastive learning by fine-grained interaction between sentences. The sentence-level objective guides the model to focus on fine-grained semantic information by feature interaction between sentences, and we design three different sentence construction strategies to explore its effect. We conduct experiments on seven Semantic Textual Similarity (STS) tasks. The experimental results show that our ConIsI models based on BERTbase and RoBERTabase achieve state-ofthe-art performance, substantially outperforming previous best models SimCSE-BERTbase and SimCSE-RoBERTabase by 2.05% and 0.77% respectively.”

## introduction learning good universal sentence representation is a fundamental task and benefits a wide range of natural language processing tasks such as text classification and machine translation, especially for largescale semantic similarity computation and information retrieval. with the rise of pre-trained language models @xcite , many downstream tasks have achieved remarkable improvements. however, the native sentence representation derived from pre-trained language models without additional supervision are usually low-quality and can not be used directly @xcite . recently, contrastive learning has become a popular approach to improve the quality of sentence representation in a self-supervised way. contrastive learning is an approach of learning effective feature representation by positive pairs and negative pairs. it generally takes different views as positive or negative pairs for each sentence using various data augmentation ways. and it works by pulling semantically close positive instances together and pushing negative instances away. however, current approaches based on contrastive learning mainly suffer two problems: train-tuned bias and fine-grained interaction deficiency. firstly, previous approaches typically input a single sentence to the encoder at a time, which is inconsistent with the pre-training stage of the language models. most language models concatenate multiple sentences as the input form at the pre-training stage. we argue that the inconsistency of input forms between the pre-training and fine-tuning stages may harm the performance. secondly, each sentence in a minibatch is encoded independently while training, which lacks fine-grained interaction information between sentences. according to previous works in text matching @xcite @xcite , modeling a proper interaction between input sentences can improve the performance of semantic feature embedding for representation-based models, but existing works on sentence representation ignore the importance of this interaction. therefore, to conquer these drawbacks of current contrastive learning based methods, we propose conisi, a contrastive framework with inter-sentences interaction for self-supervised sentence representation. firstly, we present to construct a sentence pair as positive instance for each sentence to alleviate the train-tuned bias. by referring to an original sentence and a sentence pair as a positive pair, the model can not only obtain effective representation of a single sentence, but also mitigate the train-tuned bias between the pre-training and fine-tuning stages. further, to solve the problem of lacking interaction between sentences, we propose a sentence-level objective to perform the inter-sentence interaction during encoding. we pass a pair of sentences as a text sequence into the encoder and the target semantic category of the two sentences is predicted. the sentence pair is sufficiently interacted through the internal interaction mechanism in transformer-based block @xcite during encoding. through the inter-sentence interaction, the model can encode fine-grained semantic information and achieve further improvement. moreover, for a minibatch of n sentences, there are n • (n -1)/2 interactive computations. in order to ensure the training efficiency, we do not perform an interactive operation on all data due to too many possible combinations. instead, we artificially construct a sentence for each original sentence to adjust the difficulty of the interactive objective, which only requires n interactive computations. we propose several models based on three sentence construction strategies, named conisi-o1, conisi-o2, and conisi-s, respectively. the overall model of our proposed conisi can be seen in figure 1 . our contributions can be summarized as follows: ## related work sentence representation built upon the distributional hypothesis has been widely studied and improved considerably. early works @xcite @xcite inspired by word2vec @xcite lead to strong results by predicting surrounding information of a given sentence. the emergence of pre-trained models such as bert @xcite shows much great potential for sentence representation. recently, many works have explored how to learn better sentence embeddings from the pre-trained models. supervised methods a common supervised step of learning a model is fine-tuning with labeled data in downstream training sets. several works build upon the success of using annotated natural language inference (nli) datasets (including stanford nli @xcite and multi-genre nli @xcite ) for sentence representation, which projects it as a 3-way classification task (entailment, neutral, and contradiction) to get better sentence embeddings. @xcite use a bilstmbased model as encoder, and they train it on both stanford nli and multi-genre nli datasets. universal sentence encoder @xcite uses the stanford nli dataset to enhance the unsupervised training by adopting a transformer-based model. sentence-bert @xcite that adopts a siamese network @xcite with a shared bert encoder is also trained on stanford nli and multi-genre nli datasets. ## methodology in this section, we present conisi, a contrastive framework with inter-sentence interaction for selfsupervised sentence representation, which contains two parts: (1) the conisi model of joint contrastive learning objective and inter-sentence interactive objective (section 3.1), and (2) the strategies of sentence construction in the inter-sentence interactive objective (section 3.2). ## conclusion in this paper, we propose the conisi model, which joints contrastive learning and inter-sentence interactive training objective for optimization. we propose to perform a sentence repetition operation on each sentence and then take the augmented pair as a positive instance based on contrastive learning, which alleviates the train-tuned bias of language models. we also propose the inter-sentence interactive objective, which guides the model to focus on fine-grained semantic information by feature interaction between sentences. moreover, we design three sentence construction strategies in the inter-sentence interactive objective. experimental results show our proposed conisi achieves substantial improvement over the previous state-of-the-art models. in the future, we will further explore more effective inter-sentence interactive way to enrich semantic information in sentence representation, and we hope to apply our approach to other downstream tasks such as machine translation.