An Unsupervised Sentence Embedding Method by Mutual Information Maximization (2020)

Abstract: BERT is inefficient for sentence-pair tasks such as clustering or semantic search as it needs to evaluate combinatorially many sentence pairs which is very time-consuming. Sentence BERT (SBERT) attempted to solve this challenge by learning semantically meaningful representations of single sentences, such that similarity comparison can be easily accessed. However, SBERT is trained on corpus with high-quality labeled sentence pairs, which limits its application to tasks where labeled data is extremely scarce. In this paper, we propose a lightweight extension on top of BERT and a novel self-supervised learning objective based on mutual information maximization strategies to derive meaningful sentence embeddings in an unsupervised manner. Unlike SBERT, our method is not restricted by the availability of labeled data, such that it can be applied on different domain-specific corpus. Experimental results show that the proposed method significantly outperforms other unsupervised sentence embedding baselines on common semantic textual similarity (STS) tasks and downstream supervised tasks. It also outperforms SBERT in a setting where in-domain labeled data is not available, and achieves performance competitive with supervised methods on various tasks.

## introduction bert-based pretrained language models @xcite have set new state-ofthe-art performance on various downstream nlp tasks. however, they are inefficient for sentencepair regression tasks such as clustering or semantic search because they need to evaluate combinatorially many sentence pairs during inference, which will result in a massive computational overhead. for example, finding the most similar pair in a collection of 10k sentences requires about 50 million ( 10k 2 ) inference computations with bert, which requires about 65 hours on a v100 gpu @xcite . much previous work attempted to address this problem by learning semantically meaningful representations for each sentence, such that similarity measures like cosine distance can be easily evaluated for sentence-pair regression tasks. the straightforward way to derive a fixed-size sentence embedding from bert-based models is to average the token representations at the last layer or using the output of the [cls] token. @xcite showed that both approaches yield rather unsatisfactory sentence embeddings. they proposed a model, sentence-bert (sbert), to further fine-tune bert on natural language inference (nli) tasks with labeled sentence pairs and achieved state-of-the-art performance on many semantic textual similarity tasks. however, such improvements are induced by high-quality supervision, and we find that their performance is degraded where labeled data of the target task is extremely scarce or the distribution of test set differs significantly from the nli dataset used for training. learning sentence representations in an unsupervised manner is a critical step to work with unlabeled or partially labeled dataset to address the aforementioned challenge @xcite @xcite @xcite . a common approach for unsupervised sentence representation learning is to leverage on self-supervision with large unlabeled corpus. for example, early methods explored various auto-encoders for sentence embedding @xcite . recent work such as skip-thought @xcite and fastsent @xcite assumed that a sentence is likely to have similar semantics to its context, and designed self-supervised objectives that encourage models to learn sentence representations by predicting contextual information. however, the performance of these models is far behind that of supervised learning ones on many tasks, which unveils an urgent need of better unsupervised sentence embedding methods. in this work, we propose a novel unsupervised sentence embedding model with light-weight feature extractor on top of bert for sentence encoding, and train it with a novel self-supervised learning objective. our model is not restricted by the availability of labeled data and can be applied to any domain of interest. instead of simply averaging bert token embeddings, we use convolutional neural network (cnn) layers with mean-over-time pooling that transform bert token embeddings to a global sentence embedding @xcite . moreover, we propose a novel self-supervised learning objective that maximises the mutual information (mi) between the global sentence embedding and all its local contexts embeddings, inspired by recent advances on unsupervised representation learning for images and graphs @xcite . our model is named info-sentence bert (is-bert). in is-bert, the representation of a specific sentence is encouraged to encode all aspects of its local context information, using local contexts derived from other input sentences as negative examples for contrastive learning. this learning procedure encourages the encoder to capture the unique information that is shared across all local segments of the specific input sentence while different from other inputs, leading to more expressive and semantically meaningful sentence embeddings. we evaluate our method on two groups of tasks -semantic textual similarity (sts) and senteval @xcite . empirical results show that is-bert significantly outperforms other unsupervised baselines on sts and senteval tasks. in addition, we show that is-bert substantially outperforms sbert in a setting where task-specific labeled data is not available. this demonstrates that is-bert has the flexibility to be applied to new domains without label restriction. finally, is-bert can achieve performance competitive with or even better than supervised learning methods in certain scenarios. ## model in this section, we outline a general model, the info-sentence bert (is-bert), for unsupervised sentence representation learning. we first give the problem formulation, then we present the details of our method and the corresponding neural network architecture. ## experiment following previous works @xcite , we conduct evaluation on two kinds of tasks: the results are good indicators of effectiveness on unsupervised tasks such as clustering and semantic search. we consider two groups of baselines. the first group corresponds to models trained with unlabeled sentences. this includes the unigram-tfidf mdoel, the paragraph vector model (le and mikolov, 2014), the sequential denoising auto-encoder (sdae) @xcite , the skipthought @xcite model and the fastsent @xcite ) model, all trained on the toronto book corpus @xcite consisted of 70m sentences. we also consider representing sentence with the average of glove embeddings, the average of the last layer representations of bert, and the [cls] embedding of bert, respectively. the second group consists of models trained on labeled nli data including infersent @xcite , universal sentence encoder (use) @xcite , and sentence bert (sbert-nli) @xcite . uncased-bert-base is used for all bert-related models including is-bert. ## conclusions in this paper, we proposed the is-bert model for unsupervised sentence representation learning with a novel mi maximization objective. is-bert outperforms all unsupervised sentence embedding baselines on various tasks and is competitive with supervised sentence embedding methods in certain scenarios. in addition, we show that sentence bert (sbert), the state-of-the-art supervised method, is problematic to apply to certain unsupervised tasks when the target domain significantly differs from the dataset it was trained on. is-bert achieves substantially better results in this scenario as it has the flexibility to be trained on the taskspecific corpus without label restriction. in the future, we want to explore semi-supervised methods for sentence embedding and its transferability across domains.