Bootstrapped Unsupervised Sentence Representation Learning (2021)

Abstract: As high-quality labeled data is scarce, unsupervised sentence representation learning has attracted much attention. In this paper, we propose a new framework with a two-branch Siamese Network which maximizes the similarity between two augmented views of each sentence. Specifically, given one augmented view of the input sentence, the online network branch is trained by predicting the representation yielded by the target network of the same sentence under another augmented view. Meanwhile, the target network branch is bootstrapped with a moving average of the online network. The proposed method significantly outperforms other state-of-the-art unsupervised methods on semantic textual similarity (STS) and classification tasks. It can be adopted as a post-training procedure to boost the performance of the supervised methods. We further extend our method for learning multilingual sentence representations and demonstrate its effectiveness on cross-lingual STS tasks. Our code is available at https://github.com/yanzhangnlp/BSL .

## introduction sentence representation learning aims to map sentences into vectors that capture rich semantic information. among previous approaches, supervised methods achieve state-of-the-art performance by leveraging quality sentence labels. for example, the recently proposed model sentence-bert (sbert) @xcite fine-tunes a siamese bert network on natural language inference (nli) tasks with labeled sentence pairs. it achieves state-of-the-art results on multiple semantic textual similarity (sts) tasks. however, such performance is mostly induced by high-quality supervision, while labeled data are difficult and ex-pensive to obtain in practice. @xcite showed that sbert generalizes poorly on target tasks that differ significantly from nli on which sbert is fine-tuned. many unsupervised methods learn sentence representations by optimizing over various selfsupervised learning (ssl) objectives on a largescale unlabeled corpus. early works often use auto-encoders @xcite or next-sentence prediction @xcite for sentence representation learning. recently, more efforts have been devoted to representation learning with transformer-based networks using masked language modeling (mlm). however, transformer-based methods do not directly produce meaningful sentence representations. instead, significant supervised fine-tuning steps with labeled data are commonly required to form good representations @xcite . recently, @xcite and @xcite proposed novel transformer-based frameworks to directly learn sentence representations from an unlabeled corpus, which even exhibited competitive performance to the supervised counterparts on some tasks. however, @xcite required long text during training while the contrastive learning strategy employed by @xcite need a careful treatment of negative pairs. more important, there is still great room for improvement in terms of the quality of learned sentence representations. in this paper, we introduce bootstrapped sentence representation learning (bsl), a simple and lightweight framework that directly learns sentence representations without supervised finetuning. our work is inspired by the recent success of siamese networks @xcite for unsupervised visual representation learning @xcite @xcite , especially the byol framework @xcite . these models employed various kinds of unsupervised learning objectives to maximize the similarity between two augmented views of each image, yielding performance on par with supervised methods. unlike contrastive learning-based methods, which demand a carefully negative sampling process and large batch sizes, byol could achieve great performance without negative pairs. the proposed bsl works as follows. given an input sentence, we first construct two augmented views through back-translation. these two views are simultaneously fed into the two branches of the siamese network, i.e., an online network and a target network following the terminology in @xcite . in particular, the online and target networks use two pre-trained transformer networks with the same structure, e.g., bert, to encode the two views separately. during learning, the online network is trained to predict the representation of the other augmented view generated by the target network, and its parameters are updated by minimizing a predefined prediction loss. as for the target network, we apply a stop-gradient strategy @xcite and update it with a weighted moving average of the online network. hence, the outputs of the target network are iteratively bootstrapped to serve as targets, enabling enhanced representation learning of the online network while avoiding trivial solutions. our method is evaluated through extensive experiments. empirical results show that bsl significantly outperforms strong unsupervised baselines on a standard suite of sts and classification tasks from the senteval benchmark @xcite . we also demonstrate that bsl can serve as an effective post-training approach to boost the performance of the state-of-the-art supervised sbert model. we further extend our method for learning multilingual sentence representations and demonstrate that it is able to outperform strong multilingual baselines on cross-lingual sts tasks under both unsupervised and supervised settings. detailed analysis of a few factors that could affect the model performance is provided as well to motivate future research. ## experiment design we conduct various experiments to evaluate the effectiveness of the proposed method. following prior works @xcite , our major evaluations are conducted on the semantic textual similarity (sts) tasks and the classification tasks with the senteval toolkit @xcite . to demonstrate the flexibility of the proposed method, we further extend it for learning multilingual sentence representations and evaluate it on cross-lingual sts tasks. implementation the mlp contains three linear layers. given an input vector of dimension d, the output dimensions of the three layers are @xmath0, where k is a hyperparameter controlling the hidden size. batch normalization and rectified linear units (relu) are applied to the intermediate linear layers. we use bert-base or robertabase to initialize the online and target networks in monolingual settings. hyperparameter we tune learning rate, batch size, momentum δ, and the hyperparameter k on the development set of sts-b @xcite . for all unsupervised experiments, we set learning rate to 5e-4, momentum to 0.999, and k to 8. adam (kingma and ba, 2015) is used as the optimizer. 2 baselines under a unsupervised learning setting, we compare to the unigram-tfidf model, the sequential denoising auto-encoder (sdae) @xcite , the skipthought @xcite and fastsent @xcite . those models are all trained on the toronto book corpus with 70m sentences @xcite . we also compare with sentence representations obtained with the average of glove embeddings (glove avg.), the average of bert embeddings (bert avg.), and the [cls] representation of bert (bert [cls]), as those are common ways to get sentence-level representations. we compare with bert-flow @xcite , a recent method that transforms the representation obtained by bert to an isotropic gaussian distribution. in addition, we compare with two unsupervised bert fine-tuning methods. the first is to finetune bert with masked language modeling (mlm) objective (bert-mlm) @xcite . the second is is-bert @xcite ) which employs a mutual information maximization objective for fine-tuning bert. we denote our model initialized by bert-base (roberta-base) as bsl-bert (bsl-roberta). under a supervised learning setting, we compared to infersent @xcite , universal sentence encoder (use) @xcite , and sentence bert/roberta (sbert/sroberta) @xcite , which are all trained on the snli and multinli datasets. to adapt bsl to a supervised learning setting, we first train a sbert (sroberta) model and then use the learned weights to initialize the online and target networks of bsl and perform bsl training. we denote this model variant as bsl-sbert (bsl-sroberta). ## conclusion in this paper, we propose bsl for unsupervised sentence representation learning. the experimental results demonstrate that our method could significantly outperform the state-of-the-art unsupervised methods and it can be further extended for learning multilingual sentence representations. in future work, we expect both theoretically advance of siamese networks for representation learning, e.g., why stop-gradient works so well and how to further improve the updating dynamics, as well as specifically designated ideas for nlp, e.g., augmentation or learning objectives.