Universal Sentence Representation Learning with Conditional Masked Language Model (2021)

Abstract: This paper presents a novel training method, Conditional Masked Language Modeling (CMLM), to effectively learn sentence representations on large scale unlabeled corpora. CMLM integrates sentence representation learning into MLM training by conditioning on the encoded vectors of adjacent sentences. Our English CMLM model achieves state-of-the-art performance on SentEval, even outperforming models learned using supervised signals. As a fully unsupervised learning method, CMLM can be conveniently extended to a broad range of languages and domains. We find that a multilingual CMLM model co-trained with bitext retrieval (BR) and natural language inference (NLI) tasks outperforms the previous state-of-the-art multilingual models by a large margin, e.g. 10% improvement upon baseline models on cross-lingual semantic search. We explore the same language bias of the learned representations, and propose a simple, post-training and model agnostic approach to remove the language identifying information from the representation while still retaining sentence semantics.

## introduction sentence embeddings map sentences into a vector space. the vectors capture rich semantic information that can be used to measure semantic textual similarity (sts) between sentences or train classifiers for a broad range of downstream tasks @xcite @xcite @xcite . stateof-the-art models are usually trained on supervised tasks such as natural language inference @xcite , or with semi-structured data like question-answer pairs @xcite and translation pairs @xcite ., * work done during internship at google research. in this paper, we explore an unsupervised approach, called conditional masked language modeling (cmlm), to effectively learn sentence representations from large scale unlabeled corpora. the cmlm model architecture is illustrated in fig. 1 , which integrates sentence representation learning into mlm training by conditioning on sentence level representations produced by adjacent sentences. the model therefore needs to learn effective sentence representations in order to perform good mlm. since cmlm is fully unsupervised, it can be easily extended to new languages. we explore cmlm for both english and multilingual sentence embeddings for 100+ languages. our english cmlm model achieves state-of-the-art performance on senteval @xcite , even outperforming models learned using (semi-)supervised signals. moreover, models training on the english amazon review data using our multilingual vectors exhibit strong multilingual transfer performance on translations of the amazon review evaluation data to french, german and japanese, outperforming existing multilingual sentence embedding models @xmath0. we further extend the multilingual cmlm to cotrain with parallel text (bitext) retrieval task, and finetune with cross-lingual natural language infer-"life is a box of chocolates." transformer encoder pooling sentence vector 𝑣 𝑣 ! 𝑣 " 𝑣 # . . . . transformer encoder "you never [mask] what you're [mask] get ." projection 𝑃(𝑣) 𝑣 ! 𝑣 " 𝑣 # . . . know, gonna ence (nli) data, inspired by the success of prior work on multitask sentence representation learning @xcite @xcite and nli learning @xcite . we achieve performance 3.6% better than the previous state-of-the-art multilingual sentence representation model (see details in section 4.2). on cross-lingual semantic search task, our model outperforms baseline models by 10% on average over 36 languages. language agnostic representations require semantically similar cross-lingual pairs to be closer in representation space than unrelated same-language pairs @xcite . while we find our original sentence embeddings do have a bias for same language sentences, we discover that removing the first few principal components of the embeddings eliminates the self language bias. the rest of the paper is organized as follows. section 2 describes the architecture for cmlm unsupervised learning. in section 3 we present cmlm trained on english data and evaluation results on senteval. in section 4 we apply cmlm to learn sentence multilingual sentence representations. multitask training strategies on how to effectively combining cmlm, bitext retrieval and crosslingual nli finetuning are explored. in section 5, we investigate self language bias in multilingual representations and propose a simple but effective approach to eliminate it. the pre-trained models are released at https://tfhub.dev/s?@xmath1. ## conditional masked language modeling we introduce conditional masked language modeling (cmlm) as a novel architecture for combin-ing next sentence prediction with mlm training. by "conditional", we mean the mlm task for one sentence depends on the encoded sentence level representation of the adjacent sentence. this builds on prior work on next sentence prediction that has been widely used for learning sentence level representations @xcite @xcite ), but has thus far produced poor quality sentence embeddings within bert based models using mlm loss @xcite . while existing mlms like bert include next sentence prediction tasks, they do so without any inductive bias to try to encode the meaning of a sentence within a single embedding vector. we introduce a strong inductive bias for learning sentence embeddings by structuring the task as follows. given a pair of ordered sentences, the first sentence is fed to an encoder that produces a sentence level embedding. the embedding is then provided to an encoder that conditions on the sentence embedding in order to better perform mlm prediction over the second sentence. this is notably similar to skip-thought @xcite , but replaces the generation of the complete second sentence with the mlm denoising objective. it is also similar to t5's mlm inspired unsupervised encode-decoder objective @xcite , with the second encoder acting as a sort of decoder given the representation produced for the first sentence. our method critically differs from t5's in that a sentence embedding bottleneck is used to pass information between two model components and in that the task involves denoising a second sentence when conditioning on the first rather than denoising a single text stream. fig. 1 illustrates the architecture of our model. the first sentence s 1 is tokenized and input to a transformer encoder and a sentence vector @xmath2. 1 the sentence vector v is then projected into n spaces with one of the projections being the identity mapping, i.e. v @xmath3. here we use a three-layer mlp as the projection p (•). details of p (•) are available in the supplementary material. one motivation for the projections of s 1 is that mlm of s 2 then can attend to various representations of s 1 instead of only 1. in section 5.1, we explore various different configurations of cmlm, including the number of projection spaces n . the second sentence s 2 is then masked following the procedure described in the original bert paper, including random replacement and the use of unchanged tokens. the second encoder shares the same weights with the encoder used to embed s 1 2 . tokens in the masked s 2 are first converted into token vectors. the masked language modeling of s 2 depends on s 1 such that the process involves cross-attention between s 2 token vectors and v p . in practice, this is implemented by concatenating token embeddings of s 2 with v p 3 . other implementations are also experimented (see section 5.1) and we empirically find concatenation works the best. the concatenated representations are then provided to the transformer encoder to predict the masked tokens in s 2 . at inference time, we keep the first encoding module and discard the subsequent mlm prediction. similar to skip-thought, cmlm trains the encoder to produce sentence embeddings useful for predicting material in the adjunct sentences. cmlm adapts this existing idea to mlm training. appending multiple projections performs well due to fine-grained attention between tokens and the different views of the sentence embeddings. note that cmlm differs from skipthought in the following aspects: (a) skipthought relies on an extra decoder network while cmlm only has the encoder. (b) skipthought predicts the entire sentence while cmlm predicts masked tokens only so the 1 one can equivalently choose other pooling methods, such as max pooling or use the vector output corresponding to a special token position such as the [cls] token. 2 the dual-encoder sharing encoder weights for different inputs can be also referred as "siamese encoder" ## learning english sentence representations with cmlm for training english sentence encoders with cmlm, we use three common crawl dumps. the data are filtered by a classifier which detects whether a sentence belongs to the main content of the web page or not. we use wordpiece tokenization and the vocabulary is the same as public english uncased bert. in order to enable the model to learn bidirectional information, for two consecutive sequences s 1 and s 2 , we swap their order for 50% of the time. this order-swapping process echos with the preceding and succeeding sentences prediction in skip-thought @xcite . the length of s 1 and s 2 are set to be 256 tokens (the maximum length). the number of masked tokens in s 2 are 80 (31.3%), moderately higher than classical bert. this change in the ratio of masked tokens is to make the task more challenging, due to the fact that in cmlm, language modeling has access to extra information from adjacent sentences. we train with batch size of 2048 for 1 million steps. the optimizer is lamb @xcite with learning rate of 10 -3 , β 1 = 0.9, β 2 = 0.999, warm-up in the first 10,000 steps and linear decay afterwards. we explore two transformer configurations same as in the original bert paper, i.e., base and large. the number of projections n is 15 by experimenting with multiple choices. ## learning multilingual sentence representations with cmlm as a fully unsupervised method, cmlm can be conveniently extended to multilingual modeling even for less well resourced languages. learning good multilingual sentence representations is more challenging than monolingual ones, especially when attempting to capture the semantic alignment between different languages. as cmlm does not explicitly address cross-lingual alignment, we explore several modeling approaches besides cmlm: (1) co-training cmlm with a bitext retrieval task; ## conclusion we present a novel sentence representation learning method conditional masked language modeling (cmlm) for training on large scale unlabeled corpus. cmlm outperforms the previous stateof-the-art english sentence embeddings models, including those trained with (semi-)supervised signals. for multilingual representations, we discover that co-training cmlm with bitext retrieval and cross-lingual nli finetuning achieves state-of-theart performance. we also find that multilingual representations have the same language bias and principal component removal can eliminate the bias by separating language identity information from semantics.