Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders (2021)

Abstract: Previous work has indicated that pretrained Masked Language Models (MLMs) are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective lexical and sentence encoders even without any additional data, relying simply on self-supervision. We propose an extremely simple, fast, and effective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20-30 seconds with no access to additional external knowledge. Mirror-BERT relies on identical and slightly modified string pairs as positive (i.e., synonymous) fine-tuning examples, and aims to maximise their similarity during “identity fine-tuning”. We report huge gains over off-the-shelf MLMs with Mirror-BERT both in lexical-level and in sentence-level tasks, across different domains and different languages. Notably, in sentence similarity (STS) and question-answer entailment (QNLI) tasks, our self-supervised Mirror-BERT model even matches the performance of the Sentence-BERT models from prior work which rely on annotated task data. Finally, we delve deeper into the inner workings of MLMs, and suggest some evidence on why this simple Mirror-BERT fine-tuning approach can yield effective universal lexical and sentence encoders.

## introduction transfer learning with pretrained masked language models (mlms) such as bert @xcite and roberta @xcite has been widely successful in nlp, offering unmatched performance in a large number of tasks @xcite . despite the wealth of semantic knowledge stored in the mlms @xcite , they do not produce high-quality lexical and sentence embeddings when used off-the-shelf, without further figure 1: illustration of the main concepts behind the proposed self-supervised mirror-bert method. the same text sequence can be observed from two additional "views": 1) by performing random span masking in the input space, and/or 2) by applying dropout (inside the bert/roberta mlm) in the feature space, yielding identity-based (i.e., "mirrored") positive examples for fine-tuning. a contrastive learning objective is then applied to encourage such "mirrored" positive pairs to obtain more similar representations in the embedding space relatively to negative pairs. task-specific fine-tuning @xcite . in fact, previous work has shown that their performance is sometimes even below static word embeddings and specialised sentence encoders @xcite in lexical and sentence-level semantic similarity tasks @xcite @xcite . in order to address this gap, recent work has trained dual-encoder networks on labelled external resources to convert mlms into universal language encoders. most notably, sentence-bert (sbert, reimers and gurevych 2019) further trains bert and roberta on natural language inference @xcite and sentence similarity data @xcite to obtain high-quality universal sentence embeddings. recently, sapbert @xcite self-aligns phrasal representations of the same meaning using synonyms extracted from the umls @xcite , a large biomedical knowledge base, obtaining lexical embeddings in the biomedical domain that reach state-of-the-art (sota) performance in biomedical entity linking tasks. however, both sbert and sapbert require annotated (i.e., human-labelled) data as external knowledge: it is used to instruct the model to produce similar representations for text sequences (e.g., words, phrases, sentences) of similar/identical meanings. in this paper, we fully dispose of any external supervision, demonstrating that the transformation of mlms into universal language encoders can be achieved without task-labelled data. we propose a fine-tuning framework termed mirror-bert, which simply relies on duplicating and slightly augmenting the existing text input (or their representations) to achieve the transformation, and show that it is possible to learn universal lexical and sentence encoders with such "mirrored" input data through self-supervision (see fig. 1 ). the proposed mirror-bert framework is also extremely efficient: the whole mlm transformation can be completed in less than one minute on two 2080ti gpus. our findings further confirm a general hypothesis from prior work @xcite @xcite ) that finetuning exposes the wealth of (semantic) knowledge stored in the mlms. in this case in particular, we demonstrate that the mirror-bert procedure can rewire the mlms to serve as universal language encoders even without any external supervision. we further show that data augmentation in both input space and feature space are key to the success of mirror-bert, and they provide a synergistic effect. ## mirror-bert: methodology mirror-bert consists of three main parts, described in what follows. first, we create positive pairs by duplicating the input text ( §2.1). we then further process the positive pairs by simple data augmentation operating either on the input text or on the feature map inside the model ( §2.2). finally, we apply standard contrastive learning, 'attracting' the texts belonging to the same class (i.e., positives) while pushing away the negatives ( §2.3). ## experimental setup evaluation tasks: lexical. we evaluate on domain-general and domain-specific tasks: word similarity and biomedical entity linking (bel). for the former, we rely on the multi-simlex evaluation set @xcite : it contains human-elicited word similarity scores for multiple languages. for the latter, we use ncbi-disease (ncbi, dogan et al. 2014), bc5cdr-disease, bc5cdr-chemical (bc5-d, bc5-c, @xcite , askapatient @xcite and cometa (stratified-general split, basaldella et al. 2020) as our evaluation datasets. the first three datasets are in the scientific domain (i.e., the data have been extracted from scientific papers), while the latter two are in the social media domain (i.e., extracted from online forums discussing health-related topics). we report spearman's rank correlation coefficients (ρ) for word similarity; accuracy @1/@5 is the standard evaluation measure in the bel task. evaluation tasks: sentence-level. evaluation on the intrinsic sentence textual similarity (sts) task is conducted on the standard semeval 2012-2016 datasets @xcite @xcite @xcite , sts benchmark @xcite , sick-relatedness (sick-r, @xcite for english; sts semeval-17 data is used for spanish and arabic @xcite , and we also evaluate on russian sts. 3 we report spearman's ρ rank correlation. evaluation in the question-answer entailment task is conducted on qnli @xcite . it contains 110k english qa pairs with binary entailment labels. 4 evaluation tasks: cross-lingual. we also assess the benefits of mirror-bert on cross-lingual representation learning, evaluating on cross-lingual word similarity (clws, multi-simlex is used) and bilingual lexicon induction (bli). we rely on the standard mapping-based bli setup @xcite , and training and test sets from @xcite , reporting accuracy @1 scores (with csls as the word retrieval method, @xcite . ## related work self-supervised text representations have a large body of literature. here, due to space constraints, we provide a highly condensed summary of the most related work. even prior to the emergence of large pretrained lms (plms), most representation models followed the distributional hypothesis @xcite and exploited the co-occurrence statistics of words/phrases/sentences in large corpora @xcite @xcite @xcite . recently, declutr @xcite follows the distributional hypothesis and formulates sentence embedding training as a contrastive learning task where span pairs sampled from the same document are treated as positive pairs. very recently, there has been a growing interest in using individual raw sentences for selfsupervised contrastive learning on top of plms. @xcite explore input augmentation techniques for sentence representation learning with contrastive objectives. however, they use it as an auxiliary loss during full-fledged mlm pretraining from scratch @xcite . in contrast, our post-hoc approach offers a lightweight and fast self-supervised transformation from any pretrained mlm to a universal language encoder at lexical or sentence level. @xcite use two distinct models to produce two views of the same text, where we rely on a single model, that is, we propose to use dropout and random span masking within the same model to produce the two views, and demonstrate their synergistic effect. our study also explores word-level and phrase-level representations and tasks, and to domain-specialised representations (e.g., for the bel task). simcse @xcite , a work concurrent to ours, adopts the same contrastive loss as mirror-bert, and also indicates the importance of data augmentation through dropout. however, they do not investigate random span masking as data augmentation in the input space, and limit their model to general-domain english sentence representations only, effectively rendering simcse a special case of the mirror-bert framework. other concurrent papers explore a similar idea, such as self-guided contrastive learning @xcite , consert @xcite , and bsl @xcite , inter alia. they all create two views of the same sentence for contrastive learning, with different strategies in feature extraction, data augmentation, model updating or choice of loss function. however, they offer less complete empirical findings compared to our work: we additionally evaluate on (1) lexical-level tasks, (2) tasks in a specialised biomedical domain and (3) cross-lingual tasks. ## conclusion we proposed mirror-bert, a simple, fast, selfsupervised, and highly effective approach that transforms large pretrained masked language models (mlms) into universal lexical and sentence encoders within a minute, and without any external supervision. mirror-bert, based on simple unsupervised data augmentation techniques, demonstrates surprisingly strong performance in (wordlevel and sentence-level) semantic similarity tasks, as well as on biomedical entity linking. the large gains over base mlms are observed for different languages with different scripts, and across diverse domains. moreover, we dissected and analysed the main causes behind mirror-bert's efficacy.