Are ELECTRA ’s Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity (2024)

Abstract: While BERT produces high-quality sentence embeddings, its pre-training computational cost is a significant drawback. In contrast, ELECTRA provides a cost-effective pre-training objective and downstream task performance improvements, but worse sentence embeddings. The community tacitly stopped utilizing ELECTRA’s sentence embeddings for semantic textual similarity (STS). We notice a significant drop in performance for the ELECTRA discriminator’s last layer in comparison to prior layers. We explore this drop and propose a way to repair the embeddings using a novel truncated model fine-tuning (TMFT) method. TMFT improves the Spearman correlation coefficient by over 8 points while increasing parameter efficiency on the STS Benchmark. We extend our analysis to various model sizes, languages, and two other tasks. Further, we discover the surprising efficacy of ELECTRA’s generator model, which performs on par with BERT, using significantly fewer parameters and a substantially smaller embedding size. Finally, we observe boosts by combining TMFT with word similarity or domain adaptive pre-training.

## introduction pre-trained language models (plms) have been a staple in nlp for years, leveraging self-supervised objectives to improve representations for downstream tasks. bert @xcite , one of the most widely used plms, uses a masked language modeling (mlm) objective for pre-training. the main drawbacks of mlm are the substantial compute cost due to the low masking rate, and the gap between the pre-training task and downstream tasks. to address these issues, @xcite introduce electra, which substitutes mlm with replaced token detection (rtd), achieving the same results as bert but with four times less compute. rtd uses a generator model that layer 0 @xcite a dog wags its tail @xcite token masking @xcite a dog [mask] its tail [sep] 1a 1b [cls] dog [sep] layer 0 mean pooling plm layer l layer l+1 layer l' 2 [cls] a dog wags its tail [sep] layer 0 plm layer l layer l+1 layer l' layer l' plm cross-entropy loss mean pooling figure 1 : a method for improving sentence embeddings with (2) tmft on sts. we apply mean pooling over the embeddings at layer l and fine-tune. one of the combinations can also be added for further improvement: (1a) tmft on word similarity, or (1b) dapt using mlm. while some transformer models excel across glue @xcite tasks, their sentence embedding quality may vary. sentence transformers often use plms pre-trained with a language modeling objective @xcite @xcite . in contrast, rtd pre-trained models demonstrate subpar performance in a biencoder setting @xcite , but pre-training such a model is more cost-effective. it is unclear whether this low performance is due to pre-training and whether the embeddings can be improved. to address this gap, we present a layer-wise study of electra's sentence embeddings. we examine the performance discrepancy between bert and electra's discriminator across various model sizes, languages, and tasks. we hypothesize the last few layers of the discriminator are too specialized for the pre-training task. following this, we present truncated model fine-tuning (tmft), shown in figure 1 , which utilizes the transformer up to layer l, followed by pooling and fine-tuning the whole truncated model. we show that the discriminator suffers from a drop in sts performance when fine-tuning the final layer's embeddings and that this drop is consistent across all inspected model sizes and languages. although we focus on sts, we expose a similar trend for two other tasks. when applying tmft on sts benchmark (stsb) @xcite , the embeddings significantly outperform the ones from the electra discriminator's last layer. moreover, tmft-obtained embeddings outperform the ones of bert fine-tuned on stsb up to but not including the eighth layer. further, we uncover the efficacy of the electra generator model, performing on par with bert while having a significantly smaller embedding size and substantially fewer parameters. finally, we propose two improvements of the basic tmft method: prior tmft on word similarity and prior domain adaptive pre-training (dapt) using mlm, shown in figure 1 . our contributions are: (1) a layer-wise analysis of electra's sentence embeddings for various model sizes, languages, and tasks; (2) a novel tmft method for repairing electra's embeddings that substantially improves performance on sts, paraphrase identification, and textual entailment tasks, and exposing the surprising efficacy of the generator model; (3) two additional techniques in combination with tmft for improving electra's embeddings even further on sts. 1 2 related work @xcite introduced siamese networks to transformers, motivating significant 1 we provide the source code for our work here: https: //github.com/ir2718/similarity-embedding-quality research on enhancing sentence embeddings. finetuning on an auxiliary task unrelated to sts has also been explored. @xcite first fine-tune on a textual entailment task, followed by fine-tuning on sts. dapt has also become a widely adopted method, improving performance in downstream tasks @xcite . similarly, there has been considerable interest in using different layers of a plm for sentence embeddings. @xcite assess the layerwise performance of transformer models and pooling methods on word similarity datasets. @xcite determine what combinations of hidden states perform the best for unsupervised sts, while @xcite extract layer-wise structural characteristics encoded in bert by probing. finally, @xcite explores layer-wise embedding anisotropy. two works that resemble ours the most are @xcite and @xcite . the former improves results by combining layer representations with dynamic programming, while we fine-tune the model from input embeddings up to a specific layer. @xcite , developed concurrently with our work, uses model and embedding truncation combined with fine-tuning and a novel loss function. however, this work does not address electra's performance drop. ## truncated model fine-tuning the usual approach for obtaining sentence embeddings is applying a pooling operation over the last layer's embeddings. we use mean pooling as it yields the best results, in line with previous work @xcite . the tmft method we propose for repairing electra's embeddings reduces to taking the l-th layer output followed by pooling and fine-tuning on the target task. for a sentence @xmath0, . . . , s n ) the encoder outputs a tensor @xmath1, where l ′ is the number of layers including the input embeddings (which we treat as a layer). we then apply mean pooling p over the l-th representation, p(e, l) = 1 n n @xmath2,n,: , where e l,n,: is the ddimensional output token embedding of layer l for token at position n. we apply this to both sentences, compare them using cosine similarity, and propagate the loss from layer l to the model input. 2 furthermore, we propose combining one supervised and one self-supervised method with tmft for performance gains: (1) plm fine-tuning on the word similarity task or (2) dapt using mlm. the intuition for the former is that word similarity is crucial for assessing sentence similarity, while the latter builds upon prior evidence that mlm-based plms work well in a bi-encoder @xcite . ## experiments and results we fine-tune each model on the stsb from glue. we also run the experiments on machine-translated versions of stsb in korean, german, and spanish (cf. appendix b). we choose these based on language-specific dataset and plm availability. we report the spearman correlation coefficient, suggested by @xcite . for word similarity experiments, we use word pairs present in the following datasets: rg-65 @xcite , wordsim-353 @xcite , simlex-999 @xcite , and simverb-3500 @xcite , with a random 70:15:15 train, validation, and test split. the mlm experiments are conducted on sentences from snli @xcite and multinli @xcite . we also experiment with paraphrase identification and textual entailment using the microsoft research paraphrase corpus (mrpc) and sentences involving compositional knowledge (sick) textual entailment dataset @xcite , respectively (cf. appendix a). for stsb, mrpc, and sick, we use cross-validation splits defined by the dataset authors. we use the following models from huggingface transformers @xcite : bert tiny, mini, small, medium, base, and large. for the elec-tra discriminator and generator, we use the small, base, and large models. to strengthen our findings on rtd-pre-trained models, we run experiments for the debertav3 model @xcite . for fine-tuning and pre-training, we use the adamw optimizer @xcite , a learning rate of 2e-5, and weight decay set to 1e-2. we apply gradient clipping to a max norm of 1.0. for fine-tuning on all tasks except word similarity, we use a batch size of 32 for 10 epochs. word similarity fine-tuning uses a batch size of 128 for 50 epochs. for dapt, we use a batch size of 32 with 8 gradient accumulation steps for 10 epochs with 0.15 masking probability. an exception is debertav3, with a batch size of 8 and 32 gradient accumulation steps due to memory constraints. all reported results are averaged across five seeds. test set spearman correlation coefficients on stsb correspond to the model with the highest spearman correlation coefficient on the validation set. for mrpc and sick, the test set f1 scores correspond to the models with the classification threshold optimized for the highest validation set f1 score. ## conclusion we analyze electra's sentence embeddings for sts and two other tasks, comparing them to a bert baseline on the stsb dataset. our proposed truncated model fine-tuning method significantly improves the discriminator. the generator model matches bert's performance while significantly reducing the number of parameters and producing smaller embeddings.