S EMANTIC R E -T UNING WITH C ONTRASTIVE T ENSION
∗ Evangelia Gogoulou
Erik Ylip¨a¨a RISE { firstname.lastname } @ri.se
A BSTRACT
Extracting semantically useful natural language sentence representations from pre-trained deep neural networks such as Transformers remains a challenge.
We ﬁrst demonstrate that pre-training objectives impose a signiﬁcant task bias onto the ﬁnal layers of models, with a layer-wise survey of the Semantic Textual Similarity (STS) correlations for multiple common Transformer language models.
We then propose a new self-supervised method called Contrastive Tension (CT) to counter such biases.
CT frames the training objective as a noise-contrastive task between the ﬁnal layer representations of two independent models, in turn making the ﬁnal layer representations suitable for feature extraction.
Results from multiple common unsupervised and supervised STS tasks indicate that CT outperforms previous State Of The Art (SOTA), and when combining CT with supervised data we improve upon previous SOTA results with large margins.
I NTRODUCTION
Representation learning concerns the pursuit of automatically learning representations of data that are useful for future extraction of information (Bengio et al., 2013).
Recent work has predominantly been focused on training and extracting such representations from various deep neural architectures.
However, as these deep models are mostly trained via error minimization of an objective function applied to the ﬁnal layers (Rumelhart et al., 1988), features residing in layers close to the objective function will be task-speciﬁc Yosinski et al.
Therefore, to reduce the representation’s bias towards the objective function it is common to discard one or several of the ﬁnal layers or alternatively consider features of other intermediate layers, as with AutoEncoders (Rumelhart et al., 1986).
One domain where this issue is particularly striking is learning semantic sentence embeddings with deep Transformer networks (Vaswani et al., 2017) pre-trained towards some language modeling task.
Although utilizing pre-trained Transformer models such as BERT, XLnet, ELECTRA and
GPT-2(Devlin et al., 2019; Yang et al., 2019; Clark et al., 2020; Brown et al., 2020) has become the dominant approach within the ﬁeld of Natural Language Processing (NLP), with current State Of
The Art (SOTA) results in basically all NLP tasks belonging to ﬁne-tuned versions of such models, it has been shown that simply extracting features from the layers of such models does not produce competitive sentence embeddings (Reimers & Gurevych, 2019; Liu et al., 2019a).
Our interpretation of this phenomenon, which we will demonstrate in this paper, is that the currently used language modeling objectives enforce a task-bias at the ﬁnal layers of the Transformer, and that this bias is not beneﬁcial for the learning of semantic sentence representations.
Reimers & Gurevych (2019) propose to solve this by pooling a ﬁxed size sentence embedding from the ﬁnal Transformer layer and ﬁne-tune towards a Natural Language Inference (NLI) task, an approach that when applied to Transformers is known as Sentence-BERT (or S-BERT in short).
While
Hill et al.
(2016a) empirically show that ﬁne-tuning language models towards NLI data yields good results on Semantic Textual Similarity (STS), there exists no convincing argument for why NLI is preferred over other tasks.
Hence, it is unclear whether the impressive improvements of S-BERT are to be mainly attributed to the NLI task itself, or if this merely trains the model to output sentence embeddings, in turn exposing the semantics learned during pre-training.
Since NLI requires labeled data, it would be highly valuable if an alternative method that requires no such labels was possible.
We therefore propose a fully self-supervised training objective that aims to remove the bias posed by the pre-training objective and to encourage the model to output semantically useful sentence representations.
Our method trains two separate language models on the task of maximizing the dot product between the two models’ representations for identical sentences, and minimizing the dot product between the models’ representations for different sentences.
When applied to pre-trained
BERT models, our method achieves SOTA results for multiple unsupervised STS tasks, and when applied to the S-BERT model it outperforms previous SOTA by a clear margin.
To further bolster the robustness of our method, we demonstrate that CT drastically improves STS scores for various models, across multiple languages.
Additionally, we contribute with a layer-wise STS survey for the most common Transformer-based language models, in which we ﬁnd great variability in performance between different architectures and pre-training objectives.
Finally, by introducing an alteration to the supervised regression task of
S-BERT, we are able to improve upon the supervised STS embedding results for all tested models.
In summary, the main contributions of our paper are as follows:
A novel self-supervised approach for learning sentence embeddings from pre-trained language models.
Analytical results of the layer-wise STS performance for commonly used language models.
An improvement to the supervised regression task of S-BERT that yields a higher performance for all tested models.
Code and models is available at Github.com/FreddeFrallan/Contrastive-Tension
R ELATED W ORK
Where earlier work for learning sentence embeddings focused on the composition of pre-trained word embeddings (Le & Mikolov (2014);
Wieting et al.
; Arora et al. (2016)), recent work has instead favored extracting features from deep neural networks.
The training methods of such networks can be divided into supervised and self-supervised.
A systematic comparison of preTransformer sentence embedding methods is available in the works of Hill et al.
(2016b).
Self-supervised methods typically rely on the assumption that sentences sharing similar adjacent sentences, have similar meaning.
Utilizing this assumption, Kiros et al.
introduced SkipThoughts that trains an encoder-decoder to reconstruct surrounding sentences from an encoded passage.
Logeswaran & Lee (2018) proposed QuickThoughts that instead frames the training objective as a sentence context classiﬁcation task.
Recently, and still under peer-review, Giorgi et al.
(2020) proposed DeCLUTR that uses a setup similar to QuickThoughts, but allow positive sentences to be overlapping or subsuming (one being a subsequence of the other), which further improves results.
Supervised methods utilize labeled datasets to introduce a semantic learning signal.
As the amount of explicitly labeled STS data is very limited, supervised methods often rely on various proxy tasks where more labeled data is available.
Conneau et al.
(2017) introduced InferSent that learns sentence embeddings via a siamese BiLSTM trained on NLI data.
The Universal Sentence Encoder (USE) of Cer et al.
(2018) is a Transformer encoder trained with both unlabeled data and labeled NLI data.
S-BERT by Reimers & Gurevych (2019) adopts the training objective of InferSent but instead applies pre-trained BERT models.
Finally, Wang & Kuo (2020) recently proposed S-BERT-WK, an extension to S-BERT that further increases the performance by subspace analysis of the model’s layer-wise word features.
Recently, Grill et al.
(2020) introduced the self-supervised BYOL framework that attain useful image representations, comparable with previous supervised methods.
Although their method also utilizes two untied dual networks, the main training objective and the underlying motivation for this differ greatly.
Where BYOL train using solely positive samples generated via data augmentation, our method mainly aims to dissipate negative examples and relies on two networks in order to stabilize the training process.
To the best of our knowledge, our work is the ﬁrst that suggests learning sentence representations by removing the bias imposed from the pre-training objective.
L AYER - WISE S TUDY OF T RANSFORMER M ODELS
Previous work analyzing the downstream applicability of layer-wise features in Transformer model reports similar trends of performance increasing until the middle layers before decreasing towards the ﬁnal layers.
Merchant et al.
(2020) found the best suited features for linguistic tasks such as entity typing and relation classiﬁcation reside in the intermediate layers of BERT, and Chen et al.
(2020) found the most useful representations for image classiﬁcation in the intermediate layers of
Image-GPT.
We contribute with a layer-wise study of the semantic quality of the sentence representations found in a selected number of common Transformer architectures.
Following the approach of S-BERT, we generate sentence embeddings by mean pooling over the word-piece features of a given layer.
These sentence embeddings are directly evaluated towards the STS-b test (Cer et al., 2017), without any additional training, from which we report the Spearman correlation between the cosine similarity of the embeddings and the manually collected similarity scores.
The test partition of the dataset contains 1,379 sentence pairs, with decimal human similarity scores ranging from 0.0 (two sentences having completely different meanings) to 5.0 (two sentences have identical meaning).
Figure 1 shows the results for BERT, Electra, XLNet and GPT-2, with results for additional models in appendix B.4.
Although the different models display different layer-wise patterns, a common theme is that it is not obvious where to extract features for semantic sentence embeddings; the worst-performing representations are often found in the layers close to the objective function, with the exception of RoBerta base (Liu et al., 2019b).
Considering the discrepancy between BERT and
Electra which share an almost identical architecture but differ drastically in their pre-training objectives, it is clear that the semantic quality of a model’s sentence representations is heavily impacted by the choice of pre-training objective.
is a redundant indicator of the Spearman correlation and the value 50 is included for visual comparison.
M ETHOD
To counter the negative trend found in Section 3, where the lacking STS performance of the sentence representations in the ﬁnal layers became apparent, we deﬁne a training objective meant to encourage the model to retain a semantically distinguishable sentence representation until the ﬁnal layer.
We name this method Contrastive Tension (CT), where two independent models, with identically initialized weights, are set to maximise the dot product between their sentence representations for identical sentences, and minimize the dot product for their sentence representations of differing sentences.
Hence, the CT objective is deﬁned as:
[Equation]
Where f 1 and f 2 are two independently parameterized models that given a sentence s produces a ﬁxed size vector representation and where σ refers to the Logistic function.
Following the works of Reimers & Gurevych (2019), we generate ﬁxed size sentence representations by mean pooling over the features in the ﬁnal layer of pre-trained transformer models.
Training data is randomly generated from a given corpus, where for each randomly selected sentence s , K negative sentences are sampled to generate K + 1 training samples by pairing s with the negative sentences and copying s into an identical sentence pair.
This yields one positive training sample and
K negative training samples.
We include the K + 1 training samples in the same batch and always use f 2 to embed the K negative sentences (See Appendix A.1 for a visual example).
Our approach for generating negative samples is based on the assumption that two randomly selected sentences are very likely to be semantically dissimilar.
As the models are initialized with identical weights, the CT objective creates a tension between having the two models retain similar representations for identical sentences, at the same time as the two models are encouraged to distinguish their representations for differing sentences.
Our intuition is that this creates a training dynamic where the two models acts as smooth anchors to each other, where the tension to remain synchronized mitigates the downsides of simply distancing the embeddings of differing sentences.
This makes CT a nondestructive method for distinguishing the sentence embeddings of non semantically similar sentences.
E XPERIMENTS
Unless stated otherwise, the following set of hyperparameters is applied when using CT throughout all experiments: Training data is randomly sampled from English Wikipedia (See Appendix C.2), where we collect K = 7 negative sentence pairs for each positive sentence pair.
The batch size is set to 16, which results in every batch having 2 positive sentence pairs and 14 negative sentence pairs.
We apply an RMSProp optimizer (Hinton, 2012) with a ﬁxed learning rate schedule that decreases from 1 e − 5 to 2 e − 6 (Appendix A.3).
To showcase the robustness and unsupervised applicability of
CT, we strictly perform 50,000 update steps before evaluating, and for all unsupervised tasks we report results for the worst-performing of the two models used in the CT setup.
The experiment section follows the model naming convention elaborated upon in A.2, which describes the order and what training objectives that has been applied to a model.
There exists a clear discrepancy between previously reported STS scores for various methods and models.
To improve upon this state of confusion we perform all evaluation with the SentEval package (Conneau & Kiela, 2018), to which we provide code and models for full reproducability of all tested methods.
A Discussion regarding our experience with trying to reproduce previous work is available in Appendix A.4.
A comprehensive list of all used model checkpoints is available in
Appendix C.
[Table]
5. U NSUPERVISED STS
Table 1 shows the results for CT when evaluated on the unsupervised English STS tasks of Agirre et al.
The non-ﬁne-tuned BERT models perform the worst out of all considered models, with a decrease in performance as the size of the BERT models increase.
CT clearly improves the results for all BERT based models and outperforms previous methods.
When CT is applied to the supervised S-BERT models, it sets a new SOTA with a large margin ( 3 . 03 Spearman Points).
Applying CT to BERT-Distil and S-BERT-Distil produces models that outperform S-BERT-Large while having 81% fewer parameters.
To further investigate the training dynamic of CT, we record the layer-wise STS performance for
BERT and XLNet throughout the CT training process, by unsupervised evaluation on the STS-b test set.
Figure 2 depicts the observed progression trends, and although the STS performance of the models’ ﬁnal layers differs greatly before ﬁne-tuning, with XLNet showing drastically worse performance, both models clearly beneﬁt from the CT training task.
For both models, CT mainly affects the STS performance for the latter layers, as is to be expected from the low learning rate.
5. S UPERVISED STS
Reimers & Gurevych (2019) proposed an supervised STS regression task which directly targets the cosine similarity between sentence embeddings, creating regression labels by linearly mapping the human similarity scores to the range [0 , 1] .
However, as evaluation of STS related tasks uses the
Pearson and Spearman correlation the range to which the cosine similarity labels are linearly mapped to is arbitrary.
Hence, we propose to ﬁrst investigate the spread within the models embedding space to ﬁnd a model speciﬁc linear mapping of the regression labels that imposes less change to the current embedding space.
We investigate the STS spread of a model’s embedding space by dividing the STS-b training data by their labels into 20 buckets, and measuring the mean cosine similarity between the sentence pairs within each respective bucket.
As the STS-b data is labeled in the range [0 , 5] , each bucket covers a range of 5 / 20 = 0 .
Thus the lowest bucket contains all training samples with labels between [0 , 0 . 25] and the next bucket covers the range (0 . 25 , 0 . 5] .
The STS spread results for BERT and S-BERT before and after CT is available in Figure 3.
We ﬁnd that BERT produces sentence embeddings with high cosine similarity for all sentence pairs.
Both CT and S-BERT improve the
STS performance by decreasing the mean similarity for non-similar sentence pairs, but S-BERT does this with less precision.
After attaining prior knowledge about the model’s sentence embedding space, we ﬁne-tune towards the STS-b training data using the S-BERT regression setup, but with model speciﬁc regression labels.
The cosine similarity labels are linearly mapped to the range [ M, 1] , where M is the mean cosine similarity of the lowest bucket.
For each model and label scheme we perform 10 training runs forepochs.
Table 2 shows the test results of the model that performed best on the validation set.
We see a clear increase in performance for all models when utilizing the model speciﬁc regression labels.
However, we ﬁnd no signiﬁcant increase in the supervised results when applying either CT, SBERT, or a combination of the two prior to the supervised ﬁne-tuning.
As discussed in Appendix A. we failed to reproduce the results of Reimers & Gurevych (2019), which reports a mean Spearman correlation of S-BERT-Base: 85 .
35 and S-BERT-Large: 86 .
10 , after training for 2 epochs.
[Table]
85.02 / 85.54 InferSent-GloVe 65.30 / 63.
BERT-Base 85.28 / 84.
85.11 / 85.64 USE v 78.73 / 77. BERT-Large 85.54 / 85.
85.90 / 86.35 S-BERT-Distil 73.88 / 76. S-BERT-Distil 84.22 / 84. 85.40 / 85.64 S-BERT-Base 74.15 / 76. S-BERT-Base 85.17 / 84.
85.59 / 85.81 S-BERT-Large 76.16 / 79. S-BERT-Large 85.14 / 85.
85.25 / 86.28 Our contributions
BERT-Distil-CT 79.00 / 78. BERT-Distil-CT 84.14 / 84. 85.32 / 85.82 BERT-Base-CT 77.87 / 76. BERT-Base-CT 85.13 / 84.
85.76 / 85.89 BERT-Large-CT 79.97 / 78.
BERT-Large-CT 85.20 / 84. 86.37 / 85.89 S-BERT-Base-CT 76.25 / 80. S-BERT-Distil-CT 80.09 / 84. 85.61 / 85.80 S-BERT-Base-CT 78.83 / 81.
S-BERT-Base-CT 85.26 / 85.
85.72 / 85.95 S-BERT-Large-CT 80.99 / 82. S-BERT-Large-CT 85.36 / 85. 5. M ULTILINGUAL STS [Table]
Our Contributions
[Table] We investigate the performance of CT when applied to various languages and evaluated towards STS data for Arabic, Spanish (Cer et al [Table] , 2017), Russian 1 , and Swedish (Isbister & Sahlgren, 2020).
All
CT training is performed solely with data for the respective language towards which we evaluate it, using text from a Wikipedia dump of that language (See Appendix C.2).
We perform experiments with three different types of pre-trained models, all of which have encountered the targeted evaluation language during pre-training:
Native BERT models pre-trained with text data speciﬁcally for the targeted language, a multilingual BERT pre-trained for 104 languages and an XLM-R model pre-trained on 100 languages (Conneau et al., 2020)
Results in table 3 show that CT clearly improves the performance of all models.
Prior to training with CT, we ﬁnd that the multilingual BERT performs best, with the exception of Spanish where the native BERT performs the best.
XLM-R performs the worst on all languages before CT.
After training with CT the native BERT models outperform both the multilingual models, again with the exception of Spanish, where a slight edge is seen for the Multilingual BERT.
The big performance increase seen on all models, on all languages, without requiring any labeled data, clearly demonstrate the robustness and unsupervised applicability of CT.
5. C ORPUS V ARIETY
To investigate how CT is impacted by different types of text data we vary the corpus from which training data is sampled.
The corpora we consider are as follows: a dump of all English Wikipedia pages, a large book corpus comprised of 11 , 038 books (Zhu et al., 2015), resulting in text data with a different tone and style compared to the text found on Wikipedia.
Finally, we generate random word sequences by ﬁrst uniformly sampling a sentence length in the range [20 , 75] and then ﬁlling that sequence with uniformly sampled tokens from the model’s vocabulary.
For each corpus, we train 5 models with CT, and report the mean unsupervised Pearson and Spearman correlation on the STS-b test set.
The results found in table 4 show that CT applied with different types corpus styles yields different STS results.
All models attain their highest score with the Wikipedia data, with a noticeable performance drop with the book corpus and large performance drop using random data.
Although the random corpus performs worst, it interestingly improves the performance of the smaller models while drastically worsening the performance of the large model.
While the number of models used in this experiment is too small for conclusive evidence, the performance drop between corpus types seems correlated with model size. [Table] [Table] [Table]
[Table]
D ISCUSSION
The quality of a sentence representation depends on the generating model’s ability to represent contextual interactions between the individual parts (in most cases, wordpieces) of the sentence.
We refer to this as compositionality , for the lack of a better term.
We propose that the task-bias of current Transformer language models can be seen as a form of compositionality-amnesia, where the models progressively express less compositional information throughout the layers, in favor of features speciﬁc to the pre-training objective.
Sentence embedding methods attempt to correct for this bias by applying a learning criterion that enforces compositionality in the ﬁnal layers.
In the case of CT, the learning objective is simply to maximize the dot product for identical sentences, and minimizing it for dissimilar ones.
In the case of other techniques, the learning objective takes the form of modeling adjacent sentences (Skip-thoughts, Quick-thoughts, and DeCLUTR), or classifying entailment based on two given sentences (S-BERT), both of which have semantic interpretations.
We argue that the CT objective is more suitable for the purpose of enforcing compositionality, since it only targets the composition function; there is no semantics involved in distinguishing identical from dissimilar sentences (all the necessary semantics is already learned by the language modeling objective).
The ﬁnding that CT works to some extent even with randomly generated sentences further strengthens this interpretation.
It is our intuition that CT is non-constructive, or in a sense uninformative: It does not add new information to the model, but rather forces the model to realign such that the compositional representation discriminates between different sentences.
Hence, we ﬁnd little reason to believe that the realignment enforced by CT to be beneﬁcial for ﬁne-tuning tasks where ample training data is available e.g. tasks for which a pre-trained BERT model can be ﬁne-tuned with good performance.
This is in accordance with the results available in Appendix 10, where the CT models are evaluated towards multiple supervised model tasks.
As can be seen in Figure 3, CT decreases the cosine-similarity for non-semantically similar sentences, while the cosine-similarity for highly semantically similar sentences mainly remain the same.
We believe the reason for this desired behaviour to be that all representations are generated through a common parameterized compositionality function (the Transformer model).
We thus ﬁnd it unlikely that similar results could be attained by applying CT directly to individual representations so that the manipulation of individual embeddings is performed independently (as algorithms like Word2Vec does).
Our work demonstrates the potential to produce high-quality semantic sentence representations given a pre-trained Transformer Language model, without requiring any labeled data.
In accordance with the multilingual results in Section 3, we would hence like to emphasize that this makes
CT well suited for low resource languages, as neither pre-training or ﬁne-tuning requires labeled data.
Additionally, we think that interesting future work might consider exploring the possibilities of applying CT (or similar re-tuning objectives) during the pre-training of Transformer language model, and/or applying it to different intermediate layers
Finally, it is noteworthy that our results are not to be considered ﬁnal, as many of the chosen hyperparameters for the CT experiments are yet to be thoroughly investigated.
It is therefore possible that
CT and similar methods can yield even better results, with either utilizing different language models or tuning of certain hyperparameters.
Especially considering that during the unsupervised tasks, due to the emulation of having zero labeled data, we strictly performed a ﬁxed number of iterations and assumed the worst-case scenario by reporting results for the worst performing model of the two CT models.
A higher performance is therefore expected if the training is monitored with a validation set or if one were to combine the output of both CT models.
C ONCLUSION
This paper contributed with a layer-wise survey of the unsupervised STS performance of pre-trained
Transformer language models.
Results from this survey indicates that the ﬁnal layer of most models produce the worst performing representations.
To overcome this we proposed the self-supervised method Contrastive Tension (CT) that trains the model to output semantically distinguishable sentence representations, without requiring any labeled data.
In an unsupervised setting CT achieves strong STS results when applied with various models, with varying corpora and across multiple languages.
Setting a new SOTA score for multiple well established unsupervised English STS tasks.
Additionally, this paper introduced an alteration to the supervised STS regression task proposed by Reimers & Gurevych (2019), which improves the supervised STS scores for all tested models.
Using this altered regression task, regular BERT models achieve equally good as when ﬁrst ﬁnetuned towards NLI, CT or both.
Suggesting that the current Transformer pre-training objectives themselves capture useful sentence level semantic knowledge.
A CKNOWLEDGEMENTS
This work was partially funded by Vinnova under contract 2019-02996, and the Swedish Foundation for Strategic Research (SSF) under contract RIT15-0046.
Finally, the authors wish to thank Joey ¨
Ohman, Melker Mossberg and Linus Bein Fahlander, for the spicy Taco evenings which helped us through the rough times of COVID-19.
ISUAL E XAMPLE OF C ONTRASTIVE T
[Figure]
Figure 4 demonstrates how CT is applied when K = 7 which yields 1 positive sentence pair ( S A , S A ) where the models are trained to maximize the dot product, and 7 negative sentence pairs ( S A , S B ) , ( S A , S C ) ...
( S A , S H ) where the models are trained to minimize the dot product.
As all sentence pairs are included into the same batch, the loss for all K + 1 sentence pairs are calculated before updating the models parameters.
A. M ODEL N AMING C ONVENTION
Throughout the paper we sequentially apply different ﬁne-tuning objectives on pre-trained transformer models, i.e no pre-training is performed during ﬁne-tuning and no two ﬁne-tuning tasks are applied in parallel.
The terms ”Distil” , ”Base” and ”Large” are used as descriptors of the pretrained model’s size and in the case of ”Distil” also its pre-training objective.
The two main ﬁne-tuning tasks we consider is CT and the Siamese NLI task of InferSent and SBERT.
When a model has been tuned towards the Siamese NLI task an additional ”S-” is added as a preﬁx to the model base.
When a model has been trained with the CT training objective ”-CT” is added as sufﬁx to the model name.
It is strictly the case that when both these tasks are applied to a model, the NLI task is applied before to the CT objective.
For example the model ”S-BERT-Distil-CT” , is a distilled BERT model that has been ﬁne-tuned towards the S-BERT NLI task, before ﬁnally tuned with the CT training objective
. A. L EARNING R ATE S CHEDULE
Hyperparameter search concluded that 2 e − 6 was a stable learning rate for applying CT to pre-trained
BERT models.
However, we found it possible to speedup learning during the early training stages by using a higher learning rate, leading us to the step-wise learning rate schedule seen in table 5.
We found no signiﬁcant difference in the end result when applying the learning rate schedule and when training for a longer period with a smaller learning rate.
[Table]
A. R EPRODUCIBILITY OF P REVIOUS W ORK
There exists a discrepancy regarding the reported STS scores between various previous works (Reimers & Gurevych, 2019; Wang & Kuo, 2020).
As mentioned, we perform all our STS evaluation with the SentEval framework of Conneau & Kiela (2018), the following are our observations and experiences as we compare with reported results of Reimers & Gurevych (2019), and the follow up work of Wang & Kuo (2020).
For the English Unsupervised STS tasks of SemEval 2012-2016 we found that:
Our results for BERT, S-BERT and S-BERT-WK are consistent with the work of Wang & Kuo (2020).
(Although they do not report results for S-BERT-Large-WK).
Reimers & Gurevych (2019) report the highest STS score for the S-BERT models but the lowest for BERT.
Which when compared gives an average Spearman difference of 3 . 9 for S-BERT-Base, 3 . 4 for S-BERT-Large and − 1 . 45 for BERT.
No one agrees upon the scores of either InferSent or USE.
When evaluating towards the Supervised STS tasks of STS-b test set we found that:
Our Spearman results for S-BERT-Base and S-BERT-Large, prior to ﬁne-tuning, are consistent with the work of Reimers & Gurevych (
Differing with less than 0 .
05 points.
Using the ofﬁcial S-BERT code to ﬁne-tune the released S-BERT-Large model, with the described hyperparameters, yields worse results than reported by Reimers & Gurevych (2019), both when evaluating with SentEval or the included evaluation script.
Finally, we note that the ofﬁcial S-BERT implementation continuously evaluates towards the STSb validation set throughout the NLI training, saving the copy that performs best.
This makes all models trained with this setup invalid for the unsupervised STS tasks, as the STS-b validation set is a collection of samples from these tests.
This is not to say that this is the setup that was used for the results reported by Reimers & Gurevych (2019), but something that future researchers should be aware of.
B Additional Experiments and Results B. A DDITIONAL U NSUPERVISED STS
In order to give further insights on the performance of CT we report unsupervised STS results from 10 CT training runs per considered pre-trained model.
Since CT trains two models in parallel this results in 10 CT model pairs i.e. 20 models per considered pre-trained model.
Table 6 show the worst/best performing model and the average performance of all 20 models, completely disregarding which models were paired during training.
Table 7 showcases how two paired CT models tend to differ in regards to the mean unsupervised STS score, showing the min, max and average difference over the 10 training runs.
It is apparent that out of the models whom have not been tuned towards the NLI task, BERT-DistilCT showcases the most stable performance, with very little variation between the worst and best performing model compared to both BERT-Base-CT and BERT-Large-CT who show a lower worst possible score.
If this discrepancy in performance stability is due to model size or that BERT-Distil is trained via distillation is left for future work.
Applying CT to a model who has been tuned towards an NLI task seemingly produces both more stable and better results.
[Table]
[Table]
B. D OWNSTREAM & P ROBING T ASKS
To comply with previous work
[Table]
we evaluate CT on the various set of downstream tasks supplied by the SentEval package (Conneau & Kiela, 2018).
As results in table 8 show
[Table]
we ﬁnd only minor improvements when using the representations from the ﬁne-tuned models compared to BERT.
SBERT produces a minor improvement for most non semantic related downstream tasks and CT performs slightly better on the semantic related tasks SICK-R and STS-b.
Interestingly, the results from table 2 show that BERT-CT, S-BERT and S-BERT-CT all perform better on the STS-b test set when not training an extra linear classiﬁer.
Additionally we evaluate towards the ﬁne grained analysis tasks supplied by SentEval.
The results in table 9 clearly show that S-BERT’s NLI ﬁne-tuning objective decreases the score in all tests compared to BERT.
CT also clearly decreases the performance on all tests except the Bigram Shift task, where this is done to a smaller degree.
[Table] [Table]
69.94 73.77 BERT
[Table]
Large 70.82 55.26 33.
71.23 70.95 BERT
[Table]
Distil
[Table]
NLI 71.23 61.76 31. 59.10 65.29 BERT-Base-NLI 72.12 58.65 31.
63.78 66.29 BERT-Large-NLI 59.79 54.47 29.
65.71 64.25 Our Contributions
BERT-Distil-CT 81.86 74.49 37.
59.94 71.83 BERT-Base-CT 77.68 80.69 34.
64.97 72.86 BERT-Large-CT 64.85 65.93 30. 67.37 69.84 BERT-Distil-NLI-CT 72.55 67.84 33.
59.61 67.38 BERT-Base-NLI-CT 71.44 66.42 32.
63.83 68.26 BERT-Large-NLI-CT 59.26 64.85 29.
B. GLUE B ENCHMARK
We evaluate our BERT-Base-CT model on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), and compare with BERT-Base and S-BERT-Base.
Following Devlin et al.
(2019), we chose the best performing model on the validation set for each combination of learning rate (among 5e-5, 4e-5, 3e-5, 2e-5 for BERT-base and among 5e-5, 4e-5, 3e-5, 2e-5, 1e-5, 2e-6 for the other models), model and task.
For all GLUE tasks, all models are ﬁne-tuned using a batch size of 32 for three epochs.
The results presented in Table 10 demonstrates that BERT performs the best, with a slight margin, on near all tasks.
Exception being the STS-b task, where both S-BERT and BERT-CT see a slight improvement over BERT.
However, as depicted in Table 2 both S-BERT and BERT-CT attain higher test scores with the embedding based approach, compared to feeding both sentences to the same model as is done in the GLUE tasks.
[Table]
C E XPERIMENT S ETUP C. M ODEL C HECKPOINTS
All models and checkpoints where implemented and loaded using the Huggingface API (Wolf et al., 2019).
The various model checkpoints used throughout the experiments of this paper are available in Table
[Table] [Table]
C. W IKIPEDIA D UMPS
All Wikipedia text data was pre-processed using the WikiExtractor provided by Attardi (2015).
The dump ﬁles used as text corpora throughout the experiments of this paper are available in Table
[Table] [Table]