DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Abstract
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings.
DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model.
We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other “harmful” types of augmentations.
Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE 1 by 2. absolute points on semantic textual similarity tasks.
Introduction
Learning “universal” sentence representations that capture rich semantic information and are at the same time performant across a wide range of downstream NLP tasks without task-specific finetuning is an important open issue in the field (Conneau et al., 2017; Cer et al., 2018; Kiros et al., 2015;
Logeswaran and Lee, 2018; Giorgi et al., 2020;
Yan et al., 2021; Gao et al., 2021).
Recent work has shown that finetuning pretrained language models with contrastive learning makes it possible to learn good sentence embeddings without any labeled data (Giorgi et al., 2020; Yan et al., 2021;
Gao et al., 2021).
Contrastive learning uses multiple augmentations on a single datum to construct positive pairs whose representations are trained to be more similar to one another than negative pairs.
While different data augmentations (random cropping, color jitter, rotations, etc.) have been found to be crucial for pretraining vision models (Chen et al., 2020), such augmentations have generally been unsuccessful when applied to contrastive learning of sentence embeddings.
Indeed, Gao et al.
(2021) find that constructing positive pairs via a simple dropout-based augmentation works much better than more complex augmentations such as word deletions or replacements based on synonyms or masked language models.
This is perhaps unsurprising in hindsight; while the training objective in contrastive learning encourages representations to be invariant to augmentation transformations, direct augmentations on the input (e.g., deletion, replacement) often change the meaning of the sentence.
That is, ideal sentence embeddings should not be invariant to such transformations.
We propose to learn sentence representations that are aware of, but not necessarily invariant to, such direct surface-level augmentations.
This is an instance of equivariant contrastive learning (Dangovski et al., 2021), which improves vision representation learning by using a contrastive loss on insensitive image transformations (e.g., grayscale) and a prediction loss on sensitive image transformations (e.g., rotations).
We operationalize equivariant contrastive learning on sentences by using dropout-based augmentation as the insensitive transformation (as in SimCSE (Gao et al., 2021)) and MLM-based word replacement as the sensitive transformation.
This results in an additional crossentropy loss based on the difference between the original and the transformed sentence.
We conduct experiments on 7 semantic textual similarity tasks (STS) and 7 transfer tasks from SentEval (Conneau and Kiela, 2018) and find that this difference-based learning greatly improves over standard contrastive learning.
Our DiffCSE approach can achieve around 2.3% absolute
improve-
[Figure]
[-Discriminator 0 1 0 0 0 
-]
Token Detection
[Figure]
ment on STS datasets over SimCSE, the previous state-of-the-art model.
We also conduct a set of ablation studies to justify our designed architecture.
Qualitative study and analysis are also included to look into the embedding space of DiffCSE.
Background and Related Work 2. Learning Sentence Embeddings
Learning universal sentence embeddings has been studied extensively in prior work, including unsupervised approaches such as Skip-Thought (Kiros et al., 2015), Quick-Thought (Logeswaran and Lee, 2018) and FastSent (Hill et al., 2016), or supervised methods such as InferSent (Conneau et al., 2017),
Universal Sentence Encoder (Cer et al., 2018) and
Sentence-BERT (Reimers and Gurevych, 2019).
Recently, researchers have focused on (unsupervised) contrastive learning approaches such as SimCLR (Chen et al., 2020) to learn sentence embeddings.
SimCLR (Chen et al., 2020) learns image representations by creating semantically close augmentations for the same images and then pulling these representations to be closer than representations of random negative examples.
The same framework can be adapted to learning sentence embeddings by designing good augmentation methods for natural language.
ConSERT (Yan et al., 2021) uses a combination of four data augmentation strategies: adversarial attack, token shuffling, cut-off, and dropout.
DeCLUTR (Giorgi et al., 2020) uses overlapped spans as positive examples and distant spans as negative examples for learning contrastive span representations.
Finally,
SimCSE (Gao et al., 2021) proposes an extremely simple augmentation strategy by just switching dropout masks.
While simple, sentence embeddings learned in this manner have been shown to be better than other more complicated augmentation methods.
2. Equivariant Contrastive Learning
DiffCSE is inspired by a recent generalization of contrastive learning in computer vision (CV) called equivariant contrastive learning (Dangovski et al., 2021).
We now explain how this CV technique can be adapted to natural language.
Understanding the role of input transformations is crucial for successful contrastive learning.
Past empirical studies have revealed useful transformations for contrastive learning, such as random resized cropping and color jitter for computer vision (Chen et al., 2020) and dropout for NLP (Gao et al., 2021).
Contrastive learning encourages representations to be insensitive to these transformations, i.e. the encoder is trained to be invariant to a set of manually chosen transformations.
The above studies in CV and NLP have also revealed transformations that are harmful for contrastive learning.
For example, Chen et al.
(2020) showed that making the representations insensitive to rotations decreases the ImageNet linear probe accuracy, and Gao et al.
(2021) showed that using an MLM to replace 15% of the words drastically reduces performance on
STS-B.
While previous works simply omit these transformations from contrastive pre-training, here we argue that we should still make use of these transformations by learning representations that are sensitive (but not necessarily invariant) to such transformations.
The notion of (in)sensitivity can be captured by the more general property of equivariance in mathematics.
Let T be a transformation from a group
G and let T ( x ) denote the transformation of a sentence x .
Equivariance is the property that there is an induced group transformation T ′ on the output features (Dangovski et al., 2021):
[Equation]
In the special case of contrastive learning, T ′ ’s target is the identity transformation, and we say that f is trained to be “invariant to T .”
However, invariance is just a trivial case of equivariance, and we can design training objectives where T ′ is not the identity for some transformations (such as MLM), while it is the identity for others (such as dropout).
Dangovski et al.
(2021) show that generalizing contrastive learning to equivariance in this way improves the semantic quality of features in CV, and here we show that the complementary nature of invariance and equivariance extends to the NLP domain.
The key observation is that the encoder should be equivariant to MLM-based augmentation instead of being invariant.
We can operationalize this by using a conditional discriminator that combines the sentence representation with an edited sentence, and then predicts the difference between the original and edited sentences.
This is essentially a conditional version of the ELECTRA model (Clark et al., 2020), which makes the encoder equivariant to MLM by using a binary discriminator which detects whether a token is from the original sentence or from a generator.
We hypothesize that conditioning the ELECTRA model with the representation from our sentence encoder is a useful objective for encouraging f to be “equivariant to MLM.”
To the best of our knowledge, we are the first to observe and highlight the above parallel between
CV and NLP.
In particular, we show that equivariant contrastive learning extends beyond CV, and that it works for transformations even without algebraic structures, such as diff operations on sentences.
Further, insofar as the canonical set of useful transformations is less established in NLP than is in CV, DiffCSE can serve as a diagnostic tool for NLP researchers to discover useful transformations.
Difference-based Contrastive Learning
Our approach is straightforward and can be seen as combining the standard contrastive learning objective from SimCSE (Figure 1, left) with a difference prediction objective which conditions on the sentence embedding (Figure 1, right).
Given an unlabeled input sentence x , SimCSE creates a positive example x + for it by applying different dropout masks.
By using the BERT base encoder f , we can obtain the sentence embedding h = f ( x ) for x (see section 4 for how h is obtained).
The training objective for SimCSE is:
[Equation]
where N is the batch size for the input batch { x i } Ni =1 as we are using in-batch negative examples, sim( · , · ) is the cosine similarity function, and τ is a temperature hyperparameter.
On the right-hand side of Figure 1 is a conditional version of the difference prediction objective used in ELECTRA (Clark et al., 2020), which contains a generator and a discriminator.
Given a sentence of length T , x = [ x (1) , x (2) , ..., x ( T ) ] , we first apply a random mask m =
[ m (1) , m (2) , ..., m ( T ) ] , m ( t ) ∈
[0 , 1] on x to obtain x ′ = m · x .
We use another pretrained MLM as the generator G to perform masked language modeling to recover randomly masked tokens in x ′ to obtain the edited sentence x ′′ =
G ( x ′ ) .
Then, we use a discriminator D to perform the Replaced
Token Detection (RTD) task.
For each token in the sentence, the model needs to predict whether it has been replaced or not.
The cross-entropy loss for a single sentence x is:
[Equation]
And the training objective for a batch is L RTD =
[Equation]
RTD .
Finally we optimize these two losses together with a weighting coefficient λ :
[Equation]
The difference between our model and ELECTRA is that our discriminator D is conditional , so it can use the information of x compressed in a fixeddimension vector h
= f ( x ) .
The gradient of D can be backward-propagated into f through h .
By doing so, f will be encouraged to make h informative enough to cover the full meaning of x , so that D can distinguish the tiny difference between x and x ′′ .
This approach essentially makes the conditional discriminator perform a “diff operation”, hence the name DiffCSE.
When we train our DiffCSE model, we fix the generator G , and only the sentence encoder f and the discriminator D are optimized.
After training, we discard D and only use f (which remains fixed) to extract sentence embeddings to evaluate on the downstream tasks.
Experiments 4. Setup
In our experiment, we follow the setting of unsupervised SimCSE (Gao et al., 2021) and build our model based on their PyTorch implementation.
We also use the checkpoints of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the initialization of our sentence encoder f .
We add an MLP layer with Batch Normalization (Ioffe and Szegedy, 2015) (BatchNorm) on top of the [CLS] representation as the sentence embedding.
We will compare the model with/without BatchNorm in section 5.
For the discriminator D , we use the same model as the sentence encoder f (BERT/RoBERTa).
For the generator G , we use the smaller DistilBERT and DistilRoBERTa (Sanh et al., 2019) for efficiency.
Note that the generator is fixed during training unlike the ELECTRA paper (Clark et al., 2020).
We will compare the results of using different size model for the generator in section 5.
More training details are shown in Appendix A.
4. Data
For unsupervised pretraining, we use the same 10 6 randomly sampled sentences from English
Wikipedia that are provided by the source code of SimCSE.
We evaluate our model on 7 semantic textual similarity (STS) and 7 transfer tasks in SentEval.
STS tasks includes STS 2012– 2016 (Agirre et al., 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014).
All the STS experiments are fully unsupervised, which means no STS training datasets are used and all embeddings are fixed once they are trained.
The transfer tasks are various sentence classification tasks, including MR (Pang and
Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005).
In these transfer tasks, we will use a logistic regression classifier trained on top of the frozen sentence embeddings, following the standard setup (Conneau and Kiela, 2018).
4. Results Baselines
We compare our model with many strong unsupervised baselines including SimCSE (Gao et al., 2021), IS-BERT (Zhang et al., 2020), CMLM (Yang et al., 2020), DeCLUTR (Giorgi et al., 2020), CT-BERT (Carlsson et al., 2021), SG-OPT (Kim et al., 2021) and some post-processing methods like BERT-flow (Li et al., 2020) and BERT-whitening
(Su et al., 2021) along with some naive baselines like averaged GloVe embeddings (Pennington et al., 2014) and averaged first and last layer BERT embeddings.
Semantic Textual Similarity (STS)
We show the results of STS tasks in Table 1 including
BERT base (upper part) and RoBERTa base (lower part).
We also reproduce the previous state-ofthe-art SimCSE (Gao et al., 2021).
DiffCSEBERT base can significantly outperform SimCSEBERT base and raise the averaged Spearman’s correlation from 76.25% to 78.49%.
For the RoBERTa model, DiffCSE-RoBERTa base can also improve upon SimCSE-RoBERTa base from 76.57% to 77.80%.
Transfer Tasks
We show the results of transfer tasks in Table 2.
Compared with SimCSEBERT base , DiffCSE-BERT base can improve the averaged scores from 85.56% to 86.86%.
When applying it to the RoBERTa model, DiffCSERoBERTa base also improves upon SimCSERoBERTa base from 84.84% to 87.04%.
Note that the CMLM-BERT base (Yang et al., 2020) can achieve even better performance than DiffCSE.
However, they use 1TB of the training data from
Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining.
We put their scores in Table 2 for reference.
In SimCSE, the authors propose to use MLM as an auxiliary task for the sentence encoder to further boost the performance of transfer tasks.
Compared with
[Table]
the results of SimCSE with MLM, DiffCSE still can have a little improvement around 0.2%.
Ablation Studies
In the following sections [Table] we perform an extensive series of ablation studies that support our model design.
We use BERT base model to evaluate on the development set of STS-B and transfer tasks.
Removing Contrastive Loss
In our model, both the contrastive loss and the RTD loss are crucial because they maintain what should be sensitive and what should be insensitive respectively.
If we remove the RTD loss, the model becomes a SimCSE model; if we remove the contrastive loss, the performance of STS-B drops significantly by 30%, while the average score of transfer tasks also drops by 2% (see Table 3).
This result shows that it is important to have insensitive and sensitive attributes that exist together in the representation space.
Next Sentence vs. Same Sentence
Some methods for unsupervised sentence embeddings like
Quick-Thoughts (Logeswaran and Lee, 2018) and
CMLM (Yang et al., 2020) predict the next sentence as the training objective.
We also experiment with a variant of DiffCSE by conditioning the ELECTRA loss based on the next sentence.
Note that this kind of model is not doing a “diff operation” between two similar sentences, and is not an instance of equivariant contrastive learning.
As shown in Table 3 (use next sent.
for x ′ ), the score of STS-B decreases significantly compared to DiffCSE while transfer performance remains similar.
We also tried using the same sentence and the next sentence at the same time for conditioning the ELECTRA objective (use same+next sent.
for x ′ ), and did not observe improvements.
Other Conditional Pretraining Tasks
Instead of a conditional binary difference prediction loss, we can also consider other conditional pretraining tasks such as a conditional MLM objective proposed by Yang et al.
(2020), or corrective language modeling, 5 proposed by COCO-LM (Meng et al., 2021).
We experiment with these objectives instead of the difference prediction objective in Table 3.
We observe that conditional MLM on the same sentence does not improve the performance either on
STS-B or transfer tasks compared with DiffCSE.
Conditional MLM on the next sentence performs even worse for STS-B, but slightly better than using the same sentence on transfer tasks.
Using both the same and the next sentence also does not improve the performance compared with DiffCSE.
For the corrective LM objective, the performance of STS-B decreases significantly compared with DiffCSE.
Augmentation Methods: Insert/Delete/Replace
In DiffCSE, we use MLM token replacement as the equivariant augmentation.
It is possible to use other methods like random insertion or deletion instead of replacement.
6 For insertion, we choose to
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
randomly insert mask tokens to the sentence, and then use a generator to convert mask tokens into real tokens.
The number of inserted masked tokens is 15% of the sentence length.
The task is to predict whether a token is an inserted token or the original token.
For deletion, we randomly delete 15% tokens in the sentence, and the task is to predict for each token whether a token preceding it has been deleted or not.
The results are shown in Table 4.
We can see that using either insertion or deletion achieves a slightly worse STS-B performance than using MLM replacement.
For transfer tasks, their results are similar.
Finally, we find that combining all three augmentations in the training process does not improve the MLM replacement strategy.
Pooler Choice
In SimCSE, the authors use the pooler in BERT’s original implementation (one linear layer with tanh activation function) as the final layer to extract features for computing contrastive loss.
In our implementation (see details in Appendix A), we find that it is better to use a two-layer pooler with Batch Normalization (BatchNorm)
(Ioffe and Szegedy, 2015), which is commonly used in contrastive learning framework in computer vision (Chen et al., 2020;
Grill et al., 2020; Chen and He, 2021; Hua et al., 2021).
We show the ablation results in Table 5.
We can observe that adding BatchNorm is beneficial for either
DiffCSE or SimCSE to get better performance on
STS-B and transfer tasks.
Size of the Generator
In our DiffCSE model, the generator can be in different model size from BERT large , BERT base (Devlin et al., 2019),
DistilBERT base (Sanh et al., 2019), BERT medium ,
BERT small , BERT mini , BERT tiny (Turc et al., 2019).
Their exact sizes are shown in Table 6 (L: number of layers, H: hidden dimension).
Notice that although DistilBERT base has only half the number of layers of BERT, it can retain 97% of
[Table] [Table]
BERT’s performance due to knowledge distillation.
We show our results in Table 6, we can see the performance of transfer tasks does not change much with different generators.
However, the score of STS-B decreases as we switch from BERTmedium to BERT-tiny.
This finding is not the same as ELECTRA, which works best with generators 1/4-1/2 the size of the discriminator.
Because our discriminator is conditional on sentence vectors, it will be easier for the discriminator to perform the RTD task.
As a result, using stronger generators (BERT base , DistilBERT base ) to increase the difficulty of RTD would help the discriminator learn better.
However, when using a large model like BERT large , it may be a too-challenging task for the discriminator.
In our experiment, using
DistilBERT base , which has the ability close to but slightly worse than BERT base , gives us the best performance.
Masking Ratio
In our conditional ELECTRA task, we can mask the original sentence in different ratios for the generator to produce MLM-based augmentations.
A higher masking ratio will make more perturbations to the sentence.
Our empirical result in Table 7 shows that the difference between difference masking ratios is small (in 15%-40% ), and a masking ratio of around 30% can give us the best performance.
Coefficient
λ
In Section 3, we use the λ coefficient to weight the ELECTRA loss and then add it with contrastive loss.
Because the contrastive learning objective is a relatively easier task, the scale of contrastive loss will be 100 to 1000 smaller than
[Table]
ELECTRA loss.
As a result, we need a smaller λ to balance these two loss terms.
In the Tablewe show the STS-B result under different λ values.
Note that when λ goes to zero, the model becomes a SimCSE model.
We find that using λ = 0 .can give us the best performance.
Analysis 6. Qualitative Study
A very common application for sentence embeddings is the retrieval task.
Here we show some retrieval examples to qualitatively explain why DiffCSE can perform better than SimCSE.
In this study, we use the 2758 sentences from STS-B testing set as the corpus, and then use sentence query to retrieve the nearest neighbors in the sentence embedding space by computing cosine similarities.
We show the retrieved top-3 examples in Table 9.
The first query sentence is “you can do it, too.”.
The
SimCSE model retrieves a very similar sentence but has a slightly different meaning (“you can use it, too.”)
as the rank-1 answer.
In contrast, DiffCSE can distinguish the tiny difference, so it retrieves the ground truth answer as the rank-1 answer.
The second query sentence is “this is not a problem”.
SimCSE retrieves a sentence with opposite meaning but very similar wording, while DiffCSE can retrieve the correct answer with less similar wording.
We also provide a third example where both
SimCSE and DiffCSE fail to retrieve the correct answer for a query sentence using double negation.
6. Retrieval Task
Besides the qualitative study, we also show the quantitative result of the retrieval task.
Here we also use all the 2758 sentences in the testing set of STS-B as the corpus.
There are 97 positive pairs in this corpus (with 5 out of 5 semantic similarity scores from human annotation).
For each positive pair, we use one sentence to retrieve the other one, and see whether the other sentence is in the top-1/5/10 ranking.
The recall@1/5/10 of the retrieval task are shown in Table 10.
We can observe that DiffCSE can outperform SimCSE for
[Table] [Table]
[Figure]
recall@1/5/10, showing the effectiveness of using
DiffCSE for the retrieval task.
6. Distribution of Sentence Embeddings
To look into the representation space of DiffCSE, we plot the cosine similarity distribution of sentence pairs from STS-B test set for both SimCSE and DiffCSE in Figure 2.
We observe that both
SimCSE and DiffCSE can assign cosine similarities consistent with human ratings.
However, we also observe that under the same human rating,
DiffCSE assigns slightly higher cosine similarities compared with SimCSE.
This phenomenon
[Table]
may be caused by the fact that ELECTRA and other Transformer-based pretrained LMs have the problem of squeezing the representation space, as mentioned by Meng et al.
As we use the sentence embeddings as the input of ELECTRA to perform conditional ELECTRA training, the sentence embedding will be inevitably squeezed to fit the input distribution of ELECTRA.
We follow prior studies (Wang and Isola, 2020; Gao et al., 2021) to use uniformity and alignment (details in
Appendix C) to measure the quality of representation space for DiffCSE and SimCSE in Table 11.
Compared to averaged BERT embeddings, SimCSE has similar alignment (0.177 v.s. 0.172) but better uniformity (-2.313).
In contrast, DiffCSE has similar uniformity as Avg.
BERT (-1.438 v.s.
but much better alignment (0.097).
It indicates that SimCSE and DiffCSE are optimizing the representation space in two different directions.
And the improvement of DiffCSE may come from its better alignment.
Conclusion
In this paper, we present DiffCSE, a new unsupervised sentence embedding framework that is aware of, but not invariant to, MLM-based word replacement.
Empirical results on semantic textual similarity tasks and transfer tasks both show the effectiveness of DiffCSE compared to current stateof-the-art sentence embedding methods.
We also conduct extensive ablation studies to demonstrate the different modeling choices in DiffCSE.
Qualitative study and the retrieval results also show that
DiffCSE can produce a better embedding space for sentence retrieval.
One limitation of our work is that we do not explore the supervised setting that uses human-labeled NLI datasets to further boost the performance.
We leave this topic for future work.
We believe that our work can provide researchers in the NLP community a new way to utilize augmentations for natural language and thus produce better sentence embeddings. Acknowledgements
This research was partially supported by the Centre for Perceptual and Interactive Intelligence (CPII)
Ltd under the Innovation and Technology Fund (InnoHK).
A Training Details
We use a single NVIDIA 2080Ti GPU for each experiment.
The averaged running time for DiffCSE is 3-6 hours.
We use gridsearch of batch size ∈ { 64 , 128 } learning rate ∈ { 2e-6, 3e-6, 5e-6, 7e-6, 1e-5 } and masking ratio ∈ { 0 . 15 , 0 . 20 , 0 . 30 , 0 . 40 } and λ ∈ { 0 . 1 , 0 .
The temperature τ in SimCSE is set to 0.05 for all the experiments.
During the training process, we save the checkpoint with the highest score on the STS-B development set.
And then we use STS-B development set to find the best hyperparameters (listed in Table 12) for STS task; we use the averaged score of the development sets of 7 transfer tasks to find the best hyperparameters (listed in Table 13) for transfer tasks.
All numbers in Table 1 and Table 2 are from a single run.
[Table]
[Table] [Table]
During testing, we follow SimCSE to discard the
MLP projector and only use the [CLS] output to extract the sentence embeddings.
The numbers of model parameters for BERT base and RoBERTa base are listed in Table 14.
Note that in training time DiffCSE needs two BERT models to work together (sentence encoder + discriminator), but in testing time we only need the sentence
[Table]
encoder, so the model size is the same as the SimCSE model.
Projector with BatchNorm
In Section 5, we mention that we use a projector with BatchNorm as the final layer of our model.
Here we provided the PyTorch code for its structure:
[Table]
B Using Augmentations as Positive/Negative Examples
In Section 5, we try to use different augmentations (e.g. insertion, deletion, replacement) for learning equivariance.
In Table 15 we provide the results of using these augmentations as additional positive or negative examples along with the SimCSE training paradigm.
We can observe that using these augmentations as additional positives only decreases the performance.
The only method that can improve the performance a little bit is to use MLM 15% replaced examples as additional negative examples.
Overall, none of these results can perform better than our proposed method, e.g. using these augmentations to learn equivariance.
C Uniformity and Alignment
Wang and Isola (2020) propose to use two properties, alignment and uniformity , to measure the quality of representations.
Given a distribution of positive pairs p pos and the distribution of the whole dataset p data , alignment computes the expected distance between normalized embeddings of the paired sentences:
[Equation]
Uniformity measures how well the embeddings are uniformly distributed in the representation space:
[Equation]
The smaller the values of uniformity and alignment, the better the quality of the representation space is indicated.
D Source Code
We build our model using the PyTorch implementation of SimCSE 7 Gao et al.
(2021), which is based on the HuggingFace’s Transformers package.
8 We also upload our code 9 and pretrained models (links in README.md ).
Please follow the instructions in
README.md to reproduce the results.
E Potential Risks
On the risk side, insofar as our method utilizes pretrained language models, it may inherit and propagate some of the harmful biases present in such models.
Besides that, we do not see any other potential risks in our paper.