Original: Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks
Revised: Batch-Softmax Contrastive Loss for Pairwise Sentence Scoring Tasks

Original: Abstract
Revised: Abstract

Original: The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP).
Revised: The use of contrastive loss for representation learning has become prominent in computer vision, and it is now getting attention in Natural Language Processing (NLP).

Original: Here, we explore the idea of using a batch-softmax contrastive loss when ﬁne-tuning large-scale pretrained transformer models to learn better taskspeciﬁc sentence embeddings for pairwise sentence scoring tasks.
Revised: Here, we explore the idea of using a batch-softmax contrastive loss when ﬁne-tuning large-scale pretrained transformer models to learn better taskspeciﬁc sentence embeddings for pairwise sentence scoring tasks.

Original: We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we ﬁnd that a special data shufﬂing can be quite important.
Revised: We introduce and study a number of variations in the calculation of the loss as well as in the overall training procedure; in particular, we ﬁnd that a special data shufﬂing can be quite important.

Original: Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classiﬁcation, ranking, and regression.
Revised: Our experimental results show sizable improvements on a number of datasets and pairwise sentence scoring tasks including classiﬁcation, ranking, and regression.

Original: Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP.
Revised: Finally, we offer detailed analysis and discussion, which should be useful for researchers aiming to explore the utility of contrastive loss in NLP.

Original:  Introduction
Revised:  Introduction

Original: Recent years have seen a revolution in Natural Language Processing (NLP)
Revised: Recent years have seen a revolution in Natural Language Processing (NLP)

Original: thanks to the advances in machine learning.
Revised: thanks to the advances in machine learning.

Original: A lot of attention has been paid to architectures, especially for deep learning, as well as to loss functions.
Revised: A lot of attention has been paid to architectures, especially for deep learning, as well as to loss functions.

Original: Notably, loss functions based on similar ideas were proposed in unrelated papers in different machine learning ﬁelds under different names.
Revised: Notably, loss functions based on similar ideas were proposed in unrelated papers in different machine learning ﬁelds under different names.

Original: This can cause difﬁculties when solving new problems or when designing new experiments based on previous results.
Revised: This can cause difﬁculties when solving new problems or when designing new experiments based on previous results.

Original: To a greater extent, this applies to “universal” loss functions, which can be applied in different machine learning areas and tasks such as Computer Vision (CV), Recommendation Systems and NLP.
Revised: To a greater extent, this applies to “universal” loss functions, which can be applied in different machine learning areas and tasks such as Computer Vision (CV), Recommendation Systems and NLP.

Original: An example of such loss function is the batch-softmax contrastive (BSC) loss, which we will discuss below.
Revised: An example of such loss function is the batch-softmax contrastive (BSC) loss, which we will discuss below.

Original: For many
Revised: For many

Original: NLP tasks, it is important to obtain representations of sentences for semantic matching problems, since they can be used for further analysis, e.g., for ﬁnding the best answer to a question.
Revised: NLP tasks, it is important to obtain representations of sentences for semantic matching problems, since they can be used for further analysis, e.g., for ﬁnding the best answer to a question.

Original: Sentence BERT is a recent popular approach for this (Reimers and Gurevych, 2019): it can be trained with different loss functions, and we show that the choice of a loss function is important.
Revised: Sentence BERT is a recent popular approach for this (Reimers and Gurevych, 2019): it can be trained with different loss functions, and we show that the choice of a loss function is important.

Original: Moreover, we show that it will not be optimal to take the “standard” batch-softmax contrastive loss, which is used for training SimCSE (Gao et al., 2021), a recent alternative to Sentence BERT, and we suggest ways to improve its efﬁciency.
Revised: Moreover, we show that it will not be optimal to take the “standard” batch-softmax contrastive loss, which is used for training SimCSE (Gao et al., 2021), a recent alternative to Sentence BERT, and we suggest ways to improve its efﬁciency.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We study the use of a batch-softmax contrastive loss for ﬁne-tuning large-scale transformers to learn better task-speciﬁc embeddings for pairwise sentence scoring tasks.
Revised: • We study the use of a batch-softmax contrastive loss for ﬁne-tuning large-scale transformers to learn better task-speciﬁc embeddings for pairwise sentence scoring tasks.

Original: • We introduce and study a number of novel variations in the calculation of the loss such as symmetrization, incorporating labeled negatives, aligning scores on the similarity matrix diagonal, normalizing over the batch axis, as well as in the overall training procedure, e
Revised: • We introduce and study a number of novel variations in the calculation of the loss such as symmetrization, incorporating labeled negatives, aligning scores on the similarity matrix diagonal, normalizing over the batch axis, as well as in the overall training procedure, e

Original: g., shufﬂing, trainable temperature, and sequential pre-training.
Revised: g., shufﬂing, trainable temperature, and sequential pre-training.

Original: • We demonstrate sizable improvements for a number of pairwise sentence scoring tasks such as classiﬁcation, ranking, and regression.
Revised: • We demonstrate sizable improvements for a number of pairwise sentence scoring tasks such as classiﬁcation, ranking, and regression.

Original: • We offer detailed analysis and discussion, which would be useful for future research.
Revised: • We offer detailed analysis and discussion, which would be useful for future research.

Original: • We release our code at https://github. com/aschern/BSC-loss
Revised: • We release our code at https://github. com/aschern/BSC-loss

Original:  Related Work
Revised:  Related Work

Original: The contrastive loss was proposed by Hadsell et al.
Revised: The contrastive loss was proposed by Hadsell et al.

Original: (2006) as metric learning that contrasts Euclidean distances between embeddings of samples from one class and between samples from different classes.
Revised: (2006) as metric learning that contrasts Euclidean distances between embeddings of samples from one class and between samples from different classes.

Original: Weinberger et al.
Revised: Weinberger et al.

Original: (2006) suggested the triplet loss, which is based on a similar idea, but uses triplets ( anchor , positive , negative ), and aims for the difference between the distances for ( anchor , positive ) and for ( anchor , negative ) to be larger than a margin.
Revised: (2006) suggested the triplet loss, which is based on a similar idea, but uses triplets ( anchor , positive , negative ), and aims for the difference between the distances for ( anchor , positive ) and for ( anchor , negative ) to be larger than a margin.

Original: N -pair loss was presented as a generalization of the contrastive and the triplet losses as a way to solve the problem of extensive construction of hard negative pairs and triplets (Sohn, 2016).
Revised: N -pair loss was presented as a generalization of the contrastive and the triplet losses as a way to solve the problem of extensive construction of hard negative pairs and triplets (Sohn, 2016).

Original: A batch of N pairs of examples from N different classes is sampled, and the ﬁrst element in each pair is considered to be an anchor .
Revised: A batch of N pairs of examples from N different classes is sampled, and the ﬁrst element in each pair is considered to be an anchor .

Original: Thus, for each anchor , there are one positive and N ´ 1 negative pairs.
Revised: Thus, for each anchor , there are one positive and N ´ 1 negative pairs.

Original: The loss contrasts the distances simultaneously using the softmax function over dot-product similarities.
Revised: The loss contrasts the distances simultaneously using the softmax function over dot-product similarities.

Original: The approach was used successfully in CV tasks.
Revised: The approach was used successfully in CV tasks.

Original: The same method of Multiple Negative Ranking for training Dot-Product Scoring Models was applied to ranking natural language responses to emails (Henderson et al., 2017), where the loss uses labeled pairs.
Revised: The same method of Multiple Negative Ranking for training Dot-Product Scoring Models was applied to ranking natural language responses to emails (Henderson et al., 2017), where the loss uses labeled pairs.

Original: A similar idea, called Negative
Revised: A similar idea, called Negative

Original: Sharing , was used to reduce the computational cost when training recommender systems (Chen et al., 2017).
Revised: Sharing , was used to reduce the computational cost when training recommender systems (Chen et al., 2017).

Original: Wu et al.
Revised: Wu et al.

Original: (2018) presented an approach with N -pairs like logic, as a Non-Parametric Softmax Classiﬁer, replacing the weights in the softmax with embeddings of samples from such classes.
Revised: (2018) presented an approach with N -pairs like logic, as a Non-Parametric Softmax Classiﬁer, replacing the weights in the softmax with embeddings of samples from such classes.

Original: It was also proposed to use L2 normalization and temperature.
Revised: It was also proposed to use L2 normalization and temperature.

Original: Yang et al.
Revised: Yang et al.

Original: (2018) used Multiple Negative
Revised: (2018) used Multiple Negative

Original: Ranking to train general sentence representations on data from Reddit and SNLI.
Revised: Ranking to train general sentence representations on data from Reddit and SNLI.

Original: Logeswaran and
Revised: Logeswaran and

Original: Lee (2018) presented a Quick-Thoughts approach to learn sentence embeddings, which constructs batches of contiguous sets of sentences, and for each sentence, contrasts the next sentence in the text and all other candidates.
Revised: Lee (2018) presented a Quick-Thoughts approach to learn sentence embeddings, which constructs batches of contiguous sets of sentences, and for each sentence, contrasts the next sentence in the text and all other candidates.

Original: A lot of subsequent work has focused on maximizing Mutual Information (MI).
Revised: A lot of subsequent work has focused on maximizing Mutual Information (MI).

Original: Oord et al.
Revised: Oord et al.

Original: (2018) presented a loss function based on NoiseContrastive Estimation, called InfoNCE.
Revised: (2018) presented a loss function based on NoiseContrastive Estimation, called InfoNCE.

Original: It models the “similarity” function that estimates the MI between the target (future) and the context (present) signals, and maximizes the MI between temporally nearby signals.
Revised: It models the “similarity” function that estimates the MI between the target (future) and the context (present) signals, and maximizes the MI between temporally nearby signals.

Original: If this “similarity” function expresses the dot-product between embeddings, the
Revised: If this “similarity” function expresses the dot-product between embeddings, the

Original: InfoNCE loss is equivalent to the N -pair loss up to some constants.
Revised: InfoNCE loss is equivalent to the N -pair loss up to some constants.

Original: It was also shown that InfoNCE is equivalent to the Mutual Information Neural Estimator (MINE) up to a constant (Belghazi et al., 2018), which maximizes a lower bound on MI.
Revised: It was also shown that InfoNCE is equivalent to the Mutual Information Neural Estimator (MINE) up to a constant (Belghazi et al., 2018), which maximizes a lower bound on MI.

Original: Deep InfoMax (DIM) (Hjelm et al., 2019) improves
Revised: Deep InfoMax (DIM) (Hjelm et al., 2019) improves

Original: MINE, and can be modiﬁed to incorporate some autoregression as InfoNCE.
Revised: MINE, and can be modiﬁed to incorporate some autoregression as InfoNCE.

Original: However, the effectiveness of loss functions such as DIM and InfoNCE might be primarily connected to MI, rather than to deep metric learning (Tschannen et al., 2020).
Revised: However, the effectiveness of loss functions such as DIM and InfoNCE might be primarily connected to MI, rather than to deep metric learning (Tschannen et al., 2020).

Original: The idea gained a lot of popularity in the ﬁeld of Computer Vision with the advent of SimCLR (a
Revised: The idea gained a lot of popularity in the ﬁeld of Computer Vision with the advent of SimCLR (a

Original: Simple framework for Contrastive Learning of visual Representations), which introduced NT-Xent (normalized temperature-scaled cross-entropy loss)
Revised: Simple framework for Contrastive Learning of visual Representations), which introduced NT-Xent (normalized temperature-scaled cross-entropy loss)

Original: (Chen et al., 2020).
Revised: (Chen et al., 2020).

Original: It uses self-supervised learning, where augmentations of the same image are used as positive examples and augmentations of different images are used as negative examples.
Revised: It uses self-supervised learning, where augmentations of the same image are used as positive examples and augmentations of different images are used as negative examples.

Original: Thus, the task is as follows: for each example in a batch, ﬁnd its paired positive augmentation.
Revised: Thus, the task is as follows: for each example in a batch, ﬁnd its paired positive augmentation.

Original: Here, the N -pairs loss is modiﬁed with a temperature parameter and with an L2 normalization of embeddings to the unit hypersphere.
Revised: Here, the N -pairs loss is modiﬁed with a temperature parameter and with an L2 normalization of embeddings to the unit hypersphere.

Original: The loss was further extended for supervised learning as SupCon loss (Khosla et al., 2020), which aggregates all positive examples (from the same class) in the softmax numerator.
Revised: The loss was further extended for supervised learning as SupCon loss (Khosla et al., 2020), which aggregates all positive examples (from the same class) in the softmax numerator.

Original: Subsequently, similar kinds of loss functions were also introduced to the ﬁeld of Natural Language Processing (NLP).
Revised: Subsequently, similar kinds of loss functions were also introduced to the ﬁeld of Natural Language Processing (NLP).

Original: Gunel et al.
Revised: Gunel et al.

Original: combined the SupCon loss with the cross-entropy loss and obtained state-of-the-art results for several downstream NLP tasks using RoBERTa.
Revised: combined the SupCon loss with the cross-entropy loss and obtained state-of-the-art results for several downstream NLP tasks using RoBERTa.

Original: Giorgi et al.
Revised: Giorgi et al.

Original: (2021), Fang and Xie (2020) and Meng et al.
Revised: (2021), Fang and Xie (2020) and Meng et al.

Original: (2021) used NT-Xent to pre-train Transformers, considering spans sampled from the same document, sentences augmented with back-translation as positive examples, and sequences corrupted with
Revised: (2021) used NT-Xent to pre-train Transformers, considering spans sampled from the same document, sentences augmented with back-translation as positive examples, and sequences corrupted with

Original: MLM.
Revised: MLM.

Original: Luo et al.
Revised: Luo et al.

Original: (2020) proposed to use NT-Xent in a self-supervised setting to learn noise-invariant sequence representations, where sentences augmented with masking were considered as positive examples.
Revised: (2020) proposed to use NT-Xent in a self-supervised setting to learn noise-invariant sequence representations, where sentences augmented with masking were considered as positive examples.

Original: Finally, Gao et al.
Revised: Finally, Gao et al.

Original: (2021) introduced the SimCLR loss to NLP under the name SimCSE (Simple Contrastive Learning of Sentence Embeddings), where sentences processed by a neural network with dropout served as augmentations of the original sentences.
Revised: (2021) introduced the SimCLR loss to NLP under the name SimCSE (Simple Contrastive Learning of Sentence Embeddings), where sentences processed by a neural network with dropout served as augmentations of the original sentences.

Original: Here, we explore various ways to use a similar loss function for pairwise sentence scoring tasks.
Revised: Here, we explore various ways to use a similar loss function for pairwise sentence scoring tasks.

Original: While the above-described loss functions have different names, they are all based on similar ideas.
Revised: While the above-described loss functions have different names, they are all based on similar ideas.

Original: Below, we will use the name Batch-Softmax Contrastive (BSC) loss, which we believe reﬂects the main idea best.
Revised: Below, we will use the name Batch-Softmax Contrastive (BSC) loss, which we believe reﬂects the main idea best.

Original: In our experiments below, we will use the “modern” variant of the loss: with temperature, normalization, and symmetrization components (described in more detail in Section 3.1).
Revised: In our experiments below, we will use the “modern” variant of the loss: with temperature, normalization, and symmetrization components (described in more detail in Section 3.1).

Original: These components were not used for NLP in combination before.
Revised: These components were not used for NLP in combination before.

Original: Softmax aaaa n qqq m qaaaa n aaaa n aaaa m qqq m qaa m aa
Revised: Softmax aaaa n qqq m qaaaa n aaaa n aaaa m qqq m qaa m aa

Original: T
Revised: T

Original: We further introduce a number of novel and important modiﬁcations in the deﬁnition of the loss and in the training procedure, which make it more efﬁcient, and we show that using the resulting loss yields better task-speciﬁc sentence embeddings for pairwise sentence scoring tasks.
Revised: We further introduce a number of novel and important modiﬁcations in the deﬁnition of the loss and in the training procedure, which make it more efﬁcient, and we show that using the resulting loss yields better task-speciﬁc sentence embeddings for pairwise sentence scoring tasks.

Original:  Method 3. Batch-Softmax Contrastive (BSC) Loss
Revised:  Method 3. Batch-Softmax Contrastive (BSC) Loss

Original: Pointwise approaches for training models for pairwise sentence scoring tasks, such as mean squared error (MSE), are problematic as the loss does not take the relative order into account.
Revised: Pointwise approaches for training models for pairwise sentence scoring tasks, such as mean squared error (MSE), are problematic as the loss does not take the relative order into account.

Original: For instance, for two pairs with correct target scores (0.4, 0.5), the loss function would equally penalize answers like (0.3, 0.6) and (0.5, 0.4).
Revised: For instance, for two pairs with correct target scores (0.4, 0.5), the loss function would equally penalize answers like (0.3, 0.6) and (0.5, 0.4).

Original: However, the ﬁrst pair is better, as it keeps the correct ranking, while the second one does not.
Revised: However, the ﬁrst pair is better, as it keeps the correct ranking, while the second one does not.

Original: This is addressed in pairwise approaches, e.g., in triplet loss, where the model directly learns an ordering.
Revised: This is addressed in pairwise approaches, e.g., in triplet loss, where the model directly learns an ordering.

Original: Yet, there is a problem for constructing pairs or triplets in the training set, as it is hard to ﬁnd non-trivial negatives examples.
Revised: Yet, there is a problem for constructing pairs or triplets in the training set, as it is hard to ﬁnd non-trivial negatives examples.

Original: Unlike traditional pairwise loss functions, the
Revised: Unlike traditional pairwise loss functions, the

Original: BSC loss treats all other possible pairs of examples in the batch as “negatives.”
Revised: BSC loss treats all other possible pairs of examples in the batch as “negatives.”

Original: That is, only positive pairs are needed for training.
Revised: That is, only positive pairs are needed for training.

Original: Consider a batch
Revised: Consider a batch

Original: X of pairs from a question-answering dataset.
Revised: X of pairs from a question-answering dataset.

Original: In general, let Q m ˆ n and A m ˆ n be the matrices of embeddings produced by a query model and an answer model.
Revised: In general, let Q m ˆ n and A m ˆ n be the matrices of embeddings produced by a query model and an answer model.

Original: We deﬁne the loss function as follows:
Revised: We deﬁne the loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: ´ mean ˆ
Revised: ´ mean ˆ

Original: [Equation]
Revised: [Equation]

Original: Here, softmax is applied by rows (Figure 1), and τ is the temperature.
Revised: Here, softmax is applied by rows (Figure 1), and τ is the temperature.

Original: Both components can be rewritten, e.g., L 

Revised: Both components can be rewritten, e.g., L

Original: p X q can be written as follows:
Revised: p X q can be written as follows:

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: Mathematically, this loss function is similar to the one presented in (Chen et al., 2020).
Revised: Mathematically, this loss function is similar to the one presented in (Chen et al., 2020).

Original: The difference is that we do not use augmentations, and we do not compare q i to q j (or a i to a j ) due to their different nature: we want to compare a question to an answer, not a question to a question or an answer to an answer.
Revised: The difference is that we do not use augmentations, and we do not compare q i to q j (or a i to a j ) due to their different nature: we want to compare a question to an answer, not a question to a question or an answer to an answer.

Original: Thus, we apply the symmetrization in the formula.
Revised: Thus, we apply the symmetrization in the formula.

Original: So, the difference from SimCSE (Gao et al., 2021) is that we compare not only q i to all a j , but also a i to all q j in the batch.
Revised: So, the difference from SimCSE (Gao et al., 2021) is that we compare not only q i to all a j , but also a i to all q j in the batch.

Original: Note that, although a frequent short answer may ﬁt multiple questions in a batch, such pairs are considered as “negative” examples in the loss.
Revised: Note that, although a frequent short answer may ﬁt multiple questions in a batch, such pairs are considered as “negative” examples in the loss.

Original: However, the loss learns Mutual Information (Tschannen et al., 2020), that is p p q i , a i q{p p
Revised: However, the loss learns Mutual Information (Tschannen et al., 2020), that is p p q i , a i q{p p

Original: p q i q p p a i qq , and thus it is robust to this false negatives problem.
Revised: p q i q p p a i qq , and thus it is robust to this false negatives problem.

Original: Early research has already shown the importance of properly conﬁguring and using some BSC loss settings.
Revised: Early research has already shown the importance of properly conﬁguring and using some BSC loss settings.

Original: For example, low temperatures are equivalent to optimizing for hard positives/negatives (Khosla et al., 2020), while L2 normalization of vectors to the unit hypersphere along with temperature effectively weighs different examples (Chen et al., 2020).
Revised: For example, low temperatures are equivalent to optimizing for hard positives/negatives (Khosla et al., 2020), while L2 normalization of vectors to the unit hypersphere along with temperature effectively weighs different examples (Chen et al., 2020).

Original: We further propose a number of important modiﬁcations that can have a major impact on the performance for a number of tasks.
Revised: We further propose a number of important modiﬁcations that can have a major impact on the performance for a number of tasks.

Original: 3. Batch Construction
Revised: 3. Batch Construction

Original: In computer vision, it is common to use a batch size of 5,000, which in turn would naturally be very likely to contain some hard negative examples.
Revised: In computer vision, it is common to use a batch size of 5,000, which in turn would naturally be very likely to contain some hard negative examples.

Original: In NLP, ﬁne-tuning Transformers with large batch sizes requires very large amounts of memory.
Revised: In NLP, ﬁne-tuning Transformers with large batch sizes requires very large amounts of memory.

Original: Thus, much smaller batches are used in practice, and it becomes important to make sure these batches do contain some hard negative examples.
Revised: Thus, much smaller batches are used in practice, and it becomes important to make sure these batches do contain some hard negative examples.

Original: We achieve this by ﬁxing the content of the batches at each training epoch.
Revised: We achieve this by ﬁxing the content of the batches at each training epoch.

Original: Note that this is much simpler than mining hard negatives, as we only need to increase the likelihood that there would be a hard negative example present in the batch, but we do not need to know which particular example in the batch would be hard.
Revised: Note that this is much simpler than mining hard negatives, as we only need to increase the likelihood that there would be a hard negative example present in the batch, but we do not need to know which particular example in the batch would be hard.

Original: Inside the batch, this would be controlled by the temperature parameter.
Revised: Inside the batch, this would be controlled by the temperature parameter.

Original: Example-based shufﬂing The key idea of this method is to batch several groups, so that within each group all pairs are similar based on their ﬁrst or their second elements.
Revised: Example-based shufﬂing The key idea of this method is to batch several groups, so that within each group all pairs are similar based on their ﬁrst or their second elements.

Original: In this way, each positive pair would be accompanied by hard negatives from the same group and by simpler negatives from the remaining examples inside the batch (which come from other groups).
Revised: In this way, each positive pair would be accompanied by hard negatives from the same group and by simpler negatives from the remaining examples inside the batch (which come from other groups).

Original: We use the k -nearest neighbors for an input example to form a group for it, and Faiss (Johnson et al., 2021) to quickly ﬁnd these nearest neighbors in the embedding space.
Revised: We use the k -nearest neighbors for an input example to form a group for it, and Faiss (Johnson et al., 2021) to quickly ﬁnd these nearest neighbors in the embedding space.

Original: Let the pairs be grouped by their ﬁrst elements q i .
Revised: Let the pairs be grouped by their ﬁrst elements q i .

Original: Algorithm 1 summarizes the proposed method.
Revised: Algorithm 1 summarizes the proposed method.

Original: Algorithm
Revised: Algorithm

Original: Input: sequence D , group size s initialize R Ð rs Ź sequence to store the result initialize U Ð ∅ Ź set of used examples randomly shufﬂe D for e in D do if e R U
Revised: Input: sequence D , group size s initialize R Ð rs Ź sequence to store the result initialize U Ð ∅ Ź set of used examples randomly shufﬂe D for e in D do if e R U

Original: then ﬁnd the n nearest neighbors of e from D choose the top s ´ 1 that are not in U add them and e to R and also to U return reversed R
Revised: then ﬁnd the n nearest neighbors of e from D choose the top s ´ 1 that are not in U add them and e to R and also to U return reversed R

Original: Note that in this approach we use two stages in kNN to limit the range of possible candidates and thus to reduce the computational costs (both in terms of time and memory).
Revised: Note that in this approach we use two stages in kNN to limit the range of possible candidates and thus to reduce the computational costs (both in terms of time and memory).

Original: We ﬁrst extract the top- n neighbors (for some large n , e.g., 500), and then we take the top- k from them, so that no duplicates appear in the ﬁnal sequence (for some small k “ 7 ).
Revised: We ﬁrst extract the top- n neighbors (for some large n , e.g., 500), and then we take the top- k from them, so that no duplicates appear in the ﬁnal sequence (for some small k “ 7 ).

Original: The time complexity of such a check is O(1).
Revised: The time complexity of such a check is O(1).

Original: If all such neighbors are already used, then only the considered example will be added to the resulting sequence.
Revised: If all such neighbors are already used, then only the considered example will be added to the resulting sequence.

Original: This case will often arise for the last examples, and thus batches will consist of simple 1-element groups.
Revised: This case will often arise for the last examples, and thus batches will consist of simple 1-element groups.

Original: Therefore, we reverse the sequence to start with these simple batches, as in curriculum learning.
Revised: Therefore, we reverse the sequence to start with these simple batches, as in curriculum learning.

Original: By default, we assume that there should be one positive example for each question/answer (on the diagonal of the matrix), and thus identical neighbors could be optionally ﬁltered.
Revised: By default, we assume that there should be one positive example for each question/answer (on the diagonal of the matrix), and thus identical neighbors could be optionally ﬁltered.

Original: Still, if there are the same q i in the batch X , the loss deﬁnition (eq. 1) does not change.
Revised: Still, if there are the same q i in the batch X , the loss deﬁnition (eq. 1) does not change.

Original: Indeed, let P q “ t i |
Revised: Indeed, let P q “ t i |

Original: q i “ q, p q i , a i q P X u , then @
Revised: q i “ q, p q i , a i q P X u , then @

Original: i, j P P q : p q i , a j q form a positive pair.
Revised: i, j P P q : p q i , a j q form a positive pair.

Original: According to Khosla et al.
Revised: According to Khosla et al.

Original: (2020), for each q , all ˜ q P P q should be placed in the softmax numerator and then averaging over all such ˜ q should be performed outside the logarithm.
Revised: (2020), for each q , all ˜ q P P q should be placed in the softmax numerator and then averaging over all such ˜ q should be performed outside the logarithm.

Original: Thus, in L 0 p X q (eq. 2) only the ﬁrst sum would change as follows: ÿ i P P q q Ti a i ù ÿ i P P q | P q | ÿ j P P q q Ti a j “ ÿ j P P q | P q | ÿ i P P q q T a j “ ÿ j P P q q Tj a j (3)
Revised: Thus, in L 0 p X q (eq. 2) only the ﬁrst sum would change as follows: ÿ i P P q q Ti a i ù ÿ i P P q | P q | ÿ j P P q q Ti a j “ ÿ j P P q | P q | ÿ i P P q q T a j “ ÿ j P P q q Tj a j (3)

Original: In L 1 p X q : ÿ i P P q q Ti a i ù ÿ i P P q | P q | ÿ j P P q q Tj a i “ ÿ i P P q | P q | ÿ j P P q q T a i “ ÿ i P P
Revised: In L 1 p X q : ÿ i P P q q Ti a i ù ÿ i P P q | P q | ÿ j P P q q Tj a i “ ÿ i P P q | P q | ÿ j P P q q T a i “ ÿ i P P

Original: q q Ti a i (4)
Revised: q q Ti a i (4)

Original: In order to select the groups even better, we consider task-speciﬁc embeddings.
Revised: In order to select the groups even better, we consider task-speciﬁc embeddings.

Original: To this end, we apply the current model to encode all pairs at each epoch.
Revised: To this end, we apply the current model to encode all pairs at each epoch.

Original: Algorithm 2 Shufﬂing by words
Revised: Algorithm 2 Shufﬂing by words

Original: Input: sequence D , group size k , shingle size t for e in D do e.shingle Ð random subset of t words of e (ignoring stop-words) sort D by e.shingle initialize gID Ð random uint Ź group ID initialize s ÐŹ current group size initialize prev Ð ﬁrst element of D for e in D do if e.shingle ‰ prev.shingle then gID Ð random uint if s ě k then gID Ð random uint s Ðe.gID Ð gID s Ð s `prev Ð e sort D by e.gID return D
Revised: Input: sequence D , group size k , shingle size t for e in D do e.shingle Ð random subset of t words of e (ignoring stop-words) sort D by e.shingle initialize gID Ð random uint Ź group ID initialize s ÐŹ current group size initialize prev Ð ﬁrst element of D for e in D do if e.shingle ‰ prev.shingle then gID Ð random uint if s ě k then gID Ð random uint s Ðe.gID Ð gID s Ð s `prev Ð e sort D by e.gID return D

Original: Fast shufﬂing For extremely large datasets, example-based shufﬂing is time-consuming even with Faiss; thus, we propose several effective options to perform a less-thorough shufﬂing.
Revised: Fast shufﬂing For extremely large datasets, example-based shufﬂing is time-consuming even with Faiss; thus, we propose several effective options to perform a less-thorough shufﬂing.

Original: We choose some attribute by which we will group the examples, that is, we guarantee some closeness of the examples.
Revised: We choose some attribute by which we will group the examples, that is, we guarantee some closeness of the examples.

Original: Thus, the examples are close if they share the same words, the same cluster number or the same nearest neighbors.
Revised: Thus, the examples are close if they share the same words, the same cluster number or the same nearest neighbors.

Original: First, consider the case of words and grouping by the ﬁrst elements of the pairs (the case of second elements is the same).
Revised: First, consider the case of words and grouping by the ﬁrst elements of the pairs (the case of second elements is the same).

Original: Algorithm 2 presents the shufﬂing process.
Revised: Algorithm 2 presents the shufﬂing process.

Original: We choose a random word or phrase (shingle) in each element and sort the dataset by these shingles.
Revised: We choose a random word or phrase (shingle) in each element and sort the dataset by these shingles.

Original: Since examples with the same shingles will be in a sequence, we assign group IDs sequentially.
Revised: Since examples with the same shingles will be in a sequence, we assign group IDs sequentially.

Original: We split a group if it is too large.
Revised: We split a group if it is too large.

Original: Then we shufﬂe (sort) the dataset by group
Revised: Then we shufﬂe (sort) the dataset by group

Original: IDs.
Revised: IDs.

Original: The resulting algorithmic complexity is equal to the complexity of two sorts of the dataset.
Revised: The resulting algorithmic complexity is equal to the complexity of two sorts of the dataset.

Original: To produce a shufﬂe by clusters, we apply the same algorithm, where each sentence is replaced by its cluster number.
Revised: To produce a shufﬂe by clusters, we apply the same algorithm, where each sentence is replaced by its cluster number.

Original: Thus, each individual shingle is of size t “ 1 .
Revised: Thus, each individual shingle is of size t “ 1 .

Original: To make a shufﬂe by nearest neighbors, we create shingles by “sentences,” where the words are the positions of the top- k nearest neighbors in the input sequence (for some small value of k ).
Revised: To make a shufﬂe by nearest neighbors, we create shingles by “sentences,” where the words are the positions of the top- k nearest neighbors in the input sequence (for some small value of k ).

Original: All these approaches, including k -means clustering, can be effectively implemented using
Revised: All these approaches, including k -means clustering, can be effectively implemented using

Original: MapReduce and parallel computations.
Revised: MapReduce and parallel computations.

Original: 3. Labeled Negatives
Revised: 3. Labeled Negatives

Original: Usually, when the data size is small, hard negative examples may be hard to obtain even with data shufﬂing, e.g., when all examples are semantically distant.
Revised: Usually, when the data size is small, hard negative examples may be hard to obtain even with data shufﬂing, e.g., when all examples are semantically distant.

Original: Nonetheless, if the dataset contains a labeled negative pair with some anchor, then its elements are semantically close by traditional rules of dataset construction.
Revised: Nonetheless, if the dataset contains a labeled negative pair with some anchor, then its elements are semantically close by traditional rules of dataset construction.

Original: Thus, using such a pair inside the batch, where this anchor is present, will add the necessary hard negative example.
Revised: Thus, using such a pair inside the batch, where this anchor is present, will add the necessary hard negative example.

Original: The only change that is added in the loss function is the masking of negative examples—we have no guarantees that the selected negative example is closer to the anchor than the rest of the examples inside the batch.
Revised: The only change that is added in the loss function is the masking of negative examples—we have no guarantees that the selected negative example is closer to the anchor than the rest of the examples inside the batch.

Original: Let y i be a binary label, where y i “ 1 if the i -th pair is positive.
Revised: Let y i be a binary label, where y i “ 1 if the i -th pair is positive.

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: 3. Combo Loss
Revised: 3. Combo Loss

Original: Theoretically, it is beneﬁcial to use several loss functions for training if they are calculated on the same batch (and thus do not require additional computations).
Revised: Theoretically, it is beneﬁcial to use several loss functions for training if they are calculated on the same batch (and thus do not require additional computations).

Original: That is, joint training of BSC and MSE losses combines the advantages of pointwise and of pairwise approaches.
Revised: That is, joint training of BSC and MSE losses combines the advantages of pointwise and of pairwise approaches.

Original: This ensures that for positives examples, the values on the diagonal of the dot-product matrix are not only greater than the rest, but are also close to 1 or to some target similarity.
Revised: This ensures that for positives examples, the values on the diagonal of the dot-product matrix are not only greater than the rest, but are also close to 1 or to some target similarity.

Original: Note that here
Revised: Note that here

Original: L MSE p X
Revised: L MSE p X

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 2 for target positive similarities y i , and thus we do not force all other similarities to zero.
Revised: 2 for target positive similarities y i , and thus we do not force all other similarities to zero.

Original: At the same time, the
Revised: At the same time, the

Original: BSC loss adds new examples (i.e., “negative” pairs) to the training set.
Revised: BSC loss adds new examples (i.e., “negative” pairs) to the training set.

Original: In order to use the BSC loss when training a model for tasks with non-binary labels, we modify the indicator function in equation (5) as 1 r y i ą t s , where t is a conﬁgurable binarization threshold.
Revised: In order to use the BSC loss when training a model for tasks with non-binary labels, we modify the indicator function in equation (5) as 1 r y i ą t s , where t is a conﬁgurable binarization threshold.

Original: Then, we use their convex combination with a conﬁgurable hyperparameter µ P p
Revised: Then, we use their convex combination with a conﬁgurable hyperparameter µ P p

Original: [Equation]
Revised: [Equation]

Original: 1 q
Revised: 1 q

Original: [Equation]
Revised: [Equation]

Original: 3. Normalization
Revised: 3. Normalization

Original: L2 normalization of matrices A and B means that a Ti b j will be equivalent to cosine similarity.
Revised: L2 normalization of matrices A and B means that a Ti b j will be equivalent to cosine similarity.

Original: The embeddings can also be normalized by the batch dimension (by coordinates), which can bring additional regularization.
Revised: The embeddings can also be normalized by the batch dimension (by coordinates), which can bring additional regularization.

Original: In our experiments, we conﬁrm the importance of this, e.g., new representations can be calculated with L2 normalization by coordinates or in a min-max scale.
Revised: In our experiments, we conﬁrm the importance of this, e.g., new representations can be calculated with L2 normalization by coordinates or in a min-max scale.

Original:  Datasets
Revised:  Datasets

Original: NLP tasks that compare pairs of sentences can be roughly divided into three general categories: regression (predicting a similarity score), classiﬁcation (e.g., similar vs. dissimilar), and ranking (search for the best matches).
Revised: NLP tasks that compare pairs of sentences can be roughly divided into three general categories: regression (predicting a similarity score), classiﬁcation (e.g., similar vs. dissimilar), and ranking (search for the best matches).

Original: They differ only by the quality assessment functions, and thus they all can beneﬁt from the above losses.
Revised: They differ only by the quality assessment functions, and thus they all can beneﬁt from the above losses.

Original: Note that it is important to calculate the sentence representations for ranking tasks in advance, as when independently calculating the embeddings of the individual elements in the pairs, the inference time of the model becomes linear instead of quadratic.
Revised: Note that it is important to calculate the sentence representations for ranking tasks in advance, as when independently calculating the embeddings of the individual elements in the pairs, the inference time of the model becomes linear instead of quadratic.

Original: Therefore, we use Sentence-BERT (SBERT), which is trained as a Siamese BERT model, and offers a way to obtain state-of-the-art sentence embeddings, which have been proven useful for a number of tasks (Reimers and Gurevych, 2019; Thakur et al., 2021).
Revised: Therefore, we use Sentence-BERT (SBERT), which is trained as a Siamese BERT model, and offers a way to obtain state-of-the-art sentence embeddings, which have been proven useful for a number of tasks (Reimers and Gurevych, 2019; Thakur et al., 2021).

Original: At inference time, we ﬁrst use SBERT to obtain independently a representation for the individual sentences, and then we calculate the cosine similarities between the corresponding embeddings as we construct pairs as part of the process of ranking.
Revised: At inference time, we ﬁrst use SBERT to obtain independently a representation for the individual sentences, and then we calculate the cosine similarities between the corresponding embeddings as we construct pairs as part of the process of ranking.

Original: We use a variety of public English datasets and tasks for the evaluation.
Revised: We use a variety of public English datasets and tasks for the evaluation.

Original: In particular, we have four ranking tasks (ranking answers to non-factoid questions, ranking questions by their similarity with respect to other questions, ranking comments by their similarity to a given question, ranking factchecked claims by their relevance with respect to an input claim), two binary classiﬁcation tasks (paraphrases identiﬁcation, and duplicate question identiﬁcation), and one regression task (semantic sentence similarity).
Revised: In particular, we have four ranking tasks (ranking answers to non-factoid questions, ranking questions by their similarity with respect to other questions, ranking comments by their similarity to a given question, ranking factchecked claims by their relevance with respect to an input claim), two binary classiﬁcation tasks (paraphrases identiﬁcation, and duplicate question identiﬁcation), and one regression task (semantic sentence similarity).

Original: Antique
Revised: Antique

Original: The dataset contains 2,626 non-factoid questions with answer choices (Hashemi et al., 2019), asked by users on Yahoo!
Revised: The dataset contains 2,626 non-factoid questions with answer choices (Hashemi et al., 2019), asked by users on Yahoo!

Original: Answers .
Revised: Answers .

Original: There are a total of 34,011 question–answer pairs: 27, for training and 6,589 for validation.
Revised: There are a total of 34,011 question–answer pairs: 27, for training and 6,589 for validation.

Original: Each answer is annotated with a relevance score with respect to the question on a scale from 1 to 4 (where 4 denotes the highest relevance), and the task is to rank the answers by their relevance.
Revised: Each answer is annotated with a relevance score with respect to the question on a scale from 1 to 4 (where 4 denotes the highest relevance), and the task is to rank the answers by their relevance.

Original: To model relevance as a cosine similarity, we normalize the scores to the r 0 , 1 s interval.
Revised: To model relevance as a cosine similarity, we normalize the scores to the r 0 , 1 s interval.

Original: We use Mean Reciprocal Rank (MRR) as the main evaluation measure.
Revised: We use Mean Reciprocal Rank (MRR) as the main evaluation measure.

Original: CQA-A
Revised: CQA-A

Original: This dataset was used in SemEvalTask 3 on Community Question Answering subtask A (Nakov et al., 2017).
Revised: This dataset was used in SemEvalTask 3 on Community Question Answering subtask A (Nakov et al., 2017).

Original: The goal is to rank the ﬁrst ten answers in a question thread on Qatar
Revised: The goal is to rank the ﬁrst ten answers in a question thread on Qatar

Original: Living, so that good answers are ranked higher than bad ones.
Revised: Living, so that good answers are ranked higher than bad ones.

Original: We used the clean part of the dataset, which consists of 14,110 and 2,440 labeled question–comments pairs for training and development, respectively.
Revised: We used the clean part of the dataset, which consists of 14,110 and 2,440 labeled question–comments pairs for training and development, respectively.

Original: The evaluation measure is
Revised: The evaluation measure is

Original: Mean Average Precision (MAP).
Revised: Mean Average Precision (MAP).

Original: This dataset contains important metadata, e.g., the date and time of the comment, and sorting the comments by time yields a strong baseline; yet, we only use the text.
Revised: This dataset contains important metadata, e.g., the date and time of the comment, and sorting the comments by time yields a strong baseline; yet, we only use the text.

Original: To train the model with the triplet loss, we group the pairs by the ﬁrst element (anchor).
Revised: To train the model with the triplet loss, we group the pairs by the ﬁrst element (anchor).

Original: CQA-B
Revised: CQA-B

Original: This dataset was developed for
Revised: This dataset was developed for

Original: SemEval-2017 Task 3, subtask B (Nakov et al., 2017), whose goal was to rank ten potentially related questions by their similarity with respect to an input question.
Revised: SemEval-2017 Task 3, subtask B (Nakov et al., 2017), whose goal was to rank ten potentially related questions by their similarity with respect to an input question.

Original: These questions were retrieved from the Qatar Living forum using Google and the input question as a query.
Revised: These questions were retrieved from the Qatar Living forum using Google and the input question as a query.

Original: We use the clean part of the dataset, which consists of 19,990 training and 5,500 development labeled question-question pairs.
Revised: We use the clean part of the dataset, which consists of 19,990 training and 5,500 development labeled question-question pairs.

Original: The main evaluation measure here is MAP.
Revised: The main evaluation measure here is MAP.

Original: There is additional information, e.g., the rank of the retrieved question in the Google search results, which we do not use.
Revised: There is additional information, e.g., the rank of the retrieved question in the Google search results, which we do not use.

Original: PFCC-S Shaar et al.
Revised: PFCC-S Shaar et al.

Original: (2020) presented a dataset for detecting Previously Fact-Checked Claims on
Revised: (2020) presented a dataset for detecting Previously Fact-Checked Claims on

Original: Snopes (PFCC-S), aimed at facilitating the solution of a fact-checking problem: given an input claim, it asks to rank claims that have been previously fact-checked, so that claims that can help verify the input claim are ranked as high as possible.
Revised: Snopes (PFCC-S), aimed at facilitating the solution of a fact-checking problem: given an input claim, it asks to rank claims that have been previously fact-checked, so that claims that can help verify the input claim are ranked as high as possible.

Original: The dataset has 800 positive input–veriﬁed claim pairs for training and 200 such positive pairs for testing, and they are to be matched against a database of 10,369 veriﬁed claims.
Revised: The dataset has 800 positive input–veriﬁed claim pairs for training and 200 such positive pairs for testing, and they are to be matched against a database of 10,369 veriﬁed claims.

Original: The evaluation is performed in terms of a HasPositive@k metric, which checks whether there is a positive match among the ﬁrst k results in the ranked list.
Revised: The evaluation is performed in terms of a HasPositive@k metric, which checks whether there is a positive match among the ﬁrst k results in the ranked list.

Original: There are no negative examples in the dataset, but about eight million such pairs can be created.
Revised: There are no negative examples in the dataset, but about eight million such pairs can be created.

Original: Thus, in order to train models using MSE or triplet loss, we sampled negatives according to the following scheme.
Revised: Thus, in order to train models using MSE or triplet loss, we sampled negatives according to the following scheme.

Original: First, we encoded all sentences using SBERT (we used the model pretrained on STSb and NLI).
Revised: First, we encoded all sentences using SBERT (we used the model pretrained on STSb and NLI).

Original: Then, we selected the ﬁrst element in each positive pair as an anchor and we sorted all other examples by their similarity to this anchor.
Revised: Then, we selected the ﬁrst element in each positive pair as an anchor and we sorted all other examples by their similarity to this anchor.

Original: The assumption is that positive examples will be concentrated in the beginning, e.g., among the top-100.
Revised: The assumption is that positive examples will be concentrated in the beginning, e.g., among the top-100.

Original: Thus, we selected negatives starting from 101 on, logarithmically: on positions 100 ` 2 k , k P N .
Revised: Thus, we selected negatives starting from 101 on, logarithmically: on positions 100 ` 2 k , k P N .

Original: As a result, we obtained many hard negative examples and a small number of easy ones.
Revised: As a result, we obtained many hard negative examples and a small number of easy ones.

Original: Finally, we oversampled the positive pairs to correct the balance of positive and negative examples.
Revised: Finally, we oversampled the positive pairs to correct the balance of positive and negative examples.

Original: Microsoft Research
Revised: Microsoft Research

Original: Paraphrase
Revised: Paraphrase

Original: Corpus (MRPC)
Revised: Corpus (MRPC)

Original: Dolan et al.
Revised: Dolan et al.

Original: (2004) contains 5,800 pairs of sentences, extracted from online news sources.
Revised: (2004) contains 5,800 pairs of sentences, extracted from online news sources.

Original: Each pair was labeled with a tag indicating whether the sentences are paraphrases (semantically equivalent).
Revised: Each pair was labeled with a tag indicating whether the sentences are paraphrases (semantically equivalent).

Original: There are 3,668, 407, and 1,725 pairs in the training, development, and test subsets.
Revised: There are 3,668, 407, and 1,725 pairs in the training, development, and test subsets.

Original: As it is a binary classiﬁcation task with class imbalance, it is evaluated in terms of F1.
Revised: As it is a binary classiﬁcation task with class imbalance, it is evaluated in terms of F1.

Original: Quora Question Pairs (QQP)
Revised: Quora Question Pairs (QQP)

Original: Quora presented a dataset containing over 500,000 sentences with over 400,000 lines of potential duplicate questions.
Revised: Quora presented a dataset containing over 500,000 sentences with over 400,000 lines of potential duplicate questions.

Original: Each line has a binary label indicating whether the line truly contains a duplicate pair.
Revised: Each line has a binary label indicating whether the line truly contains a duplicate pair.

Original: Due to the sampling method, which returns mostly positive pairs, the authors supplemented the dataset with negative pairs composed of “related questions.”
Revised: Due to the sampling method, which returns mostly positive pairs, the authors supplemented the dataset with negative pairs composed of “related questions.”

Original: Following
Revised: Following

Original: Thakur et al.
Revised: Thakur et al.

Original: (2021), we sampled randomly 10, examples for training, and we used F1 score as the main evaluation measure.Approach MRR
Revised: (2021), we sampled randomly 10, examples for training, and we used F1 score as the main evaluation measure.

Original: P@ nDCG@

Original: [Table]

Original: Semantic Textual Similarity Benchmark (STSb)
Revised: Semantic Textual Similarity Benchmark (STSb)

Original: The STS benchmark comprises a selection ofthe English datasets used in the STS tasks organized in the context of SemEval between 2012 and 2017 (Cer et al., 2017).
Revised: The STS benchmark comprises a selection of English datasets used in the STS tasks organized in the context of SemEval between 2012 and(Cer et al., 2017).

Original: The benchmark comprises 8,628 sentence pairs.
Revised: The benchmark comprises 8, sentence pairs.

Original: The pairs were annotated with similarity scores on a scale from 0 to 5 (5 indicating complete equivalence).
Revised: The pairs were annotated with similarity scores on a scale from 0 to 5 (with 5 indicating complete equivalence).

Original: There are a total of 5,749, 1,500 and 1,379 pairs in the training, in the development, and in the testing split, respectively.
Revised: There are a total of 5,749, 1,500 and 1,379 pairs in the training, in the development, and in the testing split, respectively.

Original: The main metric is Spearman’s rank correlation.
Revised: The main evaluation measure is Spearman’s rank correlation.

Original: As in the Antique dataset,  we normalize the scores to the r 0 , 1 s interval
Revised: As for the Antique dataset, here we also normalized the scores to the r 0 , 1 s interval

Original: and then we binarized them based on a threshold of 0 . 6 .
Revised: and then we binarized them based on a threshold of 0.6.

Original:  Experimental Setup 
Revised:  Experimental Setup 5. General Setup

Original: We used BERT-base uncased in all our experiments to be able to perform direct comparison for tasks such as MRPC, QQP and STS to previous work (Reimers and Gurevych, 2019; Thakur et al., 2021).
Revised: We used BERT-base uncased in all our experiments to be able to perform direct comparison for tasks such as MRPC, QQP, and STS to previous work (Reimers and Gurevych, 2019; Thakur et al., 2021).

Original: We considered SimCSE ( sup-simcse-bert-baseuncased checkpoint)
Revised: We considered SimCSE ( sup-simcse-bert-baseuncased checkpoint)

Original: (Gao et al., 2021) as an unsupervised baseline as it uses the base version of the BSC loss, which we modiﬁed.
Revised: (Gao et al., 2021) as an unsupervised baseline as it uses the base version of the BSC loss, which we modiﬁed.

Original: Below, by BSC we will denote using optimal settings in the tables, and variants like BSC - random shufﬂe would mean that instead of these optimal settings, we applied random shufﬂing.
Revised: Below, by BSC we will denote using optimal settings in the tables, and variants like BSC - random shufﬂe would mean that instead of these optimal settings, we applied random shufﬂing.

Original: The hyperparameter settings are described in detail in Appendix 8.
Revised: The hyperparameter settings are described in detail in Appendix ?? . 5. Hyperparameters Setting

Revised: We set the number of warm-up steps to 10% of the total steps, and we limited the input sequence length to 90 subtokens.

Revised: We used a batch size of 30 in all tasks, except for Antique and QQP, where we used 50.

Original: Results
Revised: Note that an order of magnitude larger batch sizes would probably yield better results, but they would also require much more memory. We experimented with learning rates from {5e-6, 1e5, 2e-5, 3e-5}, and we selected (on dev) 3e-5 for CQA-B and 2e-5 for all other experiments. We used the AdamW optimizer with the bias correction for the CQA tasks, and without bias correction for the rest.

Revised: We trained the model for ﬁve epochs for Antique, CQA-A and STSb, for six epochs for

Revised: QQP, MRPC and PFCC-S, and for seven epochs for CQA-B, saving a checkpoint after each one, and we selected the best checkpoint on dev.

Revised: As recommended in (Thakur et al., 2021), due to instability, we did seed optimization, running each approach ﬁve times and selecting the best result (on dev).

Revised: To train with the BSC loss, we used min-max normalization by coordinates with τ “ 1 . 2 for

Revised: PFCC-S and QQP, standard L2 normalization with τ “ 0 . 055 for CQA-A, τ “ 0 . 07 for CQA-B, and τ “ 0 .

Revised: 1 for all other tasks (to ﬁnd the optimal τ , we made it trainable for one run).

Revised: We applied examplebased shufﬂing to train with the BSC loss.

Revised: We used a group size of four in MRPC, of ﬁve in CQA-B, and of eight in all other tasks.

Revised: We iterated over µ values from the set t 0 . 1 , 0 . 5 , 0 .

Revised: 9 u , and we chose µ “ 0 . 1 to train the combo approach for CQA-A,

Revised: MRPC, QQP and STSb tasks, and µ “ 0 . 9 for the other experiments.

Revised: Our experiments demonstrated that there is only a minor effect of changing these hyperparameters.

Revised: For example, the maximum quality drop from µ ( 0 . 1 vs. 0 . 5 ) is 0.4%, i.e., almost at the noise level, while we obtained gains of up to 10% when using our proposed modiﬁcations.

Revised: We trained the triplet loss variant from (Reimers and Gurevych, 2019) with margin “ 0 . 6 for

Revised: PFCC-S, and margin “ 0 .

Revised: 5 for all other tasks.

Original: In this section
Revised: As we have no answers for the test set in MRPC, and no test sets in Antique and PFCC-S, we split the training set into 9:1 to tune the hyper-parameters. The time for training SBERT with the BSC loss (or combo loss) was almost equal to the time for training with the standard MSE loss. The most complex example-based shufﬂing only adds 8% to the training time. Note that it is several times faster than triplet loss, as only positive examples are used for training and thus the dataset becomes several times smaller.

Original: [Table]
Revised: We ran all experiments on a GeForce GTX GPU Ti. Base SBERT has 110M parameters. Its training took up to 10 minutes per epoch depending on the dataset size.

Original:  we compare the BSC loss to other loss functions: MSE and triplet loss.
Revised: In this section, we compare the BSC loss to other loss functions: MSE and triplet loss.

Original: Additionally, we make an ablation study for the BSC loss modiﬁcations we proposed.
Revised: Additionally, we make an ablation study for the BSC loss modiﬁcations we proposed.

Original: Antique
Revised: Antique

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: Our best approach of combo-training MSE and BSC losses outperforms all other variants and the approach proposed in (Hashemi et al., 2019), where speciﬁc negative sampling and a triplet loss were 
Revised: Our best approach of combo-training MSE and BSC losses outperforms all other variants and the approach proposed in (Hashemi et al., 2019), where speciﬁc negative sampling and a triplet loss were used. Besides, the best BSC conﬁguration achieves

Original: [Table] used.
Revised: [Table] [Table]

Original: Besides, the best BSC conﬁguration achieves higher scores than MSE.
Revised: higher scores than MSE.

Original: We can see the importance of using predeﬁned hand-crafted negative examples, which brings additional difﬁcult cases and increases MRR by 0.02.
Revised: We can see the importance of using predeﬁned hand-crafted negative examples, which brings additional difﬁcult cases and increases MRR by 0.02.

Original: CQA-A
Revised: CQA-A

Original: The results for CQA subtask A are shown in Table 2.
Revised: The results for CQA subtask A are shown in Table 2.

Original: A comparison with (Nakov et al., 2017) is not very fair, as we did not use the metadata, e.g., the comment position, which was crucial for the best systems.
Revised: A comparison with (Nakov et al., 2017) is not very fair, as we did not use the metadata, e.g., the comment position, which was crucial for the best systems.

Original: Besides, we use SBERT, which is inferior to a ﬁne-tuned BERT.
Revised: Besides, we use SBERT, which is inferior to a ﬁne-tuned BERT.

Original: Nevertheless, our best approach of combo training with
Revised: Nevertheless, our best approach of combo training with

Original: MSE and BSC losses yielded competitive results.
Revised: MSE and BSC losses yielded competitive results.

Original: We further compared different shufﬂing strategies.
Revised: We further compared different shufﬂing strategies.

Original: The data is ordered by questions, and keeping this order turns out to be best.
Revised: The data is ordered by questions, and keeping this order turns out to be best.

Original: That is, the model learns to distinguish positive answers for each question from manually selected negative ones and from answers to other questions.
Revised: That is, the model learns to distinguish positive answers for each question from manually selected negative ones and from answers to other questions.

Original: Also, note that random shufﬂing completely eliminates this structure, and
Revised: Also, note that random shufﬂing completely eliminates this structure, and

Original: MAP drops by 6% absolute.
Revised: MAP drops by 6% absolute.

Original: Fast shufﬂing byclusters, an advanced version of shufﬂing by words, improves these results.
Revised: Fast shufﬂing byclusters, an advanced version of shufﬂing by words, improves these results.

Original: Example-based shufﬂing ﬁnds a data order similar to the initial one, and the quality does not degrade much.
Revised: Example-based shufﬂing ﬁnds a data order similar to the initial one, and the quality does not degrade much.

Original: CQA-B
Revised: CQA-B

Original: The results for CQA-B are shown in Table 2.
Revised: The results for CQA-B are shown in Table 2.

Original: Again, we did not use the question position, which is a critically important feature for the best systems.
Revised: Again, we did not use the question position, which is a critically important feature for the best systems.

Original: We can see that the BSC loss achieved the best score, noticeably outperforming MSE and triplet losses.
Revised: We can see that the BSC loss achieved the best score, noticeably outperforming MSE and triplet losses.

Original: The experiments also demonstrate  the importance of data order when training with the BSC loss.
Revised: The experiments also demonstrate [Table] [Table] the importance of data order when training with the BSC loss.

Original: Since the dataset is small, the model overﬁts when the original data order is ﬁxed.
Revised: Since the dataset is small, the model overﬁts when the original data order is ﬁxed.

Original: PFCC-S Table 3 shows the results for PFCC-S ( HP@k stands for HasPositives@k ).
Revised: PFCC-S Table 3 shows the results for PFCC-S ( HP@k stands for HasPositives@k ).

Original: Note that the scores from (Shaar et al., 2020) are for pre-trained
Revised: Note that the scores from (Shaar et al., 2020) are for pre-trained

Original: SBERT without task-speciﬁc ﬁne-tuning.
Revised: SBERT without task-speciﬁc ﬁne-tuning.

Original: We observed that even when using oversampling to improve the balance of positive examples, ﬁne-tuning [Table] [Table] [Table]
Revised: We observed that even when using oversampling to improve the balance of positive examples, ﬁne-tuning with MSE performed worse than their results.

Original: with MSE performed worse than their results.

Original: Here, we used only positives examples to train with BSC, and normalizing by the zero dimension was the best.
Revised: Here, we used only positives examples to train with BSC, and normalizing by the zero dimension was the best.

Original: Overall, the approaches using BSC and triplet losses were comparable.
Revised: Overall, the approaches using BSC and triplet losses were comparable.

Original: However, the dataset size for training with the BSC loss was much smaller, which is also true for MSE.
Revised: However, the dataset size for training with the BSC loss was much smaller, which is also true for MSE.

Original: As a result, the BSC loss is faster, and preferable for this task.
Revised: As a result, the BSC loss is faster, and preferable for this task.

Original: MRPC Table 4 shows the results for MRPC.
Revised: MRPC Table 4 shows the results for MRPC.

Original: MSE outperformed the BSC loss, but combo achieved a slightly higher F1 score.
Revised: MSE outperformed the BSC loss, but combo achieved a slightly higher F1 score.

Original: QQP
Revised: QQP

Original: The results for QQP are presented in Table 4.
Revised: The results for QQP are presented in Table 4.

Original: We also show results for SBERT and augmented SBERT (in parentheses) from (Thakur et al., 2021).
Revised: We also show results for SBERT and augmented SBERT (in parentheses) from (Thakur et al., 2021).

Original: There score was obtained by training
Revised: There score was obtained by training

Original: SBERT with MSE using another random training sample, but nonetheless, the F1 score is close to ours.
Revised: SBERT with MSE using another random training sample, but nonetheless, the F1 score is close to ours.

Original: The combo approach outperformed separate training with BSC or MSE.
Revised: The combo approach outperformed separate training with BSC or MSE.

Original: STSb Table 5 shows the results for STS.
Revised: STSb Table 5 shows the results for STS.

Original: It is the only task where combo with the BSC loss was worse than MSE.
Revised: It is the only task where combo with the BSC loss was worse than MSE.

Original: This could be due to hard negatives not appearing in the batch in any of the shufﬂing procedures.
Revised: This could be due to hard negatives not appearing in the batch in any of the shuf- [Table] [Table] ﬂing procedures.

Original: Moreover, we observed only marginal improvement when ﬁne-tuning with a
Revised: Moreover, we observed only marginal improvement when ﬁne-tuning with a

Original: BSC model initially trained with MSE.
Revised: BSC model initially trained with MSE.

Original: However, if it was pretrained with BSC up to overﬁtting, ﬁnetuning it with MSE yielded sizable improvements.
Revised: However, if it was pretrained with BSC up to overﬁtting, ﬁnetuning it with MSE yielded sizable improvements.

Original:  Discussion
Revised:  Discussion

Original: We highlight the following observations:
Revised: We highlight the following observations:

Original: • Combo-training with BSC and MSE losses generally yields the best results (the only exception is STSb), and it outperforms the triplet loss with advanced negative sampling.
Revised: • Combo-training with BSC and MSE losses generally yields the best results (the only exception is STSb), and it outperforms the triplet loss with advanced negative sampling.

Original: • The order in which the data is presented for training can be critical, as we have seen in the cases of CQA-A and
Revised: • The order in which the data is presented for training can be critical, as we have seen in the cases of CQA-A and

Original: CQA-B. • The use of labeled negatives examples generally improves the scores by 1-2% absolute.
Revised: CQA-B. • The use of labeled negatives examples generally improves the scores by 1-2% absolute.

Original: • Embedding normalization during training is important.
Revised: • Embedding normalization during training is important.

Original: Moreover, it is useful to normalize to the zero dimension (e.g., for PFCC-S).
Revised: Moreover, it is useful to normalize to the zero dimension (e.g., for PFCC-S).

Original: • Temperature τ of order 0.1 should be used with the standard normalization, and τ of order 1-3 for coordinate normalization.
Revised: • Temperature τ of order 0.1 should be used with the standard normalization, and τ of order 1-3 for coordinate normalization.

Original: • An incorrect training setup may hurt the performance by more than 10%, as was demonstrated for ( i ) ﬁltering out negative examples for which no positives were given in the dataset (Table 1), ( ii ) using poorly formed batches (highest effect in Table 2), ( iii ) suboptimal normalization (Table 3), and ( iv ) wrong temperature value.
Revised: • An incorrect training setup may hurt the performance by more than 10%, as was demonstrated for ( i ) ﬁltering out negative examples for which no positives were given in the dataset (Table 1), ( ii ) using poorly formed batches (highest effect in Table 2), ( iii ) suboptimal normalization (Table 3), and ( iv ) wrong temperature value.

Original: • BSC is more suitable for ranking tasks (Antique, CQA, PFCC-S) , but it can help for other tasks (classiﬁcation and regression tasks, e.g., for MRPC, QQP, STSb) if applied as pretraining or in joint training with MSE.
Revised: • BSC is more suitable for ranking tasks (Antique, CQA, PFCC-S), but it can help for other tasks (classiﬁcation and regression tasks, e.g., for MRPC, QQP, STSb) if applied as pretraining or in joint training with MSE.

Original: Selecting a loss function is important.
Revised: Selecting a loss function is important.

Original: For instance, if the model optimizes Pearson correlation, it achieves a score of 85.57 on the STS task.
Revised: For instance, if the model optimizes Pearson correlation, it achieves a score of 85.57 on the STS task.

Original: Thus, it outperforms almost all considered approaches.
Revised: Thus, it outperforms almost all considered approaches.

Original: Moreover, the combination of such a loss with BSC allows the model to achieve an F1 score of 89.
Revised: Moreover, the combination of such a loss with BSC allows the model to achieve an F1 score of 89.

Original: in the MRPC task (a classiﬁcation task).
Revised: in the MRPC task (a classiﬁcation task).

Original: Finally, we would like to draw a parallel between our work and Augmented SBERT (Thakur et al., 2021).
Revised: Finally, we would like to draw a parallel between our work and Augmented SBERT (Thakur et al., 2021).

Original: When using the BSC loss, some negatives are implicitly added to the dataset.
Revised: When using the BSC loss, some negatives are implicitly added to the dataset.

Original: Augmented
Revised: Augmented

Original: SBERT adds new examples too and retrieves them using BM25 or Semantic Search samplings.
Revised: SBERT adds new examples too and retrieves them using BM25 or Semantic Search samplings.

Original: These methods are comparable to our fast shufﬂing by words ( n -grams) and to example-based shufﬂing, respectively.
Revised: These methods are comparable to our fast shufﬂing by words ( n -grams) and to example-based shufﬂing, respectively.

Original: Moreover, the task-speciﬁc model is used to encode the data in both cases.
Revised: Moreover, the task-speciﬁc model is used to encode the data in both cases.

Original: However, we do not need to label such pairs with another model (cross-encoder)
Revised: However, we do not need to label such pairs with another model (cross-encoder)

Original: due to the BSC loss deﬁnition.
Revised: due to the BSC loss deﬁnition.

Original:  Conclusion and Future Work
Revised:  Conclusion and Future Work

Original: We explored the idea of using a batch-softmax contrastive loss for ﬁne-tuning large-scale pre-trained transformers to learn better task-speciﬁc sentence embeddings for pairwise sentence scoring tasks.
Revised: We explored the idea of using a batch-softmax contrastive loss for ﬁne-tuning large-scale pre-trained transformers to learn better task-speciﬁc sentence embeddings for pairwise sentence scoring tasks.

Original: We introduced and studied a number of variations in the calculation of the loss as well as in the overall training procedure.
Revised: We introduced and studied a number of variations in the calculation of the loss as well as in the overall training procedure.

Original: Our experimental results have shown sizable improvements on a number of datasets and pairwise sentence scoring tasks including ranking, classiﬁcation, and regression.
Revised: Our experimental results have shown sizable improvements on a number of datasets and pairwise sentence scoring tasks including ranking, classiﬁcation, and regression.

Original: In future work, we want to explore new variations of the loss, and to gain better understanding of when to use which variation.
Revised: In future work, we want to explore new variations of the loss, and to gain better understanding of when to use which variation.

Original: We further plan experiments with a larger set of NLP tasks.
Revised: We further plan experiments with a larger set of NLP tasks.

Original: Ethics and Broader Impact
Revised: Ethics and Broader Impact

Original: We would like to warn that the use of large-scale
Revised: We would like to warn that the use of large-scale

Original: Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019).
Revised: Transformers requires a lot of computations and the use of GPUs/TPUs for training, which contributes to global warming (Strubell et al., 2019).

Original: This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we ﬁne-tune them on relatively small datasets.
Revised: This is a bit less of an issue in our case, as we do not train such models from scratch; rather, we ﬁne-tune them on relatively small datasets.

Original: Moreover, running on a CPU for inference, once the model is ﬁne-tuned, is perfectly feasible, and CPUs contribute much less to global warming.
Revised: Moreover, running on a CPU for inference, once the model is ﬁne-tuned, is perfectly feasible, and CPUs contribute much less to global warming.

Original: Acknowledgments
Revised: Acknowledgments

Original: The article was prepared within the framework of the HSE University Basic Research Program.
Revised: The article was prepared within the framework of the HSE University Basic Research Program.

Original: It is also part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of “fake news,” propaganda, and media bias by making users aware of what they are reading.
Revised: It is also part of the Tanbih mega-project, developed at the Qatar Computing Research Institute, HBKU, which aims to limit the impact of “fake news,” propaganda, and media bias by making users aware of what they are reading.

Original: Appendix A Hyperparameters Setting

Original: We set the number of warm-up steps to 10% of the total steps, and we limited the input sequence length to 90 subtokens.

Original: We used a batch size of 30 in all tasks, except for Antique and QQP, where we used 50.

Original: Note that an order of magnitude larger batch sizes would probably yield better results, but they would also require much more memory.

Original: We experimented with learning rates from {5e-6, 1e5, 2e-5, 3e-5}, and we selected (on dev) 3e-5 for

Original: CQA-B and 2e-5 for all other experiments.

Original: We used the AdamW optimizer with the bias correction for the CQA tasks, and without bias correction for the rest.

Original: We trained the model for ﬁve epochs for Antique, CQA-A and STSb, for six epochs for

Original: QQP, MRPC and PFCC-S, and for seven epochs for CQA-B, saving a checkpoint after each one, and we selected the best checkpoint on dev.

Original: As recommended in (Thakur et al., 2021), due to instability, we did seed optimization, running each approach ﬁve times and selecting the best result (on dev).

Original: To train with the BSC loss, we used min-max normalization by coordinates with τ “ 1 . 2 for

Original: PFCC-S and QQP, standard L2 normalization with τ “ 0 . 055 for CQA-A, τ “ 0 . 07 for CQA-B, and τ “ 0 .

Original: 1 for all other tasks (to ﬁnd the optimal τ , we made it trainable for one run).

Original: We applied examplebased shufﬂing to train with the BSC loss.

Original: We used a group size of four in MRPC, of ﬁve in CQA-B, and of eight in all other tasks.

Original: We iterated over µ values from the set t 0 . 1 , 0 . 5 , 0 .

Original: 9 u , and we chose µ “ 0 . 1 to train the combo approach for CQA-A,

Original: MRPC, QQP and STSb tasks, and µ “ 0 . 9 for the other experiments.

Original: Our experiments demonstrated that there is only a minor effect of changing these hyperparameters.

Original: For example, the maximum quality drop from µ ( 0 . 1 vs. 0 . 5 ) is 0.4%, i.e., almost at the noise level, while we obtained gains of up to 10% when using our proposed modiﬁcations.

Original: We trained the triplet loss variant from (Reimers and Gurevych, 2019) with margin “ 0 . 6 for

Original: PFCC-S, and margin “ 0 .

Original: 5 for all other tasks.

Original: As we have no answers for the test set in MRPC, and no test sets in Antique and PFCC-S, we split the training set into 9:1 to tune the hyper-parameters.

Original: The time for training SBERT with the BSC loss (or combo loss) was almost equal to the time for training with the standard MSE loss.

Original: The most complex example-based shufﬂing only adds 8% to the training time.

Original: Note that it is several times faster than triplet loss, as only positive examples are used for training and thus the dataset becomes several times smaller.

Original: We ran all experiments on a GeForce GTX

Original: GPU Ti.

Original: Base SBERT has 110M parameters.

Original: Its training took up to 10 minutes per epoch depending on the dataset size.