Original: T RANS -E NCODER : U NSUPERVISED SENTENCE - PAIR MODELLING THROUGH SELF - AND MUTUAL - DISTILLATIONS
Revised: T RANS -E NCODER : U NSUPERVISED SENTENCE - PAIR MODELLING THROUGH SELF - AND MUTUAL - DISTILLATIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g., sentence similarity and paraphrase identification).
Revised: In NLP, a large volume of tasks involve pairwise comparison between two sequences (e.g., sentence similarity and paraphrase identification).

Original: Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders.
Revised: Predominantly, two formulations are used for sentence-pair tasks: bi-encoders and cross-encoders.

Original: Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient, however, they usually underperform cross-encoders.
Revised: Bi-encoders produce fixed-dimensional sentence representations and are computationally efficient, however, they usually underperform cross-encoders.

Original: Crossencoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task finetuning and are computationally more expensive.
Revised: Crossencoders can leverage their attention heads to exploit inter-sentence interactions for better performance but they require task finetuning and are computationally more expensive.

Original: In this paper, we present a completely unsupervised sentence-pair model termed as T RANS -E NCODER that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and crossencoders.
Revised: In this paper, we present a completely unsupervised sentence-pair model termed as T RANS -E NCODER that combines the two learning paradigms into an iterative joint framework to simultaneously learn enhanced bi- and crossencoders.

Original: Specifically, on top of a pre-trained language model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations.
Revised: Specifically, on top of a pre-trained language model (PLM), we start with converting it to an unsupervised bi-encoder, and then alternate between the bi- and cross-encoder task formulations.

Original: In each alternation, one task formulation will produce pseudo-labels which are used as learning signals for the other task formulation.
Revised: In each alternation, one task formulation will produce pseudo-labels which are used as learning signals for the other task formulation.

Original: We then propose an extension to conduct such self-distillation approach on multiple
Revised: We then propose an extension to conduct such self-distillation approach on multiple

Original: PLMs in parallel and use the average of their pseudo-labels for mutual-distillation.
Revised: PLMs in parallel and use the average of their pseudo-labels for mutual-distillation.

Original: T RANS -E NCODER creates, to the best of our knowledge, the first completely unsupervised cross-encoder and also a state-of-the-art unsupervised bi-encoder for sentence similarity.
Revised: T RANS -E NCODER creates, to the best of our knowledge, the first completely unsupervised cross-encoder and also a state-of-the-art unsupervised bi-encoder for sentence similarity.

Original: Both the bi-encoder and cross-encoder formulations of T RANSE NCODER outperform recently proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT (Liu et al., 2021) and SimCSE (Gao et al., 2021) by up to 5% on the sentence similarity benchmarks. 
Revised: Both the bi-encoder and cross-encoder formulations of T RANSE NCODER outperform recently proposed state-of-the-art unsupervised sentence encoders such as Mirror-BERT (Liu et al., 2021) and SimCSE (Gao et al., 2021) by up to 5% on the sentence similarity benchmarks. Code and models are released at https://github.com/amzn/trans-encoder .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Comparing pairwise sentences is fundamental to a wide spectrum of tasks in NLP such as information retrieval (IR), natural language inference (NLI), semantic textual similarity (STS) and clustering.
Revised: Comparing pairwise sentences is fundamental to a wide spectrum of tasks in NLP such as information retrieval (IR), natural language inference (NLI), semantic textual similarity (STS) and clustering.

Original: Two general architectures usually used for sentence-pair modelling are bi-encoders and cross-encoders.
Revised: Two general architectures usually used for sentence-pair modelling are bi-encoders and cross-encoders.

Original: In a cross-encoder, two sequences are concatenated and sent into the model (usually deep Transformers like BERT/RoBERTa; Devlin et al.
Revised: In a cross-encoder, two sequences are concatenated and sent into the model (usually deep Transformers like BERT/RoBERTa; Devlin et al.

Original: Liu et al.
Revised: Liu et al.

Original: in one pass.
Revised: in one pass.

Original: The attention heads of
Revised: The attention heads of

Original: Transformers could directly model the inter-sentence interactions and output a classification/relevance score.
Revised: Transformers could directly model the inter-sentence interactions and output a classification/relevance score.

Original: However, a cross-encoder needs to recompute the encoding for different combinations of sentences in each unique sequence pair, resulting in a heavy computational overhead.
Revised: However, a cross-encoder needs to recompute the encoding for different combinations of sentences in each unique sequence pair, resulting in a heavy computational overhead.

Original: It is thus impractical in tasks like IR and clustering where massive pairwise sentence comparisons are involved.
Revised: It is thus impractical in tasks like IR and clustering where massive pairwise sentence comparisons are involved.

Original: Also, task finetuning is always required for converting PLMs to cross-encoders.
Revised: Also, task finetuning is always required for converting PLMs to cross-encoders.

Original: By contrast, in a bi-encoder, each sequence is encoded separately and mapped to a common embedding space for similarity comparison.
Revised: By contrast, in a bi-encoder, each sequence is encoded separately and mapped to a common embedding space for similarity comparison.

Original: The encoded sentences can be cached and reused.
Revised: The encoded sentences can be cached and reused.

Original: A bi-encoder is thus much more efficient.
Revised: A bi-encoder is thus much more efficient.

Original: Also, the output of a bi-encoder can be used off-the-shelf as sentence embeddings for other downstream tasks.
Revised: Also, the output of a bi-encoder can be used off-the-shelf as sentence embeddings for other downstream tasks.

Original: That said, it is well-known that in supervised learning, bi-encoders underperform cross-encoders (Humeau et al., 2020; Thakur et al., 2021) since the former could notexplicitly model the interactions between sentences but could only compare them in the embedding space in a post hoc manner.
Revised: That said, it is well-known that in supervised learning, bi-encoders underperform cross-encoders (Humeau et al., 2020; Thakur et al., 2021) since the former could not

Original: [Figure]
Revised: [Figure]

Original:  In this work, we ask the question: can we leverage the advantages of both bi- and cross-encoders and bootstrap knowledge from them in an unsupervised manner?
Revised: model but could only compare In this work, we ask the question: can we leverage the advantages of both bi- and cross-encoders and bootstrap knowledge from them in an unsupervised manner?

Original: Our proposed T RANS -E NCODER addresses this question with the following intuition: As a starting point, we can use bi-encoder representations to tune a cross-encoder.
Revised: Our proposed T RANS -E NCODER addresses this question with the following intuition: As a starting point, we can use bi-encoder representations to tune a cross-encoder.

Original: With more powerful inter-sentence modelling, the crossencoder should resurface more knowledge from the PLMs than the bi-encoder given the same data.
Revised: With more powerful inter-sentence modelling, the crossencoder should resurface more knowledge from the PLMs than the bi-encoder given the same data.

Original: In turn, the more powerful cross-encoder can distil its knowledge back to the bi-encoder.
Revised: In turn, the more powerful cross-encoder can distil its knowledge back to the bi-encoder.

Original: We can repeat this cycle to iteratively bootstrap from both the bi- and cross-encoders.
Revised: We can repeat this cycle to iteratively bootstrap from both the bi- and cross-encoders.

Original:  T RANS -E NCODER
Revised:  T RANS -E NCODER

Original: The general idea of T RANS -E NCODER is simple yet extremely effective.
Revised: The general idea of T RANS -E NCODER is simple yet extremely effective.

Original: In §2.1, we first transform an off-the-shelf PLM to a strong bi-encoder, serving as an initialisation point.
Revised: In §2.1, we first transform an off-the-shelf PLM to a strong bi-encoder, serving as an initialisation point.

Original: Then, the bi-encoder produces pseudo-labels and the PLM subsequently learns from these pseudo-labels in a cross-encoder manner (§2.2).
Revised: Then, the bi-encoder produces pseudo-labels and the PLM subsequently learns from these pseudo-labels in a cross-encoder manner (§2.2).

Original: Consecutively, the cross-encoder further produces more accurate pseudo-labels for bi-encoder learning (§2.3).
Revised: Consecutively, the cross-encoder further produces more accurate pseudo-labels for bi-encoder learning (§2.3).

Original: This self-distillation process is visualised in Fig.
Revised: This self-distillation process is visualised in Fig.

Original: Then in, §2.4, we propose a further extension called mutual-distillation that stabilises the training process and boosts the encoder performance even more.
Revised: Then in, §2.4, we propose a further extension called mutual-distillation that stabilises the training process and boosts the encoder performance even more.

Original: 2. T RANSFORM PLM S INTO E FFECTIVE B I - ENCODERS
Revised: 2. T RANSFORM PLM S INTO E FFECTIVE B I - ENCODERS

Original: Off-the-shelf PLMs are unsatisfactory bi-encoders.
Revised: Off-the-shelf PLMs are unsatisfactory bi-encoders.

Original: 1 To have a reasonably good starting point, we leverage a simple contrastive tuning procedure to transform existing PLMs to bi-encoders.
Revised: 1 To have a reasonably good starting point, we leverage a simple contrastive tuning procedure to transform existing PLMs to bi-encoders.

Original: This approach is concurrently proposed in both (Liu et al., 2021) and (Gao et al., 2021).
Revised: This approach is concurrently proposed in both (Liu et al., 2021) and (Gao et al., 2021).

Original: Let f ( · ) denote the encoder model (a Transformer-based model); X be a batch of randomly sampled raw sentences.
Revised: Let f ( · ) denote the encoder model (a Transformer-based model); X be a batch of randomly sampled raw sentences.

Original: For any sentence x i ∈ X , we send it to the encoder twice to create two views of the same data point: f ( x i ) and f ( x i ) .
Revised: For any sentence x i ∈ X , we send it to the encoder twice to create two views of the same data point: f ( x i ) and f ( x i ) .

Original: Note that the two encodings slightly differ due to the dropout layers in the encoder.
Revised: Note that the two encodings slightly differ due to the dropout layers in the encoder.

Original: We then use the standard InfoNCE loss (Oord et al., 2018) to cluster the positive pairs together and push away the negative pairs in the mini-batch:
Revised: We then use the standard InfoNCE loss (Oord et al., 2018) to cluster the positive pairs together and push away the negative pairs in the mini-batch:

Original: [Equation]
Revised: [Equation]

Original:  τ denotes a temperature parameter; N i denotes all  negatives of x i within the current data batch, which includes all x j , x j where i ̸ =
Revised: 1) τ denotes a temperature parameter; N i denotes all positive & negatives of x i within the current data batch.

Original: j (i.e., |N i | = 2 |X| − 2 ).

Original: Intuitively, the numerator is the similarity of the self-duplicated pair (the positive example) and the denominator is the sum of the similarities between x i and all other negative strings besides x i .
Revised: Intuitively, the numerator is the similarity of the self-duplicated pair (the positive example) and the denominator is the sum of the similarities between x i and all other negative strings besides x i .

Original: The loss encourages the positive pairs to be relatively close compared to the negative ones.
Revised: The loss encourages the positive pairs to be relatively close compared to the negative ones.

Original: In experiments, we use checkpoints released by SimCSE and Mirror-BERT.
Revised: In experiments, we use checkpoints released by SimCSE and Mirror-BERT.

Original: But in principle, any techniques could be used here as long as the method serves the purpose of transforming BERT to an effective bi-encoder and does not require additional labelled data (see §6).
Revised: But in principle, any techniques could be used here as long as the method serves the purpose of transforming BERT to an effective bi-encoder and does not require additional labelled data (see §6).

Original: 2. S ELF - DISTILLATION : B I - TO C ROSS - ENCODER
Revised: 2. S ELF - DISTILLATION : B I - TO C ROSS - ENCODER

Original: After obtaining a sufficiently good bi-encoder, we leverage it to label sentence pairs sampled from the task of interest.
Revised: After obtaining a sufficiently good bi-encoder, we leverage it to label sentence pairs sampled from the task of interest.

Original: Specifically, for a given sentence-pair ( sent1 , sent2 ), we input them to the bi-encoder separately and get two embeddings (we use the embedding of [CLS] from the last layer).
Revised: Specifically, for a given sentence-pair ( sent1 , sent2 ), we input them to the bi-encoder separately and get two embeddings (we use the embedding of [CLS] from the last layer).

Original: The cosine similarity between them is regarded as their relevance score.
Revised: The cosine similarity between them is regarded as their relevance score.

Original: In this way we have constructed a self-labelled sentence-pair scoring dataset in the format of ( sent1 , sent2 , score).
Revised: In this way we have constructed a self-labelled sentence-pair scoring dataset in the format of ( sent1 , sent2 , score).

Original: We then employ the same model architecture to learn from these score, but with a cross-encoder formulation.
Revised: We then employ the same model architecture to learn from these score, but with a cross-encoder formulation.

Original: The cross-encoder weights are initialised from the original PLM.
Revised: The cross-encoder weights are initialised from the original PLM.

Original: For the sentence-pair ( sent1 , sent2 ), we concatenate them to produce “ [CLS] sent1 [SEP] sent2 [SEP] ” and input it to the cross-encoder.
Revised: For the sentence-pair ( sent1 , sent2 ), we concatenate them to produce “ [CLS] sent1 [SEP] sent2 [SEP] ” and input it to the cross-encoder.

Original: A linear layer (newly initialised) then map the sequence’s encoding (embedding of the [CLS] token) to a scalar.
Revised: A linear layer (newly initialised) then map the sequence’s encoding (embedding of the [CLS] token) to a scalar.

Original: The learning objective of the cross-encoder is minimising the KL divergence between its predictions and the self-labelled scores from the bi-encoder.
Revised: The learning objective of the cross-encoder is minimising the KL divergence between its predictions and the self-labelled scores from the bi-encoder.

Original: This is equivalent to optimising the (soft) binary cross-entropy (BCE) loss:
Revised: This is equivalent to optimising the (soft) binary cross-entropy (BCE) loss:

Original: [Equation]
Revised: [Equation]

Original: where N is the data-batch size; σ ( · ) is the sigmoid activation; x n is the prediction of the cross-encoder; y n is the self-labelled ground-truth score from the bi-encoder.
Revised: where N is the data-batch size; σ ( · ) is the sigmoid activation; x n is the prediction of the cross-encoder; y n is the self-labelled ground-truth score from the bi-encoder.

Original: Note that while the cross-encoder is essentially learning from the data produced by itself (in a bi-encoder form), usually, it outperforms the original bi-encoder on held-out data.
Revised: Note that while the cross-encoder is essentially learning from the data produced by itself (in a bi-encoder form), usually, it outperforms the original bi-encoder on held-out data.

Original: The cross-encoder directly discovers the similarity between two sentences through its attention heads, finding more accurate cues to justify the relevance score.
Revised: The cross-encoder directly discovers the similarity between two sentences through its attention heads, finding more accurate cues to justify the relevance score.

Original: The ability of discovering such cues could then generalise to unseen data, resulting in stronger sentence-pair scoring capability than the original bi-encoder.
Revised: The ability of discovering such cues could then generalise to unseen data, resulting in stronger sentence-pair scoring capability than the original bi-encoder.

Original: From a knowledge distillation perspective, we can view the bi- and cross-encoder as the teacher and student respectively.
Revised: From a knowledge distillation perspective, we can view the bi- and cross-encoder as the teacher and student respectively.

Original: In this case the student outperforms the teacher, not because of stronger model capacity, but smarter task formulation.
Revised: In this case the student outperforms the teacher, not because of stronger model capacity, but smarter task formulation.

Original: By leveraging this simple yet powerful observation, we are able to design a learning scheme that iteratively boosts the performance of both bi- and cross-encoder.
Revised: By leveraging this simple yet powerful observation, we are able to design a learning scheme that iteratively boosts the performance of both bi- and cross-encoder.

Original: 2. S ELF - DISTILLATION : C ROSS - TO BI - ENCODER
Revised: 2. S ELF - DISTILLATION : C ROSS - TO BI - ENCODER

Original: With the more powerful cross-encoder at hand, a natural next step is distilling the extra gained knowledge back to a bi-encoder form, which is more useful for downstream tasks.
Revised: With the more powerful cross-encoder at hand, a natural next step is distilling the extra gained knowledge back to a bi-encoder form, which is more useful for downstream tasks.

Original: Besides, and more importantly, a better bi-encoder could produce even more self-labelled data for cross-encoder learning.
Revised: Besides, and more importantly, a better bi-encoder could produce even more self-labelled data for cross-encoder learning.

Original: In this way we could repeat §2.2 and §2.3, continually bootstrapping the encoder performance.
Revised: In this way we could repeat §2.2 and §2.3, continually bootstrapping the encoder performance.

Original: We create the self-labelled sentence-scoring dataset in the same way as §2.2 except that the crossencoder is used for producing the relevance score.
Revised: We create the self-labelled sentence-scoring dataset in the same way as §2.2 except that the crossencoder is used for producing the relevance score.

Original: The bi-encoder is initialised with the weights after SimCSE training from §2.1.
Revised: The bi-encoder is initialised with the weights after SimCSE training from §2.1.

Original: For every sentence pair, two sentence embeddings are produced separately by the bi-encoder.
Revised: For every sentence pair, two sentence embeddings are produced separately by the bi-encoder.

Original: The cosine similarity between the two embeddings are regarded as their predictions of the relevance score.
Revised: The cosine similarity between the two embeddings are regarded as their predictions of the relevance score.

Original: The aim is to regress the predictions to the self-labelled scores by the cross-encoder.
Revised: The aim is to regress the predictions to the self-labelled scores by the cross-encoder.

Original: We use a mean square error (MSE) loss:
Revised: We use a mean square error (MSE) loss:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where N is the batch size; x n is the cosine similarity between a sentence pair; y n is the self-labelled ground-truth.
Revised: where N is the batch size; x n is the cosine similarity between a sentence pair; y n is the self-labelled ground-truth.

Original: In experiments, we will show that this resulting bi-encoder is more powerful than the initial bi-encoder.
Revised: In experiments, we will show that this resulting bi-encoder is more powerful than the initial bi-encoder.

Original: Sometimes, the bi-encoder will even outperform its teacher (i.e., the cross-encoder).
Revised: Sometimes, the bi-encoder will even outperform its teacher (i.e., the cross-encoder).

Original: Choice of loss functions: maintaining student-teacher discrepancy is the key.
Revised: Choice of loss functions: maintaining student-teacher discrepancy is the key.

Original: Intuitively, MSE allows for more accurate distillations since it punishes any numerical discrepancies between predicted scores and labels on the instance level.
Revised: Intuitively, MSE allows for more accurate distillations since it punishes any numerical discrepancies between predicted scores and labels on the instance level.

Original: However, in practice, we found that using MSE for bi-tocross distillation aggravates the overfitting issue for cross-encoder: the cross-encoder, with its strong sentence-pair modelling capability, completely overfits to the mapping between concatenated sentence pairs and the pseudo scores.
Revised: However, in practice, we found that using MSE for bi-tocross distillation aggravates the overfitting issue for cross-encoder: the cross-encoder, with its strong sentence-pair modelling capability, completely overfits to the mapping between concatenated sentence pairs and the pseudo scores.

Original: This elimination of discrepancy between model predictions and pseudo labels harms generalisation and the iterative learning cycle cannot continue (as the predicted scores by the student model will be the same as the teacher model).
Revised: This elimination of discrepancy between model predictions and pseudo labels harms generalisation and the iterative learning cycle cannot continue (as the predicted scores by the student model will be the same as the teacher model).

Original: We thus use BCE loss for bi-to-cross since
Revised: We thus use BCE loss for bi-to-cross since

Original: BCE is a more forgiving loss function compared with MSE.
Revised: BCE is a more forgiving loss function compared with MSE.

Original: According to our gradient derivation (App. §A.2), BCE essentially is a temperature-sharpened version of MSE, which is more tolerant towards numerical discrepancies.
Revised: According to our gradient derivation (App. §A.2), BCE essentially is a temperature-sharpened version of MSE, which is more tolerant towards numerical discrepancies.

Original: This prevents cross-encoders from overfitting to the pseudo labels completely.
Revised: This prevents cross-encoders from overfitting to the pseudo labels completely.

Original: Similar issue does not exist for the cross-to-bi distillation as for bi-encoders, two input sequences are separately encoded and the model does not easily overfit to the labels.
Revised: Similar issue does not exist for the cross-to-bi distillation as for bi-encoders, two input sequences are separately encoded and the model does not easily overfit to the labels.

Original: We have a more thorough discussion in App.
Revised: We have a more thorough discussion in App.

Original: §A.2 and Tab. 8 to highlight the rationales of the current configuration.
Revised: §A.2 and Tab. 8 to highlight the rationales of the current configuration.

Original: 2. M UTUAL - DISTILLATION
Revised: 2. M UTUAL - DISTILLATION

Original: The aforementioned self-learning approach has a drawback: since the model regresses to its previous predictions, it tends to reinforce its errors.
Revised: The aforementioned self-learning approach has a drawback: since the model regresses to its previous predictions, it tends to reinforce its errors.

Original: To mitigate this problem, we design a simple mutualdistillation approach to smooth out the errors/biases originated from PLMs.
Revised: To mitigate this problem, we design a simple mutualdistillation approach to smooth out the errors/biases originated from PLMs.

Original: Specifically, we conduct self-distillation on multiple PLMs in parallel (for brevity, we use two in this paper: BERT and
Revised: Specifically, we conduct self-distillation on multiple PLMs in parallel (for brevity, we use two in this paper: BERT and

Original: RoBERTa, however the framework itself is compatible with any number of PLMs).
Revised: RoBERTa, however the framework itself is compatible with any number of PLMs).

Original: Each PLM does not communicate/synchronise with each other except when producing the self-labelled scores.
Revised: Each PLM does not communicate/synchronise with each other except when producing the self-labelled scores.

Original: In mutual-distillation, we use the average predictions of all models as the ground-truth for the next round of learning.
Revised: In mutual-distillation, we use the average predictions of all models as the ground-truth for the next round of learning.

Original: A graphical illustration is shown in Fig.
Revised: A graphical illustration is shown in Fig.

Original: In the following, we call all self-distillation models T RANS -E NCODER (or TE NC for short); all mutual-distillation models T RANS -E
Revised: In the following, we call all self-distillation models T RANS -E NCODER (or TE NC for short); all mutual-distillation models T RANS -E

Original: NCODER -mutual (or TE NC -mutual for short).
Revised: NCODER -mutual (or TE NC -mutual for short).

Original: Note that one
Revised: Note that one

Original: TE NC (-mutual) training produces one best bi-encoder, called TE NC (-mutual) (bi), and one best cross-encoder, called TE NC (-mutual) (cross), based on dev set.
Revised: TE NC (-mutual) training produces one best bi-encoder, called TE NC (-mutual) (bi), and one best cross-encoder, called TE NC (-mutual) (cross), based on dev set.

Original: We report numbers of both.
Revised: We report numbers of both.

Original:  E VALUATION T ASKS
Revised:  E VALUATION T ASKS

Original: Evaluation task: semantic textual similarity (STS).
Revised: Evaluation task: semantic textual similarity (STS).

Original: Following prior works (Reimers & Gurevych, 2019; Liu et al., 2021; Gao et al., 2021), we consider seven STS datasets: SemEval STS 2012-
Revised: Following prior works (Reimers & Gurevych, 2019; Liu et al., 2021; Gao et al., 2021), we consider seven STS datasets: SemEval STS 2012-

Original: (STS12-16, Agirre et al. 2012; 2013; 2014; 2015; 2016), STS Benchmark (STS-B, Cer et al. 2017) and SICK-Relatedness (SICK-R, Marelli et al. 2014).
Revised: (STS12-16, Agirre et al. 2012; 2013; 2014; 2015; 2016), STS Benchmark (STS-B, Cer et al. 2017) and SICK-Relatedness (SICK-R, Marelli et al. 2014).

Original: In all these datasets, sentence pairs are given a human-judged relevance score from 1 to 5.
Revised: In all these datasets, sentence pairs are given a human-judged relevance score from 1 to 5.

Original: We normalise them to 0 and 1 and models are asked to predict the scores.
Revised: We normalise them to 0 and 1 and models are asked to predict the scores.

Original: We report Spearman’s ρ rank correlation between the two.
Revised: We report Spearman’s ρ rank correlation between the two.

Original: Evaluation task: binary classification.
Revised: Evaluation task: binary classification.

Original: We also test T RANS -E NCODER on sentence-pair binary classification tasks where the model has to decide if a sentence-pair has certain relations.
Revised: We also test T RANS -E NCODER on sentence-pair binary classification tasks where the model has to decide if a sentence-pair has certain relations.

Original: We choose (1) the Quora Question Pair (QQP) dataset, requiring a model to judge whether two questions are duplicated; (2) a question-answering entailment dataset QNLI (Rajpurkar et al., 2016; Wang et al., 2019) in which given a question and a sentence the model needs to judge if the sentence answers the question; (3) the Microsoft Research Paraphrase Corpus (MRPC) which asks a model to decide if two sentences are paraphrases of each other.
Revised: We choose (1) the Quora Question Pair (QQP) dataset, requiring a model to judge whether two questions are duplicated; (2) a question-answering entailment dataset QNLI (Rajpurkar et al., 2016; Wang et al., 2019) in which given a question and a sentence the model needs to judge if the sentence answers the question; (3) the Microsoft Research Paraphrase Corpus (MRPC) which asks a model to decide if two sentences are paraphrases of each other.

Original: The ground truth labels of all datasets are either 0 or 1.
Revised: The ground truth labels of all datasets are either 0 or 1.

Original: Following (Li et al., 2020; Liu et al., 2021), we compute Area Under Curve (AUC) scores using the binary labels and the relevance scores predicted by models.
Revised: Following (Li et al., 2020; Liu et al., 2021), we compute Area Under Curve (AUC) scores using the binary labels and the relevance scores predicted by models.

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: Training and evaluation details.
Revised: Training and evaluation details.

Original: For each task, we use all available sentence pairs (from train, development and test sets of all datasets combined) without their labels as training data.
Revised: For each task, we use all available sentence pairs (from train, development and test sets of all datasets combined) without their labels as training data.

Original: The original
Revised: The original

Original: QQP and QNLI datasets are extremely large.
Revised: QQP and QNLI datasets are extremely large.

Original: We thus downsample QQP to have 10k, 1k and 10k pairs for train, dev and test; QNLI to have 10k train set.
Revised: We thus downsample QQP to have 10k, 1k and 10k pairs for train, dev and test; QNLI to have 10k train set.

Original: QNLI does not have public ground truth labels for testing.
Revised: QNLI does not have public ground truth labels for testing.

Original: So, we use its first 1k examples in the official dev set as our dev data and the rest in the official dev set as test data.
Revised: So, we use its first 1k examples in the official dev set as our dev data and the rest in the official dev set as test data.

Original: The dev set for MRPC is its official dev sets.
Revised: The dev set for MRPC is its official dev sets.

Original: The dev set for STS12-16, STS-B and SICK-R is the dev set of STS-B.
Revised: The dev set for STS12-16, STS-B and SICK-R is the dev set of STS-B.

Original: We save one checkpoint for everytraining steps and at the end of each epoch.
Revised: We save one checkpoint for everytraining steps and at the end of each epoch.

Original: We use the dev sets to select the best model for testing.
Revised: We use the dev sets to select the best model for testing.

Original: Dev sets are also used to tune the hyperprameters in each task.
Revised: Dev sets are also used to tune the hyperprameters in each task.

Original: For clear comparison with SimCSE and Mirror-BERT, we use their released checkpoints as initialisation points (i.e., we do not train models in §2.1 ourselves).
Revised: For clear comparison with SimCSE and Mirror-BERT, we use their released checkpoints as initialisation points (i.e., we do not train models in §2.1 ourselves).

Original: We consider four SimCSE variants.
Revised: We consider four SimCSE variants.

Original: Two base variants: SimCSE-BERT-base, SimCSE-RoBERTa-base; and two large variants: SimCSE-BERTlarge, SimCSE-RoBERTa-large.
Revised: Two base variants: SimCSE-BERT-base, SimCSE-RoBERTa-base; and two large variants: SimCSE-BERTlarge, SimCSE-RoBERTa-large.

Original: We consider two Mirror-BERT variants: Mirror-RoBERTa-base and Mirror-RoBERTa-base-drophead.
Revised: We consider two Mirror-BERT variants: Mirror-RoBERTa-base and Mirror-RoBERTa-base-drophead.

Original: For brevity, our analysis in the main text focuses on SimCSE models.
Revised: For brevity, our analysis in the main text focuses on SimCSE models.

Original: We list Mirror-BERT results in Appendix.
Revised: We list Mirror-BERT results in Appendix.

Original: We train T RANS -E NCODER models for 3 cycles on the STS task and 5 cycles on the binary classification tasks.
Revised: We train T RANS -E NCODER models for 3 cycles on the STS task and 5 cycles on the binary classification tasks.

Original: Within each cycle, all bi- and cross-encoders are trained for 10 and 1 epochs respectively for the STS task; 15 and 3 epochs for binary classification.
Revised: Within each cycle, all bi- and cross-encoders are trained for 10 and 1 epochs respectively for the STS task; 15 and 3 epochs for binary classification.

Original: 8 All models use AdamW (Loshchilov & Hutter, 2019) as the optimiser.
Revised: 8 All models use AdamW (Loshchilov & Hutter, 2019) as the optimiser.

Original: In all tasks, unless noted otherwise, we create final representations using [CLS] .
Revised: In all tasks, unless noted otherwise, we create final representations using [CLS] .

Original: We train our base models on a server with 4 * V100 (16GB) GPUs and large models on a server with 8 * A100 (40GB) GPUs.
Revised: We train our base models on a server with 4 * V100 (16GB) GPUs and large models on a server with 8 * A100 (40GB) GPUs.

Original: All main experiments have the same fixed random seed.
Revised: All main experiments have the same fixed random seed.

Original: All other hparams are listed in Appendix.
Revised: All other hparams are listed in Appendix.

Original: Mutual-distillation setup.
Revised: Mutual-distillation setup.

Original: For the two models used for mutual-distillation, they are either (1) the base variant of SimCSE-BERT and SimCSE-RoBERTa or (2) the large variant of the two.
Revised: For the two models used for mutual-distillation, they are either (1) the base variant of SimCSE-BERT and SimCSE-RoBERTa or (2) the large variant of the two.

Original: Theoretically, we could mutually distil even more models but we keep the setup simple for fast and clear comparison.
Revised: Theoretically, we could mutually distil even more models but we keep the setup simple for fast and clear comparison.

Original: Also, since mutual-distillation models use information from both PLMs, we also list (in tables) ensemble results that use the average of the predictions from two T RANS -E NCODER s.
Revised: Also, since mutual-distillation models use information from both PLMs, we also list (in tables) ensemble results that use the average of the predictions from two T RANS -E NCODER s.

Original:  R ESULTS AND D ISCUSSION 5. M AIN R ESULTS
Revised:  R ESULTS AND D ISCUSSION 5. M AIN R ESULTS

Original: STS results (Tab. 1).
Revised: STS results (Tab. 1).

Original: The main results for STS are listed in Tab. 1.
Revised: The main results for STS are listed in Tab. 1.

Original: Compared with the baseline
Revised: Compared with the baseline

Original: SimCSE, T RANS -E NCODER has brought significant improvements across the board.
Revised: SimCSE, T RANS -E NCODER has brought significant improvements across the board.

Original: With various variants of the SimCSE as the base model, T RANS -E NCODER consistently enhances the average score
Revised: With various variants of the SimCSE as the base model, T RANS -E NCODER consistently enhances the average score

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: by approximately 4-5%.
Revised: by approximately 4-5%.

Original: 9 Self-distillation usually brings an improvement of 2-3% (e.g., compare line 1.1, 1.2 with 1 in Tab. 1).
Revised: 9 Self-distillation usually brings an improvement of 2-3% (e.g., compare line 1.1, 1.2 with 1 in Tab. 1).

Original: Further, mutual-distillation brings another boost of 1-3% (e.g., compare line 3.3, 3.4 with 3.1, 3.2 in Tab. 1).
Revised: Further, mutual-distillation brings another boost of 1-3% (e [Table] g., compare line 3.3, 3.4 with 3.1, 3.2 in Tab. 1).

Original: Besides single-model results, the ensemble of mutual-distilled models are clearly better than the ensemble models of either (1) naively averaging the two initial
Revised: Besides single-model results, the ensemble of mutual-distilled models are clearly better than the ensemble models of either (1) naively averaging the two initial

Original: SimCSE models’ scores or (2) averaging predictions of two self-distilled models in a post hoc manner.
Revised: SimCSE models’ scores or (2) averaging predictions of two self-distilled models in a post hoc manner.

Original: This demonstrates the benefit of allowing models to communicate/synchronise with each other in all stages of the self-distillation training.
Revised: This demonstrates the benefit of allowing models to communicate/synchronise with each other in all stages of the self-distillation training.

Original: Binary classification results (Tab. 2).
Revised: Binary classification results (Tab. 2).

Original: On QQP, QNLI
Revised: On QQP, QNLI

Original: [Table]
Revised: [Table]

Original: and MRPC, we observe similar trends as the STS tasks, and the improvement is sometimes even more significant (see full table in Appendix
Revised: and MRPC, we observe similar trends as the STS tasks, and the improvement is sometimes even more significant (see full table in Appendix

Original: Tab. 11).
Revised: Tab. 11).

Original: We conjecture it is due to the fact that T RANS -E NCODER training also adapts models to the task domain (since tasks like QQP and QNLI are very different from Wikipedia data, which is where
Revised: We conjecture it is due to the fact that T RANS -E NCODER training also adapts models to the task domain (since tasks like QQP and QNLI are very different from Wikipedia data, which is where

Original: SimCSE models are trained on).
Revised: SimCSE models are trained on).

Original: We will discuss the domain adaptation effect in more details in analysis.
Revised: We will discuss the domain adaptation effect in more details in analysis.

Original: Another point worth discussing is the benefit of mutual-distillation is more inconsistent on binary classification tasks, compared with STS tasks (again, see full table Tab. 11 in Appendix).
Revised: Another point worth discussing is the benefit of mutual-distillation is more inconsistent on binary classification tasks, compared with STS tasks (again, see full table Tab. 11 in Appendix).

Original: We
Revised: We

Original: [Table] [Table]
Revised: [Table] [Table]

Original: suspect it is due to the performance imbalance between the BERT and RoBERTa variants on these tasks (a significant performance gap exists in the first place).
Revised: suspect it is due to the performance imbalance between the BERT and RoBERTa variants on these tasks (a significant performance gap exists in the first place).

Original: Domain transfer setup (Tab. 3).
Revised: Domain transfer setup (Tab. 3).

Original: One of the key questions we are keen to find out is how much of T RANS -E NCODER ’s success on in-domain tasks generalises/transfers to other tasks/domains.
Revised: One of the key questions we are keen to find out is how much of T RANS -E NCODER ’s success on in-domain tasks generalises/transfers to other tasks/domains.

Original: Specifically, does T RANS -E NCODER create universally better bi- and cross-encoders, or is it only task/domain-specific?
Revised: Specifically, does T RANS -E NCODER create universally better bi- and cross-encoders, or is it only task/domain-specific?

Original: To answer the question, we directly test models trained with the STS task data on binary classification tasks.
Revised: To answer the question, we directly test models trained with the STS task data on binary classification tasks.

Original: For bi-encoders, i.e., TE NC (bi), the results are inconsistent across setups.
Revised: For bi-encoders, i.e., TE NC (bi), the results are inconsistent across setups.

Original: These results hint that training on in-domain data is important for optimal performance gains for unsupervised bi-encoders.
Revised: These results hint that training on in-domain data is important for optimal performance gains for unsupervised bi-encoders.

Original: This is in line with the finding of (Liu et al., 2021).
Revised: This is in line with the finding of (Liu et al., 2021).

Original: However, for cross-encoders, surprisingly, we see extremely good transfer performance.
Revised: However, for cross-encoders, surprisingly, we see extremely good transfer performance.

Original: W/ or w/o mutualdistillation, TE NC (cross) models outperform the SimCSE baselines by large margins, despite the fact that tasks like QQP and QNLI are of a very different domain compared with STS.
Revised: W/ or w/o mutualdistillation, TE NC (cross) models outperform the SimCSE baselines by large margins, despite the fact that tasks like QQP and QNLI are of a very different domain compared with STS.

Original: In fact, they are sometimes even better than models tuned on the in-domain task data (c.f. Tab. 11).
Revised: In fact, they are sometimes even better than models tuned on the in-domain task data (c.f. Tab. 11).

Original: This finding has deep implications.
Revised: This finding has deep implications.

Original: It hints that the cross-encoder architecture (cross-attention) is by design very suitable for sentence-pair modelling (i.e., strong inductive bias is already in the architecture design), and once its sentence-pair modelling capability is ‘activated’, it is an extremely powerful universal representation that can be transferred to other tasks.
Revised: It hints that the cross-encoder architecture (cross-attention) is by design very suitable for sentence-pair modelling (i.e., strong inductive bias is already in the architecture design), and once its sentence-pair modelling capability is ‘activated’, it is an extremely powerful universal representation that can be transferred to other tasks.

Original: The reason that STS models outperform in-domain models in some tasks hint that more sentence-pairs as raw training data is welcome (since the STS train set is larger than the binary classification ones).
Revised: The reason that STS models outperform in-domain models in some tasks hint that more sentence-pairs as raw training data is welcome (since the STS train set is larger than the binary classification ones).

Original: As future work, we plan to explore mining sentence pairs for T RANS -E
Revised: As future work, we plan to explore mining sentence pairs for T RANS -E

Original: NCODER learning from a general corpus (such as Wikipedia).
Revised: NCODER learning from a general corpus (such as Wikipedia).

Original: Both lexical- (such as BM25) and semantic-level (neural models such as bi-encoders)
Revised: Both lexical- (such as BM25) and semantic-level (neural models such as bi-encoders)

Original: IR systems could be leveraged here.
Revised: IR systems could be leveraged here.

Original: 5. D ISCUSSIONS AND A NALYSIS
Revised: 5. D ISCUSSIONS AND A NALYSIS

Original: In this section we discuss some interesting phenomena we observed, justify design choices we made in an empirical way, and also show a more fine-grained understanding of what exactly happens when using T RANS -E NCODER .
Revised: In this section we discuss some interesting phenomena we observed, justify design choices we made in an empirical way, and also show a more fine-grained understanding of what exactly happens when using T RANS -E NCODER .

Original: Initialise with the original weights or train sequentially?
Revised: Initialise with the original weights or train sequentially?

Original: (Fig. 3a)
Revised: (Fig. 3a)

Original: As mentioned in §2.2 and §2.3, we initialise bi-encoders with SimCSE/Mirror-BERT weights and cross-encoders with PLMs’ weights (we call this strategy refreshing ).
Revised: As mentioned in §2.2 and §2.3, we initialise bi-encoders with SimCSE/Mirror-BERT weights and cross-encoders with PLMs’ weights (we call this strategy refreshing ).

Original: We have also tried maintaining the weights of bi- and cross-encoders sequentially.
Revised: We have also tried maintaining the weights of bi- and cross-encoders sequentially.

Original: I.e., we initialise the cross-encoder’s weights with the bi-encoder which just created the pseudo labels and vice versa (we call this strategy sequential ).
Revised: I.e., we initialise the cross-encoder’s weights with the bi-encoder which just created the pseudo labels and vice versa (we call this strategy sequential ).

Original: The benefit of doing so is keeping all the legacy knowledge learned in the process of self-distillation in the weights.
Revised: The benefit of doing so is keeping all the legacy knowledge learned in the process of self-distillation in the weights.

Original: As
Revised: As

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: suggested in Fig.
Revised: suggested in Fig.

Original: 3a (also in Appendix Tab. 13), refreshing is slightly better than sequential .
Revised: 3a (also in Appendix Tab. 13), refreshing is slightly better than sequential .

Original: We suspect it is because initialising with original weights alleviate catastrophic forgetting problem in self-supervised learning (SSL), similar to the moving average and stop-gradient strategy used in contrastive SSL methods such as MoCo (He et al., 2020).
Revised: We suspect it is because initialising with original weights alleviate catastrophic forgetting problem in self-supervised learning (SSL), similar to the moving average and stop-gradient strategy used in contrastive SSL methods such as MoCo (He et al., 2020).

Original: Self-distillation without alternating between different task formulations?
Revised: Self-distillation without alternating between different task formulations?

Original: (Fig.
Revised: (Fig.

Original: 3b)
Revised: 3b)

Original: If we disregard the bi- and cross-encoder alternating training paradigm but using the same bi-encoder architecture all along, the model is then similar to the standard self-distillation model (c.f. Figure 1 by Mobahi et al.
Revised: If we disregard the bi- and cross-encoder alternating training paradigm but using the same bi-encoder architecture all along, the model is then similar to the standard self-distillation model (c.f. Figure 1 by Mobahi et al.

Original: Theoretically, standard self-distillation could also have helped, especially considering that it adapts the model to in-domain data.
Revised: Theoretically, standard self-distillation could also have helped, especially considering that it adapts the model to in-domain data.

Original: However, as seen in Fig.
Revised: However, as seen in Fig.

Original: 3b (and also Appendix Tab. 14), standard self-distillation lags behind T RANS -E NCODER by a significant amount, and sometimes underperforms the base model.
Revised: 3b (and also Appendix Tab. 14), standard self-distillation lags behind T RANS -E NCODER by a significant amount, and sometimes underperforms the base model.

Original: Standard self-distillation essentially ensembles different checkpoints (i.e., the same model at different training phases provides different ‘views’ of the data).
Revised: Standard self-distillation essentially ensembles different checkpoints (i.e., the same model at different training phases provides different ‘views’ of the data).

Original: T RANSE NCODER can be seen as a type of self-distillation model, but instead of using previous models to provide views of data, we use different task formulations (i.e., bi- and cross-encoder) to provide different views of the data.
Revised: T RANSE NCODER can be seen as a type of self-distillation model, but instead of using previous models to provide views of data, we use different task formulations (i.e., bi- and cross-encoder) to provide different views of the data.

Original: The fact that standard self-distillation underperforms T RANS -E
Revised: The fact that standard self-distillation underperforms T RANS -E

Original: NCODER by large margins suggests the effectiveness of our proposed bi- and cross-encoder training scheme.
Revised: NCODER by large margins suggests the effectiveness of our proposed bi- and cross-encoder training scheme.

Original: How many cycles is optimal?
Revised: How many cycles is optimal?

Original: (Fig. 4)
Revised: (Fig. 4)

Original: Our approach iteratively bootstraps the performance of both bi- and cross-encoders.
Revised: Our approach iteratively bootstraps the performance of both bi- and cross-encoders.

Original: But how many iterations (cycles) are enough?
Revised: But how many iterations (cycles) are enough?

Original: We find that it heavily depends on the model and dataset, and could be unpredictable (since the ‘optimality’ depends on a relatively small dev set).
Revised: We find that it heavily depends on the model and dataset, and could be unpredictable (since the ‘optimality’ depends on a relatively small dev set).

Original: In Fig.
Revised: In Fig.

Original: 4, we plot the TE NC models’ (self-distillation only) performances on dev sets in three tasks: STS, QQP, and QNLI.
Revised: 4, we plot the TE NC models’ (self-distillation only) performances on dev sets in three tasks: STS, QQP, and QNLI.

Original: In general, the patterns are different across datasets and models.
Revised: In general, the patterns are different across datasets and models.

Original: Does more training data help?
Revised: Does more training data help?

Original: (Fig. 5)
Revised: (Fig. 5)

Original: We control the number of training samples drawn and test the T RANS -E NCODER model (using SimCSE-BERT-base) on STS test sets.
Revised: We control the number of training samples drawn and test the T RANS -E NCODER model (using SimCSE-BERT-base) on STS test sets.

Original: The average performance on all STS tasks are reported in Fig.
Revised: The average performance on all STS tasks are reported in Fig.

Original: 5, with three runs of different random seeds.
Revised: 5, with three runs of different random seeds.

Original: There are in total 37,081 data points from STS.
Revised: There are in total 37,081 data points from STS.

Original: We only draw from these in-domain data until there is none left.
Revised: We only draw from these in-domain data until there is none left.

Original: It can be seen that for in-domain training, more data points are usually always beneficial.
Revised: It can be seen that for in-domain training, more data points are usually always beneficial.

Original: Next, we test with more samples drawn from another task, SNLI (Bowman et al., 2015), containing abundant sentence-pairs.
Revised: Next, we test with more samples drawn from another task, SNLI (Bowman et al., 2015), containing abundant sentence-pairs.

Original: The performance of the model further increases till 30k extra data points and starts to gradually decrease after that point.
Revised: The performance of the model further increases till 30k extra data points and starts to gradually decrease after that point.

Original: However, it is worth mentioning that there will be discrepancy of such trends across tasks and models.
Revised: However, it is worth mentioning that there will be discrepancy of such trends across tasks and models.

Original: And when needed, one can mine as many sentence-pairs as desired from a general corpus (such as Wikipedia) for T RANS -E NCODER learning.
Revised: And when needed, one can mine as many sentence-pairs as desired from a general corpus (such as Wikipedia) for T RANS -E NCODER learning.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is closely related to (1) unsupervised sentence representation learning; (2) sentence-pair tasks using bi- and cross-encoders; and (3) self-distillation.
Revised: Our work is closely related to (1) unsupervised sentence representation learning; (2) sentence-pair tasks using bi- and cross-encoders; and (3) self-distillation.

Original: Each has a large body of work which we can only provide a highly selected summary below.
Revised: Each has a large body of work which we can only provide a highly selected summary below.

Original: Unsupervised sentence representations.
Revised: Unsupervised sentence representations.

Original: Prior to the emergence of large PLMs such as BERT, popular unsupervised sentence embedding methods included Skip-Thoughts (Kiros et al., 2015),
Revised: Prior to the emergence of large PLMs such as BERT, popular unsupervised sentence embedding methods included Skip-Thoughts (Kiros et al., 2015),

Original: FastSent (Hill et al., 2016), and Quick-Thoughts (Logeswaran & Lee, 2018).
Revised: FastSent (Hill et al., 2016), and Quick-Thoughts (Logeswaran & Lee, 2018).

Original: They all exploit the co-occurrence statistics of sentences in large corpora (i.e., sentences under similar contexts have similar meanings).
Revised: They all exploit the co-occurrence statistics of sentences in large corpora (i.e., sentences under similar contexts have similar meanings).

Original: A recent paper DeCLUTR (Giorgi et al., 2021) follows such idea and formulate the training as a contrastive learning task.
Revised: A recent paper DeCLUTR (Giorgi et al., 2021) follows such idea and formulate the training as a contrastive learning task.

Original: Very recently, there has been a growing interest in using individual raw sentences for self-supervised contrastive learning on top of PLMs.
Revised: Very recently, there has been a growing interest in using individual raw sentences for self-supervised contrastive learning on top of PLMs.

Original: Contrastive Tension (Carlsson et al., 2021), Mirror-BERT (Liu et al., 2021), SimCSE (Gao et al., 2021), Self-Guided
Revised: Contrastive Tension (Carlsson et al., 2021), Mirror-BERT (Liu et al., 2021), SimCSE (Gao et al., 2021), Self-Guided

Original: Contrastive Learning (Kim et al., 2021), ConSERT (Yan et al., 2021), BSL (Zhang et al., 2021), inter alia , all follow such an idea.
Revised: Contrastive Learning (Kim et al., 2021), ConSERT (Yan et al., 2021), BSL (Zhang et al., 2021), inter alia , all follow such an idea.

Original: Specifically, data augmentations on either input or feature space are used to create two views of the same sentence for contrastive learning.
Revised: Specifically, data augmentations on either input or feature space are used to create two views of the same sentence for contrastive learning.

Original: In our experiments, we use such methods as an initial step for creating a reasonably strong bi-encoder.
Revised: In our experiments, we use such methods as an initial step for creating a reasonably strong bi-encoder.

Original: Cross- and bi-encoder for pairwise sentence tasks.
Revised: Cross- and bi-encoder for pairwise sentence tasks.

Original: Sentence-BERT (Reimers & Gurevych, 2019) points out the speed advantage of bi-encoders and proposes to train BERT with a siamese architecture for sentence-pair tasks.
Revised: Sentence-BERT (Reimers & Gurevych, 2019) points out the speed advantage of bi-encoders and proposes to train BERT with a siamese architecture for sentence-pair tasks.

Original: Poly-encoder (Humeau et al., 2020) proposes a hybrid architecture in which intermediate encodings are cached and a final layer of cross-attention is used to exploit intersentence interactions.
Revised: Poly-encoder (Humeau et al., 2020) proposes a hybrid architecture in which intermediate encodings are cached and a final layer of cross-attention is used to exploit intersentence interactions.

Original: Conceptually similar to our work, Di-pair (Chen et al., 2020), augmented
Revised: Conceptually similar to our work, Di-pair (Chen et al., 2020), augmented

Original: Sentence-BERT (Thakur et al., 2021), RocketQA (Qu et al., 2021), and DvBERT (Cheng, 2012)
Revised: Sentence-BERT (Thakur et al., 2021), RocketQA (Qu et al., 2021), and DvBERT (Cheng, 2021)

Original: distil knowledge in supervised cross-encoders to bi-encoders.
Revised: distil knowledge in supervised cross-encoders to bi-encoders.

Original: We show that it is also possible to distil knowledge from bi- to cross-encoders.
Revised: We show that it is also possible to distil knowledge from bi- to cross-encoders.

Original: Self-distillation.
Revised: Self-distillation.

Original: In standard self-distillation, models of the same architecture train on predictions by previous checkpoint(s) with the same task formulation (Furlanello et al., 2018; Mobahi et al., 2020).
Revised: In standard self-distillation, models of the same architecture train on predictions by previous checkpoint(s) with the same task formulation (Furlanello et al., 2018; Mobahi et al., 2020).

Original: However, the standard approach does not alter the training paradigm (task formulation) as we do (biand cross-encoder training).
Revised: However, the standard approach does not alter the training paradigm (task formulation) as we do (biand cross-encoder training).

Original: In experiments, we have shown that alternating task formulation is the key to the success of our method.
Revised: In experiments, we have shown that alternating task formulation is the key to the success of our method.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose T RANS -E NCODER , an unsupervised approach of training bi- and cross-encoders for sentence-pair tasks.
Revised: We propose T RANS -E NCODER , an unsupervised approach of training bi- and cross-encoders for sentence-pair tasks.

Original: The core idea of T RANS -E NCODER is self-distillation in a smart way: alternatively training a bi-encoder and a cross-encoder (of the same architecture) with pseudo-labels created from the other.
Revised: The core idea of T RANS -E NCODER is self-distillation in a smart way: alternatively training a bi-encoder and a cross-encoder (of the same architecture) with pseudo-labels created from the other.

Original: We also propose a mutual-distillation extension to mutually bootstrap two self-distillation models trained in parallel.
Revised: We also propose a mutual-distillation extension to mutually bootstrap two self-distillation models trained in parallel.

Original: On sentence-pair tasks including sentence similarity, question de-duplication, question-answering entailment, and paraphrase identification, we show strong empirical evidence verifying the effectiveness of T RANS -E NCODER .
Revised: On sentence-pair tasks including sentence similarity, question de-duplication, question-answering entailment, and paraphrase identification, we show strong empirical evidence verifying the effectiveness of T RANS -E NCODER .

Original: We also found the surprisingly strong generalisation capability of our trained cross-encoders across domains and tasks.
Revised: We also found the surprisingly strong generalisation capability of our trained cross-encoders across domains and tasks.

Original: Finally, we conduct thorough ablation studies and analysis to verify our design choices and shed insight on the model mechanism.
Revised: Finally, we conduct thorough ablation studies and analysis to verify our design choices and shed insight on the model mechanism.

Revised:  A CKNOWLEDGEMENTS

Revised: We thank Hanchen Wang, Giorgio Giannone, Arushi Goel for providing useful feedback during internal discussions.

Original: A A PPENDIX A. R ESULTS WITH O THER C ONTRASTIVE L EARNING M ETHODS
Revised: A A PPENDIX A. R ESULTS WITH O THER C ONTRASTIVE L EARNING M ETHODS

Original: In the main text, we experimented with SimCSE, but in theory T RANS -E NCODER is compatible with any unsupervised contrastive bi-encoder learning models.
Revised: In the main text, we experimented with SimCSE, but in theory T RANS -E NCODER is compatible with any unsupervised contrastive bi-encoder learning models.

Original: In this section, we experiment with Mirror-BERT (which is similar to SimCSE) and also two additional unsupervised contrastive models Contrastive-Tension (Carlsson et al., 2021) and DeCLUTR (Giorgi et al., 2020) to verify the robustness of T RANS -E NCODER .
Revised: In this section, we experiment with Mirror-BERT (which is similar to SimCSE) and also two additional unsupervised contrastive models Contrastive-Tension (Carlsson et al., 2021) and DeCLUTR (Giorgi et al., 2021) to verify the robustness of T RANS -E NCODER .

Original: As seen in Tab. 4 and Tab.
Revised: As seen in Tab. 4 and Tab.

Original: [Table]
Revised: [Table]

Original: T RANS -E NCODER brings consistent and significant improvements to all base models, similar to what we have observed on SimCSE models.
Revised: T RANS -E NCODER brings consistent and significant improvements to all base models, similar to what we have observed on SimCSE models.

Original: After T RANS -E NCODER training, the Mirror-BERT-based models perform even slightly better than SimCSE-based models, on both the STS and binary classification tasks.
Revised: After T RANS -E NCODER training, the Mirror-BERT-based models perform even slightly better than SimCSE-based models, on both the STS and binary classification tasks.

Original: We suspect it is due to that SimCSE checkpoints are trained for more iterations (with the contrastive learning objective) and thus are more prone to overfit the training corpus.
Revised: We suspect it is due to that SimCSE checkpoints are trained for more iterations (with the contrastive learning objective) and thus are more prone to overfit the training corpus.

Original: A. M ORE D ISCUSSIONS AND A NALYSIS
Revised: A. M ORE D ISCUSSIONS AND A NALYSIS

Original: How robust is T RANS -E NCODER ?
Revised: How robust is T RANS -E NCODER ?

Original: (Tab. 6)
Revised: (Tab. 6)

Original: As mentioned, our main experiments used one fixed random seed.
Revised: As mentioned, our main experiments used one fixed random seed.

Original: Due to the scale of the experiments, performing multiple runs with different random seeds for all setups is overly expensive.
Revised: Due to the scale of the experiments, performing multiple runs with different random seeds for all setups is overly expensive.

Original: We pick STS and the base variants of BERT and RoBERTa
Revised: We pick STS and the base variants of BERT and RoBERTa

Original: [Table] [Table]
Revised: [Table] [Table]

Original: as two examples for showing results of five runs (shown in Tab. 6).
Revised: as two examples for showing results of five runs (shown in Tab. 6).

Original: In general
Revised: In general

Original: [Table]
Revised: [Table]

Original: the T RANS -E NCODER approach is quite robust and stable.
Revised: the T RANS -E NCODER approach is quite robust and stable.

Original: STS and SICK-R as different tasks (Tab. 7, Fig. 6).
Revised: STS and SICK-R as different tasks (Tab. 7, Fig. 6).

Original: It is worth noting that in the STS main results table (Tab. 1), SICK-R is the only dataset where T RANS -E NCODER models lead to performance worse than the baselines in some settings (e.g., on SimCSE-BERT-large).
Revised: It is worth noting that in the STS main results table (Tab. 1), SICK-R is the only dataset where T RANS -E NCODER models lead to performance worse than the baselines in some settings (e [Table] g., on SimCSE-BERT-large).

Original: We believe it is due to that SICK-R is essentially a different task from STS2012-2016 and STS-B since SICK-R is labelled with a different aim in mind.
Revised: We believe it is due to that SICK-R is essentially a different task from STS2012-2016 and STS-B since SICK-R is labelled with a different aim in mind.

Original: It focuses specifically on compositional distributional semantics.
Revised: It focuses specifically on compositional distributional semantics.

Original: 10 To verify our claim, we train SICK-R as a standalone task using its official dev set instead of STS-B. All sentence pairs from SICK-R are used as raw training data.
Revised: 10 To verify our claim, we train SICK-R as a standalone task using its official dev set instead of STS-B. All sentence pairs from SICK-R are used as raw training data.

Original: The results are shown in Tab. 7 and Fig.
Revised: The results are shown in Tab. 7 and Fig.

Original: As shown, around 3-4% gain can be obtained by switching to training on SICK-R only, confirming the different nature of the two tasks.
Revised: As shown, around 3-4% gain can be obtained by switching to training on SICK-R only, confirming the different nature of the two tasks.

Original: This suggests that generalisation is also dependent on the standard of how relevance is defined between sequences in the target dataset, and training with an in-domain dev set maybe crucial when the domain shift between training and testing data is significant.
Revised: This suggests that generalisation is also dependent on the standard of how relevance is defined between sequences in the target dataset, and training with an in-domain dev set maybe crucial when the domain shift between training and testing data is significant.

Original: Gradient derivation of BCE and MSE loss.
Revised: Gradient derivation of BCE and MSE loss.

Original: We show that BCE is essentially a normalised version of MSE where the gradient of x i is scaled by the sigmoid function ( σ ( · ) ).
Revised: We show that BCE is essentially a normalised version of MSE where the gradient of x i is scaled by the sigmoid function ( σ ( · ) ).

Original: Here are the gradients of the two loss functions:
Revised: Here are the gradients of the two loss functions:

Original: [Equation]
Revised: [Equation]

Original: As can be seen, the only difference lies in whether x i is scaled by σ ( · ) .
Revised: As can be seen, the only difference lies in whether x i is scaled by σ ( · ) .

Original: This gives a greater degree of freedom to L BCE especially at the areas in the two ends (close to 0 or 1 ): a large change of x i results
Revised: This gives a greater degree of freedom to L BCE especially at the areas in the two ends (close to 0 or 1 ): a large change of x i results

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: in a small change of σ ( x i ) , making the approximation of y i an easier task.
Revised: in a small change of σ ( x i ) , making the approximation of y i an easier task.

Original: In other words, in L BCE , it is in general easier for the model to push the gradient to 0 , while in L MSE , only a strict numerical match leads to 0 gradient.
Revised: In other words, in L BCE , it is in general easier for the model to push the gradient to 0 , while in L MSE , only a strict numerical match leads to 0 gradient.

Original: Loss function configurations (Tab. 8).
Revised: Loss function configurations (Tab. 8).

Original: We experimented with loss function configurations comprehensively and found that it requires caution when choosing learning objectives for cross- to bi-encoder and bi- to cross-encoder distillation.
Revised: We experimented with loss function configurations comprehensively and found that it requires caution when choosing learning objectives for cross- to bi-encoder and bi- to cross-encoder distillation.

Original: As mentioned in the main text and the paragraph above, choosing
Revised: As mentioned in the main text and the paragraph above, choosing

Original: MSE for bi-to-cross distillation causes severe overfitting to the pseudo scores and choosing BCE for cross-to-bi distillation fail to make the model converge.
Revised: MSE for bi-to-cross distillation causes severe overfitting to the pseudo scores and choosing BCE for cross-to-bi distillation fail to make the model converge.

Original: As a result, our configuration of using BCE and MSE for bi-to-cross and cross-to-bi distillations respectively becomes the only plausible solution (see Tab. 8).
Revised: As a result, our configuration of using BCE and MSE for bi-to-cross and cross-to-bi distillations respectively becomes the only plausible solution (see Tab. 8).

Original: For the regression loss used for cross-to-bi distillation, besides MSE, we also experimented with several other regression loss functions such as Log-Cosh but found no improvement.
Revised: For the regression loss used for cross-to-bi distillation, besides MSE, we also experimented with several other regression loss functions such as Log-Cosh but found no improvement.

Original: Do sentence pairs leak label information?
Revised: Do sentence pairs leak label information?

Original: (Tab. 9)
Revised: (Tab. 9)

Original: T RANS -E NCODER has leveraged the sentence pairs given by the task.
Revised: T RANS -E NCODER has leveraged the sentence pairs given by the task.

Original: One concern is that the sentence pairs may be positively correlated and the model may have exploited this na¨ıve bias.
Revised: One concern is that the sentence pairs may be positively correlated and the model may have exploited this na¨ıve bias.

Original: We highlight that this work does not have this concern as the tasks in our study test the relative similarity ranking of sentence pairs under the evaluation metrics (both spearman’s rank used in STS and AUC used in binary classification are ranking-based).
Revised: We highlight that this work does not have this concern as the tasks in our study test the relative similarity ranking of sentence pairs under the evaluation metrics (both spearman’s rank used in STS and AUC used in binary classification are ranking-based).

Original: Therefore, the overall correlation distribution of sentence pairs does not reveal what is tested by the tasks in this study.
Revised: Therefore, the overall correlation distribution of sentence pairs does not reveal what is tested by the tasks in this study.

Original: To further verify the model has not learned label distribution from the task, we run a controlled setting where we treat all sentence pairs from the tasks as (pseudo-)positive pairs and apply a standard contrastive finetuning framework (i.e. Mirror-BERT and SimCSE finetuning) with
Revised: To further verify the model has not learned label distribution from the task, we run a controlled setting where we treat all sentence pairs from the tasks as (pseudo-)positive pairs and apply a standard contrastive finetuning framework (i.e. Mirror-BERT and SimCSE finetuning) with

Original: Eq.
Revised: Eq.

Original: (1) as the loss on these pairs.
Revised: (1) as the loss on these pairs.

Original: Performance improvement from this finetuning setup will indicate that the overall positive correlation from the sentence pairs can be exploited by the model.
Revised: Performance improvement from this finetuning setup will indicate that the overall positive correlation from the sentence pairs can be exploited by the model.

Original: Results are shown in Tab. 9.
Revised: Results are shown in Tab. 9.

Original: Contrastive methods on pseudo-positive pairs almost always 11 underperform the original SimCSE and Mirror-BERT checkpoints significantly while T RANS -E NCODER models improve the checkpoints by large margins.
Revised: Contrastive methods on pseudo-positive pairs almost always 11 underperform the original SimCSE and Mirror-BERT checkpoints significantly while T RANS -E NCODER models improve the checkpoints by large margins.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: A. T RAINING T IME
Revised: A. T RAINING T IME

Original: Overall runtime.
Revised: Overall runtime.

Original: We list the runtime of T RANS -E NCODER models on different tasks and configurations in Tab. 10.
Revised: We list the runtime of T RANS -E NCODER models on different tasks and configurations in Tab. 10.

Original: All base models can be trained under 2.5 hours and large models can be trained within 5 hours.
Revised: All base models can be trained under 2.5 hours and large models can be trained within 5 hours.

Original: TE NC -mutual models usually take twice the amount of time training a self-distillation model.
Revised: TE NC -mutual models usually take twice the amount of time training a self-distillation model.

Original: Also, TE NC -mutual models require two GPUs while self-distillation models need only one.
Revised: Also, TE NC -mutual models require two GPUs while self-distillation models need only one.

Original: We have tested running T RANS -E NCODER on three types of Nvidia GPUs: V100 (16 GB), A100 (
Revised: We have tested running T RANS -E NCODER on three types of Nvidia GPUs: V100 (16 GB), A100 (

Original: GB) and 3090 (24 GB).
Revised: GB) and 3090 (24 GB).

Original: The runtimes are similar.
Revised: The runtimes are similar.

Original: Cross-encoder vs. bi-encoder.
Revised: Cross-encoder vs. bi-encoder.

Original: For training one epoch, bi-encoder takes significantly less time than cross-encoders.
Revised: For training one epoch, bi-encoder takes significantly less time than cross-encoders.

Original: However, bi-encoders take much longer time to converge (15 epochs vs. 3 epochs on binary classification tasks and 10 epochs vs. 1 epoch on STS; see Tab. 16).
Revised: However, bi-encoders take much longer time to converge (15 epochs vs. 3 epochs on binary classification tasks and 10 epochs vs. 1 epoch on STS; see Tab. 16).

Original: As a result, in practice, bi-to-cross and cross-to-bi distillations take similar amount of time.
Revised: As a result, in practice, bi-to-cross and cross-to-bi distillations take similar amount of time.

Original: A. F ULL T ABLES
Revised: A. F ULL T ABLES

Original: Here, we list the full tables of SimCSE binary classification results (Tab. 11); SimCSE domain transfer results (Tab. 12); and full tables for ablation studies (Tab. 13, Tab. 14), which are presented as figures in the main texts in Fig.
Revised: Here, we list the full tables of SimCSE binary classification results (Tab. 11); SimCSE domain transfer results (Tab. 12); and full tables for ablation studies (Tab. 13, Tab. 14), which are presented as figures in the main texts in Fig.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: + TE NC (bi) 82.10 75.
Revised: + TE NC (bi) 82.10 75.

Original: + TE NC (cross) 82.10 75.
Revised: + TE NC (cross) 82.10 75.

Original: + TE NC -mutual (bi) 
Revised: + TE NC -mutual (bi) 84.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: + TE NC -mutual (cross) 84.29 77.
Revised: + TE NC -mutual (cross) 84.29 77.

Original: SimCSE-RoBERTa-base 81.82 73.
Revised: SimCSE-RoBERTa-base 81.82 73.

Original: + TE NC (bi) 84.13 77.
Revised: + TE NC (bi) 84.13 77.

Original: + TE NC (cross) 85.16 80.
Revised: + TE NC (cross) 85.16 80.

Original: + TE NC -mutual (bi) 84.36 77.
Revised: + TE NC -mutual (bi) 84.36 77.

Original: + TE NC -mutual (cross) 84.73 77.
Revised: + TE NC -mutual (cross) 84.73 77.

Original: SimCSE-BERT-large 82.42 72.
Revised: SimCSE-BERT-large 82.42 72.

Original: + TE NC (bi) 84.53 78.
Revised: + TE NC (bi) 84.53 78.

Original: + TE NC (cross) 84.18 79.
Revised: + TE NC (cross) 84.18 79.

Original: + TE NC -mutual (bi) 85.72 81.
Revised: + TE NC -mutual (bi) 85.72 81.

Original: + TE NC -mutual (cross) 86.55 81.
Revised: + TE NC -mutual (cross) 86.55 81.

Original: SimCSE-RoBERTa-large 82.99 75.
Revised: SimCSE-RoBERTa-large 82.99 75.

Original: + TE NC (bi) 85.65 84.
Revised: + TE NC (bi) 85.65 84.

Original: + TE NC (cross) 86.16 84.
Revised: + TE NC (cross) 86.16 84.

Original: + TE NC -mutual (bi) 85.65 81.
Revised: + TE NC -mutual (bi) 85.65 81.

Original: + TE NC -mutual (cross) 86.38 83.
Revised: + TE NC -mutual (cross) 86.38 83.

Original: [Table]
Revised: [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Original: NC [Table] base
Revised: NC -base (

Original: [Table]
Revised: [Table]

Original: A. D ATA S TATISTICS
Revised: A. D ATA S TATISTICS

Original: A complete listing of train/dev/test stats of all used datasets can be found in Tab. 15.
Revised: A complete listing of train/dev/test stats of all used datasets can be found in Tab. 15.

Original: Note that for
Revised: Note that for

Original: STS 2012-2016, we dropped all sentence-pairs without a valid score.
Revised: STS 2012-2016, we dropped all sentence-pairs without a valid score.

Original: And the train sets include all sentence pairs (w/o labels) regardless of split in each task.
Revised: And the train sets include all sentence pairs (w/o labels) regardless of split in each task.

Original: A. H YPERPARAMETERS
Revised: A. H YPERPARAMETERS

Original: The full listing of hyperparameters is shown in Tab. 16.
Revised: The full listing of hyperparameters is shown in Tab. 16.

Original: Since performing a grid-search for all these hparams is ultra-expensive, we mainly used the default hparams from the sentence-BERT library.
Revised: Since performing a grid-search for all these hparams is ultra-expensive, we mainly used the default hparams from the sentence-BERT library.

Original: The hparams for TE NC and TE NC -mutual are the same.
Revised: The hparams for TE NC and TE NC -mutual are the same.

Original: A. P RE - TRAINED E NCODERS
Revised: A. P RE - TRAINED E NCODERS

Original: A complete listing of URLs for all used PLMs is provided in Tab. 17.
Revised: A complete listing of URLs for all used PLMs is provided in Tab. 17.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]