Original: C ROSS -A RCHITECTURE D ISTILLATION U SING B IDIRECTIONAL CMOW E MBEDDINGS
Revised: C ROSS -A RCHITECTURE D ISTILLATION U SING B IDIRECTIONAL CMOW E MBEDDINGS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks.
Revised: Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks.

Original: However, their sheer size is prohibitive for small laboratories or deployment on mobile devices.
Revised: However, their sheer size is prohibitive for small laboratories or deployment on mobile devices.

Original: Approaches like pruning and distillation reduce the model size but typically retain the same model architecture.
Revised: Approaches like pruning and distillation reduce the model size but typically retain the same model architecture.

Original: In contrast, we explore distilling PreLMs into a different, more efﬁcient architecture
Revised: In contrast, we explore distilling PreLMs into a different, more efﬁcient architecture

Original: CMOW, which embeds each word as a matrix and uses matrix multiplication to encode sequences.
Revised: CMOW, which embeds each word as a matrix and uses matrix multiplication to encode sequences.

Original: We extend the CMOW architecture and its CMOW/CBOW-Hybrid variant with a bidirectional component, per-token representations for distillation during pretraining, and a two-sequence encoding scheme that facilitates downstream tasks on sentence pairs such as natural language inferencing.
Revised: We extend the CMOW architecture and its CMOW/CBOW-Hybrid variant with a bidirectional component, per-token representations for distillation during pretraining, and a two-sequence encoding scheme that facilitates downstream tasks on sentence pairs such as natural language inferencing.

Original: Our results show that the embedding-based models yield scores comparable to DistilBERT on
Revised: Our results show that the embedding-based models yield scores comparable to DistilBERT on

Original: QQP and RTE, while using only half of its parameters and providing three times faster inference speed.
Revised: QQP and RTE, while using only half of its parameters and providing three times faster inference speed.

Original: We match or exceed the scores of ELMo, and only fall behind more expensive models on linguistic acceptability.
Revised: We match or exceed the scores of ELMo, and only fall behind more expensive models on linguistic acceptability.

Original: Still, our distilled bidirectional CMOW/CBOW-Hybrid model more than doubles the scores on linguistic acceptability compared to previous cross-architecture distillation approaches.
Revised: Still, our distilled bidirectional CMOW/CBOW-Hybrid model more than doubles the scores on linguistic acceptability compared to previous cross-architecture distillation approaches.

Original: Furthermore, our experiments conﬁrm the positive effects of bidirection and the two-sequence encoding scheme.
Revised: Furthermore, our experiments conﬁrm the positive effects of bidirection and the two-sequence encoding scheme.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Large pretrained language models (Devlin et al., 2019; Raffel et al., 2020)
Revised: Large pretrained language models (Devlin et al., 2019; Raffel et al., 2020)

Original: (PreLMs) have emerged as de-facto standard methods for natural language processing (Wang et al., 2018; 2019).
Revised: (PreLMs) have emerged as de-facto standard methods for natural language processing (Wang et al., 2018; 2019).

Original: The common strategy is to pretrain models on enormous amounts of unlabeled text before ﬁne-tuning them for downstream tasks.
Revised: The common strategy is to pretrain models on enormous amounts of unlabeled text before ﬁne-tuning them for downstream tasks.

Original: However, the drawback of PreLMs is that the models are becoming larger and larger with up to several billions of parameters (Brown et al., 2020).
Revised: However, the drawback of PreLMs is that the models are becoming larger and larger with up to several billions of parameters (Brown et al., 2020).

Original: This comes with high environmental and economic costs (Strubell et al., 2019) and shifts the development and research in the hands of a few global players only (Bommasani et al., 2021, pp.
Revised: This comes with high environmental and economic costs (Strubell et al., 2019) and shifts the development and research in the hands of a few global players only (Bommasani et al., 2021, pp.

Original: Even though a single pretrained model can be reused for multiple downstream tasks, the sheer model size is often prohibitive.
Revised: Even though a single pretrained model can be reused for multiple downstream tasks, the sheer model size is often prohibitive.

Original: The immense resource requirements prevent using those models in small-scale laboratories and on mobile devices, which is tied to privacy concerns (Sanh et al., 2020b).
Revised: The immense resource requirements prevent using those models in small-scale laboratories and on mobile devices, which is tied to privacy concerns (Sanh et al., 2020b).

Original: There is a need for more efﬁcient models or compressed versions of large models to make AI research more inclusive and energy-friendly, while fostering deployment in applications.
Revised: There is a need for more efﬁcient models or compressed versions of large models to make AI research more inclusive and energy-friendly, while fostering deployment in applications.

Original: Reducing the size of
Revised: Reducing the size of

Original: PreLMs using knowledge distillation (Hinton et al., 2015) or model compression (Bucila et al., 2006) is an active area of research (Sanh et al., 2020a; Jiao et al., 2020; Sun et al., 2020).
Revised: PreLMs using knowledge distillation (Hinton et al., 2015) or model compression (Bucila et al., 2006) is an active area of research (Sanh et al., 2020a; Jiao et al., 2020; Sun et al., 2020).

Original: Both knowledge distillation and model compression can be described as teacher-student setups.
Revised: Both knowledge distillation and model compression can be described as teacher-student setups.

Original: The student is trained to imitate the predictions of the teacher while using less resources.
Revised: The student is trained to imitate the predictions of the teacher while using less resources.

Original: Typically, a large PreLM takes the role of the teacher while the student is a smaller version of the same architecture.
Revised: Typically, a large PreLM takes the role of the teacher while the student is a smaller version of the same architecture.

Original: Sharing the same architecture between student and teacher enables using dedicated distillation techniques, e. g., aligning the representations of intermediate layers (Sanh et al., 2020a; Sun et al., 2020).
Revised: Sharing the same architecture between student and teacher enables using dedicated distillation techniques, e. g., aligning the representations of intermediate layers (Sanh et al., 2020a; Sun et al., 2020).

Original: However, using more efﬁcient architectures as student has already shown promising results such as the task-speciﬁc distillation approaches by Tang et al.
Revised: However, using more efﬁcient architectures as student has already shown promising results such as the task-speciﬁc distillation approaches by Tang et al.

Original: and Wasserblat et al.
Revised: and Wasserblat et al.

Original: In their works, the student models are LSTMs (Hochreiter & Schmidhuber, 1997) or models based on a continuous bag-of-words representation (CBOW) (Collobert & Weston, 2008; Mikolov et al., 2013).
Revised: In their works, the student models are LSTMs (Hochreiter & Schmidhuber, 1997) or models based on a continuous bag-of-words representation (CBOW) (Collobert & Weston, 2008; Mikolov et al., 2013).

Original: On the one hand, LSTMs are difﬁcult to parallelize as they need at least O ( n ) sequential steps to encode a sequence of length n .
Revised: On the one hand, LSTMs are difﬁcult to parallelize as they need at least O ( n ) sequential steps to encode a sequence of length n .

Original: On the other hand, CBOW-based models are not order-aware , i. e., cannot distinguish sentences with the same words but in different order (“cat eats mouse” vs. “mouse eats cat” are treated equivalent).
Revised: On the other hand, CBOW-based models are not order-aware , i. e., cannot distinguish sentences with the same words but in different order (“cat eats mouse” vs. “mouse eats cat” are treated equivalent).

Original: There are, however, more efﬁcient models such as Mai et al.
Revised: There are, however, more efﬁcient models such as Mai et al.

Original: (2019)’s continual multiplication of words (CMOW) that do capture word order by representing each token as a matrix, instead of a vector as in CBOW.
Revised: (2019)’s continual multiplication of words (CMOW) that do capture word order by representing each token as a matrix, instead of a vector as in CBOW.

Original: A sequence in CMOW is modeled by the non-commutative matrix multiplication, which makes the encoding of a sequence becomes dependent on the word order.
Revised: A sequence in CMOW is modeled by the non-commutative matrix multiplication, which makes the encoding of a sequence becomes dependent on the word order.

Original: We denote such models as matrix embeddings .
Revised: We denote such models as matrix embeddings .

Original: The present work investigates how order-aware matrix embeddings can be used as student models in cross-architecture distillation from large PreLM teachers.
Revised: The present work investigates how order-aware matrix embeddings can be used as student models in cross-architecture distillation from large PreLM teachers.

Original: Thus, we complement the existing body of works that focused predominantly on same-architecture distillation.
Revised: Thus, we complement the existing body of works that focused predominantly on same-architecture distillation.

Original: All previous cross-architecture approaches are task-speciﬁc, whereas we also explore general distillation.
Revised: All previous cross-architecture approaches are task-speciﬁc, whereas we also explore general distillation.

Original: We aim to understand to which extent order-aware embeddings are suited to capture the teacher signal of a large PreLM such as BERT.
Revised: We aim to understand to which extent order-aware embeddings are suited to capture the teacher signal of a large PreLM such as BERT.

Original: To this end, we extend Mai et al.
Revised: To this end, we extend Mai et al.

Original: (2019)’s CMOW/CBOW-Hybrid model, which is a hybrid variant unifying the strength of CBOW and CMOW, with a bidirectional representation of the sequences.
Revised: (2019)’s CMOW/CBOW-Hybrid model, which is a hybrid variant unifying the strength of CBOW and CMOW, with a bidirectional representation of the sequences.

Original: Furthermore, we add the ability to emit per-token representations to facilitate using a modern masked language model objective (Devlin et al., 2019).
Revised: Furthermore, we add the ability to emit per-token representations to facilitate using a modern masked language model objective (Devlin et al., 2019).

Original: We investigate both general distillation , i. e., the distillation is applied during pretraining on unlabeled text, as well as task-speciﬁc distillation , when an already ﬁne-tuned PreLM is distilled on a per-task basis.
Revised: We investigate both general distillation , i. e., the distillation is applied during pretraining on unlabeled text, as well as task-speciﬁc distillation , when an already ﬁne-tuned PreLM is distilled on a per-task basis.

Original: We further introduce a two-sentence encoding scheme to CMOW such that it can deal with similarity and natural language inferencing tasks.
Revised: We further introduce a two-sentence encoding scheme to CMOW such that it can deal with similarity and natural language inferencing tasks.

Original: This improves the performance by 32% compared to a naive joint encoding.
Revised: This improves the performance by 32% compared to a naive joint encoding.

Original: Our results show that large PreLMs can be distilled into efﬁcient order-aware embedding models and achieve performance comparable to ELMo (Peters et al., 2018) on the GLUE benchmark.
Revised: Our results show that large PreLMs can be distilled into efﬁcient order-aware embedding models and achieve performance comparable to ELMo (Peters et al., 2018) on the GLUE benchmark.

Original: On certain tasks, embedding-based models even challenge other size-reduced BERT models such as
Revised: On certain tasks, embedding-based models even challenge other size-reduced BERT models such as

Original: DistillBERT.
Revised: DistillBERT.

Original: In summary, our contributions are:
Revised: In summary, our contributions are:

Original: • We extend order-aware embedding models with bidirection and make them amenable for masked language model pretraining.
Revised: • We extend order-aware embedding models with bidirection and make them amenable for masked language model pretraining.

Original: • We explore using order-aware embedding models as student models in a cross-architecture distillation setup with BERT as teacher and compare general and task-speciﬁc distillation.
Revised: • We explore using order-aware embedding models as student models in a cross-architecture distillation setup with BERT as teacher and compare general and task-speciﬁc distillation.

Original: • We introduce the ﬁrst encoding scheme that enables CMOW/CBOW-Hybrid to deal with two-sentence tasks (32% increase over the naive approach).
Revised: • We introduce the ﬁrst encoding scheme that enables CMOW/CBOW-Hybrid to deal with two-sentence tasks (32% increase over the naive approach).

Original: • Our results show that the best distilled embedding models can be on-par with more expensive models such as ELMo or DistilBERT on certain tasks of the GLUE benchmark, while having a much higher encoding speed (thrice as high as DistilBERT).
Revised: • Our results show that the best distilled embedding models can be on-par with more expensive models such as ELMo or DistilBERT on certain tasks of the GLUE benchmark, while having a much higher encoding speed (thrice as high as DistilBERT).

Original: Below, we introduce our embedding models, our cross-architecture distillation setup, and our twosequence encoding scheme.
Revised: Below, we introduce our embedding models, our cross-architecture distillation setup, and our twosequence encoding scheme.

Original: The experimental procedure is described in Section 3.
Revised: The experimental procedure is described in Section 3.

Original: The results are reported in Section 4 and discussed in Section 5, where we also relate our work to the literature.
Revised: The results are reported in Section 4 and discussed in Section 5, where we also relate our work to the literature.

Original:  M ETHODS
Revised:  M ETHODS

Original: First, we introduce our bidirectional extension to the CMOW/CBOW-Hybrid model.
Revised: First, we introduce our bidirectional extension to the CMOW/CBOW-Hybrid model.

Original: Subsequently, we introduce our approach for cross-architecture distillation that we use during the pretraining and ﬁne-tuning stages.
Revised: Subsequently, we introduce our approach for cross-architecture distillation that we use during the pretraining and ﬁne-tuning stages.

Original: Finally, we introduce a two-sentence encoding scheme for order-aware embedding models that is crucial for ﬁne-tuning on downstream tasks with paired sentences.
Revised: Finally, we introduce a two-sentence encoding scheme for order-aware embedding models that is crucial for ﬁne-tuning on downstream tasks with paired sentences.

Original: 2. E XTENDING THE O RDER - AWARE E MBEDDING M ODELS
Revised: 2. E XTENDING THE O RDER - AWARE E MBEDDING M ODELS

Original: We extend the CMOW/CBOW-Hybrid embeddings of Mai et al.
Revised: We extend the CMOW/CBOW-Hybrid embeddings of Mai et al.

Original: (2019) with bidirection and the ability to emit per-token representations as a foundation for cross-architecture distillation.
Revised: (2019) with bidirection and the ability to emit per-token representations as a foundation for cross-architecture distillation.

Original: Below, we brieﬂy recapture the CMOW/CBOW-Hybrid embeddings before describing our extensions.

Original: Baseline Model: CMOW/CBOW-Hybrid CMOW/CBOW-Hybrid embeddings are a combination of matrix embeddings and vector embeddings.
Revised: CMOW/CBOWHybrid embeddings, our baseline model , are a combination of matrix embeddings and vector embeddings.

Original: Compared to vector-only embeddings, the word order can be captured because matrix multiplication is non-commutative.
Revised: Compared to vector-only embeddings, the word order can be captured because matrix multiplication is non-commutative.

Original: Given a sequence of n tokens, matrix-space embeddings X j ∈ R d × d for each different token j , and vector-space embeddings x j ∈ R d vec , the CMOW/CBOW-Hybrid embedding of a sequence of length n is the multiplication of embedding matrices
Revised: Given a sequence of n tokens, matrix-space embeddings X j ∈ R d × d for each different token j , and vector-space embeddings x j ∈ R d vec , the CMOW/CBOWHybrid embedding of a sequence of length n is the multiplication of embedding matrices

Original: X i concatenated (symbol ·||· ) to the summation of the embedding vectors x i :
Revised: X i concatenated (symbol ·||· ) to the summation of the embedding vectors x i :

Original: H (CMOW) :

Original: = X (CMOW) · X (CMOW) · · · X (CMOW) n h (CBOW) :

Original: = (cid:88) 1 ≤ j

Original: [Equation]
Revised: [Equation]

Original: n x (CBOW)

Original: j h (Hybrid) :

Original: = ﬂatten

Original: [Equation]

Original: ) where ﬂatten collapses the matrix into a vector.
Revised: where ﬂatten collapses the matrix into a vector.

Original: Following Mai et al.
Revised: The original work on CMOW (Mai et al., 2019) has extensively analyzed the CMOW and CBOW components individually and found that joint training is generally preferable. The CMOW and CBOW components can have different dimensionalities, as they are combined via concatentation.

Original: (2019), we initialize eachembedding matrix X j as identity plus Gaussian noise I d + N (0 , σ 2init ) . We use σ init = 0 .
Revised: We initialize each matrix X j as identity plus Gaussian noise I d + N (0 , σ 2init ) with σ init = 0 .

Original: [Figure]
Revised: [Figure]

Original: In this example, the model predicts the masked token at position 4 by concatenating forward and backward matrix embeddings, which are then fed 
Revised: In this example, the model predicts the masked token at position 4 by concatenating forward and backward matrix embeddings, which are then fed into a masked language modeling head.

Original: Proposed Model: Bidirectional CMOW/CBOW-Hybrid Inspired by the success of bidirection in RNNs (Schuster & Paliwal, 1997), LSTMs (Peters et al., 2018), and Transformers (Devlin et al., 2019), we extend CMOW by a bidirectional component.
Revised: Proposed Model: Bidirectional CMOW/CBOW-Hybrid Inspired by the success of bidirection in RNNs (Schuster & Paliwal, 1997), LSTMs (Peters et al., 2018), and Transformers (Devlin et al., 2019), we extend CMOW by a bidirectional component.

Original: Hence, we introduce a second set of matrixspace embeddings that are multiplied in reverse order.
Revised: Hence, we introduce a second set of matrixspace embeddings that are multiplied in reverse order.

Original: We then havetwo sets of matrix embeddings for each word: one  for the forward direction X (fw) ∈ R n vocab ×
Revised: We then have one matrix embedding for the forward direction X (fw) ∈ R n vocab ×

Original: d × d and one for the backward direction X (bw) ∈ R n vocab ×
Revised: d × d and one for the backward direction X (bw) ∈ R n vocab ×

Original: d × d .
Revised: d × d .

Original: Then, we concatenate forward and backward directions.
Revised: Then, we concatenate forward and backward directions.

Original: Figure 1 illustratesthe bidirectional CMOW component.
Revised: Figure 1 illustrates bidirectional CMOW.

Original: Furthermore, we emit one representation per token position i , which allows training with a masked language model objective (Devlin et al., 2019).
Revised: Furthermore, we emit one representation per token position i , which allows training with a masked language model objective (Devlin et al., 2019).

Original: Thus, we are able to make use of the BERT teacher signal for pretraining.
Revised: Thus, we are able to make use of the BERT teacher signal for pretraining.

Original: Since we can reuse computations, O ( n ) matrix multiplications are sufﬁcient to encode a sequence of length n .
Revised: Since we can reuse computations, O ( n ) matrix multiplications are sufﬁcient to encode a sequence of length n .

Original: For these intermediate representations, we also modify the CBOW component in a way that it yields partial sums for the forward and backward directions.
Revised: For these intermediate representations, we also modify the CBOW component in a way that it yields partial sums for the forward and backward directions.

Original: Formally, we compute the CMOW/CBOW-Hybrid representation as follows:
Revised: Formally, we compute the CMOW/CBOW-Hybrid representation as follows:

Original: [Equation]
Revised: [Equation]

Original: For ﬁne-tuning on tasks with full sentences as input, e. g., natural language inferencing, we do not need per-token representations.
Revised: For ﬁne-tuning on tasks with full sentences as input, e. g., natural language inferencing, we do not need per-token representations.

Original: In this case, we compute the representation of a token sequence as:
Revised: In this case, we compute the representation of a token sequence as:

Original: [Equation]
Revised: [Equation]

Original: Note that the forward and backward directions of the embedding vectors h (CBOW) conﬂate to equivalent formulas when we encode entire sequences.
Revised: Note that the forward and backward directions of the embedding vectors h (CBOW) conﬂate to equivalent formulas when we encode entire sequences.

Original: Thus, we only need to include a single CBOW representation along with the two CMOW components that do yield different results for the forward and backward direction.
Revised: Thus, we only need to include a single CBOW representation along with the two CMOW components that do yield different results for the forward and backward direction.

Original: At inference time, the model is parallelizable along the sequential dimension.
Revised: At inference time, the model is parallelizable along the sequential dimension.

Original: For regularization, we apply a mild dropout (p=0.1) on both the embeddings and their aggregated representations during pretraining.
Revised: For regularization, we apply a mild dropout (p=0.1) on both the embeddings and their aggregated representations during pretraining.

Original: Then, we feed them into a linear masked language modeling head, see Figure 1, or an MLP classiﬁcation head to tackle the downstream tasks.
Revised: Then, we feed them into a linear masked language modeling head, see Figure 1, or an MLP classiﬁcation head to tackle the downstream tasks.

Original: We have also experimented with linear, LSTM, and CNN classiﬁers, which are described in Appendix A. We chose an MLP because it adds nonlinearity to the model without introducing further complexity.
Revised: We have also experimented with linear, LSTM, and CNN classiﬁers, which are described in Appendix A. We chose an MLP because it adds nonlinearity to the model without introducing further complexity.

Original: The MLP has led to the best average performance across all GLUE tasks (we report the results in Appendix C).
Revised: The MLP has led to the best average performance across all GLUE tasks (we report the results in Appendix C).

Original: 2. C ROSS -A RCHITECTURE D ISTILLATION
Revised: 2. C ROSS -A RCHITECTURE D ISTILLATION

Original: A central question of our research is whether we can distill a large PreLM, e. g., BERT, into more efﬁcient, non-transformer architectures such as the proposed bidirectional CMOW/CBOW-Hybrid model.
Revised: A central question of our research is whether we can distill a large PreLM, e. g., BERT, into more efﬁcient, non-transformer architectures such as the proposed bidirectional CMOW/CBOW-Hybrid model.

Original: This requires a cross-architecture distillation approach, which we describe below.
Revised: This requires a cross-architecture distillation approach, which we describe below.

Original: In general, the idea of knowledge distillation is to compress the knowledge of a large teacher model into a smaller student model (Hinton et al., 2015; Bucila et al., 2006) .
Revised: In general, the idea of knowledge distillation is to compress the knowledge of a large teacher model into a smaller student model (Hinton et al., 2015; Bucila et al., 2006) .

Original: It involves a loss function L that is a combination of two loss terms, i. e., L = α · L hard
Revised: It involves a loss function L that is a combination of two loss terms, i. e., L = α · L hard

Original: + (1 − α ) ·
Revised: + (1 − α ) ·

Original: L soft with weighting parameter α .
Revised: L soft with weighting parameter α .

Original: L hard denotes the cross-entropy loss with respect to the ground truth and L soft =
Revised: L hard denotes the cross-entropy loss with respect to the ground truth and L soft =

Original: Σ i t i · log( s i ) is the cross-entropy between student logits s and the teacher signal t .
Revised: Σ i t i · log( s i ) is the cross-entropy between student logits s and the teacher signal t .

Original: Optionally, the softmax within
Revised: Optionally, the softmax within

Original: L soft is ﬂattened by a temperature parameter T .
Revised: L soft is ﬂattened by a temperature parameter T .

Original: We distinguish general distillation , where BERT’s teacher signal is only used during pretraining, and task-speciﬁc distillation , where the BERT teacher signal is used during ﬁne-tuning for the downstream (see related work).
Revised: We distinguish general distillation , where BERT’s teacher signal is only used during pretraining, and task-speciﬁc distillation , where the BERT teacher signal is used during ﬁne-tuning for the downstream (see related work).

Original: Considering our goal to design a cross-architecture distillation, the general distillation approach has the conceptual beneﬁt that the teacher model is not needed for ﬁne-tuning.
Revised: Considering our goal to design a cross-architecture distillation, the general distillation approach has the conceptual beneﬁt that the teacher model is not needed for ﬁne-tuning.

Original: Thus, the student model is capable of tackling downstream tasks without supervision of the large teacher.
Revised: Thus, the student model is capable of tackling downstream tasks without supervision of the large teacher.

Original: This has the beneﬁt that one does not need to carry around the BERT model for adaption to every new downstream task.
Revised: This has the beneﬁt that one does not need to carry around the BERT model for adaption to every new downstream task.

Original: Above, we have introduced the ability to emit per-token representations with lightweight (bidirectional) CMOW/CBOW-Hybrid embedding models.
Revised: Above, we have introduced the ability to emit per-token representations with lightweight (bidirectional) CMOW/CBOW-Hybrid embedding models.

Original: This enables us now to use BERT’s teacher signal during pretraining together with a masked language modeling objective.
Revised: This enables us now to use BERT’s teacher signal during pretraining together with a masked language modeling objective.

Original: With other words, this enables us to perform cross-architecture distillation with matrix embeddings.
Revised: With other words, this enables us to perform cross-architecture distillation with matrix embeddings.

Original: We consider three variants of cross-architecture distillation in our experiments: a) When using general distillation, depicted in Figure 1, BERT acts as a teacher during pretraining and the model is ﬁne-tuned to downstream tasks on its own.
Revised: We consider three variants of cross-architecture distillation in our experiments: a) When using general distillation, depicted in Figure 1, BERT acts as a teacher during pretraining and the model is ﬁne-tuned to downstream tasks on its own.

Original: b) For task-speciﬁc distillation, BERT acts as a teacher during ﬁne-tuning as shown in Figure 2.
Revised: b) For task-speciﬁc distillation, BERT acts as a teacher during ﬁne-tuning as shown in Figure 2.

Original: For this case, we have the option of either b1) starting with pretrained embeddings (from general distillation, i. e., the a) variant), or b2) starting from scratch with randomly initialized
Revised: For this case, we have the option of either b1) starting with pretrained embeddings (from general distillation, i. e., the a) variant), or b2) starting from scratch with randomly initialized

Original: embeddings
Revised: embeddings

Original: 2. T WO -S EQUENCE E NCODING WITH M ATRIX E MBEDDINGS
Revised: 2. T WO -S EQUENCE E NCODING WITH M ATRIX E MBEDDINGS

Original: When ﬁne-tuning our matrix embeddings to downstream tasks, we can deviate from BERT’s input processing, even if BERT is used as a teacher.
Revised: When ﬁne-tuning our matrix embeddings to downstream tasks, we can deviate from BERT’s input processing, even if BERT is used as a teacher.

Original: This is because the distillation loss is computed per sentence (pair) and not per token.
Revised: This is because the distillation loss is computed per sentence (pair) and not per token.

Original: The input processing of BERT encodes two sequences by joining them into one sequence.
Revised: The input processing of BERT encodes two sequences by joining them into one sequence.

Original: For example, in a natural language inferencing task, there is a sentence A that potentially entails a sentence B , which is encoded as one sequence using a special separator token.
Revised: For example, in a natural language inferencing task, there is a sentence A that potentially entails a sentence B , which is encoded as one sequence using a special separator token.

Original: This encoding scheme is less useful to our matrix embeddings without any attention component since the order-aware matrix multiplications would blend the representation of the two sequences.
Revised: This encoding scheme is less useful to our matrix embeddings without any attention component since the order-aware matrix multiplications would blend the representation of the two sequences.

Original: To develop an appropriate two-sequence encoding scheme for matrix embeddings, we take inspiration from the pre-transformer era, e. g., Mou et al.
Revised: To develop an appropriate two-sequence encoding scheme for matrix embeddings, we take inspiration from the pre-transformer era, e. g., Mou et al.

Original: (2016), and from SentenceBERT (Reimers & Gurevych, 2019).
Revised: (2016), and from SentenceBERT (Reimers & Gurevych, 2019).

Original: The key idea is to encode two sentences A and B separately before combining them.
Revised: The key idea is to encode two sentences A and B separately before combining them.

Original: As combination operation, we use the absolute elementwise difference and concatenate it to the representations of A and B , which we denote as DiffCat:
Revised: As combination operation, we use the absolute elementwise difference and concatenate it to the representations of A and B , which we denote as DiffCat:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We illustrate this separate encoding scheme during task-speciﬁc distillation in Figure 2.
Revised: We illustrate this separate encoding scheme during task-speciﬁc distillation in Figure 2.

Original: The rationale for using a concatenation of both sequence representations along with their difference is that we add a component for the similarity of the two sequence representations, without loss of expressive power.
Revised: The rationale for using a concatenation of both sequence representations along with their difference is that we add a component for the similarity of the two sequence representations, without loss of expressive power.

Original:  E XPERIMENTAL P ROCEDURE
Revised:  E XPERIMENTAL P ROCEDURE

Original: The experimental procedure is divided in pretraining on unlabeled text and ﬁne-tuning on the downstream tasks.
Revised: The experimental procedure is divided in pretraining on unlabeled text and ﬁne-tuning on the downstream tasks.

Original: We provide the details for these two stages and close the section by outlining the downstream tasks and evaluation measures.
Revised: We provide the details for these two stages and close the section by outlining the downstream tasks and evaluation measures.

Original: 3. P RETRAINING AND G ENERAL D ISTILLATION
Revised: 3. P RETRAINING AND G ENERAL D ISTILLATION

Original: In the pretraining stage, as shown in Figure 1, we train our proposed bidirectional CMOW/CBOWHybrid model with a masked language modeling objective (MLM) (Devlin et al., 2019) on large amounts of unlabeled text.
Revised: In the pretraining stage, as shown in Figure 1, we train our proposed bidirectional CMOW/CBOWHybrid model with a masked language modeling objective (MLM) (Devlin et al., 2019) on large amounts of unlabeled text.

Original: The MLM objective is to predict left-out words from their context.
Revised: The MLM objective is to predict left-out words from their context.

Original: We put equal weights on the MLM objective and the teacher signal from BERT ( α = 0 . 5 ).
Revised: We put equal weights on the MLM objective and the teacher signal from BERT ( α = 0 . 5 ).

Original: As suggested by Liu et al.
Revised: As suggested by Liu et al.

Original: and Sanh et al.
Revised: and Sanh et al.

Original: (2020a), we do not use the next-sentence-prediction objective of
Revised: (2020a), we do not use the next-sentence-prediction objective of

Original: BERT, but only the MLM objective.
Revised: BERT, but only the MLM objective.

Original: As datasets for pretraining, we use a combination of English Wikipedia and Toronto Books (Zhu et al., 2015), as used in original BERT.
Revised: As datasets for pretraining, we use a combination of English Wikipedia and Toronto Books (Zhu et al., 2015), as used in original BERT.

Original: To reduce the environmental footprint of our experiments, we have only pretrained a single bidirectional CMOW/CBOW-Hybrid model with BERT-base as a teacher on the full unlabeled training data, after pre-experiments on 10% of the training data showed that the selected CMOW/CBOW-Hybrid with distillation exceeded the performance of the baseline.
Revised: To reduce the environmental footprint of our experiments, we have only pretrained a single bidirectional CMOW/CBOW-Hybrid model with BERT-base as a teacher on the full unlabeled training data, after pre-experiments on 10% of the training data showed that the selected CMOW/CBOW-Hybrid with distillation exceeded the performance of the baseline.

Original: We use matrix embeddings of size 20 × 20 = 400 ( d = 20 ) for both the CMOW directions (forward and backward) and vector embeddings of size d vec = 400 .
Revised: We use matrix embeddings of size 20 × 20 = 400 ( d = 20 ) for both the CMOW directions (forward and backward) and vector embeddings of size d vec = 400 .

Original: We use BERT’s tokenizer and its vocabulary for both, the teacher and student.
Revised: We use BERT’s tokenizer and its vocabulary for both, the teacher and student.

Original: The BERT tokenizer relies primarily on the WordPiece algorithm (Wu et al., 2016), which yields a high coverage while maintaining a small vocabulary.
Revised: The BERT tokenizer relies primarily on the WordPiece algorithm (Wu et al., 2016), which yields a high coverage while maintaining a small vocabulary.

Original: 3. F INE - TUNING AND T ASK - SPECIFIC D ISTILLATION
Revised: 3. F INE - TUNING AND T ASK - SPECIFIC D ISTILLATION

Original: In the ﬁne-tuning stage, as shown in Figure 2, the pretrained model is adapted for each downstream task individually.
Revised: In the ﬁne-tuning stage, as shown in Figure 2, the pretrained model is adapted for each downstream task individually.

Original: The training objective for ﬁne-tuning is either cross-entropy with the ground truth (in general distillation) or a mixture of the ground truth loss and cross-entropy with respect to the teacher’s logits (in task-speciﬁc distillation).
Revised: The training objective for ﬁne-tuning is either cross-entropy with the ground truth (in general distillation) or a mixture of the ground truth loss and cross-entropy with respect to the teacher’s logits (in task-speciﬁc distillation).

Original: Again we put equal weight on ground truth and teacher signal along ( α = 0 . 5 ) with unit temperature.
Revised: Again we put equal weight on ground truth and teacher signal along ( α = 0 . 5 ) with unit temperature.

Original: To facilitate distillation on regression tasks, we follow
Revised: To facilitate distillation on regression tasks, we follow

Original: Raffel et al.
Revised: Raffel et al.

Original: and
Revised: and

Original: cast STS-B from regression to classiﬁcation by binning the scores into intervals of 0 . 2 .
Revised: cast STS-B from regression to classiﬁcation by binning the scores into intervals of 0 . 2 .

Original: To encode the inputs for two-sequence tasks, we use a sequential encoding similar to
Revised: To encode the inputs for two-sequence tasks, we use a sequential encoding similar to

Original: BERT and the proposed DiffCat encoding (see Section 2.3).
Revised: BERT and the proposed DiffCat encoding (see Section 2.3).

Original: For task-speciﬁc distillation, we employ an uncased BERT-base model 1 from the Huggingface repository that has already been ﬁne-tuned for each task of the GLUE benchmark.
Revised: For task-speciﬁc distillation, we employ an uncased BERT-base model 1 from the Huggingface repository that has already been ﬁne-tuned for each task of the GLUE benchmark.

Original: We have ﬁne-tuned the BERT model ourselves on the tasks STS-B, where we applied binning, and MNLI, since the
Revised: We have ﬁne-tuned the BERT model ourselves on the tasks STS-B, where we applied binning, and MNLI, since the

Original: Huggingface model led to subpar results.
Revised: Huggingface model led to subpar results.

Original: We use the same ﬁne-tuned BERT model as a teacher for all reported results with task-speciﬁc distillation.
Revised: We use the same ﬁne-tuned BERT model as a teacher for all reported results with task-speciﬁc distillation.

Original: We seek a fair comparison between unidirectional CMOW/CBOW-Hybrid baseline model and our bidirectional model.
Revised: We seek a fair comparison between unidirectional CMOW/CBOW-Hybrid baseline model and our bidirectional model.

Original: As such, we allow both models to equally beneﬁt from the BERT’s teacher signal during ﬁne-tuning.
Revised: As such, we allow both models to equally beneﬁt from the BERT’s teacher signal during ﬁne-tuning.

Original: For this comparison, we use random initialization for both models because the Mai et al.
Revised: For this comparison, we use random initialization for both models because the Mai et al.

Original: (2019)’s pretrained embeddings would come with a different vocabulary that covers only 53% of the one of BERT.
Revised: (2019)’s pretrained embeddings would come with a different vocabulary that covers only 53% of the one of BERT.

Original: Throughout the other experiments, we initialize our bidirectional
Revised: Throughout the other experiments, we initialize our bidirectional

Original: CMOW/CBOW-Hybrid with the pretrained embeddings from general distillation, while we isolate the effect of task-speciﬁc distillation in a dedicated experiment.
Revised: CMOW/CBOW-Hybrid with the pretrained embeddings from general distillation, while we isolate the effect of task-speciﬁc distillation in a dedicated experiment.

Original: For hyperparameter optimization, we tune learning rates in the range [10 − 3 , 10 − 6 ] .
Revised: For hyperparameter optimization, we tune learning rates in the range [10 − 3 , 10 − 6 ] .

Original: In total, we have conducted 306 training and evaluation runs for hyperparameter optimization of each GLUE task.
Revised: In total, we have conducted 306 training and evaluation runs for hyperparameter optimization of each GLUE task.

Original: To determine the best model, we use each task’s evaluation measure on the development set.
Revised: To determine the best model, we use each task’s evaluation measure on the development set.

Original: We run each model for 20 epochs with early stopping ( 5 epochs patience).
Revised: We run each model for 20 epochs with early stopping ( 5 epochs patience).

Original: We select appropriate batch sizes on the basis of preliminary experiments and training data sizes.
Revised: We select appropriate batch sizes on the basis of preliminary experiments and training data sizes.

Original: For a more detailed discussion, we refer to Appendix B.
Revised: For a more detailed discussion, we refer to Appendix B.

Original: The hyperparameters of the best-performing models are reported in Appendix C.
Revised: The hyperparameters of the best-performing models are reported in Appendix C.

Original: We have further experimented with data augmentation and using exclusively the teacher signal during task-speciﬁc distillation ( α = 1 ).
Revised: We have further experimented with data augmentation and using exclusively the teacher signal during task-speciﬁc distillation ( α = 1 ).

Original: In some tasks, we could further increase the results by a small margin, but found no consistent improvement.
Revised: In some tasks, we could further increase the results by a small margin, but found no consistent improvement.

Original: These results are also reported in Appendix C.
Revised: These results are also reported in Appendix C.

Original: 3. D OWNSTREAM T ASKS AND M EASURES
Revised: 3. D OWNSTREAM T ASKS AND M EASURES

Original: We use the GLUE benchmark (Wang et al., 2018) to evaluate our models.
Revised: We use the GLUE benchmark (Wang et al., 2018) to evaluate our models.

Original: The GLUE benchmark consists of nine tasks for English language comprehension (Wang et al., 2018).
Revised: The GLUE benchmark consists of nine tasks for English language comprehension (Wang et al., 2018).

Original: These tasks comprise natural language inference (MNLI-m, QNLI, WNLI, RTE), sentence similarity (QQP, STS-B, MRPC), linguistic acceptibility (CoLA), and sentiment analysis (SST-2).
Revised: These tasks comprise natural language inference (MNLI-m, QNLI, WNLI, RTE), sentence similarity (QQP, STS-B, MRPC), linguistic acceptibility (CoLA), and sentiment analysis (SST-2).

Original: All tasks are based on pairs of sentences except for CoLA and SST-2, which are single-sentence tasks.
Revised: All tasks are based on pairs of sentences except for CoLA and SST-2, which are single-sentence tasks.

Original: The GLUE benchmark explicitly encourages using different ﬁne-tuning strategies for different tasks.
Revised: The GLUE benchmark explicitly encourages using different ﬁne-tuning strategies for different tasks.

Original: For our evaluation, we use the GLUE development set along with its task-speciﬁc measures.
Revised: For our evaluation, we use the GLUE development set along with its task-speciﬁc measures.

Original: As such, the performance on all four NLI tasks as well as SST-2 is measured with accuracy.
Revised: As such, the performance on all four NLI tasks as well as SST-2 is measured with accuracy.

Original: CoLA is evaluated by Matthews correlation coefﬁcient.
Revised: CoLA is evaluated by Matthews correlation coefﬁcient.

Original: Similarity tasks are measured by the average of Pearson and Spearman correlation for the STS-B task, and as the average of accuracy and F 1 -score for MRPC and QQP.
Revised: Similarity tasks are measured by the average of Pearson and Spearman correlation for the STS-B task, and as the average of accuracy and F 1 -score for MRPC and QQP.

Original:  R ESULTS
Revised:  R ESULTS

Original: We present the results along the design choices introduced in Section 2, namely bidirection, crossarchitecture distillation approaches, and two-sequence encoding scheme.
Revised: We present the results along the design choices introduced in Section 2, namely bidirection, crossarchitecture distillation approaches, and two-sequence encoding scheme.

Original: Finally, we compare our best embedding methods with ELMo and BERT distillates from the literature.
Revised: Finally, we compare our best embedding methods with ELMo and BERT distillates from the literature.

Original: For brevity, we focus on reporting key results.
Revised: For brevity, we focus on reporting key results.

Original: The detailed results can be found in Appendix C.
Revised: The detailed results can be found in Appendix C.

Original: Bidirectional CMOW/CBOW-Hybrid versus Baseline To isolate the effect of the bidirectional component, we compare unidirectional CMOW/CBOW-Hybrid with bidirectional CMOW/CBOW-
Revised: Bidirectional CMOW/CBOW-Hybrid versus Baseline To isolate the effect of the bidirectional component, we compare unidirectional CMOW/CBOW-Hybrid with bidirectional CMOW/CBOW-

Original: Hybrid under equal conditions.
Revised: Hybrid under equal conditions.

Original: We train both variants from scratch for the downstream tasks, while using a BERT’s teacher signal.
Revised: We train both variants from scratch for the downstream tasks, while using a BERT’s teacher signal.

Original: Table 1 shows the results of comparing unidirectional Hybrid embeddings with the proposed bidirectional Hybrid embeddings.
Revised: Table 1 shows the results of comparing unidirectional Hybrid embeddings with the proposed bidirectional Hybrid embeddings.

Original: Bidirection helps on the tasks
Revised: Bidirection helps on the tasks

Original: MNLI, MRPC, QNLI, SST-2, STS-B, and WNLI.
Revised: MNLI, MRPC, QNLI, SST-2, STS-B, and WNLI.

Original: On the other tasks, the difference is marginal.
Revised: On the other tasks, the difference is marginal.

Original: We have an average improvement of 1% of the bidirectional model over the unidirectional model across all tasks of the GLUE benchmark.
Revised: We have an average improvement of 1% of the bidirectional model over the unidirectional model across all tasks of the GLUE benchmark.

Original: [Table]
Revised: [Table]

Original: General Distillation vs. Task-speciﬁc Distillation Next, we compare general distillation with task-speciﬁc distillation.
Revised: General Distillation vs. Task-speciﬁc Distillation Next, we compare general distillation with task-speciﬁc distillation.

Original: As shown in Table 2, using general distillation leads to better results for ﬁve tasks (MNLI, MRPC, QQP, STS-B, and RTE) compared to task-speciﬁc distillation.
Revised: As shown in Table 2, using general distillation leads to better results for ﬁve tasks (MNLI, MRPC, QQP, STS-B, and RTE) compared to task-speciﬁc distillation.

Original: For the other four tasks (CoLA, QNLI, SST-2, and WNLI), task-speciﬁc distillation achieves higher scores.
Revised: For the other four tasks (CoLA, QNLI, SST-2, and WNLI), task-speciﬁc distillation achieves higher scores.

Original: The average score of general distillation is higher than with task-speciﬁc distillation in both pretrained and randomly initialized cases.
Revised: The average score of general distillation is higher than with task-speciﬁc distillation in both pretrained and randomly initialized cases.

Original: distillation using bidirectional CMOW/CBOWHybrid embeddings and MLP classiﬁer.
Revised: distillation using bidirectional CMOW/CBOWHybrid embeddings and MLP classiﬁer.

Original: In 5 out of 9 tasks, general distillation performs best.
Revised: In 5 out of 9 tasks, general distillation performs best.

Original: [Table]
Revised: [Table]

Original: DiffCat Encoding vs. Joint Encoding Now, we compare the two encoding schemes, whose results are presented in Table 3.
Revised: DiffCat Encoding vs. Joint Encoding Now, we compare the two encoding schemes, whose results are presented in Table 3.

Original: The best scores were achieved with DiffCat encoding.
Revised: The best scores were achieved with DiffCat encoding.

Original: Most remarkable is the improvement from 18 .
Revised: Most remarkable is the improvement from 18 .

Original: 1 to 56 .
Revised: 1 to 56 .

Original: 2 on the sentence similarity task STS-B when encoding the sentence pair input via DiffCat.
Revised: 2 on the sentence similarity task STS-B when encoding the sentence pair input via DiffCat.

Original: The average improvement of DiffCat encoding is 32% across the two-sentence GLUE tasks.
Revised: The average improvement of DiffCat encoding is 32% across the two-sentence GLUE tasks.

Original: [Table]
Revised: [Table]

Original: Comparing bidirectional CMOW/CBOW-Hybrid to the Literature Finally, Table 4 shows the results of the best-performing bidirectional CMOW/CBOW-Hybrid variants using any of the three considered distillation methods.
Revised: Comparing bidirectional CMOW/CBOW-Hybrid to the Literature Finally, Table 4 shows the results of the best-performing bidirectional CMOW/CBOW-Hybrid variants using any of the three considered distillation methods.

Original: As described by Wasserblat et al.
Revised: As described by Wasserblat et al.

Original: (2020), a model needs to capture context and linguistic structure to perform well on CoLA.
Revised: (2020), a model needs to capture context and linguistic structure to perform well on CoLA.

Original: We doubled the results for CoLA and SSTcompared to the best previously reported cross-architecture distillation approaches by Wasserblat et al.
Revised: We doubled the results for CoLA and SSTcompared to the best previously reported cross-architecture distillation approaches by Wasserblat et al.

Original: Our best models scored higher than ELMo (Peters et al., 2018) on the tasks MRPC,
Revised: Our best models scored higher than ELMo (Peters et al., 2018) on the tasks MRPC,

Original: QNLI, QQP, RTE, and WNLI.
Revised: QNLI, QQP, RTE, and WNLI.

Original: We achieve higher scores than DistilBERT on RTE and WNLI.
Revised: We achieve higher scores than DistilBERT on RTE and WNLI.

Original: Runtime Performance and Parameter Count
Revised: Runtime Performance and Parameter Count

Original: To compare runtime, we generate 1,024 batches with 256 random sequences of length 64 and measure the time to encode the sequences with gradient computation disabled.
Revised: To compare runtime, we generate 1,024 batches with 256 random sequences of length 64 and measure the time to encode the sequences with gradient computation disabled.

Original: As shown in Table 5, CMOW/CBOW-Hybrid is more than 3 times faster than the fastest competitor, DistilBERT, and only uses about half of its parameters.
Revised: As shown in Table 5, CMOW/CBOW-Hybrid is more than 3 times faster than the fastest competitor, DistilBERT, and only uses about half of its parameters.

Original: The inference speed of
Revised: The inference speed of

Original: CMOW/CBOW-Hybrid could be increased even further because the O ( n ) steps to encode a sequence of length n can be parallelized to O (log n ) sequential steps, since matrix multiplication is associative.
Revised: CMOW/CBOW-Hybrid could be increased even further because the O ( n ) steps to encode a sequence of length n can be parallelized to O (log n ) sequential steps, since matrix multiplication is associative.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original:  D ISCUSSION AND R ELATED W ORK
Revised:  D ISCUSSION AND R ELATED W ORK

Original: Key Results We have shown that BERT can be distilled into efﬁcient matrix embedding models during pretraining by emitting intermediate representations.
Revised: Key Results We have shown that BERT can be distilled into efﬁcient matrix embedding models during pretraining by emitting intermediate representations.

Original: We have further introduced a bidirectional component and a separate two-sequence encoding scheme for CMOW-style models.
Revised: We have further introduced a bidirectional component and a separate two-sequence encoding scheme for CMOW-style models.

Original: We have observed that the general distillation approach, i. e., using the BERT teacher only during pretraining, leads to results that are oftentimes even better than those achieved with task-speciﬁc distillation.
Revised: We have observed that the general distillation approach, i. e., using the BERT teacher only during pretraining, leads to results that are oftentimes even better than those achieved with task-speciﬁc distillation.

Original: This is an interesting result because all previous works on cross-architecture distillation relied on task-speciﬁc distillation.
Revised: This is an interesting result because all previous works on cross-architecture distillation relied on task-speciﬁc distillation.

Original: Our proposed model offers an encoding speed at inference time that is three times faster than DistilBERT and more than ﬁve times faster than MobileBERT.
Revised: Our proposed model offers an encoding speed at inference time that is three times faster than DistilBERT and more than ﬁve times faster than MobileBERT.

Original: Limitations Currently, matrix embeddings still fall behind other BERT distillates, most notably,
Revised: Limitations Currently, matrix embeddings still fall behind other BERT distillates, most notably,

Original: MobileBERT, on many of the downstream tasks.
Revised: MobileBERT, on many of the downstream tasks.

Original: In particular, detecting linguistic acceptability remains a challenge for non-transformer methods. However, we have made progress by improving upon previous cross-architecture distillation approaches.
Revised: In particular, detecting linguistic acceptability remains a challenge for non-transformer methods, even though we improve upon previous crossarchitecture distillation approaches.

Original: In general, we argue that CMOW-style models are worth exploring because they offer a much higher encoding speed than Transformers.
Revised: So far, we have not analyzed the trade-off between embedding size and downstream performance. We rely on general arguments for the beneﬁts of increased dimensionality (Wieting & Kiela, 2019).

Original: Reﬂection w.r.t.
Revised: Reﬂection w.r.t.

Original: Distillation and Size-reduction Approaches We now discuss and relate to the existing body of works on general distillation, task-speciﬁc distillation, cross-architecture distillation, and other techniques for reducing the size of large models.

Original:  In general distillation, a PreLM is distilled into a student model during pretraining.
Revised: Distillation and Size-reduction Approaches In general distillation, a PreLM is distilled into a student model during pretraining.

Original: DistilBERT (Sanh et al., 2020a) is such a general-purpose language model that has been distilled from BERT.
Revised: DistilBERT (Sanh et al., 2020a) is such a generalpurpose language model that has been distilled from BERT.

Original: Apart from masked language modeling and distillation objectives, the authors also introduce a cosine loss term to align the student’s and teacher’s hidden states (layer transfer).
Revised: Apart from masked language modeling and distillation objectives, the authors also introduce a cosine loss term to align the student’s and teacher’s hidden states (layer transfer).

Original: Furthermore, the student is initialized with selected layers of the teacher.
Revised: Furthermore, the student is initialized with selected layers of the teacher.

Original: MobileBERT (Sun et al., 2020) introduced a bottleneck to BERT such that layers can be transferred to student models with smaller dimensions.
Revised: MobileBERT (Sun et al., 2020) introduced a bottleneck to BERT such that layers can be transferred to student models with smaller dimensions.

Original: In task-speciﬁc distillation, the teacher signal is used during ﬁne-tuning.
Revised: In task-speciﬁc distillation, the teacher signal is used during ﬁne-tuning.

Original: Sun et al.
Revised: Sun et al.

Original: (2019) use layer-wise distillation objectives and initialize with teacher weights to train BERT students with fewer layers.
Revised: (2019) use layer-wise distillation objectives and initialize with teacher weights to train BERT students with fewer layers.

Original: TinyBERT (Jiao et al., 2020) applies knowledge distillation in both stages, pretraining and ﬁne-tuning.
Revised: TinyBERT (Jiao et al., 2020) applies knowledge distillation in both stages, pretraining and ﬁne-tuning.

Revised: TinyBERT (Jiao et al., 2020) yields high throughput rates comparable to CMOW/CBOWHybrid.

Revised: Yet, TinyBERT requires keeping the teacher model for ﬁne-tuning.

Revised: We have shown that CMOW/CBOW-Hybrid is even better with general distillation only than under task-speciﬁc distillation.

Original: TinyBERT also augments the training data, which we have also considered but found no consistent improvement.
Revised: TinyBERT also augments the training data, which we have also considered but found no consistent improvement.

Original: Turc et al.
Revised: Turc et al.

Original: (2019) analyzes the interaction between pre-training and ﬁne-tuning with BERT models and ﬁnd that pretrained distillation works well, which agrees with our ﬁndings on the importance of pretraining with CMOW-style models.
Revised: (2019) analyzes the interaction between pre-training and ﬁne-tuning with BERT models and ﬁnd that pretrained distillation works well, which agrees with our ﬁndings on the importance of pretraining with CMOW-style models.

Original: LadaBERT (Mao et al., 2020) combines knowledge distillation with pruning and matrix factorization.
Revised: LadaBERT (Mao et al., 2020) combines knowledge distillation with pruning and matrix factorization.

Original: Other approaches consider distillation in multi-lingual (Tsai et al., 2019) or multi-task settings
Revised: Other approaches consider distillation in multi-lingual (Tsai et al., 2019) or multi-task settings

Original: (Yang et al., 2019).
Revised: (Yang et al., 2019).

Original: The works described above assume that the teacher and student share the same architecture.
Revised: The works described above assume that the teacher and student share the same architecture.

Original: However, the student model does not need to have the same architecture as the teacher, what we then call cross-architecture distillation.
Revised: However, the student model does not need to have the same architecture as the teacher, what we then call cross-architecture distillation.

Original: Wasserblat et al.
Revised: Wasserblat et al.

Original: (2020) use a simple feed forward network with CBOW embeddings and a bidirectional LSTM model as students.
Revised: (2020) use a simple feed forward network with CBOW embeddings and a bidirectional LSTM model as students.

Original: Both models perform well in several downstream tasks.
Revised: Both models perform well in several downstream tasks.

Original: Tang et al.
Revised: Tang et al.

Original: (2019) explore distilling BERT into a single layer BiLSTM without using any additional training data or modiﬁcations to the teacher architecture.
Revised: (2019) explore distilling BERT into a single layer BiLSTM without using any additional training data or modiﬁcations to the teacher architecture.

Original: Their distillation-based approach yields improvements compared to a plain BiLSTM without teacher signal: about 4 points on all reported tasks (QQP, MNLI, and SST-2).
Revised: Their distillation-based approach yields improvements compared to a plain BiLSTM without teacher signal: about 4 points on all reported tasks (QQP, MNLI, and SST-2).

Original: This has motivated us to investigate whether even simpler models can be used as students of a BERT teacher.
Revised: This has motivated us to investigate whether even simpler models can be used as students of a BERT teacher.

Original: Other techniques for reducing the size of a model are pruning and quantization.
Revised: Other techniques for reducing the size of a model are pruning and quantization.

Original: Pruning approaches such as Sanh et al.
Revised: Pruning approaches such as Sanh et al.

Original: (2020b) reduce the number of parameters.
Revised: (2020b) reduce the number of parameters.

Original: Still, the resulting smaller models use the same architecture as their larger counterparts and, thus, pruning does not necessarily improve inference speed.
Revised: Still, the resulting smaller models use the same architecture as their larger counterparts and, thus, pruning does not necessarily improve inference speed.

Original: Quantization is a common post-processing step to reduce model size by decreasing the ﬂoating point precision of the weights (Gupta et al., 2015; Wu et al., 2020).
Revised: Quantization is a common post-processing step to reduce model size by decreasing the ﬂoating point precision of the weights (Gupta et al., 2015; Wu et al., 2020).

Original: Pruning and quantization can be applied in conjunction with knowledge distillation (Sanh et al., 2020b; Sun et al., 2020).
Revised: Pruning and quantization can be applied in conjunction with knowledge distillation (Sanh et al., 2020b; Sun et al., 2020).

Original: Aside from techniques for reducing the model size, there is also a tremendous effort to improve the efﬁciency of the transformer architecture in the ﬁrst place, for which we refer to the recent survey of Tay et al. (2020) on efﬁcient transformers.
Revised: Aside from techniques for reducing the model size, there is also a tremendous effort to improve the efﬁciency of the transformer architecture in the ﬁrst place (Tay et al., 2020).

Original: For now, we focused on distillation, while pruning and quantization might even further increase the efﬁciency of our proposed method.

Original: Theexisting body of literatureprimarily focuses on reducing the size of PreLMs via distillation, pruning, and quantization.
Revised: The literature focuses on reducing the size of PreLMs via distillation, pruning, and quantization.

Original: Specialized techniques, such as layer transfer, depend on teacher and student sharing the same architecture, and thus, are not applicable for cross-architecture distillation.
Revised: Specialized techniques depend on teacher and student sharing the same architecture, and thus, are not applicable for cross-architecture distillation.

Original: So far, only few recent works consider distilling PreLMs into other architectures such as LSTMs and bag-of-words feed-forward nets.
Revised: So far, only few recent works consider distilling PreLMs into other architectures like LSTMs and CBOW models.

Original: In this work, we show that cross-architecture distillation with orderaware embedding models as students outperform previous cross-architecture distillation approaches and achieve scores comparable to ELMo, while using less computational resources.
Revised: In this work, we show that cross-architecture distillation with order-aware embedding models as students outperform previous cross-architecture distillation approaches and achieve scores comparable to ELMo, while using less computational resources.

Original: Finally, all previous cross-architecture distillation approaches are task-speciﬁc, while we could show that general distillation can lead to even higher scores than task-speciﬁc distillation.
Revised: Finally, all previous cross-architecture distillation approaches are task-speciﬁc, while we could show that general distillation can lead to even higher scores than task-speciﬁc distillation.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We have introduced three extensions to the CMOW/CBOW-Hybrid model: a bidirectional component, a separate two-sequence encoding scheme, and the ability to emit per-token representations.
Revised: We have introduced three extensions to the CMOW/CBOW-Hybrid model: a bidirectional component, a separate two-sequence encoding scheme, and the ability to emit per-token representations.

Original: These per-token representations allow us to distill BERT into CMOW/CBOW-Hybrid already during pretraining with a masked language modeling objective.
Revised: These per-token representations allow us to distill BERT into CMOW/CBOW-Hybrid already during pretraining with a masked language modeling objective.

Original: Our results show that a separate encoding scheme improves the performance of CMOW/CBOW-Hybrid on two-sentence GLUE tasks by more than 30%, while bidirection improves the performance by 1% compared to the unidirectional model.
Revised: Our results show that a separate encoding scheme improves the performance of CMOW/CBOW-Hybrid on two-sentence GLUE tasks by more than 30%, while bidirection improves the performance by 1% compared to the unidirectional model.

Original: Furthermore, we have shown that general distillation seems to be sufﬁcient and task-speciﬁc distillation is not necessary for most GLUE tasks.
Revised: Furthermore, we have shown that general distillation seems to be sufﬁcient and task-speciﬁc distillation is not necessary for most GLUE tasks.

Original: In comparison with more expensive models from the literature, our embedding-based approach achieves scores that match or exceed the scores of
Revised: In comparison with more expensive models from the literature, our embedding-based approach achieves scores that match or exceed the scores of

Original: ELMo and are competitive to DistilBERT on QQP and RTE with only half of its parameters and thrice its encoding speed.
Revised: ELMo and are competitive to DistilBERT on QQP and RTE with only half of its parameters and thrice its encoding speed.

Original: While linguistic acceptibility remains a challenge for non-transformer models, our approach yields notably higher scores than previous cross-architecture distillation approaches.
Revised: While linguistic acceptibility remains a challenge for non-transformer models, our approach yields notably higher scores than previous cross-architecture distillation approaches.

Original: In future work, one could further improve the efﬁciency by applying pruning (Sanh et al., 2020b) and/or quantization (Wu et al., 2020) techniques on the learned matrices to allow sparse matrix multiplication during encoding.
Revised: In future work, one could further improve the efﬁciency by applying pruning (Sanh et al., 2020b) and/or quantization (Wu et al., 2020) techniques on the learned matrices to allow sparse matrix multiplication during encoding.

Original: Future work could explore what components would be necessary to improve the scores on particularly challenging downstream tasks such as detecting linguistic acceptability.
Revised: Future work could explore what components would be necessary to improve the scores on particularly challenging downstream tasks such as detecting linguistic acceptability.

Original: This might include the introduction of a small attention module like the one of gMLP (Liu et al., 2021) to CMOW/CBOW-Hybrid.
Revised: This might include the introduction of a small attention module like the one of gMLP (Liu et al., 2021) to CMOW/CBOW-Hybrid.

Original: Since we have shown how to emit (and train) pertoken representations with CMOW, an interesting direction of future work would be to explore whether CMOW representations are suited as order-aware embeddings in transformer models, which might alleviate the need for a dedicated position embedding.
Revised: Since we have shown how to emit (and train) per-token representations with CMOW, an interesting direction of future work would be to explore whether CMOW representations are suited as order-aware embeddings in transformer models, which might alleviate the need for a dedicated position embedding.

Revised: Similarly, the per-token representations enables tackling question answering tasks with CMOW-style models, in which a span of the context ought to be predicted.

Original: Finally, matrix embeddings are technically not limited to natural language processing.
Revised: Finally, matrix embeddings are technically not limited to natural language processing.

Original: Therefore, another interesting path of future work would be to explore the use of matrix embeddings in other domains where sequences of discrete elements need to be encoded, e. g., event sequences, gene sequences, or item sequences in recommender systems.
Revised: Therefore, another interesting path of future work would be to explore the use of matrix embeddings in other domains where sequences of discrete elements need to be encoded, e. g., event sequences, gene sequences, or item sequences in recommender systems.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: There has been criticism about large-scale language models and the ampliﬁcation of bias through the reduction of model size.
Revised: There has been criticism about large-scale language models and the ampliﬁcation of bias through the reduction of model size.

Original: This is particularly a problem when the model is not explainable.
Revised: This is particularly a problem when the model is not explainable.

Original: In our proposed method, we distill large-scale pretrained language models into models that do not have a nonlinearity (except for the ﬁnal classiﬁcation head).
Revised: In our proposed method, we distill large-scale pretrained language models into models that do not have a nonlinearity (except for the ﬁnal classiﬁcation head).

Original: Since nonlinearities make explainabilty more difﬁcult, we hope to contribute to more explainable language models with our work, even though explainability is not the main focus of the present work.
Revised: Since nonlinearities make explainabilty more difﬁcult, we hope to contribute to more explainable language models with our work, even though explainability is not the main focus of the present work.

Original: Another concern with large language models is their immense consumption of resources.
Revised: Another concern with large language models is their immense consumption of resources.

Original: In our work, we have only conducted the expensive pretraining step once on the full unlabeled training data.
Revised: In our work, we have only conducted the expensive pretraining step once on the full unlabeled training data.

Original: Only afterwards, we conduct multiple “cheap” runs to optimize hyperparameters for downstream tasks.
Revised: Only afterwards, we conduct multiple “cheap” runs to optimize hyperparameters for downstream tasks.

Original: Thus, our experiments themselves have been conducted resource efﬁcient.
Revised: Thus, our experiments themselves have been conducted resource efﬁcient.

Original: Furthermore, this research contributes to the development of simpler and more efﬁcient models for natural language processing.
Revised: Furthermore, this research contributes to the development of simpler and more efﬁcient models for natural language processing.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure that our experiments are reproducible, we have tracked all hyperparameters throughout experimentation.
Revised: To ensure that our experiments are reproducible, we have tracked all hyperparameters throughout experimentation.

Original: We have described all steps for data processing, i. e., tokenization (we use the BERT tokenizer) and binning (for STS-B) in the main part.
Revised: We have described all steps for data processing, i. e., tokenization (we use the BERT tokenizer) and binning (for STS-B) in the main part.

Original: The datasets from the GLUE benchmark are well-known to the community, because of which we have omitted a detailed description, except for the metrics that we have reported.
Revised: The datasets from the GLUE benchmark are well-known to the community, because of which we have omitted a detailed description, except for the metrics that we have reported.

Original: We make the code and pretrained embeddings available upon acceptance of the paper.
Revised: We make the code and pretrained embeddings available upon acceptance of the paper.

Original: For now, we provide the code for our experiments as supplementary material.
Revised: For now, we provide the code for our experiments as supplementary material.

Original: A PPENDIX A O VERVIEW OF A RCHITECTURAL C HOICES
Revised: A PPENDIX A O VERVIEW OF A RCHITECTURAL C HOICES

Original: Figure 3 provides an overview of the architectural choices explored in this paper.
Revised: Figure 3 provides an overview of the architectural choices explored in this paper.

Original: We use pretrained
Revised: We use pretrained

Original: BERT Devlin et al.
Revised: BERT Devlin et al.

Original: (2019) as well as the embeddings from Mai et al.
Revised: (2019) as well as the embeddings from Mai et al.

Original: (2019) as teacher for general distillation.
Revised: (2019) as teacher for general distillation.

Original: Additionally, we pretrain a model CMOW/CBOW-Hybrid with our extension of masked language model training and bidirection on the same English Wikipedia + Toronto Books dataset.
Revised: Additionally, we pretrain a model CMOW/CBOW-Hybrid with our extension of masked language model training and bidirection on the same English Wikipedia + Toronto Books dataset.

Original: These pretrained embeddings may serve as initialization for the downstream classiﬁcation models.
Revised: These pretrained embeddings may serve as initialization for the downstream classiﬁcation models.

Original: We also evaluate downstream classiﬁcation models that have been initialized randomly.
Revised: We also evaluate downstream classiﬁcation models that have been initialized randomly.

Original: For the downstream classiﬁcation models, we consider three types of embeddings along with four types of classiﬁers.
Revised: For the downstream classiﬁcation models, we consider three types of embeddings along with four types of classiﬁers.

Original: [Figure]
Revised: [Figure]

Original: For training on the downstream task, we use once again BERT as a teacher for task-speciﬁc distillation, while our experiments on general distillation only beneﬁt from the initalization of the MLM-pretrained
Revised: For training on the downstream task, we use once again BERT as a teacher for task-speciﬁc distillation, while our experiments on general distillation only beneﬁt from the initalization of the MLM-pretrained

Original: CMOW/CBOW-Hybrid model.
Revised: CMOW/CBOW-Hybrid model.

Original: Throughout the main part of the paper, we have reported scores with an MLP downstream classiﬁer, which achieved the highest average scores.
Revised: Throughout the main part of the paper, we have reported scores with an MLP downstream classiﬁer, which achieved the highest average scores.

Original: We have further experimented with a linear downstream classiﬁer, LSTM, and 2D-CNN, which we brieﬂy describe below.
Revised: We have further experimented with a linear downstream classiﬁer, LSTM, and 2D-CNN, which we brieﬂy describe below.

Original: LSTM We have further experimented with pooling the sequence of embeddings with an LSTM.
Revised: LSTM We have further experimented with pooling the sequence of embeddings with an LSTM.

Original: In the past, BiLSTM models have been successfully used in sentiment analysis tasks Xu et al.
Revised: In the past, BiLSTM models have been successfully used in sentiment analysis tasks Xu et al.

Original: Hameed & Garcia-Zapirain (2020).
Revised: Hameed & Garcia-Zapirain (2020).

Original: In an LSTM network, the information at hand is propagated in the forward direction.
Revised: In an LSTM network, the information at hand is propagated in the forward direction.

Original: Thus, each state t depends on its predecessor t − 1 .
Revised: Thus, each state t depends on its predecessor t − 1 .

Original: BiLSTM are LSTM networks, in which the inputs are processed twice: once in the forward direction and once in the backward direction, generating a set of two outputs.
Revised: BiLSTM are LSTM networks, in which the inputs are processed twice: once in the forward direction and once in the backward direction, generating a set of two outputs.

Original: In order to generate the output vectors, the output of a single BiLSTM block is fed into an MLP, consisting of two consecutive linear layers with ReLU activation functions.
Revised: In order to generate the output vectors, the output of a single BiLSTM block is fed into an MLP, consisting of two consecutive linear layers with ReLU activation functions.

Original: Note that the BiLSTM operates on a sequence of token embeddings, instead of operating on pooled sentence embeddings like the other student models.
Revised: Note that the BiLSTM operates on a sequence of token embeddings, instead of operating on pooled sentence embeddings like the other student models.

Original: We apply a dropout of 0 .after the ﬁrst linear layer.
Revised: We apply a dropout of 0 .after the ﬁrst linear layer.

Original: CNN We also explore a 2D-CNN classiﬁer that induces a bias for learning two-dimensional structures within the (aggregated) embedding matrices.
Revised: CNN We also explore a 2D-CNN classiﬁer that induces a bias for learning two-dimensional structures within the (aggregated) embedding matrices.

Original: The CNN consists of one transposed convolution, which increases the matrix dimensions by a factor of four.
Revised: The CNN consists of one transposed convolution, which increases the matrix dimensions by a factor of four.

Original: Following that, we employ a block of three convolutional layers, the ﬁrst one having a single ﬁlter (or two, for hybrid variants) and a kernel size of four, with the remaining two layers having 3 (4) kernels with stride 2.
Revised: Following that, we employ a block of three convolutional layers, the ﬁrst one having a single ﬁlter (or two, for hybrid variants) and a kernel size of four, with the remaining two layers having 3 (4) kernels with stride 2.

Original: To avoid distorting the input embeddings, no padding is applied.
Revised: To avoid distorting the input embeddings, no padding is applied.

Original: ReLU is used for all activation functions.
Revised: ReLU is used for all activation functions.

Original: We apply
Revised: We apply

Original: BatchNorm for regularization before the last convolutional layer’s output is ﬂattened and passed into a linear layer, which produces the predictions.
Revised: BatchNorm for regularization before the last convolutional layer’s output is ﬂattened and passed into a linear layer, which produces the predictions.

Original: We add a dropout of 0 .
Revised: We add a dropout of 0 .

Original: 4 before the last linear layer.
Revised: 4 before the last linear layer.

Original: [Table]
Revised: [Table]

Original: B D ISCUSSION OF H YPERPARAMETERS AND L OSS F UNCTIONS FOR D ISTILLATION
Revised: B D ISCUSSION OF H YPERPARAMETERS AND L OSS F UNCTIONS FOR D ISTILLATION

Original: We list hyperparameter search spaces along with their optimization methods in Table 6.
Revised: We list hyperparameter search spaces along with their optimization methods in Table 6.

Original: For the experiments on data augmentation and using only soft loss, we keep the conﬁgurations of the best models (See Table 7 and tune the learning rate, again.
Revised: For the experiments on data augmentation and using only soft loss, we keep the conﬁgurations of the best models (See Table 7 and tune the learning rate, again.

Original: We optimize over all six initial learning rates, namely { 10 − 3 , 5 · 10 − 4 , 10 − 4 , 5 · 10 − 5 , and 10 − 5 } .
Revised: We optimize over all six initial learning rates, namely { 10 − 3 , 5 · 10 − 4 , 10 − 4 , 5 · 10 − 5 , and 10 − 5 } .

Original: All initial learning rates decay linearly over the course of training.
Revised: All initial learning rates decay linearly over the course of training.

Original: Note, we also experimented with using warmup steps versus no warmup for the learning rate schedule.
Revised: Note, we also experimented with using warmup steps versus no warmup for the learning rate schedule.

Original: As the warmup did not improve the results, we did not use it.
Revised: As the warmup did not improve the results, we did not use it.

Original: For the softmax temperature, we ﬁnd that T = 1 is often used Hinton et al.
Revised: For the softmax temperature, we ﬁnd that T = 1 is often used Hinton et al.

Original: ; Mao et al.
Revised: ; Mao et al.

Original: Jiao et al.
Revised: Jiao et al.

Original: Mishra & Marr (2018); Polino et al.
Revised: Mishra & Marr (2018); Polino et al.

Original: Since a higher temperature also ﬂattens the curve over all predictions, it could add too much noise and it is therefore better to use a smaller temperature Chen et al.
Revised: Since a higher temperature also ﬂattens the curve over all predictions, it could add too much noise and it is therefore better to use a smaller temperature Chen et al.

Original: Setting the weight α = 1 corresponds to only using hard loss and α = 0 to only using soft loss.
Revised: Setting the weight α = 1 corresponds to only using hard loss and α = 0 to only using soft loss.

Original: Since we do not want to discard any information stemming from the hard loss, we do not follow the approach of Wasserblat et al.
Revised: Since we do not want to discard any information stemming from the hard loss, we do not follow the approach of Wasserblat et al.

Original: who only use the soft loss Wasserblat et al.
Revised: who only use the soft loss Wasserblat et al.

Original: but instead, we employ a vanilla knowledge distillation approach following Hinton et al.
Revised: but instead, we employ a vanilla knowledge distillation approach following Hinton et al.

Original: Hinton et al. Hinton et al.
Revised: Hinton et al. Hinton et al.

Original: (2015) state, that using cross-entropy loss on the softmax-temperature with a large temperature, for example T = 20 , corresponds to only using the Mean Square Error (MSE) loss on the raw student and teacher logits.
Revised: (2015) state, that using cross-entropy loss on the softmax-temperature with a large temperature, for example T = 20 , corresponds to only using the Mean Square Error (MSE) loss on the raw student and teacher logits.

Original: Therefore it is also common to use this loss for the soft distillation loss Tang et al.
Revised: Therefore it is also common to use this loss for the soft distillation loss Tang et al.

Original: ; Wasserblat et al.
Revised: ; Wasserblat et al.

Original: (2020); Mukherjee & Awadallah (2020).
Revised: (2020); Mukherjee & Awadallah (2020).

Original: While
Revised: While

Original: Tang et al. Tang et al.
Revised: Tang et al. Tang et al.

Original: (2019) used the weighted hard cross-entropy loss in the overall loss calculation,
Revised: (2019) used the weighted hard cross-entropy loss in the overall loss calculation,

Original: Wasserblat et al.
Revised: Wasserblat et al.

Original: and Mukherjee et al.
Revised: and Mukherjee et al.

Original: only used the soft loss Tang et al.
Revised: only used the soft loss Tang et al.

Original: ; Wasserblat et al.
Revised: ; Wasserblat et al.

Original: (2020); Mukherjee & Awadallah (2020).
Revised: (2020); Mukherjee & Awadallah (2020).

Original: A disadvantage of MSE loss, is that every error has a huge effect on the overall loss, since it is squared.
Revised: A disadvantage of MSE loss, is that every error has a huge effect on the overall loss, since it is squared.

Original: Another point is, that Hinton et al.
Revised: Another point is, that Hinton et al.

Original: found it beneﬁcial to use small temperature values if the teacher is way bigger than the student Hinton et al.
Revised: found it beneﬁcial to use small temperature values if the teacher is way bigger than the student Hinton et al.

Original: Since using MSE loss corresponds to using big T values, this loss does not apply to our use case of using small students for lower bound knowledge distillation, but with cross-entropy loss, we still have the possibility to achieve the behavior of the MSE loss by setting the value of T to a big value.
Revised: Since using MSE loss corresponds to using big T values, this loss does not apply to our use case of using small students for lower bound knowledge distillation, but with cross-entropy loss, we still have the possibility to achieve the behavior of the MSE loss by setting the value of T to a big value.

Original: C D ETAILED R ESULTS
Revised: C D ETAILED R ESULTS

Original: In the following, we provide detailed results for task-speciﬁc distillation including the different downstream classiﬁers, unidirectional CMOW/CBOW-Hybrid, and joint two-sequence encoding.
Revised: In the following, we provide detailed results for task-speciﬁc distillation including the different downstream classiﬁers, unidirectional CMOW/CBOW-Hybrid, and joint two-sequence encoding.

Original: The best performing model per task are marked in bold.
Revised: The best performing model per task are marked in bold.

Original: We abbreviate CMOW/CBOW-Hybrid as ’Hybrid’.
Revised: We abbreviate CMOW/CBOW-Hybrid as ’Hybrid’.

Original: For the unidirectional baseline model CMOW/CBOW-Hybrid, we initialize with pretrained embeddings provided by Mai et al.
Revised: For the unidirectional baseline model CMOW/CBOW-Hybrid, we initialize with pretrained embeddings provided by Mai et al.

Original: (2019) 2 , which cover 54% of BERT’s vocabulary.
Revised: (2019) 2 , which cover 54% of BERT’s vocabulary.

Original: As initialization for the newly developed bidirectional CMOW/CBOW-Hybrid models, we use our own pretrained model obtained by general distillation with BERT, as described in Section 3.
Revised: As initialization for the newly developed bidirectional CMOW/CBOW-Hybrid models, we use our own pretrained model obtained by general distillation with BERT, as described in Section 3.

Original: Table 7 summarizes the best performing models along with their hyperparameter conﬁgurations for each task.
Revised: Table 7 summarizes the best performing models along with their hyperparameter conﬁgurations for each task.

Original: Note that we have chosen to use an MLP downstream classiﬁer for the results reported in the main part of this work.
Revised: Note that we have chosen to use an MLP downstream classiﬁer for the results reported in the main part of this work.

Original: Using an MLP downstream classiﬁer has led to the highest average scores across all GLUE tasks.
Revised: Using an MLP downstream classiﬁer has led to the highest average scores across all GLUE tasks.

Original: In Table 8, we report an extended version of the comparison with the literature.
Revised: In Table 8, we report an extended version of the comparison with the literature.

Original: Here, we also include our BERT-base teacher model, aswell as TinyBERT Jiao et al.
Revised: Here, we also include our BERT-base teacher model, aswell as TinyBERT Jiao et al.

Original: (2020) and Tang et al.
Revised: (2020) and Tang et al.

Original: (2019)’s distilled
Revised: (2019)’s distilled

Original: BiLSTM.
Revised: BiLSTM.

Original: Note that TinyBERT and BiLSTM are not fully comparable, because those numbers are reported on the ofﬁcial GLUE test set, while we have used the validation set for our experiments.
Revised: Note that TinyBERT and BiLSTM are not fully comparable, because those numbers are reported on the ofﬁcial GLUE test set, while we have used the validation set for our experiments.

Original: In Table 9, we report the results for all downstream classiﬁers without the DiffCat aggregation but with a sequential BERT-like two-sequence encoding.
Revised: In Table 9, we report the results for all downstream classiﬁers without the DiffCat aggregation but with a sequential BERT-like two-sequence encoding.

Original: It is interesting to see that the CMOW-only variant with 2D-CNN classiﬁer leads to the best scores on sentiment analysis task SST-2.
Revised: It is interesting to see that the CMOW-only variant with 2D-CNN classiﬁer leads to the best scores on sentiment analysis task SST-2.

Original: Note that all CMOW variants reported in this table are unidirectional and use task-speciﬁc distillation.
Revised: Note that all CMOW variants reported in this table are unidirectional and use task-speciﬁc distillation.

Original: In Table 10, we report the results for all downstream classiﬁers with DiffCat two-sequence encoding.
Revised: In Table 10, we report the results for all downstream classiﬁers with DiffCat two-sequence encoding.

Original: Here we observe, that pretrained CBOW with an MLP classiﬁer leads to the best results on sentence similarity (STS-B).
Revised: Here we observe, that pretrained CBOW with an MLP classiﬁer leads to the best results on sentence similarity (STS-B).

Original: Again, all CMOW variants reported in this table are unidirectional and use task-speciﬁc distillation.
Revised: Again, all CMOW variants reported in this table are unidirectional and use task-speciﬁc distillation.

Original: In Table 11, we report the results for bidirectional models with DiffCat two-sequence encoding.
Revised: In Table 11, we report the results for bidirectional models with DiffCat two-sequence encoding.

Original: From all tables combined, we see that Bidirectional CMOW/CBOW-Hybrid model leads to the highest scores on average, even though, on individual tasks, some other variations of the approach lead to higher scores.
Revised: From all tables combined, we see that Bidirectional CMOW/CBOW-Hybrid model leads to the highest scores on average, even though, on individual tasks, some other variations of the approach lead to higher scores.

Original: Thus, we regard bidrectional CMOW/CBOW-Hybrid as our primary model, whose scores we have reported in the main paper, while isolating the effect of the individual components (bidirection, DiffCat encoding, distillation strategies).
Revised: Thus, we regard bidrectional CMOW/CBOW-Hybrid as our primary model, whose scores we have reported in the main paper, while isolating the effect of the individual components (bidirection, DiffCat encoding, distillation strategies).

Original: We list the number of parameters in Tables 12 and 13.
Revised: We list the number of parameters in Tables 12 and 13.

Original: While the absolute numbers might seem high, it is important to note that we have also counted the parameters of the embeddings.
Revised: While the absolute numbers might seem high, it is important to note that we have also counted the parameters of the embeddings.

Original: As we show in the tables, the number of parameters in the classiﬁcation models is much lower.
Revised: As we show in the tables, the number of parameters in the classiﬁcation models is much lower.

Original: We have performed further experiments with the best performing model for each task: data augmentation and using only soft loss.
Revised: We have performed further experiments with the best performing model for each task: data augmentation and using only soft loss.

Original: Using Only Soft Loss We study the inﬂuence of the alpha value used in the loss function, based on the best results obtained with the initial α = 0 .
Revised: Using Only Soft Loss We study the inﬂuence of the alpha value used in the loss function, based on the best results obtained with the initial α = 0 .

Original: The goal is to investigate whether using only soft loss, i. e., setting α = 0 .
Revised: The goal is to investigate whether using only soft loss, i. e., setting α = 0 .

Original: 0 leads to different results.
Revised: 0 leads to different results.

Original: As Table 8 shows, using only soft loss improves only the MRPC task by a small margin.
Revised: As Table 8 shows, using only soft loss improves only the MRPC task by a small margin.

Original: Data Augmentation We conduct a further study with data augmentation as in TinyBERT (Jiao et al., 2020).
Revised: Data Augmentation We conduct a further study with data augmentation as in TinyBERT (Jiao et al., 2020).

Original: We employ their technique of replacing words by similar word embeddings and nearest predictions from BERT to augment the GLUE training datasets.
Revised: We employ their technique of replacing words by similar word embeddings and nearest predictions from BERT to augment the GLUE training datasets.

Original: The results are shown in Table 8.
Revised: The results are shown in Table 8.

Original: We ﬁnd that the effect of data augmentation is small.
Revised: We ﬁnd that the effect of data augmentation is small.

Original: An improvement was only observed on SST- ( +1 . 2 points) and STS-B ( +3 . 6 ).
Revised: An improvement was only observed on SST- ( +1 . 2 points) and STS-B ( +3 . 6 ).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: — CMOW
Revised: — CMOW

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CMOW
Revised: CMOW

Original: [Table]
Revised: [Table]

Original: CNN (
Revised: CNN (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CBOW
Revised: CBOW

Original: [Table]
Revised: [Table]

Original: MLP
Revised: MLP

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: MLP
Revised: MLP

Original: [Table] [Table]
Revised: [Table] [Table]

Original: — further experiments on best-performing task-
Revised: — further experiments on best-performing task-

Original: [Table]
Revised: [Table]

Original: distillation models — Only soft loss ( α = 0 )
Revised: distillation models — Only soft loss ( α = 0 )

Original: [Table]
Revised: [Table]

Original: Data augmentation
Revised: Data augmentation

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: — task-speciﬁc distillation
Revised: — task-speciﬁc distillation

Original: [Table]
Revised: [Table]

Original: ours) Bidirectional Hybrid, pretrained —
Revised: ours) Bidirectional Hybrid, pretrained —

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]