Original: L ENGTH -A DAPTIVE T RANSFORMER : T RAIN O NCE WITH L ENGTH D ROP , U SE A NYTIME WITH S EARCH
Revised: L ENGTH -A DAPTIVE T RANSFORMER : T RAIN O NCE WITH L ENGTH D ROP , U SE A NYTIME WITH S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although transformers have achieved impressive accuracies in various tasks in natural language processing, they often come with a prohibitive computational cost, that prevents their use in scenarios with limited computational resources for inference.
Revised: Although transformers have achieved impressive accuracies in various tasks in natural language processing, they often come with a prohibitive computational cost, that prevents their use in scenarios with limited computational resources for inference.

Original: This need for computational efﬁciency in inference has been addressed by for instanceGoyal et al. (2020) who proposed PoWER-BERT  which gradually decreases the length of a sequence as it is passed through layers.
Revised: This need for computational efﬁciency in inference has been addressed by for instance PoWER-BERT (Goyal et al., 2020) which gradually decreases the length of a sequence as it is passed through layers.

Original: These approaches however often assume that the target computational complexity is known in advance at the time of training.
Revised: These approaches however often assume that the target computational complexity is known in advance at the time of training.

Original: This implies that a separate model must be trained for each inference scenario with its distinct computational budget.
Revised: This implies that a separate model must be trained for each inference scenario with its distinct computational budget.

Original: In this paper, we extend PoWER-BERT to address this issue of inefﬁciency and redundancy.
Revised: In this paper, we extend PoWER-BERT to address this issue of inefﬁciency and redundancy.

Original: The proposed extension enables us to train a large-scale transformer, called LengthAdaptive Transformer , once and uses it for various inference scenarios without re-training it.
Revised: The proposed extension enables us to train a large-scale transformer, called Length-Adaptive Transformer , once and uses it for various inference scenarios without re-training it.

Original: To do so, we train a transformer with LengthDrop , a structural variant of dropout, which stochastically determines the length of a sequence at each layer.
Revised: To do so, we train a transformer with LengthDrop , a structural variant of dropout, which stochastically determines the length of a sequence at each layer.

Original: We then use a multi-objective evolutionary search to ﬁnd a length conﬁguration that maximizes the accuracy and minimizes the computational complexity under any given computational budget.
Revised: We then use a multi-objective evolutionary search to ﬁnd a length conﬁguration that maximizes the accuracy and minimizes the computational complexity under any given computational budget.

Original: Additionally, we signiﬁcantly extend the applicability of PoWER-BERT beyond sequence-level classiﬁcation into token-level classiﬁcation such as span-based question-answering, by introducing the idea of
Revised: Additionally, we signiﬁcantly extend the applicability of PoWER-BERT beyond sequence-level classiﬁcation into token-level classiﬁcation such as span-based question-answering, by introducing the idea of

Original: Drop-and-Restore .
Revised: Drop-and-Restore .

Original: With Drop-and-Restore, word-vectors are dropped temporarily in intermediate layers and restored at the last layer if necessary.
Revised: With Drop-and-Restore, word-vectors are dropped temporarily in intermediate layers and restored at the last layer if necessary.

Original: We empirically verify the utility of the proposed approach by demonstrating the superior accuracyefﬁciency trade-off under various setups, including SQuAD 1.1, MNLI-m, and
Revised: We empirically verify the utility of the proposed approach by demonstrating the superior accuracyefﬁciency trade-off under various setups, including SQuAD 1.1, MNLI-m, and

Original: SST-2.
Revised: SST-2.

Original: Upon publication, the code to reproduce our work will be open-sourced.
Revised: Upon publication, the code to reproduce our work will be open-sourced.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Pretrained language models (Peters et al., 2018; Devlin et al., 2018;
Revised: Pretrained language models (Peters et al., 2018; Devlin et al., 2018;

Original: Radford et al., 2019;
Revised: Radford et al., 2019;

Original: Yang et al., 2019) have achieved notable improvements in various natural language processing (NLP) tasks.
Revised: Yang et al., 2019) have achieved notable improvements in various natural language processing (NLP) tasks.

Original: Most of them rely on transformers (Vaswani et al., 2017), and the number of model parameters ranges from hundreds of millions to billions (Shoeybi et al., 2019; Raffel et al., 2019; Kaplan et al., 2020;
Revised: Most of them rely on transformers (Vaswani et al., 2017), and the number of model parameters ranges from hundreds of millions to billions (Shoeybi et al., 2019; Raffel et al., 2019; Kaplan et al., 2020;

Original: Brown et al., 2020).
Revised: Brown et al., 2020).

Original: Despite this high accuracy, excessive computational overhead during inference, both in terms of time and memory, has hindered its use in real applications.
Revised: Despite this high accuracy, excessive computational overhead during inference, both in terms of time and memory, has hindered its use in real applications.

Original: This level of excessive computation has further raised the concern over energy consumption as well (Schwartz et al., 2019;
Revised: This level of excessive computation has further raised the concern over energy consumption as well (Schwartz et al., 2019;

Original: Strubell et al., 2019).
Revised: Strubell et al., 2019).

Original: Recent studies have attempted at addressing these concerns regarding large-scale transformers’ computational and energy efﬁciency (see § 6 for a more extensive discussion.)
Revised: Recent studies have attempted at addressing these concerns regarding large-scale transformers’ computational and energy efﬁciency (see § 6 for a more extensive discussion.)

Original: Among these, we focus on PoWER-BERT (Goyal et al., 2020) which progressively reduces sequence length by eliminating word-vectors based on the attention values as passing layers.
Revised: Among these, we focus on PoWER-BERT (Goyal et al., 2020) which progressively reduces sequence length by eliminating word-vectors based on the attention values as passing layers.

Original: PoWER-BERT establishes the superiority of accuracy-time trade-off over earlier approaches (Sanh et al., 2019; Sun et al., 2019;
Revised: PoWER-BERT establishes the superiority of accuracy-time trade-off over earlier approaches (Sanh et al., 2019; Sun et al., 2019;

Original: Michel et al., 2019).
Revised: Michel et al., 2019).

Original: It however requires us to train a separate model for each efﬁciency constraint.
Revised: It however requires us to train a separate model for each efﬁciency constraint.

Original: In this paper, we thus develop a framework based on PoWER-BERT such that we can train a single model that can be adapted in the inference time to meet any given efﬁciency target.
Revised: In this paper, we thus develop a framework based on PoWER-BERT such that we can train a single model that can be adapted in the inference time to meet any given efﬁciency target.

Original: In order to train a transformer to cope with a diverse set of computational budgets in the inference time, we propose to train one while reducing the sequence length with a random proportion at each layer.
Revised: In order to train a transformer to cope with a diverse set of computational budgets in the inference time, we propose to train one while reducing the sequence length with a random proportion at each layer.

Original: We refer to this procedure as LengthDrop which was motivated by the nested dropout (Rippel et al., 2014).
Revised: We refer to this procedure as LengthDrop which was motivated by the nested dropout (Rippel et al., 2014).

Original: We can extract sub-models of shared weights with any length conﬁguration without requiring extra post-processing nor additional ﬁnetuning.
Revised: We can extract sub-models of shared weights with any length conﬁguration without requiring extra post-processing nor additional ﬁnetuning.

Original: Once a transformer is trained with the proposed LengthDrop, we search for the length conﬁguration that maximizes the accuracy given a computational budget.
Revised: Once a transformer is trained with the proposed LengthDrop, we search for the length conﬁguration that maximizes the accuracy given a computational budget.

Original: Because this search is combinatorial and has multiple objectives (accuracy and efﬁciency), we use an evolutionary search algorithm, which further allows us to obtain a full Pareto frontier of accuracy-efﬁciency trade-off of each model.
Revised: Because this search is combinatorial and has multiple objectives (accuracy and efﬁciency), we use an evolutionary search algorithm, which further allows us to obtain a full Pareto frontier of accuracy-efﬁciency trade-off of each model.

Original: It is not trivial to ﬁnd an optimal length conﬁguration given the inference-time computational budget, although it is extremely important in order to deploy these large-scale transformers in practice.
Revised: It is not trivial to ﬁnd an optimal length conﬁguration given the inference-time computational budget, although it is extremely important in order to deploy these large-scale transformers in practice.

Original: In this work, we propose to use evolutionary search to ﬁnd a length conﬁguration that maximizes the accuracy within a given computational budget.
Revised: In this work, we propose to use evolutionary search to ﬁnd a length conﬁguration that maximizes the accuracy within a given computational budget.

Original: We can further compute the Pareto frontier of accuracy-efﬁciency trade-off to obtain a sequence of length conﬁgurations with varying efﬁciency proﬁles.
Revised: We can further compute the Pareto frontier of accuracy-efﬁciency trade-off to obtain a sequence of length conﬁgurations with varying efﬁciency proﬁles.

Original: PoWER-BERT, which forms the foundation of the proposed two-stage procedure, is only applicable to sequence-level classiﬁcation, because by design it eliminates some of the word vectors at each layer.
Revised: PoWER-BERT, which forms the foundation of the proposed two-stage procedure, is only applicable to sequence-level classiﬁcation, because by design it eliminates some of the word vectors at each layer.

Original: In other words, it cannot be used for token-level tasks such as span-based question answering (Rajpurkar et al., 2016), because these tasks require hidden representations of the entire input sequence at the ﬁnal layer.
Revised: In other words, it cannot be used for token-level tasks such as span-based question answering (Rajpurkar et al., 2016), because these tasks require hidden representations of the entire input sequence at the ﬁnal layer.

Original: We thus propose to extend PoWER-BERT with a novel Drop-and-Restore process ( § 3.3), which eliminates this inherent limitation.
Revised: We thus propose to extend PoWER-BERT with a novel Drop-and-Restore process ( § 3.3), which eliminates this inherent limitation.

Original: Word vectors are dropped and set aside, rather than eliminated, in intermediate layers to maintain the saving of computational cost, as was with the original PoWER-BERT.
Revised: Word vectors are dropped and set aside, rather than eliminated, in intermediate layers to maintain the saving of computational cost, as was with the original PoWER-BERT.

Original: These set-aside vectors are then restored at the ﬁnal hidden layer and provided as an input to a subsequent task-speciﬁc layer, which is unlike the original PoWER-BERT.
Revised: These set-aside vectors are then restored at the ﬁnal hidden layer and provided as an input to a subsequent task-speciﬁc layer, which is unlike the original PoWER-BERT.

Original: The main contributions of this work are two-fold.
Revised: The main contributions of this work are two-fold.

Original: First, we introduce LengthDrop, a structured variant of dropout for training a single Length-Adaptive Transformer model that allows us to automatically derive multiple sub-models with different length conﬁgurations in the inference time using evolutionary search, without requiring any re-training.
Revised: First, we introduce LengthDrop, a structured variant of dropout for training a single Length-Adaptive Transformer model that allows us to automatically derive multiple sub-models with different length conﬁgurations in the inference time using evolutionary search, without requiring any re-training.

Original: Second, we design Drop-and-Restore process that makes PoWER-BERT applicable beyond classiﬁcation, which enables PoWER-BERT to be applicable to a wider range of NLP tasks such as span-based question answering.
Revised: Second, we design Drop-and-Restore process that makes PoWER-BERT applicable beyond classiﬁcation, which enables PoWER-BERT to be applicable to a wider range of NLP tasks such as span-based question answering.

Original: We empirically verify Length-Adaptive Transformer works quite well using the variants of BERT on a diverse set of
Revised: We empirically verify Length-Adaptive Transformer works quite well using the variants of BERT on a diverse set of

Original: NLP tasks, including SQuAD 1.1 (Rajpurkar et al., 2016) and two sequence-level classiﬁcation tasks in GLUE benchmark (Wang et al., 2018).
Revised: NLP tasks, including SQuAD 1.1 (Rajpurkar et al., 2016) and two sequence-level classiﬁcation tasks in GLUE benchmark (Wang et al., 2018).

Original: Our experiments reveal that the proposed approach grants us a ﬁne-grained control of computational efﬁciency and a superior accuracy-efﬁciency trade-off in the inference time, compared to existing approaches.
Revised: Our experiments reveal that the proposed approach grants us a ﬁne-grained control of computational efﬁciency and a superior accuracy-efﬁciency trade-off in the inference time, compared to existing approaches.

Original:  B ACKGROUND : T RANSFORMERS AND P O WER-BERT
Revised:  B ACKGROUND : T RANSFORMERS AND P O WER-BERT

Original: Before we describe our main approach, we review some of the building blocks in this section.
Revised: Before we describe our main approach, we review some of the building blocks in this section.

Original: In particular, we review transformers, which are a standard backbone used in natural language processing these days, and PoWER-BERT, which was recently proposed as an effective way to train a large-scale, but highly efﬁcient transformer for sequence-level classiﬁcation.
Revised: In particular, we review transformers, which are a standard backbone used in natural language processing these days, and PoWER-BERT, which was recently proposed as an effective way to train a large-scale, but highly efﬁcient transformer for sequence-level classiﬁcation.

Original: 2. T RANSFORMERS AND BERT
Revised: 2. T RANSFORMERS AND BERT

Original: A transformer is a particular neural network that has been designed to work with a variable-length sequence input and is implemented as a stack of self-attention and fully-connected layers (Vaswani et al., 2017).
Revised: A transformer is a particular neural network that has been designed to work with a variable-length sequence input and is implemented as a stack of self-attention and fully-connected layers (Vaswani et al., 2017).

Original: Here, we give a brief overview of the transformer which is the basic building block of the proposed approach.
Revised: Here, we give a brief overview of the transformer which is the basic building block of the proposed approach.

Original: Each token x t in a sequence of tokens x = ( x 1 , . .
Revised: Each token x t in a sequence of tokens x = ( x 1 , . .

Original: , x N ) , representing input text, is ﬁrst turned into a continuous vector h 0 t ∈ R H which is the sum of the token and position embedding vectors.
Revised: , x N ) , representing input text, is ﬁrst turned into a continuous vector h 0 t ∈ R H which is the sum of the token and position embedding vectors.

Original: This sequence is fed into the ﬁrst transformer layer which returns another sequence of the same length h 1 ∈ R N × H .
Revised: This sequence is fed into the ﬁrst transformer layer which returns another sequence of the same length h 1 ∈ R N × H .

Original: We repeat this procedure L times, for a transformer with L layers, to obtain h L =
Revised: We repeat this procedure L times, for a transformer with L layers, to obtain h L =

Original: ( h L 1 , . . .
Revised: ( h L 1 , . . .

Original: , h LN ) .
Revised: , h LN ) .

Original: We refer to each vector in the hidden sequence at each layer as a word vector to emphasize that there exists a correspondence between each such vector and one of the input words.
Revised: We refer to each vector in the hidden sequence at each layer as a word vector to emphasize that there exists a correspondence between each such vector and one of the input words.

Original: Although the transformer was ﬁrst introduced for the problem of machine translation, Devlin et al.
Revised: Although the transformer was ﬁrst introduced for the problem of machine translation, Devlin et al.

Original: (2018) demonstrated that the transformer can be trained and used as a masked language model.
Revised: (2018) demonstrated that the transformer can be trained and used as a masked language model.

Original: More speciﬁcally, Devlin et al.
Revised: More speciﬁcally, Devlin et al.

Original: (2018) showed that the transformer-based masked language model, called BERT, learns a universally useful parameter set that can be ﬁnetuned for any downstream task including sequence-level and token-level classiﬁcation.
Revised: (2018) showed that the transformer-based masked language model, called BERT, learns a universally useful parameter set that can be ﬁnetuned for any downstream task including sequence-level and token-level classiﬁcation.

Original: In the case of sequence-level classiﬁcation, a softmax classiﬁer is attached to the word vector h Lassociated with the special token [CLS] , and the entire network, including the softmax classiﬁer and
Revised: In the case of sequence-level classiﬁcation, a softmax classiﬁer is attached to the word vector h Lassociated with the special token [CLS] , and the entire network, including the softmax classiﬁer and

Original: BERT, is ﬁnetuned.
Revised: BERT, is ﬁnetuned.

Original: For token-level classiﬁcation, we use each h Lt as the ﬁnal hidden reprsentation of the associated t -th word in the input sequence.
Revised: For token-level classiﬁcation, we use each h Lt as the ﬁnal hidden reprsentation of the associated t -th word in the input sequence.

Original: This strategy of pretraining followed by ﬁnetuning, often referred to as transfer learning, has recently become a dominant approach to classiﬁcation in natural language processing.
Revised: This strategy of pretraining followed by ﬁnetuning, often referred to as transfer learning, has recently become a dominant approach to classiﬁcation in natural language processing.

Original: 2. P O WER-BERT
Revised: 2. P O WER-BERT

Original: PoWER-BERT keeps only the topmost l j word vectors at each layer j by eliminating redundant ones based on the signiﬁcance score, which is the total amount of attention imposed by a word on the other words (Goyal et al., 2020).
Revised: PoWER-BERT keeps only the topmost l j word vectors at each layer j by eliminating redundant ones based on the signiﬁcance score, which is the total amount of attention imposed by a word on the other words (Goyal et al., 2020).

Original: l j is the hyper-parameter determines how many vectors to keep at layer j .
Revised: l j is the hyper-parameter determines how many vectors to keep at layer j .

Original: PoWER-BERT has the same model parameters with BERT, but the extraction layers are interspersed after the self-attention layer in every transformer blocks (Vaswani et al., 2017).
Revised: PoWER-BERT has the same model parameters with BERT, but the extraction layers are interspersed after the self-attention layer in every transformer blocks (Vaswani et al., 2017).

Original: PoWER-BERT reduces inference time successfully, achieving better accuracy-time trade-off than
Revised: PoWER-BERT reduces inference time successfully, achieving better accuracy-time trade-off than

Original: DistilBERT (Sanh et al., 2019), BERT-PKD (Sun et al., 2019), and Head-Prune (Michel et al., 2019).
Revised: DistilBERT (Sanh et al., 2019), BERT-PKD (Sun et al., 2019), and Head-Prune (Michel et al., 2019).

Original: Despite the original intention of maximizing the inference efﬁciency with the minimal loss in accuracy, it is possible to set up PoWER-BERT to be both more efﬁcient and more accuracy compared to the original BERT, which was observed but largely overlooked by Goyal et al.
Revised: Despite the original intention of maximizing the inference efﬁciency with the minimal loss in accuracy, it is possible to set up PoWER-BERT to be both more efﬁcient and more accurate compared to the original BERT, which was observed but largely overlooked by Goyal et al.

Original: Training a PoWER-BERT model consists of three steps: (1) ﬁnetuning, (2) length conﬁguration search, and (3) re-training.
Revised: Training a PoWER-BERT model consists of three steps: (1) ﬁnetuning, (2) length conﬁguration search, and (3) re-training.

Original: The ﬁnetuning step is just like the standard ﬁnetuning step of BERT given a target task.
Revised: The ﬁnetuning step is just like the standard ﬁnetuning step of BERT given a target task.

Original: A length conﬁguration is a sequence of retention parameters ( l 1 , · · · l L ) , each of which corresponds to the number of word vectors that are kept at each layer.
Revised: A length conﬁguration is a sequence of retention parameters ( l 1 , · · · l L ) , each of which corresponds to the number of word vectors that are kept at each layer.

Original: These retention parameters are learned along with all the other parameters to minimize the original task loss together with an extra term that approximately measures the number of retained word vectors across layers.
Revised: These retention parameters are learned along with all the other parameters to minimize the original task loss together with an extra term that approximately measures the number of retained word vectors across layers.

Original: In the re-training step, PoWER-BERT is ﬁnetuned with the length conﬁguration ﬁxed to its learned one.
Revised: In the re-training step, PoWER-BERT is ﬁnetuned with the length conﬁguration ﬁxed to its learned one.

Original: For each computational budget, we must train a separate model going through all three steps described above.
Revised: For each computational budget, we must train a separate model going through all three steps described above.

Original: Moreover, the length conﬁguration search step above is only approximate, as it relies on relaxation of retention parameters which are inherently discrete.
Revised: Moreover, the length conﬁguration search step above is only approximate, as it relies on relaxation of retention parameters which are inherently discrete.

Original: This leads to the lack of guaranteed correlation between the success of this stage and true run-time.
Revised: This leads to the lack of guaranteed correlation between the success of this stage and true run-time.

Original: Even worse, it is a delicate act to tune the length conﬁguration given a target computational budget, because trade-off is implicitly made via a regularization coefﬁcient.
Revised: Even worse, it is a delicate act to tune the length conﬁguration given a target computational budget, because trade-off is implicitly made via a regularization coefﬁcient.

Original: Furthermore, PoWER-BERT has an inherent limitation in that it only applies to sequence-level classiﬁcation because it eliminates word vectors in intermediate layers.
Revised: Furthermore, PoWER-BERT has an inherent limitation in that it only applies to sequence-level classiﬁcation because it eliminates word vectors in intermediate layers.

Original:  L ENGTH -A DAPTIVE T RANSFORMER
Revised:  L ENGTH -A DAPTIVE T RANSFORMER

Original: In this section, we explain our proposed framework which results in a transformer that reduces the length of a sequence at each layer with an arbitrary rate.
Revised: In this section, we explain our proposed framework which results in a transformer that reduces the length of a sequence at each layer with an arbitrary rate.

Original: We call such a resulting transformer a Length-Adaptive Transformer.
Revised: We call such a resulting transformer a Length-Adaptive Transformer.

Original: We train Length-Adaptive Transformer with LengthDrop which randomly samples the number of hidden vectors to be dropped at each layer with the goal of making the ﬁnal model robust to such drop in the inference time.
Revised: We train Length-Adaptive Transformer with LengthDrop which randomly samples the number of hidden vectors to be dropped at each layer with the goal of making the ﬁnal model robust to such drop in the inference time.

Original: Once the model is trained, we search for the optimal trade-off between accuracy and efﬁciency using multi-objective evolutionary search, which allows us to use the model for any given computational budget without ﬁnetuning nor re-training.
Revised: Once the model is trained, we search for the optimal trade-off between accuracy and efﬁciency using multi-objective evolutionary search, which allows us to use the model for any given computational budget without ﬁnetuning nor re-training.

Original: At the end of this section, we describe Drop-and-Restore process as a way to greatly increase the applicability of PoWER-BERT which forms a building block of the proposed framework.
Revised: At the end of this section, we describe Drop-and-Restore process as a way to greatly increase the applicability of PoWER-BERT which forms a building block of the proposed framework.

Original: In short, we train a Length-Adaptive Transformer once with LengthDrop and Drop-and-Restore, and use it with an automatically determined length conﬁguration for inference with any target computational budget,
Revised: In short, we train a Length-Adaptive Transformer once with LengthDrop and Drop-and-Restore, and use it with an automatically determined length conﬁguration for inference with any target computational budget,

Original: 3. L ENGTH D ROP
Revised: 3. L ENGTH D ROP

Original: Earlier approaches to efﬁcient inference with transformers have focused on a scenario where the target computational budget for inference is known in advance Sanh et al.
Revised: Earlier approaches to efﬁcient inference with transformers have focused on a scenario where the target computational budget for inference is known in advance Sanh et al.

Original: ; Goyal et al.
Revised: ; Goyal et al.

Original: This greatly increases the cost of deploying transformers, as it requires us to train a separate transformer for each scenario.
Revised: This greatly increases the cost of deploying transformers, as it requires us to train a separate transformer for each scenario.

Original: Instead, we propose to train one model that could be used for a diverse set of target computational budgets without retraining.
Revised: Instead, we propose to train one model that could be used for a diverse set of target computational budgets without retraining.

Original: LengthDrop randomly generates a length conﬁguration by sequentially sampling a sequence length l i +1 at the ( i + 1)
Revised: LengthDrop randomly generates a length conﬁguration by sequentially sampling a sequence length l i +1 at the ( i + 1)

Original: -th layer based on the previous layer’s sequence length l i , following the uniform distribution U ( (cid:100) (1 − p )
Revised: -th layer based on the previous layer’s sequence length l i , following the uniform distribution U ( (cid:100) (1 − p )

Original: l i (cid:101) , l i ) , where l 0 is set to the length of the input sequence, and p is the
Revised: l i (cid:101) , l i ) , where l 0 is set to the length of the input sequence, and p is the

Original: LengthDrop probability.
Revised: LengthDrop probability.

Original: This sequential sampling results in a length conﬁguration ( l 1 , · · · , l L ) .
Revised: This sequential sampling results in a length conﬁguration ( l 1 , · · · , l L ) .

Original: Length-Adaptive Transformer can be thought of as consisting of a full model and many sub-models corresponding to different length conﬁguration, similarly to a neural network trained with dropout (Srivastava et al., 2014).
Revised: Length-Adaptive Transformer can be thought of as consisting of a full model and many sub-models corresponding to different length conﬁguration, similarly to a neural network trained with dropout (Srivastava et al., 2014).

Original: LayerDrop From the perspective of each word vector, the proposed LengthDrop could be thought of as skipping the layers between when it was set aside and the ﬁnal layer where it was restored.
Revised: LayerDrop From the perspective of each word vector, the proposed LengthDrop could be thought of as skipping the layers between when it was set aside and the ﬁnal layer where it was restored.

Original: The word vector however does not have any information based on which it can determine whether it would be dropped at any particular layer.
Revised: The word vector however does not have any information based on which it can determine whether it would be dropped at any particular layer.

Original: In our preliminary experiments, we found that this greatly hinders optimization.
Revised: In our preliminary experiments, we found that this greatly hinders optimization.

Original: We address this issue by using LayerDrop (Fan et al., 2019) which skips each layer of a transformer uniformly at random.
Revised: We address this issue by using LayerDrop (Fan et al., 2019) which skips each layer of a transformer uniformly at random.

Original: The LayerDrop encourages each word vector to be agnostic to skipping any number of layers between when it is dropped and when it is restored, just like dropout (Srivastava et al., 2014) prevents hidden neurons from co-adapting with each other by randomly dropping them.
Revised: The LayerDrop encourages each word vector to be agnostic to skipping any number of layers between when it is dropped and when it is restored, just like dropout (Srivastava et al., 2014) prevents hidden neurons from co-adapting with each other by randomly dropping them.

Original: Sandwich Rule and Inplace Distillation
Revised: Sandwich Rule and Inplace Distillation

Original: We observed that standard supervised training with
Revised: We observed that standard supervised training with

Original: LengthDrop does not work well in the preliminary experiments.
Revised: LengthDrop does not work well in the preliminary experiments.

Original: We instead borrow a pair of training techniques developed by Yu & Huang (2019) which are sandwich rule and inplace distillation, for better optimization as well as ﬁnal generalization.
Revised: We instead borrow a pair of training techniques developed by Yu & Huang (2019) which are sandwich rule and inplace distillation, for better optimization as well as ﬁnal generalization.

Original: At each update, we update the full model without LengthDrop as usual to minimize the supervised loss function.
Revised: At each update, we update the full model without LengthDrop as usual to minimize the supervised loss function.

Original: We simultaneously update n s randomly-sampled sub-models (which are called sandwiches) and the smallest-possible submodel, which corresponds to keeping only (cid:100) (1 − p ) l i (cid:101) word vectors at each layer i , using knowledge distillation (Hinton et al., 2015) from the full model. 
Revised: We simultaneously update n s randomly-sampled sub-models (which are called sandwiches) and the smallest-possible submodel, which corresponds to keeping only (cid:100) (1 − p ) l i (cid:101) word vectors at each layer i , using knowledge distillation (Hinton et al., 2015) from the full model. Here, sub-models mean models with length reduction. They are trained to their prediction close to the full model’s prediction (inplace distillation).

Original: 3. E VOLUTIONARY S EARCH OF L ENGTH C ONFIGURATIONS
Revised: 3. E VOLUTIONARY S EARCH OF L ENGTH C ONFIGURATIONS

Original: After training a Length-Adaptive Transformer with LengthDrop, we search foran appropriate length conﬁguration based on a target computational budget for inference.
Revised: After training a Length-Adaptive Transformer with LengthDrop, we search for appropriate length conﬁgurations for possible target computational budgets that will be given at inference time.

Original: The length conﬁguration determines the model performance in terms of both accuracy and efﬁciency.
Revised: The length conﬁguration determines the model performance in terms of both accuracy and efﬁciency.

Original: In order to search for the optimal length conﬁguration, we propose to use evolutionary search, similarly to Cai et al.
Revised: In order to search for the optimal length conﬁguration, we propose to use evolutionary search, similarly to Cai et al.

Original:  and Wang et al.
Revised: (2019) and Wang et al.

Original: This procedure is efﬁcient, as it only require a single pass through the relatively small validation set for each length conﬁguration, unlike re-training for a new computational budget which requires multiple passes through a signiﬁcantly larger training set for each budget.
Revised: This procedure is efﬁcient, as it only requires a single pass through the relatively small validation set for each length conﬁguration, unlike re-training for a new computational budget which requires multiple passes through a signiﬁcantly larger training set for each budget.

Original: We initialize the population with constant-ratio conﬁgurations.
Revised: We initialize the population with constant-ratio conﬁgurations.

Original: Each conﬁguration is created by l i +

Revised: Each conﬁguration is created by l i +

Original: =
Revised: =

Original: (cid:100) (1 − r ) l i (cid:101) for each layer i with r so that the amount of computation within the initial population is uniformly distributed between those of the smallest and full models.
Revised: (cid:100) (1 − r ) l i (cid:101) for each layer i with r so that the amount of computation within the initial population is uniformly distributed between those of the smallest and full models.

Original: At each iteration, we evolve the population to consist only of conﬁgurations lie on a newly updated efﬁciencyaccuracy Pareto frontier by mutation and cross-over.
Revised: At each iteration, we evolve the population to consist only of conﬁgurations lie on a newly updated efﬁciencyaccuracy Pareto frontier by mutation and cross-over.

Original: Mutation alters an original length conﬁguration ( l 1 , · · · , l L ) to ( l (cid:48) 1 , · · · , l (cid:48) L ) by sampling l (cid:48)
Revised: Mutation alters an original length conﬁguration ( l 1 , · · · , l L ) to ( l (cid:48) 1 , · · · , l (cid:48) L ) by sampling l (cid:48)

Original: i from the uniform distribution U ( l (cid:48)
Revised: i from the uniform distribution U ( l (cid:48)

Original: i − 1 , l i +1 ) with the probability p m or keeping the original length l (cid:48)
Revised: i − 1 , l i +1 ) with the probability p m or keeping the original length l (cid:48)

Original: i =
Revised: i =

Original: l i , sweeping the layers from i = 1 to i =
Revised: l i , sweeping the layers from i = 1 to i =

Original: L .
Revised: L .

Original: Crossover takes two length conﬁgurations and averages the lengths at each layer.
Revised: Crossover takes two length conﬁgurations and averages the lengths at each layer.

Original: Both of these operations are performed while ensuring the monotonicity of the lengths over the layers.
Revised: Both of these operations are performed while ensuring the monotonicity of the lengths over the layers.

Original: We repeat this iteration G times, while maintaining n m mutated conﬁgurations and n c crossover’d conﬁgurations.
Revised: We repeat this iteration G times, while maintaining n m mutated conﬁgurations and n c crossover’d conﬁgurations.

Original: Repeating this procedure pushes the Pareto frontier further to identify the best trade-off between two objectives, efﬁciency and accuracy, without requiring any continuous relaxation of length conﬁgurations nor using a proxy objective function.
Revised: Repeating this procedure pushes the Pareto frontier further to identify the best trade-off between two objectives, efﬁciency and accuracy, without requiring any continuous relaxation of length conﬁgurations nor using a proxy objective function.

Original: [Figure]
Revised: [Figure]

Original: 3. D ROP - AND -R ESTORE P ROCESS
Revised: 3. D ROP - AND -R ESTORE P ROCESS

Original: The applicability of the PoWER-BERT, based on which our main contribution above was made, is limited to sequence-level classiﬁcation, because it eliminates word vectors at each layer.
Revised: The applicability of the PoWER-BERT, based on which our main contribution above was made, is limited to sequence-level classiﬁcation, because it eliminates word vectors at each layer.

Original: In addition to our main contribution above, we thus propose to extend the PoWER-BERT so that it is applicable to token-level classiﬁcation, such as span-based question-answering.
Revised: In addition to our main contribution above, we thus propose to extend the PoWER-BERT so that it is applicable to token-level classiﬁcation, such as span-based question-answering.

Original: Our proposal, to which we refer as Drop-and-Restore, does not eliminate word vectors at each layer according to the length conﬁguration but instead sets them aside until the ﬁnal hidden layer.
Revised: Our proposal, to which we refer as Drop-and-Restore, does not eliminate word vectors at each layer according to the length conﬁguration but instead sets them aside until the ﬁnal hidden layer.

Original: At the ﬁnal hidden layer, these word vectors are brought back to form the full hidden sequence, as illustrated graphically in Fig.
Revised: At the ﬁnal hidden layer, these word vectors are brought back to form the full hidden sequence, as illustrated graphically in Fig.

Original:  E XPERIMENT S ETUP
Revised:  E XPERIMENT S ETUP

Original: Datasets We test the proposed approach on both sequence-level and token-level tasks, the latter of which could not have been done with the original PoWER-BERT unless for the proposed Dropand-Restore.
Revised: Datasets We test the proposed approach on both sequence-level and token-level tasks, the latter of which could not have been done with the original PoWER-BERT unless for the proposed Dropand-Restore.

Original: We use MNLI-m and SST-2 from GLUE benchmark (Wang et al., 2018), as was done to test PoWER-BERT earlier, for sequence-level classiﬁcation.
Revised: We use MNLI-m and SST-2 from GLUE benchmark (Wang et al., 2018), as was done to test PoWER-BERT earlier, for sequence-level classiﬁcation.

Original: We choose them because consistent accuracy scores from standard training on them due to their sufﬁciently large training set imply that they are reliable to verify our approach.
Revised: We choose them because consistent accuracy scores from standard training on them due to their sufﬁciently large training set imply that they are reliable to verify our approach.

Original: We use SQuAD 1.1 (Rajpurkar et al., 2016) for token-level classiﬁcation.
Revised: We use SQuAD 1.1 (Rajpurkar et al., 2016) for token-level classiﬁcation.

Original: Evaluation metrics We use the number of ﬂoating operations (FLOPs) as a main metric to measure the inference efﬁciency given any length conﬁguration, as it is agnostic to the choice of underlying hardware, unlike other alternatives such as hardware-aware latency (Wang et al., 2020) or energy consumption (Henderson et al., 2020).
Revised: Evaluation metrics We use the number of ﬂoating operations (FLOPs) as a main metric to measure the inference efﬁciency given any length conﬁguration, as it is agnostic to the choice of underlying hardware, unlike other alternatives such as hardware-aware latency (Wang et al., 2020) or energy consumption (Henderson et al., 2020).

Original: We later demonstrate that FLOPs and wall-clock time on
Revised: We later demonstrate that FLOPs and wall-clock time on

Original: GPU and CPU correlate well with the proposed approach, which is not necessarily the case for other approaches, such as unstructured weight pruning (Han et al., 2015; See et al., 2016).
Revised: GPU and CPU correlate well with the proposed approach, which is not necessarily the case for other approaches, such as unstructured weight pruning (Han et al., 2015; See et al., 2016).

Original: Pretrained transformers Since BERT was introduced by Devlin et al.
Revised: Pretrained transformers Since BERT was introduced by Devlin et al.

Original: (2018), it has become a standard practice to start from a pretrained (masked) language model and ﬁnetune it for each downstream task.
Revised: (2018), it has become a standard practice to start from a pretrained (masked) language model and ﬁnetune it for each downstream task.

Original: We follow the same strategy in this paper and test two pretrained transformerbased language models; BERT BASE (Devlin et al., 2018) and DistilBERT (Sanh et al., 2019), which allows us to demonstrate that the usefulness and applicability of our approach are not tied to any speciﬁc architectural choice, such as the number of layers and the maximum length of input sequence.
Revised: We follow the same strategy in this paper and test two pretrained transformerbased language models; BERT BASE (Devlin et al., 2018) and DistilBERT (Sanh et al., 2019), which allows us to demonstrate that the usefulness and applicability of our approach are not tied to any speciﬁc architectural choice, such as the number of layers and the maximum length of input sequence.

Original: Although we focus on BERT-based masked language models here, the proposed approach is readily applicable to any transformer-based models.
Revised: Although we focus on BERT-based masked language models here, the proposed approach is readily applicable to any transformer-based models.

Original: Learning We train a Length-Adaptive Transformer with LengthDrop probability and LayerDrop probability both set to 0 .
Revised: Learning We train a Length-Adaptive Transformer with LengthDrop probability and LayerDrop probability both set to 0 .

Original: We use n s = 2 randomly sampled intermediate sub-models in addition to the full model and smallest model for applying the sandwich learning rule.
Revised: We use n s = 2 randomly sampled intermediate sub-models in addition to the full model and smallest model for applying the sandwich learning rule.

Original: For token-level classiﬁcation, we start ﬁnetuning the pretrained transformer without Drop-and-Restore ﬁrst, just as Goyal et al.
Revised: We start ﬁnetuning the pretrained transformer without Drop-and-Restore ﬁrst, just as Goyal et al.

Original: (2020) did with PoWER-BERT.
Revised: (2020) did with PoWER-BERT.

Original: We then continue ﬁnetuning it for another ﬁve epochs with Drop-and-Restore.
Revised: We then continue ﬁnetuning it for another ﬁve epochs with Drop-andRestore.

Original: This is unlike the recommended three epochs by Devlin et al.
Revised: This is unlike the recommended three epochs by Devlin et al.

Revised: (2018), as learning progresses

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (2018), as learning progresses slower due to a higher level of stochasticity introduced by LengthDrop and LayerDrop.
Revised: slower due to a higher level of stochasticity introduced by LengthDrop and LayerDrop.

Revised: We use the batch size of 32 , the learning rate of 5 e − 5 for SQuAD v1.1 and 2 e − 5 for MNLI-m and SST and the maximum sequence length of 384 for SQuAD v1.1 and 128 for MNLI-m and SST.

Original: Search We run up to G = 30 iterations of evolutionary search, using n m = 30 mutated conﬁgurations with mutation probability p m = 0 .
Revised: Search We run up to G = 30 iterations of evolutionary search, using n m = 30 mutated conﬁgurations with mutation probability p m = 0 .

Original: 5 and n c = 30 crossover’d conﬁgurations, to ﬁnd the Pareto frontier of accuracy and efﬁciency.
Revised: 5 and n c = 30 crossover’d conﬁgurations, to ﬁnd the Pareto frontier of accuracy and efﬁciency.

Original:  R ESULTS AND A NALYSIS
Revised:  R ESULTS AND A NALYSIS

Original: Efﬁciency-accuracy trade-off We use SQuAD 1.1 to examine the effect of the proposed approach on the efﬁciency-accuracy trade-off.
Revised: Efﬁciency-accuracy trade-off We use SQuAD 1.1 to examine the effect of the proposed approach on the efﬁciency-accuracy trade-off.

Original: When the underlying classiﬁer was not trained with LengthDrop, as proposed in this paper, the accuracy drops even more dramatically as more word vectors are dropped at each layer.
Revised: When the underlying classiﬁer was not trained with LengthDrop, as proposed in this paper, the accuracy drops even more dramatically as more word vectors are dropped at each layer.

Original: The difference between standard transformer and Length-Adaptive Transformer is stark in Fig.
Revised: The difference between standard transformer and Length-Adaptive Transformer is stark in Fig.

Original: This veriﬁes the importance of training a transformer in a way that makes it malleable for inference-time re-conﬁguration.
Revised: This veriﬁes the importance of training a transformer in a way that makes it malleable for inference-time re-conﬁguration.

Original: When the model was trained with the proposed LengthDrop, we notice the efﬁcacy of the proposed approach of using evolutionary search to ﬁnd the optimal trade-off between inference efﬁciency and accuracy.
Revised: When the model was trained with the proposed LengthDrop, we notice the efﬁcacy of the proposed approach of using evolutionary search to ﬁnd the optimal trade-off between inference efﬁciency and accuracy.

Original: The trade-off curve from the proposed search strategy has a larger area-under-curve (AUC) than when constant-rate length reduction was used to meet a target computational budget.
Revised: The trade-off curve from the proposed search strategy has a larger area-under-curve (AUC) than when constant-rate length reduction was used to meet a target computational budget.

Original: It demonstrates the importance of using both LengthDrop and evolutionary search.
Revised: It demonstrates the importance of using both LengthDrop and evolutionary search.

Original: We make a minor observation that the proposed approach ends up with a signiﬁcantly higher accuracy than DistillBERT when enough computational budget is allowed for inference ( log FLOPs > 10 ).
Revised: We make a minor observation that the proposed approach ends up with a signiﬁcantly higher accuracy than DistillBERT when enough computational budget is allowed for inference ( log FLOPs > 10 ).

Original: This makes our approach desirable in a wide array of scenarios, as it does not require any additional pretraining stage, as does DistilBERT.
Revised: This makes our approach desirable in a wide array of scenarios, as it does not require any additional pretraining stage, as does DistilBERT.

Original: With a severe constraint on the computational budget, the proposed approach could be used on DistilBERT to signiﬁcantly improve the efﬁciency without compromising the accuracy.
Revised: With a severe constraint on the computational budget, the proposed approach could be used on DistilBERT to signiﬁcantly improve the efﬁciency without compromising the accuracy.

Original: Maximizing inference efﬁciency We consider all three tasks, SQuAD 1.1, MNLI-m and SST-2, and investigate how much efﬁciency can be gained by the proposed approach with minimal sacriﬁce of
Revised: Maximizing inference efﬁciency We consider all three tasks, SQuAD 1.1, MNLI-m and SST-2, and investigate how much efﬁciency can be gained by the proposed approach with minimal sacriﬁce of

Original: [Table]
Revised: [Table]

Original: Iteration
Revised: Iteration

Original: A UC 88.
Revised: A UC 88.

Original: accuracy.
Revised: accuracy.

Original: First, we look at how much efﬁciency could be gained without losing on the accuracy.
Revised: First, we look at how much efﬁciency could be gained without losing on the accuracy.

Original: That is, we use the length conﬁguration that maximizes the inference efﬁciency (i e., minimize the FLOPs) while ensuring that the accuracy is above or same as the accuracy of the standard approach without any drop of word vectors.
Revised: That is, we use the length conﬁguration that maximizes the inference efﬁciency (i e., minimize the FLOPs) while ensuring that the accuracy is above or same as the accuracy of the standard approach without any drop of word vectors.

Original: The results are presented in the rows marked with Lengnth-Adaptive † from
Revised: The results are presented in the rows marked with Length-Adaptive † from

Original: Table 1.
Revised: Table 1.

Original: For example, in the case of BERT Base , the proposed approach reduce FLOPs by more than half across all three tasks.
Revised: For example, in the case of BERT Base , the proposed approach reduce FLOPs by more than half across all three tasks.

Original: From Fig.
Revised: From Fig.

Original: 2, we have observed that the proposed Length-Adaptive Transformer generalize better than the standard, base model in some cases.
Revised: 2, we have observed that the proposed Length-Adaptive Transformer generalize better than the standard, base model in some cases.

Original: We thus try to maximize both the inference efﬁciency and accuracy, in order to see whether it is possible for the proposed algorithm to ﬁnd a length conﬁguration that both maximizes inference efﬁciency and improves accuracy.
Revised: We thus try to maximize both the inference efﬁciency and accuracy, in order to see whether it is possible for the proposed algorithm to ﬁnd a length conﬁguration that both maximizes inference efﬁciency and improves accuracy.

Original: We present the results in the rows marked with Lengnth-Adaptive (cid:63) from Table 1.
Revised: We present the results in the rows marked with Length-Adaptive (cid:63) from Table 1.

Original: For all cases, Length-Adaptive Transformer achieves higher accuracy than a standard transformer does while reducing FLOPs signiﬁcantly.
Revised: For all cases, Length-Adaptive Transformer achieves higher accuracy than a standard transformer does while reducing FLOPs signiﬁcantly.

Original: Although it is not apparent from the table, tor MNLI-m and SST-2, the accuracy of the smallest sub-model is already greater than or equal to that of a standard transformer.
Revised: Although it is not apparent from the table, tor MNLI-m and SST-2, the accuracy of the smallest sub-model is already greater than or equal to that of a standard transformer.

Original: FLOPs vs. Latency As has been discussed in recent literature (see, e.g., (Li et al., 2020; Chin et al., 2020)), FLOPs is not a perfect indicator of the real latency measured in wall-clock time, as the latter is affected by the combination of hardware choice and network architecture.
Revised: FLOPs vs. Latency As has been discussed in recent literature (see, e.g., (Li et al., 2020; Chin et al., 2020)), FLOPs is not a perfect indicator of the real latency measured in wall-clock time, as the latter is affected by the combination of hardware choice and network architecture.

Original: To understand the real-world impact of the proposed approach, we study the relationship between FLOPs, obtained by the proposed procedure, and wall-clock time measured on both CPU and GPU by measuring them while varying length conﬁgurations.
Revised: To understand the real-world impact of the proposed approach, we study the relationship between FLOPs, obtained by the proposed procedure, and wall-clock time measured on both CPU and GPU by measuring them while varying length conﬁgurations.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 3, FLOPs and latency exhibit near-linear correlation on GPU, when the minibatch size is ≥ 16 , and regardless of the minibatch size, on CPU.
Revised: 3, FLOPs and latency exhibit near-linear correlation on GPU, when the minibatch size is ≥ 16 , and regardless of the minibatch size, on CPU.

Original: In other words, the reduction in FLOPs with the proposed approach directly implies the reduction in wall-clock time.
Revised: In other words, the reduction in FLOPs with the proposed approach directly implies the reduction in wall-clock time.

Original: Convergence of search Although the proposed approach is efﬁcient in that it requires only one round of training, it needs a separate search stage for each target budget.
Revised: Convergence of search Although the proposed approach is efﬁcient in that it requires only one round of training, it needs a separate search stage for each target budget.

Original: It is important for evoluationary search to converge quickly in the number of forward sweeps of a validation set.
Revised: It is important for evolutionary search to converge quickly in the number of forward sweeps of a validation set.

Original: As exempliﬁed in
Revised: As exempliﬁed in

Original: Fig.
Revised: Fig.

Original: 4,it converges rapidly evolutionary search converges after about ﬁfteen iterations.
Revised: 4, evolutionary search converges after about ﬁfteen iterations.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The main purpose of the proposed algorithm is to improve the inference efﬁciency of a large-scale transformer.
Revised: The main purpose of the proposed algorithm is to improve the inference efﬁciency of a large-scale transformer.

Original: This goal has been pursued from various directions, and in this section, we provide a brief overview of these earlier, and some concurrent, attempts in the context of the proposed approach.
Revised: This goal has been pursued from various directions, and in this section, we provide a brief overview of these earlier, and some concurrent, attempts in the context of the proposed approach.

Original: Weight pruning Weight pruning (Han et al., 2015) focuses on reducing the number of parameters which directly reﬂects the memory footprint of a model and indirectly correlates with inference speed.
Revised: Weight pruning Weight pruning (Han et al., 2015) focuses on reducing the number of parameters which directly reﬂects the memory footprint of a model and indirectly correlates with inference speed.

Original: However, their actual speed-up in runtime is usually not signiﬁcant, especially while executing a model with parallel computation using GPU devices (Tang et al., 2018; Li et al., 2020).
Revised: However, their actual speed-up in runtime is usually not signiﬁcant, especially while executing a model with parallel computation using GPU devices (Tang et al., 2018; Li et al., 2020).

Original: Adaptive architecture There are three major axes along which computation can be reduced in a neural network; (1) input size/length, (2) network depth and (3) network width.
Revised: Adaptive architecture There are three major axes along which computation can be reduced in a neural network; (1) input size/length, (2) network depth and (3) network width.

Original: The proposed approach, based on PoWER-BERT, adaptively reduces the input length as the input sequence is processed by the transformer layers.
Revised: The proposed approach, based on PoWER-BERT, adaptively reduces the input length as the input sequence is processed by the transformer layers.

Original: In our knowledge, Goyal et al.
Revised: In our knowledge, Goyal et al.

Original: (2020) is the ﬁrst work in this direction for transformers.
Revised: (2020) is the ﬁrst work in this direction for transformers.

Original: More recently, Funnel-Transformer (Dai et al., 2020) and multi-scale transformer language models (Subramanian et al., 2020) also successfully reduce sequence length in the middle and rescale to full length for the ﬁnal computation.
Revised: More recently, Funnel-Transformer (Dai et al., 2020) and multi-scale transformer language models (Subramanian et al., 2020) also successfully reduce sequence length in the middle and rescale to full length for the ﬁnal computation.

Original: However, their inference complexity is ﬁxed unlike PoWER-BERT because they are not designed for the control of efﬁciency.
Revised: However, their inference complexity is ﬁxed unlike PoWER-BERT because they are not designed for the control of efﬁciency.

Original: LayerDrop (Fan et al., 2019) drops random layers during the training to be robust to pruning inspired by Huang et al.
Revised: LayerDrop (Fan et al., 2019) drops random layers during the training to be robust to pruning inspired by Huang et al.

Original: Word-level adaptive depth in Elbayad et al.
Revised: Word-level adaptive depth in Elbayad et al.

Original: (2019) might seemingly resemble with length reduction, but word vectors reached the maximal layer are used for self-attention computation without updating themselves.
Revised: (2019) might seemingly resemble with length reduction, but word vectors reached the maximal layer are used for self-attention computation without updating themselves.

Original: Escaping a network early (Teerapittayanon et al., 2016;
Revised: Escaping a network early (Teerapittayanon et al., 2016;

Original: Huang et al., 2017)
Revised: Huang et al., 2017)

Original: based on the conﬁdence of the prediction (Xin et al., 2020; Schwartz et al., 2020;
Revised: based on the conﬁdence of the prediction (Xin et al., 2020; Schwartz et al., 2020;

Original: Liu et al., 2020) also offers a control over accuracy-efﬁciency trade-off, but it is difﬁcult to tune a threshold for a desired computational budget because of the example-wise adaptive computation.
Revised: Liu et al., 2020) also offers a control over accuracy-efﬁciency trade-off, but it is difﬁcult to tune a threshold for a desired computational budget because of the example-wise adaptive computation.

Original: DynaBERT (Hou et al., 2020) can run at adaptive width (the number of attention heads and intermediate hidden dimension) and depth.
Revised: DynaBERT (Hou et al., 2020) can run at adaptive width (the number of attention heads and intermediate hidden dimension) and depth.

Original: Hardware-aware Transformers (Wang et al., 2020) construct a design space with arbitrary encoder-decoder attention and heterogeneous layers in terms of different numbers of layers, attention heads, hidden dimension, and embedding dimension.
Revised: Hardware-aware Transformers (Wang et al., 2020) construct a design space with arbitrary encoder-decoder attention and heterogeneous layers in terms of different numbers of layers, attention heads, hidden dimension, and embedding dimension.

Original: Structured dropout
Revised: Structured dropout

Original: A major innovation we introduce over the existing PoWER-BERT is the use of stochastic, structured regularization to make a transformer robust to the choice of length conﬁguration in the inference time.
Revised: A major innovation we introduce over the existing PoWER-BERT is the use of stochastic, structured regularization to make a transformer robust to the choice of length conﬁguration in the inference time.

Original: Rippel et al.
Revised: Rippel et al.

Original: (2014) proposes a nested dropout to learn ordered representations.
Revised: (2014) proposes a nested dropout to learn ordered representations.

Original: Similar to LengthDrop, it samples an index form a prior distribution and drops all units having a larger index than sampled one.
Revised: Similar to LengthDrop, it samples an index form a prior distribution and drops all units having a larger index than sampled one.

Original: Search There have been a series of attempts at ﬁnding the optimal network conﬁguration by solving a combinatorial optimization problem.
Revised: Search There have been a series of attempts at ﬁnding the optimal network conﬁguration by solving a combinatorial optimization problem.

Original: In computer vision, Once-for-All (Cai et al., 2019) use an evolutionary search (Real et al., 2019) to ﬁnd a better conﬁguration in dimensions of depth, width, kernel size, and resolution given computational budget.
Revised: In computer vision, Once-for-All (Cai et al., 2019) use an evolutionary search (Real et al., 2019) to ﬁnd a better conﬁguration in dimensions of depth, width, kernel size, and resolution given computational budget.

Original: Similarly but differently, our evolutionary search is mutli-objective to ﬁnd length conﬁgurations on the Pareto accuracy-efﬁciency frontier to cope with any possible computational budgets.
Revised: Similarly but differently, our evolutionary search is mutli-objective to ﬁnd length conﬁgurations on the Pareto accuracy-efﬁciency frontier to cope with any possible computational budgets.

Original: Moreover, we only change the sequence length of hidden vectors instead of architectural model size like dimensions.
Revised: Moreover, we only change the sequence length of hidden vectors instead of architectural model size like dimensions.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this work, we propose a new framework for training a transformer once and using it for efﬁcient inference under any computational budget.
Revised: In this work, we propose a new framework for training a transformer once and using it for efﬁcient inference under any computational budget.

Original: With the help of training with LengthDrop and Drop-andRestore process followed by the evolutionary search, our proposed Length-Adaptive Transformer allows any given transformer models to be used with any inference-time computational budget for both sequence-level and token-level classiﬁcation tasks.
Revised: With the help of training with LengthDrop and Drop-andRestore process followed by the evolutionary search, our proposed Length-Adaptive Transformer allows any given transformer models to be used with any inference-time computational budget for both sequence-level and token-level classiﬁcation tasks.

Original: Our experiments, on SQuAD 1.1, MNLI-m and SST-2, have revealed that the proposed algorithmic framework signiﬁcantly pushes a better Pareto frontier on the trade-off between inference efﬁciency and accuracy.
Revised: Our experiments, on SQuAD 1.1, MNLI-m and SST-2, have revealed that the proposed algorithmic framework signiﬁcantly pushes a better Pareto frontier on the trade-off between inference efﬁciency and accuracy.

Original: Furthermore, we have observed that the proposed Length-Adaptive Transformer could achieve up to 3x speed-up over the standard transformer without sacriﬁcing accuracy, both in terms of FLOPs and wallclock time.
Revised: Furthermore, we have observed that the proposed Length-Adaptive Transformer could achieve up to 3x speed-up over the standard transformer without sacriﬁcing accuracy, both in terms of FLOPs and wallclock time.

Original: Although our approach ﬁnds an optimal length conﬁguration of a trained classiﬁer per computational budget, it leaves open a question whether the proposed approach could be further extended to support per-instance length conﬁguration by for instance training a small, auxiliary neural network for each computational budget.
Revised: Although our approach ﬁnds an optimal length conﬁguration of a trained classiﬁer per computational budget, it leaves open a question whether the proposed approach could be further extended to support per-instance length conﬁguration by for instance training a small, auxiliary neural network for each computational budget.

Original: Yet another aspect we have not investigated in this paper is the applicability of the proposed approach to sequence generation, such as machine translation.
Revised: Yet another aspect we have not investigated in this paper is the applicability of the proposed approach to sequence generation, such as machine translation.

Original: We leave both of these research directions for the future.
Revised: We leave both of these research directions for the future.

Original: Our approach is effective, as we have shown in this paper, and also quite simple to implement on top of existing language models.
Revised: Our approach is effective, as we have shown in this paper, and also quite simple to implement on top of existing language models.

Original: We will release our implementation, which is based on HuggingFace’s
Revised: We will release our implementation, which is based on HuggingFace’s

Original: Transformers library (Wolf et al., 2019), publicly and plan to adapt it for a broader set of transformerbased models and downstream tasks.
Revised: Transformers library (Wolf et al., 2019), publicly and plan to adapt it for a broader set of transformerbased models and downstream tasks.