Original: R ESERVOIR T RANSFORMERS
Revised: R ESERVOIR T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.
Revised: We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated.

Original: Inspired by old and wellestablished ideas in machine learning, we explore a variety of non-linear “reservoir” layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.
Revised: Inspired by old and wellestablished ideas in machine learning, we explore a variety of non-linear “reservoir” layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers (Vaswani et al., 2017) have dominated natural language processing (NLP) in recent years, from large scale machine translation (Ott et al., 2018) to pre-trained (masked) language modeling (Devlin et al., 2018;
Revised: Transformers (Vaswani et al., 2017) have dominated natural language processing (NLP) in recent years, from large scale machine translation (Ott et al., 2018) to pre-trained (masked) language modeling (Devlin et al., 2018;

Original: Radford et al., 2018), and are becoming more popular in other ﬁelds as well, from reinforcement learning (Vinyals et al., 2019) to speech recognition (Baevski et al., 2019) and computer vision (Carion et al., 2020).
Revised: Radford et al., 2018), and are becoming more popular in other ﬁelds as well, from reinforcement learning (Vinyals et al., 2019) to speech recognition (Baevski et al., 2019) and computer vision (Carion et al., 2020).

Original: Their success is enabled in part by ever increasing computational demands, which has naturally led to an increased interest in improving their efﬁciency.
Revised: Their success is enabled in part by ever increasing computational demands, which has naturally led to an increased interest in improving their efﬁciency.

Original: Scalability gains in transformers could facilitate bigger, deeper networks with longer contexts (Kitaev et al., 2020;
Revised: Scalability gains in transformers could facilitate bigger, deeper networks with longer contexts (Kitaev et al., 2020;

Original: Wang et al., 2020; Beltagy et al., 2020; Kaplan et al., 2020; Tay et al., 2020b).
Revised: Wang et al., 2020; Beltagy et al., 2020; Kaplan et al., 2020; Tay et al., 2020b).

Original: Conversely, improved efﬁciency could reduce environmental costs (Strubell et al., 2019) and hopefully help democratize the technology.
Revised: Conversely, improved efﬁciency could reduce environmental costs (Strubell et al., 2019) and hopefully help democratize the technology.

Original: In this work, we explore a simple question: if some layers of the transformer are kept frozen—i.e., never updated after random initialization—can we match the performance of fully learned transformers, while being more efﬁcient?
Revised: In this work, we explore a simple question: if some layers of the transformer are kept frozen—i.e., never updated after random initialization—can we match the performance of fully learned transformers, while being more efﬁcient?

Original: Surprisingly, the answer is resoundingly yes; and what is more, we ﬁnd that freezing layers may actually improve performance.
Revised: Surprisingly, the answer is resoundingly yes; and what is more, we ﬁnd that freezing layers may actually improve performance.

Original: Beyond desirable efﬁciency gains, random layers are interesting for several additional reasons.
Revised: Beyond desirable efﬁciency gains, random layers are interesting for several additional reasons.

Original: Fixed randomly initialized networks (Gallicchio & Scardapane, 2020) converge to Gaussian processes in the limit of inﬁnite width (Daniely et al., 2016), have intriguing interpretations in metric learning (Rosenfeld & Tsotsos, 2019; Giryes et al., 2016), and have been shown to provide excellent “priors” either for subsequent learning (Ulyanov et al., 2018) or pruning (Frankle & Carbin, 2018).
Revised: Fixed randomly initialized networks (Gallicchio & Scardapane, 2020) converge to Gaussian processes in the limit of inﬁnite width (Daniely et al., 2016), have intriguing interpretations in metric learning (Rosenfeld & Tsotsos, 2019; Giryes et al., 2016), and have been shown to provide excellent “priors” either for subsequent learning (Ulyanov et al., 2018) or pruning (Frankle & Carbin, 2018).

Original: Fixed layers allow for efﬁcient low-cost hardware implementations (Schrauwen et al., 2007) and can be characterized using only a random number generator and its seed, which might have repercussions in distributed training and enables highly efﬁcient deployment to edge devices.
Revised: Fixed layers allow for efﬁcient low-cost hardware implementations (Schrauwen et al., 2007) and can be characterized using only a random number generator and its seed, which might have repercussions in distributed training and enables highly efﬁcient deployment to edge devices.

Original: The strong performance of networks with ﬁxed layers also sheds new light on the inner workings of
Revised: The strong performance of networks with ﬁxed layers also sheds new light on the inner workings of

Original: BERT (Devlin et al., 2018), and layer-wise interpretations of such models (Rogers et al., 2020; Tenney et al., 2019).
Revised: BERT (Devlin et al., 2018), and layer-wise interpretations of such models (Rogers et al., 2020; Tenney et al., 2019).

Original: It appears that “not all layers are created equal” (Zhang et al., 2019) is true to such an extent that some layers can simply remain random and ﬁxed.
Revised: It appears that “not all layers are created equal” (Zhang et al., 2019) is true to such an extent that some layers can simply remain random and ﬁxed.

Original: These ideas have a long history in machine learning.
Revised: These ideas have a long history in machine learning.

Original: By Cover’s theorem (Cover, 1965), any highdimensional non-linear transformation is more likely to be linearly separable than its lower-or-equaldimensional input space.
Revised: By Cover’s theorem (Cover, 1965), any highdimensional non-linear transformation is more likely to be linearly separable than its lower-or-equaldimensional input space.

Original: By Johnson-Lindenstrauss (Johnson & Lindenstrauss, 1984), random projections distort Euclidean distances very little under mild assumptions, which is useful e.g. for dimensionality reduction and random indexing (Sahlgren, 2005).
Revised: By Johnson-Lindenstrauss (Johnson & Lindenstrauss, 1984), random projections distort Euclidean distances very little under mild assumptions, which is useful e.g. for dimensionality reduction and random indexing (Sahlgren, 2005).

Original: Fixed random layers in neural networks pre-date deep learning by far (Gamba et al., 1961; Baum, 1988).
Revised: Fixed random layers in neural networks pre-date deep learning by far (Gamba et al., 1961; Baum, 1988).

Original: Indeed, random kernel methods have been an impactful idea in machine learning (Rahimi & Recht, 2008; 2009).
Revised: Indeed, random kernel methods have been an impactful idea in machine learning (Rahimi & Recht, 2008; 2009).

Original: One way to think of such layers is as “reservoirs” (Lukoˇseviˇcius & Jaeger, 2009), where a highly non-linear high-dimensional black box representation is provided to a lightweight “readout” network, as in echo state networks (Jaeger, 2003) and liquid state machines (Maass et al., 2002).
Revised: One way to think of such layers is as “reservoirs” (Lukoˇseviˇcius & Jaeger, 2009), where a highly non-linear high-dimensional black box representation is provided to a lightweight “readout” network, as in echo state networks (Jaeger, 2003) and liquid state machines (Maass et al., 2002).

Original: The beneﬁt of such an approach is that the reservoir has ﬁxed parameters and is computationally efﬁcient, as it can be pre-computed and does not (necessarily) require backpropagation.
Revised: The beneﬁt of such an approach is that the reservoir has ﬁxed parameters and is computationally efﬁcient, as it can be pre-computed and does not (necessarily) require backpropagation.

Original: In NLP, Wieting & Kiela (2019) showed that random sentence encoders present a strong baseline for text classiﬁcation, with subsequent work showing applications in a variety of NLP tasks (Enguehard et al., 2019;
Revised: In NLP, Wieting & Kiela (2019) showed that random sentence encoders present a strong baseline for text classiﬁcation, with subsequent work showing applications in a variety of NLP tasks (Enguehard et al., 2019;

Original: Garg et al., 2020; Pilault et al., 2020).
Revised: Garg et al., 2020; Pilault et al., 2020).

Original: To our knowledge, this work is the ﬁrst to examine this phenomenon in transformers, and the ﬁrst to recursively alternate reservoirs with subsequent transformer layers acting as readout functions.
Revised: To our knowledge, this work is the ﬁrst to examine this phenomenon in transformers, and the ﬁrst to recursively alternate reservoirs with subsequent transformer layers acting as readout functions.

Original: We introduce “reservoir transformers”, wherein ﬁxed random reservoir layers are interspersed with regular updateable transformer layers.
Revised: We introduce “reservoir transformers”, wherein ﬁxed random reservoir layers are interspersed with regular updateable transformer layers.

Original: The goal of this work is not necessarily to set a new state of the art, but to put our understanding of transformer models on a more solid footing by providing empirical evidence of their capabilities even when some of their parameters are ﬁxed.
Revised: The goal of this work is not necessarily to set a new state of the art, but to put our understanding of transformer models on a more solid footing by providing empirical evidence of their capabilities even when some of their parameters are ﬁxed.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We introduce a new area under the convergence curve metric for measuring performanceefﬁciency trade-offs, and show that replacing regular transformer layers with reservoir layers leads to better results on that metric.
Revised: • We introduce a new area under the convergence curve metric for measuring performanceefﬁciency trade-offs, and show that replacing regular transformer layers with reservoir layers leads to better results on that metric.

Original: • We show that the addition of reservoir layers in fact leads to improved test set generalization on a variety of tasks in a variety of settings.
Revised: • We show that the addition of reservoir layers in fact leads to improved test set generalization on a variety of tasks in a variety of settings.

Original: • We show that pre-trained masked language modelling architectures like BERT and RoBERTa (Liu et al., 2019) can beneﬁt from having some of their layers frozen, both during pre-training as well as when ﬁne-tuning on downstream tasks.
Revised: • We show that pre-trained masked language modelling architectures like BERT and RoBERTa (Liu et al., 2019) can beneﬁt from having some of their layers frozen, both during pre-training as well as when ﬁne-tuning on downstream tasks.

Original: • In addition, we experiment with different types of reservoir layers, including convolutional and recurrent neural network-based ones.
Revised: • In addition, we experiment with different types of reservoir layers, including convolutional and recurrent neural network-based ones.

Original: We also show empirical evidence that the backward pass can be entirely skipped by approximating top-layer gradients using an approach we call backskipping , with a relatively small sacriﬁce in performance.
Revised: We also show empirical evidence that the backward pass can be entirely skipped by approximating top-layer gradients using an approach we call backskipping , with a relatively small sacriﬁce in performance.

Original:  A PPROACH
Revised:  A PPROACH

Original: This paper is based on a very simple idea.
Revised: This paper is based on a very simple idea.

Original: Neural networks are trained via backpropagation, which involves consecutive steps of matrix addition and multiplication, i.e.,
Revised: Neural networks are trained via backpropagation, which involves consecutive steps of matrix addition and multiplication, i.e.,

Original: [Equation]
Revised: [Equation]

Original: for some objective J , parameterization θ and learning rate η , with the gradient computed via the chain rule, where L i is the i -th layer of the neural network and x is the input.
Revised: for some objective J , parameterization θ and learning rate η , with the gradient computed via the chain rule, where L i is the i -th layer of the neural network and x is the input.

Original: Let L =
Revised: Let L =

Original: Transformer ( X ) be a single layer in a Transformer network (Vaswani et al., 2017), i.e.,
Revised: Transformer ( X ) be a single layer in a Transformer network (Vaswani et al., 2017), i.e.,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: MultiHeadSelfAttn
Revised: MultiHeadSelfAttn

Original: [Equation]
Revised: [Equation]

Original: Now, during every “backward pass”, we compute the Jacobian for parameters θ L at layer L , which are used to update the parameters of L , θ Lt , as well as to compute the next layer’s
Revised: Now, during every “backward pass”, we compute the Jacobian for parameters θ L at layer L , which are used to update the parameters of L , θ Lt , as well as to compute the next layer’s

Original: Jacobian, thus back-propagating the gradients.
Revised: Jacobian, thus back-propagating the gradients.

Original: In this work however, for some of the layers, we still backpropagate through them to compute gradients for earlier layers, but we never update their parameters .
Revised: In this work however, for some of the layers, we still backpropagate through them to compute gradients for earlier layers, but we never update their parameters .

Original: As a result, these layers stay ﬁxed at their random initialization, saving computational resources.
Revised: As a result, these layers stay ﬁxed at their random initialization, saving computational resources.

Original: 2. B ACKGROUND
Revised: 2. B ACKGROUND

Original: Naturally, never updating some of the parameters is computationally more efﬁcient, as some matrix addition operations can be skipped in the backward pass, but why is this not detrimental to the performance of the network?
Revised: Naturally, never updating some of the parameters is computationally more efﬁcient, as some matrix addition operations can be skipped in the backward pass, but why is this not detrimental to the performance of the network?

Original: In the early days of neural networks, the bottom layers were often kept ﬁxed as “associators” (Block, 1962), or what Minsky & Papert (2017) called the Gamba perceptron (Gamba et al., 1961; Borsellino & Gamba, 1961).
Revised: In the early days of neural networks, the bottom layers were often kept ﬁxed as “associators” (Block, 1962), or what Minsky & Papert (2017) called the Gamba perceptron (Gamba et al., 1961; Borsellino & Gamba, 1961).

Original: Fixed random networks (Baum, 1988; Schmidt et al., 1992; Pao et al., 1994) have been explored from many angles, including as “random kitchen sink” kernel machines (Rahimi & Recht, 2008; 2009), “extreme learning machines” (Huang et al., 2006) and reservoir computing (Jaeger, 2003; Maass et al., 2002; Lukoˇseviˇcius & Jaeger, 2009).
Revised: Fixed random networks (Baum, 1988; Schmidt et al., 1992; Pao et al., 1994) have been explored from many angles, including as “random kitchen sink” kernel machines (Rahimi & Recht, 2008; 2009), “extreme learning machines” (Huang et al., 2006) and reservoir computing (Jaeger, 2003; Maass et al., 2002; Lukoˇseviˇcius & Jaeger, 2009).

Original: In reservoir computing, input data are represented through ﬁxed random high-dimensional non-linear representations, called “reservoirs”, which are followed by a regular (often but not necessarily linear) “readout” network to make the ﬁnal classiﬁcation decision.
Revised: In reservoir computing, input data are represented through ﬁxed random high-dimensional non-linear representations, called “reservoirs”, which are followed by a regular (often but not necessarily linear) “readout” network to make the ﬁnal classiﬁcation decision.

Original: The theoretical justiﬁcation for these approaches lies in two well-known results in machine learning: Cover’s theorem (Cover, 1965) on the separability of patterns states that high-dimensional non-linear transformations are more likely to be linearly separable; and the Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) shows that random projections distort Euclidean distances very little under mild assumptions.
Revised: The theoretical justiﬁcation for these approaches lies in two well-known results in machine learning: Cover’s theorem (Cover, 1965) on the separability of patterns states that high-dimensional non-linear transformations are more likely to be linearly separable; and the Johnson-Lindenstrauss lemma (Johnson & Lindenstrauss, 1984) shows that random projections distort Euclidean distances very little under mild assumptions.

Original: Practically, random layers can be seen as a cheap way to increase network depth.
Revised: Practically, random layers can be seen as a cheap way to increase network depth.

Original: There are interesting advantages to this approach.
Revised: There are interesting advantages to this approach.

Original: Fixed layers are known to have particularly low-cost hardware requirements and can be easily implemented on high-bandwidth FPGAs with low power consumption (Hadaeghi et al., 2017; Tanaka et al., 2019), or on optical devices (Hicke et al., 2013).
Revised: Fixed layers are known to have particularly low-cost hardware requirements and can be easily implemented on high-bandwidth FPGAs with low power consumption (Hadaeghi et al., 2017; Tanaka et al., 2019), or on optical devices (Hicke et al., 2013).

Original: This might yield interesting possibilities for training in a distributed fashion across multiple devices, as well as for neurmorphic hardware (Neftci et al., 2017).
Revised: This might yield interesting possibilities for training in a distributed fashion across multiple devices, as well as for neurmorphic hardware (Neftci et al., 2017).

Original: This approach also facilitates lower-latency deployment of neural networks to edge devices, since weights can be shared simply by sending the seed number, assuming the random number generator is known on both ends.
Revised: This approach also facilitates lower-latency deployment of neural networks to edge devices, since weights can be shared simply by sending the seed number, assuming the random number generator is known on both ends.

Original: 2. R ESERVOIR T RANSFORMERS
Revised: 2. R ESERVOIR T RANSFORMERS

Original: This work explores inserting random non-linear transformations, or what we call reservoir layers, into transformer networks.
Revised: This work explores inserting random non-linear transformations, or what we call reservoir layers, into transformer networks.

Original: Speciﬁcally, we experiment with a variety of reservoir layers:
Revised: Speciﬁcally, we experiment with a variety of reservoir layers:

Original: • Transformer Reservoir: The standard transformer layer as described above, but with all parameters ﬁxed after initialization, including the self-attention module.
Revised: • Transformer Reservoir: The standard transformer layer as described above, but with all parameters ﬁxed after initialization, including the self-attention module.

Original: • FFN Reservoir: A transformer-style ﬁxed feed-forward layer without any self-attention, i.e., FFN(LayerNorm ( Previous layer ))
Revised: • FFN Reservoir: A transformer-style ﬁxed feed-forward layer without any self-attention, i.e., FFN(LayerNorm ( Previous layer ))

Original: + Previous layer.
Revised: + Previous layer.

Original: • BiGRU Reservoir: A ﬁxed bidirectional Gated Recurrent Unit (Cho et al., 2014) layer, which is closer in spirit to previous work on reservoir computing, most of which builds on recurrent neural network architectures.
Revised: • BiGRU Reservoir: A ﬁxed bidirectional Gated Recurrent Unit (Cho et al., 2014) layer, which is closer in spirit to previous work on reservoir computing, most of which builds on recurrent neural network architectures.

Original: • CNN Reservoir:
Revised: • CNN Reservoir:

Original: A ﬁxed Convolutional Neural Network (LeCun et al., 1998) layer, speciﬁcally light dynamical convolution layers (Wu et al., 2019), which are known to be competitive with transformers in sequence-to-sequence tasks.
Revised: A ﬁxed Convolutional Neural Network (LeCun et al., 1998) layer, speciﬁcally light dynamical convolution layers (Wu et al., 2019), which are known to be competitive with transformers in sequence-to-sequence tasks.

Original: We ﬁnd that all these approaches work well, to a certain extent.
Revised: We ﬁnd that all these approaches work well, to a certain extent.

Original: For clarity, we focus primarily on the ﬁrst two reservoir layers, but include a broader comparison in Appendix A.
Revised: For clarity, we focus primarily on the ﬁrst two reservoir layers, but include a broader comparison in Appendix A.

Original: In each case, contrary to traditional reservoir computing, our reservoir layers are interspersed throughout a regular transformer network, or what we call a reservoir transformer.
Revised: In each case, contrary to traditional reservoir computing, our reservoir layers are interspersed throughout a regular transformer network, or what we call a reservoir transformer.

Original: A good justiﬁcation for this approach is that while random projections are not learned and might introduce noise, subsequent normal transformer “readout” layers might allow us to recover from any adverse effects of randomness.
Revised: A good justiﬁcation for this approach is that while random projections are not learned and might introduce noise, subsequent normal transformer “readout” layers might allow us to recover from any adverse effects of randomness.

Original: For example, previous work has shown that ResNets, with all of their parameters ﬁxed except for the scale and shift parameters of batch normalization, can still achieve high performance, simply by scaling and shifting random features (Frankle et al., 2020).
Revised: For example, previous work has shown that ResNets, with all of their parameters ﬁxed except for the scale and shift parameters of batch normalization, can still achieve high performance, simply by scaling and shifting random features (Frankle et al., 2020).

Original: Adding noise to the parameters of neural networks is also known to help convergence and generalization (Jim et al., 1995; 1996;
Revised: Adding noise to the parameters of neural networks is also known to help convergence and generalization (Jim et al., 1995; 1996;

Original: Gulcehre et al., 2016; Noh et al., 2017).
Revised: Gulcehre et al., 2016; Noh et al., 2017).

Original:  E VALUATION
Revised:  E VALUATION

Original: We evaluate the proposed approach on a variety of well-known tasks in natural language processing, namely: machine translation, language modelling and masked language model pre-training.
Revised: We evaluate the proposed approach on a variety of well-known tasks in natural language processing, namely: machine translation, language modelling and masked language model pre-training.

Original: In this work, we are not necessarily interested in obtaining the state of the art on any task or even in improving overall task performance via this method.
Revised: In this work, we are not necessarily interested in obtaining the state of the art on any task or even in improving overall task performance via this method.

Original: The main objective is to examine efﬁciency, i.e. the relationship between compute time and task performance.
Revised: The main objective is to examine efﬁciency, i.e. the relationship between compute time and task performance.

Original: This is closely related
Revised: This is closely related

Original: [Figure]
Revised: [Figure]

Original: a li

Original: [Figure]

Original: L

Original: [Figure]

Original: U A U

Original: CC

Original: [Figure]

Original: Model # Layers

Original: Frozen

Original: Max BLEU

Original: Train time until max (in hours)

Original: Ratio

Original: Transformer  34.

Original: T Reservoir  34.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: FFN Reservoir

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] to efforts in Green AI, which are concerned with the trade-offs between compute, data, and performance (Schwartz et al., 2019).
Revised: to efforts in Green AI, which are concerned with the trade-offs between compute, data, and performance (Schwartz et al., 2019).

Original: We propose a new metric for our purposes, the area under the convergence curve (AUCC): similarly to how the area under the receiver operating characteristic (Bradley, 1997, AUC-ROC) measures a classiﬁer’s performance independent of the classiﬁcation threshold, AUCC measures a model’s performance independent of the speciﬁc compute budget.
Revised: We propose a new metric for our purposes, the area under the convergence curve (AUCC): similarly to how the area under the receiver operating characteristic (Bradley, 1997, AUC-ROC) measures a classiﬁer’s performance independent of the classiﬁcation threshold, AUCC measures a model’s performance independent of the speciﬁc compute budget.

Original: Speciﬁcally, AUCC is computed as follows:
Revised: Speciﬁcally, AUCC is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: where f is the network and g is the evaluation metric, measured until convergence time ˆ T , which is the maximum convergence time of all models included in the comparison.
Revised: where f is the network and g is the evaluation metric, measured until convergence time ˆ T , which is the maximum convergence time of all models included in the comparison.

Original: Note that time here is wall-clock time, not iterations.
Revised: Note that time here is wall-clock time, not iterations.

Revised: By convergence, we mean that validation performance has stopped improving, and hence the convergence curve whose area we measure plots the desired metric over time.

Original: Runs are averaged over multiple seeds and reported with standard deviation.
Revised: Runs are averaged over multiple seeds and reported with standard deviation.

Original: We normalize raw AUCC scores by their maximum score to ensure a more easily interpretable [0 − 1] range.
Revised: We normalize raw AUCC scores by their maximum score to ensure a more easily interpretable [0 − 1] range.

Original: On potential downside of this approach is that the AUCC metric could lead to higher scores for a model that converges quickly but to ultimately worse performance, if measured in a small window.
Revised: One potential downside of this approach is that the AUCC metric could lead to higher scores for a model that converges quickly but to ultimately worse performance, if measured in a small window.

Original: We account for this by making sure that ˆ T is set sufﬁciently high.
Revised: We account for this by making sure that ˆ T is set sufﬁciently high.

Original: We include the raw validation curves in the appendix and also report test set generalization in each experiment.
Revised: We include the raw validation curves in the appendix and also report test set generalization in each experiment.

Original: 3. E XPERIMENTAL S ETTINGS AND I MPLEMENTATION D ETAILS
Revised: 3. E XPERIMENTAL S ETTINGS AND I MPLEMENTATION D ETAILS

Original: We use 8 Volta V100 GPUs for all the experiments except for  IWSLT, which used a single V100.
Revised: We evaluate on IWSLT de-en (Cettolo et al., 2015) and WMT en-de (Bojar et al., 2014) for machine translation; enwiki8 (LLC, 2009) for language modelling; and experiment with RoBERTa (Liu et al., 2019) in our pretraining experiments. For IWSLT, we follow the pre-processing steps in Edunov et al. The train/val/test split is 129k/10k/6.8k sentences.

Revised: For WMT, we follow the pre-processing steps in Ott et al.

Revised: The train/val/test split is 4.5M/16.5k/3k sentences.

Revised: For enwiki8, we follow the pre-processing steps in Dai et al.

Revised: The train/val/test split is 1M/54k/56k sentences.

Revised: For RoBERTa pretraining, we follow the pre-processing steps in Liu et al.

Revised: We use 8 Volta V100 GPUs for WMT and enwik8, 32 V100 GPUs for RoBERTa and a single V for IWSLT.

Original: The hyperparameters for IWSLT14 and WMT16 were set to the best-performing values from Ott et al.
Revised: The hyperparameters for IWSLT14 and WMT16 were set to the best-performing values from Ott et al.

Original: (2018) and Kasai et al.
Revised: (2018) and Kasai et al.

Original: (2020) respectively.
Revised: (2020) respectively.

Original: The enwik8 experiment settings followed Bachlechner et al.
Revised: The enwik8 experiment settings followed Bachlechner et al.

Original: and the RoBERTa experiments followed Liu et al.
Revised: and the RoBERTa experiments followed Liu et al.

Original: All experiments were conducted using fairseq (Ott et al., 2019).
Revised: All experiments were conducted using fairseq (Ott et al., 2019).

Original: Our code and experimental settings will be made open source at [ANONYMIZED-GITHUB-URL]. 
Revised: Our code and experimental settings will be made open source at [ANONYMIZED-GITHUB-URL]. [Table]

Original: All the experiments in this paper were run with 3 random seeds and the mean and standard deviation are reported.
Revised: All the experiments in this paper were run with 3 random seeds and the mean and standard deviation are reported.

Original: For the relatively small IWSLT, the ˆ T value in the AUCC metric was set to 4 hours.
Revised: For the relatively small IWSLT, the ˆ T value in the AUCC metric was set to 4 hours.

Original: For WMT, which is larger, we set it to 20 hours.
Revised: For WMT, which is larger, we set it to 20 hours.

Original: For enwiki8, it was 30 hours; and for the RoBERTa pre-training experiments, it was set to 60 hours.
Revised: For enwiki8, it was 30 hours; and for the RoBERTa pre-training experiments, it was set to 60 hours.

Original: The projection weights in random layers were initialized using orthogonal initialization (Saxe et al., 2013), which makes sense since random orthogonal projections should be most informationpreserving, and which was found to work well empirically for initializing ﬁxed random representations in previous work (Wieting & Kiela, 2019).
Revised: The projection weights in random layers were initialized using orthogonal initialization (Saxe et al., 2013), which makes sense since random orthogonal projections should be most informationpreserving, and which was found to work well empirically for initializing ﬁxed random representations in previous work (Wieting & Kiela, 2019).

Original: Biases and layer norm parameters were initialized using their respective PyTorch defaults (based on Xavier init; Glorot & Bengio, 2010).
Revised: Biases and layer norm parameters were initialized using their respective PyTorch defaults (based on Xavier init; Glorot & Bengio, 2010).

Original: We intersperse reservoir layers in alternating fashion starting from the middle.
Revised: We intersperse reservoir layers in alternating fashion starting from the middle.

Original: Speciﬁcally, we alternate one reservoir layer with one transformer layer, and place the alternating block in the middle.
Revised: Speciﬁcally, we alternate one reservoir layer with one transformer layer, and place the alternating block in the middle.

Original: For example: a 7 -layer encoder LLLLLLL in which we replace three layers with reservoirs becomes
Revised: For example: a 7 -layer encoder LLLLLLL in which we replace three layers with reservoirs becomes

Original: LRLRLRL, and with two becomes LLRLRLL.
Revised: LRLRLRL, and with two becomes LLRLRLL.

Original: See Appendix C for a study comparing this strategy to alternative approaches (e.g., freezing in the bottom, middle or top).
Revised: See Appendix C for a study comparing this strategy to alternative approaches (e.g., freezing in the bottom, middle or top).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In what follows, we ﬁrst show our main result: reservoir transformers often have better AUCC metrics,  and even improved test set generalization metrics, on a variety of tasks.
Revised: In what follows, we ﬁrst show our main result: reservoir transformers often have better AUCC metrics, less training time per epoch, less convergence time until the best validation performance is achieved, and even improved test set generalization metrics, on a variety of tasks.

Revised: As a strong baseline method, we compare to LayerDrop (Fan et al., 2019).

Revised: LayerDrop can also be seen as a method that dynamically bypasses parts of the computation during Transformer training in an attempt to improve efﬁciency, and is a suitable comparison to examine our methods..

Original: We also examine whether we can minimize the expectation over the gradients of upper layers in the transformer network such that we do not have to pass the true gradients through the reservoir for further efﬁciency.
Revised: We also examine whether we can minimize the expectation over the gradients of upper layers in the transformer network such that we do not have to pass the true gradients through the reservoir for further efﬁciency.

Original: 4. M ACHINE T RANSLATION
Revised: 4. M ACHINE T RANSLATION

Original: Machine translation (MT) is one of the core tasks of NLP.
Revised: Machine translation (MT) is one of the core tasks of NLP.

Original: We demonstrate on two well-known MT datasets, IWSLT’14 German-English and WMT’16 English-German, that reservoir transformers obtain a better AUCC.
Revised: We demonstrate on two well-known MT datasets, IWSLT’14 German-English and WMT’16 English-German, that reservoir transformers obtain a better AUCC.

Original: For the raw validation plots over time that were used to calculate the AUCC, please refer to Appendix F.
Revised: For the raw validation plots over time that were used to calculate the AUCC, please refer to Appendix F.

Original: Following Kasai et al.
Revised: Following Kasai et al.

Original: (2020), the architecture of the network is an N-layer reservoir transformer encoder, followed by a regular shallow one- or two-layer decoder.
Revised: (2020), the architecture of the network is an N-layer reservoir transformer encoder, followed by a regular shallow one- or two-layer decoder.

Original: This design choice has been shown to lead to very good speed and efﬁciency trade-offs, and serves as a good baseline for our experiments.
Revised: This design choice has been shown to lead to very good speed and efﬁciency trade-offs, and serves as a good baseline for our [Figure] [Figure] [Figure]

Original:  Moreover, shallow decoders make it easier to decide where to place reservoir layers (in the encoder) and makes it more straightforward to identify where performance gains come from.
Revised: of regular and transformers for varying depths. experiments. Moreover, shallow decoders make it easier to decide where to place reservoir layers (in the encoder) and makes it more straightforward to identify where performance gains come from.

Original: Figure 1 shows the results for IWSLT.
Revised: Figure 1 shows the results for IWSLT.

Original: On the y-axis we show validation AUCC for the BLEU metric; on the x-axis we show the number of updatable layers in the encoder.
Revised: On the y-axis we show validation AUCC for the BLEU metric; on the x-axis we show the number of updatable layers in the encoder.

Original: The performance of a regular transformer encoder with 6 layers and a reservoir transformer encoder with 6 layers plus N additional reservoir layers are plotted for the same x-axis value to show the total number of updated layers.
Revised: The performance of a regular transformer encoder with 6 layers and a reservoir transformer encoder with 6 layers plus N additional reservoir layers are plotted for the same x-axis value to show the total number of updated layers.

Original: Plots for the total number of layers (updatable plus not-updatable, so essentially shifted versions) are shown in Appendix E. Table 1 shows the time it took to achieve the maximum validation BLEU score and how that relates to the regular transformer, demonstrating that reservoir transformers consistently converge faster in terms of wall-clock time, up to 22% as much with the same number of updateable layers. 
Revised: Plots for the total number of layers (updatable plus not-updatable, so essentially shifted versions) are shown in Appendix E. Table 1 shows the time it took to achieve the maximum validation BLEU score and how that relates to the regular transformer, demonstrating that reservoir transformers consistently converge faster in terms of wall-clock time, up to 22% as much with the same number of updateable layers. We save as much as 27% time until convergence a 24 layer model on WMT, as shown in Table 3.

Revised: One other noticeable point is that we can see that the T Reservoir achieves similar performance to LayerDrop on IWSLT and WMT in terms of wall-clock per epoch and wall-clock time to the best performance.

Revised: However, on both tasks, FFN Reservoir performs much better than LayerDrop in terms of efﬁciency per epoch and achieves better/similar performance in less time in each case.

Original: As a point of reference, a half hour gain on IWSLT translates to a gain of several days in the training of bigger transformer models like GPT-3 (Brown et al., 2020).
Revised: As a point of reference, a half hour gain on IWSLT translates to a gain of several days in the training of bigger transformer models like GPT-3 (Brown et al., 2020).

Original: We observe that reservoir transformers consistently perform better than, or are competitive to, regular transformers, both in terms of validation BLEU AUCC as well as test time BLEU, for all examined encoder depths. 
Revised: We observe that reservoir transformers consistently perform better than, or are competitive to, regular transformers, both in terms of validation BLEU AUCC as well as test time BLEU, for all examined encoder depths. [Figure] v a li d a cc u r

Original: [Figure]
Revised: [Figure]

Revised: c y

Revised: [Figure]

Revised: [Figure]

Revised: v a li

Original: [Figure]
Revised: [Figure]

Original: Comparison of regular and reservoir transformers for varying depths.
Revised: r [Figure] c

Original:  Figure 2 shows a similar trend for WMT.
Revised: [Figure] Figure 2 shows a similar trend for WMT.

Original: WMT is much larger and requires a much deeper encoder, as illustrated by the fact that a certain minimum depth is required for reservoir transformers to achieve a comparable validation AUCC.
Revised: WMT is much larger and requires a much deeper encoder, as illustrated by the fact that a certain minimum depth is required for reservoir transformers to achieve a comparable validation AUCC.

Original: At test time, reservoir transformers outperform regular transformers for almost all encoder depths.
Revised: At test time, reservoir transformers outperform regular transformers for almost all encoder depths.

Original: The FFN reservoir transformer seems to work best in both cases, which is surprising because it does not have any self-attention component at all.
Revised: The FFN reservoir transformer seems to work best in both cases, which is surprising because it does not have any self-attention component at all.

Original: This ﬁnding shows that self-attention, or the mechanism to summarize context information, should be learned if present.
Revised: This ﬁnding shows that self-attention, or the mechanism to summarize context information, should be learned if present.

Original: Once the context features have been gathered, a random projection via a ﬁxed
Revised: Once the context features have been gathered, a random projection via a ﬁxed

Original: FFN module appears to be beneﬁcial, at least for MT.
Revised: FFN module appears to be beneﬁcial, at least for MT.

Original: 4. L ANGUAGE M ODELLING
Revised: 4. L ANGUAGE M ODELLING

Original: To examine whether the same ﬁndings hold for other tasks, we evaluate on the enwiki8 (Mahoney, 2011) language modelling task.
Revised: To examine whether the same ﬁndings hold for other tasks, we evaluate on the enwiki8 (LLC, 2009) language modelling task.

Original: We examine the BPC (bits per character) rate for a variety of network depths (since the task is language modelling, these layers are in the decoder).
Revised: We examine the BPC (bits per character) rate for a variety of network depths (since the task is language modelling, these layers are in the decoder).

Original: The results show that we obtain consistently better BPC for lower depths, except for the 64 -layer regular transformer, which appears to be particularly optimal for this task.
Revised: The results show that we obtain consistently better BPC for lower depths, except for the 64 -layer regular transformer, which appears to be particularly optimal for this task.

Original: We observe similar trends during test time.
Revised: We observe similar trends during test time.

Original: 4. M ASKED L ANGUAGE M ODEL P RETRAINING
Revised: 4. M ASKED L ANGUAGE M ODEL P RETRAINING

Original: We train RoBERTa
Revised: We train RoBERTa

Original: (Liu et al., 2019) models from scratch at a variety of depths, both in the normal and reservoir setting.
Revised: (Liu et al., 2019) models from scratch at a variety of depths, both in the normal and reservoir setting.

Original: We ﬁnd that these networks show minor differences in their best perplexity and similar AUCC perplexity (see Appendix D).
Revised: We ﬁnd that these networks show minor differences in their best perplexity and similar AUCC perplexity (see Appendix D).

Original: We then examine the performance of these models when ﬁne-tuned on downstream tasks, speciﬁcally the well known SST-2 (Socher et al., 2013) [Figure] v a li d a cc u r a c y [Figure]
Revised: We then examine the performance of these models when ﬁne-tuned on downstream tasks, speciﬁcally the well known SST-2 (Socher et al., 2013) and MultiNLI 1 (Williams et al., 2017) tasks.

Original: [Figure]

Original: v a li d a cc u r a c y

Original: [Figure]

Original: and MultiNLI 1 (Williams et al., 2017) tasks.

Original: When ﬁne-tuning the reservoir models, we keep the reservoir layers ﬁxed (including them in ﬁne-tuning did not work very well, see Appendix D).
Revised: When ﬁne-tuning the reservoir models, we keep the reservoir layers ﬁxed (including them in ﬁne-tuning did not work very well, see Appendix D).

Original: Figure 4 shows the results of ﬁne-tuning.
Revised: Figure 4 shows the results of ﬁne-tuning.

Original: We observe that the reservoir transformer outperforms normal RoBERTa at all depths in both tasks.
Revised: We observe that the reservoir transformer outperforms normal RoBERTa at all depths in both tasks.

Original: At lower depth, the improvements are substantial.
Revised: At lower depth, the improvements are substantial.

Original: As a sanity check, we also experiment with freezing some of the layers in normal RoBERTa during ﬁne-tuning (Transformer frozen ﬁnetuned) and show that this helps a little but is still outperformed by the reservoir transformer.
Revised: As a sanity check, we also experiment with freezing some of the layers in normal RoBERTa during ﬁne-tuning (Transformer frozen ﬁnetuned) and show that this helps a little but is still outperformed by the reservoir transformer.

Original: These ﬁndings suggest that you can train a RoBERTa model without updating all of the layers, achieve similar perplexity at a similar computational cost, but with better downstream performance.
Revised: These ﬁndings suggest that you can train a RoBERTa model without updating all of the layers, achieve similar perplexity at a similar computational cost, but with better downstream performance.

Original: The fact that some layers can be kept random and entirely ﬁxed during training, without sacriﬁcing any performance, raises intriguing questions for “BERTology” (Rogers et al., 2020) and for the study of what different layers in transformers learn. 
Revised: The fact that some layers can be kept random and entirely ﬁxed during training, without sacriﬁcing any performance, raises intriguing questions for “BERTology” (Rogers et al., 2020) and for the study of what different layers in transformers learn. [Figure] [Figure]

Original: 4. B ACKSKIPPING
Revised: B ACKSKIPPING

Original: With the reservoir transformers as described above, we obtain better efﬁciency by skipping the “gradient application” matrix addition step in some of the layers (i.e., updating the weights).
Revised: With the reservoir transformers as described above, we obtain better efﬁciency by skipping the “gradient application” matrix addition step in some of the layers (i.e., updating the weights).

Original: One step further would be to investigate skipping the entire backward pass for reservoirs altogether, which would save us from having to do the much more expensive matrix multiplication for these layers that is required for the propagation of gradients.
Revised: One step further would be to investigate skipping the entire backward pass for reservoirs altogether, which would save us from having to do the much more expensive matrix multiplication for these layers that is required for the propagation of gradients.

Original: We report on preliminary experiments where in the backward pass we replace the gradients for the layer L i going into the reservoir L i +1 with a noisy estimate (Jaderberg et al., 2017; Czarnecki et al., 2017).
Revised: We report on preliminary experiments where in the backward pass we replace the gradients for the layer L i going into the reservoir L i +1 with a noisy estimate (Jaderberg et al., 2017; Czarnecki et al., 2017).

Original: Promisingly, Oktay et al.
Revised: Promisingly, Oktay et al.

Original: (2020) recently asked “why spend resources on exact gradients when we’re going to use stochastic optimization?” and show that you can do randomized auto-differentiation quite successfully.
Revised: (2020) recently asked “why spend resources on exact gradients when we’re going to use stochastic optimization?” and show that you can do randomized auto-differentiation quite successfully.

Original: Here, rather than minimizing the actual gradients ∂L i ∂θ Li , we minimize their expectation and train via continuous-action REINFORCE (Williams, 1992).
Revised: Here, rather than minimizing the actual gradients ∂L i ∂θ Li , we minimize their expectation and train via continuous-action REINFORCE (Williams, 1992).

Original: That is, L i becomes a policy π a : s → µ where we sample actions a ∼ N ( µ, 1) .
Revised: That is, L i becomes a policy π a : s → µ where we sample actions a ∼ N ( µ, 1) .

Original: We train to minimize the gradient prediction loss via MSE, i.e., 1 n (cid:80) ni =

Revised: We train to minimize the gradient prediction loss via MSE, i.e., 1 n (cid:80) ni =

Original: ( R i − V i ( a )) 2 , and the REINFORCE loss E a [log( a ) ( R − V ( a ))] , where the value network V acts as the baseline.
Revised: ( R i − V i ( a )) 2 , and the REINFORCE loss E a [log( a ) ( R − V ( a ))] , where the value network V acts as the baseline.

Original: R is deﬁned as the mean of the gradients of the top layer L i +2 , with the sign ﬂipped.
Revised: R is deﬁned as the mean of the gradients of the top layer L i +2 , with the sign ﬂipped.

Original: Thus, simply put, we train to minimize the expectation of the true gradients at the layer directly following the reservoir.
Revised: Thus, simply put, we train to minimize the expectation of the true gradients at the layer directly following the reservoir.

Original: We employ an annealing scheme where we ﬁrst train the value network and propagate the true gradients during warmup.
Revised: We employ an annealing scheme where we ﬁrst train the value network and propagate the true gradients during warmup.

Original: Afterwards, we anneal the probability of backskipping rather than performing a true backward pass (multiplying the probability by 0 .every iteration until we only backskip).
Revised: Afterwards, we anneal the probability of backskipping rather than performing a true backward pass (multiplying the probability by 0 .every iteration until we only backskip).

Original: We experimented with setting R to the negation of the total loss as well but found the current reward to work better.
Revised: We experimented with setting R to the negation of the total loss as well but found the current reward to work better.

Original: We call this approach backskipping .
Revised: We call this approach backskipping .

Original: Figure 5 shows the results as validation BLEU over time.
Revised: Figure 5 shows the results as validation BLEU over time.

Original: We observe that this approach helps especially during the earlier stages of training.
Revised: We observe that this approach helps especially during the earlier stages of training.

Original: Although it does not match the performance of[Figure] the approach with true gradients quite yet, it actually performs competitively.
Revised: Although it does not match the performance of the approach with true gradients quite yet, it actually performs competitively.

Original: Backskipping looks promising as an approach to further reduce computational costs, and would be even more efﬁcient from a hardware perspective since the circuitry for such layers (which do not need to propagate gradients) can effectively be hardwired entirely.
Revised: Backskipping looks promising as an approach to further reduce computational costs, and would be even more efﬁcient from a hardware perspective since the circuitry for such layers (which do not need to propagate gradients) can effectively be hardwired entirely.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Recent work has shown that modern NLP models are able to function with different numbers of layers for different examples (Elbayad et al., 2019; Fan et al., 2019); that different layers specialize for different purposes (Zhang et al., 2019); that layers can be compressed (Li et al., 2020); and, that layers can be reordered (Press et al., 2019).
Revised: Recent work has shown that modern NLP models are able to function with different numbers of layers for different examples (Elbayad et al., 2019; Fan et al., 2019); that different layers specialize for different purposes (Zhang et al., 2019); that layers can be compressed (Li et al., 2020); and, that layers can be reordered (Press et al., 2019).

Original: There is a growing body of work in efﬁcient self-attention networks (Tay et al., 2020b), such as linear attention (Wang et al., 2020), on how to process long context information (Beltagy et al., 2020) and on approximations to make transformers more scalable (Kitaev et al., 2020; Katharopoulos et al., 2020).
Revised: There is a growing body of work in efﬁcient self-attention networks (Tay et al., 2020b), such as linear attention (Wang et al., 2020), on how to process long context information (Beltagy et al., 2020) and on approximations to make transformers more scalable (Kitaev et al., 2020; Katharopoulos et al., 2020).

Original: BigBIRD (Zaheer et al., 2020) provides random keys as additional inputs to its attention mechanism.
Revised: BigBIRD (Zaheer et al., 2020) provides random keys as additional inputs to its attention mechanism.

Original: Locality sensitive hashing (LSH) as employed e.g. in Reformer (Kitaev et al., 2020) utilizes a ﬁxed random projection.
Revised: Locality sensitive hashing (LSH) as employed e.g. in Reformer (Kitaev et al., 2020) utilizes a ﬁxed random projection.

Original: Performer (Choromanski et al., 2020) computes the transformer’s multi-head attention weights as a ﬁxed orthogonal random projection.
Revised: Performer (Choromanski et al., 2020) computes the transformer’s multi-head attention weights as a ﬁxed orthogonal random projection.

Original: Closely related to this work, Tay et al.
Revised: Closely related to this work, Tay et al.

Original: (2020a) showed that randomized alignment matrices in their “Synthesizer” architecture are sufﬁcient for many NLP tasks.
Revised: (2020a) showed that randomized alignment matrices in their “Synthesizer” architecture are sufﬁcient for many NLP tasks.

Original: While these works focus on random attention, we show that entire layers can be random and ﬁxed.
Revised: While these works focus on random attention, we show that entire layers can be random and ﬁxed.

Original: We also show that entire layers can be replaced by ﬁxed random projections that do not have any attention whatsoever
Revised: We also show that entire layers can be replaced by ﬁxed random projections that do not have any attention whatsoever.

Original: Beyond transformers, random features have been extensively explored.
Revised: Beyond transformers, random features have been extensively explored.

Original: Examples of this include
Revised: Examples of this include

Original: FreezeOut (Brock et al., 2017), deep reservoir computing networks (Scardapane & Wang, 2017;
Revised: FreezeOut (Brock et al., 2017), deep reservoir computing networks (Scardapane & Wang, 2017;

Original: Gallicchio & Micheli, 2017), as well as applications in domains as varied as text classiﬁcation (Conneau et al., 2017; Zhang & Bowman, 2018; Wieting & Kiela, 2019) or music classiﬁcation (Pons & Serra, 2019).
Revised: Gallicchio & Micheli, 2017), as well as applications in domains as varied as text classiﬁcation (Conneau et al., 2017; Zhang & Bowman, 2018; Wieting & Kiela, 2019) or music classiﬁcation (Pons & Serra, 2019).

Original: It is well known that randomly initialized networks can display impressive performance on their own (Ulyanov et al., 2018; Rosenfeld & Tsotsos, 2019; Ramanujan et al., 2020), which underlies, for example, the recently popularized lottery ticket hypothesis (Frankle & Carbin, 2018;
Revised: It is well known that randomly initialized networks can display impressive performance on their own (Ulyanov et al., 2018; Rosenfeld & Tsotsos, 2019; Ramanujan et al., 2020), which underlies, for example, the recently popularized lottery ticket hypothesis (Frankle & Carbin, 2018;

Original: Zhou et al., 2019).
Revised: Zhou et al., 2019).

Original: We know that learning deep overparameterized networks appears to help in general (Li & Liang, 2018; Du et al., 2019).
Revised: We know that learning deep overparameterized networks appears to help in general (Li & Liang, 2018; Du et al., 2019).

Original: Our method represents an easy and cheap way to add both depth and parameters to transformer networks.
Revised: Our method represents an easy and cheap way to add both depth and parameters to transformer networks.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This work demonstrated that state-of-the-art transformer architectures can be trained without updating all of the layers.
Revised: This work demonstrated that state-of-the-art transformer architectures can be trained without updating all of the layers.

Original: This complements a long history in machine learning of harnessing the power of random features.
Revised: This complements a long history in machine learning of harnessing the power of random features.

Original: In most cases, “reservoir transformers” achieve better performance-efﬁciency trade-offs as measured by our newly introduced AUCC metric, and better test set generalization, on a variety of tasks and in a variety of settings.
Revised: In most cases, “reservoir transformers” achieve better performance-efﬁciency trade-offs as measured by our newly introduced AUCC metric, and better test set generalization, on a variety of tasks and in a variety of settings.

Original: Future work includes further investigating hybrid networks and backskipping architectures, as well as utilizing pruning strategies at inference time, in order to try to obtain even better performance/efﬁciency trade-offs.arXiv preprint [Figure] [Figure]
Revised: Future work includes further investigating hybrid networks and backskipping architectures, as well as utilizing pruning strategies at inference time, in order to try to obtain even better performance/efﬁciency trade-offs.

Original: A H YBRID N ETWORKS AND N ON -T RANSFORMER R ESERVOIRS
Revised: A H YBRID N ETWORKS AND N ON -T RANSFORMER R ESERVOIRS

Original: We investigate whether reservoir layers need to be transformer-based (or transformers-withoutattention, i
Revised: We investigate whether reservoir layers need to be transformer-based (or transformers-withoutattention, i

Original: [Figure]
Revised: [Figure]

Original: e., FFN).
Revised: e., FFN).

Original: We examine two different alternatives: bidirectional Gated Recurrent Units (Cho et al., 2014) and Convolutional Neural Networks (LeCun et al., 1998; Kim, 2014), speciﬁcally light dynamical convolutions (Wu et al., 2019).
Revised: We examine two different alternatives: bidirectional Gated Recurrent Units (Cho et al., 2014) and Convolutional Neural Networks (LeCun et al., 1998; Kim, 2014), speciﬁcally light dynamical convolutions (Wu et al., 2019).

Original: Figure 6 shows the results for these hybrids: depending on the setting, they may obtain a better AUCC than the regular transformer
Revised: Figure 6 shows the results for these hybrids: depending on the setting, they may obtain a better AUCC than the regular transformer

Original: [Figure]
Revised: [Figure]

Original: but this is less consistent than with the other reservoir layers, most likely because these layers have different computational properties.
Revised: but this is less consistent than with the other reservoir layers, most likely because these layers have different computational properties.

Original: It’s possible that these hybrids simply require further tuning, as we found e.g. up-projecting to help for BiGRUs, but studying this is outside of the scope of the current work.
Revised: It’s possible that these hybrids simply require further tuning, as we found e.g. up-projecting to help for BiGRUs, but studying this is outside of the scope of the current work.

Original: B D EEP DECODERS
Revised: B D EEP DECODERS

Original: We show that the same results hold for a 6-layer decoder on IWSLT (although less pronounced for
Revised: We show that the same results hold for a 6-layer decoder on IWSLT (although less pronounced for

Original: AUCC, probably because the decoder is computationally heavier).
Revised: AUCC, probably because the decoder is computationally heavier).

Original: See Figure 7 and Table 2.
Revised: See Figure 7 and Table 2.

Original: C F REEZING S TRATEGY
Revised: C F REEZING S TRATEGY

Original: We explored different strategies for the placement of reservoir layers and found the “alternating” strategy reported in the main body of the paper to work best.
Revised: We explored different strategies for the placement of reservoir layers and found the “alternating” strategy reported in the main body of the paper to work best.

Original: Generally, we found repetitive application of reservoirs to yield diminishing returns, as might be expected.
Revised: Generally, we found repetitive application of reservoirs to yield diminishing returns, as might be expected.

Original: See Figure 8.
Revised: See Figure 8.

Original: [Table] 
Revised: [Table] [Table]

Original: [Figure]
Revised: (averaged over multiple runs) saved for WMT for different model types and encoder depths. Max BLEU is for validation. Number of layers is for encoder, decoder depth is kept ﬁxed at 1. Ratio is computed compared to comparable number of layers in the normal case.

Original: D R O BERT A R ESULTS
Revised: D R O BERT A R ESULTS

Original: Here we present the additional RoBERTa results for convergence plot and AUCC in various decoder depth setting.
Revised: Here we present the additional RoBERTa results for convergence plot and AUCC in various decoder depth setting in Figure 10.

Original: As stated in the main paper, the difference of AUCC / ConvergencePlot between
Revised: As stated in the main paper, the difference of AUCC / Convergence

Original:  RoBERTa model with or without Reservoir layers are limited.
Revised: Plot between RoBERTa model with or without Reservoir layers are limited.

Original: [Figure] [Figure]
Revised: Moreover, we plot the downstream task performance for SST-2 and MNLI compared to the pretraining wall-clock time in Figure 9.

Original: [Figure]
Revised: It can be seen that the FFN Reservoir can achieve up to 25% and 10% pretraining time savings while matching the best performance of vanilla transformers for MNLI-m and SST2, respectively.

Original: E R ESERVOIR R ESULTS FOR TOTAL LAYERS
Revised: E R ESERVOIR R ESULTS FOR TOTAL LAYERS

Original: Here we present the shifted Reservoir Results for IWSLT14, WMT16, Enwik8 and RoBERTa ﬁnetuning in Figure10, 11, 12, 13,  respectively.
Revised: Here we present the shifted Reservoir Results for IWSLT14, WMT16, Enwik8 and RoBERTa ﬁnetuning in Figure 11, 12, 13, 14, respectively.

Original: We show the same results also hold when it comes to replace normal transformer blocks with Reservoir blocks at least for MT. 
Revised: We show the same results also hold when it comes to replace normal transformer blocks with Reservoir blocks at least for MT. [Table] [Table]

Original: F V ALIDATION P LOTS
Revised: F V ALIDATION P LOTS

Original: Here we present the validation plot for training a 8-layer encoder, 2-layer decoder model for
Revised: Here we present the validation plots for training a 8-layer encoder, 2-layer decoder model for

Original: IWSLT14, a 24-layer encoder, 1-layer decoder model for WMT14, a 48-layer decoder model for enwik8 and a 12-layer decoder model for RoBERTa for detailed steps to calculate the AUCC.
Revised: IWSLT14, a 24-layer encoder, 1-layer decoder model for WMT14, a 48-layer decoder model for enwik8 and a 12-layer decoder model for RoBERTa for detailed steps to calculate the AUCC.

Original: It can be clearly observed that given the conﬁgurations from Section 3.1, all the models haveessentially converged.
Revised: It can be clearly observed that given the conﬁgurations from Section 3.1, all the models have converged.

Original: So when we compute the area under the convergence curve, it depicted the training efﬁciency of the model (basically time x performance) till convergence.
Revised: So when we compute the area under the convergence curve, this depicts the training efﬁciency of the model (basically time x performance) until convergence.

Original: [Figure]
Revised: Speciﬁcally, we set T sufﬁciently high for computing the AUCC, which is 4h for IWSLT, 20h for WMT, 30h for enwik8 and 60h for RoBERTa pretraning. From the training plot in the appendix, we can see that each model has converged at that point. The Reservoir model in Figure 15 has 2 layers frozen for IWSLT14, 8 layers frozen for enwik8, and 4 layers frozen for WMT14 and RoBERTa. G R O BERT A P ROBING

Original: [Figure]
Revised: We follow Jawahar et al. and investigate what the frozen layers in the Reservoir Transformer have actually “learned” (while being forzen) as measured by probing tasks, reported in Table 6.

Original: [Figure]
Revised: The results are gathered over 3 random seeds for reporting the mean and standard deviation. From the table, we can see that generally probing performance is quite similar between Transformer and the T Reservoir model. We also noticed that the representations collected after the frozen layer (3, 5, 7, 9) in the T Reservoir actually have signiﬁcantly better performance over the regular Transformer representations across all the probing tasks.

Original: a li d a t i o n B
Revised: This has interesting repercussions for the study of “BERTology”, as it clearly shows, somewhat confusingly, that even completely random and frozen layers represent linguistic phenomena. [Figure] [Figure] [Figure]

Original: P C
Revised: [Figure] [Figure] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: [Table]