Original: A DDING R ECURRENCE TO P RETRAINED T RANSFORMERS
Revised: A DDING R ECURRENCE TO P RETRAINED T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
Revised: Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.

Original: While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures.
Revised: While the results from these models are impressive, applying them can be extremely computationally expensive, as is pretraining new models with the latest architectures.

Original: We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time.
Revised: We present a novel method for applying pretrained transformer language models which lowers their memory requirement both at training and inference time.

Original: An additional beneﬁt is that our method removes the ﬁxed context size constraint that most transformer models have, allowing for more ﬂexible use.
Revised: An additional beneﬁt is that our method removes the ﬁxed context size constraint that most transformer models have, allowing for more ﬂexible use.

Original: When applied to the GPT-2 language model, we ﬁnd that our method attains better perplexity than an unmodiﬁed GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.
Revised: When applied to the GPT-2 language model, we ﬁnd that our method attains better perplexity than an unmodiﬁed GPT-2 model on the PG-19 and WikiText-103 corpora, for a given amount of computation or memory.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent progress in NLP has been dominated by large pretrained transformer neural networks (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), and GPT-2 (Radford et al., 2019).
Revised: Recent progress in NLP has been dominated by large pretrained transformer neural networks (Vaswani et al., 2017), such as BERT (Devlin et al., 2019), and GPT-2 (Radford et al., 2019).

Original: However, these models have a memory footprint that is quadratic in input sequence length.
Revised: However, these models have a memory footprint that is quadratic in input sequence length.

Original: Although architectural innovations such as those of Kitaev et al.
Revised: Although architectural innovations such as those of Kitaev et al.

Original: and Rae et al.
Revised: and Rae et al.

Original: (2019) mitigate this and the issue of a predetermined maximum context size, large pretrained models applying these techniques are not available at this time.
Revised: (2019) mitigate this and the issue of a predetermined maximum context size, large pretrained models applying these techniques are not available at this time.

Original: Even if large pretrained models of this kind are released in the future, they will likely not cover the wide range of domains that BERT-family models have been published for.
Revised: Even if large pretrained models of this kind are released in the future, they will likely not cover the wide range of domains that BERT-family models have been published for.

Original: For example, there have been BERT-based models trained for other languages such as French (Le et al., 2020; Martin et al., 2020), Italian (Polignano et al., 2019), and many other languages (see Nozza et al.
Revised: For example, there have been BERT-based models trained for other languages such as French (Le et al., 2020; Martin et al., 2020), Italian (Polignano et al., 2019), and many other languages (see Nozza et al.

Original: for an overview) as well as speciﬁc domains such as scientiﬁc papers (Beltagy et al., 2019), biomedical papers (Lee et al., 2020), and health records (Rasmy et al., 2020).
Revised: for an overview) as well as speciﬁc domains such as scientiﬁc papers (Beltagy et al., 2019), biomedical papers (Lee et al., 2020), and health records (Rasmy et al., 2020).

Original: Individuals working with these models may not have the resources to train new models from scratch using the latest tricks, as the computation requirements for pretraining are extremely high.
Revised: Individuals working with these models may not have the resources to train new models from scratch using the latest tricks, as the computation requirements for pretraining are extremely high.

Original: As such, identifying ways that already existing models can be improved could be widely impactful.
Revised: As such, identifying ways that already existing models can be improved could be widely impactful.

Original: Another drawback of this family of models is that they have an a priori ﬁxed maximum context size (typically 512 or 1024 tokens for the currently available pretrained models).
Revised: Another drawback of this family of models is that they have an a priori ﬁxed maximum context size (typically 512 or 1024 tokens for the currently available pretrained models).

Original: A typical application of pretrained language models is producing contextual embeddings for a document.
Revised: A typical application of pretrained language models is producing contextual embeddings for a document.

Original: If the document is simply chunked into disjoint segments of 512 tokens, tokens at the boundary of a window will have less contextual information than tokens in the center of a window.
Revised: If the document is simply chunked into disjoint segments of 512 tokens, tokens at the boundary of a window will have less contextual information than tokens in the center of a window.

Original: This can be mitigated by striding the evaluation of the model, and only keeping the embedding for a token which has the largest context—but this adds quite a bit of wasted computation.
Revised: This can be mitigated by striding the evaluation of the model, and only keeping the embedding for a token which has the largest context—but this adds quite a bit of wasted computation.

Original: In this paper, we propose a method for augmenting and ﬁne-tuning pretrained transformer language models to use context without directly attending to it.
Revised: In this paper, we propose a method for augmenting and ﬁne-tuning pretrained transformer language models to use context without directly attending to it.

Original: Our method simultaneously allows for increasing the context size a transformer processes, while allowing a controllable trade-off between computation and perplexity.
Revised: Our method simultaneously allows for increasing the context size a transformer processes, while allowing a controllable trade-off between computation and perplexity.

Original: We accomplish this by adding a small recurrence module that computes a ﬁxed size representation from the transformer hidden states in a window of text.
Revised: We accomplish this by adding a small recurrence module that computes a ﬁxed size representation from the transformer hidden states in a window of text.

Original: Then, the representation for that window is used during processing of the next window.
Revised: Then, the representation for that window is used during processing of the next window.

Original: Shrinking the window size is then a way to reduce the memory footprint of the model, with less loss of performance than would occur with a standard transformer.
Revised: Shrinking the window size is then a way to reduce the memory footprint of the model, with less loss of performance than would occur with a standard transformer.

Original: Our experiments add recurrence GPT-2 language models, and ﬁne-tune them on the PG-19 (Rae et al., 2019) and WikiText-103 corpora (Merity et al., 2016), and require only the same amount of memory used for standard ﬁne-tuning of a pretrained PretrainedTransformer z PretrainedTransformer MLP t 1: T t ( T +1):2 T P (cid:0) t 2:( T +1) (cid:1)
Revised: Our experiments add recurrence GPT-2 language models, and ﬁne-tune them on the PG-19 (Rae et al., 2019) and WikiText-103 corpora (Merity et al., 2016), and require only the same amount of memory used for standard ﬁne-tuning of a pretrained language model.

Original: P (cid:0) t ( T +2):(2 T +1) (cid:1) h ( ‘ ins − 1)h ( ‘ ins − 1)h ( ‘ ins − 1) T ... h ( ‘ ins )h ( ‘ ins )h ( ‘ ins ) T ... h ( ‘ ins +1)h ( ‘ ins +1)h ( ‘ ins +1) T ...

Original: h prev language model.

Original: We demonstrate improvements in perplexity compared to a baseline model using the same amount of computation.
Revised: We demonstrate improvements in perplexity compared to a baseline model using the same amount of computation.

Original: Qualitative analysis shows that our recurrent module propagates certain information from previous windows of text, which can facilitate handling of long-distance dependencies with ﬁxed-size input windows.
Revised: Qualitative analysis shows that our recurrent module propagates certain information from previous windows of text, which can facilitate handling of long-distance dependencies with ﬁxed-size input windows.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Many methods have been proposed to lower the memory footprint or computation time of transformer language models, or allow them to be used on larger contexts.
Revised: Many methods have been proposed to lower the memory footprint or computation time of transformer language models, or allow them to be used on larger contexts.

Original: The Transformer-XL (Dai et al., 2019) allows a position within an attention window to attend to tokens from the previous windows by introducing relative position embeddings.
Revised: The Transformer-XL (Dai et al., 2019) allows a position within an attention window to attend to tokens from the previous windows by introducing relative position embeddings.

Original: While that mechanism, like ours, allows information to ﬂow between windows, existing BERT and GPT-2 models do not use relative position embeddings, so training from scratch would be necessary to take advantage of this architecture.
Revised: While that mechanism, like ours, allows information to ﬂow between windows, existing BERT and GPT-2 models do not use relative position embeddings, so training from scratch would be necessary to take advantage of this architecture.

Revised: Additionally, each layer in the Transformer-XL attends to the previous layer in the previous window, so the maximum attention horizon is ﬁnite.

Revised: Our recurrent method could theoretically pass information across an arbitrary distance, although one would not expect it to exceed the Transformer-XL’s horizon without a much larger scale of data than we experiment with.

Original: We list here some other modiﬁcations of the transformer architecture, somewhat imprecisely grouping them for brevity.
Revised: We list here some other modiﬁcations of the transformer architecture, somewhat imprecisely grouping them for brevity.

Original: For a more detailed discussion, see Tay et al.
Revised: For a more detailed discussion, see Tay et al.

Original: (2020b).
Revised: (2020b).

Original: Child et al.
Revised: Child et al.

Original: Qiu et al.
Revised: Qiu et al.

Original: (2019), Kitaev et al.
Revised: (2019), Kitaev et al.

Original: (2019), Sukhbaatar et al.
Revised: (2019), Sukhbaatar et al.

Original: (2019), and Roy et al.
Revised: (2019), and Roy et al.

Original: (2020) introduce sparsity to self-attention in various forms, reducing its memory cost.
Revised: (2020) introduce sparsity to self-attention in various forms, reducing its memory cost.

Original: Rae et al.
Revised: Rae et al.

Original: (2019) and Beltagy et al.
Revised: (2019) and Beltagy et al.

Original: (2020)—dynamically and statically respectively—add extra tokens to attend to which allow for global passing of information.
Revised: (2020)—dynamically and statically respectively—add extra tokens to attend to which allow for global passing of information.

Original: Tay et al.
Revised: Tay et al.

Original: (2020a)
Revised: (2020a)

Original: and Wu et al.
Revised: and Wu et al.

Original: (2019) replace dynamically computed self-attention with cheaper alternatives.
Revised: (2019) replace dynamically computed self-attention with cheaper alternatives.

Original: While the above methods all allow for a reduction in computation, they also all require training from scratch.
Revised: While the above methods all allow for a reduction in computation, they also all require training from scratch.

Original: Our goal is to allow more efﬁcient and powerful use of the wide array of existing pre-trained models that cover many domains.
Revised: Our goal is to allow more efﬁcient and powerful use of the wide array of existing pre-trained models that cover many domains.

Original: Cao et al.
Revised: Cao et al.

Original: (2020) propose the DeFormer, which also modiﬁes the execution of a pretrained transformer.
Revised: (2020) propose the DeFormer, which also modiﬁes the execution of a pretrained transformer.

Original: However, unlike our method, they decompose a single window into multiple windows by removing the attention interactions between these windows.
Revised: However, unlike our method, they decompose a single window into multiple windows by removing the attention interactions between these windows.

Original: This is largely orthogonal to our method, as one could both decompose windows of text, and additionally use our method to allow information to be passed between neighboring windows.
Revised: This is largely orthogonal to our method, as one could both decompose windows of text, and additionally use our method to allow information to be passed between neighboring windows.

Original: Similarly, distilled versions of pre-trained models such as DistilBERT (Sanh et al., 2019) provide more computational efﬁciency, but could be combined with our method to apply them to longer contexts, or reduce the quadratic cost of self-attention.
Revised: Similarly, distilled versions of pre-trained models such as DistilBERT (Sanh et al., 2019) provide more computational efﬁciency, but could be combined with our method to apply them to longer contexts, or reduce the quadratic cost of self-attention.

Original: Hao et al.
Revised: Hao et al.

Original: (2019) apply pre-trained transformers recurrently for machine translation, but do so by using an attention network to embed the document, applying a recurrent encoder to those embeddings, and using the recurrent encoder alongside a typical transformer encoder.
Revised: (2019) apply pre-trained transformers recurrently for machine translation, but do so by using an attention network to embed the document, applying a recurrent encoder to those embeddings, and using the recurrent encoder alongside a typical transformer encoder.

Original: This differs from our method as we are ﬁne-tuning language models, which are transformer decoders, and directly modifying the transformer’s computation with a recurrent connection, rather than running an RNN on top of embeddings produced by a transformer.
Revised: This differs from our method as we are ﬁne-tuning language models, which are transformer decoders, and directly modifying the transformer’s computation with a recurrent connection, rather than running an RNN on top of embeddings produced by a transformer.

Original:  M ETHOD
Revised:  M ETHOD

Original: The main idea of our method is to take a transformer that was pretrained in a ﬁxed context size setting and add recurrence at the level of T -token windows of text.
Revised: The main idea of our method is to take a transformer that was pretrained in a ﬁxed context size setting and add recurrence at the level of T -token windows of text.

Original: For example, instead of executing the model on one 1000 token window of text, we could instead execute our model with 10 windows of 100 tokens.
Revised: For example, instead of executing the model on one 1000 token window of text, we could instead execute our model with 10 windows of 100 tokens.

Original: The ﬁrst window is processed by the transformer model as normal, but for subsequent windows we add a supplementary embedding, which is generated using the hidden states from the preceding window (see Figure 1).
Revised: The ﬁrst window is processed by the transformer model as normal, but for subsequent windows we add a supplementary embedding, which is generated using the hidden states from the preceding window (see Figure 1).

Original: The recurrence module is extremely small compared to the size of transformer language model, so the additional computation required is negligible.
Revised: The recurrence module is extremely small compared to the size of transformer language model, so the additional computation required is negligible.

Original: 3. A DDING RECURRENCE TO PRETRAINED TRANSFORMERS
Revised: 3. A DDING RECURRENCE TO PRETRAINED TRANSFORMERS

Original: Starting by deﬁning terms, we will consider a pretrained transformer with L layers, a hidden state size of k , and a maximum context size of T tokens.
Revised: Starting by deﬁning terms, we will consider a pretrained transformer with L layers, a hidden state size of k , and a maximum context size of T tokens.

Original: Let h ( (cid:96) )
Revised: Let h ( (cid:96) )

Original: i ∈
Revised: i ∈

Original: R k be the output of the (cid:96) -th layer of the pretrained model, at position t .
Revised: R k be the output of the (cid:96) -th layer of the pretrained model, at position i .

Original: To produce a ﬁxed-size representation of tokens t 1 , t 2 , . . 
Revised: To produce a ﬁxed-size representation of tokens t 1 , t 2 , . . . , t T ,

Revised: PretrainedTransformer z

Revised: PretrainedTransformer MLP t 1: T t ( T +1):2 T

Revised: P (cid:0) t 2:( T +1) (cid:1)

Revised: P (cid:0) t ( T +2):(2 T +1) (cid:1) h ( ‘ ins − 1)h ( ‘ ins − 1)h ( ‘ ins − 1) T ... h ( ‘ ins )h ( ‘ ins )h ( ‘ ins ) T ... h ( ‘ ins +1)h ( ‘ ins +1)h ( ‘ ins +1) T ...

Original: . , t T , the embeddings produced by the pretrained transformer are mean-pooled as follows:
Revised: h prev the embeddings produced by the pretrained transformer are mean-pooled as follows:

Original: [Equation]
Revised: [Equation]

Original: where w (cid:96) are weights softmax-normalized from learned parameters α (cid:96) :
Revised: where w (cid:96) are weights softmax-normalized from learned parameters α [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁxed-size representation, z 1 , is passed through a feedforward network to produce an embedding h prev , 1 which represents the tokens processed so far, t 1: T .
Revised: The ﬁxed-size representation, z 1 , is passed through a feedforward network to produce an embedding h prev , 1 which represents the tokens processed so far, t 1: T .

Original: Next, instead of evaluating the pretrained transformer without modiﬁcation on positions T + 1 through 2 T , h prev , 1 is inserted at a single layer (denoted (cid:96) ins ) of the pretrained model, as an additional embedding that may be used in the computation of attention, as shown in Figure 2.
Revised: Next, instead of evaluating the pretrained transformer without modiﬁcation on positions T + 1 through 2 T , h prev , 1 is inserted at a single layer (denoted (cid:96) ins ) of the pretrained model, as an additional embedding that may be used in the computation of attention, as shown in Figure 2.

Original: To keep the number of embeddings per layer ﬁxed, this embedding is only used as a key and a value, but not a query, in the self-attention layer.
Revised: To keep the number of embeddings per layer ﬁxed, this embedding is only used as a key and a value, but not a query, in the self-attention layer.

Original: That is, for a window size of 300 tokens, there are 301 inputs to layer (cid:96) ins , but still only 300 outputs.
Revised: That is, for a window size of 300 tokens, there are 301 inputs to layer (cid:96) ins , but still only 300 outputs.

Original: The embeddings for positions T + 1 to 2 T are then pooled in the same way as Equation 1 to produce z 2 and passed through the feedforward network, outputting h prev , 2 .
Revised: The embeddings for positions T + 1 to 2 T are then pooled in the same way as Equation 1 to produce z 2 and passed through the feedforward network, outputting h prev , 2 .

Original: h prev , 2 is used to modify the execution of the pretrained language model on tokens 2 T + 1 through 3 T , and so on.
Revised: h prev , 2 is used to modify the execution of the pretrained language model on tokens 2 T + 1 through 3 T , and so on.

Original: Because the model is now being applied recurrently, it is trained end-to-end with backpropagation through time.
Revised: Because the model is now being applied recurrently, it is trained end-to-end with backpropagation through time.

Original: One could consider more complex recurrence modules, other methods for pooling the previous window’s embeddings, or for inserting h prev into the computation for the next window.
Revised: One could consider more complex recurrence modules, other methods for pooling the previous window’s embeddings, or for inserting h prev into the computation for the next window.

Original: We experimented with modiﬁcations such as max pooling instead of mean pooling, inserting multiple embeddings into the next window, inserting an embedding at all layers of the transformer for the next window, and using ﬁxed key attention as the pooling function.
Revised: We experimented with modiﬁcations such as max pooling instead of mean pooling, inserting multiple embeddings into the next window, inserting an embedding at all layers of the transformer for the next window, and using ﬁxed key attention as the pooling function.

Original: However the performance after each of these changes was not  signiﬁcantly better than the model presented above, so  we do not include those modiﬁcations here.
Revised: However during our preliminary experiments, we were not successful in ﬁnding a signiﬁcantly higher performing architecture than the one given above, so it is the one we present results for.

Original: 3. G RADIENT CHECKPOINTING IN NETWORKS WITH BOTTLENECKS
Revised: 3. G RADIENT CHECKPOINTING IN NETWORKS WITH BOTTLENECKS

Original: While our method can reduce the quadratic cost of attention by splitting the input into windows, we can also easily apply it to much longer contexts by use of gradient checkpointing (Chen et al., 2016).
Revised: While our method can reduce the quadratic cost of attention by splitting the input into windows, we can also easily apply it to much longer contexts by use of gradient checkpointing (Chen et al., 2016).

Original: Gradient checkpointing is a method for lowering the peak memory requirement of training large neural networks.
Revised: Gradient checkpointing is a method for lowering the peak memory requirement of training large neural networks.

Original: This is accomplished by storing only a subset of activations during the forward pass, and recomputing forward from those cached states during the backwards pass.
Revised: This is accomplished by storing only a subset of activations during the forward pass, and recomputing forward from those cached states during the backwards pass.

Original: For example, in a 100 layer feedforward network with uniformly wide layers, one could store the output of only every 10th layer.
Revised: For example, in a 100 layer feedforward network with uniformly wide layers, one could store the output of only every 10th layer.

Original: Then, during the backward pass, in order to compute the gradients for the 95th layer, one would re-compute layers 91 through 99 using the stored 90th layer activations.
Revised: Then, during the backward pass, in order to compute the gradients for the 95th layer, one would re-compute layers 91 through 99 using the stored 90th layer activations.

Original: The overall memory cost is reduced to √ L at the cost of a single additional forward pass.
Revised: The overall memory cost is reduced to √ L at the cost of a single additional forward pass.

Revised: [Figure]

Original: In a network with variable width, the memory reduction can be even larger.
Revised: In a network with variable width, the memory reduction can be even larger.

Original: When gradient checkpointing is applied to transformers, the outputs of each layer are usually stored ( k × L × T values), so that at most one set of self-attention activations is in memory at once.
Revised: When gradient checkpointing is applied to transformers, the outputs of each layer are usually stored ( k × L × T values), so that at most one set of self-attention activations is in memory at once.

Original: In the case of our recurrent models, we have an even narrower bottleneck: the z i ’s and h prev ,i ’s.
Revised: In the case of our recurrent models, we have an even narrower bottleneck: the z i ’s and h prev ,i ’s.

Original: Storing only these values means that the maximum number of activations present in memory while training on sequences N tokens[Figure] in length is M +2 k (cid:100) NT (cid:101) , where M is the number of activations stored when training the transformer on an individual window of length T .
Revised: Storing only these values means that the maximum number of activations present in memory while training on sequences N tokens in length is M +2 k (cid:100) N T (cid:101) , where M is the number of activations stored when training the transformer on an individual window of length T .

Original: Because k is extremely small compared to M , our model can be applied to very long contexts on any GPU on which the pretrained model can be ﬁne-tuned.
Revised: Because k is extremely small compared to M , our model can be applied to very long contexts on any GPU on which the pretrained model can be ﬁne-tuned.

Original:  R EVISITING THE EVALUATION OF TRANSFORMER LANGUAGE MODELS
Revised:  R EVISITING THE EVALUATION OF TRANSFORMER LANGUAGE MODELS

Original: Before describing the empirical evaluation of our method, we discuss how transformer language models are evaluated in related work.
Revised: Before describing the empirical evaluation of our method, we discuss how transformer language models are evaluated in related work.

Original: The standard way of measuring perplexity uses extra computation in order to make as much context available for each token prediction.
Revised: The standard way of measuring perplexity uses extra computation in order to make as much context available for each token prediction.

Original: This yields low perplexities, but does not reﬂect how practitioners use transformer language models in applications.
Revised: This yields low perplexities, but does not reﬂect how practitioners use transformer language models in applications.

Original: In this section, we describe the situation in detail and propose practical solutions that achieve relatively low perplexities while being closer to how transformers are used in practice.
Revised: In this section, we describe the situation in detail and propose practical solutions that achieve relatively low perplexities while being closer to how transformers are used in practice.

Original: 4. P OTENTIAL MISALIGNMENT BETWEEN LM EVALUATION AND APPLICATION
Revised: 4. P OTENTIAL MISALIGNMENT BETWEEN LM EVALUATION AND APPLICATION

Original: Transformers are often described as having quadratic time complexity in comparison to RNNs which have linear time complexity.
Revised: Transformers are often described as having quadratic time complexity in comparison to RNNs which have linear time complexity.

Original: However, this can be somewhat misleading when it comes to evaluation of perplexity.
Revised: However, this can be somewhat misleading when it comes to evaluation of perplexity.

Original: Given a test set of length N , an RNN requires O ( N ) time to evaluate—but reaching the best perplexity for a transformer requires O ( NT 2 ) , where T is its maximum context size.
Revised: Given a test set of length N , an RNN requires O ( N ) time to evaluate—but reaching the best perplexity for a transformer requires O ( NT 2 ) , where T is its maximum context size.

Original: (These time complexities exclude hidden state size, number of layers, and batch size.)
Revised: (These time complexities exclude hidden state size, number of layers, and batch size.)

Original: This much higher time complexity is due to the fact that a transformer may be run with its full context size once for each token in the test set, so that the maximum context is available for each prediction.
Revised: This much higher time complexity is due to the fact that a transformer may be run with its full context size once for each token in the test set, so that the maximum context is available for each prediction.

Original: Re-execution of the whole model for each token is required for models with absolute position embeddings, since hidden state reuse is only possible up to the maximum context size of the network.
Revised: Re-execution of the whole model for each token is required for models with absolute position embeddings, since hidden state reuse is only possible up to the maximum context size of the network.

Original: Note that it is possible to achieve smaller wall-clock time by splitting evaluation of a test set over multiple GPUs, but this is not applicable to the generation setting where outputs depend on prior ones.
Revised: Note that it is possible to achieve smaller wall-clock time by splitting evaluation of a test set over multiple GPUs, but this is not applicable to the generation setting where outputs depend on prior ones.

Original: To illustrate why re-computation is necessary, consider executing GPT-2 (which has 1024 position embeddings) on a test set.
Revised: To illustrate why re-computation is necessary, consider executing GPT-2 (which has 1024 position embeddings) on a test set.

Original: Each of the ﬁrst 1024 tokens of a test set will have been passed into the network using a distinct position embedding.
Revised: Each of the ﬁrst 1024 tokens of a test set will have been passed into the network using a distinct position embedding.

Original: Having exhausted the position embeddings, one option is to start again with the 1025th token being treated as position 1—we will refer to this as disjoint execution , illustrated in Figure 3a.
Revised: Having exhausted the position embeddings, one option is to start again with the 1025th token being treated as position 1—we will refer to this as disjoint execution , illustrated in Figure 3a.

Original: The issue with disjoint execution is that it requires predicting the tokens at the beginning of a window from a very small amount of context.
Revised: The issue with disjoint execution is that it requires predicting the tokens at the beginning of a window from a very small amount of context.

Original: The alternative, which is used for standard test set evaluation, is overlapped execution , as shown in
Revised: The alternative, which is used for standard test set evaluation, is overlapped execution , as shown in

Original: Figure 3b.
Revised: Figure 3b.

Original: The position embeddings are advanced by one position for each prediction, meaning that
Revised: The position embeddings are advanced by one position for each prediction, meaning that

Original: T − 1 tokens are repeated between consecutive evaluations of the transformer, requiring much more computation.
Revised: T − 1 tokens are repeated between consecutive evaluations of the transformer, requiring much more computation.

Original: The beneﬁt of this method is that it allows a model with T position embeddings to have T tokens of context for each prediction, as opposed to a variable amount between 1 and T .
Revised: The beneﬁt of this method is that it allows a model with T position embeddings to have T tokens of context for each prediction, as opposed to a variable amount between 1 and T .

Original: Stepping a transformer decoder forward one token at a time measures the best that such a model could perform, but it reﬂects a generative story that does not align with how the models may be used in practice.
Revised: Stepping a transformer decoder forward one token at a time measures the best that such a model could perform, but it reﬂects a generative story that does not align with how the models may be used in practice.

Original: A perplexity that only measures the ability of GPT-2 to generate the 1024th token given a context of 1023 tokens is not necessarily indicative of the model’s performance when generating from a smaller context.
Revised: A perplexity that only measures the ability of GPT-2 to generate the 1024th token given a context of 1023 tokens is not necessarily indicative of the model’s performance when generating from a smaller context.

Original: For example, the popular website Talk To Transformer 1 generates samples from GPT-2, but only provides 150 tokens of output.
Revised: For example, the popular website Talk To Transformer 1 generates samples from GPT-2, but only provides 150 tokens of output.

Original: The evaluation of GPT-2 by stepping forward one token at a time provides little information about the quality of such generations.
Revised: The evaluation of GPT-2 by stepping forward one token at a time provides little information about the quality of such generations.

Original: An example where the discrepancy is length instead of brevity is the GPT backed text adventure game AI Dungeon.
Revised: An example where the discrepancy is length instead of brevity is the GPT backed text adventure game AI Dungeon.

Original: In this setting, the number of tokens can easily reach and exceed the full context size GPT-2 was trained on.
Revised: In this setting, the number of tokens can easily reach and exceed the full context size GPT-2 was trained on.

Original: Using overlapped execution as described above, generating each token would be 1024 times slower than with disjoint execution, so perplexity calculated by overlapped execution does not match this use case either.
Revised: Using overlapped execution as described above, generating each token would be 1024 times slower than with disjoint execution, so perplexity calculated by overlapped execution does not match this use case either.

Original: While lower perplexity seems to correspond to better generation with shorter contexts in practice (perhaps due to parameter sharing between all sequence positions), there is no reason that this need be the case in principle.
Revised: While lower perplexity seems to correspond to better generation with shorter contexts in practice (perhaps due to parameter sharing between all sequence positions), there is no reason that this need be the case in principle.

Original: To demonstrate an extreme case of the concern being discussed, let F be a transformer model with vocabulary V , which uses the previous 1023 tokens as context, and consider the following generative story for generating token t i :
Revised: To demonstrate an extreme case of the concern being discussed, let F be a transformer model with vocabulary V , which uses the previous 1023 tokens as context, and consider the following generative story for generating token t i :

Original: [Equation]
Revised: [Equation]

Original: Clearly the above generative model would not be of any practical use for generation or otherwise.
Revised: Clearly the above generative model would not be of any practical use for generation or otherwise.

Original: However, because perplexity is calculated per token, increasing the size of the test set will lead to a measured perplexity that approaches that of a standard evaluation of the model F .
Revised: However, because perplexity is calculated per token, increasing the size of the test set will lead to a measured perplexity that approaches that of a standard evaluation of the model F .

Original: This example is not representative of the models that are trained in practice, as even generations much shorter than the maximum context size from a GPT-2 model are quite impressive.
Revised: This example is not representative of the models that are trained in practice, as even generations much shorter than the maximum context size from a GPT-2 model are quite impressive.

Original: However, it does demonstrate that the criteria that we use to compare models, or to select the best model during early stopping, place very high weight on the ability of the model to produce text given a full context, and a potentially vanishingly small amount on its ability to generate text using shorter contexts.
Revised: However, it does demonstrate that the criteria that we use to compare models, or to select the best model during early stopping, place very high weight on the ability of the model to produce text given a full context, and a potentially vanishingly small amount on its ability to generate text using shorter contexts.

Original: 4. V ARYING OVERLAP FOR EVALUATION
Revised: 4. V ARYING OVERLAP FOR EVALUATION

Original: As we are interested in increasing computational efﬁciency at evaluation time for pretrained models, we investigate their performance using overlapped execution, but with a reduced degree of overlap between windows.
Revised: As we are interested in increasing computational efﬁciency at evaluation time for pretrained models, we investigate their performance using overlapped execution, but with a reduced degree of overlap between windows.

Original: Varying the overlap lets us investigate the connection between degree of overlap and perplexity.
Revised: Varying the overlap lets us investigate the connection between degree of overlap and perplexity.

Original: The overlap used in evaluation will be deﬁned to be the number of tokens from each input window that are repeated in the next window (see Figure 3).
Revised: The overlap used in evaluation will be deﬁned to be the number of tokens from each input window that are repeated in the next window (see Figure 3).

Original: For example, consider a window size T = 10 and an overlap of 3.
Revised: For example, consider a window size T = 10 and an overlap of 3.

Original: The windows that the transformer will be executed are then t 1:10 , t 8:17 , t 15:24 , ..., t 1+7 n :10+7 n where n indexes the window.
Revised: The windows that the transformer will be executed are then t 1:10 , t 8:17 , t 15:24 , ..., t 1+7 n :10+7 n where n indexes the window.

Original: These input windows are used to predict the spans of tokens t 2:11 , t 12:18 , t 19:25 , ..., t 5+7 n :11+7 n .
Revised: These input windows are used to predict the spans of tokens t 2:11 , t 12:18 , t 19:25 , ..., t 5+7 n :11+7 n .

Original: Figure 3c illustrates an intermediate overlap setting with T = 3 and an overlap of 1.
Revised: Figure 3c illustrates an intermediate overlap setting with T = 3 and an overlap of 1.

Original: The perplexity-minimizing evaluation setting is then the extreme with an overlap T − 1 , and an overlap of 0 corresponds to disjoint execution.
Revised: The perplexity-minimizing evaluation setting is then the extreme with an overlap T − 1 , and an overlap of 0 corresponds to disjoint execution.

Original: While a transformer can be evaluated with any degree of overlap, our augmentation method produces the embedding h prev , which is used during training to help predict the ﬁrst token of a window.
Revised: While a transformer can be evaluated with any degree of overlap, our augmentation method produces the embedding h prev , which is used during training to help predict the ﬁrst token of a window.

Original: If we change the overlap at test time, the alignment of the text represented by h prev and the current window will be different than the model was trained for, and so performance will degrade.
Revised: If we change the overlap at test time, the alignment of the text represented by h prev and the current window will be different than the model was trained for, and so performance will degrade.

Original: To address this, we use the same overlap that will be used at test time during training for the recurrent models.
Revised: To address this, we use the same overlap that will be used at test time during training for the recurrent models.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now present experiments comparing our proposed technique to the default usage of transformer language models.
Revised: We now present experiments comparing our proposed technique to the default usage of transformer language models.

Original: We describe experiments on the WikiText-103 corpus and a subset of the PGcorpus, using the GPT-2-small language model as the pretrained transformer in our models. 
Revised: We describe experiments on the WikiText-103 corpus and a subset of the PG- is corpus, using the GPT-2-small language model as the pretrained transformer in our models. We also provide proof-of-concept experiments using RoBERTa

Revised: (Liu et al., 2019) on the HotpotQA (Yang et al., 2018) question answering dataset, indicating that our method can improve encoder performance for tasks other than language modeling.

Revised: All of our experiments are based on the Hugging

Original:  WikiText-103 is a standard language modeling corpus composed of approximately 29,000 documents from English Wikipedia, containing 103 million words.We use the WikiText-103 “raw” pooled.
Revised: Face Transformers library (Wolf et al., 2019). WikiText-103 is a standard language modeling corpus composed of approximately 29,000 documents from English Wikipedia, containing 103 million words.

Original: We found that it led to a slight increase in perplexity, so we report results with training and evaluation matching.

Original: [Table] corpus, which does not have rare words replaced by “UNK”.
Revised: We use the WikiText-103 “raw” corpus, which does not have rare words replaced by “UNK”.

Original: While GPT-2 uses BPE tokenization, we compute perplexity using the number of words rather than the number of BPE tokens for clarity.
Revised: While GPT-2 uses BPE tokenization, we compute perplexity using the number of words rather than the number of BPE tokens for clarity.

Original: Although WikiText-103 does test long term dependencies, many of the documents are still shorter than the context size of the models we test.
Revised: Although WikiText-103 does test long term dependencies, many of the documents are still shorter than the context size of the models we test.

Original: Therefore, we also use PG-19, which consists of books from the Project Gutenberg corpus.
Revised: Therefore, we also use PG-19, which consists of books from the Project Gutenberg corpus.

Original: The average length of a WikiText-103 document is 3.6K words, while PG-19 documents (i.e. books) average 69K words, which far exceeds the context size of the models we test.
Revised: The average length of a WikiText-103 document is 3.6K words, while PG-19 documents (i.e. books) average 69K words, which far exceeds the context size of the models we test.

Original: However, the full PG-19 dataset is over 20 times larger than WikiText-103, so we use only a subset of it for training due to computational constraints.
Revised: However, the full PG-19 dataset is over 20 times larger than WikiText-103, so we use only a subset of it for training due to computational constraints.

Original: Speciﬁcally, we use only the ﬁrst (alphabetically by ﬁlename) 1250 books of the PG-19 corpus, and use only the ﬁrst 15000 tokens of each of the books in the validation set for early stopping.
Revised: Speciﬁcally, we use only the ﬁrst (alphabetically by ﬁlename) 1250 books of the PG-19 corpus, and use only the ﬁrst 15000 tokens of each of the books in the validation set for early stopping.

Original: We make no modiﬁcations to the test set.
Revised: We make no modiﬁcations to the test set.

Original: In all our experiments we use the HuggingFace implementation of the pretrained GPT-2 small model (12 layers, 768-dimensional hidden state).
Revised: In all our experiments we use the HuggingFace implementation of the pretrained GPT-2 small model (12 layers, 768-dimensional hidden state).

Original: For both the recurrent and baseline models, the GPTmodel was ﬁne-tuned, not left frozen.
Revised: For both the recurrent and baseline models, the GPTmodel was ﬁne-tuned, not left frozen.

Original: We selected learning rates for both our models and the baseline separately, by evaluating on WikiText-103 for the same set of candidate learning rates.
Revised: We selected learning rates for both our models and the baseline separately, by evaluating on WikiText-103 for the same set of candidate learning rates.

Original: We used the same learning rates for the PG-19 experiments without further hyperparameter search.
Revised: We used the same learning rates for the PG-19 experiments without further hyperparameter search.

Original: We ﬁne-tune all models for 2 epochs, measuring the validation loss every 2 million tokens.
Revised: We ﬁne-tune all models for 2 epochs, measuring the validation loss every 2 million tokens.

Original: All models were trained with Adam (Kingma & Ba, 2014), warming the learning rate up linearly from 0 to its ﬁnal value over 100 steps.
Revised: All models were trained with Adam (Kingma & Ba, 2014), warming the learning rate up linearly from 0 to its ﬁnal value over 100 steps.

Original: The feedforward network used to produce h prev ,i from window i − 1 consisted of 3 hidden layers with dimension 200.
Revised: The feedforward network used to produce h prev ,i from window i − 1 consisted of 3 hidden layers with dimension 200.

Original: We ﬁxed (cid:96) ins to be 2.
Revised: We ﬁxed (cid:96) ins to be 2.

Original: Recall from Section 4 that we are interested in evaluating the models in a setting similar to how they would be used in practice.
Revised: Recall from Section 4 that we are interested in evaluating the models in a setting similar to how they would be used in practice.

Original: To that end, we report separate perplexities for different degrees of overlap between adjacent windows of text, as described in Section 4.2.
Revised: To that end, we report separate perplexities for different degrees of overlap between adjacent windows of text, as described in Section 4.2.

Original: For our models, we train with the same overlap that we test with, as unlike the baseline models, they cannot be trained with no overlap between adjacent windows and then tested with an overlap.
Revised: For our models, we train with the same overlap that we test with, as unlike the baseline models, they cannot be trained with no overlap between adjacent windows and then tested with an overlap.

Original: This is because the embedding of the previous window of text is expected to represent all tokens up until the ﬁrst token of the current window, but with an overlap of 30 for example, that embedding would be representing all tokens up until the 30th token of the current window.
Revised: This is because the embedding of the previous window of text is expected to represent all tokens up until the ﬁrst token of the current window, but with an overlap of 30 for example, that embedding would be representing all tokens up until the 30th token of the current window.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: We ﬁrst show that with the same amount of ﬁne-tuning, our method achieves lower perplexity than a baseline GPT-2 model when evaluated using the same window size and degree of overlap between adjacent windows of text.
Revised: We ﬁrst show that with the same amount of ﬁne-tuning, our method achieves lower perplexity than a baseline GPT-2 model when evaluated using the same window size and degree of overlap between adjacent windows of text.

Original: It is important to emphasize that the perplexities we report are based on pretrained models, and so should not be compared to models trained from scratch on these datasets.
Revised: It is important to emphasize that the perplexities we report are based on pretrained models, and so should not be compared to models trained from scratch on these datasets.

Original: The GPT-2 models were trained on text from a web crawl from which all Wikipedia documents are removed, but this still leaves open the possibility of quotes from Wikipedia having been encountered, or text from PG-19.
Revised: The GPT-2 models were trained on text from a web crawl from which all Wikipedia documents are removed, but this still leaves open the possibility of quotes from Wikipedia having been encountered, or text from PG-19.

Original: Table 1 shows the perplexity of our models and the non-recurrent GPT-2 models on the WikiText103 dataset.
Revised: Table 1 shows the perplexity of our models and the non-recurrent GPT-2 models on the WikiTextdataset.

Original: The models compared here all use windows of 300 tokens, with varying degrees of 
Revised: The models compared here all use windows of 300 tokens, with varying degrees of overlap.

Original: [Table]

Original: overlap. The baseline models can only access information from the previous window of text through the overlapping tokens, while the recurrent models have a ﬁxed size representation of the longer context.
Revised: The baseline models can only access information from the previous window of text through the overlapping tokens, while the recurrent models have a ﬁxed size representation of the longer context.

Original: Our addition of recurrence increases the performance of the GPT-2 models in this setting, but by a relatively small amount.
Revised: Our addition of recurrence increases the performance of the GPT-2 models in this setting, but by a relatively small amount.

Original: Increasing the overlap between each window of text decreases the perplexities of the baseline model as expected
Revised: Increasing the overlap between each window of text decreases the perplexities of the baseline model as expected, but also decreases the perplexity of the recurrent models.

Original: [Table]
Revised: [Table]

Original: but also decreases the perplexity of the recurrent models.

Original: This indicates that there is room to increase the capacity of the recurrence mechanism, as if it passed all relevant information about the previous window forward

Original: [Table]
Revised: [Table]

Original:  the  overlapping tokenswould be redundant.
Revised: This indicates that there is room to increase the capacity of the recurrence mechanism (potentially requiring more training data), as an ideal recurrence mechanism would render these overlapping tokens redundant.

Original: On the other hand, some useful information beyond what is contained in the local context is being propagated, as otherwise the baseline model should catch up in perplexity at higher overlaps.
Revised: On the other hand, some useful information beyond what is contained in the local context is being propagated, as otherwise the baseline model should catch up in perplexity at higher overlaps.

Original: To investigate this further, we also experiment with the PG-19 dataset.
Revised: To investigate this further, we also experiment with the PG-19 dataset.

Original: The results for the PG-19 experiments are shown in Table 2.
Revised: The results for the PG-19 experiments are shown in Table 2.

Original: While we ﬁnd only small increases in performances on the WikiText-103 dataset, we see larger improvements on PG-19, conﬁrming our prediction that the gains would be larger on a dataset that has a larger context available for each prediction on average.
Revised: While we ﬁnd only small increases in performances on the WikiText-103 dataset, we see larger improvements on PG-19, conﬁrming our prediction that the gains would be larger on a dataset that has a larger context available for each prediction on average.

Original: We ﬁnd that adding our recurrence module leads to a model that gives as good a perplexity with no overlap between adjacent windows as an unmodiﬁed model does when evaluated with an overlap of 30 out of 300 tokens in each window.
Revised: We ﬁnd that adding our recurrence module leads to a model that gives as good a perplexity with no overlap between adjacent windows as an unmodiﬁed model does when evaluated with an overlap of 30 out of 300 tokens in each window.

Original: Training the recurrent model with a 5 token overlap gives perplexity lower than the baseline perplexity with an overlap of 50 or even 75.
Revised: Training the recurrent model with a 5 token overlap gives perplexity lower than the baseline perplexity with an overlap of 50 or even 75.

Original: In terms of FLOPs, adding our recurrence module and overlapping adjacent windows of tokens by 50 is less than half as costly as using a non-recurrent model with an overlap of 200.
Revised: In terms of FLOPs, adding our recurrence module and overlapping adjacent windows of tokens by 50 is less than half as costly as using a non-recurrent model with an overlap of 200.

Original: 5. E FFECT OF WINDOW SIZE
Revised: 5. E FFECT OF WINDOW SIZE

Original: As one of our motivations is to retain performance while decreasing compute requirements, we experiment with varying the window size used by our model and an unmodiﬁed GPT-2 model.
Revised: As one of our motivations is to retain performance while decreasing compute requirements, we experiment with varying the window size used by our model and an unmodiﬁed GPT-2 model.

Original: At smaller window sizes the recurrent model has access to much more information than GPT-2, which can only attend to the current window.
Revised: At smaller window sizes the recurrent model has access to much more information than GPT-2, which can only attend to the current window.

Original: Because of this, we expect our augmentation to cause the performance to fall off less rapidly with decreasing window size.
Revised: Because of this, we expect our augmentation to cause the performance to fall off less rapidly with decreasing window size.

Original: The results, shown in Figure 4, conﬁrm this prediction, as the performance gap widens with smaller windows.
Revised: The results, shown in Figure 4, conﬁrm this prediction, as the performance gap widens with smaller windows.

Original: Figure 5 contains the same points (and additional baseline curves for various overlaps), but in terms of FLOPs rather than window size.
Revised: Figure 5 contains the same points (and additional baseline curves for various overlaps), but in terms of FLOPs rather than window size.

Original: All of the results of the recurrent models lie on the Pareto frontier, meaning that to improve perplexity or computational cost, one must worsen the other.
Revised: All of the results of the recurrent models lie on the Pareto frontier, meaning that to improve perplexity or computational cost, one must worsen the other.

Original: The non-monotonicity of the overlap 30 and 50 curves is due to the fact that at smaller window sizes, an overlap represents a higher fraction of the computation being used for positions that predictions were already produced for.
Revised: The non-monotonicity of the overlap 30 and 50 curves is due to the fact that at smaller window sizes, an overlap represents a higher fraction of the computation being used for positions that predictions were already produced for.

Original: Also note that while the baseline with overlap 50 curve has the lowest absolute perplexity in
Revised: Also note that while the baseline with overlap 50 curve has the lowest absolute perplexity in

Original: Figure 5, the recurrent models trained with overlaps shown in Table 2 still perform better.
Revised: Figure 5, the recurrent models trained with overlaps shown in Table 2 still perform better.

Original: 5. W HAT INFORMATION IS BEING PROPAGATED BETWEEN WINDOWS ?
Revised: 5. W HAT INFORMATION IS BEING PROPAGATED BETWEEN WINDOWS ?

Original: We now discuss some features that our models display when we compute continuations(using greedy decoding) from contexts in the PG-19 validation set, which illustrate types of information that the recurrent module passes (or fails to pass) forward.
Revised: We now discuss some features that our models display in greedily decoded continuations from contexts in the PG-19 validation set, which illustrate types of information that the recurrent module passes (or fails to pass) forward.

Original: Samples are included in Tables 3 andin the appendix.
Revised: Samples are included in Tables 4 and 5 in the appendix.

Original: The most common phenomenon we identify in these samples is successful propagation of topical information between adjacent windows.
Revised: The most common phenomenon we identify in these samples is successful propagation of topical information between adjacent windows.

Original: For instance, we see in Table 3 a context discussing geography and rivers, followed by a continuation maintaining the same topic, and we see a context discussing the topic of payment, leading to a mention of money in the continuation. 
Revised: For instance, we see in Table 4 a context discussing geography and rivers, followed by a continuation maintaining the same topic, and we see a context discussing the topic of payment, leading to a mention of money in the continuation. We give more rigorous quantitative support of this claim in Section 5.3.1.

Original: Beyond passing of topical information, another success case in the generations is passing of certain information about characters between windows—in Table 4 we see that pronouns in the continuations often reﬂect characters mentioned in the context, and we see an example in which the continuation includes “the two women”, after a context mentioning “the aunts”.
Revised: Beyond passing of topical information, another success case in the generations is passing of certain information about characters between windows—in Table 5 we see that pronouns in the continuations often reﬂect characters mentioned in the context, and we see an example in which the continuation includes “the two women”, after a context mentioning “the aunts”.

Original: This behavior was likely learned due to the fact that PG-19 consists of narratives, so correctly passing character information between windows is quite beneﬁcial.
Revised: This behavior was likely learned due to the fact that PG-19 consists of narratives, so correctly passing character information between windows is quite beneﬁcial.

Original: However, these examples also contain discontinuities between the context and the continuation, in terms of local syntax or facts of the narrative.
Revised: However, these examples also contain discontinuities between the context and the continuation, in terms of local syntax or facts of the narrative.

Original: We see that some sentences are not completed in the expected form (for instance, “There are lots of ways of being” is continued with a new quote rather than completion of the thought), and new characters are sometimes invented rather than continuing to reference those described in the context.
Revised: We see that some sentences are not completed in the expected form (for instance, “There are lots of ways of being” is continued with a new quote rather than completion of the thought), and new characters are sometimes invented rather than continuing to reference those described in the context.

Original: One sample has a closing quotation mark, predicted from the previous window, being interpreted as an opening quotation mark.
Revised: One sample has a closing quotation mark, predicted from the previous window, being interpreted as an opening quotation mark.

Original: These are the types of issues that an overlap between adjacent windows easily addresses—a fact that likely accounts in part for the gap between the recurrent model with disjoint and overlapped execution in Table 2.
Revised: These are the types of issues that an overlap between adjacent windows easily addresses—a fact that likely accounts in part for the gap between the recurrent model with disjoint and overlapped execution in Table 2.

Original: A higher capacity recurrent module might ﬁx these issues in exchange for additional computation.
Revised: A higher capacity recurrent module might ﬁx these issues in exchange for additional computation.

Revised: 5.3. Q UANTITATIVE EVALUATION OF TOPIC PROPAGATION

Revised: To verify the trend we identiﬁed of topic propagation in continuations generated by our recurrent models, we ﬁt an LDA topic model (Blei et al., 2003) with 20 topics to 5000 books from the PG-

Revised: [Table]

Revised: training set.

Revised: Given a bag of words, this topic model will assign a distribution over topics, so we can use a statistical distance as a metric for the similarity between the topics of two segments of text.

Original: C ONCLUSION AND F UTURE WORK
Revised: We sampled 8000 contexts of 300 tokens from the PG-19 validation set, and computed argmax decoded continuations of 30 tokens from the same models used to generate Table 4 6 . We then computed the Jensen-Shannon divergence (JSD) between the topic distribution of each context and the corresponding continuations. This procedure ﬁnds that continuations from the recurrent model have an average topic JSD of 0.5331, while those from the baseline model have an average topic JSD of 0.5951.

Revised: For a given context, the continuation given by the recurrent model is likely to have a lower JSD at least 60% of the time ( p < 0 . 00001 ).

Revised: 5. Q UESTION ANSWERING EXPERIMENTS

Revised: To investigate whether our recurrence method would be helpful in tasks other than language modeling, we ran a small experiment on the HotpotQA extractive question answering task, in the distractor setting.

Revised: In this setting, 10 paragraphs of context are given which must be used to answer the given question.

Revised: HotpotQA’s inputs can greatly exceed one 512 token window in length, making it an ideal test of our method.

Revised: The questions are a mix of span-based and yes/no questions.

Revised: In order to be able to reduce training time, we use a subset of 30000 randomly sampled questions from the training set.

Revised: We use the RoBERTa-base model for both the baseline and the recurrently augmented model.

Revised: To evaluate whether recurrence improves encoder performance on this task, we directly ﬁnetune the models to predict answer span start and end tokens, as done for question answering by Devlin et al.

Revised: (2019), and max pool the embedding of the [CLS] token across windows and use the result for three way classiﬁcation between “span”, “yes”, and “no”.

Revised: Because the non-recurrent baseline cannot process an entire example at once, we begin each input window with the question, separated from the text by a [SEP] token.

Revised: We use this input format for both the baseline and recurrent models.

Revised: For both models, we use a learning rate of 2e-5 and train for 4 epochs.

Revised: For the recurrent model, we mean-pool the ﬁnal RoBERTa layer for the previous window, and use a 2 layer, 768-dimensional hidden layer MLP to produce an embedding which is inserted at the second layer of the next window (i.e., (cid:96) ins = 2 ).

Revised: Table 3 shows F1 and exact match scores for both models on the HotpotQA dev set.

Revised: Adding the recurrence module improves both scores by about 1 point, indicating that our method of propagating information between windows can be beneﬁcial for question answering in addition to language modeling.

Revised: It should be noted that these values are not directly comparable to scores on the HotpotQA leaderboard, as we only used a subset of the training set, in addition to evaluating on the dev set rather than the private test set.

Revised: Nonetheless, we ﬁnd these initial experiments to be highly promising, especially given the lack of hyperparameter tuning.

Revised:  C ONCLUSION AND F UTURE WORK

Original: We showed that augmenting a pretrained language model with a recurrence module during ﬁnetuning can allow increased performance given a ﬁxed computational budget.
Revised: We showed that augmenting a pretrained language model with a recurrence module during ﬁnetuning can allow increased performance given a ﬁxed computational budget.

Original: Our method can be similarly applied to improve the computational efﬁciency of pretrained models that already exist for many languages and domains, as well as for future models that will be developed.
Revised: Our method can be similarly applied to improve the computational efﬁciency of pretrained models that already exist for many languages and domains, as well as for future models that will be developed.

Original: It can also allow their application to longer contexts than they were trained for, increasing their ﬂexibility.
Revised: It can also allow their application to longer contexts than they were trained for, increasing their ﬂexibility.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: Here we provide some example continuations for contexts from the PG-19 validation set.
Revised: Here we provide some example continuations for contexts from the PG-19 validation set.

Original: The samples were generated with greedy argmax decoding which leads to a large amount of repetition, however we were more concerned with reducing variance and identifying the most likely continuation than optimizing for sample quality
Revised: The samples were generated with greedy argmax decoding, which leads to a large amount of repetition, however we were more concerned with reducing variance and identifying the most likely continuation than optimizing for sample quality.

Original: [Table]
Revised: [Table]

Original: Topic ... It drains a territory of 300,000 square miles, and, traced back to the rise of its principal source, is 2,000 miles long.
Revised: Topic ... It drains a territory of 300,000 square miles, and, traced back to the rise of its principal source, is 2,000 miles long.

Original: At two points, Needles and Yuma on the California boundary, it is crossed by a railroad.
Revised: At two points, Needles and Yuma on the California boundary, it is crossed by a railroad.

Original: Elsewhere its course lies far from Caucasian settlements and far from the routes of common travel, in the heart of a vast region fenced on the one hand by arid plains or deep forests and on the other by formidable mountains .
Revised: Elsewhere its course lies far from Caucasian settlements and far from the routes of common travel, in the heart of a vast region fenced on the one hand by arid plains or deep forests and on the other by formidable mountains .

Original: The ﬁrst of the two great rivers, the
Revised: The ﬁrst of the two great rivers, the

Original: Mississippi, was the ﬁrst to be navigated by the Americans, and the ﬁrst to be navigated by the Indians.
Revised: Mississippi, was the ﬁrst to be navigated by the Americans, and the ﬁrst to be navigated by the Indians.

Original: The other was the ﬁrst to be navigated by the Americans, and the ﬁrst to be navigated by the Indians.
Revised: The other was the ﬁrst to be navigated by the Americans, and the ﬁrst to be navigated by the Indians.

Original: The ﬁrst of the two great streams, the Mississippi, was navigable by the Indians, and the ﬁrst by the English...
Revised: The ﬁrst of the two great streams, the Mississippi, was navigable by the Indians, and the ﬁrst by the English...

Original: Rivers ... “Mr. Hammerby will have to wait for his money.”
Revised: Rivers ... “Mr. Hammerby will have to wait for his money.”

Original: “Will he?”
Revised: “Will he?”

Original: “I don’t see what else he can do.
Revised: “I don’t see what else he can do.

Original: But, aside from that, three dollars and a half won’t keep us a month.
Revised: But, aside from that, three dollars and a half won’t keep us a month.

Original: I’ll have to get the money, and he’ll be glad to see you again.”
Revised: I’ll have to get the money, and he’ll be glad to see you again.”

Original: “I’m glad to see you again, Tom.
Revised: “I’m glad to see you again, Tom.

Original: I’m glad to see you again.
Revised: I’m glad to see you again.

Original: I’m glad to see you again. I’m glad to see you again.
Revised: I’m glad to see you again. I’m glad to see you again.

Original: I’m glad to see you again
Revised: I’m glad to see you again

Original: Money ...
Revised: Money ...

Original: Long’s his sister, Thankful, lived, we couldn’t say nothing, of course.
Revised: Long’s his sister, Thankful, lived, we couldn’t say nothing, of course.

Original: If she wanted to slave and work so’s her brother could live in idleness and sloth, why, that was her business.
Revised: If she wanted to slave and work so’s her brother could live in idleness and sloth, why, that was her business.

Original: There ain’t any law against a body’s making a fool of herself, more’s the pity.
Revised: There ain’t any law against a body’s making a fool of herself, more’s the pity.

Original: But she’s been dead a year, and he’s done nothing since but live on those that’ll trust him, and ask help from the town.
Revised: But she’s been dead a year, and he’s done nothing since but live on those that’ll trust him, and ask help from the town.

Original: He ain’t sick–except sick of work.
Revised: He ain’t sick–except sick of work.

Original: [...] to-mor rer and his wife, and the whole family.
Revised: [...] to-mor rer and his wife, and the whole family.

Original: He was a good man, and he was a good man. He was a good man, and he was a good man.
Revised: He was a good man, and he was a good man. He was a good man, and he was a good man.

Original: He was a good man, and he was a good man. He was a good man, and he was a good man.
Revised: He was a good man, and he was a good man. He was a good man, and he was a good man.

Original: He
Revised: He

Original: Family,
Revised: Family,

Original: Description of a man ... and any form of thought or any institution tending to suppress education or destroy intelligence strikes at the very essence of the government, and constitutes a treason which no law can meet, and for which no punishment is adequate.
Revised: Description of a man ... and any form of thought or any institution tending to suppress education or destroy intelligence strikes at the very essence of the government, and constitutes a treason which no law can meet, and for which no punishment is adequate.

Original: Education, then, as universally diffused as the elements of God, is the life-blood of our body politic.
Revised: Education, then, as universally diffused as the elements of God, is the life-blood of our body politic.

Original: The intelligence of the people is the one great fact of our civilization and our prosperity,–it is the only way to preserve the peace of the world.
Revised: The intelligence of the people is the one great fact of our civilization and our prosperity,–it is the only way to preserve the peace of the world.

Original: The only way to preserve the peace of the world is to preserve the liberty of the people.
Revised: The only way to preserve the peace of the world is to preserve the liberty of the people.

Original: The ﬁrst step to be taken is to abolish the power of the government.
Revised: The ﬁrst step to be taken is to abolish the power of the government.

Original: The ﬁrst step to be taken is to abolish the power of the people.
Revised: The ﬁrst step to be taken is to abolish the power of the people.

Original: The ﬁrst step to ...
Revised: The ﬁrst step to ...

Original: Government
Revised: Government

Revised: Context

Original: [Table]
Revised: [Table]

Original: Elaine, however, did not gratify their hopes; she referred to the ball with the detachment she would have shown in describing a drawing-room show of cottage industries.
Revised: Elaine, however, did not gratify their hopes; she referred to the ball with the detachment she would have shown in describing a drawing-room show of cottage industries.

Original: It was not difﬁcult to discern in her description of the affair the confession that she had been slightly bored.
Revised: It was not difﬁcult to discern in her description of the affair the confession that she had been slightly bored.

Original: From Courtenay, later in the day, the aunts received a much livelier impression of the house, and the two women were standing in the doorway.
Revised: From Courtenay, later in the day, the aunts received a much livelier impression of the house, and the two women were standing in the doorway.

Original: “I am afraid I have not been very well,” she said, as she looked at the two men.
Revised: “I am afraid I have not been very well,” she said, as she looked at the two men.

Original: “I have been very much worried about you, and I have been very much worried about you.
Revised: “I have been very much worried about you, and I have been very much worried about you.

Original: I have been very much worried about you, and I have been very much worried about you.
Revised: I have been very much worried about you, and I have been very much worried about you.

Original: I have been very much pleased with you, and I have been very much ...
Revised: I have been very much pleased with you, and I have been very much ...

Original: “Oh, he’s not at all a bookworm,” said Suzette quickly, “though he’s tremendously well-read.
Revised: “Oh, he’s not at all a bookworm,” said Suzette quickly, “though he’s tremendously well-read.

Original: He’s quite the man of action.”
Revised: He’s quite the man of action.”

Original: “Does he hunt?”
Revised: “Does he hunt?”

Original: asked Elaine.
Revised: asked Elaine.

Original: “No, he doesn’t get much time or opportunity for riding.”
Revised: “No, he doesn’t get much time or opportunity for riding.”

Original: “What a pity,” commented Elaine; “I don’t think I could marry a man who wasn ” “I don’t know,” said the Sergeant, “but I have heard that you are a very good friend.”
Revised: “What a pity,” commented Elaine; “I don’t think I could marry a man who wasn ” “I don’t know,” said the Sergeant, “but I have heard that you are a very good friend.”

Original: “I am not,” said the girl, “I am not.”
Revised: “I am not,” said the girl, “I am not.”

Original: “I am not,” said the
Revised: “I am not,” said the

Original: Sergeant, “I am not.”
Revised: Sergeant, “I am not.”

Original: “I am not,” said the girl, “I am not.
Revised: “I am not,” said the girl, “I am not.

Original: ... Gregg was still the soft-stepping scoundrel who invited the toe of honesty, and Mornway, as he entered, was conscious of a sharp revulsion of feeling.
Revised: ... Gregg was still the soft-stepping scoundrel who invited the toe of honesty, and Mornway, as he entered, was conscious of a sharp revulsion of feeling.

Original: But it was impossible to evade the interview, and he sat silent while the man stated his case.
Revised: But it was impossible to evade the interview, and he sat silent while the man stated his case.

Original: Mrs. Mornway had represented the stenographer as being in desperate straits, and ready to accept any job that could be found, but though his appearance might have seemed to corroborate her account, he evidently took a less hopeless view of his case, and
Revised: Mrs. Mornway had represented the stenographer as being in desperate straits, and ready to accept any job that could be found, but though his appearance might have seemed to corroborate her account, he evidently took a less hopeless view of his case, and

Original: the
Revised: the

Original: Governor found with surprise that he had ﬁxed his eye on a clerkship that he had been in the habit of doing, and that he had been in the habit of doing.
Revised: Governor found with surprise that he had ﬁxed his eye on a clerkship that he had been in the habit of doing, and that he had been in the habit of doing.

Original: “I’m not going to be a fool,” he said, “and I’m not going to be a fool,...
Revised: “I’m not going to be a fool,” he said, “and I’m not going to be a fool,...

Original: Many transactions too, from a desire to avoid worrying me, were carried through without my knowledge, although formerly, as a matter of course, they would have been submitted to me.
Revised: Many transactions too, from a desire to avoid worrying me, were carried through without my knowledge, although formerly, as a matter of course, they would have been submitted to me.

Original: Strangers, when they called, asked to see Johnson or Marsh.
Revised: Strangers, when they called, asked to see Johnson or Marsh.

Original: I directed the messenger that they were to be shown into my room if I was disengaged.
Revised: I directed the messenger that they were to be shown into my room if I was disengaged.

Original: This was a failure, for, when they came, I was obliged to ask for help, which was not given very generously.
Revised: This was a failure, for, when they came, I was obliged to ask for help, which was not given very generously.

Original: Sometimes I sent for the papers, but it took a long time to read them, and my visitors became impatient.
Revised: Sometimes I sent for the papers, but it took a long time to read them, and my visitors became impatient.

Original: During one of these interviews, I remember that I was sorely perplexed, but I had managed to get the money, and I was glad to get it.
Revised: During one of these interviews, I remember that I was sorely perplexed, but I had managed to get the money, and I was glad to get it.

Original: The next day I went to the Court of the
Revised: The next day I went to the Court of the

Original: Emperor, and found him standing in the courtyard of the Palace, with a large number of soldiers.
Revised: Emperor, and found him standing in the courtyard of the Palace, with a large number of soldiers.

Original: He was dressed in a very dirty uniform, and wore a very dirty hat. He was dressed in a very dirty uniform, and wore a very dirty hat.
Revised: He was dressed in a very dirty uniform, and wore a very dirty hat. He was dressed in a very dirty uniform, and wore a very dirty hat.

Original: He was dressed in a very dirty hat, and wore a very dirty hat ...
Revised: He was dressed in a very dirty hat, and wore a very dirty hat ...

Original: I know at which crossings to look out–I know what
Revised: I know at which crossings to look out–I know what

Original: I’m going to see in the shop-windows.
Revised: I’m going to see in the shop-windows.

Original: It saves a lot of wear and tear to know what’s coming.
Revised: It saves a lot of wear and tear to know what’s coming.

Original: For a good many years I never did know, from one minute to another, and now I like to think that everything’s cut-and-dried, and nothing unexpected can jump out at me like a tramp from a ditch.”
Revised: For a good many years I never did know, from one minute to another, and now I like to think that everything’s cut-and-dried, and nothing unexpected can jump out at me like a tramp from a ditch.”

Original: He paused calmly to knock the ashes from his cigar, and Garnett said with a smile: “Doesn’t such a plan of life cut off nearly all the possibilities?”
Revised: He paused calmly to knock the ashes from his cigar, and Garnett said with a smile: “Doesn’t such a plan of life cut off nearly all the possibilities?”

Original: The old gentleman made a contemptuous motion.
Revised: The old gentleman made a contemptuous motion.

Original: “Possibilities of what?
Revised: “Possibilities of what?

Original: Of being multifariously miserable?
Revised: Of being multifariously miserable?

Original: There are lots of ways of being ”I’m afraid I’m not going to be able to do that,” he said.
Revised: There are lots of ways of being ”I’m afraid I’m not going to be able to do that,” he said.

Original: “I’m going to have to go to the station.
Revised: “I’m going to have to go to the station.

Original: I’m going to have to go to the station.
Revised: I’m going to have to go to the station.

Original: I want to see the station.
Revised: I want to see the station.

Original: I want to see the station. I want to see the station.
Revised: I want to see the station. I want to see the station.

Original: I want to see the station. I want to see the station.
Revised: I want to see the station. I want to see the station.

Original: I want to see the station. I want to see the station.
Revised: I want to see the station. I want to see the station.

Original: I want to see
Revised: I want to see