Original: T RAIN S HORT , T EST L ONG : A TTENTION WITH L INEAR B IASES E NABLES I NPUT L ENGTH E XTRAPOLATION
Revised: T RAIN S HORT , T EST L ONG : A TTENTION WITH L INEAR B IASES E NABLES I NPUT L ENGTH E XTRAPOLATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Since the introduction of the transformer model by Vaswani et al.
Revised: Since the introduction of the transformer model by Vaswani et al.

Original: (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?
Revised: (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?

Original: We ﬁrst show that extrapolation can be enabled by simply changing the position representation method, though we ﬁnd that current methods do not allow for efﬁcient extrapolation.
Revised: We ﬁrst show that extrapolation can be enabled by simply changing the position representation method, though we ﬁnd that current methods do not allow for efﬁcient extrapolation.

Original: We therefore introduce a simpler and more efﬁcient position method,
Revised: We therefore introduce a simpler and more efﬁcient position method,

Original: Attention with Linear Biases (ALiBi).
Revised: Attention with Linear Biases (ALiBi).

Original: ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance.
Revised: ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance.

Original: We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory.
Revised: We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory.

Original: ALiBi’s inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.
Revised: ALiBi’s inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: When constructing a transformer-based language model, a major design decision is the length of training sequences, denoted L herein, which has to date been equivalent to the length of inference sequences.
Revised: When constructing a transformer-based language model, a major design decision is the length of training sequences, denoted L herein, which has to date been equivalent to the length of inference sequences.

Original: More context, achieved by larger L , improves predictions at inference time.
Revised: More context, achieved by larger L , improves predictions at inference time.

Original: But longer sequences are more expensive to train on.
Revised: But longer sequences are more expensive to train on.

Original: Before transformers, RNN language models were trained on shorter- L sequences and assumed to generalize to longer contexts at inference time (Mikolov et al., 2010; Mikolov & Zweig, 2012;
Revised: Before transformers, RNN language models were trained on shorter- L sequences and assumed to generalize to longer contexts at inference time (Mikolov et al., 2010; Mikolov & Zweig, 2012;

Original: Zaremba et al., 2014).
Revised: Zaremba et al., 2014).

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017), introducing the transformer, speculated that it “may [...] extrapolate to sequence lengths longer than the ones encountered during training.”
Revised: (2017), introducing the transformer, speculated that it “may [...] extrapolate to sequence lengths longer than the ones encountered during training.”

Original: We deﬁne extrapolation as a model’s ability to continue performing well as the number of input tokens during validation increases beyond the number of tokens on which the the model was trained.
Revised: We deﬁne extrapolation as a model’s ability to continue performing well as the number of input tokens during validation increases beyond the number of tokens on which the the model was trained.

Original: We ﬁnd that transformer language models (LMs) that use sinusoidal position embeddings have very weak extrapolation abilities; see Figure 1.
Revised: We ﬁnd that transformer language models (LMs) that use sinusoidal position embeddings have very weak extrapolation abilities; see Figure 1.

Original: We demonstrate that this failure to extrapolate is caused by the position embedding method.
Revised: We demonstrate that this failure to extrapolate is caused by the position embedding method.

Original: As shown in Figure 1, recent alternatives to the original sinusoidal position method (Su et al., 2021;
Revised: As shown in Figure 1, recent alternatives to the original sinusoidal position method (Su et al., 2021;

Original: Raffel et al., 2020) have improved extrapolation.
Revised: Raffel et al., 2020) have improved extrapolation.

Original: However, the better of these, the T5 bias, is considerably slower than the sinusoidal approach and uses extra memory and parameters (Figure 2).
Revised: However, the better of these, the T5 bias, is considerably slower than the sinusoidal approach and uses extra memory and parameters (Figure 2).

Original: We therefore introduce Attention with Linear Biases (ALiBi) to facilitate efﬁcient extrapolation.
Revised: We therefore introduce Attention with Linear Biases (ALiBi) to facilitate efﬁcient extrapolation.

Original: ALiBi negatively biases attention scores with a linearly decreasing penalty proportional to the distance between the relevant key and query.
Revised: ALiBi negatively biases attention scores with a linearly decreasing penalty proportional to the distance between the relevant key and query.

Original: Our simple approach eliminates position embeddings.
Revised: Our simple approach eliminates position embeddings.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Inference Input Tokens
Revised: Inference Input Tokens

Original: P e r p l e x it y ( )
Revised: P e r p l e x it y ( )

Original: Extrapolation for Models Trained on 512 Tokens
Revised: Extrapolation for Models Trained on 512 Tokens

Original: Sinusoidal
Revised: Sinusoidal

Original: Rotary
Revised: Rotary

Original: T5 Bias
Revised: T5 Bias

Original: ALiBi 

Revised: ALiBi

Original: Inference Input Tokens
Revised: Inference Input Tokens

Original: P e r p l e x it y ( )
Revised: P e r p l e x it y ( )

Original: Extrapolation for Models Trained on 1024 Tokens
Revised: Extrapolation for Models Trained on 1024 Tokens

Original: Sinusoidal
Revised: Sinusoidal

Original: Rotary
Revised: Rotary

Original: T5 Bias
Revised: T5 Bias

Original: ALiBi
Revised: ALiBi

Original: Compared to a sinusoidal model trained on the same input length, our method requires no additional runtime or parameters and incurs a negligible (0–0.7%) memory increase.
Revised: Compared to a sinusoidal model trained on the same input length, our method requires no additional runtime or parameters and incurs a negligible (0–0.7%) memory increase.

Original: ALiBi can be implemented by changing only a few lines of existing transformer code.
Revised: ALiBi can be implemented by changing only a few lines of existing transformer code.

Original: Using ALiBi, a transformer LM can be trained on short- L sequences and therefore at much lower cost, and it can still be reliably applied to long sequences at runtime.
Revised: Using ALiBi, a transformer LM can be trained on short- L sequences and therefore at much lower cost, and it can still be reliably applied to long sequences at runtime.

Original: For example, a 1.3 billion parameter LM trained on L = 1024 tokens with ALiBi achieves the same perplexity as a sinusoidal model trained on L = 2048 when both are tested on sequences of 2048 tokens, even though our model is 11% faster and uses 11% less memory.
Revised: For example, a 1.3 billion parameter LM trained on L = 1024 tokens with ALiBi achieves the same perplexity as a sinusoidal model trained on L = 2048 when both are tested on sequences of 2048 tokens, even though our model is 11% faster and uses 11% less memory.

Original: Though performance peaks at around two times the number of tokens that the model was trained on,
Revised: Though performance peaks at around two times the number of tokens that the model was trained on,

Original: ALiBi maintains strong performance even on sequences of length 10,000.
Revised: ALiBi maintains strong performance even on sequences of length 10,000.

Original: In recently explored settings where NLP training examples are given as context to an LM (Brown et al., 2020), our approach will allow exposure to more examples.
Revised: In recently explored settings where NLP training examples are given as context to an LM (Brown et al., 2020), our approach will allow exposure to more examples.

Original: Additionally, it enables generation of longer outputs.
Revised: Additionally, it enables generation of longer outputs.

Original:  C URRENT A PPROACHES D O N OT E XTRAPOLATE E FFICIENTLY
Revised:  C URRENT A PPROACHES D O N OT E XTRAPOLATE E FFICIENTLY

Original: We show for the ﬁrst time that the sinusoidal position method, which technically should be able to extrapolate, in practice has very limited extrapolation capabilities.
Revised: We show for the ﬁrst time that the sinusoidal position method, which technically should be able to extrapolate, in practice has very limited extrapolation capabilities.

Original: Though the rotary position method improves over the sinusoidal one, it still does not achieve satisfying results.
Revised: Though the rotary position method improves over the sinusoidal one, it still does not achieve satisfying results.

Original: Holding everything else constant, we are the ﬁrst to observe that the T5 biasposition method leads to better extrapolates than either of these, and so we conclude that extrapolation ability depends heavily on the position embedding.
Revised: Holding everything else constant, we are the ﬁrst to observe that the T5 bias method leads to better extrapolation than either of these, and so we conclude that extrapolation ability depends heavily on the position embedding.

Original: Unfortunately, the T5 bias is computationally costly (Figure 2).
Revised: Unfortunately, the T5 bias is computationally costly (Figure 2).

Original: 2. B ACKGROUND AND E XPERIMENTAL S ETUP
Revised: 2. B ACKGROUND AND E XPERIMENTAL S ETUP

Original: A transformer LM receives a list of tokens and outputs a probability distribution representing its prediction for the next token.
Revised: A transformer LM receives a list of tokens and outputs a probability distribution representing its prediction for the next token.

Original: We call the input list the current input subsequence since the inputs to language models are typically subsequences from (much longer) training or evaluation sequences.
Revised: We call the input list the current input subsequence since the inputs to language models are typically subsequences from (much longer) training or evaluation sequences.

Original: During both training and perplexity evaluation (i.e., scoring a ﬁxed sequence), many predictions can be calculated at once; this is done using a “causal mask” that ensures each position’s prediction is inﬂuenced only by tokens to its left.
Revised: During both training and perplexity evaluation (i.e., scoring a ﬁxed sequence), many predictions can be calculated at once; this is done using a “causal mask” that ensures each position’s prediction is inﬂuenced only by tokens to its left.

Original: Let L be the length of each input subsequence during training; it includes L predictions, which on average have access to L +12 tokens of (left) context.
Revised: Let L be the length of each input subsequence during training; it includes L predictions, which on average have access to L +12 tokens of (left) context.

Original: To explore a model’s extrapolation abilities, we are interested in cases where sequences of length L valid >
Revised: To explore a model’s extrapolation abilities, we are interested in cases where sequences of length L valid >

Original: L are considered at evaluation time.
Revised: L are considered at evaluation time.

Original: When L differs between inference and training, we use L to refer to the length of subsequences during training and L valid to refer to their length at validation.
Revised: When L differs between inference and training, we use L to refer to the length of subsequences during training and L valid to refer to their length at validation.

Original: [Figure]
Revised: [Figure]

Original: Nonoverlapping Inference To train on or evaluate a sequence longer than L tokens, it is typical to segment the sequence into L -length subsequences and train on or evaluate them independently.
Revised: Nonoverlapping Inference To train on or evaluate a sequence longer than L tokens, it is typical to segment the sequence into L -length subsequences and train on or evaluate them independently.

Original: Unless otherwise stated, we use nonoverlapping inference to report perplexity scores.
Revised: Unless otherwise stated, we use nonoverlapping inference to report perplexity scores.

Original: Extrapolation During Inference Formally, the functions that deﬁne a transformer layer are agnostic to input length; 3 they map from some arbitrary, unﬁxed number of input vectors to the same number of output vectors.
Revised: Extrapolation During Inference Formally, the functions that deﬁne a transformer layer are agnostic to input length; 3 they map from some arbitrary, unﬁxed number of input vectors to the same number of output vectors.

Original: When transformers are applied to data that is inherently sequential, like text, positional information is injected into the inputs in various ways.
Revised: When transformers are applied to data that is inherently sequential, like text, positional information is injected into the inputs in various ways.

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017) discussed two options for embedding positions into vectors to be added to word embeddings: learning embeddings for speciﬁc positions and unlearned sinusoidal embeddings.
Revised: (2017) discussed two options for embedding positions into vectors to be added to word embeddings: learning embeddings for speciﬁc positions and unlearned sinusoidal embeddings.

Original: They observed similar performance between these two but preferred the sinusoidal approach, which they argued might extrapolate to longer input sequences during inference.
Revised: They observed similar performance between these two but preferred the sinusoidal approach, which they argued might extrapolate to longer input sequences during inference.

Original: We ﬁnd that this model cannot extrapolate to more than a few dozen tokens beyond L .
Revised: We ﬁnd that this model cannot extrapolate to more than a few dozen tokens beyond L .

Original: Experiment Setup We ﬁrst test the extrapolation abilities of various position methods on the
Revised: Experiment Setup We ﬁrst test the extrapolation abilities of various position methods on the

Original: WikiText-103 corpus (Merity et al., 2016) using the transformer language model of Baevski & Auli (2018).
Revised: WikiText-103 corpus (Merity et al., 2016) using the transformer language model of Baevski & Auli (2018).

Original: We use this model because of its prominent role in recent language modeling developments (Khandelwal et al., 2020; Press et al., 2021).
Revised: We use this model because of its prominent role in recent language modeling developments (Khandelwal et al., 2020; Press et al., 2021).

Original: The training set is about 103 million tokens from English Wikipedia (half a gigabyte).
Revised: The training set is about 103 million tokens from English Wikipedia (half a gigabyte).

Original: The model has 16 transformer layers of dimension 1024 , with 8 heads, and a feedforward inner dimension of 4096 .
Revised: The model has 16 transformer layers of dimension 1024 , with 8 heads, and a feedforward inner dimension of 4096 .

Original: This model ties the word embedding and softmax matrices (Press & Wolf, 2017; Inan et al., 2017).
Revised: This model ties the word embedding and softmax matrices (Press & Wolf, 2017; Inan et al., 2017).

Original: In our experiments, other than varying the position method and training subsequence length, we modify no other hyperparameters, including the random seed and number of training epochs (205).
Revised: In our experiments, other than varying the position method and training subsequence length, we modify no other hyperparameters, including the random seed and number of training epochs (205).

Original: 2. M EASURING E XTRAPOLATION
Revised: 2. M EASURING E XTRAPOLATION

Original: Sinusoidal Position Embeddings Sinusoidal position embeddings (Vaswani et al. (2017); §3.5) are constant, non-learned vectors that are added to token embeddings on input to the ﬁrst layer of the transformer.
Revised: Sinusoidal Position Embeddings Sinusoidal position embeddings (Vaswani et al., 2017; §3.5) are constant, non-learned vectors that are added to token embeddings on input to the ﬁrst layer of the transformer.

Original: They are frequently used in transformer language modeling (Baevski & Auli, 2018; Lewis et al., 2021) and machine translation (Vaswani et al., 2017; Ott et al., 2018) models.
Revised: They are frequently used in transformer language modeling (Baevski & Auli, 2018; Lewis et al., 2021) and machine translation (Vaswani et al., 2017; Ott et al., 2018) models.

Original: We ﬁrst consider the unmodiﬁed model of Baevski & Auli (2018), which uses sinusoidal position embeddings, and train it on L = 512 tokens; we then run inference with it on the validation set on L + k tokens, with k ranging from 0 to 15,000.
Revised: We ﬁrst consider the unmodiﬁed model of Baevski & Auli (2018), which uses sinusoidal position embeddings, and train it on L = 512 tokens; we then run inference with it on the validation set on L + k tokens, with k ranging from 0 to 15,000.

Original: Figure 1 (left) and the corresponding Table(in the appendix) show that while the model improves perplexity up to k = 20 , performance stops improving and stays steady from k = 20 to k = 50 and then begins degrading.
Revised: Figure 1 (left) and the corresponding Table(in the appendix) show that while the model improves perplexity up to k = 20 , performance stops improving and stays steady from k = 20 to k = 50 and then begins degrading.

Original: Similar results are obtained for a model trained with L = 1024 tokens (Figure 1 (right) and Table 3 in the appendix).
Revised: Similar results are obtained for a model trained with L = 1024 tokens (Figure 1 (right) and Table 3 in the appendix).

Original: That model improves for up to L valid =
Revised: That model improves for up to L valid =

Original: L + 50 tokens, after which performance declines.
Revised: L + 50 tokens, after which performance declines.

Original: Rotary Position Embeddings
Revised: Rotary Position Embeddings

Original: The rotary method was introduced by Su et al.
Revised: The rotary method was introduced by Su et al.

Original: and has recently been popularized by the open source GPT-3 (Brown et al., 2020) implementation GPTJ (Wang & Komatsuzaki, 2021).
Revised: and has recently been popularized by the open source GPT-3 (Brown et al., 2020) implementation GPTJ (Wang & Komatsuzaki, 2021).

Original: Instead of adding sinusoidal embeddings at the bottom of the transformer, they multiply the keys and queries of every attention layer by sinusoidal embeddings.
Revised: Instead of adding sinusoidal embeddings at the bottom of the transformer, they multiply the keys and queries of every attention layer by sinusoidal embeddings.

Original: Unlike the sinusoidal or learned positional embedding approach, the rotary method injects position information into the model at every layer, not just at the initial one.
Revised: Unlike the sinusoidal or learned positional embedding approach, the rotary method injects position information into the model at every layer, not just at the initial one.

Original: In addition, it adds no position information to the values of the self-attention sublayer.
Revised: In addition, it adds no position information to the values of the self-attention sublayer.

Original: The output of a self-attention sublayer is a linearly transformed, weighted sum of the input value vectors; therefore, by not inserting position information into the values, the outputs of each transformer-layer contain no explicit position information.
Revised: The output of a self-attention sublayer is a linearly transformed, weighted sum of the input value vectors; therefore, by not inserting position information into the values, the outputs of each transformer-layer contain no explicit position information.

Original: We suspect that this segregation of position information may be beneﬁcial for extrapolation, and we draw inspiration from it in the design of our method (§3).
Revised: We suspect that this segregation of position information may be beneﬁcial for extrapolation, and we draw inspiration from it in the design of our method (§3).

Original: We apply the rotary position embedding method to our Baevski & Auli baseline.
Revised: We apply the rotary position embedding method to our Baevski & Auli baseline.

Original: The perplexity results (Figure 1 and Appendix Tables 2 and 3) are better than the sinusoidal approach: the model with L = 

Revised: The perplexity results (Figure 1 and Appendix Tables 2 and 3) are better than the sinusoidal approach: the model with L =

Original: ( L = 1024 ) improves perplexity with up to k = 200 ( k = 100 ) more tokens than it saw during training, but this comes at the cost of slower training and inference (Figure 2).
Revised: ( L = 1024 ) improves perplexity with up to k = 200 ( k = 100 ) more tokens than it saw during training, but this comes at the cost of slower training and inference (Figure 2).

Original: T5 Bias Though most models use trained or sinusoidal position embeddings, the T5 model of Raffel et al.
Revised: T5 Bias Though most models use trained or sinusoidal position embeddings, the T5 model of Raffel et al.

Original: (2020) uses a relative position method (Shaw et al., 2018;
Revised: (2020) uses a relative position method (Shaw et al., 2018;

Original: Huang et al., 2019) that adds no position information to word embeddings (as in the previous method).
Revised: Huang et al., 2019) that adds no position information to word embeddings (as in the previous method).

Original: Instead, it modiﬁes the way attention values are computed.
Revised: Instead, it modiﬁes the way attention values are computed.

Original: We refer to this as the “T5 bias” method.
Revised: We refer to this as the “T5 bias” method.

Original: 6 To compute attention values in the unmodiﬁed transformer, we compute the dot product of every query with every relevant key and then softmax these attention values.
Revised: 6 To compute attention values in the unmodiﬁed transformer, we compute the dot product of every query with every relevant key and then softmax these attention values.

Original: In this method, we compute the attention values as before, but then we add a learned, shared bias to each query-key score that is dependent on just the distance between the query and key.
Revised: In this method, we compute the attention values as before, but then we add a learned, shared bias to each query-key score that is dependent on just the distance between the query and key.

Original: Therefore, all query-key scores where the query and key distance are zero (i.e., the query and key represent the same token) get a speciﬁc learned bias, all scores where the query and key are one word away get a different learned bias, and so on, up to a certain point, from where multiple different distances share the same learned bias (which might be beneﬁcial for extrapolation).
Revised: Therefore, all query-key scores where the query and key distance are zero (i.e., the query and key represent the same token) get a speciﬁc learned bias, all scores where the query and key are one word away get a different learned bias, and so on, up to a certain point, from where multiple different distances share the same learned bias (which might be beneﬁcial for extrapolation).

Original: As in the rotary method, the T5 bias injects position information into the model at every layer and integrates no explicit position information into the self-attention value vectors.
Revised: As in the rotary method, the T5 bias injects position information into the model at every layer and integrates no explicit position information into the self-attention value vectors.

Original: Raffel et al.
Revised: Raffel et al.

Original: (2020) propose that the T5 bias may allow extrapolation, but they did not report experiments testing this.
Revised: (2020) propose that the T5 bias may allow extrapolation, but they did not report experiments testing this.

Original: Here, we show that the T5 bias does allow language models to extrapolate.
Revised: Here, we show that the T5 bias does allow language models to extrapolate.

Original: We do this by again modifying the Baevski & Auli model, this time to insert the T5 bias into it.
Revised: We do this by again modifying the Baevski & Auli model, this time to insert the T5 bias into it.

Original: As Figure 1 shows, the T5 bias improves perplexity with longer sequences than the ones it was trained on, i.e., k = 600 ( k = 800 ) extra tokens for a model trained on L = 

Revised: As Figure 1 shows, the T5 bias improves perplexity with longer sequences than the ones it was trained on, i.e., k = 600 ( k = 800 ) extra tokens for a model trained on L =

Original: ( L = 1024 ) input tokens.
Revised: ( L = 1024 ) input tokens.

Original: Unfortunately, this impressive performance comes at a cost: training is at least twice as slow as with the sinusoidal model.
Revised: Unfortunately, this impressive performance comes at a cost: training is at least twice as slow as with the sinusoidal model.

Original: Therefore, this model’s extrapolation ability provides no efﬁciency advantage.
Revised: Therefore, this model’s extrapolation ability provides no efﬁciency advantage.

Original: For example, to do inference on 1024 tokens, we could either train the sinusoidal model with L = 1024 or train the T5 bias model on L = 512 tokens and extrapolate to 1024 for inference.
Revised: For example, to do inference on 1024 tokens, we could either train the sinusoidal model with L = 1024 or train the T5 bias model on L = 512 tokens and extrapolate to 1024 for inference.

Original: However, the L = 1024 sinusoidal model runs at 28.5k words per second (WPS), while the L = 512 T5 bias model runs at 14.4k WPS (Appendix Table 1), so there is no speedup when training on shorter sequences with this method.
Revised: However, the L = 1024 sinusoidal model runs at 28.5k words per second (WPS), while the L = 512 T5 bias model runs at 14.4k WPS (Appendix Table 1), so there is no speedup when training on shorter sequences with this method.

Original: linked to the ofﬁcial repository of Su et al roformer we running our with the could be optimized optimization would extrapolation q 1 kq 2 k
Revised: linked to the ofﬁcial repository of Su et al roformer we running our with the could be optimized optimization would extrapolation q 1 kq 2 k

Original: 1 q 2 kq 3 k 1 q 3 k 2 q 3 kq 4 k 1 q 4 k 2 q 4 k 3 q 4 kq 5 k 1 q 5 k 2 q 5 k 3 q 5 k 4 q 5 k 1 1  1   1+ m
Revised: 1 q 2 kq 3 k 1 q 3 k 2 q 3 kq 4 k 1 q 4 k 2 q 4 k 3 q 4 kq 5 k 1 q 5 k 2 q 5 k 3 q 5 k 4 q 5 k 1 1  1   1+ m

Original:  A TTENTION WITH L INEAR B IASES (AL I B I )
Revised:  A TTENTION WITH L INEAR B IASES (AL I B I )

Original: In the transformer model of Vaswani et al.
Revised: In the transformer model of Vaswani et al.

Original: (2017), position embeddings are added to the word embeddings at the bottom of the network.
Revised: (2017), position embeddings are added to the word embeddings at the bottom of the network.

Original: For an input subsequence of length L , the attention sublayer computes the attention scores for the i th query q
Revised: For an input subsequence of length L , the attention sublayer computes the attention scores for the i th query q

Original: i ∈ R 1 × d , ( 1 ≤ i ≤ L ) in each head, given the ﬁrst i keys K ∈ R
Revised: i ∈ R 1 × d , ( 1 ≤ i ≤ L ) in each head, given the ﬁrst i keys K ∈ R

Original: i × d , where d is the head dimension:
Revised: i × d , where d is the head dimension:

Original: [Equation]
Revised: [Equation]

Original: These attention scores are then multiplied by the values to return the output of the attention sublayer.
Revised: These attention scores are then multiplied by the values to return the output of the attention sublayer.

Original: When using ALiBi, we do not add position embeddings at any point in the network.
Revised: When using ALiBi, we do not add position embeddings at any point in the network.

Original: The only modiﬁcation we apply is after the query-key dot product, where we add a static, non-learned bias 10 :
Revised: The only modiﬁcation we apply is after the query-key dot product, where we add a static, non-learned bias:

Original: [Equation]
Revised: [Equation]

Original: where scalar m is a head-speciﬁc slope ﬁxed before training.
Revised: where scalar m is a head-speciﬁc slope ﬁxed before training.

Original: Figure 3 offers a visualization.
Revised: Figure 3 offers a visualization.

Original: For our models with 8 heads, the slopes that we used are the geometric sequence: 12 1 , 12 2 , ..., 12 8 .
Revised: For our models with 8 heads, the slopes that we used are the geometric sequence: 12 1 , 12 2 , ..., 12 8 .

Original: For models that require 16 heads, we interpolate those 8 slopes by geometrically averaging every consecutive pair, resulting in the geometric sequence that starts at 1 √ 2 and has the ratio of 1 √ 2 : 12 0 .
Revised: For models that require 16 heads, we interpolate those 8 slopes by geometrically averaging every consecutive pair, resulting in the geometric sequence that starts at 1 √ 2 and has the ratio of 1 √ 2 : 12 0 .

Original: In general, for n heads, our set of slopes is the geometric sequence that starts at 2 − 8 n and uses that same value as its ratio.
Revised: In general, for n heads, our set of slopes is the geometric sequence that starts at 2 − 8 n and uses that same value as its ratio.

Original: In §4, we observe that this set of slopes works on a wide variety of text domains and model sizes.
Revised: In §4, we observe that this set of slopes works on a wide variety of text domains and model sizes.

Original: Therefore, we do not believe that it is necessary to tune these slope values every time a new model is trained on a new dataset.
Revised: Therefore, we do not believe that it is necessary to tune these slope values every time a new model is trained on a new dataset.

Original: This makes our method similar to the sinusoidal approach, where the hyperparameters (the start and end of the geometric progression of wavelengths) were set once by Vaswani et al.
Revised: This makes our method similar to the sinusoidal approach, where the hyperparameters (the start and end of the geometric progression of wavelengths) were set once by Vaswani et al.

Original: and then reused in different models of different sizes on different datasets.
Revised: and then reused in different models of different sizes on different datasets.

Original: ALiBi has an inductive bias towards recency; it penalizes attention scores between distant query-key pairs, with the penalty increasing as the distance between a key and a query grows.
Revised: ALiBi has an inductive bias towards recency; it penalizes attention scores between distant query-key pairs, with the penalty increasing as the distance between a key and a query grows.

Original: The different heads increase their penalties at different rates, depending on the slope magnitude.
Revised: The different heads increase their penalties at different rates, depending on the slope magnitude.

Original: We initially experimented with making the slopes trainable, but this did not yield strong extrapolation results.
Revised: We initially experimented with making the slopes trainable, but this did not yield strong extrapolation results.

Original: A brief manual exploration of around ten slope sets led us to discover the set of slopes that we ﬁnally picked.
Revised: A brief manual exploration of around ten slope sets led us to discover the set of slopes that we ﬁnally picked.

Original: Our main insight from this exploration is that the slope sets that work best are those with slopes in the (0 , 1) range, with the slopes’ density increasing as we get closer to 0 .
Revised: Our main insight from this exploration is that the slope sets that work best are those with slopes in the (0 , 1) range, with the slopes’ density increasing as we get closer to 0 .

Original: We also found our method to be robust to slope choice.
Revised: We also found our method to be robust to slope choice.

Original: Even randomly sampling from the exponential distribution worked well in some cases (although that method had high variance).
Revised: Even randomly sampling from the exponential distribution worked well in some cases (although that method had high variance).

Original: Since ALiBi is a relative position method, we add position information at every layer to the keys and queries but not to the values, as is done in the T5 bias and rotary methods.
Revised: Since ALiBi is a relative position method, we add position information at every layer to the keys and queries but not to the values, as is done in the T5 bias and rotary methods.

Original: We hypothesize that these properties might be beneﬁcial for extrapolation.
Revised: We hypothesize that these properties might be beneﬁcial for extrapolation.

Original: Implementation.
Revised: Implementation.

Original: ALiBi is easy to implement, with all changes accomplished in a few lines of code.
Revised: ALiBi is easy to implement, with all changes accomplished in a few lines of code.

Original: We implement it by modifying the mask matrix by adding the linear biases to it (in practice, when training a transformer LM, query q i attends only to keys 1 to i , this is implemented by adding a mask matrix to the query-key dot product before the softmax operation is applied).
Revised: We implement it by modifying the mask matrix by adding the linear biases to it (in practice, when training a transformer LM, query q i attends only to keys 1 to i ; this is implemented by adding a mask matrix to the query-key dot product before the softmax operation is applied).

Original: This means that there is no runtime penalty when using our method since we add no operations to the network.
Revised: This means that there is no runtime penalty when using our method since we add no operations to the network.

Original: Compared to the sinusoidal model trained on the same input lengths, AliBi incurs a memory increase (up to 100MB in some of our experiments): in the unmodiﬁed transformer, the mask is of size L × L ; when using ALiBi, the mask is a slightly larger n × L × L (where n is the number of heads) since the linear biases added for each head uses a different slope.
Revised: Compared to the sinusoidal model trained on the same input lengths, AliBi incurs a memory increase (up to 100MB in some of our experiments): in the unmodiﬁed transformer, the mask is of size L × L ; when using ALiBi, the mask is a slightly larger n × L × L (where n is the number of heads) since the linear biases added for each head uses a different slope.

Original: But, as we show, ALiBi enables training on much smaller sequences while still achieving (and occasionally surpassing) results obtained using sinusoidal embeddings on longer sequences, which saves multiple gigabytes of memory.
Revised: But, as we show, ALiBi enables training on much smaller sequences while still achieving (and occasionally surpassing) results obtained using sinusoidal embeddings on longer sequences, which saves multiple gigabytes of memory.

Original:  R ESULTS
Revised:  R ESULTS

Original: We ﬁrst show that on WikiText103 ALiBi is efﬁcient and enables training models with short input subsequences that outperform strong baselines even when the ALiBi models extrapolate to more than six times the number of tokens that they were trained on.
Revised: We ﬁrst show that on WikiText103 ALiBi is efﬁcient and enables training models with short input subsequences that outperform strong baselines even when the ALiBi models extrapolate to more than six times the number of tokens that they were trained on.

Original: We then take the same hyperparameters for our method (the set of slopes) that worked on WikiText-103 and show that – with no modiﬁcation – they provide strong results on a dataset in a very different domain: books.
Revised: We then take the same hyperparameters for our method (the set of slopes) that worked on WikiText-103 and show that – with no modiﬁcation – they provide strong results on a dataset in a very different domain: books.

Original: Finally, we show that a 1.3B parameter model trained with AliBi on a much larger (461 GB) dataset with much more compute provides a superior alternative to the sinusoidal method since it achieves similar perplexity scores while running faster and using less memory (since it is trained on shorter inputs).
Revised: Finally, we show that a 1.3B parameter model trained with AliBi on a much larger (461 GB) dataset with much more compute provides a superior alternative to the sinusoidal method since it achieves similar perplexity scores while running faster and using less memory (since it is trained on shorter inputs).

Original: While multiple alternatives to the position methods presented in Vaswani et al.
Revised: While multiple alternatives to the position methods presented in Vaswani et al.

Original: (2017) have been proposed, few have been adopted in large (1B or more parameter) LMs since that setting is much more challenging than the smaller scale experiments.
Revised: (2017) have been proposed, few have been adopted in large (1B or more parameter) LMs since that setting is much more challenging than the smaller scale experiments.

Original: GPT-3 and Jurassic-1 (Lieber et al., 2021) use the learned position embedding method from Vaswani et al., and GPT-J uses the rotary method.
Revised: GPT-3 and Jurassic-1 (Lieber et al., 2021) use the learned position embedding method from Vaswani et al., and GPT-J uses the rotary method.

Original: Our results on the 1.3B parameter model show our method’s ability to generalize to larger models, dataset sizes and training durations without retuning the hyperparameter.
Revised: Our results on the 1.3B parameter model show our method’s ability to generalize to larger models, dataset sizes and training durations without retuning the hyperparameter.

Original: 4. R ESULTS ON W IKI T EXT -103 AND T ORONTO B OOK C ORPUS
Revised: 4. R ESULTS ON W IKI T EXT -103 AND T ORONTO B OOK C ORPUS

Original: [Figure]
Revised: [Figure]

Original: We ﬁrst develop our method on the WikiText-103 corpus (Merity et al., 2016), replacing the sinusoidal position embeddings in the language model of Baevski & Auli (2018) with ALiBi.
Revised: We ﬁrst develop our method on the WikiText-103 corpus (Merity et al., 2016), replacing the sinusoidal position embeddings in the language model of Baevski & Auli (2018) with ALiBi.

Original: Figure 4 (and the corresponding Appendix Table 5) show our results for models trained with varying numbers of input subsequence tokens ( L ), extrapolating to longer subsequence lengths on the validation dataset.
Revised: Figure 4 (and the corresponding Appendix Table 5) show our results for models trained with varying numbers of input subsequence tokens ( L ), extrapolating to longer subsequence lengths on the validation dataset.

Original: Our ﬁrst observation is that, without extrapolation, for every L , our models outperform those using the sinusoidal method, sometimes by a signiﬁcant amount.
Revised: Our ﬁrst observation is that, without extrapolation, for every L , our models outperform those using the sinusoidal method, sometimes by a signiﬁcant amount.

Original: For example, the Baevski &
Revised: For example, the Baevski &

Original: Auli model achieves 18.67 ± 0.24 (std. dev.)
Revised: Auli model achieves 18.67 ± 0.24 (std. dev.)

Original: perplexity when trained with L = 3072 input tokens, but our L = 3072 model achieves 17.60 perplexity (when both models evaluate with L valid = 3072).
Revised: perplexity when trained with L = 3072 input tokens, but our L = 3072 model achieves 17.60 perplexity (when both models evaluate with L valid = 3072).

Original: Our second observation is that all of our models can extrapolate, and they obtain improved perplexity scores when handling more tokens than they observed during training.
Revised: Our second observation is that all of our models can extrapolate, and they obtain improved perplexity scores when handling more tokens than they observed during training.

Original: For example, our model trained on 512 tokens (which achieves 19.73 perplexity when evaluating subsequences of length 512 in the development set) achieves a perplexity score of 18.40 on the development set when extrapolating to subsequences of length 3072.
Revised: For example, our model trained on 512 tokens (which achieves 19.73 perplexity when evaluating subsequences of length 512 in the development set) achieves a perplexity score of 18.40 on the development set when extrapolating to subsequences of length 3072.

Original: Surprisingly, this surpasses the score that the L = 3072 sinusoidal model obtains on the development set by a statistically signiﬁcant margin.
Revised: Surprisingly, this surpasses the score that the L = 3072 sinusoidal model obtains on the development set by a statistically signiﬁcant margin.

Original: Note that all our models trained on L = 512 to L = 2048 outperform the sinusoidal baseline trained on L = 3072 when extrapolating to L valid = 3072 even though those models all take much less time to train since they train on shorter subsequences (Appendix Figure 8 compares training speed to perplexity for these models)!
Revised: Note that all our models trained on L = 512 to L = 2048 outperform the sinusoidal baseline trained on L = 3072 when extrapolating to L valid = 3072 even though those models all take much less time to train since they train on shorter subsequences (Appendix Figure 8 compares training speed to perplexity for these models)!

Original: The L = 512 model is 1.84 times faster to train and yet still outperforms the L = 3072 sinusoidal model when extrapolating to L valid = 3072 .
Revised: The L = 512 model is 1.84 times faster to train and yet still outperforms the L = 3072 sinusoidal model when extrapolating to L valid = 3072 .

Original: In addition, training the L = 3072 sinusoidal model requires a GPU with more than 16 GB of memory to ﬁt the large attention matrices, which our L = 512 outperforms even though it can be trained on a GPU with much less memory due to much smaller attention matrices.
Revised: In addition, training the L = 3072 sinusoidal model requires a GPU with more than 16 GB of memory to ﬁt the large attention matrices, which our L = 512 outperforms even though it can be trained on a GPU with much less memory due to much smaller attention matrices.

Original: Additionally, Table 5 (in the appendix) also shows that, for L s of 1024 and 3072, our method performs better than the rotary and T5 bias models even when L valid =
Revised: Additionally, Table 5 (in the appendix) also shows that, for L s of 1024 and 3072, our method performs better than the rotary and T5 bias models even when L valid =

Original: L (i.e., no extrapolation is occurring).
Revised: L (i.e., no extrapolation is occurring).

Original: Figure 1 (and the corresponding Appendix Tables 2 and 3) more broadly explore our method vs the other position methods.
Revised: Figure 1 (and the corresponding Appendix Tables 2 and 3) more broadly explore our method vs. the other position methods.

Original: They show that the T5 bias (the best of the baselines) improves perplexity until L valid is around 2 L , but on the WikiText-103 dataset our method continually improves perplexity until at least around 3 L , with the L = 512 model improving perplexity even when L valid exceeds 12k tokens.
Revised: They show that the T5 bias (the best of the baselines) improves perplexity until L valid is around 2 L , but on the WikiText-103 dataset our method continually improves perplexity until at least around 3 L , with the L = 512 model improving perplexity even when L valid exceeds 12k tokens.

Original: Even when unable to improve perplexity given longer sequences,
Revised: Even when unable to improve perplexity given longer sequences,

Original: ALiBi always maintains strong performance as more tokens are added.
Revised: ALiBi always maintains strong performance as more tokens are added.

Original: Appendix Table 6 shows that our results on the validation set also transfer to the test set of WikiText103.
Revised: Appendix Table 6 shows that our results on the validation set also transfer to the test set of WikiText103.

Original: Currently, almost all models that present results on WikiText-103 use sliding window evaluation (deﬁned in §B) to compute perplexities.
Revised: Currently, almost all models that present results on WikiText-103 use sliding window evaluation (deﬁned in §B) to compute perplexities.

Original: We apply that method to our (and to the sinusoidal, rotary and T5 bias) models in Appendix Table 7.
Revised: We apply that method to our (and to the sinusoidal, rotary and T5 bias) models in Appendix Table 7.

Original: We ﬁnd that our L = 3072 model surpasses the performance of Transformer-XL (Dai et al., 2019), the Sandwich (Press et al., 2020), and Shortformer (Press et al., 2021) models.
Revised: We ﬁnd that our L = 3072 model surpasses the performance of Transformer-XL (Dai et al., 2019), the Sandwich (Press et al., 2020), and Shortformer (Press et al., 2021) models.

Original: Our results are similar to the ones obtained with staged training (Press et al., 2021) but fall short of results obtained by Routing Transformer (Roy et al., 2020) and
Revised: Our results are similar to the ones obtained with staged training (Press et al., 2021) but fall short of results obtained by Routing Transformer (Roy et al., 2020) and

Original: kNN-LM (Khandelwal et al., 2020).
Revised: kNN-LM (Khandelwal et al., 2020).

Original: The methods used in those models are orthogonal to ours, and we hypothesize that combining them with ours might lead to even larger performance increases.
Revised: The methods used in those models are orthogonal to ours, and we hypothesize that combining them with ours might lead to even larger performance increases.

Original: After developing our method on WikiText-103, in Appendix Section A.3, we run one set of experiments on a different domain (books) using a similar model architecture and without modifying any of the ALiBi hyperparameters (the slopes) and show that our results fully transfer to this new domain.
Revised: After developing our method on WikiText-103, in Appendix Section A.3, we run one set of experiments on a different domain (books) using a similar model architecture and without modifying any of the ALiBi hyperparameters (the slopes) and show that our results fully transfer to this new domain.

Original: Our models are able to both surpass the sinusoidal baseline when not extrapolating while also outperforming it when extrapolating to longer sequences.
Revised: Our models are able to both surpass the sinusoidal baseline when not extrapolating while also outperforming it when extrapolating to longer sequences.

Original: 4. R ESULTS ON THE CC100+R O BERT A C ORPUS
Revised: 4. R ESULTS ON THE CC100+R O BERT A C ORPUS

Original: Our ﬁnal set of experiments investigates whether ALiBi transfers to a larger model trained with a larger computational budget on a larger dataset than the ones we previously used.
Revised: Our ﬁnal set of experiments investigates whether ALiBi transfers to a larger model trained with a larger computational budget on a larger dataset than the ones we previously used.

Original: We show that our method achieves strong results in this more challenging setting, obtaining similar performance to the sinusoidal baseline while using signiﬁcantly less memory, since we train on shorter subsequences.
Revised: We show that our method achieves strong results in this more challenging setting, obtaining similar performance to the sinusoidal baseline while using signiﬁcantly less memory, since we train on shorter subsequences.

Original: The dataset we choose is a combination of the datasets used to train the RoBERTa (Liu et al., 2019) implementation of BERT (Devlin et al., 2019) and the English part of the CC-100 corpus introduced in Conneau et al.
Revised: The dataset we choose is a combination of the datasets used to train the RoBERTa (Liu et al., 2019) implementation of BERT (Devlin et al., 2019) and the English part of the CC-100 corpus introduced in Conneau et al.

Original: (2020), for a total of 461 GB.
Revised: (2020), for a total of 461 GB.

Original: The RoBERTa training corpus—i.e., the
Revised: The RoBERTa training corpus—i.e., the

Original: Toronto Book Corpus (Zhu et al., 2015), English Wikipedia, CC-News (Nagel, 2016), OpenWebText (Gokaslan & Cohen, 2019) and Stories (Trinh & Le, 2018))—is 161 gigabytes, and the English part of the CC-100 corpus is 300 gigabytes.
Revised: Toronto Book Corpus (Zhu et al., 2015), English Wikipedia, CC-News (Nagel, 2016), OpenWebText (Gokaslan & Cohen, 2019) and Stories (Trinh & Le, 2018))—is 161 gigabytes, and the English part of the CC-100 corpus is 300 gigabytes.

Original: The validation set contains 649K tokens.
Revised: The validation set contains 649K tokens.

Original: Our models for this dataset have 25 transformer layers with 16 heads and a dimension of 2048, with an 8192 hidden dimension of the feedforward sublayers.
Revised: Our models for this dataset have 25 transformer layers with 16 heads and a dimension of 2048, with an 8192 hidden dimension of the feedforward sublayers.

Original: These models have 1.3B parameters.
Revised: These models have 1.3B parameters.

Original: We train our models for one epoch, which is 50k updates on 128 V100 GPUs.
Revised: We train our models for one epoch, which is 50k updates on 128 V100 GPUs.

Original: In Figure 5 (left), we compare the validation perplexity for L valid = 1024 throughout the training process for an ALiBi model trained with L = 512 compared to the sinusoidal model trained with
Revised: In Figure 5 (left), we compare the validation perplexity for L valid = 1024 throughout the training process for an ALiBi model trained with L = 512 compared to the sinusoidal model trained with

Original: L = 1024.
Revised: L = 1024.

Original: Since our model is trained on shorter sequences, it is 7% faster and uses 1.6 GB less
Revised: Since our model is trained on shorter sequences, it is 7% faster and uses 1.6 GB less

Original: [Figure]
Revised: [Figure]

Original: memory.
Revised: memory.

Original: We halt training of the sinusoidal baseline when our model reaches the end of its training (one epoch).
Revised: We halt training of the sinusoidal baseline when our model reaches the end of its training (one epoch).

Original: At that time, our model is just 0.06 perplexity away from the baseline even though it was trained on sequences that are half the length of those the baseline used and requires less memory.
Revised: At that time, our model is just 0.06 perplexity away from the baseline even though it was trained on sequences that are half the length of those the baseline used and requires less memory.

Original: In Figure 5 (right), results become even more impressive, showing that our model trained on L = 1024 outperforms by 0.09 perplexity the sinusoidal model trained on L = 2048 (when evaluating with L valid = 2048) even though our model uses 3.1 GB less memory.
Revised: In Figure 5 (right), results become even more impressive, showing that our model trained on L = 1024 outperforms by 0.09 perplexity the sinusoidal model trained on L = 2048 (when evaluating with L valid = 2048) even though our model uses 3.1 GB less memory.

Original: Our model maintains a lead in perplexity over the sinusoidal model during the entire training process.
Revised: Our model maintains a lead in perplexity over the sinusoidal model during the entire training process.

Original: By sampling ﬁve evenly distributed points across the training process, we compute that our L = 1024 model reaches a given perplexity value, on average, 11% faster than the sinusoidal model does.
Revised: By sampling ﬁve evenly distributed points across the training process, we compute that our L = 1024 model reaches a given perplexity value, on average, 11% faster than the sinusoidal model does.

Original: Since our models in these comparisons use much less memory, they allow for stacking more layers, which would further improve performance (with negligible, if any, runtime cost).
Revised: Since our models in these comparisons use much less memory, they allow for stacking more layers, which would further improve performance (with negligible, if any, runtime cost).

Original: To keep our experiments as straightforward as possible, however, we do not add layers to our models.
Revised: To keep our experiments as straightforward as possible, however, we do not add layers to our models.

Original: Appendix Table 12 presents additional results comparing our models to the sinusoidal baseline when both are trained on the same L , showing that ALiBi performs similarly to the sinusoidal baseline when not extrapolating.
Revised: Appendix Table 12 presents additional results comparing our models to the sinusoidal baseline when both are trained on the same L , showing that ALiBi performs similarly to the sinusoidal baseline when not extrapolating.

Original: This contrasts with the results presented on the smaller datasets, where
Revised: This contrasts with the results presented on the smaller datasets, where

Original: ALiBi consistently outperforms other position methods even when not extrapolating, suggesting that ALiBi’s inductive bias provides additional beneﬁts for lower-resource language modeling.
Revised: ALiBi consistently outperforms other position methods even when not extrapolating, suggesting that ALiBi’s inductive bias provides additional beneﬁts for lower-resource language modeling.

Original: [Figure]
Revised: [Figure]

Original: Figure 6 shows that our models trained on L = 512 and L = 1024 achieve the best results when extrapolating to about double the tokens that they were trained on.
Revised: Figure 6 shows that our models trained on L = 512 and L = 1024 achieve the best results when extrapolating to about double the tokens that they were trained on.

Original: Speciﬁcally, the L = 512 model (that obtains 9.79 perplexity when L valid = 512) achieves its best score (9.3) when extrapolating to
Revised: Speciﬁcally, the L = 512 model (that obtains 9.79 perplexity when L valid = 512) achieves its best score (9.3) when extrapolating to

Original: 1012 tokens, and the L = 1024 model (that obtains 9.16 perplexity when L valid = 1024) achieves its best score (8.9) when extrapolating to 2024 tokens.
Revised: 1012 tokens, and the L = 1024 model (that obtains 9.16 perplexity when L valid = 1024) achieves its best score (8.9) when extrapolating to 2024 tokens.

Original: One possible explanation is that the subsequences the model observes during training are up to L tokens long.
Revised: One possible explanation is that the subsequences the model observes during training are up to L tokens long.

Original: When performing inference on subsequences of length 2 L , half of the subsequences the model consumes are as long as the examples seen during training.
Revised: When performing inference on subsequences of length 2 L , half of the subsequences the model consumes are as long as the examples seen during training.

Original: When inference is performed on subsequences of length 2 L + 1 or longer, less than half of the predictions the model makes are on subsequences of lengths seen during training, and that might degrade performance.
Revised: When inference is performed on subsequences of length 2 L + 1 or longer, less than half of the predictions the model makes are on subsequences of lengths seen during training, and that might degrade performance.

Original: The sinusoidal model cannot extrapolate at all in this setting, with its performance degrading for both the L = 512 and 1024 models as soon as one token more than L is added during evaluation.
Revised: The sinusoidal model cannot extrapolate at all in this setting, with its performance degrading for both the L = 512 and 1024 models as soon as one token more than L is added during evaluation.

Original: In Appendix B, we ﬁnd that ALiBi edge over sinusoidal embeddings is largely explained by its improved avoidance of the early token curse.
Revised: In Appendix B, we ﬁnd that ALiBi’s edge over sinusoidal embeddings is largely explained by its improved avoidance of the early token curse.

Original: We posit that future work building on ALiBi might achieve further gains by more efﬁciently exploiting longer histories.
Revised: We posit that future work building on ALiBi might achieve further gains by more efﬁciently exploiting longer histories.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In parallel with our work, Wennberg & Henter (2021) introduce a relative position method that, like our method, adds a bias to attention scores that is a function of the distance between the key and query elements.
Revised: In parallel with our work, Wennberg & Henter (2021) introduce a relative position method that, like our method, adds a bias to attention scores that is a function of the distance between the key and query elements.

Original: Unlike our ALiBi method, which uses a non-learned linear function, their method uses a radial-basis function, with multiple trainable parameters (in our experiments, this led to a slight decrease in runtime).
Revised: Unlike our ALiBi method, which uses a non-learned linear function, their method uses a radial-basis function, with multiple trainable parameters (in our experiments, this led to a slight decrease in runtime).

Original: In addition, they present experiments on text classiﬁcation, not on language modeling.
Revised: In addition, they present experiments on text classiﬁcation, not on language modeling.

Original: They do not explore extrapolation.
Revised: They do not explore extrapolation.

Original: The Distance Aware Transformer (Wu et al., 2021)
Revised: The Distance Aware Transformer (Wu et al., 2021)

Original: multiplies attention scores by a bias that is a function of the distance between the key and query.
Revised: multiplies attention scores by a bias that is a function of the distance between the key and query.

Original: This function uses a different, learned parameter in every head.
Revised: This function uses a different, learned parameter in every head.

Original: They show results only on text classiﬁcation.
Revised: They show results only on text classiﬁcation.

Original: In our experiments (not presented), multiplying attention scores by the bias (instead of adding, as in ALiBi) degraded performance.
Revised: In our experiments (not presented), multiplying attention scores by the bias (instead of adding, as in ALiBi) degraded performance.

Original: Transformer-XL(Dai et
Revised: Transformer-XL

Original:  al., 2019) presented a language model that uses a cache and can attend to more tokens during inference than it was trained on (by increasing the length of the cache).
Revised: (Dai et al., 2019) presented a language model that uses a cache and can attend to more tokens during inference than it was trained on (by increasing the length of the cache).

Original: However, this work presents results only where output length is limited to the L (the training length), and their relative position method is very slow (Press et al., 2021).
Revised: However, this work presents results only where output length is limited to the L (the training length), and their relative position method is very slow (Press et al., 2021).

Original: The Longformer (Beltagy et al., 2020) adapts models trained on shorter sequences to document-level tasks.
Revised: The Longformer (Beltagy et al., 2020) adapts models trained on shorter sequences to document-level tasks.

Original: However, to achieve this they had to partially train their models on longer sequences.
Revised: However, to achieve this they had to partially train their models on longer sequences.

Original: Our ALiBi method enables extrapolation without any additional training on longer sequences.
Revised: Our ALiBi method enables extrapolation without any additional training on longer sequences.

Original: To our knowledge, extrapolation has not been previously explored in transformer language modeling, but it has been investigated previously and concurrently with transformers on other tasks, such as machine translation (Rosendahl et al., 2019; Neishi & Yoshinaga, 2019; Newman et al., 2020;
Revised: To our knowledge, extrapolation has not been previously explored in transformer language modeling, but it has been investigated previously and concurrently with transformers on other tasks, such as machine translation (Rosendahl et al., 2019; Neishi & Yoshinaga, 2019; Newman et al., 2020;

Original: Kiyono et al., 2021), sequence-to-sequence models trained on an artiﬁcial dataset (Hupkes et al., 2020), pretrained sequence-to-sequence models tested on arithmetic tasks (Nogueira et al., 2021,
Revised: Kiyono et al., 2021), sequence-to-sequence models trained on an artiﬁcial dataset (Hupkes et al., 2020), pretrained sequence-to-sequence models tested on arithmetic tasks (Nogueira et al., 2021,

Original: Appendix C), models trained with reinforcement learning (Lampinen et al., 2021), image, speech recognition, and machine translation models (Likhomanenko et al., 2021), and protein structure prediction (Jumper et al., 2021, Appendix 1.5).
Revised: Appendix C), models trained with reinforcement learning (Lampinen et al., 2021), image, speech recognition, and machine translation models (Likhomanenko et al., 2021), and protein structure prediction (Jumper et al., 2021, Appendix 1.5).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We showed that the sinusoidal position embedding approach does not enable transformers to extrapolate to inputs longer than the ones they were trained on.
Revised: We showed that the sinusoidal position embedding approach does not enable transformers to extrapolate to inputs longer than the ones they were trained on.

Original: We then established that extrapolation in transformers can be enabled by just changing the position method.
Revised: We then established that extrapolation in transformers can be enabled by just changing the position method.

Original: We showed that our ALiBi method offers an extremely simple replacement for existing position approaches and allow models to extrapolate.
Revised: We showed that our ALiBi method offers an extremely simple replacement for existing position approaches and allow models to extrapolate.

Original: In addition, when not extrapolating, our method achieves either better perplexity than the sinusoidal method (in models smaller than 1B parameters, trained on less data) or similar perplexity (in larger, billion parameter models trained on much more data).
Revised: In addition, when not extrapolating, our method achieves either better perplexity than the sinusoidal method (in models smaller than 1B parameters, trained on less data) or similar perplexity (in larger, billion parameter models trained on much more data).

Original: ALiBi is simple to implement and does not slow down runtime or require extra parameters (but does occasionally require a negligible amount of extra memory).
Revised: ALiBi is simple to implement and does not slow down runtime or require extra parameters (but does occasionally require a negligible amount of extra memory).

Original: Using our method, we sped up the training of a 1.3 billion parameter model evaluated on the same input sequence length as GPT-3 (2048).
Revised: Using our method, we sped up the training of a 1.3 billion parameter model evaluated on the same input sequence length as GPT-3 (2048).

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We thank Tim Dettmers, Gabriel Ilharco, Jungo Kasai, Hao Peng, Sewon Min, Soﬁa Serrano, Sam
Revised: We thank Tim Dettmers, Gabriel Ilharco, Jungo Kasai, Hao Peng, Sewon Min, Soﬁa Serrano, Sam

Original: Shleifer, Luke Zettlemoyer, Julian Michael, Nikolaos Pappas,and Yizhong Wang for their valuable feedback and fruitful discussions.
Revised: Shleifer, Luke Zettlemoyer, Julian Michael, Nikolaos Pappas, Yizhong Wang, and the anonymous reviewers for their valuable feedback and fruitful discussions.

Original: I NTRODUCTION
Revised: I NTRODUCTION

Original: The training speed of transformer LMs gets slower as the input subsequence length L increases.
Revised: The training speed of transformer LMs gets slower as the input subsequence length L increases.

Original: Figure 7 visualizes this.
Revised: Figure 7 visualizes this.

Original: [Figure]
Revised: [Figure]

Original: P
Revised: P

Original: [Figure]
Revised: [Figure]

Revised: ec

Original: [Figure] Sinusoidal
Revised: [Figure] [Figure]

Original: ALiBi
Revised: ALiBi

Original: Table 1 contains the runtimes and memory use statistics for models using the various position methods discussed in this work.
Revised: Table 1 contains the runtimes and memory use statistics for models using the various position methods discussed in this work.

Original: [Table]
Revised: [Table]

Original: 28.5k 82.1k 15.
Revised: 28.5k 82.1k 15.3 GB

Original: [Table]
Revised: [Table]

Original: 26.0k 77.8k 19. 15.3k 42.4k 15. 20.0k 43.4k 17.
Revised: 26.0k 77.8k 19.2 GB 15.3k 42.4k 15.1 GB 20.0k 43.4k 17.8 GB

Original: Rotary 17.7k 39.4k 22. 11.5k 29.5k 17. 14.4k 21.8k 16.
Revised: Rotary 17.7k 39.4k 22.8 GB 11.5k 29.5k 17.8 GB 14.4k 21.8k 16.9 GB

Original: T5 Bias 13.0k 20.2k 20. 4.3k 4.9k 15. 28.3k 85.8k 15.
Revised: T5 Bias 13.0k 20.2k 20.9 GB 4.3k 4.9k 15.9 GB 28.3k 85.8k 15.3 GB

Original: ALiBi 25.8k 76.4k 19. 15.5k 42.2k15.
Revised: ALiBi 25.8k 76.4k 19.3 GB 15.5k 42.2k

Original:  Tables 2, 3, and 4 show the perplexity and 
Revised: [Equation] GB Tables 2, 3, and 4 show the perplexity and runtime of models using the sinusoidal, rotary T5 bias, and ALiBi position methods when extrapolating to sequences longer than the ones they were trained on. The models used in these tables were trained on L = 512 , 1024 and 3072 tokens.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Figure]
Revised: [Figure]

Original: AL

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: P e r p l e ( it ) ( ← →
Revised: P e r p l e ( it ) ( ← →

Original: [Figure]
Revised: [Figure]

Original: ALiBi
Revised: ALiBi

Original: Figure 8 depicts a cross section of Figure 4, showing our models with different train lengths and the sinusoidal baseline, all evaluated on L valid = 3072 tokens.
Revised: Figure 8 depicts a cross section of Figure 4, showing our models with different train lengths and the sinusoidal baseline, all evaluated on L valid = 3072 tokens.

Original: We observe that all our models with 512 ≤ L < 3072 are faster to train than the sinusoidal model with L = 3072, but they all achieve greater perplexity scores on the validation set.
Revised: We observe that all our models with 512 ≤ L < 3072 are faster to train than the sinusoidal model with L = 3072, but they all achieve greater perplexity scores on the validation set.

Original: Our model with L = 3072 trains just as fast as the sinusoidal one but bests its score by more than one perplexity point; (the standard deviation for the the sinusoidal model with L = 3072 is 0.24).
Revised: Our model with L = 3072 trains just as fast as the sinusoidal one but bests its score by more than one perplexity point; (the standard deviation for the the sinusoidal model with L = 3072 is 0.24).

Original: Table 5 shows the perplexity values obtained when 8 different ALiBi models, trained on L values between 64 and 3072, extrapolating to L valid values longer than the ones they were trained on.
Revised: Table 5 shows the perplexity values obtained when 8 different ALiBi models, trained on L values between 64 and 3072, extrapolating to L valid values longer than the ones they were trained on.

Original: In addition, we present results for the sinusoidal, rotary and T5 bias models, with L valid =
Revised: In addition, we present results for the sinusoidal, rotary and T5 bias models, with L valid =

Original: L .
Revised: L .

Original: [Table]
Revised: [Table]

Original: Sinusoidal ∗ 28.
Revised: Sinusoidal ∗ 28.

Original: Rotary ∗ - - - 20.
Revised: Rotary ∗ - - - 20.

Original: T5 Bias ∗ - - - 19.
Revised: T5 Bias ∗ - - - 19.

Original: Table 6 compares ALiBi to the sinusoidal, rotary and T5 bias baselines on the test set of WikiText103, and Table 7 compares ALiBi to the current state of the art models on that test set.
Revised: Table 6 compares ALiBi to the sinusoidal, rotary and T5 bias baselines on the test set of WikiText103, and Table 7 compares ALiBi to the current state of the art models on that test set.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Khandelwal et al., 2020) 247M 15.
Revised: Khandelwal et al., 2020) 247M 15.

Original: Sinusoidal, L =247M 17.
Revised: Sinusoidal, L =247M 17.

Original: Rotary, L =247M 17.
Revised: Rotary, L =247M 17.

Original: T5 Bias, L =247M 17.
Revised: T5 Bias, L =247M 17.

Original: A L i B i L = 512, L valid =247M 18.
Revised: A L i B i L = 512, L valid =247M 18.

Original: L = 3072, L valid =247M 16.
Revised: L = 3072, L valid =247M 16.

Original: A. R ESULTS ON THE T ORONTO B OOK C ORPUS
Revised: A. R ESULTS ON THE T ORONTO B OOK C ORPUS

Original: To ensure that our results are not speciﬁc to the WikiText-103 corpus, we next apply our model and the baselines to a different domain while using a similar model architecture and the same ALiBi slopes as those used in the previous subsection.
Revised: To ensure that our results are not speciﬁc to the WikiText-103 corpus, we next apply our model and the baselines to a different domain while using a similar model architecture and the same ALiBi slopes as those used in the previous subsection.

Original: We emphasize that our set of slopes was chosen by running experiments on the WikiText-103 corpus, and here we apply that set of slopes to a model trained on a very different text domain.
Revised: We emphasize that our set of slopes was chosen by running experiments on the WikiText-103 corpus, and here we apply that set of slopes to a model trained on a very different text domain.

Original: Throughout the entire process of developing this method, we ran only one set of experiments on this domain using the previously selected set of slopes.
Revised: Throughout the entire process of developing this method, we ran only one set of experiments on this domain using the previously selected set of slopes.

Original: Speciﬁcally, we use the Toronto BooksCorpus (Zhu et al., 2015), which has been used to train
Revised: Speciﬁcally, we use the Toronto BooksCorpus (Zhu et al., 2015), which has been used to train

Original: BERT (Devlin et al., 2019) (in conjuction with the English Wikipedia).
Revised: BERT (Devlin et al., 2019) (in conjuction with the English Wikipedia).

Original: The corpus is about 700M tokens (2.9 GB).
Revised: The corpus is about 700M tokens (2.9 GB).

Original: We use the same train/validation/test split as Khandelwal et al.
Revised: We use the same train/validation/test split as Khandelwal et al.

Original: (2020) and their tokenization, which uses BERT’s vocabulary of 29K byte-pair encodings.
Revised: (2020) and their tokenization, which uses BERT’s vocabulary of 29K byte-pair encodings.

Original: Since the vocabulary is much smaller than
Revised: Since the vocabulary is much smaller than

Original: WikiText-103’s, we replace the adaptive word embedding and softmax of Baevski & Auli (2018) with a tied word embedding and softmax matrix (Press & Wolf, 2017; Inan et al., 2017).
Revised: WikiText-103’s, we replace the adaptive word embedding and softmax of Baevski & Auli (2018) with a tied word embedding and softmax matrix (Press & Wolf, 2017; Inan et al., 2017).

Original: Our results in Figure 9 (and Table 8) replicate our success on the WikiText-103 dataset.
Revised: Our results in Figure 9 (and Table 8) replicate our success on the WikiText-103 dataset.

Original: Our model surpasses the sinusoidal baseline when trained on the same amount of input tokens ( L ) and, in
Revised: Our model surpasses the sinusoidal baseline when trained on the same amount of input tokens ( L ) and, in

Original: Validation Input Length (L valid ) 13.
Revised: Validation Input Length (L valid ) 13.

Original: P e r p l e x it y
Revised: P e r p l e x it y

Original: ALiBi Extrap lating n T r nt B kC rpus
Revised: ALiBi Extrap lating n T r nt B kC rpus

Original: L =
Revised: L =

Original: Sinus idal← L =
Revised: Sinus idal← L =

Original: L =
Revised: L =

Original: Sinus idal← L =
Revised: Sinus idal← L =

Original: L =
Revised: L =

Original: Sinus idal← L =
Revised: Sinus idal← L =

Original: Our models extrapolate to longer sequence lengths and outperform the sinusoidal baseline even when trained on much shorter sequences.
Revised: Our models extrapolate to longer sequence lengths and outperform the sinusoidal baseline even when trained on much shorter sequences.

Original: addition, our model is able to extrapolate to longer sequences at inference.
Revised: addition, our model is able to extrapolate to longer sequences at inference.

Original: This occurs even though our set of slopes was not tuned on this dataset.
Revised: This occurs even though our set of slopes was not tuned on this dataset.

Original: This result establishes the generality of ALiBi and the particular set of slopes we found and suggests that they may be used on different text domains without further hyperparameter tuning.
Revised: This result establishes the generality of ALiBi and the particular set of slopes we found and suggests that they may be used on different text domains without further hyperparameter tuning.

Original: Tables 9 and 10 present the perplexities for our ALiBi models, the baselines, and the current state of the art on the Toronto BookCorpus validation and test sets.
Revised: Tables 9 and 10 present the perplexities for our ALiBi models, the baselines, and the current state of the art on the Toronto BookCorpus validation and test sets.

Original: Our results here mirror our results on
Revised: Our results here mirror our results on

Original: WikiText-103: we improve over the sinusoidal baseline even when AliBi is trained on fewer tokens.
Revised: WikiText-103: we improve over the sinusoidal baseline even when AliBi is trained on fewer tokens.

Original: those
Revised: those

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Param.
Revised: Param.

Original: ↓
Revised: ↓

Original: Valid ↓
Revised: Valid ↓

Original: Test ↓
Revised: Test ↓

Original: [Table]
Revised: [Table]

Original: , L =247M 14.
Revised: 247M 14.

Revised: [Table]

Original: A L i B i
Revised: A L i B i

Original: [Equation]
Revised: [Equation]

Original: train = 512, L valid =247M 13.
Revised: train = [Table] L valid = [Table]

Original:  L train = 3072, L valid =247M 13.
Revised: 247M 13. L train = 3072, L valid =247M 13.

Original: A. R ESULTS ON THE CC100+R O BERT A C ORPUS
Revised: A. R ESULTS ON THE CC100+R O BERT A C ORPUS

Original: Table 11 compares our 1.3 billion parameter ALiBi models when extrapolating to two times the number of tokens that they were trained on.
Revised: Table 11 compares our 1.3 billion parameter ALiBi models when extrapolating to two times the number of tokens that they were trained on.

Original: We use the sinusoidal model as our baseline, and train it for the same amount of time as we train the ALiBi model that we compare it to (and so since our
Revised: We use the sinusoidal model as our baseline, and train it for the same amount of time as we train the ALiBi model that we compare it to (and so since our

Original: ALiBi models run faster in this setting, the sinusoidal models complete less updates).
Revised: ALiBi models run faster in this setting, the sinusoidal models complete less updates).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Sinusoidal, L train =26.2 GB 46.7k 5.5k9.
Revised: Sinusoidal, L train =26.2 GB 46.7k 5.5k

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: ALiBi, L train =24.6 GB 50.0k 5.5k 9.
Revised: ALiBi, L train =24.6 GB 50.0k 5.5k 9.

Original: Sinusoidal, L train =
Revised: Sinusoidal, L train =

Original: 29.3 GB 44.2k 5.9k9.
Revised: 29.3 GB 44.2k 5.9k9.

Original: ALiBi, L train =26.2 GB 50.0k 5.9k8.
Revised: ALiBi, L train =26.2 GB 50.0k 5.9k8.

Original: Table 12 compares our 1.3 billion parameter ALiBi models to the sinusoidal baselines, with and without extrapolation, with all models completing 50,000 updates
Revised: Table 12 compares our 1.3 billion parameter ALiBi models to the sinusoidal baselines, with and without extrapolation, with all models completing 50,000 updates

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B A NALYSIS
Revised: B A NALYSIS

Original: In this section we investigate why ALiBi works so effectively.
Revised: In this section we investigate why ALiBi works so effectively.

Original: We ﬁnd that ALiBi’s decrease in perplexity when given longer sequences is largely explained by its improved avoidance of the early token curse.
Revised: We ﬁnd that ALiBi’s decrease in perplexity when given longer sequences is largely explained by its improved avoidance of the early token curse.

Original: We hypothesize that future work building on ALiBi might achieve further gains by more efﬁciently exploiting longer histories.
Revised: We hypothesize that future work building on ALiBi might achieve further gains by more efﬁciently exploiting longer histories [Table]

Original: The big gray cat sat on the mat
Revised: The big gray cat sat on the mat

Original: Sliding Window Inference As mentioned in Section 2, nonoverlapping inference is commonly used to evaluate sequences longer than L (the number of tokens in each training subsequence).
Revised: Sliding Window Inference As mentioned in Section 2, nonoverlapping inference is commonly used to evaluate sequences longer than L (the number of tokens in each training subsequence).

Original: An alternative is to use a sliding window during evaluation (Baevski & Auli, 2018).
Revised: An alternative is to use a sliding window during evaluation (Baevski & Auli, 2018).

Original: A stride S is picked between 1 and L − 1 , and the window is advanced by S tokens after each forward pass.
Revised: A stride S is picked between 1 and L − 1 , and the window is advanced by S tokens after each forward pass.

Original: This means that L − S tokens from the previous subsequence are re-encoded, and only S new tokens are output.
Revised: This means that L − S tokens from the previous subsequence are re-encoded, and only S new tokens are output.

Original: The advantage is that all outputs in each subsequence after the ﬁrst have at least L − S previous tokens to condition on.
Revised: The advantage is that all outputs in each subsequence after the ﬁrst have at least L − S previous tokens to condition on.

Original: However, since tokens must be re-encoded multiple times, this approach is much slower than the nonoverlapping one.
Revised: However, since tokens must be re-encoded multiple times, this approach is much slower than the nonoverlapping one.

Original: When S = 1 , we output one token
Revised: When S = 1 , we output one token

Original: every inference pass, each using the maximal context window that the model can handle; however, this is the slowest approach.
Revised: every inference pass, each using the maximal context window that the model can handle; however, this is the slowest approach.

Original: Figure 10 is a visualization of the nonoverlapping and sliding window evaluation approaches.
Revised: Figure 10 is a visualization of the nonoverlapping and sliding window evaluation approaches.

Original: We use sliding window inference as a tool to analyze our models, but we note that it is normally prohibitively slow in practice (Press et al., 2021).
Revised: We use sliding window inference as a tool to analyze our models, but we note that it is normally prohibitively slow in practice (Press et al., 2021).

Original: Early Token Curse Splitting an evaluation set into subsequences means that predictions occuring early in each subsequence cannot access many previous context tokens (appearing at the end of the previous subsequence).
Revised: Early Token Curse Splitting an evaluation set into subsequences means that predictions occuring early in each subsequence cannot access many previous context tokens (appearing at the end of the previous subsequence).

Original: The result, referred to as the early token curse (Press et al., 2021), increases (i.e., degrades) perplexity scores.
Revised: The result, referred to as the early token curse (Press et al., 2021), increases (i.e., degrades) perplexity scores.

Original: A workaround is to evaluate the model using a sliding window, giving each prediction more context.
Revised: A workaround is to evaluate the model using a sliding window, giving each prediction more context.

Original: This solution is slow since it requires many more forward passes of the model.
Revised: This solution is slow since it requires many more forward passes of the model.

Original: B. E XTRAPOLATION REDUCES THE EARLY TOKEN CURSE
Revised: B. E XTRAPOLATION REDUCES THE EARLY TOKEN CURSE

Original: We presented results showing that our ALiBi method (and, to a lesser extent, the T5 bias) allow
Revised: We presented results showing that our ALiBi method (and, to a lesser extent, the T5 bias) allows

Original: LMs to extrapolate during inference.
Revised: LMs to extrapolate during inference.

Original: Two reasons could explain why these methods enable LMs to achieve better perplexity given longer input subsequences:
Revised: Two reasons could explain why these methods enable LMs to achieve better perplexity given longer input subsequences:

Original: Performance improves because the models can use longer contexts to make more accurate predictions.
Revised: Performance improves because the models can use longer contexts to make more accurate predictions.

Original: For example, the average article length in the WikiText-103 corpus is about 3600 tokens; therefore, if a model trained on L = 512 tokens extrapolates to L valid = 3072 tokens during inference and achieves better results, that might be because it can spot patterns occurring across more than 512 tokens.
Revised: For example, the average article length in the WikiText-103 corpus is about 3600 tokens; therefore, if a model trained on L = 512 tokens extrapolates to L valid = 3072 tokens during inference and achieves better results, that might be because it can spot patterns occurring across more than 512 tokens.

Original: Performance improves because longer input sequences mean the early token curse is reduced.
Revised: Performance improves because longer input sequences mean the early token curse is reduced.

Original: For example, during nonoverlapping evaluation on sequences of length L valid = 1000, 10% of predictions have 100 tokens of context or less.
Revised: For example, during nonoverlapping evaluation on sequences of length L valid = 1000, 10% of predictions have 100 tokens of context or less.

Original: If we rerun nonoverlapping evaluation on that model with L valid = 2000 tokens, now only 5% of predictions have
Revised: If we rerun nonoverlapping evaluation on that model with L valid = 2000 tokens, now only 5% of predictions have

Original: Nonoverlapping
Revised: Nonoverlapping

Original: tokens of context or less.
Revised: tokens of context or less.

Original: So, by simply being able to handle longer sequences, a model can substantially reduce the early token curse and improve performance.
Revised: So, by simply being able to handle longer sequences, a model can substantially reduce the early token curse and improve performance.

Original: To better understand what might be occurring, we re-evaluate the development set of WikiTextwith our models and the sinusoidal baseline with L = 512 , 1024 , 3072 .
Revised: To better understand what might be occurring, we re-evaluate the development set of WikiTextwith our models and the sinusoidal baseline with L = 512 , 1024 , 3072 .

Original: However, this time we use sliding window evaluation with a stride of S = 1 , meaning that we move the sliding window just one token after every inference pass, giving each prediction the maximum number of context tokens that the model can use.
Revised: However, this time we use sliding window evaluation with a stride of S = 1 , meaning that we move the sliding window just one token after every inference pass, giving each prediction the maximum number of context tokens that the model can use.

Original: [Figure]
Revised: [Figure]

Original: ALiBi and
Revised: ALiBi and

Original: [Figure]
Revised: [Figure]

Original: (oidal Evalua)ion (w
Revised: (oidal Evalua)ion (w

Original: [Figure]
Revised: [Figure]

Original: Sliding Window) on WikiTextSinusoidal, L =
Revised: Sliding Window) on WikiTextSinusoidal, L =

Original: Sinusoidal, L = Sinusoidal, L =
Revised: Sinusoidal, L = Sinusoidal, L =

Original: ALiBi, L = ALiBi, L =
Revised: ALiBi, L = ALiBi, L =

Original: ALiBi, L =
Revised: ALiBi, L =

Original: The results are shown in Figure 11 and in the corresponding Tables 13 (sinusoidal) and 14 (ALiBi).
Revised: The results are shown in Figure 11 and in the corresponding Tables 13 (sinusoidal) and 15 (ALiBi).

Original: Unsurprisingly, for the sinusoidal model, as in §2, increasing L valid causes an explosion in perplexity even when using sliding window evaluation.
Revised: Unsurprisingly, for the sinusoidal model, as in §2, increasing L valid causes an explosion in perplexity even when using sliding window evaluation.

Original: Our ALiBi models cannot improve perplexity when looking at longer sequences in this setting, but they keep perplexity ﬂat when L valid increases.
Revised: Our ALiBi models cannot improve perplexity when looking at longer sequences in this setting, but they keep perplexity ﬂat when L valid increases.

Original: This leads us to believe that our perplexity improvement when increasing L valid and using nonoverlapping evaluation is caused by explanation 2, not explanation 1.
Revised: This leads us to believe that our perplexity improvement when increasing L valid and using nonoverlapping evaluation is caused by explanation 2, not explanation 1.

Original: Because sliding window evaluation provides long context windows for every prediction made, it curtails the early token curse.
Revised: Because sliding window evaluation provides long context windows for every prediction made, it curtails the early token curse.

Original: In this setting, ALiBi’s performance remains ﬂat when L valid increases, leading us to hypothesize that the gains seen while increasing L valid in §4 were the result of larger L valid values mitigating the early token curse.
Revised: In this setting, ALiBi’s performance remains ﬂat when L valid increases, leading us to hypothesize that the gains seen while increasing L valid in §4 were the result of larger L valid values mitigating the early token curse.

Original: Our ALiBi results mirror what occurs in the model using the T5 bias: when using sliding window evaluation, perplexity remains relatively ﬂat when evaluating longer sequences (see Table 15).
Revised: Our ALiBi results mirror what occurs in the model using the T5 bias: when using sliding window evaluation, perplexity remains relatively ﬂat when evaluating longer sequences (see Table 14).

Original: Our analysis reveals that when L valid > L , ALiBi might not be using contexts longer than the ones it was trained on.
Revised: Our analysis reveals that when L valid > L , ALiBi might not be using contexts longer than the ones it was trained on.

Original: This highlights a research direction that could be pursued in future work.
Revised: This highlights a research direction that could be pursued in future work.

Original: These ﬁndings do not lessen the value of ALiBi.
Revised: These ﬁndings do not lessen the value of ALiBi.

Original: When L valid =
Revised: When L valid =

Original: L , ALiBi achieves either superior or similar results to the sinusoidal method and other alternatives even though it is simpler and requires no learned parameters.
Revised: L , ALiBi achieves either superior or similar results to the sinusoidal method and other alternatives even though it is simpler and requires no learned parameters.

Original: When evaluating L valid > L tokens, even if ALiBi does not attend to more than L tokens, it yields better results than the other alternatives that can be used in this case, i.e., standard nonoverlapping inference (which is cheap, but does not perform as well) and the more accurate sliding window approach (which is very slow).
Revised: When evaluating L valid > L tokens, even if ALiBi does not attend to more than L tokens, it yields better results than the other alternatives that can be used in this case, i.e., standard nonoverlapping inference (which is cheap, but does not perform as well) and the more accurate sliding window approach (which is very slow).

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]