Paper: On the Long Range Abilities of Transformers (ICLR-2024)

Reviewer comment (Theory):
The paper contains many assumptions that are inaccurate or unjustified. For example:  
> The lack of effectiveness of transformers in this setting [long context]  

implies that transformers are ineffective, in general, on long inputs. I am not sure that this is the case - I agree that transformers could be improved, but the main problem seems to me that most transformers cannot even process long inputs. When the input *does* fit in the transformer's context window, most results that I've seen are not that bad. In other words, claiming that "transformers are ineffective in long context" requires some experiments and justification, to show exactly what the authors mean, instead of relying on one paper who said so a few years ago.

Author response:
Thank you for the comprehensive review and the constructive feedback.

> The paper contains many assumptions that are inaccurate or unjustified

We disagree that the paper contains many inaccurate or unjustified assumptions. It appears the reviewer may not be fully cognizant of recent developments in the domain, which our paper discusses in the introduction and background sections. 

> For example: The lack of effectiveness of transformers in this setting [long context].. implies that transformers are ineffective, in general, on long inputs. I am not sure that this is the case - I agree that transformers could be improved, but the main problem seems to me that most transformers cannot even process long inputs. When the input does fit in the transformer's context window, most results that I've seen are not that ba,  In other words, claiming that "transformers are ineffective in long context" requires some experiments and justification, to show exactly what the authors mean, instead of relying on one paper who said so a few years ago. 


The statement that the reviewer extracted from our manuscript ("transformers are ineffective in long context") is immediately substantiated in the next two paragraphs. 

It seems that the reviewer may not be aware of critical and well-established results in long-range modeling, which are also discussed in our introduction and background. Specifically, state-space layers such as S4 [S4] (ICLR 2022), DSS [DSS] (NeurIPS 2022), H3 [H3] (ICLR 2023), S5 (ICLR 2023), and other variants such as Gated state-space [GSS] and Liquid state-space [LSS] are known to achieve significantly better results than transformers on long-range tasks (please see full reference list in the shared response).

 Notably, these layers outperform the best-known transformer variants by 20 points on the Long Range Arena Benchmark, a key benchmark in the field, and they do so with fewer parameters and even when compared to transformers trained on the entire sequence length. For more details, please refer to Table 1. 

Additionally, studies such as [H3, GSS] have shown that the interleaving of such layers enhances performance on long-range language modeling tasks. This trend is also evident in RL [RL1,RL2], speech processing [S1,S2,S3], and other areas [F1,HY2] (see citations in the shared response).

Reviewer comment (Theory):
Another inaccurate and unjustified claim appears in Section 3:  
> the observed sub-optimal performance of transformers on long-range tasks does not arise necessarily from issues of optimization or expressiveness, which are inherent to the architecture. Rather, it is a matter of generalization  

This claim assumes that transformers are sub-optimal (compared to what?) on long-range tasks (which?), and claims that the issue is generalization. This is a bit of a vacuous claim, since "Generalization" can contain anything. If the model reaches a zero training loss but is worse at test time, is that generalization? Is that the kind of "generalization" that the authors refer to?

Author response:
> This claim assumes that transformers are sub-optimal (compared to what?) on long-range tasks (which?), and claims that the issue is generalization. This is a bit of a vacuous claim, since "Generalization" can contain anything. If the model reaches a zero training loss but is worse at test time, is that generalization? Is that the kind of "generalization" that the authors refer to? “compared to what? ..” :

Please refer to the first sentence in the section you cited: “This research starts with a systematic attempt to understand the reasons behind the inferior performance of transformers in long-range modeling, **compared to state-space layers and long convolution layers**”.  

*Additionally, please consider the third paragraph in the introduction:** “Motivated by recent advances in deep long sequence modeling, we delve into the question of **why long-range layers such as state-space layers** (Gu et al., 2021b;a; Gupta et al., 2022a) and long convolutions (Li et al., 2022; Fu et al., 2023) perform well on the LRA benchmark and other long-range tasks”. 

Moreover, the first sentence in the abstract states “.. transformer architectures exhibit sub-optimal performance on long-range tasks **compared to recent layers that are specifically designed for this purpose**. In this work, drawing inspiration from **key attributes of long-range layers, such as state-space layers, linear RNN layers, and global convolution layers”.** 

For more discussion regarding these questions, please see point #4 in the shared response.


> Long-range tasks (which?) : 

The main focus of the paper is the Long Range Arena benchmark. However, the findings are also applicable to other long-range tasks, including long-range NLP (as seen in [H3,DSS]), RL [RL1,RL2], speech processing [S1,S2,S3], and more [F1,S4,HY2] (please see citations in the shared response).

> “Generalization" can contain anything”

Generalization is a well-defined term widely used in machine learning papers. It encompasses various issues such as the absence of regularization, inadequate inductive bias, or a small training dataset. It does not pertain to everything. For instance, it does not cover multiple aspects of expressiveness or optimization (despite being related). 

> “If the model reaches a zero training loss but is worse at test time, is that generalization?” 

Yes, this is an example of a lack of generalization.

> Is that the kind of "generalization" that the authors refer to? 

Yes.

Reviewer comment (Theory):
The paper contains further inaccurate claims like:  
> Long-range dependencies are often associated with optimization issues, such as exploding and vanishing gradient problems.  

which were not justified and references were not provided. I agree that in the past RNN era, this was indeed the common belief. But can this claim be said on "long-range dependencies" in general?

Author response:
> “Long-range dependencies are often associated with optimization issues, such as exploding and vanishing gradient problems.” which were not justified and references were not provided. I agree that in the past RNN era, this was indeed the common belief. But can this claim be said on "long-range dependencies" in general?

We do not agree that the sentence is unjustified. For example, a few lines after the one the reviewer cited, it is written, "Moreover, in the insightful work of Orvieto et al. (2023), it was empirically demonstrated that vanishing and exploding gradient issues on the LRA benchmarks arise from a high number of non-linear operations between distant tokens".

Reviewer comment (Evaluation):
There is a potential contradiction in the claims in Section 3, where it is stated that:  
> on the LRA benchmarks including the validation set, large transformers can achieve near 100% accuracy,  

but then also stated that:  
> the existing transformers underfit the long-range data.  

If not contradictory, these statements require justification, numbers, and references.

Author response:
> “the existing transformers underfit the long-range data”

Thanks for catching this! This is a very confusing typo which is fixed in the revised manuscript.

Reviewer comment (Theory):
Motivation – the authors motivate their solution with the assumption that:  
> smooth and exponentially decaying kernels are associated with a long-range inductive bias  

This "axiom" isn't clear – I am not sure what exactly the authors mean by "smoothness". And in general, why? Why would *decaying* any kind of attention improve any long-range modeling? I was not convinced that these are indeed related to the problem/solution.

Author response:
> Motivation - the authors motivate their solution with the assumption that “smooth and exponentially decaying kernels are associated with a long-range inductive bias”. This "axiom" isn't clear - I am not sure what exactly the authors mean by "smoothness"

 Please refer to paragraph 4 in section 4 (method), which is one line above the line you cited: “Following this, Fu et al. (2023) **empirically demonstrate that smoothness can be a powerful tool for kernel regularization. This smoothness** can be achieved by reducing the size of kernel weights in the time domain using a squashing operator, which enforces sparsity, leading to smoothness in the frequency domain. Moreover, layers such as EMA or linear-state space layers are naturally smooth in their design, as can be seen in Fig. 1(b).”

Reviewer comment (Writing):
Clarity – there are many parts of the paper which are unclear. For example, the introduction says:  
> We discern two simple yet significant conditions (i) an exponential decaying positional structure, and (ii) a regularized smooth global operator.  
> furthermore (iv) present an SL-chunk variation  

At this point in reading the paper, this is meaningless to me.

Author response:
> “furthermore (iv) present an SL-chunk variation” .. At this point in reading the paper, this is meaningless to me.

Using chunk variants of transformers is a standard for long-range evaluations. Basically, it employs the same approach, but in chunks. We introduce the main idea a few lines before the one the reviewer cites: “Building upon these two principles, we introduce Local and Smooth Attention …”

Reviewer comment (Presentation):
The section on identifying common design choices of prior work feels a bit handy wavy. Perhaps could be better presented in a table.

Author response:
Thank you for the suggestion. We have added a comparison table and some discussion to Appendix E (in the revised manuscript).

Reviewer comment (Theory):
There is an error either in Equation 3 or Figure 2. They transpose the softmax and average pooling. My guess is the error is in Equation 3.

Author response:
Thanks for pointing out this typo. We fixed Figure 2 in the revised manuscript.

Reviewer comment (Theory):
There seems to be only empirical evidence about why seemingly "unintuitive" methods work. Some theoretical justification would make the paper stronger.

Author response:
Please refer to Theorem 1 in Appendix B of the revised manuscript (see point #1 in the shared response). Transformers are shown to have the capacity to express several long-range layers, such as S4. This indicates that the performance gap in such tasks is not a problem of expressiveness, but rather, a problem of generalization.

Regarding the 'unintuitive' method, while we agree that smoothness and locality may seem counterintuitive, we believe that the rationale for trying such a method becomes apparent after explaining that we took inspiration from long-range layers (see Figure 1 and the first paragraphs in Section 4).

Additionally, regarding why smoothness and locality enhance the model's ability to manage long-range dependencies, we hypothesize that in long-range tasks across common modalities such as images, text, and speech, long-range dependencies manifest through a hierarchical combination of local and smooth dependencies. As argued in Section 3, the difficulty in capturing these long-range dependencies stems from a generalization challenge, which can be mitigated by an appropriate inductive bias. Hence, such bias, when suitably applied, can be highly effective.

Reviewer comment (Theory):
What is meant by "necessary conditions for achieving success in long-range tasks"? Is it meant in the strict mathematical sense? If so, I would expect to see a proof along the lines of success implies these conditions.

Author response:
In this context, 'necessary conditions' is not used as a mathematical term. Due to the confusion, we have replaced "necessary conditions" with "desired properties."

Reviewer comment (Experiments):
Since this mechanism works in a causal manner, how well does it perform in decoding tasks like language modeling or translation?

Author response:
Indeed, the current model is causal, but it can be adapted for non-causal tasks by converting it into a bidirectional model. This can be achieved by applying each layer twice, once in each direction, similar to other global convolution layers (like S4). Alternatively, we can adopt an approach similar to that in [A1], which extends Alibi to bidirectional by carefully defining the values in the upper part of the attention matrix.

[A1] Lee, Minchul, Kijong Han, and Myeong Cheol Shin. "LittleBird: Efficient Faster & Longer Transformer for Question Answering." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

Reviewer comment (Experiments):
While experiments that study the impact of context length and sample size are intriguing, interpreting the results is challenging without comparing the patterns to any other baseline. For instance, in the experiment where you restrict the context window size to examine LaS's dependence on long-range dependencies, do we have insights into how a standard transformer might be influenced by a reduced context window size?

Author response:
The results of the full-length vanilla transformer are presented in the first line of Table 1, and they are relatively similar to those of simple local attention (which we have added to Table 1 in the revised manuscript). As can be seen, in all tasks except Pathfinder, LaS-chunk performs much better than both of the requested baselines. Therefore, we omit them from the ablations in Figures 3 and 4 for clarity. Furthermore, Figure 4 was created to strengthen the claims of Section 3, and therefore is not directly related to abating the LaS mechanism.

Reviewer comment (Theory):
I believe the arguments about expressivity limitations and optimization challenges might be wrong.

Author response:
We recognize that certain edge cases may arise from the model's lack of expressiveness or optimization challenges. Examples include an inherent inability to express long-range dependencies or optimization issues such as vanishing gradients that prevent the model from learning these dependencies.

However, according to the evidence provided by Theorem 1, the transformer architecture does not suffer from a lack of expressiveness, as it is more expressive than other layers that perform well on this dataset. Additionally, optimization is unlikely to be problematic since the generalization improves as the number of training samples increases (refer to Figure 4 and the detailed analysis in Section 3).
 
Finally, please note that we have refined our claims based on this discussion (“we claim that the primary factor behind the suboptimal performance of transformers on long-range tasks is **probably** the lack of generalization”, “Rather, it is **likely** a matter of generalization”).

Reviewer comment (Evaluation):
For instance, a model can fit the training data without necessarily capturing long-range dependencies, simply by leveraging spurious features.

Author response:
We recognize that certain edge cases may arise from the model's lack of expressiveness or optimization challenges. Examples include an inherent inability to express long-range dependencies or optimization issues such as vanishing gradients that prevent the model from learning these dependencies.

However, according to the evidence provided by Theorem 1, the transformer architecture does not suffer from a lack of expressiveness, as it is more expressive than other layers that perform well on this dataset. Additionally, optimization is unlikely to be problematic since the generalization improves as the number of training samples increases (refer to Figure 4 and the detailed analysis in Section 3).
 
Finally, please note that we have refined our claims based on this discussion (“we claim that the primary factor behind the suboptimal performance of transformers on long-range tasks is **probably** the lack of generalization”, “Rather, it is **likely** a matter of generalization”).

Reviewer comment (Theory):
For example, I believe there are elements within the transformer block where some kind of diminishing effect might transpire as the context window becomes exceedingly long.

Author response:
Our solution shows that generalization is possible even without modifying these elements. However, it may be possible that ablating parts of the transformer would also be beneficial.

Reviewer comment (Presentation):
Could you explain how one should read Figure 1? What is the x-axis? What is the y-axis?

Author response:
Given a kernel k:= (k_1, k_2, .. k_L) of length L, axes y represent the value of the kernel (k_i) on position i. Axes x is the position i.

Reviewer comment (Theory):
Can you elaborate a bit more on the arguments about smoothness and its relationship to improving the model's capability to handle long-range dependencies?

Author response:
First, it is important to recall that we utilize these fundamentals by drawing inspiration from global convolutional layers. For instance, Figure 1 presents relatively local and smooth kernels.

Second, regarding how smoothness and locality enhance the model's ability to manage long-range dependencies, we hypothesize that in long-range tasks across common modalities such as images, text, and speech, long-range dependencies manifest through a hierarchical combination of local and smooth dependencies. Since we argue that the problem of capturing long-range dependencies reflects as a problem of generalization, such inductive bias can be very effective.

Reviewer comment (Theory):
Same question as above about locality! What's the intuition?

Author response:
First, it is important to recall that we utilize these fundamentals by drawing inspiration from global convolutional layers. For instance, Figure 1 presents relatively local and smooth kernels.

Second, regarding how smoothness and locality enhance the model's ability to manage long-range dependencies, we hypothesize that in long-range tasks across common modalities such as images, text, and speech, long-range dependencies manifest through a hierarchical combination of local and smooth dependencies. Since we argue that the problem of capturing long-range dependencies reflects as a problem of generalization, such inductive bias can be very effective.

Reviewer comment (Experiments):
How does the Transformer architecture perform on other tasks (e.g., standard language modeling or typical image classification)? What do we sacrifice by biasing the models towards solutions that generalize better in long-range context settings?

Author response:
Refer to point #3 in the shared response. We evaluate LaS-attention (compared to vanilla attention and additional ablations) on standard language modeling tasks in Appendix C. It appears that the locality does not negatively impact the results, and there might be a slight improvement with further tuning of the B hyper-parameter (see the last paragraph of Section 4). However, the smoothness seems to negatively affect the perplexity, possibly reducing the transformer's ability to model complex dependencies between tokens (As the 1D Avg Pooling regularizes the bias toward the pairwise interactions).

Reviewer comment (Experiments):
Have you considered learning the biases or explored applying different patterns? How does LaS compare to a model like Synthesizer?

Author response:
Yes, we have considered such variants (and many more), but we didn't observe any major improvements. Please refer to our code for the exact settings and other variants at  \src\models\baselines\transformer.py, line 843:

`self.alpha = alpha if not learn_local_param else torch.nn.Parameter(alpha)`

Reviewer comment (Presentation):
What does the final attention score matrix look like?

Author response:
Please refer to point #2 in the shared response, as well as Appendix D. Evidently, the LaS attention matrices are better tuned to long-range dependencies, particularly in the upper layers, unlike the vanilla transformer which tends to focus on short-range dependencies. Additionally, LaS attention yields smoother attention matrices, mitigating the bias of self-attention towards pairwise interactions.

Reviewer comment (Experiments):
Are there any interactions or side effects from the positional encoding?

Author response:
Based on the experiments we conducted on Wikitext-103, it appears that our method, akin to Alibi, enhances length generalization. However, further experiments are needed to confirm this phenomenon, even though it seems plausible.

Reviewer comment (Novelty):
Methodology: Although the insights are novel, significant, and interesting to read, the methodological novelty is limited. The proposed modification is a simple modification 1) of the pre-softmax linear attention matrix by pointwise multiplication with an exponential decay term, and then 2) smoothing of the activated attention matrix. Especially that a similar modified version appeared in previous work in (Press et al. 2021), and its main difference from 1) is effectively that the distance matrix is exponentiated.

Author response:
>.. the methodological novelty is limited:
Our method (Local and Smooth) is deliberately simple since we seek minimal modifications to the transformer architecture that could improve its long-range capabilities. We believe that controlled and simple experiments are essential for providing clear insights. 

The main novelty of the paper lies in the insights we offer. We have characterized the main challenge of the long range problem (generalization) and identified fundamental principles that make long-range layers, such as state-space layers, much better than transformers on those tasks. Please refer to “Framing of the Contribution” (point #4)  in the shared response for a more comprehensive description of the insights we provide.

Reviewer comment (Experiments):
Experiments: I also found the experimental aspect somewhat lacking, as only LRA and sequential MNIST is considered, where on LRA there is improvement but not up to par yet with global convolutions. It would also be interesting to know if the proposed upgrade can provide improvements on other Transformer tasks, where it already performs as SOTA.

Author response:
> Experiments: I also found the experimental aspect somewhat lacking, as only LRA and sequential MNIST is considered, where on LRA there is improvement but not up to par yet with global convolutions. It would also be interesting to know if the proposed upgrade can provide improvements on other Transformer tasks, where it already performs as SOTA.

First, please see the experiment result on language modeling (point #3 in the main response, Appendix C). Second, we want to highlight that the LRA benchmark is the central benchmark in the domain.

Reviewer comment (Theory):
Question 3: Another question is whether the authors have any intuition about why the exponentiation of the distance matrix helps the attention compared to Alibi? This seems weird to me because applying the decay matrix before the softmax decreases large query-key dot-product values as intended, but negative dot-product values are actually increased, which actually degrades the ability of the model to reduce the dependence on certain tokens. I wonder if this is a desired effect due to some implicit regularization phenomenon of being less likely to attend to a very small select few tokens, or if this might be harmful?

Author response:
> Another question is whether the authors have any intuition about why the exponentiation of the distance matrix helps the attention compared to Alibi? This seems weird to me because applying the decay matrix before the softmax decreases large query-key dot-product values as intended, but negative dot-product values are actually increased, which actually degrades the ability of the model to reduce the dependence on certain tokens. I wonder if this is a desired effect due to some implicit regularization phenomenon of being less likely to attend to a very small select few tokens, or if this is might be harmful?

We investigate this question through experiments. It appears that the model learns to control such phenomena by creating a non-symmetric distribution at the input to the softmax layer. Specifically, when iterating over 1,000 examples with a model trained using Large-Scale (LaS) attention, we observe that the values before the softmax layer are in the range of [-100,000,000 to 100], which could potentially handle your scenario.