Paper: Functional Interpolation for Relative Positions improves Long Context Transformers (ICLR-2024)

Reviewer comment (Experiments):
Heavy reliance on handcrafted input features: The authors attempt to use a neural network to learn a relative position encoding. However, the input features of this network still heavily rely on manual design. Although some ablation experiments are conducted in section B.1, I believe the baseline should be a simpler approach like b(i, j) = f(i - j). For example, [1], [2] has achieved good results by learning relative positional relationships using a similar approach.

Author response:
**Regarding Weakness 1.** Thanks for the reference! We note that [1] and [2] propose new deep learning architectures for sequence modeling, while our work focuses on the Transformer variants. Besides, The idea of "simple" baseline $b(i,j)=f(i-j)$ is exactly T5's RPE, which has been extensively compared in our experiments. This "simple" baseline, along with many existing methods, are mainly evaluated using perplexity-based measures in previous works. Our systematic experiments show that they suffer significant performance drops with longer contexts across diverse evaluation tasks. Those methods typically have hard-coded inductive biases (with fixed RPE patterns or the same encoding for distant tokens), which prevent them from being universally effective. Our method addresses these drawbacks by using a learnable approach with interpolation to truly enable length generalization. 

To reiterate, progressive interpolation is proposed to map relative distances to numbers in $[0,1]$ and avoid out-of-distribution issues during inferences. The $\psi$ transform and thresholding technique further enhance modeling local positions and short sequences. We emphasize that these transformations do not limit the expressive power of FIRE to learn arbitrary biases, which is theoretically shown in Theorem 3.1. Moreover, we carefully ablate these design choices through experiments in Appendix B. Thus, we believe the design of FIRE is both practically effective and theoretically principled.

Reviewer comment (Experiments):
Trade-off between efficiency and performance: Compared to Alibi and RoPE, the learnable FIRE raises the question of how much slower it makes the training process for language modeling. However, this aspect is not mentioned in the paper. Therefore, the authors need to evaluate and discuss the training speed.

Author response:
**Regarding Weakness 2.**  Please note that the performance gap for long context settings between FIRE and baselines is so large ($>2$ perplexity points and $>1$ average SCROLLS score), shifting speed-accuracy pareto curve significantly. In fact, even base-sized FIRE is better than or on par with many large-sized baselines on SCROLLS. Further, in Sec. 4.6, we show that FIRE can be accelerated by layerwise sharing the positional encoding (FIRE-S) and provides the inference speed evaluations. We also follow your suggestion to additionally compare the speed and accuracy of RoPE, Alibi, and FIRE below:

|               | C4 log perplexity (length 8192) | SCROLLS average score | Training speed (steps/sec) |
|:-------------:|:-------------------------------:|:---------------------:|:--------------------------:|
|  RoPE (base)  |               3.52              |         20.83         |           12.80            |
|  RoPE (large) |               3.26              |         22.39         |            4.79            |
|  Alibi (base) |               3.54              |         15.06         |           12.20            |
| Alibi (large) |               3.30              |         16.58         |            4.58            |
| FIRE-S (base) |               3.10              |         24.83         |           10.44            |

It's clear that our method obtains much stronger performance with decent training speed. We will include a full version of the table in the final draft.

Reviewer comment (Experiments):
Lack of certain baselines: For instance, the paper fails to compare its approach with NTK-RoPE[3], YaRN[4], and NTK-ALiBi[5]. These methods have also made advancements in relative position encoding. The absence of comparison with these baselines is a drawback.

Author response:
**Regarding Weakness 3.** Thanks for the references! We would like to first highlight that all these 3 references appeared only after May 28, 2023, with the YaRN paper being uploaded on August 31st, just a month before the ICLR deadline. Hence we did not have comparisons in our submission.

According to ICLR policy, _authors are not required to compare their own work to papers published on or after May 28, 2023_. As all the references have appeared after that and have not been peer-reviewed (we are not even able to find paper drafts for [3] and [5]), they should be considered _concurrent_ to our work.

Among these references, YaRN [4] seems to be the only one with a formal preprint paper and a complete codebase. We follow your suggestion and additionally compare FIRE with YaRN. The results are shown below:

-  **Language modeling on C4**

| Sequence length |  512 | 1024 | 2048 | 4096 | 8192 |
|:---------------:|:----:|:----:|:----:|:----:|:----:|
|       YaRN      | 3.46 | 3.23 | 3.16 | 3.19 | 3.28 |
|       FIRE      | 3.16 | 3.09 | 3.06 | 3.05 | 3.06 |

-  **Language modeling on arXiv**

| Sequence length |  512 | 1024 | 2048 | 4096 | 8192 |
|:---------------:|:----:|:----:|:----:|:----:|:----:|
|       YaRN      | 4.16 | 3.62 | 3.29 | 3.15 | 3.19 |
|       FIRE      | 3.73 | 3.41 | 3.17 | 3.00 | 3.05 |

-  **Language modeling on github**

| Sequence length |  512 | 1024 | 2048 | 4096 | 8192 |
|:---------------:|:----:|:----:|:----:|:----:|:----:|
|       YaRN      | 3.14 | 2.66 | 2.41 | 2.34 | 2.40 |
|       FIRE      | 2.78 | 2.49 | 2.31 | 2.21 | 2.20 |


-  **SCROLLS benchmark**

|      |  QAS  |  CNLI |  QMS  |  NQA |  SumS |  GovR |  QuAL | Average |
|:----:|:-----:|:-----:|:-----:|:----:|:-----:|:-----:|:-----:|:-------:|
| YaRN | 14.64 | 73.10 | 13.83 | 9.46 | 16.20 | 16.74 | 23.11 |  23.87  |
| FIRE | 16.24 | 82.93 | 14.58 | 9.55 | 15.87 | 16.31 | 24.02 |  25.64  |

The performance of YaRN is worse than FIRE on most datasets and sequence lengths, though it improves over RoPE PI. In [4], YaRN is claimed to be better than NTK-RoPE, so FIRE should also be better than NTK-RoPE. Besides, our experiments show that Alibi-based models consistently underperform RoPE/FIRE-based models. Therefore, we believe NTK-Alibi may be less promising to compare with.

[1] Qin, Zhen, et al. "Toeplitz Neural Network for Sequence Modeling." arXiv preprint arXiv:2305.04749 (2023).

[2] Fu, Daniel Y., et al. "Simple hardware-efficient long convolutions for sequence modeling." arXiv preprint arXiv:2302.06646 (2023).

[3] https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/

[4] Peng, Bowen, et al. "Yarn: Efficient context window extension of large language models." arXiv preprint arXiv:2309.00071 (2023).

[5] https://github.com/keezen/ntk_alibi/blob/main/readme_en.md

We sincerely hope that our responses address your concerns and you reevaluate our work based on the responses. We are also willing to discuss with you if you have any further questions.

Reviewer comment (Experiments):
Also regarding the thresholding parameter, it is mentioned that the parameter $L$ is learnable. Could the authors demonstrate what the learned parameter $L$ looks like after training? Also, will model performance significantly change if we set $L$ to a fixed value?

Author response:
**Regarding the learned thresholding parameter $L$.** This is a good question. For the base-sized model with pretraining sequence length 2048, the learned parameter $L$ is between 1200 to 1600 across different layers.

We agree that setting $L$ to a fixed value is another viable option. In our preliminary exploration, FIRE with a fixed/learnable $L$ achieved 24.65/25.64 average SCROLLS scores respectively. Both versions outperform all the baselines, while the learnable variant leads to better performances. Moreover, the fixed variant introduces one more hyper-parameter and may require more tuning. Thus, FIRE uses learnable $L$ as the default choice.

Reviewer comment (Evaluation):
Another weakness is that FIRE is not better than existing methods when the sequence length is short. Even with the thresholding parameter and the slower inference speed, it seems FIRE is not better than RoPE on GLUE/SuperGLUE.

Author response:
**Regarding the short sequence performance.** We agree with the reviewer that FIRE does not lead to performance gains on GLUE/SuperGLUE, but is on par with existing approaches. We emphasize that the primary focus of FIRE is length generalization on long sequences, and our experiments on GLUE/SuperGLUE only serve as an additional sanity check of performances on shorter sequences. RoPE shows good performances on GLUE/SuperGLUE (normal sequence length), but it significantly underperforms FIRE on long sequences (Sec. 4.1-4.3). Besides, Sec. 4.6 shows that FIRE-S (with layer-wise sharing) is faster than RoPE, and the GLUE/SuperGLUE performance is comparable to RoPE (FIRE-S 71.04 v.s. RoPE 71.15). We believe it is a merit of our work to conduct more comprehensive evaluations rather than cherry-picking experimental settings.