Original: CAB: C OMPREHENSIVE A TTENTION B ENCHMARKING ON L ONG S EQUENCE M ODELING
Revised: CAB: C OMPREHENSIVE A TTENTION B ENCHMARKING ON L ONG S EQUENCE M ODELING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformer has achieved remarkable success in language, image, and speech processing.
Revised: Transformer has achieved remarkable success in language, image, and speech processing.

Original: Recently, various efficient attention architectures have been proposed to improve transformer’s efficiency while largely preserving its efficacy, especially in modeling long sequences.
Revised: Recently, various efficient attention architectures have been proposed to improve transformer’s efficiency while largely preserving its efficacy, especially in modeling long sequences.

Original: A widely-used benchmark to test these efficient methods’ capability on long-range modeling is Long Range Arena ( LRA ).
Revised: A widely-used benchmark to test these efficient methods’ capability on long-range modeling is Long Range Arena ( LRA ).

Original: However,
Revised: However,

Original: LRA only focuses on the standard bidirectional (or noncausal) self attention, and completely ignores cross attentions and unidirectional (or causal) attentions, which are equally important to downstream applications.
Revised: LRA only focuses on the standard bidirectional (or noncausal) self attention, and completely ignores cross attentions and unidirectional (or causal) attentions, which are equally important to downstream applications.

Original: Although designing cross and causal variants of an attention method is straightforward for vanilla attention, it is often challenging for efficient attentions with subquadratic time and memory complexity.
Revised: Although designing cross and causal variants of an attention method is straightforward for vanilla attention, it is often challenging for efficient attentions with subquadratic time and memory complexity.

Original: In this paper, we propose Comprehensive Attention Benchmark ( CAB ) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions.
Revised: In this paper, we propose Comprehensive Attention Benchmark ( CAB ) under a fine-grained attention taxonomy with four distinguishable attention patterns, namely, noncausal self, causal self, noncausal cross, and causal cross attentions.

Original: CAB collects seven real-world tasks from different research areas to evaluate efficient attentions under the four attention patterns.
Revised: CAB collects seven real-world tasks from different research areas to evaluate efficient attentions under the four attention patterns.

Original: Among these tasks, CAB validates efficient attentions in eight backbone networks to show their generalization across neural architectures.
Revised: Among these tasks, CAB validates efficient attentions in eight backbone networks to show their generalization across neural architectures.

Original: We conduct exhaustive experiments to benchmark the performances of nine widely-used efficient attention architectures designed with different philosophies on CAB .
Revised: We conduct exhaustive experiments to benchmark the performances of nine widely-used efficient attention architectures designed with different philosophies on CAB .

Original: Extensive experimental results also shed light on the fundamental problems of efficient attentions, such as efficiency length against vanilla attention, performance consistency across attention patterns, the benefit of attention mechanisms, and interpolation/extrapolation on long-context language modeling.
Revised: Extensive experimental results also shed light on the fundamental problems of efficient attentions, such as efficiency length against vanilla attention, performance consistency across attention patterns, the benefit of attention mechanisms, and interpolation/extrapolation on long-context language modeling.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformer has achieved great breakthroughs in natural language processing (Devlin et al., 2019;
Revised: Transformer has achieved great breakthroughs in natural language processing (Devlin et al., 2019;

Original: Radford et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;
Revised: Radford et al., 2019; Brown et al., 2020), computer vision (Dosovitskiy et al., 2020;

Original: Liu et al., 2021b), speech processing (Schneider et al., 2019;
Revised: Liu et al., 2021b), speech processing (Schneider et al., 2019;

Original: Ren et al., 2021), and biology (Jumper et al., 2021; Brandes et al., 2022).
Revised: Ren et al., 2021), and biology (Jumper et al., 2021; Brandes et al., 2022).

Original: A major drawback of the transformer architecture is its quadratic complexity in both time and memory.
Revised: A major drawback of the transformer architecture is its quadratic complexity in both time and memory.

Original: The problem has been more evident with the ever-increasing need in applying transformers for longer sequence modeling in different domains.
Revised: The problem has been more evident with the ever-increasing need in applying transformers for longer sequence modeling in different domains.

Original: Recent research on efficient attention mechanisms seeks to respond to this problem by improving attention efficiency while preserving efficacy (Wang et al., 2020; Kitaev et al., 2019; Zaheer et al., 2020; Choromanski et al., 2020; Zheng et al., 2022).
Revised: Recent research on efficient attention mechanisms seeks to respond to this problem by improving attention efficiency while preserving efficacy (Wang et al., 2020; Kitaev et al., 2019; Zaheer et al., 2020; Choromanski et al., 2020; Zheng et al., 2022).

Original: The commonly adopted test bed for benchmarking efficient transformers in the context of processing long sequences, is the Long Range Arena (Tay et al., 2020a; LRA ), consisting of both synthetic probing tasks and real-world tasks.
Revised: The commonly adopted test bed for benchmarking efficient transformers in the context of processing long sequences, is the Long Range Arena (Tay et al., 2020a; LRA ), consisting of both synthetic probing tasks and real-world tasks.

Original: However, all of these tasks focus on the self attention setting, ignoring cross attention and causal attention , which are equally important and often more challenging.
Revised: However, all of these tasks focus on the self attention setting, ignoring cross attention and causal attention , which are equally important and often more challenging.

Original: In other words, the transformer model is only used as a sequence encoder in LRA , while in real applications, cross attention is essential for conditionality modeling tasks such as sequence-to-sequence (Bahdanau et al., 2015), data-to-text (Duˇsek et al., 2020) and knowledge-enhanced models (Liu et al., 2021a), and causal attention is critical for causality modeling tasks such as text generation (Vaswani et al., 2017;
Revised: In other words, the transformer model is only used as a sequence encoder in LRA , while in real applications, cross attention is essential for conditionality modeling tasks such as sequence-to-sequence (Bahdanau et al., 2015), data-to-text (Duˇsek et al., 2020) and knowledge-enhanced models (Liu et al., 2021a), and causal attention is critical for causality modeling tasks such as text generation (Vaswani et al., 2017;

Original: Zhang et al., 2020), language modeling (Radford et al., 2019; Brown et al., 2020) and speech synthesis (Li et al., 2019).
Revised: Zhang et al., 2020), language modeling (Radford et al., 2019; Brown et al., 2020) and speech synthesis (Li et al., 2019).

Original: Another potential drawback of LRA as discovered by researchers recently, is that with proper tuning, the performance gap between different transformer variants in these tasks can be insignificant (Xiong et al., 2022; Ivgi et al., 2022; Shaham et al., 2022), impairing its effectiveness as a standard benchmark.
Revised: Another potential drawback of LRA as discovered by researchers recently, is that with proper tuning, the performance gap between different transformer variants in these tasks can be insignificant (Xiong et al., 2022; Ivgi et al., 2022; Shaham et al., 2022), impairing its effectiveness as a standard benchmark.

Original: To address these problems, we propose a comprehensive attention benchmark ( CAB ) for long sequence modeling.
Revised: To address these problems, we propose a comprehensive attention benchmark ( CAB ) for long sequence modeling.

Original: Above all, we present a fine-grained attention taxonomy, considering attentive functionality on conditionality and causality modeling.
Revised: Above all, we present a fine-grained attention taxonomy, considering attentive functionality on conditionality and causality modeling.

Original: We present four patterns of attentions, namely, noncausal self, causal self, noncausal cross, and causal cross, representing the distinguishable attentive functionality to sequence modeling (§2).
Revised: We present four patterns of attentions, namely, noncausal self, causal self, noncausal cross, and causal cross, representing the distinguishable attentive functionality to sequence modeling (§2).

Original: With that in mind, we then collect seven real-world tasks from diverse fields of computer vision, natural language processing, speech processing, and time series forecasting (§3).
Revised: With that in mind, we then collect seven real-world tasks from diverse fields of computer vision, natural language processing, speech processing, and time series forecasting (§3).

Original: Among these tasks, CAB includes rich backbone architectures to evaluate the attention mechanisms, testing their performances and generalization abilities.
Revised: Among these tasks, CAB includes rich backbone architectures to evaluate the attention mechanisms, testing their performances and generalization abilities.

Original: Given four attention patterns defined by the attention taxonomy, we advocate a pattern-wise comparison between attention mechanisms, evaluating their attentive functionality respectively.
Revised: Given four attention patterns defined by the attention taxonomy, we advocate a pattern-wise comparison between attention mechanisms, evaluating their attentive functionality respectively.

Original: We conduct exhaustive experiments on CAB , assessing nine widely-used efficient attentions designed with different philosophies (§4).
Revised: We conduct exhaustive experiments on CAB , assessing nine widely-used efficient attentions designed with different philosophies (§4).

Original: The experimental results reveal several insights into designing efficient attention architectures.
Revised: The experimental results reveal several insights into designing efficient attention architectures.

Original: First, we show that existing efficient transformers claiming comparable or even superior performances to vanilla attention, often achieve less competitive results in the causal cross scenario, indicating efficient attentions’ modeling capability cannot always generalize across different attention patterns.
Revised: First, we show that existing efficient transformers claiming comparable or even superior performances to vanilla attention, often achieve less competitive results in the causal cross scenario, indicating efficient attentions’ modeling capability cannot always generalize across different attention patterns.

Original: Second, by quantifying the efficiency of attentions in long sequence contexts using efficiency length (i.e., the minimum length for a sub-quadratic efficient model to surpass vanilla attention in efficiency), we disclose the underlying inefficiency problem of existing efficient attention methods in modeling relatively short sequences.
Revised: Second, by quantifying the efficiency of attentions in long sequence contexts using efficiency length (i.e., the minimum length for a sub-quadratic efficient model to surpass vanilla attention in efficiency), we disclose the underlying inefficiency problem of existing efficient attention methods in modeling relatively short sequences.

Original: Third, we investigate interpolation and extrapolation on a long-context language model, and find that it is promising for efficient attention, such as local attention and LongShort Transformer, to scale to long-context language modeling.
Revised: Third, we investigate interpolation and extrapolation on a long-context language model, and find that it is promising for efficient attention, such as local attention and LongShort Transformer, to scale to long-context language modeling.

Original: We hope CAB and elaborated experimental efforts can shed light on the fundamental problems of efficient attentions, and inspire the design of advanced attention mechanisms.
Revised: We hope CAB and elaborated experimental efforts can shed light on the fundamental problems of efficient attentions, and inspire the design of advanced attention mechanisms.

Original: CAB and all the related codes will be released at https://github.com/Anonymous .
Revised: CAB and all the related codes will be released at https://github.com/Anonymous .

Original:  A TTENTION T AXONOMY
Revised:  A TTENTION T AXONOMY

Original: Attention methods are designed to capture token-to-token dependency within and across sequences.
Revised: Attention methods are designed to capture token-to-token dependency within and across sequences.

Original: Let Y =
Revised: Let Y =

Original: { y 1 , y 2 , . . .
Revised: { y 1 , y 2 , . . .

Original: y n } and Y ∈ R n × d be a target sequence and its feature matrix, and X = { x 1 , x 2 , . . .
Revised: y n } and Y ∈ R n × d be a target sequence and its feature matrix, and X = { x 1 , x 2 , . . .

Original: , x m } and X ∈ R m × d be a source sequence and its feature matrix, where n and m are target and source sequence lengths respectively, and d denotes dimensionality.
Revised: , x m } and X ∈ R m × d be a source sequence and its feature matrix, where n and m are target and source sequence lengths respectively, and d denotes dimensionality.

Original: Note that X could be the same as Y in the case of self attention.
Revised: Note that X could be the same as Y in the case of self attention.

Original: The target and source feature matrices are transformed to
Revised: The target and source feature matrices are transformed to

Original: Q uery- K ey- V alue feature matrices as Q =
Revised: Q uery- K ey- V alue feature matrices as Q =

Original: Y W Q , K = X W K , V = X W V with learnt parametric matrices W Q , W K , W V .
Revised: Y W Q , K = X W K , V = X W V with learnt parametric matrices W Q , W K , W V .

Original: Vanilla transformer (Vaswani et al., 2017) learns to integrate key-value feature matrices K ∈ R m × d , V ∈ R m × d into a query one Q = { q 1 , q 2 , . .
Revised: Vanilla transformer (Vaswani et al., 2017) learns to integrate key-value feature matrices K ∈ R m × d , V ∈ R m × d into a query one Q = { q 1 , q 2 , . .

Original: . , q n } ∈ R n × d with token-to-token attention:
Revised: . , q n } ∈ R n × d with token-to-token attention:

Original: [Equation]
Revised: [Equation]

Original: in which we omit the multihead notation without loss of generality.
Revised: in which we omit the multihead notation without loss of generality.

Original: Vanilla attention in Eq.advocates a token-to-token alignment matrix QK ⊤ , which results in quadratic complexity O ( nm ) of computational time and memory usage.
Revised: Vanilla attention in Eq.advocates a token-to-token alignment matrix QK ⊤ , which results in quadratic complexity O ( nm ) of computational time and memory usage.

Original: It is observed that Attn( · ) in Eq.
Revised: It is observed that Attn( · ) in Eq.

Original: 1 acts as an integration model, projecting query features Q by integrating key-value features without changing Q ’s shape.
Revised: 1 acts as an integration model, projecting query features Q by integrating key-value features without changing Q ’s shape.

Original: Beyond the vanilla token-to-token attention mechanism, we extend Eq.
Revised: Beyond the vanilla token-to-token attention mechanism, we extend Eq.

Original: 1 to a more general form of attention family as:
Revised: 1 to a more general form of attention family as:

Original: [Equation]
Revised: [Equation]

Original: where Attn
Revised: where Attn

Original: [Equation]
Revised: [Equation]

Original: where Q is treated as the main variable and K , V are conditional ones.
Revised: where Q is treated as the main variable and K , V are conditional ones.

Original: Comparing with Eq.
Revised: Comparing with Eq.

Original: 1, Eq.covers efficient attention methods without explicitly modeling the token-to-token alignment.
Revised: 1, Eq.covers efficient attention methods without explicitly modeling the token-to-token alignment.

Original: In usage, attentions are modified to fit generative models with underlying structures, such as conditional models and causal models, to achieve specific attentive functionality.
Revised: In usage, attentions are modified to fit generative models with underlying structures, such as conditional models and causal models, to achieve specific attentive functionality.

Original: In this section, we put forward a fine-grained attention taxonomy, considering the conditionality and causality of attentions.
Revised: In this section, we put forward a fine-grained attention taxonomy, considering the conditionality and causality of attentions.

Original: We show their challenges and potential impact on real-world applications in modeling conditionality and causality.
Revised: We show their challenges and potential impact on real-world applications in modeling conditionality and causality.

Original: Under the taxonomy, we present four attention patterns with different attentive functionality as shown in Figure 1.
Revised: Under the taxonomy, we present four attention patterns with different attentive functionality as shown in Figure 1.

Original: [Figure]
Revised: [Figure]

Original: Self Attention & Cross Attention Self attention (Lin et al., 2017; Vaswani et al., 2017; Dosovitskiy et al., 2020) learns contextual features within a sequence whereas cross attention (Bahdanau et al., 2015) learns to integrate features across sequences.
Revised: Self Attention & Cross Attention Self attention (Lin et al., 2017; Vaswani et al., 2017; Dosovitskiy et al., 2020) learns contextual features within a sequence whereas cross attention (Bahdanau et al., 2015) learns to integrate features across sequences.

Original: Both self attention and cross attention are highly effective in sequence learning, but designing cross attention is more challenging for efficient attentions without explicit token-to-token alignment.
Revised: Both self attention and cross attention are highly effective in sequence learning, but designing cross attention is more challenging for efficient attentions without explicit token-to-token alignment.

Original: On the aspect of computation, Q ∈ R n × d and
Revised: On the aspect of computation, Q ∈ R n × d and

Original: K , V ∈ R m × d are of different shapes and are impossible to perform elementwise operations along length dimension, such as cross-covariance used in XCiT (Ali et al., 2021).
Revised: K , V ∈ R m × d are of different shapes and are impossible to perform elementwise operations along length dimension, such as cross-covariance used in XCiT (Ali et al., 2021).

Original: Besides, different from self attention, cross attention lacks implicit alignment between Q and K , V , such as neighborhood information.
Revised: Besides, different from self attention, cross attention lacks implicit alignment between Q and K , V , such as neighborhood information.

Original: Cross attention learns to integrate conditional features K , V into the main features Q , and has been successfully applied to real-world applications, such as machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), data-to-text generation (Duˇsek et al., 2020;
Revised: Cross attention learns to integrate conditional features K , V into the main features Q , and has been successfully applied to real-world applications, such as machine translation (Bahdanau et al., 2015; Vaswani et al., 2017), data-to-text generation (Duˇsek et al., 2020;

Original: Shen et al., 2020) and knowledge enhanced model (Jin et al., 2020; Liu et al., 2021a).
Revised: Shen et al., 2020) and knowledge enhanced model (Jin et al., 2020; Liu et al., 2021a).

Original: Noncausal Attention & Causal Attention
Revised: Noncausal Attention & Causal Attention

Original: While modeling a sequence, noncausal models have a holistic view of the whole sequence whereas causal ones only access to previous tokens up to now.
Revised: While modeling a sequence, noncausal models have a holistic view of the whole sequence whereas causal ones only access to previous tokens up to now.

Original: This restriction forbids efficient attention like Nystr ¨ omformer (Xiong et al., 2021), which must be conditioned on the whole target sequence to achieve linearity, to be applied in causal models.
Revised: This restriction forbids efficient attention like Nystr ¨ omformer (Xiong et al., 2021), which must be conditioned on the whole target sequence to achieve linearity, to be applied in causal models.

Original: Furthermore, when training a causal model, efficient attentions such as RFA (Peng et al., 2021) and
Revised: Furthermore, when training a causal model, efficient attentions such as RFA (Peng et al., 2021) and

Original: ABC (Peng et al., 2022) have to provide different sets of features from Y for each query token.
Revised: ABC (Peng et al., 2022) have to provide different sets of features from Y for each query token.

Original: This mechanism in these attentions, therefore, leads to multiplied memory consumption compared to their noncausal counterparts.
Revised: This mechanism in these attentions, therefore, leads to multiplied memory consumption compared to their noncausal counterparts.

Original: We provide a detailed analysis of the increasing computation costs in
Revised: We provide a detailed analysis of the increasing computation costs in

Original: Appendix B.
Revised: Appendix B.

Original: In real-world applications, causal attentions bring up strong generation capability and is widely adopted in text generation (Bahdanau et al., 2015; Vaswani et al., 2017; Zhang et al., 2020), text-to-speech synthesis (Li et al., 2019) and language modeling (Radford et al., 2019;
Revised: In real-world applications, causal attentions bring up strong generation capability and is widely adopted in text generation (Bahdanau et al., 2015; Vaswani et al., 2017; Zhang et al., 2020), text-to-speech synthesis (Li et al., 2019) and language modeling (Radford et al., 2019;

Original: Brown et al., 2020; Raffel et al., 2020; Chen et al., 2021a).
Revised: Brown et al., 2020; Raffel et al., 2020; Chen et al., 2021a).

Original: Attention Patterns As discussed above, conditionality and causality are two orthogonal aspects of modeling sequences.
Revised: Attention Patterns As discussed above, conditionality and causality are two orthogonal aspects of modeling sequences.

Original: Thus we obtain four attention patterns via Cartesian product as { noncausal , causal } × { self , cross } : noncausal self (NS), causal self (CS), noncausal cross (NC), and causal cross (CC).
Revised: Thus we obtain four attention patterns via Cartesian product as { noncausal , causal } × { self , cross } : noncausal self (NS), causal self (CS), noncausal cross (NC), and causal cross (CC).

Original: Following the general form of attentions in Eq.
Revised: Following the general form of attentions in Eq.

Original: 2, we formulate the four attention patterns as below:
Revised: 2, we formulate the four attention patterns as below:

Original: [Equation]
Revised: [Equation]

Original: Figure 1 illustrates the computation diagrams of these four patterns.
Revised: Figure 1 illustrates the computation diagrams of these four patterns.

Original: These attention patterns construct typical sequence models with attention mechanisms.
Revised: These attention patterns construct typical sequence models with attention mechanisms.

Original: For instance, Transformer (Vaswani et al., 2017) adopt NS attention in encoder, and CS/CC ones in decoder; recently prevailing Non-Autoregressive
Revised: For instance, Transformer (Vaswani et al., 2017) adopt NS attention in encoder, and CS/CC ones in decoder; recently prevailing Non-Autoregressive

Original: Transformer (Gu et al., 2018) models adopt NS attention in the encoder, and NS/NC attention in the decoder.
Revised: Transformer (Gu et al., 2018) models adopt NS attention in the encoder, and NS/NC attention in the decoder.

Original: Previous efficient attention benchmark LRA only tests attentions with the NS pattern but
Revised: Previous efficient attention benchmark LRA only tests attentions with the NS pattern but

Original:  = { y 1 , y 2 , . . .
Revised: Y = { y 1 , y 2 , . . .

Original: y n } autoregressively, neural models (Vaswani et al., 2017; Li et al., 2019; Brown et al., 2020) usually adopt a causal graph G ( V, E ) with vertices V = { y i ∈ Y } and edges
Revised: y n } autoregressively, neural models (Vaswani et al., 2017; Li et al., 2019; Brown et al., 2020) usually adopt a causal graph G ( V, E ) with vertices V = { y i ∈ Y } and edges

Original: E = { y i → y j | i ∈ [1 , . . . , n ] , j ∈ [1 , . . . , n ] , i ≤ j } .
Revised: E = { y i → y j | i ∈ [1 , . . . , n ] , j ∈ [1 , . . . , n ] , i ≤ j } .

Original: [Table]
Revised: [Table]

Original: without the others, leading to an insufficient evaluation of attentive functionality.
Revised: without the others, leading to an insufficient evaluation of attentive functionality.

Original: In the subsequent sections
Revised: In the subsequent sections

Original: [Table]
Revised: [Table]

Original: we will introduce a comprehensive attention benchmark CAB that evaluates attentions comprehensively on the four patterns
Revised: we will introduce a comprehensive attention benchmark CAB that evaluates attentions comprehensively on the four patterns

Original: [Table]
Revised: [Table]

Original:  C OMPREHENSIVE A TTENTION B ENCHMARK
Revised:  C OMPREHENSIVE A TTENTION B ENCHMARK

Original: We propose Comprehensive Attention Benchmark ( CAB ) that collects seven tasks covering four research fields, including computer vision, natural language processing, speech processing, and time series forecasting.
Revised: We propose Comprehensive Attention Benchmark ( CAB ) that collects seven tasks covering four research fields, including computer vision, natural language processing, speech processing, and time series forecasting.

Original: We introduce these tasks with their datasets, data lengths, evaluation metrics, and backbone neural networks.
Revised: We introduce these tasks with their datasets, data lengths, evaluation metrics, and backbone neural networks.

Original: Then we present a compositional index ( CI ) to combine individual task performance into a compositional one for direct performance comparison across tasks and models.
Revised: Then we present a compositional index ( CI ) to combine individual task performance into a compositional one for direct performance comparison across tasks and models.

Original: 3. L ONG S EQUENCE M ODELING T ASKS
Revised: 3. L ONG S EQUENCE M ODELING T ASKS

Original: We collect seven real-world long-sequence modeling tasks in CAB with data lengths from 300 to 16,000 and include eight widely-used models as backbones in CAB to assess typical efficient attention mechanisms.
Revised: We collect seven real-world long-sequence modeling tasks in CAB with data lengths from 300 to 16,000 and include eight widely-used models as backbones in CAB to assess typical efficient attention mechanisms.

Original: Table 1 summarizes the tasks’ statistics, evaluation metrics, backbone neural networks and the required attention patterns (§2) are also checked.
Revised: Table 1 summarizes the tasks’ statistics, evaluation metrics, backbone neural networks and the required attention patterns (§2) are also checked.

Original: Text-to-Speech Synthesis (TTS)
Revised: Text-to-Speech Synthesis (TTS)

Original: This task requires models to convert input text sequences, descriptions, or narrations produced by a single speaker, to synthesized audio clips sharing the same timbre as the provided speaker.
Revised: This task requires models to convert input text sequences, descriptions, or narrations produced by a single speaker, to synthesized audio clips sharing the same timbre as the provided speaker.

Original: The CAB incorporates the LJSpeech dataset (Ito, 2017) whose audio clips are sampled with 22,050 Hz.
Revised: The CAB incorporates the LJSpeech dataset (Ito, 2017) whose audio clips are sampled with 22,050 Hz.

Original: Under this set of relatively high sample rates, the average sequence length of processed audio clips is 559.
Revised: Under this set of relatively high sample rates, the average sequence length of processed audio clips is 559.

Original: We use non-autoregressive FastSpeech 2 (Ren et al., 2021) and autoregressive Transformer-TTS (Li et al., 2019) as backbone networks.
Revised: We use non-autoregressive FastSpeech 2 (Ren et al., 2021) and autoregressive Transformer-TTS (Li et al., 2019) as backbone networks.

Original: We adopt Mean Cepstral
Revised: We adopt Mean Cepstral

Original: Distortion (MCD) and Mel Spectral Distortion (MSD) for objective evaluation.
Revised: Distortion (MCD) and Mel Spectral Distortion (MSD) for objective evaluation.

Original: Summarization (Sum)
Revised: Summarization (Sum)

Original: This task is one that makes a comprehensive summary of multiple input documents without losing important information.
Revised: This task is one that makes a comprehensive summary of multiple input documents without losing important information.

Original: We consider multi-document summarization task and use Multi-News datasets (Fabbri et al., 2019) for long sequence modeling.
Revised: We consider multi-document summarization task and use Multi-News datasets (Fabbri et al., 2019) for long sequence modeling.

Original: To make the task more challenging, we set the maximum source text and target summary lengths to be 4,096 and 400, respectively.
Revised: To make the task more challenging, we set the maximum source text and target summary lengths to be 4,096 and 400, respectively.

Original: Transformer (Vaswani et al., 2017)  serves the evaluating backbone and ROUGE (R-N) (Lin, 2004) is used to evaluate each attention.
Revised: Transformer (Vaswani et al., 2017) implemented on P ARA G EN (Feng et al., 2022) serves the evaluating backbone and ROUGE (R-N) (Lin, 2004) is used to evaluate each attention.

Original: Long Sequence Time-series Forecasting (LSTF)
Revised: Long Sequence Time-series Forecasting (LSTF)

Original: Long sequence time-series forecasting is to predict long-term future behavior based on past states.
Revised: Long sequence time-series forecasting is to predict long-term future behavior based on past states.

Original: This task evaluates models on three datasets,[Table] including Electricity Transformer Temperature (ETT), Electricity Consuming Load (ECL) [Table]
Revised: This task evaluates models on three datasets, including Electricity Transformer Temperature (ETT), Electricity Consuming Load (ECL), and

Original: and

Original: Weather (Zhou et al., 2021).
Revised: Weather (Zhou et al., 2021).

Original: Following (Zhou et al., 2021) [Table]
Revised: Following (Zhou et al., 2021), we use Informer as backbone and conduct univariate and multivariate evaluations.

Original: we also conduct univariate and multivariate evaluations and average their Mean Square Error (MSE) and Mean Absolute Error (MAE) to obtain the final scores.
Revised: We average their Mean Square Error (MSE) and Mean Absolute Error (MAE) to obtain the final scores.

Original: Point Cloud Completion (PCC)
Revised: Point Cloud Completion (PCC)

Original: This task is to complete a frame of point cloud data when providing partial points.
Revised: This task is to complete a frame of point cloud data when providing partial points.

Original: CAB adopts PCN dataset (Griffiths & Boehm, 2019) which inputs 2, 3D point coordinates and requires models to output the complete point cloud data comprising 14,336 points.
Revised: CAB adopts PCN dataset (Griffiths & Boehm, 2019) which inputs 2, 3D point coordinates and requires models to output the complete point cloud data comprising 14,336 points.

Original: PoinTr (Yu et al., 2021) is adopted here serving as evaluating backbone.
Revised: PoinTr (Yu et al., 2021) is adopted here serving as evaluating backbone.

Original: Chamfer
Revised: Chamfer

Original: Distance (CD) (Huang et al., 2020) and F-Score (Tatarchenko et al., 2019) are selected to evaluate each attention.
Revised: Distance (CD) (Huang et al., 2020) and F-Score (Tatarchenko et al., 2019) are selected to evaluate each attention.

Original: For clearer comparison, CD metrics are enlarged by 1,000 times.
Revised: For clearer comparison, CD metrics are enlarged by 1,000 times.

Original: Language Modeling (LM)
Revised: Language Modeling (LM)

Original: The language modeling task requires the language model to predict the next text based on the previous information.
Revised: The language modeling task requires the language model to predict the next text based on the previous information.

Original: In this task, we consider a long-context language modeling where the context length is prolonged to 8,192.
Revised: In this task, we consider a long-context language modeling where the context length is prolonged to 8,192.

Original: We use PG-19 dataset (Rae et al., 2019) for evaluating language models.
Revised: We use PG-19 dataset (Rae et al., 2019) and conduct evaluations based on the backbone model GPT-2 (Radford et al., 2019) implemented on FAIRSEQ (Ott et al., 2019).

Original: Perplexity (PPL) is used to serve as the evaluation metric.
Revised: Perplexity (PPL) is used to serve as the evaluation metric.

Original: Masked Language Modeling (MLM) Different from the language modeling task, masked language modeling (Devlin et al., 2019) uses full context to predict masked tokens.
Revised: Masked Language Modeling (MLM) Different from the language modeling task, masked language modeling (Devlin et al., 2019) uses full context to predict masked tokens.

Original: Roberta-base (Liu et al., 2019)  is adopted as the evaluation backbone.
Revised: Roberta-base (Liu et al., 2019) implemented on FAIRSEQ is adopted as the evaluation backbone.

Original: We use the same dataset and evaluation metric as LM task while the context length is set to 2,048.
Revised: We use the same dataset and evaluation metric as LM task while the context length is set to 2,048.

Original: Super-Resolution (SR)
Revised: Super-Resolution (SR)

Original: In this task, we aims to convert low-resolution ( 16 × 16 ) face images into high-resolution ( 128 × 128 ) images.
Revised: In this task, we aims to convert low-resolution ( 16 × 16 ) face images into high-resolution ( 128 × 128 ) images.

Original: Following Saharia et al.
Revised: Following Saharia et al.

Original: , we train super-resolution models on Flickr-Faces-HQ (FFHQ) dataset (Karras et al., 2019) and evaluate them on CelebAHQ dataset (Karras et al., 2018).
Revised: (2022), we train the backbone model SR3 on Flickr-Faces-HQ (FFHQ) dataset (Karras et al., 2019) and conduct evaluation on CelebA-HQ dataset (Karras et al., 2018).

Original: Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM)
Revised: Peak Signal-to-Noise Ratio (PSNR) and Structural SIMilarity (SSIM)

Original: (Wang et al., 2004) are selected to evaluate models from different aspects.
Revised: (Wang et al., 2004) are selected to evaluate models from different aspects.

Original: To test attentions under different patterns, we assign each pattern with tasks as suggested in Table 1.
Revised: To test attentions under different patterns, we assign each pattern with tasks as suggested in Table 1.

Original: We exclude LSTF and PCC tasks from noncausal self attention test for the limited contribution of attentions on them (see §4.2).
Revised: We exclude LSTF and PCC tasks from noncausal self attention test for the limited contribution of attentions on them (see §4.2).

Original: 3. C OMPOSITIONAL I NDEXES
Revised: 3. C OMPOSITIONAL I NDEXES

Original: We compute a compositional index ( CI ) from seven tasks for performance comparison.
Revised: We compute a compositional index ( CI ) from seven tasks for performance comparison.

Original: CI is a normalized score to balance the influence among evaluation metrics, and high CI represents excellence.
Revised: CI is a normalized score to balance the influence among evaluation metrics, and high CI represents excellence.

Original: It is computed as follows: a) we transform all evaluation metrics beforehand, so that a higher score indicates better performance; b) we then normalize each transformed metric with
Revised: It is computed as follows: a) we transform all evaluation metrics beforehand, so that a higher score indicates better performance; b) we then normalize each transformed metric with

Original: Z-score normalization; c) after normalization, the score of each evaluation metric is averaged within each task, and is further averaged across tasks.
Revised: Z-score normalization; c) after normalization, the score of each evaluation metric is averaged within each task, and is further averaged across tasks.

Original: We include details of CI calculation in Appendix C.
Revised: We include details of CI calculation in Appendix C.

Original: [Table]

Original:  E XPERIMENTS 4. M AIN R ESULTS
Revised:  E XPERIMENTS 4. M AIN R ESULTS

Original: We conduct exhaustive experiments on CAB by applying various efficient attentions to the backbone models and measure their performance under noncausal self, causal self, noncausal cross [Table] and causal cross attention patterns.We extensively study the fundamental problems of efficient attentions
Revised: We conduct exhaustive experiments on CAB by applying various efficient attentions to the backbone models and measure their performance under noncausal self, causal self, noncausal cross, and causal cross attention patterns.

Original: [Table] in terms of efficiency length against vanilla attention, performance consistency across attention patterns,[Figure] the benefit of attention, and interpolation/extrapolation on long-context language modeling.
Revised: We extensively study the fundamental problems of efficient attentions, in terms of efficiency length against vanilla attention, performance consistency across attention patterns, the benefit of attention, and interpolation/extrapolation on long-context language modeling.

Original: The details of experimental settings for tasks are shown in Appendix D.
Revised: The details of experimental settings for tasks are shown in Appendix D.

Original: Compared Efficient Attentions We benchmark nine typical efficient attentions designed with different philosophies, including sparse attention ( local attention (Luong et al., 2015)), kernelbased linearized attention ( Performer (Choromanski et al., 2020), cosFormer (Qin et al., 2021),
Revised: Compared Efficient Attentions We benchmark nine typical efficient attentions designed with different philosophies, including sparse attention ( local attention (Luong et al., 2015)), kernelbased linearized attention ( Performer (Choromanski et al., 2020), cosFormer (Qin et al., 2021),

Original: LARA (Zheng et al., 2022)), low-rank factorization ( Nystr¨omformer (Xiong et al., 2021)), prototype ( ABC 2 (Peng et al., 2022), ProbSparse (Zhou et al., 2021)), hybrid attention ( LongShort
Revised: LARA (Zheng et al., 2022)), low-rank factorization ( Nystr¨omformer (Xiong et al., 2021)), pro- [Table] totype ( ABC 2 (Peng et al., 2022), ProbSparse (Zhou et al., 2021)), hybrid attention ( LongShort

Original: Transformer (Zhu et al., 2021)), and state space ( S4D (Gu et al., 2022)) and evaluate these attention architectures in terms of different attention patterns described in §2.
Revised: Transformer (Zhu et al., 2021)), and state space ( S4D (Gu et al., 2022)) and evaluate these attention architectures in terms of different attention patterns described in §2.

Original: We provide a brief overview of these attention architectures in Table 2 and leave a detailed description in Appendix F.
Revised: We provide a brief overview of these attention architectures in Table 2 and leave a detailed description in Appendix F.

Original: The main results are shown in Table 3.
Revised: The main results are shown in Table 3.

Original: It is observed that efficient attention achieves comparable performance with vanilla attention under noncausal self attention pattern, and some of them even surpass vanilla attention.
Revised: It is observed that efficient attention achieves comparable performance with vanilla attention under noncausal self attention pattern, and some of them even surpass vanilla attention.

Original: It suggests that the current efficient attention indeed improves the ability to learn contextual features in long sequences of various modalities.
Revised: It suggests that the current efficient attention indeed improves the ability to learn contextual features in long sequences of various modalities.

Original: Although the performance of efficient attention is remarkable in the noncausal self pattern, it is less competitive in the other three patterns.
Revised: Although the performance of efficient attention is remarkable in the noncausal self pattern, it is less competitive in the other three patterns.

Original: As shown in Table 3d, causal cross attentions are extremely hard to design and are inferior to vanilla attention by a significant margin.
Revised: As shown in Table 3d, causal cross attentions are extremely hard to design and are inferior to vanilla attention by a significant margin.

Original: Due to the challenge of causality modeling, it is notable that when compared with autoregressive models (i.e. Transformer and Transformer-TTS), non-autoregressive or parallel generation models (i.e. FastSpeech 2, SR3, and RoBERTa) are more likely to have their efficient variants without significant performance drops.
Revised: Due to the challenge of causality modeling, it is notable that when compared with autoregressive models (i.e. Transformer and Transformer-TTS), non-autoregressive or parallel generation models (i.e. FastSpeech 2, SR3, and RoBERTa) are more likely to have their efficient variants without significant performance drops.

Original: On the dimension of efficient attention, we find that local attention is a strongly competitive efficient model on most real-world tasks, which agrees with the previous studies (Xiong et al., 2022), although it is utterly defeated in LRA .
Revised: On the dimension of efficient attention, we find that local attention is a strongly competitive efficient model on most real-world tasks, which agrees with the previous studies (Xiong et al., 2022), although it is utterly defeated in LRA .

Original: It achieves the highest score of 0.978 under the noncausal self pattern and ranks second under the causal self pattern.
Revised: It achieves the highest score of 0.978 under the noncausal self pattern and ranks second under the causal self pattern.

Original: In contrast, S4D, which achieves impressive results on LRA , appears less inspiring than expected on Sum and MLM tasks.
Revised: In contrast, S4D, which achieves impressive results on LRA , appears less inspiring than expected on Sum and MLM tasks under the noncausal self attention pattern.

Original: Efficiency Analysis We conduct a simulation efficiency experiment under noncausal self attention pattern.
Revised: Efficiency Analysis We conduct a simulation efficiency experiment under noncausal self attention pattern.

Original: The experiment is performed on a single A100 GPU, where attention mechanisms are fed a set of dummy sequences with lengths of { 256 , 512 , 1024 , 2048 , 4096 , 8192 } .
Revised: The experiment is performed on a single A100 GPU, where attention mechanisms are fed a set of dummy sequences with lengths of { 256 , 512 , 1024 , 2048 , 4096 , 8192 } .

Original: Table 2 shows the time and memory consumption against sequence length by each attentions.
Revised: Table 2 shows the time and memory consumption against sequence length by each attentions.

Original: We can find that the advantage of efficient attention gradually expands as the sequence length increases.
Revised: We can find that the advantage of efficient attention gradually expands as the sequence length increases.

Original: Moreover, kernel-based and low-rank attentions are more efficient on time and space.
Revised: Moreover, kernel-based and low-rank attentions are more efficient on time and space.

Original: Further, we report efficiency length to measure the utility of efficient attentions.
Revised: Further, we report efficiency length to measure the utility of efficient attentions.

Original: The efficiency length is defined as the intersection point of computational time and memory curves by efficient models and vanilla attention.
Revised: The efficiency length is defined as the intersection point of computational time and memory curves by efficient models and vanilla attention.

Original: The efficiency length represents the minimum length that a sub-quadratic efficient model surpasses vanilla attention in efficiency.
Revised: The efficiency length represents the minimum length that a sub-quadratic efficient model surpasses vanilla attention in efficiency.

Original: Specifically, we perform regression on vanilla and efficient attentions by their theoretical complexity.
Revised: Specifically, we perform regression on vanilla and efficient attentions by their theoretical complexity.

Original: Then we calculate the intersection of the regression line and the quadratic regression[Figure] curve to get an efficiency length.
Revised: Then we calculate the intersection of the regression line and the quadratic regression curve to get an efficiency length.

Original: Figure 2c shows that efficient attentions gain superiority over vanilla attention once the sequence contains thousands of tokens.
Revised: Figure 2c shows that efficient attentions gain superiority over vanilla attention once the sequence contains thousands of tokens.

Original: For noncausal self attention, ABC and
Revised: For noncausal self attention, ABC and

Original: Performer have the shortest efficiency lengths on running time and memory usage, respectively.
Revised: Performer have the shortest efficiency lengths on running time and memory usage, respectively.

Original: The details of efficiency lengths and experiment settings can be found in Appendix G.
Revised: The details of efficiency lengths and experiment settings can be found in Appendix G.

Revised: [Table]

Original: 4. D ISCUSSIONS
Revised: 4. D ISCUSSIONS

Original: Performance Consistency Across Attention Patterns We investigate whether efficient attentions perform consistently across different attention patterns.
Revised: Performance Consistency Across Attention Patterns We investigate whether efficient attentions perform consistently across different attention patterns.

Original: To achieve this, we compute Pearson correlation (Benesty et al., 2009) over CI between each pair of attention patterns against the same arrangement of efficient attentions 3 for intra-benchmark comparison.
Revised: To achieve this, we compute Pearson correlation (Benesty et al., 2009) over CI between each pair of attention patterns against the same arrangement of efficient attentions 3 for intra-benchmark comparison.

Original: Note that we also include CI computed from LRA for inter-benchmark comparison between CAB ’s four attention patterns and LRA . 
Revised: Note that we also include CI computed from LRA for inter-benchmark comparison between CAB ’s four attention patterns and LRA . In Figure 3a

Revised: [Table]

Original: In Figure 3a, CAB ’s NS attention pattern is highly correlated with LRA , reflecting that efficient NS  attentions developed on LRA do contribute to a wide range of real-world applications.
Revised: CAB ’s NS attention pattern is highly correlated with LRA , reflecting that efficient NS [Figure] [Figure] c) the performance of efficient attentions based on different context lengths in the LM task during the test phase.

Revised: S4D fails on the context with 16,384 tokens

Revised: attentions developed on LRA do contribute to a wide range of real-world applications.

Original: In contrast, cross attention patterns share non-positive correlations with self attention patterns, indicating their inconsistent performance on self and cross attention.
Revised: In contrast, cross attention patterns share non-positive correlations with self attention patterns, indicating their inconsistent performance on self and cross attention.

Original: On the causality dimension, the performances of causal and noncausal models are highly related but not strictly consistent.
Revised: On the causality dimension, the performances of causal and noncausal models are highly related but not strictly consistent.

Original: We also include task-to-task correlation analysis in Appendix I.
Revised: We also include task-to-task correlation analysis in Appendix I.

Original: Benefit of Attention Self attention is designed to capture contextual features while cross attention integrates non-homologous information.
Revised: Benefit of Attention Self attention is designed to capture contextual features while cross attention integrates non-homologous information.

Original: But how much do these features benefit the model?
Revised: But how much do these features benefit the model?

Original: We conduct an ablation study without the attention mechanism.
Revised: We conduct an ablation study without the attention mechanism.

Original: In a preliminary experiment, we find that causal and cross patterns are indispensable to the backbone models as expected, while noncausal self attention only has a limited contribution.
Revised: In a preliminary experiment, we find that causal and cross patterns are indispensable to the backbone models as expected, while noncausal self attention only has a limited contribution.

Original: Therefore, we focus on discussing the attentive functionality of noncausal self attention.
Revised: Therefore, we focus on discussing the attentive functionality of noncausal self attention.

Original: Figure 3b depicts the CI of backbone models with and without attention on tasks.
Revised: Figure 3b depicts the CI of backbone models with and without attention on tasks.

Original: Detailed CI can be found in Appendix H. Results show that attention mechanism improves performance on most tasks.
Revised: Detailed CI can be found in Appendix H. Results show that attention mechanism improves performance on most tasks.

Original: Embarrassingly, after removing the attention mechanism, the PoinTr and
Revised: Embarrassingly, after removing the attention mechanism, the PoinTr and

Original: Informer achieve improvement on PCC and LSTF tasks, respectively.
Revised: Informer achieve improvement on PCC and LSTF tasks, respectively.

Original: A possible reason is that there are alternative contextualizing structures such as the convolutional neural network in the backbone models and the contribution of the attention mechanism is weakened.
Revised: A possible reason is that there are alternative contextualizing structures such as the convolutional neural network in the backbone models and the contribution of the attention mechanism is weakened.

Original: Besides, we also notice that the current efficient attention family has already surpassed vanilla attention in modeling noncausal self relationships on most tasks.
Revised: Besides, we also notice that the current efficient attention family has already surpassed vanilla attention in modeling noncausal self relationships on most tasks.

Original: Interpolation and Extrapolation on Long-Context Language Modeling Language models with efficient attentions are more likely to face unexpected long context that exceeds predefined lengths, resulting in an extrapolation problem.
Revised: Interpolation and Extrapolation on Long-Context Language Modeling Language models with efficient attentions are more likely to face unexpected long context that exceeds predefined lengths, resulting in an extrapolation problem.

Original: It is critical for efficient attentions to scale to longer sequences while preserving the performance on short ones.
Revised: It is critical for efficient attentions to scale to longer sequences while preserving the performance on short ones.

Original: Therefore, we conduct an experiment on the language modeling task, where we train the model with the input of 8,192 tokens and calculate the perplexity of contexts with 256 - 16 , 384 tokens during the test.
Revised: Therefore, we conduct an experiment on the language modeling task, where we train the model with the input of 8,192 tokens and calculate the perplexity of contexts with 256 - 16 , 384 tokens during the test.

Original: Figure 3c shows the results on both of interpolation and extrapolation.
Revised: Figure 3c shows the results on both of interpolation and extrapolation.

Original: On one hand, as the size of context grows to 8,192, language models achieve decreasing perplexity.
Revised: On one hand, as the size of context grows to 8,192, language models achieve decreasing perplexity.

Original: It means that longer contexts indeed improve language modeling, and efficient attentions do well in interpolation as expected.
Revised: It means that longer contexts indeed improve language modeling, and efficient attentions do well in interpolation as expected.

Original: On the other hand, for sequences with more than 8,192 tokens, the perplexities of all these efficient attention-equipped language models become higher.
Revised: On the other hand, for sequences with more than 8,192 tokens, the perplexities of all these efficient attention-equipped language models become higher.

Original: Notably, local attention and LongShort with local features have only a slight drop in performance in extrapolation, while ABC with only global features fails to extrapolate to longer sequences.
Revised: Notably, local attention and LongShort with local features have only a slight drop in performance in extrapolation, while ABC with only global features fails to extrapolate to longer sequences.

Revised: Although

Revised: S4D performs well as causal attention, it has a large perplexity with sequence input longer than 8,192, indicating that S4D has relatively poor extrapolation ability.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Long Sequence Modeling Most efficient attention architectures are designed for long sequence modeling.
Revised: Long Sequence Modeling Most efficient attention architectures are designed for long sequence modeling.

Original: Previous works evaluate their proposed attention modules separately on Language
Revised: Previous works evaluate their proposed attention modules separately on Language

Original: Model (Choromanski et al., 2020; Qin et al., 2021; Peng et al., 2022), Masked Language Model (Qin et al., 2021; Peng et al., 2022), Image Classification (Choromanski et al., 2020; Zheng et al., 2022), and Machine Translation (Peng et al., 2022; 2021).
Revised: Model (Choromanski et al., 2020; Qin et al., 2021; Peng et al., 2022), Masked Language Model (Qin et al., 2021; Peng et al., 2022), Image Classification (Choromanski et al., 2020; Zheng et al., 2022), and Machine Translation (Peng et al., 2022; 2021).

Original: Without unified criteria for evaluating attention modules, researchers find it hard to distinguish between them and select an appropriate attention module for their research.
Revised: Without unified criteria for evaluating attention modules, researchers find it hard to distinguish between them and select an appropriate attention module for their research.

Original: Later, Tay et al.
Revised: Later, Tay et al.

Original: (2020a) proposed Long Range Arena (LRA), a benchmark that contains 5 different classification tasks across language and image domainsHowever, their benchmark only considers the noncausal self attention pattern and contains most synthetic tasks which are far from real-world applications.
Revised: (2020a) proposed Long Range Arena (LRA), a benchmark that contains 5 different classification tasks across language and image domainsHowever, their benchmark only considers the noncausal self attention pattern and contains most synthetic tasks which are far from real-world applications.

Original: Besides, different attention modules perform nearly equally on
Revised: Besides, different attention modules perform nearly equally on

Original: LRA (Xiong et al., 2022), which causes LRA a limited benchmark.
Revised: LRA (Xiong et al., 2022), which causes LRA a limited benchmark.

Original: Shaham et al.
Revised: Shaham et al.

Original: (2022) proposed
Revised: (2022) proposed

Original: SCROLLS for long language sequences but ignores long sequence modeling tasks in other fields.
Revised: SCROLLS for long language sequences but ignores long sequence modeling tasks in other fields.

Original: Efficient Attention
Revised: Efficient Attention

Original: The emergence and prevalence of Transformer models (Vaswani et al., 2017) as well as its strong capability for encoding and generation have imposed impressiveness on various applications.
Revised: The emergence and prevalence of Transformer models (Vaswani et al., 2017) as well as its strong capability for encoding and generation have imposed impressiveness on various applications.

Original: To improve the efficiency of attention in Transformer models, previous work explored efficient attention mechanisms, including sparse attention (Guo et al., 2019a; Ainslie et al., 2020;
Revised: To improve the efficiency of attention in Transformer models, previous work explored efficient attention mechanisms, including sparse attention (Guo et al., 2019a; Ainslie et al., 2020;

Original: Beltagy et al., 2020), low-rank attention (Guo et al., 2019b; Chen et al., 2020; Xiong et al., 2021), kernel-based linearized attention (Choromanski et al., 2020; Katharopoulos et al., 2020), prototype attention (Vyas et al., 2020;
Revised: Beltagy et al., 2020), low-rank attention (Guo et al., 2019b; Chen et al., 2020; Xiong et al., 2021), kernel-based linearized attention (Choromanski et al., 2020; Katharopoulos et al., 2020), prototype attention (Vyas et al., 2020;

Original: Zhou et al., 2021), prior attention (Zhang et al., 2018;
Revised: Zhou et al., 2021), prior attention (Zhang et al., 2018;

Original: Ying et al., 2021;
Revised: Ying et al., 2021;

Original: Tay et al., 2021), and memory compress (Liu* et al., 2018; Lee et al., 2019; Wang et al., 2020).
Revised: Tay et al., 2021), and memory compress (Liu* et al., 2018; Lee et al., 2019; Wang et al., 2020).

Original: For interested readers, we refer them to (Tay et al., 2020b) and (Lin et al., 2021) for a more detailed description of efficient attention.
Revised: For interested readers, we refer them to (Tay et al., 2020b) and (Lin et al., 2021) for a more detailed description of efficient attention.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose to evaluate efficient attentions under a more fine-grained attention taxonomy with four distinguishable attention patterns, each of which reflects different attentive functionality.
Revised: In this paper, we propose to evaluate efficient attentions under a more fine-grained attention taxonomy with four distinguishable attention patterns, each of which reflects different attentive functionality.

Original: Under the taxonomy, we present Comprehensive Attention Benchmark ( CAB ), consisting of seven real-world tasks and eight backbone networks.
Revised: Under the taxonomy, we present Comprehensive Attention Benchmark ( CAB ), consisting of seven real-world tasks and eight backbone networks.

Original: We conduct exhaustive experiments to benchmark nine typical efficient attentions on CAB .
Revised: We conduct exhaustive experiments to benchmark nine typical efficient attentions on CAB .

Original: The empirical results show that existing efficient attentions make significant progress in noncausal self attention, but are still struggling to catch up with vanilla attention in conditionality and causality modeling.
Revised: The empirical results show that existing efficient attentions make significant progress in noncausal self attention, but are still struggling to catch up with vanilla attention in conditionality and causality modeling.

Original: The efficiency analysis shows that although existing efficient attentions enjoy lower computational complexity, most of them only start to achieve significant efficiency gain when applied to sequences with more than one thousand tokens, especially in the causal attention scenario.
Revised: The efficiency analysis shows that although existing efficient attentions enjoy lower computational complexity, most of them only start to achieve significant efficiency gain when applied to sequences with more than one thousand tokens, especially in the causal attention scenario.

Original: Moreover, we find that attention mechanisms, in the noncausal self case, are sometimes not helpful or even harmful to neural models that already contain feature mixing.
Revised: Moreover, we find that attention mechanisms, in the noncausal self case, are sometimes not helpful or even harmful to neural models that already contain feature mixing.

Original: Our study sheds light on long-context language modeling and shows that efficient attentions such as local attention and LongShort Transformer are promising for extrapolation problems in long-context language modeling.
Revised: Our study sheds light on long-context language modeling and shows that efficient attentions such as local attention and LongShort Transformer are promising for extrapolation problems in long-context language modeling.

Original: A E FFICIENT A TTENTIONS WITH P ERFORMANCE D IFFERENCES
Revised: A E FFICIENT A TTENTIONS WITH P ERFORMANCE D IFFERENCES

Original: cosFormer requires a cos -reweighting mechanism to behave like vanilla softmax attention.
Revised: cosFormer requires a cos -reweighting mechanism to behave like vanilla softmax attention.

Original: However, its reweighting mechanism needs the maximum length between the source and target sequences in Eq.
Revised: However, its reweighting mechanism needs the maximum length between the source and target sequences in Eq.

Original: Under causal self pattern, this operation requires different L values for each token of Q and K , which is ignored in its released codebase repository 5 .
Revised: Under causal self pattern, this operation requires different L values for each token of Q and K , which is ignored in its released codebase repository 5 .

Original: Therefore, the released code cannot model causality.
Revised: Therefore, the released code cannot model causality.

Original: Performer uses random feature matrices to project Q and K .
Revised: Performer uses random feature matrices to project Q and K .

Original: However, before transforming Q , K with the softmax kernel, its released codebase 6 uses the golden target sequence length to compute random feature maps under causal self pattern.
Revised: However, before transforming Q , K with the softmax kernel, its released codebase 6 uses the golden target sequence length to compute random feature maps under causal self pattern.

Original: However, this operation brings inconsistency between training and inference, because the golden target length is unavailable during inference.
Revised: However, this operation brings inconsistency between training and inference, because the golden target length is unavailable during inference.

Original: Thus, we cannot use the released code to perform causal self tasks.
Revised: Thus, we cannot use the released code to perform causal self tasks.

Original: B I NCREASING C OMPUTATION C OSTS FOR C AUSAL E FFICIENT A TTENTION
Revised: B I NCREASING C OMPUTATION C OSTS FOR C AUSAL E FFICIENT A TTENTION

Original: In this section, we use an efficient attention example RFA (Peng et al., 2021) to illustrate enforcing causality might bring an increase in computation for efficient attention.
Revised: In this section, we use an efficient attention example RFA (Peng et al., 2021) to illustrate enforcing causality might bring an increase in computation for efficient attention.

Original: In RFA, the noncausal version models the query q t as:
Revised: In RFA, the noncausal version models the query q t as:

Original: [Equation]
Revised: [Equation]

Original: where S t ∈ R 2 D × d and z t ∈ R d and 2 D is the dimensionality of kernel ϕ .
Revised: where S t ∈ R 2 D × d and z t ∈ R d and 2 D is the dimensionality of kernel ϕ .

Original: In its causal version, the hidden state S t and z t are recurrently modified to maintain the causality:
Revised: In its causal version, the hidden state S t and z t are recurrently modified to maintain the causality:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We need to store S t and z t explicitly during training, which contains the historical information specific for each query q t .
Revised: We need to store S t and z t explicitly during training, which contains the historical information specific for each query q t .

Original: As a result, these recurrent operations add additional O ( ndD ) time and memory consumption.
Revised: As a result, these recurrent operations add additional O ( ndD ) time and memory consumption.

Original: [Equation]
Revised: [Equation]

Original: OMPOSITIONAL I NDEX
Revised: OMPOSITIONAL I NDEX

Original: We define s kij as the score of i -th attention model on k -th metric in the j -th task, where k = 1 , . . .
Revised: We define s kij as the score of i -th attention model on k -th metric in the j -th task, where k = 1 , . . .

Original: , K and j = 1 , . .
Revised: , K and j = 1 , . .

Original: , M .
Revised: , M .

Original: To obtain Compositional index ( CI ), we first normalize s k ij .
Revised: To obtain Compositional index ( CI ), we first normalize s k ij .

Original: For the metric whose value is higher indicating high performance, s kij is normalized to s kij by s kij =
Revised: For the metric whose value is higher indicating high performance, s kij is normalized to s kij by s kij =

Original: s kij − µ kj σ kj , where µ kj and σ kj denote the mean and standard deviation of the models’ scores on this metric.
Revised: s kij − µ kj σ kj , where µ kj and σ kj denote the mean and standard deviation of the models’ scores on this metric.

Original: Otherwise, we normalize s kij by s kij = µ kj − s kij σ kj for the metric whose value is lower indicating high performance.
Revised: Otherwise, we normalize s kij by s kij = µ kj − s kij σ kj for the metric whose value is lower indicating high performance.

Original: Then we average the normalized score of each metric to obtain a task score CI ij =
Revised: Then we average the normalized score of each metric to obtain a task score CI ij =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 s kij K , which is further converted to the final score CI
Revised: Kk =1 s kij K , which is further converted to the final score CI

Original: i
Revised: i

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: Mj =1 CI ij M .
Revised: Mj =1 CI ij M .

Original: D H YPERPARAMETERS
Revised: D H YPERPARAMETERS

Original: Hyperparameters for all tasks are shown in Table 4.
Revised: Hyperparameters for all tasks are shown in Table 4.

Original: We also report the hyperparameters of efficient attentions in Table 5.
Revised: We also report the hyperparameters of efficient attentions in Table 5.

Original: [Table]
Revised: [Table]

Original: performance of LongShort with num landmarks = 64 is significantly lower than its performance with num landmarks = 16 on Sum task under the noncausal self attention pattern.
Revised: performance of LongShort with num landmarks = 64 is significantly lower than its performance with num landmarks = 16 on Sum task under the noncausal self attention pattern.

Original: So we finally use LongShort with num landmarks =
Revised: So we finally use LongShort with num landmarks =

Original: [Table]
Revised: [Table]

Original: E T ASK S CORE FOR C OMPARISON
Revised: E T ASK S CORE FOR C OMPARISON

Original: We show the compositional index of each task in Table 6.
Revised: We show the compositional index of each task in Table 6.

Original: For a new efficient attention, it can use our provided means and variances to obtain the normalized score.
Revised: For a new efficient attention, it can use our provided means and variances to obtain the normalized score.

Original: The means and standard deviations of metrics are shown in the Table 7, Table 8, Table
Revised: The means and standard deviations of metrics are shown in the Table 7, Table 8, Table

Original: [Table]
Revised: [Table]

Original: and Table 10.
Revised: and Table 10.

Original: F D ETAILED D ESCRIPTION OF E FFICIENT A RCHITECTURES
Revised: F D ETAILED D ESCRIPTION OF E FFICIENT A RCHITECTURES

Original: We here only present the noncausal self forms of these efficient architectures.
Revised: We here only present the noncausal self forms of these efficient architectures.

Original: In the subsequent discussion, we denote n and m as the target and source sequence lengths, respectively.
Revised: In the subsequent discussion, we denote n and m as the target and source sequence lengths, respectively.

Original: Also, we omit the factor 1 / √ d for simplicity.
Revised: Also, we omit the factor 1 / √ d for simplicity.

Original: Local Attention Local attention (Luong et al., 2015) intuitively forces each query token to attend to its neighborhood tokens within a fixed window size w to reduce the computation costs.
Revised: Local Attention Local attention (Luong et al., 2015) intuitively forces each query token to attend to its neighborhood tokens within a fixed window size w to reduce the computation costs.

Original: In the full attention map, the neighborhood tokens contribute a large proportion of the attention score for the querying token (Qin et al., 2021).
Revised: In the full attention map, the neighborhood tokens contribute a large proportion of the attention score for the querying token (Qin et al., 2021).

Original: Therefore, local attention intuitively selects w ( w/ 2 for causal
Revised: Therefore, local attention intuitively selects w ( w/ 2 for causal

Original: [Table]
Revised: [Table]

Original: self attention pattern)
Revised: self attention pattern)

Original: tokens around a querying token to serve as K w and V w .
Revised: tokens around a querying token to serve as K w and V w .

Original: Its formulation is as follows:
Revised: Its formulation is as follows:

Original: [Equation]
Revised: [Equation]

Original: where K w , V w ∈ R w × d denotes the neighborhood tokens within w size around the query token q i in K and V .
Revised: where K w , V w ∈ R w × d denotes the neighborhood tokens within w size around the query token q i in K and V .

Original: The complexity is of linear complexity O ( nwd ) when w is predefined and independent of sequence length n .
Revised: The complexity is of linear complexity O ( nwd ) when w is predefined and independent of sequence length n .

Original: Due to the locality inductive bias, local attention is competent only on causal self and noncausal self attention patterns.
Revised: Due to the locality inductive bias, local attention is competent only on causal self and noncausal self attention patterns.

Original: Nystr¨omformer Nystr ¨ omformer (Xiong et al., 2021) adopts the Nystr ¨ om method to approximate the softmax attention matrix.
Revised: Nystr¨omformer Nystr ¨ omformer (Xiong et al., 2021) adopts the Nystr ¨ om method to approximate the softmax attention matrix.

Original: It selects r landmarks for Q and K with Segment-means (Shen et al., 2018) and produces (cid:101) Q and (cid:101) K .
Revised: It selects r landmarks for Q and K with Segment-means (Shen et al., 2018) and produces (cid:101) Q and (cid:101) K .

Original: Using iterative Moore-Penrose pseudoinverse (Razavi et al., 2014),
Revised: Using iterative Moore-Penrose pseudoinverse (Razavi et al., 2014),

Original: Nystr ¨ oformer decomposes the softmax attention matrix into three sub-softmax matrices as follows:
Revised: Nystr ¨ oformer decomposes the softmax attention matrix into three sub-softmax matrices as follows:

Original: [Equation]
Revised: [Equation]

Original: where (softmax( (cid:101) Q (cid:101) K ⊤ ))
Revised: where (softmax( (cid:101) Q (cid:101) K ⊤ ))

Original: + is the Moore-Penrose pseudoinverse of softmax( (cid:101) Q (cid:101) K ⊤ ) .
Revised: + is the Moore-Penrose pseudoinverse of softmax( (cid:101) Q (cid:101) K ⊤ ) .

Original: The complexity
Revised: The complexity

Original: Moore-Penrose pseudoinverse is linear with respect to n when its iteration time is constant.
Revised: Moore-Penrose pseudoinverse is linear with respect to n when its iteration time is constant.

Original: Therefore, the overall computation cost is O ( nrd ) when r is predefined.
Revised: Therefore, the overall computation cost is O ( nrd ) when r is predefined.

Original: Because it needs to select the same number of landmarks for Q and K and requires the full Q representation, it is only adaptable in the noncausal self attention pattern.
Revised: Because it needs to select the same number of landmarks for Q and K and requires the full Q representation, it is only adaptable in the noncausal self attention pattern.

Original: Performer Performer (Choromanski et al., 2020) is another linear transformer using random feature map ϕ ( · ) :
Revised: Performer Performer (Choromanski et al., 2020) is another linear transformer using random feature map ϕ ( · ) :

Original: R d → R r + (for some r > 0 ) as a kernel function to reduce computational complexity.
Revised: R d → R r + (for some r > 0 ) as a kernel function to reduce computational complexity.

Original: Formally, the random feature map is defined to approximate the exponential kernel as
Revised: Formally, the random feature map is defined to approximate the exponential kernel as

Original: [Equation]
Revised: [Equation]

Original: [Equation] where ϕ ( x ) = exp( W r x − 12 ∥ x
Revised: (12) where ϕ ( x ) = exp( W r x − 12 ∥ x

Original: ∥ 2 1 r ) ∈ R r and all elements of W r are independently drawn from a standard Gaussian q ( w ) =
Revised: ∥ 2 1 r ) ∈ R r and all elements of W r are independently drawn from a standard Gaussian q ( w ) =

Original: N ( w ; 0 , 1) .
Revised: N ( w ; 0 , 1) .

Original: Given the defined random feature map, the whole softmax attention for each query can then be approximated as
Revised: Given the defined random feature map, the whole softmax attention for each query can then be approximated as

Original: [Equation]
Revised: [Equation]

Original: Thanks to the kernel approximation, Performer achieves O ( nrd ) complexity in both noncausal self and noncausal cross attention patterns; however, for causal self and causal cross attentions, Performer has to store the cumulative summation for both (cid:80) mj
Revised: Thanks to the kernel approximation, Performer achieves O ( nrd ) complexity in both noncausal self and noncausal cross attention patterns; however, for causal self and causal cross attentions, Performer has to store the cumulative summation for both (cid:80) mj

Original: =1 ϕ ( k j ) v ⊤ j and (cid:80) mj
Revised: =1 ϕ ( k j ) v ⊤ j and (cid:80) mj

Original: =1 ϕ ( k j ) , which requires a dedicated CUDA operator to accelerate the computation.
Revised: =1 ϕ ( k j ) , which requires a dedicated CUDA operator to accelerate the computation.

Original: LARA LARA (Zheng et al., 2022) further extends the random feature approximation from estimating individual exponential kernels to directly estimating the whole softmax function.
Revised: LARA LARA (Zheng et al., 2022) further extends the random feature approximation from estimating individual exponential kernels to directly estimating the whole softmax function.

Original: In particular, there exists a distribution p that instantiates the random feature (Zheng et al., 2022) such as:
Revised: In particular, there exists a distribution p that instantiates the random feature (Zheng et al., 2022) such as:

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: By viewing softmax attention as an expectation over a potentially complicated distribution p , Zheng et al.
Revised: By viewing softmax attention as an expectation over a potentially complicated distribution p , Zheng et al.

Original: (2022) noted that Performer can be understood as performing self-normalized importance sampling to approximate this expectation, with standard Gaussian as the proposal distribution.
Revised: (2022) noted that Performer can be understood as performing self-normalized importance sampling to approximate this expectation, with standard Gaussian as the proposal distribution.

Original: LARA generalizes this view by adopting multiple adaptive proposal distributions to further improve the approximation fidelity.
Revised: LARA generalizes this view by adopting multiple adaptive proposal distributions to further improve the approximation fidelity.

Original: It is shown that LARA takes the following form:
Revised: It is shown that LARA takes the following form:

Original: [Equation]
Revised: [Equation]

Original: (15) where A n is a query-specific diagonal matrix that ensures the computation overall still yields a valid self-normalized importance sampling estimation; and the elements of sampled W r in ϕ ( x ) =
Revised: where A n is a query-specific diagonal matrix that ensures the computation overall still yields a valid self-normalized importance sampling estimation; and the elements of sampled W r in ϕ ( x ) =

Original: exp( W r x − 12 ∥ x ∥ 2 1 r ) are allowed to be drawn from different distributions beyond standard Gaussian.
Revised: exp( W r x − 12 ∥ x ∥ 2 1 r ) are allowed to be drawn from different distributions beyond standard Gaussian.

Original: cosFormer
Revised: cosFormer

Original: Following the same method defined in Eq. 13, cosFormer (Qin et al., 2021) adopts another kernel function ReLU( · ) instead of random feature map.
Revised: Following the same method defined in Eq. 13, cosFormer (Qin et al., 2021) adopts another kernel function ReLU( · ) instead of random feature map.

Original: Due to the importance of the softmax operator that can ensure the non-negativeness of value weights and stabilize the training process, cosFormer designed a new cos -based reweighting mechanism similar to softmax.
Revised: Due to the importance of the softmax operator that can ensure the non-negativeness of value weights and stabilize the training process, cosFormer designed a new cos -based reweighting mechanism similar to softmax.

Original: Denote q ′ i
Revised: Denote q ′ i

Original: =
Revised: =

Original: ReLU( q i ) and k ′ j = ReLU( k j ) , the re-weight formula is expressed as follows:
Revised: ReLU( q i ) and k ′ j = ReLU( k j ) , the re-weight formula is expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where L = max( n, m ) .
Revised: where L = max( n, m ) .

Original: Using this formula and Ptolemy’s theorem, the final expression of cosFormer is as follows:
Revised: Using this formula and Ptolemy’s theorem, the final expression of cosFormer is as follows:

Original: [Equation]
Revised: [Equation]

Original: where q cos i = q i cos( πi 2 L ) , q sin i
Revised: where q cos i = q i cos( πi 2 L ) , q sin i

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: sin

Original: [Equation]
Revised: [Equation]

Original: Similar to
Revised: Similar to

Original: Performer, it can efficiently fulfill noncausal self and noncausal cross attentions with linear complexity
Revised: Performer, it can efficiently fulfill noncausal self and noncausal cross attentions with linear complexity

Original: O ( nd 2 ) , but its causal attention relies on a specialized CUDA operator to achieve high efficiency.
Revised: O ( nd 2 ) , but its causal attention relies on a specialized CUDA operator to achieve high efficiency.

Original: LongShort Transformer LongShort Transformer (Zhu et al., 2021) models the long-range and short-term dependency of a sequence with two different methods discussed above.
Revised: LongShort Transformer LongShort Transformer (Zhu et al., 2021) models the long-range and short-term dependency of a sequence with two different methods discussed above.

Original: For long-range modeling, it utilizes the low-rank property of the softmax function to convert key and value to a more compact representation using a learnable matrix W P ∈ R d × r : K = P ⊤ K ∈ R r × d and
Revised: For long-range modeling, it utilizes the low-rank property of the softmax function to convert key and value to a more compact representation using a learnable matrix W P ∈ R d × r : K = P ⊤ K ∈ R r × d and

Original: V = P ⊤ V ∈ R r × d where P = softmax( K W P ) and r is the compressed dimension.
Revised: V = P ⊤ V ∈ R r × d where P = softmax( K W P ) and r is the compressed dimension.

Original: For shortterm modeling, it borrows local attention to output a token representation fused with neighborhood features via segment-wise local attention.
Revised: For shortterm modeling, it borrows local attention to output a token representation fused with neighborhood features via segment-wise local attention.

Original: Let l be the segment length and s i =
Revised: Let l be the segment length and s i =

Original: ⌊ i/l ⌋ be the segment id for i -th token, segment-wise local attention produces K w =
Revised: ⌊ i/l ⌋ be the segment id for i -th token, segment-wise local attention produces K w =

Original: K
Revised: K

Original: ( s i − 1) l + l 2 :(
Revised: ( s i − 1) l + l 2 :(

Original: s i +1)
Revised: s i +1)

Original: l + l
Revised: l + l

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: and V w = V ( s i − 1) l + l 2 :( s i +1)
Revised: and V w = V ( s i − 1) l + l 2 :( s i +1)

Original: l + l 2 .
Revised: l + l 2 .

Original: Combined with these two methods, the output of LongShort
Revised: Combined with these two methods, the output of LongShort

Original: Transformer is formalized as follows:
Revised: Transformer is formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: where [ · ; · ] is a concatenation operator at the first dimension.
Revised: where [ · ; · ] is a concatenation operator at the first dimension.

Original: The complexity O ( nrd + nwd ) is also linear w.r
Revised: The complexity O ( nrd + nwd ) is also linear w.r

Original: [Table]
Revised: [Table]

Original: t. the target length n .
Revised: t. the target length n .

Original: Due to its introduction of local attention, LongShort Transformer can only complete tasks under causal self and noncausal self attention patterns.
Revised: Due to its introduction of local attention, LongShort Transformer can only complete tasks under causal self and noncausal self attention patterns.

Original: ProbSparse Attention ProbSparse (Zhou et al., 2021) calculates attention scores for few ( Q , K ) dot-product pairs based on sparsity of self attention scores.
Revised: ProbSparse Attention ProbSparse (Zhou et al., 2021) calculates attention scores for few ( Q , K ) dot-product pairs based on sparsity of self attention scores.

Original: Specifically, ProbSparse selects some “important” queries whose attention distributions on keys are non-uniform.
Revised: Specifically, ProbSparse selects some “important” queries whose attention distributions on keys are non-uniform.

Original: Then we only need to calculate the attention scores between the “important” queries and all keys and output the weighted sum of V .
Revised: Then we only need to calculate the attention scores between the “important” queries and all keys and output the weighted sum of V .

Original: For other trivial queries, ProbSparse takes the mean of V as outputs.
Revised: For other trivial queries, ProbSparse takes the mean of V as outputs.

Original: To distinguish the “important” queries, ProbSparse first samples some keys K , and then measures the sparsity of query q i by ¯ M (cid:0) q i , K (cid:1) as follows:
Revised: To distinguish the “important” queries, ProbSparse first samples some keys K , and then measures the sparsity of query q i by ¯ M (cid:0) q i , K (cid:1) as follows:

Original: [Equation]
Revised: [Equation]

Original: where L K denotes the number of keys.
Revised: where L K denotes the number of keys.

Original: Finally, the Top- u queries with large ¯ M (cid:0) q i , K (cid:1) are the “important” queries.
Revised: Finally, the Top- u queries with large ¯ M (cid:0) q i , K (cid:1) are the “important” queries.

Original: ABC In this paper, we use ABC to denote ABC MLP (Peng et al., 2022).
Revised: ABC In this paper, we use ABC to denote ABC MLP (Peng et al., 2022).

Original: ABC MLP bounds the amount of memory slot for K and V to be r < m , independent of both n and m .
Revised: ABC MLP bounds the amount of memory slot for K and V to be r < m , independent of both n and m .

Original: ABC MLP uses a learnable matrix to fill each memory slot with reweighted key and value representations:
Revised: ABC MLP uses a learnable matrix to fill each memory slot with reweighted key and value representations:

Original: [Equation]
Revised: [Equation]

Original: where W Kϕ , W Vϕ ∈ R r × d is a learnable matrix representing the bounded memory.
Revised: where W Kϕ , W Vϕ ∈ R r × d is a learnable matrix representing the bounded memory.

Original: Leveraging the bounded representation (cid:101) K and (cid:101) V , ABC MLP executes following softmax attention:
Revised: Leveraging the bounded representation (cid:101) K and (cid:101) V , ABC MLP executes following softmax attention:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ABC MLP is again of linear complexity O ( nrd ) .
Revised: ABC MLP is again of linear complexity O ( nrd ) .

Original: S4D S4D
Revised: S4D S4D

Original: (Gu et al., 2022) is different from the above attention-based methods, as it relies on State
Revised: (Gu et al., 2022) is different from the above attention-based methods, as it relies on State

Original: Space Matrix called “HiPPO matrix” (Gu et al., 2020) which enables the whole model to be viewed as a convolutional model.
Revised: Space Matrix called “HiPPO matrix” (Gu et al., 2020) which enables the whole model to be viewed as a convolutional model.

Original: It does not require the introduction of Q , K , V as it takes the whole modeling process as a linear time-invariant (LTI) system u ( t ) (cid:55)→ y ( t ) where u ( t ) is the input sequence.
Revised: It does not require the introduction of Q , K , V as it takes the whole modeling process as a linear time-invariant (LTI) system u ( t ) (cid:55)→ y ( t ) where u ( t ) is the input sequence.

Original: To parameterize the system, S4D designs a Convolution Kernel K ( t )
Revised: To parameterize the system, S4D designs a Convolution Kernel K ( t )

Original: ∈ R n × d and introduces a inner state dimension r during parameterization of K ( t ) :
Revised: ∈ R n × d and introduces a inner state dimension r during parameterization of K ( t ) :

Original: [Equation]
Revised: [Equation]

Original: where A, B, C are all initialized with certain values and trained with the whole model.
Revised: where A, B, C are all initialized with certain values and trained with the whole model.

Original: During implementation, S4D processes u and K in the frequency domain and transforms back the output to the time domain:
Revised: During implementation, S4D processes u and K in the frequency domain and transforms back the output to the time domain:

Original: [Equation]
Revised: [Equation]

Original: Therefore, S4D has the complexity of O ( nrd ) .
Revised: Therefore, S4D has the complexity of O ( nrd ) .

Original: G E FFICIENCY L ENGTH
Revised: G E FFICIENCY L ENGTH

Original: We show efficiency lengths under noncausal self attention pattern in Table 11, where we set hyperparameters as emb dim = 512 , head = 8 , batch size = 4 , wsize = 16 , num landmarks = 16 , approx attn dim = 16 , d state = 16 , conv kernel size = 5 .
Revised: We show efficiency lengths under noncausal self attention pattern in Table 11, where we set hyperparameters as emb dim = 512 , head = 8 , batch size = 4 , wsize = 16 , num landmarks = 16 , approx attn dim = 16 , d state = 16 , conv kernel size = 5 .

Original: We use the adaptive factor for ProbSparse introduced in Skyformer (Chen et al., 2021b) repository 7 .
Revised: We use the adaptive factor for ProbSparse introduced in Skyformer (Chen et al., 2021b) repository 7 .

Original: [Table]
Revised: [Table]

Original: H B ENEFIT OF A TTENTION
Revised: H B ENEFIT OF A TTENTION

Original: We report the CI of backbone models with and without attention in Table
Revised: We report the CI of backbone models with and without attention in Table

Original: [Table] [Table]
Revised: [Table] [Table]

Original: I I NTER - TASK C ORRELATION
Revised: I I NTER - TASK C ORRELATION

Original: Figure 4 shows the task-to-task correlation.
Revised: Figure 4 shows the task-to-task correlation.

Original: Except for the causal cross attention pattern, tasks within the other three patterns scarcely correlate positively with each other.
Revised: Except for the causal cross attention pattern, tasks within the other three patterns scarcely correlate positively with each other.

Original: This suggests that there does hardly exist a single task that can comprehensively test the capability of some attention module under an attention pattern
Revised: This suggests that there does hardly exist a single task that can comprehensively test the capability of some attention module under an attention pattern

Original: [Figure]
Revised: [Figure]

Original: 1& && /5$ 1 & 1& &&
Revised: 1& && /5$ 1 & 1& &&

Original: [Figure]
Revised: [Figure]