Original: C OMPOSITE S LICE T RANSFORMER : A N E FFICIENT T RANSFORMER WITH C OMPOSITION OF M ULTI -S CALE M ULTI -R ANGE A TTENTIONS
Revised: C OMPOSITE S LICE T RANSFORMER : A N E FFICIENT T RANSFORMER WITH C OMPOSITION OF M ULTI -S CALE M ULTI -R ANGE A TTENTIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Since the introduction of Transformers, researchers have tackled thenotoriously expensive quadratic complexity problem.
Revised: Since the introduction of Transformers, researchers have tackled the expensive quadratic complexity of the attention mechanism.

Original: While significant computational efficiency improvements have been achieved, they  come at the cost of reduced accuracy trade-offs.
Revised: While significant complexity improvements have been achieved, they often come at the cost of reduced accuracy.

Original: In this paper, we propose Composite Slice Transformer ( CST ), a Transformer-based network equipped with a composition of multi-scale multirange attentions, boosting both efficiency and modeling capability.
Revised: In this paper, we propose Composite Slice Transformer ( CST ), a Transformerbased network equipped with a composition of multi-scale multi-range attentions, boosting both efficiency and modeling capability.

Original: After stacking fixed-length slices of the input sequence, each layer in CST performs a pair of fine-and-coarse-grained attentions with short-long ranges in a sequential manner, coupled with  volatile instant positional embedding, enabling efficient token interactions and improving expressiveness of the model.
Revised: After stacking fixed-length slices of the input sequence, each layer in CST performs a pair of fine-and-coarse-grained attentions with short and long ranges in a sequential manner, coupled with a volatile instant positional embedding.

Original: In addition to significantly reduced O ( NL + N 2 /L 2 ) complexity for sequence length N and slice length L , CST achieves superior performance on a variety of tasks.
Revised: In addition to significantly reduced O ( NL + N 2 /L 2 ) complexity for sequence length N and slice length L , CST achieves superior performance on a variety of tasks.

Original: We show that CST surpasses recently published efficient Transformers on the Long Range Arena benchmark, demonstrating the bidirectional long-range dependency modeling capability of our model.
Revised: We show that CST surpasses recently published efficient Transformers on the Long Range Arena benchmark, demonstrating the bidirectional long-range dependency modeling capability of our model with a comparable complexity.

Original: It  outperforms the standard Transformer by a margin of 6 . 9 % in average accuracy across the five classification tasks of the benchmark, while being of complexity comparable to other efficient transformers.
Revised: It also outperforms the standard Transformer by a margin of 6 . 9 % in average accuracy across the five classification tasks.

Original: Furthermore, on the word-level autoregressive language modeling task with  the WikiTextdataset, CST performs competitively against the Transformer model with only 2 % gap in the test perplexity while outperforming other efficient Transformers.
Revised: On the wordlevel WikiText-103 autoregressive language modeling task with various sequence lengths, and the masked language modeling followed by GLUE benchamrk, CST outperforms most other efficient Transformers while being competitive against the Transformer.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers (Vaswani et al., 2017)and their variants are among the most pivotal contributions to recent advances in artificial intelligence (AI).
Revised: Transformers (Vaswani et al., 2017) are one of the most important recent advances in artificial intelligence.

Original: Since they can be combined in a straightforward fashion with advanced training methods and auxiliary modules, Transformers have proven extremely effective as a versatile backbone architecture for achieving state-of-the-art performance in many domains such as natural language processing (Devlin et al. ; Yang
Revised: Since they can be combined in a straightforward fashion with advanced training methods and auxiliary modules, Transformers have proven extremely effective as a versatile backbone architecture for achieving state-of-the-art performance in many domains such as natural language processing (Devlin et al., 2018;

Original: ; Brown  reinforcement 
Revised: Yang et al., 2019; Brown et al., 2020; Raffel et al., 2020; Sanh et al., 2022), vision processing (Dosovitskiy et al., 2020; Liu et al., 2021b; Radford et al., 2021), visual language modeling (Alayrac et al., 2022), speech recognition (Dong et al., 2018; Gulati et al., 2020; Shi et al., 2021), and reinforcement learning (Chen et al., 2021b; Janner et al., 2021).

Original: Despite this versatility and wide applicability, Transformers possess an expensive memory and computational complexity of O ( N 2 ) with respect to input length N ; as a result of these complexity drawbacks, Transformers are often not applied totasks with long sequence data.
Revised: Despite this versatility, Transformers possess an expensive memory and computational complexity of O ( N 2 ) with respect to input length N in the multi-head self-attention computation. As a result, Transformers are often not applied to long sequence data.

Original: Since the introduction of Transformers in (Vaswani et al., 2017), significant research has concentrated on improving Transformer complexity through variousapproaches and techniques, achieving efficiency improvements in computation and memory  (Tay et al., 2020c), even of O ( N ) complexity (Wang et al., 2020; Katharopoulos et al., 2020; Ma et al., 2021).
Revised: Since the introduction of Transformers in (Vaswani et al., 2017), recent work has focused on improving Transformer complexity through various techniques, achieving efficiency gains in complexity and memory requirements (Tay et al., 2020c), with several models attaining O ( N ) complexity (Wang et al., 2020; Katharopoulos et al., 2020; Ma et al., 2021).

Original: Unfortunately,despite these clear advances in complexity reduction, these accelerations come at the expense of decreased accuracy and simultaneously lack the modeling capability of fine-graineddependencies among all token pairs, hamstringing the ability to apply these recent techniques to  practical problems.
Revised: Unfortunately, these efficiencies come at the cost of reduced accuracy and often lack the ability to model fine-grained token dependencies, limiting the application of these recent improvements to a wider range of practical problems.

Original: Recent studies (Zhu et al., 2021; Ren et al., 2021;
Revised: Recent studies (Zhu et al., 2021; Ren et al., 2021;

Original: Nguyen et al., 2021; Zhu & Soricut, 2021; Han et al., 2021) show that combining efficient techniques for global sequence modeling with fine-grained limited range attention can improve accuracy, maintain low complexity, and enable the longer-range context windows, allowing such models to outperform the standard, full-resolution Transformer on certain tasks.
Revised: Nguyen et al., 2021; Zhu & Soricut, 2021; Han et al., 2021) show that combining efficient techniques for global sequence modeling with fine-grained limited range attention can improve accuracy, maintain low complexity, and enable longer context windows, allowing such models to outperform the standard, full-resolution Transformer on certain tasks.

Original: However, optimally combining sequence modeling at different levels of granularity remains an open problem.
Revised: However, optimally combining sequence modeling at different levels of granularity remains an open problem.

Original: Through the development of a technique for efficiently combining local and global token dependency modeling, we have observed better performance than the standard Transformers can be achieved.

Original: To this end, we propose thewider adoption of our Composite Slice Transformer ( CST ), an efficientand innovative Transformer-based network architecture, consisting of a composition of attentions applied to a stacked, slice representation of the input sequence at different scales, coupled with a multi-scale volatile instant positional embedding.
Revised: To this end, we propose the Composite Slice Transformer ( CST ), an efficient Transformer-based network architecture, consisting of a composition of attentions applied to a stacked slice representation of the input sequence at different scales, coupled with a multi-scale volatile instant positional embedding.

Original: For fixed slice length L , CST has a complexity of O ( NL + N 2 /L 2 ) ,  comparable  to linear complexity in many practical settings (Section A.3).
Revised: For fixed slice length L , CST has a complexity of O ( NL + N 2 /L 2 ) , which is comparable or more efficient to linear complexity in many practical settings (Section A.4).

Original: Since slicing restricts fine-grained token interaction across boundaries, CST also leverages  extended attention (or overlapped attention), preventing context fragmentation (Dai et al., 2019b) and enabling seamless sequence modeling.
Revised: Since slicing restricts fine-grained token interaction across boundaries, CST also leverages an extended local attention, preventing context fragmentation (Dai et al., 2019b) and enabling seamless sequence modeling.

Revised: These improvements allow CST to outperform the standard Transformer on several benchmark tasks.

Original: Similar to (Dosovitskiy et al., 2020), CST abstracts the input sequence into another with fewer tokens for low-resolution attention with longer range with fixed-size segment-wise embedding.
Revised: Similar to (Dosovitskiy et al., 2020), CST abstracts the input sequence into another with fewer tokens to compute a low-resolution attention with longer range and a fixed-size segment-wise embedding.

Original: When the segment lengthis L ,as the  sequence  lengthbecomes N/L , the quadratic complexity N 2 becomes a decimated quadratic one N 2 /L 2  .
Revised: With a segment length L , the reduced sequence has length N/L ; evaluating attention on this reduced sequence has complexity O ( N 2 /L 2 ) .

Original: With a proper L , the model can benefit from a significantamount of complexity reduction while having comparable performance.
Revised: With an appropriately chosen L , CST can achieve a significant complexity reduction without a loss of performance.

Original: Along with the segmentlevel abstractive attention, CST also leverages a full-resolution local attention to form a multi-scale multi-range attention (MSMRA) similar to (Zhu & Soricut, 2021; Zhu et al., 2021;
Revised: Along with the segment-level attention, CST also leverages a full-resolution local attention to form a multi-scale multi-range attention (MSMRA) through a composition with the global attention, improving model expressiveness (Section A.5), unlike (Zhu & Soricut, 2021; Zhu et al., 2021;

Original: Nguyen et al., 2021; Ren et al., 2021; Han et al., 2021).
Revised: Nguyen et al., 2021; Ren et al., 2021; Han et al., 2021).

Original: Additionally, CST extends the ideas of position-infused attentionin (Press et al., 2021) and applies them to MSMRA, which we refer to as multi-scale volatile instant positional embedding (MS-VIPE).
Revised: CST extends the ideas of position-infused attention (Press et al., 2021) and applies them to MSMRA, which we refer to as multi-scale volatile instant positional embedding (MS-VIPE).

Original: In addition to its effectiveness in embedding of positional information, MS-VIPEfurther provides  parameter efficiency since it only requires storage for reduced lengths of local attention L  and global attention N/L  instead of the full  length N .
Revised: In addition to its effectiveness as a positional embedding, MS-VIPE provides further parameter efficiency by only requiring storage of reduced lengths of local ( L ) and global ( N/L ) attentions instead of the full sequence length N .

Original: We evaluate our model on bidirectional long-range dependency modeling tasks and an autoregressive language modeling task.
Revised: We evaluate our model on bidirectional long-range dependency modeling tasks, an autoregressive language modeling task, and natural language understanding tasks.

Original: In our experiments (Section 5), CST achieves state-of-the-art performance among all Transformer-based models (including the standard Transformer and recently proposed efficient Transformers) as well as demonstrates strong performance on a wide variety of tasks.
Revised: In our experiments (Section 5), CST achieves state-of-the-art performance among all Transformer-based models (including the standard Transformer and recently proposed efficient Transformers) and demonstrates strong performance on a wide variety of tasks.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2 and Section 3, we discuss recent efficient Transformer developments within a single logical framework, discuss strengths and weaknesses, and outline the inspiration for CST .
Revised: In Section 2 and Section 3, we discuss recent efficient Transformer developments, discuss strengths and weaknesses, and outline the inspiration for CST .

Original: In Section 4, we present our model, Composite Slice Transformer, detailing its features.
Revised: In Section 4, we present our model, Composite Slice Transformer, and describe its features.

Original: In Section 5, we show our experimental results on the Long Range Arena benchmark and WikiText-103 autoregressive language modeling benchmark demonstrating that CST is efficient in practice and applicable to a wide spectrum of different tasks.
Revised: In Section 5, we provide our experimental results on the Long Range Arena, WikiText-103 autoregressive language modeling, and the GLUE benchmarks, demonstrating the efficiency and versatility of CST .

Original: In Section 6, we conclude with a summary of our work and discuss about potential future works in this direction.
Revised: In Section 6, we conclude with a summary of our work.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORK

Original: Tay et al.
Revised: Tay et al.

Original: (2020c) provides a comprehensive study of proposed approaches to accelerate the attention mechanism of Vaswani et al.
Revised: (2020c) provides a comprehensive study of proposed approaches to accelerate the attention mechanism of Vaswani et al.

Original: The key differentiating factor among the approaches presented is the modeling assumptions used to approximate the original attention map.
Revised: The key differentiating factor among the approaches presented is the modeling assumptions used to approximate the original attention map.

Original: For example, common assumptions include sparsity (Child et al., 2019;
Revised: For example, common assumptions include sparsity (Child et al., 2019;

Original: Ho et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020; Tay et al., 2020a; Kitaev et al., 2019) and low-rankness (Wang et al., 2020;
Revised: Ho et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020; Tay et al., 2020a; Kitaev et al., 2019) and low-rankness (Wang et al., 2020;

Original: Choromanski et al., 2020; Katharopoulos et al., 2020).
Revised: Choromanski et al., 2020; Katharopoulos et al., 2020).

Original: Chen et al.
Revised: Chen et al.

Original: (2021a)complementarily combines these two assumptions.
Revised: (2021a) combines these two assumptions.

Original: Other architectures leverage additional memory for compressing the global context (Lee et al., 2019; Ma et al., 2021; Jaegle et al., 2021).
Revised: Other architectures leverage additional memory for compressing the global context (Lee et al., 2019; Ma et al., 2021; Jaegle et al., 2021).

Original: A group of approaches aim to reduce the dimensions of the operands involved in computing attention (Wang et al., 2020;

Original: Ma et al., 2021; Dosovitskiy et al., 2020) as abstractive attention .

Original: ( CST ) leverages a one-dimensional version of the token abstraction used in (Dosovitskiy et al., 2020) for the global attention, but with additional composition of local attention.

Original: In order to capture fine-grained token interactions that might be missing in the abstractive attention,
Revised: In order to capture fine-grained token interactions that might be missing in the abstractive attention,

Original: (Zhu & Soricut, 2021; Zhu et al., 2021; Nguyen et al., 2021;
Revised: (Zhu & Soricut, 2021; Zhu et al., 2021; Nguyen et al., 2021;

Original: Ren et al., 2021) use a process akin to leverage a full-resolution local attention to form a multi-scale multi-range attention (MSMRA).
Revised: Ren et al., 2021) use a process akin to leverage a full-resolution local attention to form a multi-scale multi-range attention (MSMRA).

Original: These approaches, however, do not compose the local full-resolution attention and global reduced-resolution attentions in a series.
Revised: These approaches, however, do not compose the local full-resolution attention and global reduced-resolution attentions in a series.

Original: Absent from this literature, serial composition and positional embedding would improve accuracy while preserving the efficiency (see the following sections for how we address this enhancement).
Revised: Absent from this literature, serial composition and positional embedding would improve accuracy while preserving the efficiency (see the following sections for how we address this enhancement).

Revised: More recently, there also have been proposed sequence models with structures or strong inductive biases achieving significant improvements in benchmarks such as (Gu et al., 2021; Mehta et al., 2022; Ma et al., 2022;

Original:  (Press et al., 2021) propose the position-infused attention (PIA), which is an attention module with a layer-wise positional embedding applied only to queries and keys, to address token representation reusability issue in increasing window attention due to positional information draft, while avoiding expensive computation of relative position encoding (Dai et al., 2019b).
Revised: Li et al., 2022). (Press et al., 2021) propose the position-infused attention (PIA), which is an attention module with a layer-wise positional embedding applied only to queries and keys, to address token representation reusability issue in increasing window attention due to positional information draft, while avoiding expensive computation of relative position encoding (Dai et al., 2019b).

Original: Another study of adding positional embedding to values is conducted in (Tsai et al., 2019), concluding that positional embedding on value does not lead to performance improvement.
Revised: Another study of adding positional embedding to values is conducted in (Tsai et al., 2019), concluding that positional embedding on value does not lead to performance improvement.

Original: We extends these ideas in( CST) and apply in MSMRA.
Revised: We extend these ideas in CST and apply in MSMRA.

Original:  P RELIMINARIES 3. T RANSFORMERS AND M ULTI -H EAD S ELF -A TTENTION
Revised:  P RELIMINARIES 3. T RANSFORMERS AND M ULTI -H EAD S ELF -A TTENTION

Original: A Transformer layer consists of a multi-head self-attention sublayer followed by a feed-forward network (Vaswani et al., 2017) with an optional cross-attention sublayer in case of the decoder.
Revised: A Transformer layer consists of a multi-head self-attention sublayer followed by a feed-forward network (Vaswani et al., 2017) with an optional cross-attention sublayer when used as a decoder.

Original: The multi-head self-attention is defined as the concatenation of the self-attention output of all attention heads:
Revised: The multi-head self-attention is defined as the concatenation of the self-attention output of all attention heads:

Original: [Equation]
Revised: [Equation]

Original: ) where [ · ] r denotes concatenation in the r th dimension, and each of the outputsY h ∈ R N ×
Revised: where [ · ] r denotes concatenation in the r th dimension, and each of the outputs

Original:  d h is a scaled dot-product attention computed from the input X ∈ R N × D as
Revised: Y h ∈ R N × d h is a scaled dot-product attention computed from the input X ∈ R N × D as

Original: [Equation]
Revised: [Equation]

Original: In Eq.equation 2, Q h = XW q,h are queries, K h = XW k,h are keys, and V h = XW v,h are values expressed as linear transformations of the input X as W · ,h ∈ R D 
Revised: In Eq. 2, Q h = XW q,h , K h = XW k,h , and V h = XW v,h are queries, keys and values, respectively, expressed as linear transformations of the input X by W · ,h ∈ R D × d h .

Original: × d h . We assume the queries, keys, and values have the same hidden dimension: d h = D/H .
Revised: We assume the queries, keys, and values have the same hidden dimension: d h = D/H .

Original: For the rest of the paper, we omit the head index h and scaling factor 1 / √ d for simplicity.
Revised: For the rest of the paper, we omit the head index h and scaling factor 1 / √ d for simplicity.

Original: We denote the query, key and value at some position index i by k i , v i , q i ∈ R 1 × d , respectively.
Revised: We denote the query, key and value at some position index i by k i , v i , q i ∈ R 1 × d , respectively.

Original: In this context, the attention output at the i th token position y
Revised: In this context, the attention output at the i th token position y

Original: i ∈ R 1 ×
Revised: i ∈ R 1 ×

Original: d h corresponds to
Revised: d h corresponds to

Original: [Equation]
Revised: [Equation]

Original: Due to the nonlinearity and normalization property of the softmax function, QK ⊤ must be computed to obtain the attention weight, followed by value aggregation through the attention weights, AV , resulting in quadratic complexity with respect to the sequence length N for the self-attention; i.e., O ( N 2 ) computational and memory complexities.
Revised: Due to the nonlinearity and normalization property of the softmax function, QK ⊤ must be computed to obtain the attention weight, followed by value aggregation through the attention weights, AV , resulting in O ( N 2 ) complexity with respect to the sequence length N for the self-attention.

Original: 3. A BSTRACTIVE A TTENTIONS AND M ULTI -S CALE M ULTI -R ANGE A TTENTION (MSMRA)
Revised: 3. A BSTRACTIVE A TTENTIONS AND M ULTI -S CALE M ULTI -R ANGE A TTENTION (MSMRA)

Original: Abstractive attentions are the family of efficient attention approaches in which the lengths of the attention operands are reduced to M( < N) by applying an abstraction function ϕ ( · ) , resulting in reducedcomplexity of the attention  while retaining the form of basic attention computation in Eq.3.
Revised: We refer to the family of efficient attention approaches in which the lengths of the attention operands are reduced to M < N by applying an abstraction function ϕ ( · ) as abstractive attentions . This approach results in reduced attention complexity while retaining the form of basic attention computation in Eq.

Revised: Many recent approaches follow this template (Wang et al., 2020; Ma et al., 2021;

Revised: Dosovitskiy et al., 2020).

Original: We focus onthe cases that all the operands of attention, i.e., query, key, and value, are abstracted.
Revised: We focus on cases where the operands of attention, i.e., query, key, and value, are abstracted, noting that there are other possible choices, e.g., abstracting only query and key.

Original: In such cases, the attention mechanism is reduced to
Revised: In such cases, the attention mechanism is reduced to

Original: [Equation]
Revised: [Equation]

Original: 4) where Q , K , and V are  queries, keys, and values, respectively,abstracted, q i ′ is i ′
Revised: where Q , K , and V are abstracted queries, keys, and values, respectively, q i ′ is i ′

Original: th row of Q , and y i ′ is the attention output token with the abstracted query token q i ′ obtained by
Revised: th row of Q , and y i ′ is the attention output token with the abstracted query token q i ′ obtained by

Original: [Equation]
Revised: [Equation]

Original: In order to restore the resolution of the output,  we define a one-to-many mapping function ψy ( · ) as[Equation]
Revised: In order to restore the resolution of the output, since the query is abstracted, we define a one-to-many mapping function ψ ( · ) as

Original: [Equation]
Revised: [Equation]

Original: Detailed description of abstractive attention is presented in Section A.1.
Revised: A more detailed description of abstractive attention is presented in Section A.1.

Original: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically come at the cost of degraded benchmark performance.
Revised: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically result in degraded benchmark performance.

Original: However, Transformerbased models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer.
Revised: However, Transformer-based models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer.

Original: In fact, these models can outperform the standard Transformer on certain tasks, while still maintaining efficiency (Zhu & Soricut, 2021; Zhu et al., 2021;
Revised: In fact, these models can outperform the standard Transformer on certain tasks, while still maintaining efficiency (Zhu & Soricut, 2021; Zhu et al., 2021;

Original: Nguyen et al., 2021; Ren et al., 2021).
Revised: Nguyen et al., 2021; Ren et al., 2021).

Original: While other types of MSMRA are described in Section A.2, our proposed attention mechanism is essentially an MSMRA of the form
Revised: While other types of MSMRA are described in Section A.2, our proposed attention mechanism is essentially an MSMRA of the form

Original: [Equation]
Revised: [Equation]

Original: where y l ,i is the local attention output and y g ,i is the global attention output.
Revised: where y l ,i is the local attention output and y g ,i is the global attention output, leveraging a onedimensional version of the token abstraction used in (Dosovitskiy et al., 2020) for the global attention, but with additional composition of local attention. [Figure]

Original:  C OMPOSITE S LICE T RANSFORMER
Revised:  C OMPOSITE S LICE T RANSFORMER

Original: We describe the key ideas of Composite Slice Attention ( CSA ) and CST network, a Transformerbased model with CSA replacing the full softmax dot-product attention in the standard Transformer.
Revised: We describe the key ideas of Composite Slice Attention ( CSA ) and CST network, a Transformerbased model with CSA replacing the full softmax dot-product attention in the standard Transformer.

Original: CSA leverages both full-resolution attention in limited ranges and abstracted attention to capture long-range interactions.
Revised: CSA leverages both full-resolution attention in limited ranges and abstracted attention to capture long-range interactions.

Original: Unlike previous approaches, the multi-scale, multi-range attentions are combined through function composition in a serial connection, which allows information routing across representations at different scales and improves expressiveness of the model.
Revised: Unlike previous approaches, the multi-scale, multi-range attentions are combined through function composition in a serial connection, which allows information passing across representations at different scales and improves expressiveness of the model.

Original: See Figure 1 for an illustration of theCST network and the CSA module.
Revised: See Figure 1 for an illustration of the CSA module and Appendix A.3 for full architecture diagrams.

Original: While focusing on the core components of CST in this section, we expand on further details of CSA in various appendices.

Original: First, we present the extension of the local attention in Section A.4, to address the fine-grained contextual fragmentation problem (Dai et al., 2019b) caused by hard segmentation in the stacked slice representation of a sequence, and improve token-level dependency modeling.

Original: In Section A.5, we design a multi-scale causal mask and modify the CSA extension to prevent undesirable look-ahead due to the query abstraction, and guarantee true causality for autoregressive application.

Original: In Section A.6, we mathematically motivate CSA with approximation theory by arguing that a composition of attentions can express a larger functional space than common alternatives.

Original: 4. S EQUENCE R EPRESENTATION AS S TACKED S LICES AND CSA
Revised: 4. S EQUENCE R EPRESENTATION AS S TACKED S LICES AND CSA

Original: In a high-level categorization, the multi-scale multi-range attention of the CST corresponds to the combination of block-wise local window attention (Beltagy et al., 2020) and patch-based global attention (Dosovitskiy et al., 2020) in a one-dimensional form.
Revised: In a high-level categorization, the multi-scale multi-range attention of the CST corresponds to the combination of block-wise local window attention (Beltagy et al., 2020) and patch-based global attention (Dosovitskiy et al., 2020) in a one-dimensional form.

Original: CSA layer starts with converting the input sequence X ∈ R N × D into a stack of slices S ∈ R N/L × L × D by slicing it with a fixed length
Revised: CSA layer starts with converting the input sequence X ∈ R N × D into a stack of slices S ∈ R N/L × L × D by slicing it with a fixed length

Original: L , implemented simply as a Reshape operation.
Revised: L , implemented simply as a Reshape operation.

Original: Two attentions with different granularity are then performed sequentially in each direction.
Revised: Two attentions with different granularity are then performed sequentially in each direction.

Original: First, the local attention is performed across the token within each slice while considering the number of slices as a batch, andthen the number of slices N/L is combined with the batch size and parallelized together:
Revised: First, the batch size B and the number of slices N/L is combined as a new batch size BN/L , so that we parallelize the local attention for all slices. Then, the local attention is performed across the tokens within each of these new batches:

Original: [Equation]
Revised: [Equation]

Original:  where Q l , K l , and V l ∈ R N/L × L × d are the queries, keys, and values for each local attention head, computed as SW l ,q , SW l ,k , and SW l ,v , respectively, with W l ∈ R D × d .
Revised: (8) where Q l , K l , and V l ∈ R BN/L × L × d are the queries, keys, and values for each local attention head, computed as SW l ,q , SW l ,k , and SW l ,v , respectively, with W l ∈ R D × d .

Original: Next, the dimension of length L in the local attention output is collapsed using an abstraction function ϕy to get the slice embedding S ∈ R N/L × D .
Revised: Next, the dimension of length L in the local attention output is collapsed using an abstraction function ϕ to get the slice embedding S ∈ R BN/L × D .

Original: Specifically, we use a simple mean pooling S s = ϕ
Revised: Specifically, we use a simple mean pooling S s = ϕ

Original: y ( Y l ,s ) =
Revised: ( Y l ,s ) =

Original: (cid:80) L − 1 l =

Revised: (cid:80) L − 1 l =

Original: m s,l Y s,l / (cid:80) L − 1 l =

Revised: m s,l Y s,l / (cid:80) L − 1 l =

Original: m s,l , where s and l denote the slice index and the token index, respectively, and m ∈ R N/L × L is the stack of slices of the binary attention mask that is optionally given in the case that the input sequence is padded; e.g., when a data batch has samples with different lengths.
Revised: m s,l , where s and l denote the slice index and the token index, respectively, and m ∈ R N/L × L is the stack of slices of the binary attention mask that is optionally given in the case that the input sequence is padded; e.g., when a data batch has samples with different lengths.

Original: Normalization with the sum of the binary mask, i.e., the number of nonzero mask values, instead of the slice length L , avoids biases in the mean computation induced by masked tokens.

Original: Then the second attention along the slice dimension is performed to model full-range slice-wise interaction in a reduced resolution:
Revised: Then the second, global attention along the slice dimension is performed to model full-range slice-wise interaction in a reduced resolution:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where Q g , K g , and V g is the abstracted queries, keys, and values for the global attention obtained by applying W g ,q , W g ,k , and W g ,v to S .
Revised: where Q g , K g , and V g is the abstracted queries, keys, and values for the global attention obtained by applying W g ,q , W g ,k , and W g ,v to S .

Original: Finally, the output of the global attention is interpolated to the original resolution followed by being added to the local output:
Revised: Finally, the output of the global attention is broadcasted to the original resolution and added to the local output:

Original: [Equation]
Revised: [Equation]

Original: (10) where ψ ( Y g ,i ) is a broadcasting function that restores only the sequence length, i.e., Y g ,i
Revised: where ψ ( Y g ,i ) is a broadcasting function that restores only the sequence length, i.e., Y g ,

Original:  =
Revised: i =

Original: Y g ,i ′ for i ∈ Ω i ′ , since the output of the local attention still holds the full resolution.
Revised: Y g ,i ′ for i ∈ Ω i ′ , since the output of the local attention still holds the full resolution.

Original: This process can be implemented as a simple broadcast-add operation as shown in Figure 1a.
Revised: This process can be implemented as a simple broadcast-add operation.

Revised: Slice Extension: Addressing Fine-Grained Context Fragmentation

Revised: The local attention of the stacked slice representation of the sequence S is strictly bounded, resulting in potential context fragmentation (Dai et al., 2019b).

Revised: Although the global attention models the slice-wise interactions, it may not be sufficient to capture the important fine-grained token dependencies.

Original: 4. M ULTI - SCALE VOLATILE INSTANT POSITIONAL EMBEDDING (MS-VIPE)
Revised: To allow token-level interaction between slices, we slightly extend each slice into its neighbors, allowing slices to share tokens. This extended local attention can be computed by having keys K l, ext and values V l, ext transformed from extended stacked slices S ext , where α denotes the extension ratio. The extension can be implemented by concatenating shifted-slices in keys and values: [Equation]

Revised: 11) for α ≤ 3 where the Python notation of indices selection is used.

Revised: Slice Extension and Multi-Scale Causal Mask for Autoregressive Sequence Modeling

Revised: In order to apply CST to autoregressive sequence processing tasks, we propose custom causal masking and slice extension schemes.

Revised: For the local token-level attention, we apply an L × L causal mask map with −∞ above the main diagonal and zero elsewhere to the score tensor Q l K ⊤ l .

Revised: However, since the leftmost tokens in each slice have few (and possibly zero) tokens to attend to, we extend the keys and values only on the left-hand side to encourage better fine-grained dependency modeling, i.e.,

Revised: [Equation]

Revised: [Equation]

Revised: For the global attention, an extra care must be taken to prevent leftward information leakage while computing the slice embedding via mean pooling.

Revised: The diagonal elements in the N/L × N/L casual mask are set to be −∞ in contrast to the local counterpart.

Revised: In addition, at the slice index t , the shifted query Q g ,t − 1 is used for query instead of Q g ,t (note that K g ,t is handled by the global casual mask).

Revised: Increased Expressiveness of CST We mathematically motivate the improved performance of

Revised: CST over competing efficient Transformer models.

Revised: We show that a given function that CST is able to represent is ϵ away from a rational function with Euclidean degree 4 d ( ϵ ) , while competing approaches such as H-Transformer-1D Zhu & Soricut (2021) can only approach a rational function with Euclidean degree d ( ϵ ) .

Revised: This implies that CST is more expressive than other approaches that do not involve compositions of multi-scale attentions.

Revised: We state the main result below and present the details in Appendix A.5.

Revised: Proposition 1.

Revised: For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )

Revised: =

Revised: O (log( ϵ )) such that,

Revised: [Equation]

Revised: where M corresponds to the total number of weights in CST , and S ϵM,d is the space of ratios of real analytic functions that are ϵ away from a rational function with Euclidean degree d with input in R M .

Revised: 4. M ULTI - SCALE VOLATILE INSTANT POSITIONAL EMBEDDING (MS-VIPE)

Original: Since we reduce the input lengths of both global and local attentions, the full positional embedding of the maximum input sequence length is no longer necessary.
Revised: Since we reduce the input lengths of both global and local attentions, the full positional embedding of the maximum input sequence length is no longer necessary.

Original: For the local attention, we can limit the positional embedding length to the attention range; i.e., the slice length L , sharing the embedding across slices.
Revised: For the local attention, we can limit the positional embedding length to the attention range (i.e., the slice length L ), sharing the embedding across slices.

Original: In addition, as we aggregate the tokens from each slice for the global attention resulting in different granularity, it is more natural to have a separate positional embedding of length N/L at the scale of slice embedding instead of aggregating the full-resolutionfull-length positional embedding by the same process for the token embedding.
Revised: In addition, as we aggregate the tokens from each slice for the global attention, it is more natural to have a separate positional embedding of length N/L at the scale of slice embedding instead of aggregating the full-resolution positional embedding with the same process as the token embedding.

Original: To that end, CST utilizes the multi-scale positional embedding P l ∈ R 1 × L × D and P g ∈ R N/L × D . Although it is similar to the positional embedding used in (Han et al., 2021), there are crucial differences. First, instead of adding the positional embedding to the stacked slices of token embedding at the embedding layer and having aggregated positional information as the layers stack up (Press et al., 2021), CST applies them instantly at the corresponding attentions in each layer before the linear transformations.
Revised: To this end, CST uses two positional embeddings P l ∈ R 1 × L × D and P g ∈ R N/L × D applied at different scales, in a fashion similar to that used by (Han et al., 2021), but with a few crucial differences: first, instead of adding the positional embedding to the stacked slices of token embedding at the embedding layer and having aggregated positional information as the layers stack up (Press et al., 2021), CST applies them instantly at the corresponding attentions in each layer before the linear transformations.

Original: Second, as the positional embedding in all layers are added, they can be accumulated over the layers and  undesirably dominate the contextual information at top layers which potentially leads to performance degradation.In order to prevent this issue, CST does not add the positional embedding to values.
Revised: Second, the positional embeddings are applied for the queries and the keys, not for the values, to prevent accumulation of positional information in sequence representations. Since the positional embedding in all layers are added, they can be accumulated over the layers and can undesirably dominate the contextual information at top layers which potentially leads to performance degradation.

Original: Our experiments in Section A.7.2 show that the multi-scale volatile instant positional embedding (MS-VIPE) is more effective as compared to the conventional absolute full-length positional embedding.
Revised: Our experiments in Section A.6.2 show that the multi-scale volatile instant positional embedding (MS-VIPE) is more effective as compared to the conventional absolute full-length positional embedding.

Original: Equations 8 and 9 can now be rewritten as:
Revised: Equations (8) and (9) are rewritten as:

Original: [Equation]
Revised: [Equation]

Revised: For the extended local attention, we modify the corresponding positional embedding P l to have the extended length αL , similarly to K l, ext .

Original: 4. C OMPLEXITY AND P ARAMETER R EDUCTION IN CST
Revised: 4. C OMPLEXITY AND P ARAMETER R EDUCTION IN CST

Original: Compared to the quadratic O ( N 2 ) complexity of the standard Transformer, CST has linear plus decimated quadratic complexity O ( NL) +O ( N 2 /L 2 ) .
Revised: CST has linear plus decimated quadratic complexity O ( NL + N 2 /L 2 ) compared to the O ( N 2 ) complexity of the standard Transformer.

Revised: However, in a practical range of sequence lengths, e.g., from a few hundreds to a few tens of thousands, CST has a comparable or better efficiency than other efficient transformers with linear complexity O ( NM ) with typical choices of M , e.g., from 64 to 256 , or higher for even longer sequences.

Original: However, as shown in Section A.3, since the slice length L  is typically less than the abstraction lengths M in other linear-complexity models,
Revised: This is because the slice length L for CST is typically less than the abstraction lengths M of other linear-complexity abstractive attentions. See Section A. for more details.

Original:  CST has comparable efficiency to other efficient Transformer models for practical lengths of input sequences.
Revised: Furthermore, unlike most efficient transformers that can have similar or higher complexity than the standard transformers with short input sequences, CST has better efficiency even in such cases.

Revised: See Section A.4 for a complexity comparison.

Original: Since the global attention in CST takes thecollapsed local attention outputs as inputs, it requires  a slight increase in computation for linear transformers to get queries, keys, and values, corresponding to 3( N/L ) d 2 .
Revised: Additionally, since the global attention in CST takes the local attention output as input, it requires only a slight in crease in computation compared with linear attention (i.e. 3( N/L ) d 2 versus ( NMd ) to get queries, keys, and values.

Revised: Using extended local attention with extension ratio α leads to a complexity increase for the local attention to O ( αNL ) and a parameter count increase of αLD for the local positional embedding.

Original: Two sets of W q , W k , W v transformations in CST may also affect the parameter count.
Revised: Two sets of W q , W k , W v transformations in CST may also affect the parameter count.

Original: While TNT (Han et al., 2021) reduces the hidden dimension for local attention to limit the network size increase,in CST we use the same dimension for representations at different scale with tying the weights instead , resulting in no increase  parameter count.
Revised: While TNT (Han et al., 2021) reduces the hidden dimension for local attention to limit the network size increase, we keep the same dimension for representations at different scales and share the weights between local and global attentions, resulting in no increase in parameter count.

Original: Another benefit of having the stacked slice representation is the reduction in storage requirement for the positional embedding.
Revised: Another benefit of having the stacked slice representation is the reduction in storage requirement for the positional embedding.

Original: Since the lengths for  attentions are L and N/L for local and global attentions, respectively, CST has fewer parameters ( ( L + N/L )· D ) than that of the conventional positional embedding number N · D .
Revised: Since the lengths for local and global attentions are L and N/L , respectively, CST has fewer parameters (( L + N/L ) D ) than that of a conventional positional embedding ( ND ) .

Original: This benefit can be significant for tasks with very long sequences while it can be relatively small when the network size is very large.
Revised: This benefit can be significant for tasks with very long sequences while it can be relatively small when the network size is very large.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the computational efficiency and sequence modeling capability of CST , we evaluate our model in three different contexts: (1) bidirectional long-range dependency modeling on classification tasks (2) word-level auto-regressive language modeling and (3) masked language modeling and  natural language understanding on short sequences.
Revised: To demonstrate the computational efficiency and sequence modeling capability of CST , we evaluate our model in three different contexts: (1) bidirectional long-range dependency modeling on classification tasks (2) word-level auto-regressive language modeling and (3) masked language modeling and transfer learning to natural language understanding on short sequences.

Original: Throughout this section, unless stated otherwise, it is implied that we train each model from scratch, reporting its test performance from a model with the best validation results.
Revised: Throughout this section, unless stated otherwise, it is implied that we train each model from random initialization, reporting its test performance from the model with the best validation result in each case.

Original: 5. B IDIRECTIONAL L ONG -R ANGE D EPENDENCY M ODELING
Revised: 5. B IDIRECTIONAL L ONG -R ANGE D EPENDENCY M ODELING

Original: Datasets and Baselines The Long Range Arena (LRA) benchmark (Tay et al., 2020b) is a suite of classification tasks that evaluate long-range dependency modeling capabilities with datasets from several different modalities with lengths ranging from 1k to 16k.
Revised: Datasets and Baselines The Long Range Arena (LRA) benchmark (Tay et al., 2020b) is a suite of classification tasks that evaluate long-range dependency modeling capabilities with datasets from several different modalities with lengths ranging from 1k to 16k.

Original: We evaluate CST on the five tasks broadly used in literature, ListOps, Text, Retrieval, Image , and Pathfinder , where the maximum sequence length is 4k.
Revised: We evaluate CST on the five tasks broadly used in literature, ListOps, Text, Retrieval, Image , and Pathfinder , where the maximum sequence length is 4k.

Original: We compare CST ’s efficiency and performance against several state-ofthe-art efficient Transformer models.
Revised: We compare CST ’s efficiency and performance against several state-ofthe-art efficient Transformer models.

Original: Specifically, we evaluate our model and other Transformermodels from a group of non-abstractive attention methods: FNet (Lee-Thorp et al., 2021), Linear
Revised: Specifically, we evaluate our model and other Transformer

Original: Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020), and another group of abstractive attention methods: Reformer (Kitaev et al., 2019), Linformer (Wang et al., 2020), Luna (Ma et al., 2021), Nystr ¨ omformer (Xiong et al., 2021), that of multi-scale multi-range attention [Figure]
Revised: 5 Relative Speed

Original: L R A S c o r e
Revised: L R A S c o r e

Original: FNet
Revised: FNet

Original: H-Transformer-1D
Revised: H-Transformer-1D

Original: Linear Transformer
Revised: Linear Transformer

Original: Linformer
Revised: Linformer

Original: LS-Transformer
Revised: LS-Transformer

Original: Luna Nystromformer
Revised: Luna Nystromformer

Original: Performer
Revised: Performer

Original: Reformer
Revised: Reformer

Original: Transformer
Revised: Transformer

Original: CST (L=16) CST (L=8) CST (L=8)
Revised: CST (L=16) CST (L=8) CST (L=8)

Original: CST (L=16) 0.
Revised: CST (L=16) 0.

Original: P a r a m e t e r C o un t ( M ) methods: H-Transformer-1D (Zhu & Soricut, 2021), Long Short Transformer (Zhu et al., 2021), and
Revised: P a r a m e t e r C o un t ( M ) [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: FNet

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Linear

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Performer 37.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Reformer 18.

Revised: Linformer 38.

Revised: Luna 38.

Revised: Nystr ¨ omformer 37.

Revised: H-Transformer-1D 38.

Revised: Long Short Transformer 38.

Revised: Scatterbrain 38.

Revised: CST ( L = 8 ) 37.

Revised: CST ( L = 16 ) 37. models from a group of non-abstractive attention methods: FNet (Lee-Thorp et al., 2021), Linear

Revised: Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020), Cosformer (Qin et al., 2021), a group of abstractive attention methods: Reformer (Kitaev et al., 2019), Linformer (Wang et al., 2020), Luna (Ma et al., 2021), Nystr¨omformer (Xiong et al., 2021), and another group of multi-scale multi-range attention methods: H-Transformer-1D (Zhu & Soricut, 2021), Long Short

Original:  Scatterbrain (Chen et al., 2021a), as well as the standard Transformer (Vaswani et al., 2017).
Revised: Transformer (Zhu et al., 2021), and Scatterbrain (Chen et al., 2021a), as well as the standard Transformer (Vaswani et al., 2017).

Original: We outline experimental setup for the Long Range Arena in Appendix A.7.
Revised: We outline experimental setup for the Long Range Arena in Appendix A.6.1.

Revised: Our setup is favorable to the baseline Transformer to provide a fair condition to the challengers, explaining the difference in performances of Transformers from previous work.

Original: Results Our experimental results on the LRA benchmark are given in Table 1.
Revised: Our experimental results on the LRA benchmark are given in Table 1.

Original: We observe that
Revised: We observe that

Original: CST surpasses the state-of-the-art efficient Transformers with large margins.
Revised: CST surpasses the state-of-the-art efficient Transformers with large margins.

Original: The best performance is achieved using a slice length 8 , outperforming the Transformer baseline by 4 . 23 points in average score across all tasks.
Revised: The best performance is achieved using a slice length 8 , outperforming the Transformer baseline by 4 . 23 points in average score across all tasks.

Original: The largest improvements are observed on the Text, Retrieval , and Image tasks, while a minimal degradation is observed on the ListOps task.
Revised: The largest improvements are observed on the Text, Retrieval , and Image tasks, while a minimal degradation is observed on the ListOps task.

Original: In addition, as discussed in
Revised: In addition, as discussed in

Original: Section 4.1, CST has the added benefit of requiring fewer parameters for a given model size thanks to its positional embedding characteristics, although this benefit may be marginal as the network size increases. 
Revised: Section 4.1, CST has the added benefit of requiring fewer parameters for a given model size thanks to its positional embedding characteristics, although this benefit may be marginal as the network size increases. [Table]

Original: Aside from the average score across all tasks, we measure relative speed, and relative GPU memory usage for each model under comparison on a NVIDIA Tesla V100 GPU with 32GB memory on Retrieval dataset with the sequence length of 4k and report the results in Figure 2.
Revised: [Table] [Table] [Table] We also measure relative speed, and relative GPU memory usage for each model under comparison on a NVIDIA Tesla V100 GPU with 32GB memory on Retrieval dataset with the sequence length of 4k and report the results in Figure 2.

Original: Here, relative speed is computed as the number of training steps per second divided by that of Transformer, and GPU memory usage is measured using the PyTorch command, torch.cuda.max_memory_allocated() , both on the Retrieval task.
Revised: Here, relative speed is computed as the number of training steps per second divided by that of Transformer, and GPU memory usage is measured using the PyTorch command, torch.cuda.max_memory_allocated() , both on the Retrieval task.

Original: It is clearly seen that CST outperforms all models while having competitive computation and memory efficiency to other efficient Transformer models.
Revised: It is clearly seen that CST outperforms all models while having competitive computation and memory efficiency to other efficient Transformer models.

Original: Wenote that these measures can be different according to implementation and hardware under test. We present an ablation study  on  LRA benchamrk in Section A.7.2.
Revised: We present an ablation study regarding the choice of positional embedding and aggregation method on the LRA benchmark in Section A.6.2.

Original: As can be seen in Table 5 and Table 6, all variants of CST outperformed the baseline models, except that switching the MSMRA composition order, i.e. attentions in global-local order, performs slightly worse than Long Short Transformer (Zhu et al., 2021).
Revised: In summary, variants of CST consistently outperform all other efficient models (Table 7), except that switching the MSMRA composition order, i.e. attentions in global-local order, performs slightly worse than Long Short Transformer (Zhu et al., 2021).

Original: We demonstrate that the composition of MSMRA in the local-global order and the proposed positional embedding are the most contributing components  to the accuracy.
Revised: We demonstrate that the composition of MSMRA in the local-global order and the proposed positional embedding are the most important components contributing to CST ’s accuracy.

Original: While the best overall performance is achieved with L = 8 , we find that the slice length L has less effect than other hyperparameters.
Revised: While the best overall performance is achieved with L = 8 , we find that the slice length L has less effect than other hyperparameters.

Original: [Table] [Table] [Table]
Revised: We also find that attention outperforms an MLP-based local token mixing scheme and that mean pooling aggregation outperforms max pooling and linear projection (Table 8).

Original:  A UTOREGRESSIVE L ANGUAGE M ODELING
Revised: 5. A UTOREGRESSIVE L ANGUAGE M ODELING

Original: In this section, to evaluate its extendability aspractical language models, we conduct an autoregressive language modeling experiment on the word-level WikiText-103 dataset (Merity et al., 2016).
Revised: In this section, to evaluate its applicability as language model, we conduct an autoregressive language modeling experiment on the word-level WikiText-103 dataset (Merity et al., 2016).

Original: Dataset and Experimental Setup
Revised: Dataset and Experimental Setup

Original: The WikiText-103 (Merity et al., 2016) dataset is a collection of featured articles on Wikipedia.
Revised: The WikiText-103 (Merity et al., 2016) dataset is a collection of featured articles on Wikipedia.

Original: It has 103 M/ 218 K/
Revised: It has 103 M/ 218 K/

Original: 246 K words in training/validation/test sets, respectively.
Revised: 246 K words in training/validation/test sets, respectively.

Original: The task in this experiment is to predict the next word given the history of past words.
Revised: The task in this experiment is to predict the next word given the history of past words.

Original: We follow the experimental setup in (Nguyen et al., 2021) and train CST and carefully match the network configuration and training hyperparameters to those given in (Nguyen et al., 2021) and  compare  with their reported results.
Revised: We follow the experimental setup in (Nguyen et al., 2021) and that in (Chen et al., 2021a) for the context window length 256 and 1024 , respectively, and train CST s to compare the results with their reported results.

Original: Specifically, we train a small network with 16 layers, 8 attention heads, the model dimension of 128 , and the hidden dimension of the feed-forward network of 2 , 048 .

Original: We train the model for a total of 500 K steps.

Original: While the dataset allows longer context, we used the 256  context window length to matchwith the baselines.
Revised: While the dataset allows for a larger context, we used the 256 and 1024 context window lengths to match the baselines.

Original: Results For CST , we report the perplexities of the best performing model of the slice length of  32 and the extension ratio α of 3 . Results for other configurations can be found in Section A.8.2.
Revised: CST also uses the causal mask and slice embedding described in Section 4.1. More discussion of the experimental setup can be found in Appendix A.7.1. Results We report the perplexities of the best performing variant of CST ( L = 32 , α = 3 ) compared to other state-of-the-art efficient Transformer models in Table 2 and Table 4.

Original: As can be seen from Table 2, CST outperforms other efficient Transformer including Linear Transformer (Katharopoulos et al.,  2020),a kernel method-based linear-complexity model, and FMMformer (Nguyen et al., 2021), an MSMRA with kernel method-based global attention.
Revised: CST outperforms other efficient Transformer including Linear Transformer (Katharopoulos et al., 2020) and Performer (Choromanski et al., 2020), kernel method-based linear-complexity models, and FMMformer, while being comparable to Scatterbrain (Chen et al., 2021a), noting that the latter two [Table]

Original: While methods for the baselines are limited, the results indicate that composite attention can be more effective than kernel method-based attention.
Revised: [Table] are MSMRA-based models. In Table 3, we provide an ablation study demonstrating the impact of

Original: Our model reduces theautoregressive language modeling perplexity gap between the standard Transformer and efficient Transformers to 2 %.
Revised: L and α on validation and test perplexity. We observe that longer local attention length leads to better perplexities while being much shorter than the the context window length, e.g. CST with L = 32 , α = 3 has the local window length 64 that is 4 and 16 times smaller than the context window lengths 256 and 1024 , respectively. We believe that addressing the missing global context, as discussed in A.9, and a better hyperparameter search will improve the perplexity.

Original: We further observe consistent 1 .
Revised: We also observe consistent 1 .

Original: 5 x speed-ups across configurations, compared to the standard Transformer, as shown in 
Revised: 5 x speed-ups across configurations compared to the standard Transformer in Table 3.

Original: Section A.8.2. We also perform an ablation study for CST on the language modeling task.
Revised: Additional Experiment on PG-19 We additionally conduct an experiment on PG-19 dataset (Rae et al., 2019).

Original: While the full results can be found in Section A.8.2, we observe that for smaller context lengths, larger slice length with [Table] smaller local attention extension is more effective and efficient than having smaller slice length with larger local attention extension.
Revised: We use various combinations of sequence length N , slice length L , and extension ratio α to match the attention complexity of CST s to those of Transformers with N = 256 and 512 .

Original: In addition [Table] significant performance improvement is observed with increased context length to 1024, as shown in Table [Table]
Revised: While the validation and the test perplexities with discussion can be found in A.7.2, we observe CST consistently outperforms the Transformer counterparts with the same attention complexities.

Original: 5. B IDIRECTIONAL L ANGUAGE M ODELING AND T RANSFER L EARNING
Revised: 5. B IDIRECTIONAL L ANGUAGE M ODELING AND T RANSFER L EARNING

Original: We further evaluate CST on bidirectional language modeling and transfer learning  with relatively short input sequences, i [Table] e., N = 128 .
Revised: We further evaluate CST on bidirectional language modeling and transfer learning on GLUE benchmark with relatively short input sequences, i.e., N = 128 .

Original: Masked language modeling (MLM) was proposed in (Devlin et al., 2018) as a pretraining method of transformer encoder models and it greatly improves downstream natural language understanding (NLU) performances when fine-tuning the pretrained models.
Revised: Masked language modeling (MLM) was proposed in (Devlin et al., 2018) as a pretraining method of transformer encoder models and it greatly improves downstream natural language understanding (NLU) performances when fine-tuning the pretrained models.

Original: We follow the experimental setup of (Devlin et al., 2018) for conducting both pre-training and fine-tuning including datasets, masking rate, batch size, optimizer settings, and evaluation metrics with a few exceptions.
Revised: We follow the experimental setup of (Devlin et al., 2018) for conducting both pre-training and fine-tuning including datasets, masking rate, batch size, optimizer settings, and evaluation metrics with a few exceptions.

Original: We report the MLM validation perplexities and GLUE scores with accuracy on each task  in Table 3.
Revised: We report the MLM validation perplexities and GLUE scores with accuracy on each task for a variation of L and α in Table 5.

Original: Further experimental details, more experimental results, and discussion can be found in A.9.
Revised: Further experimental details, more experimental results, and discussion can be found in A.8.

Original: As can be seen in the table, CST consistently outperforms other efficient transformers and closes the gap to the baseline BERT model by 0.46 validation perplexity and 2.45 points in GLUE score.
Revised: CST consistently outperforms other efficient transformers and closes the gap to the baseline BERT model by 0.46 validation perplexity and 2.45 points in GLUE score.

Original: In the Section A.9, we discuss the sources of performance degradation compared to BERT and future research directions to address them.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present Composite Slice Transformer ( CST ), an efficient Transformer-based network equipped with composition of multi-scale multi-range attentions.
Revised: In this paper, we present Composite Slice Transformer ( CST ), an efficient Transformer-based network equipped with composition of multi-scale multi-range attentions.

Original: Using stacked slice representation of input sequences, a CST layer performs a set of local fine-grained attention and global coarsegrained attention in a sequential manner at a low complexity cost ofthe sum of linear and decimated quadratic complexities O ( NL + N 2 /L 2 ) .
Revised: Using stacked slice representation of input sequences, a CST layer performs a set of local fine-grained attention and global coarsegrained attention in a sequential manner at a low complexity cost of O ( NL + N 2 /L 2 ) .

Original: In addition to the complexity benefit, we also show that CST improves performance on various sequence modeling tasks.
Revised: In addition to the reduced complexity, we also show that CST improves performance on various sequence modeling tasks.

Original: Onthe Long Range Arena benchmark and word-level autoregressive language modelingbenchmark on WikiText-103 dataset, CST significantly surpasses strong baselines includingthe standard Transformer and the recently proposed efficient Transformers.
Revised: On Long Range Arena, word-level autoregressive language modeling on WikiText-103, masked language modeling, and natural language understanding benchmarks, CST significantly surpasses strong baselines including recently proposed efficient Transformers, and sometimes the standard

Original: After the observation that CST achieves superior modeling capability as well as relaxing the complexity bottleneck of Transformers, we observe some limitations of CST as summarized in A. and consider extending our research in several directions.

Original: First, we can easily generalize our work to multi-level slicing with a higher level than two.

Original: This will lead to more improvement in expressiveness of the model while adding more degree of freedoms for better efficiency while enabling its application to much longer sequences.

Original: Second, as discussed in Section A.5, more advanced design of autoregressive sequence modeling scheme will further improve the performance.

Original: Third, while CST is based on fixed-length slicing of the sequence, dynamic or semantic slicing would further improve applicability to non-stationary sequences.

Original: With end-to-end training, the model can learn the optimal slicing of the data for given tasks and provide layer-wise dynamic tokens.

Original: To this end, efficient realization of the dynamic slicing would be the most challenging task.

Original: Finally, as a meaningful next step, modification of CST to better fit vision tasks or training a CST -based large language model would provide general and practical impact.

Original: R EPRODUCIBILITY S TATEMENT

Original: Model Implementation We implemented the composite slice attention (CSA) and formed the composite slice transformer (CST) by replacing the self-attention block with the CSA block in the baseline transformer network.

Original: We are following our internal open-source policy and trying to open-source the code, at least the model implementation, to provide it to the reviewers in the rebuttal period.

Original: Nonetheless, since the implementation of CSA is straightforward in the sense that it does not involve any special kernels or operators, we believe any reader with a moderate knowledge and experience of deep learning would be able to implement it based on Section 4, especially,

Original: Figure 1a. For other models, we used the implementations from different github repositories, i.e., Linformer, Performer, Reformer from https://github.com/NVIDIA/transformer-ls ), FNet from https://github.com/huggingface/transformers , Nystr¨omformer, Linear Transformer from https://github.com/mlpen/Nystromformer , H-Transformer-1D from https://github.com/lucidrains/h-transformer-1d , and Scatterbrain from https://github.com/HazyResearch/fly .
Revised: Transformer.

Original: Benchmark Implementation and Hyperparameters To facilitate reproducibility, we used a well-established open-source code for each benchmark as below. For Long Range Arena benchmark, we followed the training and evaluation pipeline in the opensource code of Long-Short Transformer (Zhu et al., 2021) ( https://github.com/NVIDIA/ transformer-ls , including the dataset preparation. The experimental setup including the training hyperparameters are described in A.7.1.
Revised: We highlight limitations and potential directions for future work in A.9.

Original: As different models require different sets of hyperparameters, we described them in A.7.1 as well, mainly taken from the default or best settings from each paper.

Original: For autoregressive language modeling task, we used the language modeling pipeline implementation from https://github.com/IDSIA/lmtool-fwp .

Original: Here, we swap in our CST implementation in place of the softmax transformers and applied autoregressive masking as described in section

Original: A.5 with the hyperparameters described in A.8.1.

Original: For masked language modeling, we followed the training procedure and the training hyperparameter setting of BERT (Devlin et al., 2018) as described in A.9.1, up to the phase 1 pretraining, i.e., we only trained each model for 900k with the sequence length 128.

Original: We used the Huggingface MLM pretraining pipeline in huggingface transformers ( https://github.com/huggingface/transformers ) and BERT implementation ( https://github.com/huggingface/transformers/blob/main/src/ transformers/models/bert/modeling_bert.py ) as the baseline transformer model to form different transformer variants by replacing the self-attention module.

Original: For GLUE benchmark, with the MLM pretrained version of each transformer model, we mostly followed the evaluation procedure in Devlin et al.

Original: (2018) for GLUE benchmark, using the Jiant code ( https://github.com/nyu-mll/jiant ).

Original: As described in A.9.1, we used mean pooling of the model output sequence for classification input instead of using [CLS] token.

Original: A A PPENDIX A. A BSTRACTIVE A TTENTIONS
Revised: A A PPENDIX A. A BSTRACTIVE A TTENTIONS

Original: Abstractive attentions are the family of efficient attention approaches in which the lengths of the attention operands are reduced to M ( < N ) by applying an abstraction function, resulting in reduced complexity of the attention while retaining the form of basic attention computation in Eq. 3.
Revised: Abstractive attentions are the family of efficient attention approaches in which the lengths of the attention operands are reduced to M ( < N ) by applying an abstraction function, resulting in reduced complexity of the attention while retaining the form of basic attention computation in Eq. 3.

Original: Abstractive attentions can be further categorized to either resolution preserving or non-preserving , according to which operands are chosen to be abstracted.
Revised: Abstractive attentions can be further categorized to either resolution preserving or non-preserving , according to which operands are chosen to be abstracted.

Original: Resolution non-preserving attention is the result of abstracted queries, further producing abstracted output.
Revised: Resolution non-preserving attention is the result of abstracted queries, further producing abstracted output.

Original: This categorization is determined by the requirement of the given task.
Revised: This categorization is determined by the requirement of the given task.

Original: For example, tasks such as language modeling and machine translation require the full resolution at the output to be retained.
Revised: For example, tasks such as language modeling and machine translation require the full resolution at the output to be retained.

Original: In such cases, it is common to have only abstracted keys and values while the query resolution is retained; the abstractive attention can be expressed as[Equation] [Equation] [Equation]
Revised: In such cases, it is common to have only abstracted keys and values while the query resolution is retained; the abstractive attention can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where Ω j ′ denotes the abstraction range with the cardinality | Ω j ′ |
Revised: where Ω j ′ denotes the abstraction range with the cardinality | Ω j ′ |

Original: = M k for the j ′ th key abstraction k j ′ and ϕ ( · ) : K Ω j ′ ∈ R | Ω j ′ |× d h → k j ′ ∈ R 1 ×
Revised: = M k for the j ′ th key abstraction k j ′ and ϕ ( · ) : K Ω j ′ ∈ R | Ω j ′ |× d h → k j ′ ∈ R 1 ×

Original: d h is a many-to-one abstraction function.
Revised: d h is a many-to-one abstraction function.

Original: The abstracted values V j ′ can be expressed similarly.
Revised: The abstracted values V j ′ can be expressed similarly.

Original: The queries Q can be similarly abstracted to Q as
Revised: The queries Q can be similarly abstracted to Q as

Original: [Equation]
Revised: [Equation]

Original: (16) where q i ′ is the i ′ th row of Q .
Revised: 19) where q i ′ is the i ′ th row of Q .

Original: The attention mechanism is reduced to
Revised: The attention mechanism is reduced to

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where an attention output vector y i ′ is obtained at each abstract position i ′ .
Revised: where an attention output vector y i ′ is obtained at each abstract position i ′ .

Original: In order to restore the resolution of the output, we define a one-to-many mapping function ψ y as
Revised: In order to restore the resolution of the output, we define a one-to-many mapping function ψ y as

Original: [Equation]
Revised: [Equation]

Original: Resolution non-preserving abstraction is often used for the tasks where the full-resolution output is not needed such as sequence-level classification problem.
Revised: Resolution non-preserving abstraction is often used for the tasks where the full-resolution output is not needed such as sequence-level classification problem.

Original: However, in some cases, with additional processing leveraging representations at a lower layer, e.g., cross attention with input tokens, it is possible to restore the resolution (Dai et al., 2020; Jaegle et al., 2021).
Revised: However, in some cases, with additional processing leveraging representations at a lower layer, e.g., cross attention with input tokens, it is possible to restore the resolution (Dai et al., 2020; Jaegle et al., 2021).

Original: A. M ULTI -S CALE M ULTI -R ANGE A TTENTIONS (MSMRA)
Revised: A. M ULTI -S CALE M ULTI -R ANGE A TTENTIONS (MSMRA)

Original: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically come at the cost of degraded benchmark performance.
Revised: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically come at the cost of degraded benchmark performance.

Original: However, Transformerbased models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer; in fact, these models can outperform the standard Transformer on some benchmarks, while still maintaining efficiency (Zhu & Soricut, 2021;
Revised: However, Transformerbased models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer; in fact, these models can outperform the standard Transformer on some benchmarks, while still maintaining efficiency (Zhu & Soricut, 2021;

Original: Zhu et al., 2021; Nguyen et al., 2021;
Revised: Zhu et al., 2021; Nguyen et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: The local attention, also known as sliding window attention, limits the attention range to the vicinity of query locations.
Revised: The local attention, also known as sliding window attention, limits the attention range to the vicinity of query locations.

Original: That is, the key abstraction with the whole abstraction range and the query locationdependent abstraction function becomes K l ,i =
Revised: That is, the key abstraction with the whole abstraction range and the query locationdependent abstraction function becomes K l ,i =

Original: ϕ sliding k,i ( K )
Revised: ϕ sliding k,i ( K )

Original: =
Revised: =

Original: K ⊙ ( H ( i − j − w/ 2) − H ( i − j + w/ 2)) for the i th query token, where H is the Heaviside step function, w is the window length, and ⊙ is the element-wise product.
Revised: K ⊙ ( H ( i − j − w/ 2) − H ( i − j + w/ 2)) for the i th query token, where H is the Heaviside step function, w is the window length, and ⊙ is the element-wise product.

Original: This results in the following equation:
Revised: This results in the following equation:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For better computational efficiency, block-wise key abstraction in Equation equation 19 can be adopted as K l ,i = ϕ block k,i ( K )
Revised: For better computational efficiency, block-wise key abstraction in Equation equation 22 can be adopted as K l ,i = ϕ block k,i ( K )

Original: =
Revised: =

Original: K ⊙ ( H ( t i − j − w/ 2) − H ( t i − j + w/ 2)) for a block-wise attention where t i =
Revised: K ⊙ ( H ( t i − j − w/ 2) − H ( t i − j + w/ 2)) for a block-wise attention where t i =

Original: ( b − 1 / 2) w for the block index b such that ( b − 1) w ≤ i < bw .
Revised: ( b − 1 / 2) w for the block index b such that ( b − 1) w ≤ i < bw .

Original: For the global attention, abstractive attention with either positional abstractions (Zhu & Soricut, 2021;
Revised: For the global attention, abstractive attention with either positional abstractions (Zhu & Soricut, 2021;

Original: Yang et al., 2021; Ren et al., 2021) or contextual abstractions (Ma et al., 2021; Zhu et al., 2021) can be employed.
Revised: Yang et al., 2021; Ren et al., 2021) or contextual abstractions (Ma et al., 2021; Zhu et al., 2021) can be employed.

Original: The former can be loosely seen as having patch embedding in ViT (Dosovitskiy et al., 2020).
Revised: The former can be loosely seen as having patch embedding in ViT (Dosovitskiy et al., 2020).

Original: MSMRAs can also be categorized according to how the two attentions are combined.
Revised: MSMRAs can also be categorized according to how the two attentions are combined.

Original: While one approach involves concatenating the abstractions of multi-scale keys and values for a single attention operation (Zhu & Soricut,2021; Zhu et al., 2021; Yang et al., 2021):
Revised: While one approach involves concatenating the abstractions of multi-scale keys and values for a single attention operation (Zhu & Soricut,

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: another separates attentions at different scales and combining the outputs (Han et al., 2021)
Revised: another separates attentions at different scales and combining the outputs (Han et al., 2021)

Original: (possibly with some weighting coefficients):
Revised: (possibly with some weighting coefficients):

Original: [Equation]
Revised: [Equation]

Original: In this case, other non-attentive methods such as kernel method (Nguyen et al., 2021) can also be used for the global attention.
Revised: In this case, other non-attentive methods such as kernel method (Nguyen et al., 2021) can also be used for the global attention.

Original: CST belongs to the latter of the two approaches, where the local and global attentions are performed separately and their outputs are combined.
Revised: CST belongs to the latter of the two approaches, where the local and global attentions are performed separately and their outputs are combined.

Original: This is closely related to Transformer-In-Transformer ( TNT ) (Han et al., 2021); however, since TNT has a path for the local attention that is independent of the global attention, information exchange between patches is not allowed for the features in the local attention path.
Revised: This is closely related to Transformer-In-Transformer ( TNT ) (Han et al., 2021); however, since TNT has a path for the local attention that is independent of the global attention, information exchange between patches is not allowed for the features in the local attention path.

Original: Unlike TNT , the composition of multi-granular attentions in CST enables two-way information routing.
Revised: Unlike TNT , the composition of multi-granular attentions in CST enables two-way information passing.

Original: This is more suitable for modeling highly non-stationary data, such as natural language text data for example, where the locality assumption does not hold.
Revised: This is more suitable for modeling highly non-stationary data, such as natural language text data for example, where the locality assumption does not hold.

Revised: A. A RCHITECTURE OF C OMPOSITE S LICE T RANSFORMER N ETWORK

Revised: We present the overall architecture of CST alongside the detailed CSA architecture in Figure 3.

Revised: CST consists of a fine-grained local attention with computational complexity O ( NL ) and a coarse-grained global attention with complexity O

Revised: ( N 2 /L 2 ) .

Revised: A shared multi-scale positional embedding, MS-VIPE, is applied to all layers.

Original:  A. C ONSIDERATION ON P RACTICAL E FFICIENCY AND E FFECTIVENESS OF CST
Revised: CSA replaces self-attention sublayer in each Transformer block to form a Composite Slice Transformer ( CST ) network. [Figure] A. C ONSIDERATION ON P RACTICAL E FFICIENCY AND E FFECTIVENESS OF CST

Original: In this section, we provide an intuition for how and why a multi-scale multi-range attention with
Revised: In this section [Figure] we provide an intuition for how and why a multi-scale multi-range attention with

Original: O ( NL + N 2 /L 2 ) complexity can be a better alternative to the vanilla self-attention than abstraction attentions with linear complexities in terms of both complexity and modeling capability.
Revised: O ( NL + N 2 /L 2 ) complexity can be a better alternative to the vanilla self-attention than abstraction attentions with linear complexities in terms of both complexity and modeling capability.

Original: Improved expressiveness from composition of them is further discussed in Section A.6.
Revised: Improved expressiveness from composition of them is further discussed in Section A.5.

Original: A linear complexity can be easily considered to be more efficient than a quadratic complexity.
Revised: A linear complexity can be easily considered to be more efficient than a quadratic complexity.

Original: A linear complexity often accompanies another variable; i.e., the abstraction length M in the case of abstractive attention such as (Wang et al., 2020; Ma et al., 2021), resulting in O ( NM ) complexity.
Revised: A linear complexity often accompanies another variable; i [Figure] e., the abstraction length M in the case of abstractive attention such as (Wang et al., 2020; Ma et al., 2021), resulting in O ( NM ) complexity.

Original: It is obvious that the quadratic complexity O ( N 2 ) is higher than the linear complexity whenever the abstraction length
Revised: It is obvious that the quadratic complexity O ( N 2 ) is higher than the linear complexity whenever the abstraction length

Original: M is smaller than the sequence length, which is mostly true by the definition of abstraction .
Revised: M is smaller than the sequence length, which is mostly true by the definition of abstraction .

Original: However, when a notion of sequence decimation comes into play, we come to a little different conclusion.
Revised: However, when a notion of sequence decimation comes into play, we come to a little different conclusion.

Original: Consider an attention on an abstracted sequence of patch embeddings in (Dosovitskiy et al., 2020) or slice embeddings in CST .
Revised: Consider an attention on an abstracted sequence of patch embeddings in (Dosovitskiy et al., 2020) or slice embeddings in CST .

Original: Then the complexity of the abstractive attention becomes a decimated quadratic O ( N 2 /L 2 ) .
Revised: Then the complexity of the abstractive attention becomes a decimated quadratic O ( N 2 /L 2 ) .

Original: In addition to this, if a full-resolution local attention is used as in CST or (Han et al., 2021) as a multi-scale multi-range attention, the complexity becomes
Revised: In addition to this, if a full-resolution local attention is used as in CST or (Han et al., 2021) as a multi-scale multi-range attention, the complexity becomes

Original: O ( N 2 /L 2 + N/L · L 2 )
Revised: O ( N 2 /L 2 + N/L · L 2 )

Original: =
Revised: =

Original: O ( N 2 /L 2 + NL ) .
Revised: O ( N 2 /L 2 + NL ) .

Original: We plot the comparison of the linear complexity O ( NM ) and O ( N 2 /L 2 + NL ) in Figure 3a with several practical choices of the number of abstractions M and the decimation ratio, e.g., slice length, L .
Revised: We plot the comparison of the linear complexity O ( NM ) and O ( N 2 /L 2 + NL ) in Figure 4a with several practical choices of the number of abstractions M and the decimation ratio, e.g., slice length, L .

Original: Here, one can find that a decimated quadratic complexity attention can have better efficiency a linear complexity attention, when the sequence length is less than a few tens of thousands which is considered as a practical range of sequence lengths in many tasks and data types.
Revised: Here, one can find that a decimated quadratic complexity attention can have better efficiency a linear complexity attention, when the sequence length is less than a few tens of thousands which is considered as a practical range of sequence lengths in many tasks and data types.

Original: Figures 3b shows effective number of tokens for each abstraction for both cases with the same choices of M and L .
Revised: Figures 4b shows effective number of tokens for each abstraction for both cases with the same choices of M and L .

Original: While a decimation, e.g., slicing, retains constant numbers of tokens in each abstraction, the linear complexity attention methods that uses fixed number of abstraction has linearly[Figure] increasing effective number of tokens per abstraction with respect to the sequence length.
Revised: While a decimation, e.g., slicing, retains constant numbers of tokens in each abstraction, the linear complexity attention methods that uses fixed number of abstraction has linearly increasing effective number of tokens per abstraction with respect to the sequence length.

Original: Given a fixed hidden dimension, larger number of effective tokens per abstraction requires the abstraction process to compress more information, resulting in loss of potentially important information and negatively affecting the modeling capability.
Revised: Given a fixed hidden dimension, larger number of effective tokens per abstraction requires the abstraction process to compress more information, resulting in loss of potentially important information and negatively affecting the modeling capability.

Original: Since the complexity of CST has a decimated quadratic term N 2 /L 2 , the efficiency benefit compared to linear-complexity method keeps closing as the sequence length becomes larger [Figure] and eventually the efficiency order is reversed, as it can be seen in 3a.
Revised: Since the complexity of CST has a decimated quadratic term N 2 /L 2 , the efficiency benefit compared to linear-complexity method keeps closing as the sequence length becomes larger, and eventually the efficiency order is reversed, as it can be seen in 4a.

Original: However, we argue that our model can still be beneficial in two aspects.
Revised: However, we argue that our model can still be beneficial in two aspects.

Original: First, it is still more efficient than the standard Transformer.
Revised: First, it is still more efficient than the standard Transformer.

Original: Second, it is possible to find an optimal set of hyperparameters such as the slice length L and the extension ratio α which result in comparably efficient and more effective in modeling capability to the linearcomplexity counterpart, considering the discussion made in this section on the effective number of tokens per abstraction.
Revised: Second, it is possible to find an optimal set of hyperparameters such as the slice length L and the extension ratio α which result in comparably efficient and more effective in modeling capability to the linearcomplexity counterpart, considering the discussion made in this section on the effective number of tokens per abstraction.

Original: A. A DDRESSING F INE -G RAINED C ONTEXT F RAGMENTATION

Original: With the stacked slice representation of the sequence described in the previous section, the local attention is strictly bounded resulting in potential context fragmentation (Dai et al., 2019b).

Original: While the global attention models the slice-wise interaction to relax this issue, it may not be sufficient to capture the important fine-grained token dependencies.

Original: A simple and intuitive way to allow token-level interaction between slices is to have overlaps in the local attention.

Original: The overlapped local attention (or extended local attention) can be computed by having keys K l, ext and values V l, ext transformed from extended stacked slices S ext ∈ R N/L × αL × D , where α ( ≤ 3) denotes the extension ratio.

Original: The extension can be implemented by concatenating shifted-slices in keys and values as below

Original: [Equation]

Original: (22) where the Python notation of indices selection is used.

Original: As the local attention has longer range, the corresponding positional embedding P l needs to have the extended length αL for K l, ext .

Original: Extended local attention leads to the complexity increase to O ( αNL ) for the local attention and the parameter count increase to αLD for the local positional embedding.

Original: A. S LICE E XTENSION AND M ULTI -S CALE C AUSAL M ASK FOR A UTOREGRESSIVE S EQUENCE M ODELING

Original: For the use of CST in autoregressive sequence processing, e.g., autoregressive language modeling and generation, we additionally design causal masks and slice extension scheme.

Original: In the local token-level attention, it is sufficient to apply a L × L causal mask map to the score tensor Q l K ⊤ l , to guarantee causality where lower triangular including the diagonal elements to be zero and −∞ elsewhere.

Original: However, since the leftmost tokens in each slice has no or very little tokens to attend, we extend the keys and values only on the left-hand side to encourage better fine-grained dependency modeling, i.e.,

Original: [Equation]

Original: For the slice-wise global attention, extra care should be taken.

Original: First, to prevent leftward information leak during obtaining the slice embedding via mean pooling, the diagonal elements in the N/L × N/L mask map are set to be −∞ in contrast to the local counterpart.

Original: Second, at the slice index (coarse time step) t , the time-shifted query Q g ,t − 1 is used for query instead of Q g ,t for the same reason.

Original: In our experiments on autoregressive language modeling, we use this approach.

Original: Since, in the global attention, no information of tokens in current slice is taken into account because of the query shift and the attention mask excluding diagonal elements, there is missing information from coarse-grained modeling in the current design.

Original: We leave design of more advanced autoregressive modeling as a future work.

Original: As an alternative, one may consider using token-level accumulated query slice embedding at a larger computational cost, which is essentially equivalent to performing attention with abstraction in only keys and values.

Original: A. I MPROVED E XPRESSIVENESS OF CST S LICE A TTENTION
Revised: A. I MPROVED E XPRESSIVENESS OF CST S LICE A TTENTION

Original: Recall from Eq.equation 10 that the attention mechanism under consideration takes the form,
Revised: Recall from Eq.equation 10 that the attention mechanism under consideration takes the form,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Y l represents local attention, Y g represents global attention and ψ ( · ) is a (linear) one-to-many map.
Revised: where Y l represents local attention, Y g represents global attention and ψ ( · ) is a (linear) one-to-many map.

Original: In this section, we will compare CST with the H-transformer described in Zhu & Soricut (2021).
Revised: In this section, we will compare CST with the H-transformer described in Zhu & Soricut (2021).

Original: In particular, we argue that, for a given number of weights (degrees of freedom), our proposed attention mechanism, which involves the composition of attention mechanisms, is more expressive than the H-matrix-based approach proposed Zhu & Soricut (2021).Zhu & Soricut (2021) shares similar characteristics with CST , but does not involve composition, which explains our superior prediction performance (Section 5).
Revised: In particular, we argue that, for a given number of weights (degrees of freedom), our proposed attention mechanism, which involves the composition of attention mechanisms, is more expressive than the H-matrix-based approach proposed Zhu & Soricut (2021).Zhu & Soricut (2021) shares similar characteristics with CST , but does not involve composition, which explains our superior prediction performance (Section 5).

Original: To explain why this is so, we first define
Revised: To explain why this is so, we first define

Original: [Equation]
Revised: [Equation]

Original: [Equation] which is the standard attention mechanism found in Eq.equation 2.
Revised: (26) which is the standard attention mechanism found in Eq.equation 2.

Original: Here, X is generally a matrix or a 3-tensor; in the latter case, we apply attention independently to each matrix slice along the third dimension.
Revised: Here, X is generally a matrix or a 3-tensor; in the latter case, we apply attention independently to each matrix slice along the third dimension.

Original: Expressing our local and global attention using this notation leads to,
Revised: Expressing our local and global attention using this notation leads to,

Original: [Equation]
Revised: [Equation]

Original: [Equation] 27) where µ = (cid:2) 1 L , 1 L , ... , 1 L (cid:3) T is a fixed vector of mask values, S corresponds to the stack of L -length slices of X described in Section 4.1, and × 2 indicates a tensor product along the second dimension, i.e.,
Revised: where µ = (cid:2) 1 L , 1 L , ... , 1 L (cid:3) T is a fixed vector of mask values, S corresponds to the stack of L -length slices of X described in Section 4.1, and × 2 indicates a tensor product along the second dimension, i.e.,

Original: [Equation]
Revised: [Equation]

Original: which corresponds to mean pooling within each slice.
Revised: which corresponds to mean pooling within each slice.

Original: For the purpose of comparison, we focus on a 2-level H-transformer where each element at level 1 has L children at level 0.
Revised: For the purpose of comparison, we focus on a 2-level H-transformer where each element at level 1 has L children at level 0.

Original: In this case, it can be shown that the H-tranformer approximation takes the form of Eq.
Revised: In this case, it can be shown that the H-tranformer approximation takes the form of Eq.

Original: equation 24 with,
Revised: equation 25 with,

Original: [Equation]
Revised: [Equation]

Original: Note that the main difference between H-Transformer these and CST lies in Eq.equation 30 and
Revised: Note that the main difference between H-Transformer these and CST lies in Eq.equation 31 and

Original: Eq.equation 27; in the former, the argument to the global attention is S × 2 µ , whereas it is Y CS l × 2 µ in the latter.
Revised: Eq.equation 28; in the former, the argument to the global attention is S × 2 µ , whereas it is Y CS l × 2 µ in the latter.

Original: To demonstrate our claim, let us first introduce some quantities of interest, starting with the families of parameterized functions that the aforementioned attention mechanisms encompass. {+First, assume 
+}
Revised: To demonstrate our claim, let us first introduce some quantities of interest, starting with the families of parameterized functions that the aforementioned attention mechanisms encompass. {+First, assume 
+}

Revised: In practice, it is common to have equal weights at local and global scales, i.e., W l, · = W g, · .

Revised: This is a special case of the analysis presented here and changes the conclusion in no significant way.

Original: First, assume without loss of generality that all the weights are limited to the interval [ − 1 , 1] .
Revised: Refer to without loss of generality that all the weights are limited to the interval [ − 1 , 1] .

Original: Then, any attention function learned during the training process must belong to the following sets,
Revised: Then, any attention function learned during the training process must belong to the following sets,

Original: [Equation]
Revised: [Equation]

Original: in the case of CST and,
Revised: in the case of CST and,

Original: [Equation]
Revised: [Equation]

Original: in the case of H-transformer.
Revised: in the case of H-transformer.

Original: We want to show that Y CS g has more expressive set than Y H g , which explains why CST exhibits better empirical prediction performance than its competitors.
Revised: We want to show that Y CS g has more expressive set than Y H g , which explains why CST exhibits better empirical prediction performance than its competitors.

Original: To this end, let us introduce the following quantities:  Let p ( x ) , q ( x ) be polynomials of Euclidean degree d from R M to R .
Revised: To this end, let us introduce the following quantities: Definition 1. Let p ( x ) , q ( x ) be polynomials of Euclidean degree d from R M to R .

Revised: We denote by

Original: We denote by S M,d the set of rational functions with numerator and denominator of Euclidean degree at most d , i.e.,
Revised: S M,d the set of rational functions with numerator and denominator of Euclidean degree at most d , i [Equation] e.,

Original: [Equation]
Revised: [Equation]

Original: imbued with the metric,
Revised: imbued with the metric,

Original: [Equation]
Revised: [Equation]

Original: where || f ( x ) || ∞ =
Revised: where || f ( x ) || ∞ =

Original: max x ∈ [ − 1 , 1] M | f ( x ) | .
Revised: max x ∈ [ − 1 , 1] M | f ( x ) | .

Original: We also introduce S M, ∞ to indicate the ratio of real analytic functions, which is a vector space.
Revised: We also introduce S M, ∞ to indicate the space of ratios of real analytic functions, which is a vector space.

Original: Further, given any fixed positive ϵ ∈ R , we define,
Revised: Further, given any fixed positive ϵ ∈ R , we define,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: as the ϵ -ball surrounding S M,d in the topology induced on S M, ∞ by the metric.
Revised: as the ϵ -ball surrounding S M,d in the topology induced on S M, ∞ by the metric.

Original: Clearly, for any fixed dimension M , ϵ > 0 and Euclidean degrees d ′ > d , the family of functions S ϵM,d ′ is more expressive than S ϵM,d , since the latter is a proper subset of the former, i.e., S M,d ⊂ S M
Revised: Clearly, for any fixed dimension M , ϵ > 0 and Euclidean degrees d ′ > d , the family of functions S ϵM,d ′ is more expressive than S ϵM,d , since the latter is a proper subset of the former, i.e., S M,d ⊂ S M,d ′ .

Original: Next, we show that the families of functions Y CS g and Y H g are in fact subsets of S ϵM,d ′ and S ϵM,d respectively, for appropriately-chosen values of ϵ , d ′ and d with d ′ > d .
Revised: Next, we show that the families of functions Y CS g and Y H g are in fact subsets of S ϵM,d ′ and S ϵM,d respectively, for appropriately-chosen values of ϵ , d ′ and d with d ′ > d .

Original: To do so, we need the following result adapted from Trefethen (2017),
Revised: To do so, we need the following result adapted from Trefethen (2017),

Revised: Theorem 1.

Original: Let f ( x ) be an analytic function from R M to R .
Revised: Let f ( x ) be an analytic function from R M to R .

Original: Then for every fixed ϵ > 0 , there exists a polynomial p ( x ) of Euclidean degree d ( ϵ )
Revised: Then for every fixed ϵ > 0 , there exists a polynomial p ( x ) of Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that,
Revised: O (log( ϵ )) such that,

Original: [Equation]
Revised: [Equation]

Original: [Equation] We are now ready to demonstrate our main result,  For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )
Revised: We are now ready to demonstrate our main result, Proposition 2. For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that,
Revised: O (log( ϵ )) such that,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: where M corresponds to the total number of weights.
Revised: where M corresponds to the total number of weights.

Revised: Proof.

Original: First, recall that (assuming a single head) the parameterized functions γ ( · ) from Eq.
Revised: First, recall that (assuming a single head) the parameterized functions γ ( · ) from Eq.

Original: equation 25 takes the following form
Revised: equation 26 takes the following form

Original: [Equation]
Revised: [Equation]

Original: This should be recognized as the ratio of two analytic functions (because the exponential is analytic) paramatrized by the elements of the weight matrices.
Revised: This should be recognized as the ratio of two analytic functions (because the exponential is analytic) paramatrized by the elements of the weight matrices.

Original: Following Theorem A.6, for every fixed set of weights W g =
Revised: Following Theorem 1, for every fixed set of weights W g =

Original: [ W q , W k , W v ] , there exists polynomials { p W g ij ( x ) } , { q W g ij ( x ) } of Euclidean degree d ( ϵ )
Revised: [ W q , W k , W v ] , there exists polynomials { p W g ij ( x ) } , { q W g ij ( x ) } of Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that
Revised: O (log( ϵ )) such that

Original: [Equation]
Revised: [Equation]

Original: This means that
Revised: This means that

Original: [Equation]
Revised: [Equation]

Revised: and that γ ij ( · ; W q

Revised: [Equation]

Revised: k ,

Revised: [Equation]

Original: and that γ ij ( · ; W q , W k , W v ) ∈ S ϵM,d ( ϵ ) . In particular, this shows that Y H g ⊂ S ϵM g ,d ( ϵ ) since every element of Y H g has the form of Eq.equation 40.
Revised: ∈ [Equation] ϵ ) . In particular, this shows that Y H g ⊂ S ϵM g ,d ( ϵ ) since every element of Y H g has the form of Eq.equation 41.

Original: A similar conclusion can be reached for Y H l and
Revised: A similar conclusion can be reached for Y H l and

Original: Y CS l since elements of these families have the same functional form as those of Y H g .
Revised: Y CS l since elements of these families have the same functional form as those of Y H g .

Original: All that remains is Y CS g .
Revised: All that remains is Y CS g .

Original: In this case, it suffices to note that the result of polynomial composition creates polynomials with Euclidean degree bounded by the sum of the degree of the polynomials involved in the composition.
Revised: In this case, it suffices to note that the result of polynomial composition creates polynomials with Euclidean degree bounded by the sum of the degree of the polynomials involved in the composition.

Original: Indeed, the above analysis shows that elements of Y CS g are within ϵ distance from rational functions of the form,
Revised: Indeed, the above analysis shows that elements of Y CS g are within ϵ distance from rational functions of the form,

Original: [Equation]
Revised: [Equation]

Original: for some polynomials { p W l ijk ( x [Equation] , { q W l ijk ( x ) } of degree d ( ϵ ) .
Revised: for some polynomials { p W l ijk ( x ) } , { q W l ijk ( x ) } of degree d ( ϵ ) .

Original: Upon expanding the polynomials and expressing terms using a common denominator, this expression should be recognized as the ratio of two polynomials, each of which has degree 4 d ( ϵ ) , following polynomial composition.
Revised: Upon expanding the polynomials and expressing terms using a common denominator, this expression should be recognized as the ratio of two polynomials, each of which has degree 4 d ( ϵ ) , following polynomial composition.

Original: Thus, we conclude that Y CSg ⊂ S ϵM, 4 d ( ϵ ) .
Revised: Thus, we conclude that Y CSg ⊂ S ϵM, 4 d ( ϵ ) .

Original: This proves our claim.
Revised: This proves our claim.

Original: In other words, Proposition A. shows that our proposed family of attention mechanisms belongs to S ϵM, 4 d ( ϵ ) which possesses more expressive power (since 4 d ( ϵ ) >
Revised: In other words, Proposition 2 shows that our proposed family of attention mechanisms belongs to S ϵM, 4 d ( ϵ ) which possesses more expressive power (since 4 d ( ϵ ) >

Original: d ( ϵ ) ) than the family of functions S ϵM,d ( ϵ ) to which global attention mechanism without composition, such as H-transfomers ( Y H l + ψ (cid:0) Y H g (cid:1) ), belong.
Revised: d ( ϵ ) ) than the family of functions S ϵM,d ( ϵ ) to which global attention mechanism without composition, such as H-transfomers ( Y H l + ψ (cid:0) Y H g (cid:1) ), belong.

Original: We claim that this is the reason why our proposed approach performs better than the approach proposed by (Zhu & Soricut, 2021) and other similar mechanisms where linear combinations of attentions, rather than composition, are used.
Revised: We claim that this is the reason why our proposed approach performs better than the approach proposed by (Zhu & Soricut, 2021) and other similar mechanisms where linear combinations of attentions, rather than composition, are used.

Original: In other words, for the same number of parameters (weights), our proposed approach can capture more complex attentive interactions than that of (Zhu & Soricut, 2021) and related mechanisms, which leads to a richer set of attentions over which to train the network and, ultimately, better performance.
Revised: In other words, for the same number of parameters (weights), our proposed approach can capture more complex attentive interactions than that of (Zhu & Soricut, 2021) and related mechanisms, which leads to a richer set of attentions over which to train the network and, ultimately, better performance.

Original: A. M ORE I NFORMATION AND D ETAILS ON LRA B ENCHMARK A.7. E XPERIMENTAL S ETUP
Revised: A. M ORE I NFORMATION AND D ETAILS ON LRA B ENCHMARK A.6. E XPERIMENTAL S ETUP

Original: We follow the experimental setup described in (Xiong et al., 2021) with a few exceptions in hyperparameters.
Revised: We follow the experimental setup described in (Xiong et al., 2021) with a few exceptions in hyperparameters.

Original: Specifically, we use the same Transformer encoder network backbone that consists oftransformer layers with pre-norm (Xiong et al., 2020) with 4 attention heads and the model dimension 256 across all tasks.
Revised: Specifically, we use the same Transformer encoder network backbone that consists of [Table] transformer layers with pre-norm (Xiong et al., 2020) with 4 attention heads and the model dimension 256 across all tasks.

Original: Further, we replace the self-attention sub-layer by efficient attention alternatives including the slice attention of CST .
Revised: Further, we replace the self-attention sub-layer by efficient attention alternatives including the slice attention of CST .

Original: To obtain label output for classification, we additionally use a classification head network with 2 fully-connected layers of the same hidden dimension with the feed-forward network in the backbone network and a ReLU layer between them.
Revised: To obtain label output for classification, we additionally use a classification head network with 2 fully-connected layers of the same hidden dimension with the feed-forward network in the backbone network and a ReLU layer between them.

Original: We aggregate the output sequence from the encoder by a mean pooling across the length dimension and feed it to the classification head.
Revised: We aggregate the output sequence from the encoder by a mean pooling across the length dimension and feed it to the classification head.

Original: For a fair comparison between models with different characteristics, we optimize the learning rates for the baseline Transformer model on each task while keeping all other hyperparameters fixed.
Revised: For a fair comparison between models with different characteristics, we optimize the learning rates for the baseline Transformer model on each task while keeping all other hyperparameters fixed.

Original: For the experimental setup, this allows for a strong baseline model, advantageous for the standard Transformer.
Revised: For the experimental setup, this allows for a strong baseline model, advantageous for the standard Transformer.

Original: We then evaluate each model with a learning rate search within a small bracket [0 .
Revised: We then evaluate each model with a learning rate search within a small bracket [0 .

Original: 5 l 0 , l 0 , 2 l 0 ] , where l 0 is the base learning rate providing the highest validation accuracy of the Transformer in each task, and report the test accuracy of the model trained with a learning rate that gives the highest validation accuracy among the bracket.
Revised: 5 l 0 , l 0 , 2 l 0 ] , where l 0 is the base learning rate providing the highest validation accuracy of the Transformer in each task, and report the test accuracy of the model trained with a learning rate that gives the highest validation accuracy among the bracket.

Original: We note that our experiments have stronger baselines than those found in literature (Tay et al., 2020b; Xiong et al., 2021) as highlighted in Table 1.
Revised: We note that our experiments have stronger baselines than those found in literature (Tay et al., 2020b; Xiong et al., 2021) as highlighted in Table 1.

Original: For other efficient Transformer models, we perform a model-specific hyperparameter search as described below and report the test accuracy of the best models.
Revised: For other efficient Transformer models, we perform a model-specific hyperparameter search as described below and report the test accuracy of the best models.

Original: For the Long Range Arena benchmark, we performed experiments on our PyTorch version implementation of the original open-source JAX/Flax code (Tay et al., 2020b) where we followed its dataset preparation procedure and verified the results, and implemented our model on it.
Revised: For the Long Range Arena benchmark, we performed experiments on our PyTorch version implementation of the original open-source JAX/Flax code (Tay et al., 2020b) where we followed its dataset preparation procedure and verified the results, and implemented our model on it.

Original: Since we found the performances of LRA tasks have relatively high variances, especially on certain tasks such as ListOps , we first performed a hyperparameter search with the standard Transformer with the base setup in (Xiong et al., 2021).
Revised: Since we found the performances of LRA tasks have relatively high variances, especially on certain tasks such as ListOps , we first performed a hyperparameter search with the standard Transformer with the base setup in (Xiong et al., 2021).

Original: The hyperparameters we used for the search include number of layers { 2, 4 } , number of attention heads { 2, 4 } , and model dimension { 128, 256 } , and learning rate { 1 , 2 , 5 } × 10 {− 3 , − 4 , − 5 , − 6 } .
Revised: The hyperparameters we used for the search include number of layers { 2, 4 } , number of attention heads { 2, 4 } , and model dimension { 128, 256 } , and learning rate { 1 , 2 , 5 } × 10 {− 3 , − 4 , − 5 , − 6 } .

Original: In addition, for better convergence, we also increased the number of training steps in some tasks.
Revised: In addition, for better convergence, we also increased the number of training steps in some tasks.

Original: The hyperparameters we determined after the search can be found in
Revised: The hyperparameters we determined after the search can be found in

Original: Table 4.
Revised: Table 6.

Original: As this step is essentially a hyperparameter tuning for the baseline Transformer, we note that we have stronger baseline Transformer model than those reported in (Tay et al., 2020b) and (Xiong et al., 2021), and compare them in Table 1.
Revised: As this step is essentially a hyperparameter tuning for the baseline Transformer, we note that we have stronger baseline Transformer model than those reported in (Tay et al., 2020b) and (Xiong et al., 2021), and compare them in Table 1.

Original: While fixing all other hyperparameters, we performed a learning rate sweep using a learning rate bracket as described in 5.1.
Revised: While fixing all other hyperparameters, we performed a learning rate sweep using a learning rate bracket as described in 5.1.

Original: For each model and task, we pick the best model on the validation dataset with early stopping, and report its accuracy on the test dataset.
Revised: For each model and task, we pick the best model on the validation dataset with early stopping, and report its accuracy on the test dataset.

Original: For the model training, we used AdamW optimizer (Loshchilov & Hutter, 2017) with linear learning rate warmup and decay, but without weight decay and other training hyperparameters as shown in Table 4.
Revised: For the model training, we used AdamW optimizer (Loshchilov & Hutter, 2017) with linear learning rate warmup and decay, but without weight decay and other training hyperparameters as shown in Table 6.

Original: Since each efficient Transformer has its own hyperparameters and they can affect the performance, we tried a few different values, around the default values used in each paper.
Revised: Since each efficient Transformer has its own hyperparameters and they can affect the performance, we tried a few different values, around the default values used in each paper.

Original: Specifically, they are: the number of random features { 64, 128, 256 } in Performer (Choromanski et al., 2020), the projection length { 64, 128, 256 } in Linformer (Wang et al., 2020) and Luna (Ma et al., 2021), the number of landmarks { 64, 128, 256 } in Nystr¨omformer (Xiong et al., 2021), the number of hashes { 2, 4 } in Reformer (Kitaev et al., 2019), the numerical rank of off-diagonal blocks { 16 ,32, 64, 128 } in H-Transformer-1D (Zhu & Soricut, 2021), and the local window segment size { 8, 16, 32 } and  the rank of dynamic projection { 32, 64 } in Long Short Transformer (Zhu et al., 2021).
Revised: Specifically, they are: the number of random features { 64, 128, 256 } in Performer (Choromanski et al., 2020), the projection length { 64, 128, 256 } in Linformer (Wang et al., 2020) and Luna (Ma et al., 2021), the number of landmarks { 64, 128, 256 } in Nystr¨omformer (Xiong et al., 2021), the number of hashes { 2, 4 } in Reformer (Kitaev et al., 2019), the numerical rank of off-diagonal blocks { 16 ,32, 64, 128 } in H-Transformer-1D (Zhu & Soricut, 2021), and the local window segment size { 8, 16, 32 } and [Table] [Table] the rank of dynamic projection { 32, 64 } in Long Short Transformer (Zhu et al., 2021).

Original: For Long
Revised: For Long

Original: Short Transformer, we optionally used additional 1D depth-wise convolution with the kernel 35 as described in (Zhu et al., 2021).
Revised: Short Transformer, we optionally used additional 1D depth-wise convolution with the kernel 35 as described in (Zhu et al., 2021).

Original: For scatterbrain (Chen et al., 2021a), we follow the default setting for the model specific hyperparameters as in the official code release.
Revised: For scatterbrain (Chen et al., 2021a), we follow the default setting for the model specific hyperparameters as in the official code release.

Original: Specifically, for text task, we have cluster size for query/key = 16, number of features = 16.
Revised: Specifically, for text task, we have cluster size for query/key = 16, number of features = 16.

Original: For Listops task, we have cluster size[Table] [Table] for query/key = 32 and number of features = 32.
Revised: For Listops task, we have cluster size for query/key = 32 and number of features = 32.

Original: For image and pathfinder task, we have cluster size = 16 and number of features = 16.
Revised: For image and pathfinder task, we have cluster size = 16 and number of features = 16.

Original: For retrieval task, we have cluster size for query/key =and number of features = 64.
Revised: For retrieval task, we have cluster size for query/key =and number of features = 64.

Original: Additionally, we have number of hashes =
Revised: Additionally, we have number of hashes =

Original: 2 for all tasks.
Revised: 2 for all tasks.

Original: While each option produces different performances requiring different computation and memory costs, we found variants in each model have similar accuracy and report only the best accuracy among them and the corresponding complexity as comparing them is out of scope of this paper.
Revised: While each option produces different performances requiring different computation and memory costs, we found variants in each model have similar accuracy and report only the best accuracy among them and the corresponding complexity as comparing them is out of scope of this paper.

Original: A.7. A BLATION S TUDY ON LRA
Revised: A.6. A BLATION S TUDY ON LRA

Original: We investigate the effect of each component in CST by performing an ablation study.
Revised: We investigate the effect of each component in CST by performing an ablation study.

Original: We set the slice length of 16 without extension for local attention as the base configuration.
Revised: We set the slice length of 16 without extension for local attention as the base configuration.

Original: Then, we train a CST variant from scratch where one component is changed.
Revised: Then, we train a CST variant from scratch where one component is changed.

Original: Specifically, we consider the variations in: 1)
Revised: Specifically, we consider the variations in: 1)

Original: Slice length, 2) Extended local attention, 3)
Revised: Slice length, 2) Extended local attention, 3)

Original: Positional embedding, and 4) Composition of MSMRA.
Revised: Positional embedding, and 4) Composition of MSMRA.

Original: The test accuracies of the variants are given in Table 5.
Revised: The test accuracies of the variants are given in Table 7.

Original: We note that all variants of CST consistently outperforms other models, except for one case of the global first composition.
Revised: We note that all variants of CST consistently outperforms other models, except for one case of the global first composition.

Original: Slice Length First, we vary the slice length L and test how the local attention range and abstraction length for the global attention affects the accuracy.
Revised: Slice Length First, we vary the slice length L and test how the local attention range and abstraction length for the global attention affects the accuracy.

Original: Intuitively, one can expect a trade-off between fine-grained token interaction modeling and coarse-grained global context capturing, since in the extreme cases either local attention or global attention converges to the standard self-attention 6 .
Revised: Intuitively, one can expect a trade-off between fine-grained token interaction modeling and coarse-grained global context capturing, since in the extreme cases either local attention or global attention converges to the standard self-attention 7 .

Original: While each task has its own best L , shorter slice length tends to give better overall accuracy.
Revised: While each task has its own best L , shorter slice length tends to give better overall accuracy.

Original: One possible
Revised: One possible

Original: [Table]
Revised: CSA converges to the standard self-attention with the global attention at L or the local attention

Original:  interpretation is that L = 8 or 16 provides better-formed tokens to the global attention in a dynamic manner by the local attention.
Revised: L interpretation is that L = 8 or 16 provides better-formed tokens to the global attention in a dynamic manner by the local attention.

Original: Local Attention Extension Ratio While slight improvements are observed on Retrieval and Image tasks, on the contrary to our expectation, extended local attention does not always improve the accuracy on LRA tasks.
Revised: Local Attention Extension Ratio While slight improvements are observed on Retrieval and Image tasks, on the contrary to our expectation, extended local attention does not always improve the accuracy on LRA tasks.

Original: We conjecture it is because LRA datasets has tokens of very fine granularity, i.e., byte-level or pixel-level, and abstraction of the extended local attention output may result in redundant information for global attention.
Revised: We conjecture it is because LRA datasets has tokens of very fine granularity, i.e., byte-level or pixel-level, and abstraction of the extended local attention output may result in redundant information for global attention.

Original: Thus, we evaluate the effect of the extension ratio on a dataset with a coarser-grained tokenization in the Section 5.2.
Revised: Thus, we evaluate the effect of the extension ratio on a dataset with a coarser-grained tokenization in the Section 5.2.

Original: Position Embedding We compare the MS-VIPE with the conventional learned positional embedding that is applied at the bottom embedding layer of the network.
Revised: Position Embedding We compare the MS-VIPE with the conventional learned positional embedding that is applied at the bottom embedding layer of the network.

Original: MS-VIPE shows consistent improvement over the conventional positional embedding except for the ListOps task where the fined-grained order of each token is essential in this particular task.
Revised: MS-VIPE shows consistent improvement over the conventional positional embedding except for the ListOps task where the fined-grained order of each token is essential in this particular task.

Original: Combination of MSMRA Instead of the proposed composition of MSMRA that has a serial connection of MSMRA in the local-global order, we try two different connections, i.e., parallel and global-local connection.
Revised: Combination of MSMRA Instead of the proposed composition of MSMRA that has a serial connection of MSMRA in the local-global order, we try two different connections, i.e., parallel and global-local connection.

Original: Both of them degrade the accuracy compared to the local-global composition, supporting effectiveness of our design.
Revised: Both of them degrade the accuracy compared to the local-global composition, supporting effectiveness of our design.

Original: Local Modeling with MLP As shown in (Tolstikhin et al., 2021) and (Liu et al., 2021a), a multilayer perceptron (MLP) can effectively replace the attention without loss of accuracy in some applications.
Revised: Local Modeling with MLP As shown in (Tolstikhin et al., 2021) and (Liu et al., 2021a), a multilayer perceptron (MLP) can effectively replace the attention without loss of accuracy in some applications.

Original: In CST , since the slice length is fixed and relatively short, use of MLP instead of attention can be considered as another source of efficiency with a small increase of parameters.
Revised: In CST , since the slice length is fixed and relatively short, use of MLP instead of attention can be considered as another source of efficiency with a small increase of parameters.

Original: We evaluate a variant of CST with 2-layer MLP replacing the local attention, with 2 different hidden dimensions d hidden = 16 or 32 while L = 16 and α = 1 .
Revised: We evaluate a variant of CST with 2-layer MLP replacing the local attention, with 2 different hidden dimensions d hidden = 16 or 32 while L = 16 and α = 1 .

Original: MLP in each slice is equivalent to 1D convolution with the kernel size and the stride set to the same with the slice length.
Revised: MLP in each slice is equivalent to 1D convolution with the kernel size and the stride set to the same with the slice length.

Original: We observe that using attention for local modeling module is still preferred in terms of accuracy.
Revised: We observe that using attention for local modeling module is still preferred in terms of accuracy.

Original: Slice aggregation We further evaluate use of different aggregation methods from the simple mean pooling.
Revised: Slice aggregation We further evaluate use of different aggregation methods from the simple mean pooling.

Original: Similarly to (Rae et al., 2019), we try max pooling and linear projection.
Revised: Similarly to (Rae et al., 2019), we try max pooling and linear projection.

Original: Again, a linear projection is equivalent to 1D convolution with the kernel size and stride set to L .
Revised: Again, a linear projection is equivalent to 1D convolution with the kernel size and stride set to L .

Original: While showing comparable results, they lead to degraded performances in Image and Pathfinder tasks.
Revised: While showing comparable results, they lead to degraded performances in Image and Pathfinder tasks.

Original: One can expect better accuracy from the linear projection as it is a general form of other pooling methods, but this result can be interpreted as limitation of a simple linear projection for modeling sequence translations.
Revised: One can expect better accuracy from the linear projection as it is a general form of other pooling methods, but this result can be interpreted as limitation of a simple linear projection for modeling sequence translations.

Original: The mean pooling has already generalized well, and with better initialization, linear projection is expected to converge to the mean pooling or slightly better.
Revised: The mean pooling has already generalized well, and with better initialization, linear projection is expected to converge to the mean pooling or slightly better.

Original: A. M ORE  ON A UTOREGRESSIVE L ANGUAGE M ODELING A.8. E XPERIMENTAL D ETAIL 
Revised: A. M ORE I NFORMATION AND D ETAILS ON A UTOREGRESSIVE L ANGUAGE M ODELING A.7. E XPERIMENTAL D ETAIL OF A UTOREGRESSIVE L ANGUAGE M ODELING

Original: For the autoregressive language modeling experiment, we use an open-source language modeling experiment framework (Schlag et al., 2021) and plug in our CST implementation.
Revised: For the autoregressive language modeling experiment, we use an open-source language modeling experiment framework (Schlag et al., 2021) and plug in our CST implementation.

Original: We follow all training setup including dataset preparation and hyperparameters given in (Dai et al., 2019a), and evaluate with the small-sized network configuration: 16 layers, 8 attention heads, 128 model dimension, and 2,048 hidden dimension.We train all models using the Adam optimizer (Kingma [Table]
Revised: We follow all training setup including dataset preparation and hyperparameters given in (Dai et al., 2019a), and evaluate with the small-sized network configuration: 16 layers, 8 attention heads, 128 model dimension, and 2,048 hidden dimension.

Original: [Table] & Ba, 2014) , the cosine annealing learning rate scheduler with 2000 warmup steps while the base learning rate is 2 .
Revised: We train all models using the Adam optimizer (Kingma & Ba, 2014), the cosine annealing learning rate scheduler with 2000 warmup steps while the base learning rate is 2 .

Original: 5 × 10 − 4 for 500K steps with the batch size of 96.
Revised: 5 × 10 − 4 for 500K steps with the batch size of 96.

Original: We also conduct autoregressive language modeling experiments with 1024 sequence length.
Revised: We also conduct autoregressive language modeling experiments with 1024 sequence length.

Original: Here, we roughly follow the experiment setup used in (Chen et al., 2021a).
Revised: Here, we roughly follow the experiment setup used in (Chen et al., 2021a).

Original: The larger model has 512 model dimension instead of 128, and we change the learning rate to 5 × 10 − 4 , number of steps to 90K, and batch size to 32, while keeping all the remaining hyperparameters same as that of the small model.
Revised: The larger model has 512 model dimension instead of 128, and we change the learning rate to 5 × 10 − 4 , number of steps to 90K, and batch size to 32, while keeping all the remaining hyperparameters same as that of the small model.

Original: Different choices of slice length L and extension ratio α results in the same local attention range.
Revised: Different choices of slice length L and extension ratio α results in the same local attention range.

Original: For instance, the combination of L = 8 , α = 3 has the same key/value lengths with those in the
Revised: For instance, the combination of L = 8 , α = 3 has the same key/value lengths with those in the

Original: L = 16 , α = 1 setting, i.e., L × (3 − 1) / 2 = L .
Revised: L = 16 , α = 1 setting, i.e., L × (3 − 1) / 2 = L .

Original: However, since the abstraction length and the resulting granularity of the global attention only depends on L , the choices of these hyperparameters can affect  the performance differently.
Revised: However, since the abstraction length and the resulting granularity of the global attention only depends on L , the choices of these hyperparameters can affect [Table] the performance differently.

Original: We examine how the performance is affected by this configuration.
Revised: We examine how the performance is affected by this configuration.

Original: To this end, we evaluate CST with combinations of a set of slice lengths { 8 , 16 , 32 } and a set of extension ratios α { 1 , 2 , 3 } .
Revised: To this end [Table] we evaluate CST with combinations of a set of slice lengths { 8 , 16 , 32 } and a set of extension ratios α { 1 , 2 , 3 } .

Original: To ensure that the current query token has no interaction with the future key tokens, we design the causal mask and local attention extension scheme to better suit CSA as described in Section A.5, and use it throughout this experiment.
Revised: To ensure that the current query token has no interaction with the future key tokens [Table] we design the causal mask and local attention extension scheme to better suit CSA as described in Section 4.

Original: A.8. A BLATION S TUDY 
Revised: [Table] and use it throughout this experiment. A.7. A UTOREGRESSIVE L ANGUAGE M ODELING WITH M ATCHED A TTENTION C OMPLEXITY TO T RANSFORMERS ON PG-

Original: We show how different CSThyperparameters affects the performance on WikiText-103 autoregressive language modeling in Table 7. These results are computed using the MS-VIPE positional embedding.
Revised: We additionally perform evaluation of CST on PG-19 dataset (Rae et al., 2019) compare to Transformers where their attention complexities are matched, i.e., O ( N 2 /N 2 +0 .

Original: It can be observed that models with smaller slice length with local attention extension performs comparable to models with larger slice length without extension, while being more efficient than their non-extended counterparts.
Revised: ( α +1) NL ) ≃ O ( N 2 ) .

Original: We demonstrate the relative speedup obtained by dividing the  training [-runtime of the regular transformer model with the runtime of CST on NVIDIA RTX 
 GPU with-] the batch size of 16.
Revised: We use GPT-2 (Radford et al., 2019) implementation in Huggingface transformers library (Wolf et al., 2020), and use the same architecture to implement CST where only the attention layer is replaced. We train all models for 400 k training [-runtime of the regular transformer model with the runtime of CST on NVIDIA RTX 
 GPU with-]steps using the batch size 32 .

Original: We report the average speed up over 100 training steps.
Revised: We use AdamW optimizer with the learning rate = 1 e − 3 , β = [0 . 9 , 0 . 99] , and the weight decay = 0 .

Original: In Table 8 we compare the result for our training on 1024 context length with other models. It’s seen that CST outperforms Reformer and Performer while achieveing competitive performance with
Revised: We report the validation and test perplexities in Table 9. Efficient attention in CST enables longer context length with the same complexity in attention computation, leading to significantly improved perplexity compared to the Transformer baseline.

Original: Scatterbrain.
Revised: Note that, with the current architecture in these cases, the overall complexity can be higher in CST since there still are complexity increases in linear layers when they are processing longer inputs.

Original: We believe a better hyperparameter search will further improve the results here. [Table]
Revised: We leave application of caching techniques as future work, that saves complexity in linear layers to match the overall network complexity while enjoying the long-range attention.

Original: A. M ORE I NFORMATION AND D ETAILS ON B IDIRECTIONAL M ASKED L ANGUAGE M ODELING AND GLUE B ENCHMARK
Revised: A. M ORE I NFORMATION AND D ETAILS ON B IDIRECTIONAL M ASKED L ANGUAGE M ODELING AND GLUE B ENCHMARK

Original: In this section, we present more details about the experimental setup and results on the bidirectional masked language modeling (MLM) task Devlin et al.
Revised: In this section, we present more details about the experimental setup and results on the bidirectional masked language modeling (MLM) task Devlin et al.

Original: and the natural language understanding tasks in the GLUE benchmark Wang et al.
Revised: and the natural language understanding tasks in the GLUE benchmark Wang et al.

Original: (2018) with discussion.
Revised: (2018) with discussion.

Original: A.9. E XPERIMENTAL S ETUP
Revised: A.8. E XPERIMENTAL S ETUP

Original: We pre-train transformer models with the masked language modeling (MLM) objective on BookCorpus Zhu et al.
Revised: We pre-train transformer models with the masked language modeling (MLM) objective on BookCorpus Zhu et al.

Original: and English Wikipedia.
Revised: and English Wikipedia.

Original: After pre-training, the models are fine-tuned on downstream natural language understanding tasks from GLUE benchmark Wang et al.
Revised: After pre-training, the models are fine-tuned on downstream natural language understanding tasks from GLUE benchmark Wang et al.

Original: We follow the experimental setup of Devlin et al.
Revised: We follow the experimental setup of Devlin et al.

Original: (2018) for conducting both pre-training and fine-tuning experiment, including datasets, masking rate, batch size, and optimizer settings, with several exceptions.
Revised: (2018) for conducting both pre-training and fine-tuning experiment, including datasets, masking rate, batch size, and optimizer settings, with several exceptions.

Original: We pre-train the models for 900k steps with sequences of 128 tokens.
Revised: We pre-train the models for 900k steps with sequences of 128 tokens.

Original: This corresponds to the first phase of BERT pre-training and we consider it to be enough for evaluating on GLUE tasks while expediting the experiment.
Revised: This corresponds to the first phase of BERT pre-training and we consider it to be enough for evaluating on GLUE tasks while expediting the experiment.

Original: With the baseline network architecture of BERT base Devlin et al.
Revised: With the baseline network architecture of BERT base Devlin et al.

Original: (2018), we replace its self-attention layer with different types of efficient attentions such as Nystr ¨ omformer, Performer, Luna, and FNet.
Revised: (2018), we replace its self-attention layer with different types of efficient attentions such as Nystr¨omformer, Performer, Luna, and FNet.

Original: For downstream NLU tasks, we also follow the setup of Devlin et al.
Revised: For downstream NLU tasks, we also follow the setup of Devlin et al.

Original: (2018) with one minor change: instead of prepending [CLS] token in the input sequence, we use mean pooling to get the classifier input vector from the sequence output, as discussed in the following part of this section.
Revised: (2018) with one minor change: instead of prepending [CLS] token in the input sequence, we use mean pooling to get the classifier input vector from the sequence output, as discussed in the following part of this section.

Original: A.9. R ESULTS AND D ISCUSSION
Revised: A.8. R ESULTS AND D ISCUSSION

Original: As shown in Table 9, we can see that CST outperforms competing efficient transformers in terms of
Revised: As shown in Table 5, we can see that CST outperforms competing efficient transformers in terms of

Original: MLM perplexity.
Revised: MLM perplexity.

Original: It can be seen that, while CST s achieve consistently lower perplexity than most of the efficient transformers, its perplexity decreases as slice length and extension ratio increases.
Revised: It can be seen that, while CST s achieve consistently lower perplexity than most of the efficient transformers, its perplexity decreases as slice length and extension ratio increases.

Original: However, CST does not outperform BERT base on contrary to our results on the LRA benchmark.
Revised: However, CST does not outperform BERT base on contrary to our results on the LRA benchmark.

Original: We conjecture this is due to fine-grained masking strategy in the original MLM objective.
Revised: We conjecture this is due to fine-grained masking strategy in the original MLM objective.

Original: The MLM pre-training stage involves replacing random tokens in the input sequence with a dummy token that the model must predict.
Revised: The MLM pre-training stage involves replacing random tokens in the input sequence with a dummy token that the model must predict.

Original: After applying the positional embedding, replacing a standard token with a dummy token, i.e., [MASK], translates to injecting high-frequency noise into the input sequence.
Revised: After applying the positional embedding, replacing a standard token with a dummy token, i.e., [MASK], translates to injecting high-frequency noise into the input sequence.

Original: CST and other efficient attention alternatives are approximations to the full attention, which must truncate the high-frequency modes in exchange for faster evaluation.
Revised: CST and other efficient attention alternatives are approximations to the full attention, which must truncate the high-frequency modes in exchange for faster evaluation.

Original: In MLM pre-training, all of the input sequences have high-frequency content that attention approximations have difficulty capturing.
Revised: In MLM pre-training, all of the input sequences have high-frequency content that attention approximations have difficulty capturing.

Original: Meanwhile, the sequences in the LRA benchmark are long, low-frequency sequences which can be efficiently compressed.
Revised: Meanwhile, the sequences in the LRA benchmark are long, low-frequency sequences which can be efficiently compressed.

Original: An alternative pre-training method could eventually be developed specifically for efficient transformers that circumvents this issue.
Revised: An alternative pre-training method could eventually be developed specifically for efficient transformers that circumvents this issue.

Original: On downstream NLU classification tasks, CST also outperforms among the efficient transformers.
Revised: On downstream NLU classification tasks, CST also outperforms among the efficient transformers.

Original: But with under-performing pre-trained models compared to BERT, similar degradation trend is observed in evaluation on downstream tasks after fine-tuning them.
Revised: But with under-performing pre-trained models compared to BERT, similar degradation trend is observed in evaluation on downstream tasks after fine-tuning them.

Original: One drawback of the current version of CST in the original setting is that the use of [CLS] token is not trivial.
Revised: One drawback of the current version of CST in the original setting is that the use of [CLS] token is not trivial.

Original: [CLS] token can be seen as a global token that summarizes the sequence in the context of the given classification task, and it is often prepended in the input sequence to summarize information from the sequence for the classification purpose.
Revised: [CLS] token can be seen as a global token that summarizes the sequence in the context of the given classification task, and it is often prepended in the input sequence to summarize information from the sequence for the classification purpose.

Original: In CST , naively prepending it limits attention range in local attention part and also make the summarization in the global attention highly implicit.
Revised: In CST , naively prepending it limits attention range in local attention part and also make the summarization in the global attention highly implicit.

Original: We believe this can be addressed by an additional well-designed architecture that uses global tokens and input them to classifier.
Revised: We believe this can be addressed by an additional well-designed architecture that uses global tokens and input them to classifier.

Original: We leave this as a future work and we instead use mean pooling for the experiment for all models.
Revised: We leave this as a future work and we instead use mean pooling for the experiment for all models.

Original: While the efficiency gain is reduced due to the short sequence length, one can still expect reduction in total FLOPS from using CST .
Revised: While the efficiency gain is reduced due to the short sequence length, one can still expect reduction in total FLOPS from using CST .

Original: For instance, in the simple case of N = 128 , L = 64 and α = 1 , the complexity of CST is O (128 · 64+128 2 / 64 2 ) which is almost half of that of the standard transformer
Revised: For instance, in the simple case of N = 128 , L = 64 and α = 1 , the complexity of CST is O (128 · 64+128 2 / 64 2 ) which is almost half of that of the standard transformer

Original: O (128 2 ) .
Revised: O (128 2 ) .

Original: A. L IMITATIONS 
Revised: A. L IMITATIONS AND F UTURE W ORK A.9. L IMITATIONS

Original: We summarize limitations of CST discussed in the previous sections.
Revised: We summarize limitations of CST discussed in the previous sections.

Original: As discussed in A.9, there is a mismatch of using token-wise random masking for MLM pretraining.
Revised: As discussed in A.8, there is a mismatch of using token-wise random masking for MLM pretraining.

Original: In the abstraction of sequence slices for global attention, high-frequency modes are truncated, i.e., smoothed out, that leads to loss of information.
Revised: In the abstraction of sequence slices for global attention, high-frequency modes are truncated, i.e., smoothed out, that leads to loss of information.

Original: While similar arguments can be made to other transformers with abstractive attention, we plan to carefully analyze it and address the issue by designing an alternative pretraining method.
Revised: While similar arguments can be made to other transformers with abstractive attention, we plan to carefully analyze it and address the issue by designing an alternative pretraining method.

Original:  InA.5, we discussed about the missing information of the latest slice in global attention step.
Revised: There are some limitations on the casual mask and slice embedding introduced in Section 4.1, In the global attention, no information of tokens in current slice is taken into account because of the query shift and the attention mask excluding diagonal elements. As a result, there is missing information from coarse-grained modeling in the current design.

Original: This is only happening during the training phase, not in the testing, since the next token prediction with sliding window always has access to all tokens in the sequence up to the current time step.
Revised: This is only happening during the training phase, not in the testing, since the next token prediction with sliding window always has access to all tokens in the sequence up to the current time step.

Original: This mismatch may prohibits the full modeling capability of CST .
Revised: Therefore, this mismatch may prohibit the full modeling capability of CST .

Original: To address the issue, we plan to design more suitable yet efficient training objective that allows full use of abstractions in the global attention during training.
Revised: To address the issue, we plan to design more suitable yet efficient training objective that allows full use of abstractions in the global attention during training.

Revised: We leave design of more advanced autoregressive modeling as a future work.

Original: Additional computation of the complexity O ( N/L )is required in CST for the Q , K , and V transformation  from the local attention output.
Revised: CST requires additional computations of the complexity O ( N/L ) for the Q , K , and V transformation for the global attention, from the local attention output.

Original: This increase in complexity is negligible compared to the overall computation, especially when N/L is small.
Revised: However, this increase in complexity is negligible compared to the overall computation, especially when N/L is small, as discussed in 4.3. A  F UTURE W ORKS

Revised: There are several potential avenues for extending CST .

Revised: First, we can easily generalize our work to multi-level slicing with a higher level than two.

Revised: This will lead to more improvement in expressiveness of the model while adding more degree of freedoms for better efficiency while enabling its application to much longer sequences.

Revised: Second, as discussed in Section 4.1, more advanced design of autoregressive sequence modeling scheme will further improve the performance.

Revised: Third, while CST is based on fixed-length slicing of the sequence, dynamic or semantic slicing would further improve applicability to non-stationary sequences.

Revised: With end-to-end training, the model can learn the optimal slicing of the data for given tasks and provide layer-wise dynamic tokens.

Revised: To this end, efficient realization of the dynamic slicing would be the most challenging task.

Revised: Finally, as a meaningful next step, modification of CST to better fit vision tasks or training a CST -based large language model would lead to more general and practical impacts.