Original: C OMPOSITE S LICE T RANSFORMER : A N E FFICIENT T RANSFORMER WITH C OMPOSITION OF M ULTI -S CALE M ULTI -R ANGE A TTENTIONS
Revised: C OMPOSITE S LICE T RANSFORMER : A N E FFICIENT T RANSFORMER WITH C OMPOSITION OF M ULTI -S CALE M ULTI -R ANGE A TTENTIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Since the introduction of Transformers, researchers have tackled the expensive quadratic complexity of the attention mechanism.
Revised: Since the introduction of Transformers, researchers have tackled the expensive quadratic complexity of the attention mechanism.

Original: While significant complexity improvements have been achieved, they often come at the cost of reduced accuracy.
Revised: While significant complexity improvements have been achieved, they often come at the cost of reduced accuracy.

Original: In this paper, we propose Composite Slice Transformer ( CST ), a Transformerbased network equipped with a composition of multi-scale multi-range attentions, boosting both efficiency and modeling capability.
Revised: In this paper, we propose Composite Slice Transformer ( CST ), a Transformerbased network equipped with a composition of multi-scale multi-range attentions, boosting both efficiency and modeling capability.

Original: After stacking fixed-length slices of the input sequence, each layer in CST performs a pair of fine-and-coarse-grained attentions with short and long ranges in a sequential manner, coupled with a volatile instant positional embedding.
Revised: After stacking fixed-length slices of the input sequence, each layer in CST performs a pair of fine-and-coarse-grained attentions with short and long ranges in a sequential manner, coupled with a volatile instant positional embedding.

Original: In addition to significantly reduced O ( NL + N 2 /L 2 ) complexity for sequence length N and slice length L , CST achieves superior performance on a variety of tasks.
Revised: In addition to significantly reduced O ( NL + N 2 /L 2 ) complexity for sequence length N and slice length L , CST achieves superior performance on a variety of tasks.

Original: We show that CST surpasses recently published efficient Transformers on the Long Range Arena benchmark, demonstrating the bidirectional long-range dependency modeling capability of our model with a comparable complexity.
Revised: We show that CST surpasses recently published efficient Transformers on the Long Range Arena benchmark, demonstrating the bidirectional long-range dependency modeling capability of our model with a comparable complexity.

Original: It also outperforms the standard Transformer by a margin of 6 . 9 % in average accuracy across the five classification tasks.
Revised: It also outperforms the standard Transformer by a margin of 6 . 9 % in average accuracy across the five classification tasks.

Original: On the wordlevel WikiText-103 autoregressive language modeling task with various sequence lengths, and the masked language modeling followed by GLUE benchamrk, CST outperforms most other efficient Transformers while being competitive against the
Revised: On the wordlevel WikiText-103 autoregressive language modeling task with various sequence lengths, and the masked language modeling followed by GLUE benchamrk, CST outperforms most other efficient Transformers while being competitive against the

Original: Transformer.
Revised: Transformer.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Transformers (Vaswani et al., 2017) are one of the most important recent advances in artificial intelligence.
Revised: Transformers (Vaswani et al., 2017) are one of the most important recent advances in artificial intelligence.

Original: Since they can be combined in a straightforward fashion with advanced training methods and auxiliary modules, Transformers have proven extremely effective as a versatile backbone architecture for achieving state-of-the-art performance in many domains such as natural language processing (Devlin et al., 2018;
Revised: Since they can be combined in a straightforward fashion with advanced training methods and auxiliary modules, Transformers have proven extremely effective as a versatile backbone architecture for achieving state-of-the-art performance in many domains such as natural language processing (Devlin et al., 2018;

Original: Yang et al., 2019; Brown et al., 2020; Raffel et al., 2020; Sanh et al., 2022), vision processing (Dosovitskiy et al., 2020; Liu et al., 2021b;
Revised: Yang et al., 2019; Brown et al., 2020; Raffel et al., 2020; Sanh et al., 2022), vision processing (Dosovitskiy et al., 2020; Liu et al., 2021b;

Original: Radford et al., 2021), visual language modeling (Alayrac et al., 2022), speech recognition (Dong et al., 2018; Gulati et al., 2020;
Revised: Radford et al., 2021), visual language modeling (Alayrac et al., 2022), speech recognition (Dong et al., 2018; Gulati et al., 2020;

Original: Shi et al., 2021), and reinforcement learning (Chen et al., 2021b; Janner et al., 2021).
Revised: Shi et al., 2021), and reinforcement learning (Chen et al., 2021b; Janner et al., 2021).

Original: Despite this versatility, Transformers possess an expensive memory and computational complexity of O ( N 2 ) with respect to input length N in the multi-head self-attention computation.
Revised: Despite this versatility, Transformers possess an expensive memory and computational complexity of O ( N 2 ) with respect to input length N in the multi-head self-attention computation.

Original: As a result,
Revised: As a result,

Original: Transformers are often not applied to long sequence data.
Revised: Transformers are often not applied to long sequence data.

Original: Since the introduction of Transformers in (Vaswani et al., 2017), recent work has focused on improving Transformer complexity through various techniques, achieving efficiency gains in complexity and memory requirements (Tay et al., 2020c), with several models attaining O ( N ) complexity (Wang et al., 2020; Katharopoulos et al., 2020; Ma et al., 2021).
Revised: Since the introduction of Transformers in (Vaswani et al., 2017), recent work has focused on improving Transformer complexity through various techniques, achieving efficiency gains in complexity and memory requirements (Tay et al., 2020c), with several models attaining O ( N ) complexity (Wang et al., 2020; Katharopoulos et al., 2020; Ma et al., 2021).

Original: Unfortunately, these efficiencies come at the cost of reduced accuracy and often lack the ability to model fine-grained token dependencies, limiting the application of these recent improvements to a wider range of practical problems.
Revised: Unfortunately, these efficiencies come at the cost of reduced accuracy and often lack the ability to model fine-grained token dependencies, limiting the application of these recent improvements to a wider range of practical problems.

Original: Recent studies (Zhu et al., 2021; Ren et al., 2021;
Revised: Recent studies (Zhu et al., 2021; Ren et al., 2021;

Original: Nguyen et al., 2021; Zhu & Soricut, 2021; Han et al., 2021) show that combining efficient techniques for global sequence modeling with fine-grained limited range attention can improve accuracy, maintain low complexity, and enable longer context windows, allowing such models to outperform the standard, full-resolution Transformer on certain tasks.
Revised: Nguyen et al., 2021; Zhu & Soricut, 2021; Han et al., 2021) show that combining efficient techniques for global sequence modeling with fine-grained limited range attention can improve accuracy, maintain low complexity, and enable longer context windows, allowing such models to outperform the standard, full-resolution Transformer on certain tasks.

Original: However, optimally combining sequence modeling at different levels of granularity remains an open problem.
Revised: However, optimally combining sequence modeling at different levels of granularity remains an open problem.

Original: To this end, we propose the Composite Slice Transformer ( CST ), an efficient Transformer-based network architecture, consisting of a composition of attentions applied to a stacked slice representation of the input sequence at different scales, coupled with a multi-scale volatile instant positional embedding.
Revised: To this end, we propose the Composite Slice Transformer ( CST ), an efficient Transformer-based network architecture, consisting of a composition of attentions applied to a stacked slice representation of the input sequence at different scales, coupled with a multi-scale volatile instant positional embedding.

Original: For fixed slice length L , CST has a complexity of O ( NL + N 2 /L 2 ) , which is comparable or more efficient to linear complexity in many practical settings (Section A.4).
Revised: For fixed slice length L , CST has a complexity of O ( NL + N 2 /L 2 ) , which is comparable or more efficient to linear complexity in many practical settings (Section A.4).

Original: Since slicing restricts fine-grained token interaction across boundaries, CST also leverages an extended local attention, preventing context fragmentation (Dai et al., 2019b) and enabling seamless sequence modeling.
Revised: Since slicing restricts fine-grained token interaction across boundaries, CST also leverages an extended local attention, preventing context fragmentation (Dai et al., 2019b) and enabling seamless sequence modeling.

Original: These improvements allow CST to outperform the standard Transformer on several benchmark tasks.
Revised: These improvements allow CST to outperform the standard Transformer on several benchmark tasks.

Original: Similar to (Dosovitskiy et al., 2020), CST abstracts the input sequence into another with fewer tokens to compute a low-resolution attention with longer range and a fixed-size segment-wise embedding.
Revised: Similar to (Dosovitskiy et al., 2020), CST abstracts the input sequence into another with fewer tokens to compute a low-resolution attention with longer range and a fixed-size segment-wise embedding.

Original: With a segment length L , the reduced sequence has length N/L ; evaluating attention on this reduced sequence has complexity O ( N 2 /L 2 ) .
Revised: With a segment length L , the reduced sequence has length N/L ; evaluating attention on this reduced sequence has complexity O ( N 2 /L 2 ) .

Original: With an appropriately chosen L , CST can achieve a significant complexity reduction without a loss of performance.
Revised: With an appropriately chosen L , CST can achieve a significant complexity reduction without a loss of performance.

Original: Along with the segment-level attention, CST also leverages a full-resolution local attention to form a multi-scale multi-range attention (MSMRA) through a composition with the global attention, improving model expressiveness (Section A.5), unlike (Zhu & Soricut, 2021; Zhu et al., 2021;
Revised: Along with the segment-level attention, CST also leverages a full-resolution local attention to form a multi-scale multi-range attention (MSMRA) through a composition with the global attention, improving model expressiveness (Section A.5), unlike (Zhu & Soricut, 2021; Zhu et al., 2021;

Original: Nguyen et al., 2021; Ren et al., 2021; Han et al., 2021).
Revised: Nguyen et al., 2021; Ren et al., 2021; Han et al., 2021).

Original: CST extends the ideas of position-infused attention (Press et al., 2021) and applies them to MSMRA, which we refer to as multi-scale volatile instant positional embedding (MS-VIPE).
Revised: CST extends the ideas of position-infused attention (Press et al., 2021) and applies them to MSMRA, which we refer to as multi-scale volatile instant positional embedding (MS-VIPE).

Original: In addition to its effectiveness as a positional embedding, MS-VIPE provides further parameter efficiency by only requiring storage of reduced lengths of local ( L ) and global ( N/L ) attentions instead of the full sequence length N .
Revised: In addition to its effectiveness as a positional embedding, MS-VIPE provides further parameter efficiency by only requiring storage of reduced lengths of local ( L ) and global ( N/L ) attentions instead of the full sequence length N .

Original: We evaluate our model on bidirectional long-range dependency modeling tasks, an autoregressive language modeling task, and natural language understanding tasks.
Revised: We evaluate our model on bidirectional long-range dependency modeling tasks, an autoregressive language modeling task, and natural language understanding tasks.

Original: In our experiments (Section 5), CST achieves state-of-the-art performance among all Transformer-based models (including the standard Transformer and recently proposed efficient Transformers) and demonstrates strong performance on a wide variety of tasks.
Revised: In our experiments (Section 5), CST achieves state-of-the-art performance among all Transformer-based models (including the standard Transformer and recently proposed efficient Transformers) and demonstrates strong performance on a wide variety of tasks.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2 and Section 3, we discuss recent efficient Transformer developments, discuss strengths and weaknesses, and outline the inspiration for CST .
Revised: In Section 2 and Section 3, we discuss recent efficient Transformer developments, discuss strengths and weaknesses, and outline the inspiration for CST .

Original: In Section 4, we present our model, Composite Slice Transformer, and describe its features.
Revised: In Section 4, we present our model, Composite Slice Transformer, and describe its features.

Original: In Section 5, we provide our experimental results on the Long Range Arena, WikiText-103 autoregressive language modeling, and the GLUE benchmarks, demonstrating the efficiency and versatility of CST .
Revised: In Section 5, we provide our experimental results on the Long Range Arena, WikiText-103 autoregressive language modeling, and the GLUE benchmarks, demonstrating the efficiency and versatility of CST .

Original: In Section 6, we conclude with a summary of our work.
Revised: In Section 6, we conclude with a summary of our work.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Tay et al.
Revised: Tay et al.

Original: (2020c) provides a comprehensive study of proposed approaches to accelerate the attention mechanism of Vaswani et al.
Revised: (2020c) provides a comprehensive study of proposed approaches to accelerate the attention mechanism of Vaswani et al.

Original: The key differentiating factor among the approaches presented is the modeling assumptions used to approximate the original attention map.
Revised: The key differentiating factor among the approaches presented is the modeling assumptions used to approximate the original attention map.

Original: For example, common assumptions include sparsity (Child et al., 2019;
Revised: For example, common assumptions include sparsity (Child et al., 2019;

Original: Ho et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020; Tay et al., 2020a; Kitaev et al., 2019) and low-rankness (Wang et al., 2020;
Revised: Ho et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020; Tay et al., 2020a; Kitaev et al., 2019) and low-rankness (Wang et al., 2020;

Original: Choromanski et al., 2020; Katharopoulos et al., 2020).
Revised: Choromanski et al., 2020; Katharopoulos et al., 2020).

Original: Chen et al.
Revised: Chen et al.

Original: (2021a) combines these two assumptions.
Revised: (2021a) combines these two assumptions.

Original: Other architectures leverage additional memory for compressing the global context (Lee et al., 2019; Ma et al., 2021; Jaegle et al., 2021).
Revised: Other architectures leverage additional memory for compressing the global context (Lee et al., 2019; Ma et al., 2021; Jaegle et al., 2021).

Original: In order to capture fine-grained token interactions that might be missing in the abstractive attention,
Revised: In order to capture fine-grained token interactions that might be missing in the abstractive attention,

Original: (Zhu & Soricut, 2021; Zhu et al., 2021; Nguyen et al., 2021;
Revised: (Zhu & Soricut, 2021; Zhu et al., 2021; Nguyen et al., 2021;

Original: Ren et al., 2021) use a process akin to leverage a full-resolution local attention to form a multi-scale multi-range attention (MSMRA).
Revised: Ren et al., 2021) use a process akin to leverage a full-resolution local attention to form a multi-scale multi-range attention (MSMRA).

Original: These approaches, however, do not compose the local full-resolution attention and global reduced-resolution attentions in a series.
Revised: These approaches, however, do not compose the local full-resolution attention and global reduced-resolution attentions in a series.

Original: Absent from this literature, serial composition and positional embedding would improve accuracy while preserving the efficiency (see the following sections for how we address this enhancement).
Revised: Absent from this literature, serial composition and positional embedding would improve accuracy while preserving the efficiency (see the following sections for how we address this enhancement).

Original: More recently, there also have been proposed sequence models with structures or strong inductive biases achieving significant improvements in benchmarks such as (Gu et al., 2021; Mehta et al., 2022; Ma et al., 2022;
Revised: More recently, there also have been proposed sequence models with structures or strong inductive biases achieving significant improvements in benchmarks such as (Gu et al., 2021; Mehta et al., 2022; Ma et al., 2022;

Original: Li et al., 2022).
Revised: Li et al., 2022).

Original: (Press et al., 2021) propose the position-infused attention (PIA), which is an attention module with a layer-wise positional embedding applied only to queries and keys, to address token representation reusability issue in increasing window attention due to positional information draft, while avoiding expensive computation of relative position encoding (Dai et al., 2019b).
Revised: (Press et al., 2021) propose the position-infused attention (PIA), which is an attention module with a layer-wise positional embedding applied only to queries and keys, to address token representation reusability issue in increasing window attention due to positional information draft, while avoiding expensive computation of relative position encoding (Dai et al., 2019b).

Original: Another study of adding positional embedding to values is conducted in (Tsai et al., 2019), concluding that positional embedding on value does not lead to performance improvement.
Revised: Another study of adding positional embedding to values is conducted in (Tsai et al., 2019), concluding that positional embedding on value does not lead to performance improvement.

Original: We extend these ideas in
Revised: We extend these ideas in

Original: CST and apply in MSMRA.
Revised: CST and apply in MSMRA.

Original:  P RELIMINARIES 3. T RANSFORMERS AND M ULTI -H EAD S ELF -A TTENTION
Revised:  P RELIMINARIES 3. T RANSFORMERS AND M ULTI -H EAD S ELF -A TTENTION

Original: A Transformer layer consists of a multi-head self-attention sublayer followed by a feed-forward network (Vaswani et al., 2017) with an optional cross-attention sublayer when used as a decoder.
Revised: A Transformer layer consists of a multi-head self-attention sublayer followed by a feed-forward network (Vaswani et al., 2017) with an optional cross-attention sublayer when used as a decoder.

Original: The multi-head self-attention is defined as the concatenation of the self-attention output of all attention heads:
Revised: The multi-head self-attention is defined as the concatenation of the self-attention output of all attention heads:

Original: [Equation]
Revised: [Equation]

Original:  where [ · ] r denotes concatenation in the r th dimension, and each of the outputs 
Revised: [Equation] ) where [ · ] r denotes concatenation in the r th dimension, and each of the outputs Y h ∈ R N ×

Original: Y h ∈ R N × d h is a scaled dot-product attention computed from the input X ∈ R N × D as
Revised: d h is a scaled dot-product attention computed from the input X ∈ R N × D as

Original: [Equation]
Revised: [Equation]

Original: In Eq. 2, Q h = XW q,h , K h = XW k,h , and V h = XW v,h are queries, keys and values, respectively, expressed as linear transformations of the input X by W · ,h ∈ R D × d h .
Revised: In Eq. 2, Q h = XW q,h , K h = XW k,h , and V h = XW v,h are queries, keys and values, respectively, expressed as linear transformations of the input X by W · ,h ∈ R D × d h .

Original: We assume the queries, keys, and values have the same hidden dimension: d h = D/H .
Revised: We assume the queries, keys, and values have the same hidden dimension: d h = D/H .

Original: For the rest of the paper, we omit the head index h and scaling factor 1 / √ d for simplicity.
Revised: For the rest of the paper, we omit the head index h and scaling factor 1 / √ d for simplicity.

Original: We denote the query, key and value at some position index i by k i , v i , q i ∈ R 1 × d , respectively.
Revised: We denote the query, key and value at some position index i by k i , v i , q i ∈ R 1 × d , respectively.

Original: In this context, the attention output at the i th token position y
Revised: In this context, the attention output at the i th token position y

Original: i ∈ R 1 ×
Revised: i ∈ R 1 ×

Original: d h corresponds to
Revised: d h corresponds to

Original: [Equation]
Revised: [Equation]

Original: Due to the nonlinearity and normalization property of the softmax function, QK ⊤ must be computed to obtain the attention weight, followed by value aggregation through the attention weights, AV , resulting in O ( N 2 ) complexity with respect to the sequence length N for the self-attention.
Revised: Due to the nonlinearity and normalization property of the softmax function, QK ⊤ must be computed to obtain the attention weight, followed by value aggregation through the attention weights, AV , resulting in O ( N 2 ) complexity with respect to the sequence length N for the self-attention.

Original: 3. A BSTRACTIVE A TTENTIONS AND M ULTI -S CALE M ULTI -R ANGE A TTENTION (MSMRA)
Revised: 3. A BSTRACTIVE A TTENTIONS AND M ULTI -S CALE M ULTI -R ANGE A TTENTION (MSMRA)

Original: We refer to the family of efficient attention approaches in which the lengths of the attention operands are reduced to M < N by applying an abstraction function ϕ ( · ) as abstractive attentions .
Revised: We refer to the family of efficient attention approaches in which the lengths of the attention operands are reduced to M < N by applying an abstraction function ϕ ( · ) as abstractive attentions .

Original: This approach results in reduced attention complexity while retaining the form of basic attention computation in Eq.
Revised: This approach results in reduced attention complexity while retaining the form of basic attention computation in Eq.

Original: Many recent approaches follow this template (Wang et al., 2020; Ma et al., 2021;
Revised: Many recent approaches follow this template (Wang et al., 2020; Ma et al., 2021;

Original: Dosovitskiy et al., 2020).
Revised: Dosovitskiy et al., 2020).

Original: We focus on cases where the operands of attention, i.e., query, key, and value, are abstracted, noting that there are other possible choices, e.g., abstracting only query and key.
Revised: We focus on cases where the operands of attention, i.e., query, key, and value, are abstracted, noting that there are other possible choices, e.g., abstracting only query and key.

Original: In such cases, the attention mechanism is reduced to
Revised: In such cases, the attention mechanism is reduced to

Original: [Equation]
Revised: [Equation]

Original: where Q , K , and V are abstracted queries, keys, and values, respectively, q i ′ is i ′
Revised: where Q , K , and V are abstracted queries, keys, and values, respectively, q i ′ is i ′

Original: th row of Q , and y i ′ is the attention output token with the abstracted query token q i ′ obtained by
Revised: th row of Q , and y i ′ is the attention output token with the abstracted query token q i ′ obtained by

Original: [Equation]
Revised: [Equation]

Original: In order to restore the resolution of the output, since the query is abstracted, we define a one-to-many mapping function ψ ( · ) as
Revised: In order to restore the resolution of the output, since the query is abstracted, we define a one-to-many mapping function ψ ( · ) as

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: A more detailed description of abstractive attention is presented in Section A.1.
Revised: A more detailed description of abstractive attention is presented in Section A.1.

Original: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically result in degraded benchmark performance.
Revised: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically result in degraded benchmark performance.

Original: However, Transformer-based models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer.
Revised: However, Transformer-based models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer.

Original: In fact, these models can outperform the standard
Revised: In fact, these models can outperform the standard

Original: Transformer on certain tasks, while still maintaining efficiency (Zhu & Soricut, 2021; Zhu et al., 2021;
Revised: Transformer on certain tasks, while still maintaining efficiency (Zhu & Soricut, 2021; Zhu et al., 2021;

Original: Nguyen et al., 2021; Ren et al., 2021).
Revised: Nguyen et al., 2021; Ren et al., 2021).

Original: While other types of MSMRA are described in Section A.2, our proposed attention mechanism is essentially an MSMRA of the form
Revised: While other types of MSMRA are described in Section A.2, our proposed attention mechanism is essentially an MSMRA of the form

Original: [Equation]
Revised: [Equation]

Original: where y l ,i is the local attention output and y g ,i is the global attention output, leveraging a onedimensional version of the token abstraction used in (Dosovitskiy et al., 2020) for the global attention, but with additional composition of local attention.[Figure]
Revised: where y l ,i is the local attention output and y g ,i is the global attention output, leveraging a onedimensional version of the token abstraction used in (Dosovitskiy et al., 2020) for the global attention, but with additional composition of local attention.

Original:  C OMPOSITE S LICE T RANSFORMER
Revised:  C OMPOSITE S LICE T RANSFORMER

Original: We describe the key ideas of Composite Slice Attention ( CSA ) and CST network, a Transformerbased model with CSA replacing the full softmax dot-product attention in the standard Transformer.
Revised: We describe the key ideas of Composite Slice Attention ( CSA ) and CST network, a Transformerbased model with CSA replacing the full softmax dot-product attention in the standard Transformer.

Original: CSA leverages both full-resolution attention in limited ranges and abstracted attention to capture long-range interactions.
Revised: CSA leverages both full-resolution attention in limited ranges and abstracted attention to capture long-range interactions.

Original: Unlike previous approaches, the multi-scale, multi-range attentions are combined through function composition in a serial connection, which allows information passing across representations at different scales and improves expressiveness of the model.
Revised: Unlike previous approaches, the multi-scale, multi-range attentions are combined through function composition in a serial connection, which allows information passing across representations at different scales and improves expressiveness of the model.

Original: See Figure 1 for an illustration of the CSA module and Appendix A.3 for full architecture diagrams.
Revised: See Figure 1 for an illustration of the CSA module and Appendix A.3 for full architecture diagrams.

Original: 4. S EQUENCE R EPRESENTATION AS S TACKED S LICES AND CSA
Revised: 4. S EQUENCE R EPRESENTATION AS S TACKED S LICES AND CSA

Original: In a high-level categorization, the multi-scale multi-range attention of the CST corresponds to the combination of block-wise local window attention (Beltagy et al., 2020) and patch-based global attention (Dosovitskiy et al., 2020) in a one-dimensional form.
Revised: In a high-level categorization, the multi-scale multi-range attention of the CST corresponds to the combination of block-wise local window attention (Beltagy et al., 2020) and patch-based global attention (Dosovitskiy et al., 2020) in a one-dimensional form.

Original: CSA layer starts with converting the input sequence X ∈ R N × D into a stack of slices S ∈ R N/L × L × D by slicing it with a fixed length
Revised: CSA layer starts with converting the input sequence X ∈ R N × D into a stack of slices S ∈ R N/L × L × D by slicing it with a fixed length

Original: L , implemented simply as a Reshape operation.
Revised: L , implemented simply as a Reshape operation.

Original: Two attentions with different granularity are then performed sequentially in each direction.
Revised: Two attentions with different granularity are then performed sequentially in each direction.

Original: First, the batch size B and the number of slices N/L is combined as a new batch size BN/L , so that we parallelize the local attention for all slices.
Revised: First, the batch size B and the number of slices N/L is combined as a new batch size BN/L , so that we parallelize the local attention for all slices.

Original: Then, the local attention is performed across the tokens within each of these new batches:
Revised: Then, the local attention is performed across the tokens within each of these new batches:

Original: [Equation]
Revised: [Equation]

Original: (8) where Q l , K l , and V l ∈ R BN/L × L × d are the queries, keys, and values for each local attention head, computed as SW l ,q , SW l ,k , and SW l ,v , respectively, with W l ∈ R D × d .
Revised: [Equation] where Q l , K l , and V l ∈ R BN/L × L × d are the queries, keys, and values for each local attention head, computed as SW l ,q , SW l ,k , and SW l ,v , respectively, with W l ∈ R D × d .

Original: Next, the dimension of length L in the local attention output is collapsed using an abstraction function ϕ to get the slice embedding S ∈ R BN/L × D .
Revised: Next, the dimension of length L in the local attention output is collapsed using an abstraction function ϕ to get the slice embedding S ∈ R BN/L × D .

Original: Specifically, we use a simple mean pooling S s = ϕ
Revised: Specifically, we use a simple mean pooling S s = ϕ

Original: ( Y l ,s ) =
Revised: ( Y l ,s ) =

Original: [-(cid:80) L − 1 l =
-]
Revised: [-(cid:80) L − 1 l =
-][Equation]

Original:  m s,l Y s,l / (cid:80)[-L − 1 l =
-]
Revised: = [Equation] m s,l Y s,l / (cid:80)[-L − 1 l =
-]

Original: {+[Equation] − 1 l =
+} m s,l , where s and l denote the slice index and the token index, respectively, and m ∈ R N/L × L is the stack of slices of the binary attention mask that is optionally given in the case that the input sequence is padded; e.g., when a data batch has samples with different lengths.
Revised: {+[Equation] − 1 l =
+} m s,l , where s and l denote the slice index and the token index, respectively, and m ∈ R N/L × L is the stack of slices of the binary attention mask that is optionally given in the case that the input sequence is padded; e.g., when a data batch has samples with different lengths.

Original: Then the second, global attention along the slice dimension is performed to model full-range slice-wise interaction in a reduced resolution:
Revised: Then the second, global attention along the slice dimension is performed to model full-range slice-wise interaction in a reduced resolution:

Original: [Equation]
Revised: [Equation]

Original: where Q g , K g , and V g is the abstracted queries, keys, and values for the global attention obtained by applying W g ,q , W g ,k , and W g ,v to S .
Revised: where Q g , K g , and V g is the abstracted queries, keys, and values for the global attention obtained by applying W g ,q , W g ,k , and W g ,v to S .

Original: Finally, the output of the global attention is broadcasted to the original resolution and added to the local output:
Revised: Finally, the output of the global attention is broadcasted to the original resolution and added to the local output:

Original: [Equation]
Revised: [Equation]

Original: where ψ ( Y g ,i ) is a broadcasting function that restores only the sequence length, i.e., Y g ,
Revised: where ψ ( Y g ,i ) is a broadcasting function that restores only the sequence length, i.e., Y g ,

Original: i =
Revised: i =

Original: Y g ,i ′ for i ∈ Ω i ′ , since the output of the local attention still holds the full resolution.
Revised: Y g ,i ′ for i ∈ Ω i ′ , since the output of the local attention still holds the full resolution.

Original: This process can be implemented as a simple broadcast-add operation.
Revised: This process can be implemented as a simple broadcast-add operation.

Original: Slice Extension: Addressing Fine-Grained Context Fragmentation
Revised: Slice Extension: Addressing Fine-Grained Context Fragmentation

Original: The local attention of the stacked slice representation of the sequence S is strictly bounded, resulting in potential context fragmentation (Dai et al., 2019b).
Revised: The local attention of the stacked slice representation of the sequence S is strictly bounded, resulting in potential context fragmentation (Dai et al., 2019b).

Original: Although the global attention models the slice-wise interactions, it may not be sufficient to capture the important fine-grained token dependencies.
Revised: Although the global attention models the slice-wise interactions, it may not be sufficient to capture the important fine-grained token dependencies.

Original: To allow token-level interaction between slices, we slightly extend each slice into its neighbors, allowing slices to share tokens.
Revised: To allow token-level interaction between slices, we slightly extend each slice into its neighbors, allowing slices to share tokens.

Original: This extended local attention can be computed by having keys K l, ext and values V l, ext transformed from extended stacked slices S ext , where α denotes the extension ratio.
Revised: This extended local attention can be computed by having keys K l, ext and values V l, ext transformed from extended stacked slices S ext , where α denotes the extension ratio.

Original: The extension can be implemented by concatenating shifted-slices in keys and values:
Revised: The extension can be implemented by concatenating shifted-slices in keys and values:

Original: [Equation]
Revised: [Equation]

Original: 11) for α ≤ 3 where the Python notation of indices selection is used.
Revised: for α ≤ 3 where the Python notation of indices selection is used.

Original: Slice Extension and Multi-Scale Causal Mask for Autoregressive Sequence Modeling
Revised: Slice Extension and Multi-Scale Causal Mask for Autoregressive Sequence Modeling

Original: In order to apply CST to autoregressive sequence processing tasks, we propose custom causal masking and slice extension schemes.
Revised: In order to apply CST to autoregressive sequence processing tasks, we propose custom causal masking and slice extension schemes.

Original: For the local token-level attention, we apply an L × L causal mask map with −∞ above the main diagonal and zero elsewhere to the score tensor Q l K ⊤ l .
Revised: For the local token-level attention, we apply an L × L causal mask map with −∞ above the main diagonal and zero elsewhere to the score tensor Q l K ⊤ l .

Original: However, since the leftmost tokens in each slice have few (and possibly zero) tokens to attend to, we extend the keys and values only on the left-hand side to encourage better fine-grained dependency modeling, i.e.,[Equation]
Revised: However, since the leftmost tokens in each slice have few (and possibly zero) tokens to attend to, we extend the keys and values only on the left-hand side to encourage better fine-grained dependency modeling, i.e.,

Original: [Equation]
Revised: [Equation]

Original: For the global attention, an extra care must be taken to prevent leftward information leakage while computing the slice embedding via mean pooling.
Revised: For the global attention, an extra care must be taken to prevent leftward information leakage while computing the slice embedding via mean pooling.

Original: The diagonal elements in the N/L × N/L casual mask are set to be −∞ in contrast to the local counterpart.
Revised: The diagonal elements in the N/L × N/L casual mask are set to be −∞ in contrast to the local counterpart.

Original: In addition, at the slice index t , the shifted query Q g ,t − 1 is used for query instead of Q g ,t (note that K g ,t is handled by the global casual mask).
Revised: In addition, at the slice index t , the shifted query Q g ,t − 1 is used for query instead of Q g ,t (note that K g ,t is handled by the global casual mask).

Original: Increased Expressiveness of CST We mathematically motivate the improved performance of
Revised: Increased Expressiveness of CST We mathematically motivate the improved performance of

Original: CST over competing efficient Transformer models.
Revised: CST over competing efficient Transformer models.

Original: We show that a given function that CST is able to represent is ϵ away from a rational function with Euclidean degree 4 d ( ϵ ) , while competing approaches such as H-Transformer-1D Zhu & Soricut (2021) can only approach a rational function with Euclidean degree d ( ϵ ) .
Revised: We show that a given function that CST is able to represent is ϵ away from a rational function with Euclidean degree 4 d ( ϵ ) , while competing approaches such as H-Transformer-1D (Zhu & Soricut, 2021) can only approach a rational function with Euclidean degree d ( ϵ ) .

Original: This implies that CST is more expressive than other approaches that do not involve compositions of multi-scale attentions.
Revised: This implies that CST is more expressive than other approaches that do not involve compositions of multi-scale attentions.

Original: We state the main result below and present the details in Appendix A.5.
Revised: We state the main result below and present the details in Appendix A.5.

Original: Proposition 1.
Revised: Proposition 1.

Original: For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )
Revised: For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that,
Revised: O (log( ϵ )) such that,

Original: [Equation]
Revised: [Equation]

Original: where M corresponds to the total number of weights in CST , and S ϵM,d is the space of ratios of real analytic functions that are ϵ away from a rational function with Euclidean degree d with input in R M .
Revised: where M corresponds to the total number of weights in CST , and S ϵM,d is the space of ratios of real analytic functions that are ϵ away from a rational function with Euclidean degree d with input in R M .

Original: 4. M ULTI - SCALE VOLATILE INSTANT POSITIONAL EMBEDDING (MS-VIPE)
Revised: 4. M ULTI - SCALE VOLATILE INSTANT POSITIONAL EMBEDDING (MS-VIPE)

Original: Since we reduce the input lengths of both global and local attentions, the full positional embedding of the maximum input sequence length is no longer necessary.
Revised: Since we reduce the input lengths of both global and local attentions, the full positional embedding of the maximum input sequence length is no longer necessary.

Original: For the local attention, we can limit the positional embedding length to the attention range (i.e., the slice length L ), sharing the embedding across slices.
Revised: For the local attention, we can limit the positional embedding length to the attention range (i.e., the slice length L ), sharing the embedding across slices.

Original: In addition, as we aggregate the tokens from each slice for the global attention, it is more natural to have a separate positional embedding of length N/L at the scale of slice embedding instead of aggregating the full-resolution positional embedding with the same process as the token embedding.
Revised: In addition, as we aggregate the tokens from each slice for the global attention, it is more natural to have a separate positional embedding of length N/L at the scale of slice embedding instead of aggregating the full-resolution positional embedding with the same process as the token embedding.

Revised: The total number of parameters is (( L + N/L ) D ) , less than that of a conventional positional embedding ( ND ) .

Original: To this end, CST uses two positional embeddings P l ∈ R 1 × L × D and P g ∈ R N/L × D applied at different scales, in a fashion similar to that used by (Han et al., 2021), but with a few crucial differences: first, instead of adding the positional embedding to the stacked slices of token embedding at the embedding layer and having aggregated positional information as the layers stack up (Press et al., 2021), CST applies them instantly at the corresponding attentions in each layer before the linear transformations.
Revised: To this end, CST uses two positional embeddings P l ∈ R 1 × L × D and P g ∈ R N/L × D applied at different scales, in a fashion similar to that used by (Han et al., 2021), but with a few crucial differences: first, instead of adding the positional embedding to the stacked slices of token embedding at the embedding layer and having aggregated positional information as the layers stack up (Press et al., 2021), CST applies them instantly at the corresponding attentions in each layer before the linear transformations.

Original: Second, the positional embeddings are applied for the queries and the keys, not for the values, to prevent accumulation of positional information in sequence representations.
Revised: Second, the positional embeddings are applied for the queries and the keys, not for the values, to prevent accumulation of positional information in sequence representations.

Original: Since the positional embedding in all layers are added, they can be accumulated over the layers and can undesirably dominate the contextual information at top layers which potentially leads to performance degradation.
Revised: Since the positional embedding in all layers are added, they can be accumulated over the layers and can undesirably dominate the contextual information at top layers which potentially leads to performance degradation.

Original: Our experiments in Section A.6.2 show that the multi-scale volatile instant positional embedding (MS-VIPE) is more effective as compared to the conventional absolute full-length positional embedding.
Revised: Our experiments in Section A.6.2 show that the multi-scale volatile instant positional embedding (MS-VIPE) is more effective as compared to the conventional absolute full-length positional embedding.

Original: Equations (8) and (9) are rewritten as:
Revised: Equations (8) and (9) are rewritten as:

Original: [Equation]
Revised: [Equation]

Original: For the extended local attention, we modify the corresponding positional embedding P l to have the extended length αL , similarly to K l, ext .
Revised: For the extended local attention, we modify the corresponding positional embedding P l to have the extended length αL , similarly to K l, ext .

Original: 4. C OMPLEXITY AND P ARAMETER R EDUCTION IN CST
Revised: 4. C OMPLEXITY AND P ARAMETER R EDUCTION IN CST

Original: CST has linear plus decimated quadratic complexity
Revised: CST has linear plus decimated quadratic complexity

Original: O ( NL + N 2 /L 2 )
Revised: O ( NL + N 2 /L 2 )

Original: compared to the O ( N 2 ) complexity of the standard Transformer.
Revised: compared to the O ( N 2 ) complexity of the standard Transformer.

Original: However, in a practical range of sequence lengths, e.g., from a few hundreds to a few tens of thousands, CST has a comparable or better efficiency than other efficient transformers with linear complexity O ( NM ) withtypical choices of  M , e.g., from 64 to 256, or higher for even longer sequences. This is because the slice length L for CST is typically less thanthe abstraction lengths M of other linear-complexity abstractive attentions. See Section A. for more details.
Revised: However, in a practical range of sequence lengths, e.g., from a few hundreds to a few tens of thousands, CST has a comparable or better efficiency than other efficient transformers with linear complexity O ( NM ) with choices of the abstraction lengths M , e.g., from 64 to 256 or higher for even longer sequences, since the slice length L for CST is typically less than M , even with additional for query, key, and value transformations of 3( N/L ) d 2 , which is almost negligible.

Original: Furthermore, unlike most efficient transformers that can have similar or higher complexity than the standard transformers with short input sequences, CST has better efficiency even in such cases.
Revised: Furthermore, unlike most efficient transformers that can have similar or higher complexity than the standard transformers with short input sequences, CST has better efficiency even in such cases.

Original: See Section A.4 for a complexity comparison.
Revised: See Section A.4 for more details of practical complexity analysis.

Original: Additionally, since the global attention in CST takes the local attention output as input, it requires only a slight in crease in computation compared with linear attention (i.e. 3( N/L ) d 2 versus ( NMd ) to get queries, keys, and values.

Original: Using extended local attention with extension ratio α leads to a complexity increase for the local attention to O ( αNL ) and a parameter count increase of αLD for the local positional embedding.

Original: Two sets of W q , W k , W v transformations in CST may also affect the parameter count.

Original: While TNT (Han et al., 2021) reduces the hidden dimension for local attention to limit the network size increase, we keep the same dimension for representations at different scales and share the weights between local and global attentions, resulting in no increase in parameter count.

Original: Another benefit of having the stacked slice representation is the reduction in storage requirement for the positional embedding.

Original: Since the lengths for local and global attentions are L and N/L , respectively, CST has fewer parameters (( L + N/L ) D ) than that of a conventional positional embedding ( ND ) .

Original: This benefit can be significant for tasks with very long sequences while it can be relatively small when the network size is very large.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To demonstrate the computational efficiency and sequence modeling capability of CST , we evaluate our model in three different contexts: (1) bidirectional long-range dependency modeling on classification tasks (2) word-level auto-regressive language modeling and (3) masked language modeling and transfer learning to natural language understanding on short sequences.
Revised: To demonstrate the computational efficiency and sequence modeling capability of CST , we evaluate our model in three different contexts: (1) bidirectional long-range dependency modeling on classification tasks (2) word-level auto-regressive language modeling and (3) masked language modeling and transfer learning to natural language understanding on short sequences.

Original: Throughout this section, unless stated otherwise, it is implied that we train each model from random initialization, reporting its test performance from the model with the best validation result in each case.
Revised: Throughout this section, unless stated otherwise, it is implied that we train each model from random initialization, reporting its test performance from the model with the best validation result in each case.

Original: 5. B IDIRECTIONAL L ONG -R ANGE D EPENDENCY M ODELING
Revised: 5. B IDIRECTIONAL L ONG -R ANGE D EPENDENCY M ODELING

Original: Datasets and Baselines The Long Range Arena (LRA) benchmark (Tay et al., 2020b) is a suite of classification tasks that evaluate long-range dependency modeling capabilities with datasets from several different modalities with lengths ranging from 1k to 16k.
Revised: Datasets and Baselines The Long Range Arena (LRA) benchmark (Tay et al., 2020b) is a suite of classification tasks that evaluate long-range dependency modeling capabilities with datasets from several different modalities with lengths ranging from 1k to 16k.

Original: We evaluate CST on the five tasks broadly used in literature, ListOps, Text, Retrieval, Image , and Pathfinder , where the maximumsequence length is 4k.
Revised: We evaluate CST on the five tasks broadly used in literature, ListOps, Text, Retrieval, Image , and Pathfinder , where the maximum

Original: We compare CST ’s efficiency and performance against several state-ofthe-art efficient Transformer models.

Original: Specifically, we evaluate our model and other Transformer

Original: 5 Relative Speed
Revised: 5 Relative Speed

Original: L R A S c o r e
Revised: L R A S c o r e

Original: FNet
Revised: FNet

Original: H-Transformer-1D
Revised: H-Transformer-1D

Original: Linear Transformer
Revised: Linear Transformer

Original: Linformer
Revised: Linformer

Original: LS-Transformer
Revised: LS-Transformer

Original: Luna Nystromformer
Revised: Luna Nystromformer

Original: Performer
Revised: Performer

Original: Reformer
Revised: Reformer

Original: Transformer
Revised: Transformer

Original: CST (L=16) CST (L=8) CST (L=8)
Revised: CST (L=16) CST (L=8) CST (L=8)

Original: CST (L=16) 0.
Revised: CST (L=16) 0.

Original: P a r a m e t e r C o un t ( M )
Revised: P a r a m e t e r C o un t ( M )

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FNet

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Linear
Revised: Linear

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Performer37.
Revised: Performer

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Reformer 18.

Original: Linformer 38.
Revised: Linformer 38.

Original: Luna 38.
Revised: Luna 38.

Original: Nystr ¨ omformer 37.
Revised: Nystr ¨ omformer 37.

Original: H-Transformer-1D 38.
Revised: H-Transformer-1D 38.

Original: Long Short Transformer 38.
Revised: Long Short Transformer 38.

Original: Scatterbrain 38.
Revised: Scatterbrain 38.

Original: CST ( L = 8 ) 37.
Revised: CST ( L = 8 ) 37.

Original: CST ( L = 16 ) 37.  models from a group of non-abstractive attention methods: FNet (Lee-Thorp et al., 2021), Linear
Revised: CST ( L = 16 ) 37. sequence length is 4k. We compare CST ’s efficiency and performance against several state-ofthe-art efficient Transformer models. Specifically, we evaluate our model and other Transformer models from a group of non-abstractive attention methods: FNet (Lee-Thorp et al., 2021), Linear

Original: Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020), Cosformer (Qin et al., 2021), a group of abstractive attention methods: Reformer (Kitaev et al., 2019), Linformer (Wang et al., 2020), Luna (Ma et al., 2021), Nystr¨omformer (Xiong et al., 2021), and another group of multi-scale multi-range attention methods: H-Transformer-1D (Zhu & Soricut, 2021), Long Short
Revised: Transformer (Katharopoulos et al., 2020), Performer (Choromanski et al., 2020), Cosformer (Qin et al., 2021), a group of abstractive attention methods: Reformer (Kitaev et al., 2019), Linformer (Wang et al., 2020), Luna (Ma et al., 2021), Nystr¨omformer (Xiong et al., 2021), and another group of multi-scale multi-range attention methods: H-Transformer-1D (Zhu & Soricut, 2021), Long Short

Original: Transformer (Zhu et al., 2021), and Scatterbrain (Chen et al., 2021a), as well as the standard
Revised: Transformer (Zhu et al., 2021), and Scatterbrain (Chen et al., 2021a), as well as the standard

Original: Transformer (Vaswani et al., 2017).
Revised: Transformer (Vaswani et al., 2017).

Original: We outline experimental setup for the Long Range Arena in
Revised: We outline experimental setup for the Long Range Arena in

Original: Appendix A.6.1.
Revised: Appendix A.6.1.

Original: Our setup is favorable to the baseline Transformer to provide a fair condition to the challengers, explaining the difference in performances of Transformers from previous work.
Revised: Our setup is favorable to the baseline Transformer to provide a fair condition to the challengers, explaining the difference in performances of Transformers from previous work.

Original: Results
Revised: Results

Original: Our experimental results on the LRA benchmark are given in Table 1.
Revised: Our experimental results on the LRA benchmark are given in Table 1.

Original: We observe that
Revised: We observe that

Original: CST surpasses the state-of-the-art efficient Transformers with large margins.
Revised: CST surpasses the state-of-the-art efficient Transformers with large margins.

Original: The best performance is achieved using a slice length 8 , outperforming the Transformer baseline by 4 . 23 points in average score across all tasks.
Revised: The best performance is achieved using a slice length 8 , outperforming the Transformer baseline by 4 . 23 points in average score across all tasks.

Original: The largest improvements are observed on the Text, Retrieval , and Image tasks, while a minimal degradation is observed on the ListOps task.
Revised: The largest improvements are observed on the Text, Retrieval , and Image tasks, while a minimal degradation is observed on the ListOps task.

Original: In addition, as discussed in
Revised: In addition, as discussed in

Original: Section 4.1, CST has the added benefit of requiring fewer parameters for a given model size thanksto its positional embedding characteristics, although this benefit may be marginal as the network size increases.
Revised: Section 4.1, CST has the added benefit of requiring fewer parameters for a given model size thanks

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]

Original: [Table]
Revised: [Table]

Revised: to its positional embedding characteristics, although this benefit may be marginal as the network size increases.

Original: We also measure relative speed, and relative GPU memory usage for each model under comparison on a NVIDIA Tesla V100 GPU with 32GB memory on Retrieval dataset with the sequence length of 4k and report the results in Figure 2.
Revised: We also measure relative speed, and relative GPU memory usage for each model under comparison on a NVIDIA Tesla V100 GPU with 32GB memory on Retrieval dataset with the sequence length of 4k and report the results in Figure 2.

Original: Here, relative speed is computed as the number of training steps per second divided by that of Transformer, and GPU memory usage is measured using the
Revised: Here, relative speed is computed as the number of training steps per second divided by that of Transformer [Table] and GPU memory usage is measured using the

Original: PyTorch command, torch.cuda.max_memory_allocated() , both on the Retrieval task.
Revised: PyTorch command, torch.cuda.max_memory_allocated() , both on the Retrieval task.

Original: It is clearly seen that CST outperforms all models while having competitive computation and memory efficiency to other efficient Transformer models.
Revised: It is clearly seen that CST outperforms all models while having competitive computation and memory efficiency to other efficient Transformer models.

Original: We present an ablation study regarding the choice of positional embedding and aggregation method on the LRA benchmark in Section A.6.2.
Revised: We present an ablation study regarding the choice of positional embedding and aggregation method on the LRA benchmark in Section A.6.2.

Original: In summary, variants of CST consistently outperform all other efficient models (Table 7), except that switching the MSMRA composition order, i.e. attentions in global-local order, performs slightly worse than Long Short Transformer (Zhu et al., 2021).
Revised: In summary, variants of CST consistently outperform all other efficient models (Table 7), except that switching the MSMRA composition order, i.e. attentions in global-local order, performs slightly worse than Long Short Transformer (Zhu et al., 2021).

Original: We demonstrate that the composition of MSMRA in the local-global order and the proposed positional embedding are the most important components contributing to CST ’s accuracy.
Revised: We demonstrate that the composition of MSMRA in the local-global order and the proposed positional embedding are the most important components contributing to CST ’s accuracy.

Original: While the best overall performance is achieved with L = 8 , we find that the slice length L has less effect than other hyperparameters.
Revised: While the best overall performance is achieved with L = 8 , we find that the slice length L has less effect than other hyperparameters.

Original: We also find that attention outperforms an MLP-based local token mixing scheme and that mean pooling aggregation outperforms max pooling and linear projection (Table 8).
Revised: We also find that attention outperforms an MLP-based local token mixing scheme and that mean pooling aggregation outperforms max pooling and linear projection (Table 8).

Original: 5. A UTOREGRESSIVE L ANGUAGE M ODELING
Revised: 5. A UTOREGRESSIVE L ANGUAGE M ODELING

Original: In this section, to evaluate its applicability as language model, we conduct an autoregressive language modeling experiment on the word-level WikiText-103 dataset (Merity et al., 2016).
Revised: In this section, to evaluate its applicability as language model, we conduct an autoregressive language modeling experiment on the word-level WikiText-103 dataset (Merity et al., 2016).

Original: Dataset and Experimental Setup
Revised: Dataset and Experimental Setup

Original: The WikiText-103 (Merity et al., 2016) dataset is a collection of featured articles on Wikipedia.
Revised: The WikiText-103 (Merity et al., 2016) dataset is a collection of featured articles on Wikipedia.

Original: It has 103 M/ 218 K/
Revised: It has 103 M/ 218 K/

Original: 246 K words in training/validation/test sets, respectively.
Revised: 246 K words in training/validation/test sets, respectively.

Original: The task in this experiment is to predict the next word given the history of past words.
Revised: The task in this experiment is to predict the next word given the history of past words.

Original: We follow the experimental setup in (Nguyen et al., 2021) and that in (Chen et al., 2021a) for the context window length 256 and 1024 , respectively, and train CST s to compare the results with their reported results.
Revised: We follow the experimental setup in (Nguyen et al., 2021) and that in (Chen et al., 2021a) for the context window length 256 and 1024 , respectively, and train CST s to compare the results with their reported results.

Original: While the dataset allows for a larger context, we used the 256 and 1024 context window lengths to match the baselines.
Revised: While the dataset allows for a larger context, we used the 256 and 1024 context window lengths to match the baselines.

Original: CST also uses the causal mask and slice embedding described in Section 4.1.
Revised: CST also uses the causal mask and slice embedding described in Section 4.1.

Original: More discussion of the experimental setup can be found in Appendix A.7.1.
Revised: More discussion of the experimental setup can be found in Appendix A.7.1.

Original: Results We report the perplexities of the best performing variant of CST ( L = 32 , α = 3 ) compared to other state-of-the-art efficient Transformer models in Table 2 and Table 4.
Revised: Results We report the perplexities of the best performing variant of CST ( L = 32 , α = 3 ) compared to other state-of-the-art efficient Transformer models in Table 2 and Table 4.

Original: CST outperforms other efficient Transformer including Linear Transformer (Katharopoulos et al., 2020) and Performer (Choromanski et al., 2020), kernel method-based linear-complexity models, and
Revised: CST outperforms other efficient Transformer including Linear Transformer (Katharopoulos et al., 2020) and Performer (Choromanski et al., 2020), kernel method-based linear-complexity models, and

Original: FMMformer, while being comparable to Scatterbrain (Chen et al., 2021a), noting that the latter two

Original: [Table]
Revised: [Table]

Original: [Table] are MSMRA-based models.
Revised: FMMformer, while being comparable to Scatterbrain (Chen et al., 2021a), noting that the latter two are MSMRA-based models.

Original: In Table 3, we provide an ablation study demonstrating the impact of
Revised: In Table 3, we provide an ablation study demonstrating the impact of

Original: L and α on validation and test perplexity.
Revised: L and α on validation and test perplexity.

Original: We observe that longer local attention length leads to better perplexities while being much shorter than the the context window length, e.g. CST with
Revised: We observe that longer local attention length leads to better perplexities while being much shorter than the the context window length, e.g. CST with

Original: L = 32 , α = 3 has the local window length 64 that is 4 and 16 times smaller than the context window lengths 256 and 1024 , respectively.
Revised: L = 32 , α = 3 has the local window length 64 that is 4 and 16 times smaller than the context window lengths 256 and 1024 , respectively.

Original: We believe that addressing the missing global context, as discussed in A.9, and a better hyperparameter search will improve the perplexity.
Revised: We believe that addressing the missing global context, as discussed in A.9, and a better hyperparameter search will improve the perplexity.

Original: We also observe consistent 1 .
Revised: We also observe consistent 1 .

Original: 5 x speed-ups across configurations compared to the standard Transformer in Table 3.
Revised: 5 x speed-ups across configurations compared to the standard Transformer in Table 3.

Original: Additional Experiment on PG-19 We additionally conduct an experiment on PG-19 dataset (Rae et al., 2019).
Revised: Additional Experiment on PG-19 We additionally conduct an experiment on PG-19 dataset (Rae et al., 2019).

Original: We use various combinations of sequence length N , slice length L , and extension ratio α to match the attention complexity of CST s to those of Transformers with N = 256 and 512 .
Revised: We use various combinations of sequence length N , slice length L , and extension ratio α to match the attention complexity of CST s to those of Transformers with N = 256 and 512 .

Original: While the validation and the test perplexities with discussion can be found in A.7.2, we observe CST consistently outperforms the Transformer counterparts with the same attention complexities.
Revised: While the validation and the test perplexities with discussion can be found in A.7.2, we observe CST consistently outperforms the Transformer counterparts with the same attention complexities.

Original: 5. B IDIRECTIONAL L ANGUAGE M ODELING AND T RANSFER L EARNING
Revised: 5. B IDIRECTIONAL L ANGUAGE M ODELING AND T RANSFER L EARNING

Original: We further evaluate CST on bidirectional language modeling and transfer learning on GLUE benchmark with relatively short input sequences, i.e., N = 128 .
Revised: We further evaluate CST on bidirectional language modeling and transfer learning on GLUE benchmark with relatively short input sequences, i.e., N = 128 .

Original: Masked language modeling (MLM) was proposed in (Devlin et al., 2018) as a pretraining method of transformer encoder models and it greatly improves downstream natural language understanding (NLU) performances when fine-tuning the pretrained models.
Revised: Masked language modeling (MLM) was proposed in (Devlin et al., 2018) as a pretraining method of transformer encoder models and it greatly improves downstream natural language understanding (NLU) performances when fine-tuning the pretrained models.

Original: We follow the experimental setup of (Devlin et al., 2018) for conducting both pre-training and fine-tuning including datasets, masking rate, batch size, optimizer settings, and evaluation metrics with a few exceptions.
Revised: We follow the experimental setup of (Devlin et al., 2018) for conducting both pre-training and fine-tuning including datasets, masking rate, batch size, optimizer settings, and evaluation metrics with a few exceptions.

Original: We report the MLM validation perplexities and GLUE scores with accuracy on each task for a variation of L and α in Table 5.
Revised: We report the MLM validation perplexities and GLUE scores with accuracy on each task for a variation of L and α in Table 5.

Original: Further experimental details, more experimental results, and discussion can be found in A.8.
Revised: Further experimental details, more experimental results, and discussion can be found in A.8.

Original: CST consistently outperforms other efficient transformers and closes the gap to the baseline BERT model by 0.46 validation perplexity and 2.45 points in GLUE score.
Revised: CST consistently outperforms other efficient transformers and closes the gap to the baseline BERT model by 0.46 validation perplexity and 2.45 points in GLUE score.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present Composite Slice Transformer ( CST ), an efficient Transformer-based network equipped with composition of multi-scale multi-range attentions.
Revised: In this paper, we present Composite Slice Transformer ( CST ), an efficient Transformer-based network equipped with composition of multi-scale multi-range attentions.

Original: Using stacked slice representation of input sequences, a CST layer performs a set of local fine-grained attention and global coarsegrained attention in a sequential manner at a low complexity cost of O ( NL + N 2 /L 2 ) .
Revised: Using stacked slice representation of input sequences, a CST layer performs a set of local fine-grained attention and global coarsegrained attention in a sequential manner at a low complexity cost of O ( NL + N 2 /L 2 ) .

Original: In addition to the reduced complexity, we also show that CST improves performance on various sequence modeling tasks.
Revised: In addition to the reduced complexity, we also show that CST improves performance on various sequence modeling tasks.

Original: On Long Range Arena, word-level autoregressive language modeling on WikiText-103, masked language modeling, and natural language understanding benchmarks, CST significantly surpasses strong baselines including recently proposed efficient Transformers, and sometimes the standard
Revised: On Long Range Arena, word-level autoregressive language modeling on WikiText-103, masked language modeling, and natural language understanding benchmarks, CST significantly surpasses strong baselines including recently proposed efficient Transformers, and sometimes the standard

Original: Transformer.
Revised: Transformer.

Original: We highlight limitations and potential directions for future work in A.9.
Revised: We highlight limitations and potential directions for future work in A.9.

Original: A A PPENDIX A. A BSTRACTIVE A TTENTIONS
Revised: A A PPENDIX A. A BSTRACTIVE A TTENTIONS

Original: Abstractive attentions are the family of efficient attention approaches in which the lengths of the attention operands are reduced to M ( < N ) by applying an abstraction function, resulting in reduced complexity of the attention while retaining the form of basic attention computation in Eq. 3.
Revised: Abstractive attentions are the family of efficient attention approaches in which the lengths of the attention operands are reduced to M ( < N ) by applying an abstraction function, resulting in reduced complexity of the attention while retaining the form of basic attention computation in Eq. 3.

Original: Abstractive attentions can be further categorized to either resolution preserving or non-preserving , according to which operands are chosen to be abstracted.
Revised: Abstractive attentions can be further categorized to either resolution preserving or non-preserving , according to which operands are chosen to be abstracted.

Original: Resolution non-preserving attention is the result of abstracted queries, further producing abstracted output.
Revised: Resolution non-preserving attention is the result of abstracted queries , further producing abstracted output.

Original: This categorization is determined by the requirement of the given task.
Revised: This categorization is determined by the requirement of the given task.

Original: For example, tasks such as language modeling and machine translation require the full resolution at the output to be retained.
Revised: For example, tasks such as language modeling and machine translation require the full resolution at the output to be retained.

Original: In such cases, it is common to have only abstracted keys and values while the query resolution is retained; the abstractive attention can be expressed as
Revised: In such cases, it is common to have only abstracted keys and values while the query resolution is retained; the abstractive attention can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where Ω j ′ denotes the abstraction range with the cardinality | Ω j ′ |
Revised: where Ω j ′ denotes the abstraction range with the cardinality | Ω j ′ |

Original: = M k for the j ′ th key abstraction k j ′ and ϕ ( · ) : K Ω j ′ ∈ R | Ω j ′ |× d h → k j ′ ∈ R 1 ×
Revised: = M k for the j ′ th key abstraction k j ′ and ϕ ( · ) : K Ω j ′ ∈ R | Ω j ′ |× d h → k j ′ ∈ R 1 ×

Original: d h is a many-to-one abstraction function.
Revised: d h is a many-to-one abstraction function.

Original: The abstracted values V j ′ can be expressed similarly.
Revised: The abstracted values V j ′ can be expressed similarly.

Original: The queries Q can be similarly abstracted to Q as
Revised: The queries Q can be similarly abstracted to Q as

Original: [Equation]
Revised: [Equation]

Original: 19) where q i ′ is the i ′ th row of Q .
Revised: 19) where q i ′ is the i ′ th row of Q .

Original: The attention mechanism is reduced to
Revised: The attention mechanism is reduced to

Original: [Equation]
Revised: [Equation]

Original: where an attention output vector y i ′ is obtained at each abstract position i ′ .
Revised: where an attention output vector y i ′ is obtained at each abstract position i ′ .

Original: In order to restore the resolution of the output, we define a one-to-many mapping function ψ y as
Revised: In order to restore the resolution of the output, we define a one-to-many mapping function ψ y as

Original: [Equation]
Revised: [Equation]

Original: Resolution non-preserving abstraction is often used for the tasks where the full-resolution output is not needed such as sequence-level classification problem.
Revised: Resolution non-preserving abstraction is often used for the tasks where the full-resolution output is not needed such as sequence-level classification problem.

Original: However, in some cases, with additional processing leveraging representations at a lower layer, e.g., cross attention with input tokens, it is possible to restore the resolution (Dai et al., 2020; Jaegle et al., 2021).
Revised: However, in some cases, with additional processing leveraging representations at a lower layer, e.g., cross attention with input tokens, it is possible to restore the resolution (Dai et al., 2020; Jaegle et al., 2021).

Original: A. M ULTI -S CALE M ULTI -R ANGE A TTENTIONS (MSMRA)
Revised: A. M ULTI -S CALE M ULTI -R ANGE A TTENTIONS (MSMRA)

Original: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically come at the cost of degraded benchmark performance.
Revised: Although many previous abstractive attention approaches have achieved sub-quadratic or linear complexities, they typically come at the cost of degraded benchmark performance.

Original: However, Transformerbased models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer; in fact, these models can outperform the standard Transformer on some benchmarks, while still maintaining efficiency (Zhu & Soricut, 2021;
Revised: However, Transformerbased models that leverage multi-scale attention by combining local attention and global attention perform competitively against the standard Transformer; in fact, these models can outperform the standard Transformer on some benchmarks, while still maintaining efficiency (Zhu & Soricut, 2021;

Original: Zhu et al., 2021; Nguyen et al., 2021;
Revised: Zhu et al., 2021; Nguyen et al., 2021;

Original: Ren et al., 2021).
Revised: Ren et al., 2021).

Original: The local attention, also known as sliding window attention, limits the attention range to the vicinity of query locations.
Revised: The local attention, also known as sliding window attention, limits the attention range to the vicinity of query locations.

Original: That is, the key abstraction with the whole abstraction range and the query locationdependent abstraction function becomes K l ,i =
Revised: That is, the key abstraction with the whole abstraction range and the query locationdependent abstraction function becomes K l ,i =

Original: ϕ sliding k,i ( K )
Revised: ϕ sliding k,i ( K )

Original: =
Revised: =

Original: K ⊙ ( H ( i − j − w/ 2) − H ( i − j + w/ 2)) for the i th query token, where H is the Heaviside step function, w is the window length, and ⊙ is the element-wise product.
Revised: K ⊙ ( H ( i − j − w/ 2) − H ( i − j + w/ 2)) for the i th query token, where H is the Heaviside step function, w is the window length, and ⊙ is the element-wise product.

Original: This results in the following equation:
Revised: This results in the following equation:

Original: [Equation]
Revised: [Equation]

Original: For better computational efficiency, block-wise key abstraction in Equation equation 22 can be adopted as K l ,i = ϕ block k,i ( K )
Revised: For better computational efficiency, block-wise key abstraction in Equation equation 22 can be adopted as K l ,i = ϕ block k,i ( K )

Original: =
Revised: =

Original: K ⊙ ( H ( t i − j − w/ 2) − H ( t i − j + w/ 2)) for a block-wise attention where t i =
Revised: K ⊙ ( H ( t i − j − w/ 2) − H ( t i − j + w/ 2)) for a block-wise attention where t i =

Original: ( b − 1 / 2) w for the block index b such that ( b − 1) w ≤ i < bw .
Revised: ( b − 1 / 2) w for the block index b such that ( b − 1) w ≤ i < bw .

Original: For the global attention, abstractive attention with either positional abstractions (Zhu & Soricut, 2021;
Revised: For the global attention, abstractive attention with either positional abstractions (Zhu & Soricut, 2021;

Original: Yang et al., 2021; Ren et al., 2021) or contextual abstractions (Ma et al., 2021; Zhu et al., 2021) can be employed.
Revised: Yang et al., 2021; Ren et al., 2021) or contextual abstractions (Ma et al., 2021; Zhu et al., 2021) can be employed.

Original: The former can be loosely seen as having patch embedding in ViT (Dosovitskiy et al., 2020).
Revised: The former can be loosely seen as having patch embedding in ViT (Dosovitskiy et al., 2020).

Original: MSMRAs can also be categorized according to how the two attentions are combined.
Revised: MSMRAs can also be categorized according to how the two attentions are combined.

Original: While one approach involves concatenating the abstractions of multi-scale keys and values for a single attention operation (Zhu & Soricut,
Revised: While one approach involves concatenating the abstractions of multi-scale keys and values for a single attention operation (Zhu & Soricut,

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: another separates attentions at different scales and combining the outputs (Han et al., 2021)
Revised: another separates attentions at different scales and combining the outputs (Han et al., 2021)

Original: (possibly with some weighting coefficients):
Revised: (possibly with some weighting coefficients):

Original: [Equation]
Revised: [Equation]

Original: In this case, other non-attentive methods such as kernel method (Nguyen et al., 2021) can also be used for the global attention.
Revised: In this case, other non-attentive methods such as kernel method (Nguyen et al., 2021) can also be used for the global attention.

Original: CST belongs to the latter of the two approaches, where the local and global attentions are performed separately and their outputs are combined.
Revised: CST belongs to the latter of the two approaches, where the local and global attentions are performed separately and their outputs are combined.

Original: This is closely related to Transformer-In-Transformer ( TNT ) (Han et al., 2021); however, since TNT has a path for the local attention that is independent of the global attention, information exchange between patches is not allowed for the features in the local attention path.
Revised: This is closely related to Transformer-In-Transformer ( TNT ) (Han et al., 2021); however, since TNT has a path for the local attention that is independent of the global attention, information exchange between patches is not allowed for the features in the local attention path.

Original: Unlike TNT , the composition of multi-granular attentions in CST enables two-way information passing.
Revised: Unlike TNT , the composition of multi-granular attentions in CST enables two-way information passing.

Original: This is more suitable for modeling highly non-stationary data, such as natural language text data for example, where the locality assumption does not hold.
Revised: This is more suitable for modeling highly non-stationary data, such as natural language text data for example, where the locality assumption does not hold.

Original: A. A RCHITECTURE OF C OMPOSITE S LICE T RANSFORMER N ETWORK
Revised: A. A RCHITECTURE OF C OMPOSITE S LICE T RANSFORMER N ETWORK

Original: We present the overall architecture of CST alongside the detailed CSA architecture in Figure 3.
Revised: We present the overall architecture of CST alongside the detailed CSA architecture in Figure 3.

Original: CST consists of a fine-grained local attention with computational complexity O ( NL ) and a coarse-grained global attention with complexity O
Revised: CST consists of a fine-grained local attention with computational complexity O ( NL ) and a coarse-grained global attention with complexity O

Original: ( N 2 /L 2 ) .
Revised: ( N 2 /L 2 ) .

Original: A shared multi-scale positional embedding, MS-VIPE, is applied to all layers.
Revised: A shared multi-scale positional embedding, MS-VIPE, is applied to all layers.

Original: CSA replaces self-attention sublayer in each Transformer block to form a
Revised: CSA replaces self-attention sublayer in each Transformer block to form a

Original: Composite Slice Transformer ( CST ) network.
Revised: Composite Slice Transformer ( CST ) network.

Original: [Figure]
Revised: Two sets of W q , W k , W v transformations in CST may also affect the parameter count. While TNT (Han et al., 2021) reduces the hidden dimension for local attention to limit the network size increase, we keep the same dimension for representations at different scales and share the weights between local and global attentions, resulting in no increase in parameter count.

Original: A. C ONSIDERATION ON P RACTICAL E FFICIENCY AND E FFECTIVENESS OF CST
Revised: A. C ONSIDERATION ON P RACTICAL E FFICIENCY AND E FFECTIVENESS OF CST

Original: In this section [Figure] we provide an intuition for how and why a multi-scale multi-range attention with
Revised: In this section, we provide an intuition for how and why a multi-scale multi-range attention with

Original: O ( NL + N 2 /L 2 ) complexity can be a better alternative to the vanilla self-attention than abstraction attentions with linear complexities in terms of both complexity and modeling capability.
Revised: O ( NL + N 2 /L 2 ) complexity can be a better alternative to the vanilla self-attention than abstraction attentions with linear complexities in terms of both complexity and modeling capability.

Original: Improved expressiveness from composition of them is further discussed in Section A.5.
Revised: Improved expressiveness from composition of them is further discussed in Section A.5.

Original: A linear complexity can be easily considered to be more efficient than a quadratic complexity.
Revised: A linear complexity can be easily considered to be more efficient than a quadratic complexity.

Original: A linear complexity often accompanies another variable; i [Figure] e., the abstraction length M in the case of abstractive attention such as (Wang et al., 2020; Ma et al., 2021), resulting in O ( NM ) complexity.
Revised: A linear complexity often accompanies another variable; i.e., the abstraction length M in the case of abstractive attention such as (Wang et al., 2020; Ma et al., 2021), resulting in O ( NM ) complexity.

Original: It is obvious that the quadratic complexity O ( N 2 ) is higher than the linear complexity whenever the abstraction length
Revised: It is obvious that the quadratic complexity O ( N 2 ) is higher than the linear complexity whenever the abstraction length

Original: M is smaller than the sequence length, which is mostly true by the definition of abstraction .
Revised: M is smaller than the sequence length, which is mostly true by the definition of abstraction .

Original: However, when a notion of sequence decimation comes into play, we come to a little different conclusion.
Revised: However, when a notion of sequence decimation comes into play, we come to a little different conclusion.

Original: Consider an attention on an abstracted sequence of patch embeddings in (Dosovitskiy et al., 2020) or slice embeddings in CST .
Revised: Consider an attention on an abstracted sequence of patch embeddings in (Dosovitskiy et al., 2020) or slice embeddings in CST .

Original: Then the complexity of the abstractive attention becomes a decimated quadratic O ( N 2 /L 2 ) .
Revised: Then the complexity of the abstractive attention becomes a decimated quadratic O ( N 2 /L 2 ) .

Original: In addition to this, if a full-resolution local attention is used as in CST or (Han et al., 2021) as a multi-scale multi-range attention, the complexity becomes
Revised: In addition to this, if a full-resolution local attention is used as in CST or (Han et al., 2021) as a multi-scale multi-range attention, the complexity becomes

Original: O ( N 2 /L 2 + N/L · L 2 )
Revised: O ( N 2 /L 2 + N/L · L 2 )

Original: =
Revised: =

Original: O ( N 2 /L 2 + NL ) .
Revised: O ( N 2 /L 2 + NL ) .

Original: We plot the comparison of the linear complexity O ( NM ) and O ( N 2 /L 2 + NL ) in Figure 4a with several practical choices of the number of abstractions M and the decimation ratio, e.g., slice length, L .
Revised: We plot the comparison of the linear complexity O ( NM ) and O ( N 2 /L 2 + NL ) in Figure 4a with several practical choices of the number of abstractions M and the decimation ratio, e.g., slice length, L .

Original: Here, one can find that a decimated quadratic complexity attention can have better efficiency a linear complexity attention, when the sequence length is less than a few tens of thousands which is considered as a practical range of sequence lengths in many tasks and data types.
Revised: Here, one can find that a decimated quadratic complexity attention can have better efficiency a linear complexity attention, when the sequence length is less than a few tens of thousands which is considered as a practical range of sequence lengths in many tasks and data types.

Original: Figures 4b shows effective number of tokens for each abstraction for both cases with the same choices of M and L .
Revised: Figures 4b shows effective number of tokens for each abstraction for both cases with the same choices of M and L .

Original: While a decimation, e.g., slicing, retains constant numbers of tokens in each abstraction, the linear complexity attention methods that uses fixed number of abstraction has linearly increasing effective number of tokens per abstraction with respect to the sequence length.
Revised: While a decimation, e.g., slicing, retains constant numbers of tokens in each abstraction, the linear complexity attention methods that uses fixed number of abstraction has linearly increasing effective number of tokens per abstraction with respect to the sequence length.

Original: Given a fixed hidden dimension, larger number of effective tokens per abstraction requires the abstraction process to compress more information, resulting in loss of potentially important information and negatively affecting the modeling capability.
Revised: Given a fixed hidden dimension, larger number of effective tokens per abstraction requires the abstraction process to compress more information, resulting in loss of potentially important information and negatively affecting the modeling capability.

Original: Since the complexity of CST has a decimated quadratic term N 2 /L 2 , the efficiency benefit compared to linear-complexity method keeps closing as the sequence length becomes larger, and eventually  the efficiency order is reversed, as it can be seen in 4a.
Revised: Since the complexity of CST has a decimated quadratic term N 2 /L 2 , the efficiency benefit compared to linear-complexity method keeps closing as the sequence length becomes larger, and eventually [Figure] the efficiency order is reversed, as it can be seen in 4a.

Original: However, we argue that our model can still be beneficial in two aspects.
Revised: However, we argue that our model can still be beneficial in two aspects.

Original: First, it is still more efficient than the standard Transformer.
Revised: First, it is still more efficient than the standard Transformer.

Original: Second, it is possible to find an optimal set of hyperparameters such as the slice length L and the extension ratio α which result in comparably efficient and more effective in modeling capability to the linearcomplexity counterpart, considering the discussion made in this section on the effective number of tokens per abstraction.
Revised: Second, it is possible to find an optimal set of hyperparameters such as the slice length L and the extension ratio α which result in comparably efficient and more effective in modeling capability to the linearcomplexity counterpart, considering the discussion made in this section on the effective number of tokens per abstraction.

Original: A. I MPROVED E XPRESSIVENESS OF CST S LICE A TTENTION
Revised: A. I MPROVED E XPRESSIVENESS OF CST S LICE A TTENTION

Original: Recall from Eq.equation 10 that the attention mechanism under consideration takes the form,
Revised: Recall from Eq.equation 10 that the attention mechanism under consideration takes the form,

Original: [Equation]
Revised: [Equation]

Original: where Y l represents local attention, Y g represents global attention and ψ ( · ) is a (linear) one-to-many map.
Revised: where Y l represents local attention, Y g represents global attention and ψ ( · ) is a (linear) one-to-many map.

Original: In this section, we will compare CST with the H-transformer described in Zhu & Soricut (2021).
Revised: In this section, we will compare CST with the H-transformer described in Zhu & Soricut (2021).

Original: In particular, we argue that, for a given number of weights (degrees of freedom), our proposed attention mechanism, which involves the composition of attention mechanisms, is more expressive than the H-matrix-based approach proposed Zhu & Soricut (2021).Zhu & Soricut (2021) shares similar characteristics with CST , but does not involve composition, which explains our superior prediction performance (Section 5).
Revised: In particular, we argue that, for a given number of weights (degrees of freedom), our proposed attention mechanism, which involves the composition of attention mechanisms, is more expressive than the H-matrix-based approach proposed Zhu & Soricut (2021).Zhu & Soricut (2021) shares similar characteristics with CST , but does not involve composition, which explains our superior prediction performance (Section 5).

Original: To explain why this is so, we first define
Revised: To explain why this is so, we first define

Original: [Equation]
Revised: [Equation]

Original: (26) which is the standard attention mechanism found in Eq.equation 2.
Revised: (26) which is the standard attention mechanism found in Eq.equation 2.

Original: Here, X is generally a matrix or a 3-tensor; in the latter case, we apply attention independently to each matrix slice along the third dimension.
Revised: Here, X is generally a matrix or a 3-tensor; in the latter case, we apply attention independently to each matrix slice along the [Table] [Figure] dimension.

Original: Expressing our local and global attention using this notation leads to,
Revised: Expressing our local and global attention using this notation leads to,

Original: [Equation]
Revised: [Equation]

Original:  where µ = (cid:2) 1 L , 1 L , ... , 1 L (cid:3) T is a fixed vector of mask values, S corresponds to the stack of L -length slices of X described in Section 4.1, and × 2 indicates a tensor product along the second dimension, i.e.,
Revised: [Equation] (28) where µ = (cid:2) 1 L , 1 L , ... , 1 L (cid:3) T is a fixed vector of mask values, S corresponds to the stack of L -length slices of X described in Section 4. [Figure] and × 2 indicates a tensor product along the second dimension, i.e.,

Original: [Equation]
Revised: [Equation]

Original: which corresponds to mean pooling within each slice.
Revised: which corresponds to mean pooling within each slice.

Original: For the purpose of comparison, we focus on a 2-level H-transformer where each element at level 1 has L children at level 0.
Revised: For the purpose of comparison, we focus on a 2-level H-transformer where each element at level 1 has L children at level 0.

Original: In this case, it can be shown that the H-tranformer approximation takes the form of Eq.
Revised: In this case, it can be shown that the H-tranformer approximation takes the form of Eq.

Original: equation 25 with,
Revised: equation 25 with,

Original: [Equation]
Revised: [Equation]

Original: Note that the main difference between H-Transformer these and CST lies in Eq.equation 31 and
Revised: Note that the main difference between H-Transformer these and CST lies in Eq.equation 31 and

Original: Eq.equation 28; in the former, the argument to the global attention is S × 2 µ , whereas it is Y CS l × 2 µ in the latter.
Revised: Eq.equation 28; in the former, the argument to the global attention is S × 2 µ , whereas it is Y CS l × 2 µ in the latter.

Original: To demonstrate our claim, let us first introduce some quantities of interest, starting with the families of parameterized functions that the aforementioned attention mechanisms encompass.
Revised: Refer to Eq. 29) in Zhu & Soricut 2021) for two levels, i.e., Y = Y (0) + P (0) ˜ Y (1)

Original: [-First, assume 
-]
Revised: [-First, assume 
-]In this case

Original: In practice, it is common to have equal weights at local and global scales, i.e., W l, · = W g, · .
Revised: P ( to one to- ψ ( · ) whereas ˜ Y level- A context this is analogous , S µ

Original: This is a special case of  the analysis presented here and changes the conclusion in no significant way.
Revised: To demonstrate our claim, let us first introduce some quantities of interest, starting with the families of parameterized functions that the aforementioned attention mechanisms encompass.

Original: Refer to without loss of generality that all the weights are limited to the interval [ − 1 , 1] .
Revised: First, assume without loss of generality that all the weights are limited to the interval [ − 1 , 1] .

Original: Then, any attention function learned during the training process must belong to the following sets,
Revised: Then, any attention function learned during the training process must belong to the following sets,

Original: [Equation]
Revised: [Equation]

Original: in the case of CST and,
Revised: in the case of CST and,

Original: [Equation]
Revised: [Equation]

Original: in the case of H-transformer.
Revised: in the case of H-transformer.

Original: We want to show that Y CS g has more expressive set than Y H g , which explains why CST exhibits better empirical prediction performance than its competitors.
Revised: We want to show that Y CS g has more expressive set than Y H g , which explains why CST exhibits better empirical prediction performance than its competitors.

Original: To this end, let us introduce the following quantities:
Revised: To this end, let us introduce the following quantities:

Original: Definition 1.
Revised: Definition 1.

Original: Let p ( x ) , q ( x ) be polynomials of Euclidean degree d from R M to R .
Revised: Let p ( x ) , q ( x ) be polynomials of Euclidean degree d from R M to R .

Original: We denote by
Revised: We denote by

Original: S M,d the set of rational functions with numerator and denominator of Euclidean degree at most d , i [Equation] e.,
Revised: S M,d the set of rational functions with numerator and denominator of Euclidean degree at most d , i.e.,

Original: [Equation]
Revised: [Equation]

Original: imbued with the metric,
Revised: imbued with the metric,

Original: [Equation]
Revised: [Equation]

Original: where || f ( x ) || ∞ =
Revised: where || f ( x ) || ∞ =

Original: max x ∈ [ − 1 , 1] M | f ( x ) | .
Revised: max x ∈ [ − 1 , 1] M | f ( x ) | .

Original: We also introduce S M, ∞ to indicate the space of ratios of real analytic functions, which is a vector space.
Revised: We also introduce S M, ∞ to indicate the space of ratios of real analytic functions, which is a vector space.

Original: Further, given any fixed positive ϵ ∈ R , we define,
Revised: Further, given any fixed positive ϵ ∈ R , we define,

Original: [Equation]
Revised: [Equation]

Original: as the ϵ -ball surrounding S M,d in the topology induced on S M, ∞ by the metric.
Revised: as the ϵ -ball surrounding S M,d in the topology induced on S M, ∞ by the metric.

Original: Clearly, for any fixed dimension M , ϵ > 0 and Euclidean degrees d ′ > d , the family of functions
Revised: Clearly, for any fixed dimension M , ϵ > 0 and Euclidean degrees d ′ > d , the family of functions

Original: S ϵM,d ′ is more expressive than S ϵM,d , since the latter is a proper subset of the former, i.e., S M,d ⊂
Revised: S ϵM,d ′ is more expressive than S ϵM,d , since the latter is a proper subset of the former, i.e., S M,d ⊂

Original: S M,d ′ .
Revised: S M,d ′ .

Original: Next, we show that the families of functions Y CS g and Y H g are in fact subsets of S ϵM,d ′ and S ϵM,d respectively, for appropriately-chosen values of ϵ , d ′ and d with d ′ > d .
Revised: Next, we show that the families of functions Y CS g and Y H g are in fact subsets of S ϵM,d ′ and S ϵM,d respectively, for appropriately-chosen values of ϵ , d ′ and d with d ′ > d .

Original: To do so, we need the following result adapted from Trefethen (2017),
Revised: To do so, we need the following result adapted from Trefethen (2017),

Original: Theorem 1.
Revised: Theorem 1.

Original: Let f ( x ) be an analytic function from R M to R .
Revised: Let f ( x ) be an analytic function from R M to R .

Original: Then for every fixed ϵ > 0 , there exists a polynomial p ( x ) of Euclidean degree d ( ϵ )
Revised: Then for every fixed ϵ > 0 , there exists a polynomial p ( x ) of Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that,
Revised: O (log( ϵ )) such that,

Original: [Equation]
Revised: [Equation]

Original: We are now ready to demonstrate our main result,
Revised: We are now ready to demonstrate our main result,

Original: Proposition 2.
Revised: Proposition 2.

Original: For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )
Revised: For any fixed ϵ > 0 , there exists some Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that,
Revised: O (log( ϵ )) such that,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: where M corresponds to the total number of weights.
Revised: where M corresponds to the total number of weights.

Original: Proof.
Revised: Proof.

Original: First, recall that (assuming a single head) the parameterized functions γ ( · ) from Eq.
Revised: First, recall that (assuming a single head) the parameterized functions γ ( · ) from Eq.

Original: equation 26 takes the following form
Revised: equation 26 takes the following form

Original: [Equation]
Revised: [Equation]

Original: This should be recognized as the ratio of two analytic functions (because the exponential is analytic) paramatrized by the elements of the weight matrices.
Revised: This should be recognized as the ratio of two analytic functions (because the exponential is analytic) paramatrized by the elements of the weight matrices.

Original: Following Theorem 1, for every fixed set of weights W g =
Revised: Following Theorem 1, for every fixed set of weights W g =

Original: [ W q , W k , W v ] , there exists polynomials { p W g ij ( x ) } , { q W g ij ( x ) } of Euclidean degree d ( ϵ )
Revised: [ W q , W k , W v ] , there exists polynomials { p W g ij ( x ) } , { q W g ij ( x ) } of Euclidean degree d ( ϵ )

Original: =
Revised: =

Original: O (log( ϵ )) such that
Revised: O (log( ϵ )) such that

Original: [Equation]
Revised: [Equation]

Original: This means that
Revised: This means that

Original: [Equation]
Revised: [Equation]

Original: and that γ ij ( · ; W q [Equation] k , 
Revised: and that γ ij ( · ; W q , W k , W v ) ∈ S ϵM,d

Original: [Equation]
Revised: [Equation]

Original: ∈ [Equation] ϵ ) . In particular, this shows that Y H g ⊂ S ϵM g ,d ( ϵ ) since every element of Y H g has the form of Eq.equation 41.
Revised: In particular, this shows that Y H g ⊂ S ϵM g ,d ( ϵ ) since every element of Y H g has the form of Eq.equation 41.

Original: A similar conclusion can be reached for Y H l and
Revised: A similar conclusion can be reached for Y H l and

Original: Y CS l since elements of these families have the same functional form as those of Y H g .
Revised: Y CS l since elements of these families have the same functional form as those of Y H g .

Original: All that remains is Y CS g .
Revised: All that remains is Y CS g .

Original: In this case, it suffices to note that the result of polynomial composition creates polynomials with Euclidean degree bounded by the sum of the degree of the polynomials involved in the composition.
Revised: In this case, it suffices to note that the result of polynomial composition creates polynomials with Euclidean degree bounded by the sum of the degree of the polynomials involved in the composition.

Original: Indeed, the above analysis shows that elements of Y CS g are within ϵ distance from rational functions of the form,
Revised: Indeed, the above analysis shows that elements of Y CS g are within ϵ distance from rational functions of the form,

Original: [Equation]
Revised: [Equation]

Original: for some polynomials { p W l ijk ( x ) } , { q W l ijk ( x ) } of degree d ( ϵ ) .
Revised: for some polynomials { p W l ijk ( x ) } , { q W l ijk ( x ) } of degree d ( ϵ ) .

Original: Upon expanding the polynomials and expressing terms using a common denominator, this expression should be recognized as the ratio of two polynomials, each of which has degree 4 d ( ϵ ) , following polynomial composition.
Revised: Upon expanding the polynomials and expressing terms using a common denominator, this expression should be recognized as the ratio of two polynomials, each of which has degree 4 d ( ϵ ) , following polynomial composition.

Original: Thus, we conclude that Y CSg ⊂ S ϵM, 4 d ( ϵ ) .
Revised: Thus, we conclude that Y CSg ⊂ S ϵM, 4 d ( ϵ ) .

Original: This proves our claim.
Revised: This proves our claim.

Original: In other words, Proposition 2 shows that our proposed family of attention mechanisms belongs to S ϵM, 4 d ( ϵ ) which possesses more expressive power (since 4 d ( ϵ ) >
Revised: In other words, Proposition 2 shows that our proposed family of attention mechanisms belongs to S ϵM, 4 d ( ϵ ) which possesses more expressive power (since 4 d ( ϵ ) >

Original: d ( ϵ ) ) than the family of functions S ϵM,d ( ϵ ) to which global attention mechanism without composition, such as H-transfomers ( Y H l + ψ (cid:0) Y H g (cid:1) ), belong.
Revised: d ( ϵ ) ) than the family of functions S ϵM,d ( ϵ ) to which global attention mechanism without composition, such as H-transfomers ( Y H l + ψ (cid:0) Y H g (cid:1) ), belong.

Original: We claim that this is the reason why our proposed approach performs better than the approach proposed by (Zhu & Soricut, 2021) and other similar mechanisms where linear combinations of attentions, rather than composition, are used.
Revised: We claim that this is the reason why our proposed approach performs better than the approach proposed by (Zhu & Soricut, 2021) and other similar mechanisms where linear combinations of attentions, rather than composition, are used.

Original: In other words, for the same number of parameters (weights), our proposed approach can capture more complex attentive interactions than that of (Zhu & Soricut, 2021) and related mechanisms, which leads to a richer set of attentions over which to train the network and, ultimately, better performance.
Revised: In other words, for the same number of parameters (weights), our proposed approach can capture more complex attentive interactions than that of (Zhu & Soricut, 2021) and related mechanisms, which leads to a richer set of attentions over which to train the network and, ultimately, better performance.

Original: A. M ORE I NFORMATION AND D ETAILS ON LRA B ENCHMARK A.6. E XPERIMENTAL S ETUP
Revised: A. M ORE I NFORMATION AND D ETAILS ON LRA B ENCHMARK A.6. E XPERIMENTAL S ETUP

Original: We follow the experimental setup described in (Xiong et al., 2021) with a few exceptions in hyperparameters.
Revised: We follow the experimental setup described in (Xiong et al., 2021) with a few exceptions in hyperparameters.

Original: Specifically, we use the same Transformer encoder network backbone that consists of
Revised: Specifically, we use the same Transformer encoder network backbone that consists of

Original: [Table]
Revised: [Table]

Original: transformer layers with pre-norm (Xiong et al., 2020) with 4 attention heads and the model dimension 256 across all tasks.
Revised: transformer layers with pre-norm (Xiong et al., 2020) with 4 attention heads and the model dimension 256 across all tasks.

Original: Further, we replace the self-attention sub-layer by efficient attention alternatives including the slice attention of CST .
Revised: Further, we replace the self-attention sub-layer by efficient attention alternatives including the slice attention of CST .

Original: To obtain label output for classification, we additionally use a classification head network with 2 fully-connected layers of the same hidden dimension with the feed-forward network in the backbone network and a ReLU layer between them.
Revised: To obtain label output for classification, we additionally use a classification head network with 2 fully-connected layers of the same hidden dimension with the feed-forward network in the backbone network and a ReLU layer between them.

Original: We aggregate the output sequence from the encoder by a mean pooling across the length dimension and feed it to the classification head.
Revised: We aggregate the output sequence from the encoder by a mean pooling across the length dimension and feed it to the classification head.

Original: For a fair comparison between models with different characteristics, we optimize the learning rates for the baseline Transformer model on each task while keeping all other hyperparameters fixed.
Revised: For a fair comparison between models with different characteristics, we optimize the learning rates for the baseline Transformer model on each task while keeping all other hyperparameters fixed.

Original: For the experimental setup, this allows for a strong baseline model, advantageous for the standard Transformer.
Revised: For the experimental setup, this allows for a strong baseline model, advantageous for the standard Transformer.

Original: We then evaluate each model with a learning rate search within a small bracket [0 .
Revised: We then evaluate each model with a learning rate search within a small bracket [0 .

Original: 5 l 0 , l 0 , 2 l 0 ] , where l 0 is the base learning rate providing the highest validation accuracy of the Transformer in each task, and report the test accuracy of the model trained with a learning rate that gives the highest validation accuracy among the bracket.
Revised: 5 l 0 , l 0 , 2 l 0 ] , where l 0 is the base learning rate providing the highest validation accuracy of the Transformer in each task, and report the test accuracy of the model trained with a learning rate that gives the highest validation accuracy among the bracket.

Original: We note that our experiments have stronger baselines than those found in literature (Tay et al., 2020b; Xiong et al., 2021) as highlighted in Table 1.
Revised: We note that our experiments have stronger baselines than those found in literature (Tay et al., 2020b; Xiong et al., 2021) as highlighted in Table 1.

Original: For other efficient Transformer models, we perform a model-specific hyperparameter search as described below and report the test accuracy of the best models.
Revised: For other efficient Transformer models, we perform a model-specific hyperparameter search as described below and report the test accuracy of the best models.

Original: For the Long Range Arena benchmark, we performed experiments on our PyTorch version implementation of the original open-source JAX/Flax code (Tay et al., 2020b) where we followed its dataset preparation procedure and verified the results, and implemented our model on it.
Revised: For the Long Range Arena benchmark, we performed experiments on our PyTorch version implementation of the original open-source JAX/Flax code (Tay et al., 2020b) where we followed its dataset preparation procedure and verified the results, and implemented our model on it.

Original: Since we found the performances of LRA tasks have relatively high variances, especially on certain tasks such as ListOps , we first performed a hyperparameter search with the standard Transformer with the base setup in (Xiong et al., 2021).
Revised: Since we found the performances of LRA tasks have relatively high variances, especially on certain tasks such as ListOps , we first performed a hyperparameter search with the standard Transformer with the base setup in (Xiong et al., 2021).

Original: The hyperparameters we used for the search include number of layers { 2, 4 } , number of attention heads { 2, 4 } , and model dimension { 128, 256 } , and learning rate { 1 , 2 , 5 } × 10 {− 3 , − 4 , − 5 , − 6 } .
Revised: The hyperparameters we used for the search include number of layers { 2, 4 } , number of attention heads { 2, 4 } , and model dimension { 128, 256 } , and learning rate { 1 , 2 , 5 } × 10 {− 3 , − 4 , − 5 , − 6 } .

Original: In addition, for better convergence, we also increased the number of training steps in some tasks.
Revised: In addition, for better convergence, we also increased the number of training steps in some tasks.

Original: The hyperparameters we determined after the search can be found in
Revised: The hyperparameters we determined after the search can be found in

Original: Table 6.
Revised: Table 6.

Original: As this step is essentially a hyperparameter tuning for the baseline Transformer, we note that we have stronger baseline Transformer model than those reported in (Tay et al., 2020b) and (Xiong et al., 2021), and compare them in Table 1.
Revised: As this step is essentially a hyperparameter tuning for the baseline Transformer, we note that we have stronger baseline Transformer model than those reported in (Tay et al., 2020b) and (Xiong et al., 2021), and compare them in Table 1.

Original: While fixing all other hyperparameters, we performed a learning rate sweep using a learning rate bracket as described in 5.1.
Revised: While fixing all other hyperparameters, we performed a learning rate sweep using a learning rate bracket as described in 5.1.

Original: For each model and task, we pick the best model on the validation dataset with early stopping, and report its accuracy on the test dataset.
Revised: For each model and task, we pick the best model on the validation dataset with early stopping, and report its accuracy on the test dataset.

Original: For the model training, we used AdamW optimizer (Loshchilov & Hutter, 2017) with linear learning rate warmup and decay, but without weight decay and other training hyperparameters as shown in Table 6.
Revised: For the model training, we used AdamW optimizer (Loshchilov & Hutter, 2017) with linear learning rate warmup and decay, but without weight decay and other training hyperparameters as shown in Table 6.

Original: Since each efficient Transformer has its own hyperparameters and they can affect the performance, we tried a few different values, around the default values used in each paper.
Revised: Since each efficient Transformer has its own hyperparameters and they can affect the performance, we tried a few different values, around the default values used in each paper.

Original: Specifically, they are: the number of random features { 64, 128, 256 } in Performer (Choromanski et al., 2020), the projection length { 64, 128, 256 } in Linformer (Wang et al., 2020) and Luna (Ma et al., 2021), the number of landmarks { 64, 128, 256 } in Nystr¨omformer (Xiong et al., 2021), the number of hashes { 2, 4 } in Reformer (Kitaev et al., 2019), the numerical rank of off-diagonal blocks { 16 ,32, 64, 128 } in H-Transformer-1D (Zhu & Soricut, 2021), and the local window segment size { 8, 16, 32 } and
Revised: Specifically, they are: the number of random features { 64, 128, 256 } in Performer (Choromanski et al., 2020), the projection length { 64, 128, 256 } in Linformer (Wang et al., 2020) and Luna (Ma et al., 2021), the number of landmarks { 64, 128, 256 } in Nystr¨omformer (Xiong et al., 2021), the number of hashes { 2, 4 } in Reformer (Kitaev et al., 2019), the numerical rank of off-diagonal blocks { 16 ,32, 64, 128 } in H-Transformer-1D (Zhu & Soricut, 2021), and the local window segment size { 8, 16, 32 } and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: the rank of dynamic projection { 32, 64 } in Long Short Transformer (Zhu et al., 2021).
Revised: the rank of dynamic projection { 32, 64 } in Long Short Transformer (Zhu et al., 2021).

Original: For Long
Revised: For Long

Original: Short Transformer, we optionally used additional 1D depth-wise convolution with the kernel 35 as described in (Zhu et al., 2021).
Revised: Short Transformer, we optionally used additional 1D depth-wise convolution with the kernel 35 as described in (Zhu et al., 2021).

Original: For scatterbrain (Chen et al., 2021a), we follow the default setting for the model specific hyperparameters as in the official code release.
Revised: For scatterbrain (Chen et al., 2021a), we follow the default setting for the model specific hyperparameters as in the official code release.

Original: Specifically, for text task, we have cluster size for query/key = 16, number of features = 16.
Revised: Specifically, for text task, we have cluster size for query/key = 16, number of features = 16.

Original: For Listops task, we have cluster size for query/key = 32 and number of features = 32.
Revised: For Listops task, we have cluster size for query/key = 32 and number of features = 32.

Original: For image and pathfinder task, we have cluster size = 16 and number of features = 16.
Revised: For image and pathfinder task, we have cluster size = 16 and number of features = 16.

Original: For retrieval task, we have cluster size for query/key =and number of features = 64.
Revised: For retrieval task, we have cluster size for query/key =and number of features = 64.

Original: Additionally, we have number of hashes =
Revised: Additionally, we have number of hashes =

Original: 2 for all tasks.
Revised: 2 for all tasks.

Original: While each option produces different performances requiring different computation and memory costs, we found variants in each model have similar accuracy and report only the best accuracy among them and the corresponding complexity as comparing them is out of scope of this paper.
Revised: While each option produces different performances requiring different computation and memory costs, we found variants in each model have similar accuracy and report only the best accuracy among them and the corresponding complexity as comparing them is out of scope of this paper.

Original: A.6. A BLATION S TUDY ON LRA
Revised: A.6. A BLATION S TUDY ON LRA

Original: We investigate the effect of each component in CST by performing an ablation study.
Revised: We investigate the effect of each component in CST by performing an ablation study.

Original: We set the slice length of 16 without extension for local attention as the base configuration.
Revised: We set the slice length of 16 without extension for local attention as the base configuration.

Original: Then, we train a CST variant from scratch where one component is changed.
Revised: Then, we train a CST variant from scratch where one component is changed.

Original: Specifically, we consider the variations in: 1)
Revised: Specifically, we consider the variations in: 1)

Original: Slice length, 2) Extended local attention, 3)
Revised: Slice length, 2) Extended local attention, 3)

Original: Positional embedding, and 4) Composition of MSMRA.
Revised: Positional embedding, and 4) Composition of MSMRA.

Original: The test accuracies of the variants are given in Table 7.
Revised: The test accuracies of the variants are given in Table 7.

Original: We note that all variants of CST consistently outperforms other models, except for one case of the global first composition.
Revised: We note that all variants of CST consistently outperforms other models, except for one case of the global first composition.

Original: Slice Length First, we vary the slice length L and test how the local attention range and abstraction length for the global attention affects the accuracy.
Revised: Slice Length First, we vary the slice length L and test how the local attention range and abstraction length for the global attention affects the accuracy.

Original: Intuitively, one can expect a trade-off between fine-grained token interaction modeling and coarse-grained global context capturing, since in the extreme cases either local attention or global attention converges to the standard self-attention 7 .
Revised: Intuitively, one can expect a trade-off between fine-grained token interaction modeling and coarse-grained global context capturing, since in the extreme cases either local attention or global attention converges to the standard self-attention 7 .

Original: While each task has its own best L , shorter slice length tends to give better overall accuracy.
Revised: While each task has its own best L , shorter slice length tends to give better overall accuracy.

Original: One possible

Original: CSA converges to the standard self-attention with the global attention at L or the local attention

Original: L interpretation is that L = 8 or 16 provides better-formed tokens to the global attention in a dynamic manner by the local attention.
Revised: One possible interpretation is that L = 8 or 16 provides better-formed tokens to the global attention in a dynamic manner by the local attention.

Original: Local Attention Extension Ratio While slight improvements are observed on Retrieval and Image tasks, on the contrary to our expectation, extended local attention does not always improve the accuracy on LRA tasks.
Revised: Local Attention Extension Ratio While slight improvements are observed on Retrieval and Image tasks, on the contrary to our expectation, extended local attention does not always improve the accuracy on LRA tasks.

Original: We conjecture it is because LRA datasets has tokens of very fine granularity, i.e., byte-level or pixel-level, and abstraction of the extended local attention output may result in redundant information for global attention.
Revised: We conjecture it is because LRA datasets has tokens of very fine granularity, i.e., byte-level or pixel-level, and abstraction of the extended local attention output may result in redundant information for global attention.

Original: Thus, we evaluate the effect of the extension ratio on a dataset with a coarser-grained tokenization in the Section 5.2.
Revised: Thus, we evaluate the effect of the extension ratio on a dataset with a coarser-grained tokenization in the Section 5.2.

Original: Position Embedding We compare the MS-VIPE with the conventional learned positional embedding that is applied at the bottom embedding layer of the network.
Revised: Position Embedding We compare the MS-VIPE with the conventional learned positional embedding that is applied at the bottom embedding layer of the network.

Original: MS-VIPE shows consistent improvement over the conventional positional embedding except for the ListOps task where the fined-grained order of each token is essential in this particular task.
Revised: MS-VIPE shows consistent improvement over the conventional positional embedding except for the ListOps task where the fined-grained order of each token is essential in this particular task.

Original: Combination of MSMRA Instead of the proposed composition of MSMRA that has a serial connection of MSMRA in the local-global order, we try two different connections, i.e., parallel and global-local connection.
Revised: Combination of MSMRA Instead of the proposed composition of MSMRA that has a serial connection of MSMRA in the local-global order, we try two different connections, i.e., parallel and global-local connection.

Original: Both of them degrade the accuracy compared to the local-global composition, supporting effectiveness of our design.
Revised: Both of them degrade the accuracy compared to the local-global composition, supporting effectiveness of our design.

Original: Local Modeling with MLP As shown in (Tolstikhin et al., 2021) and (Liu et al., 2021a), a multilayer perceptron (MLP) can effectively replace the attention without loss of accuracy in some applications.
Revised: Local Modeling with MLP As shown in (Tolstikhin et al., 2021) and (Liu et al., 2021a), a multilayer perceptron (MLP) can effectively replace the attention without loss of accuracy in some applications.

Original: In CST , since the slice length is fixed and relatively short, use of MLP instead of attention can be considered as another source of efficiency with a small increase of parameters.
Revised: In CST , since the slice length is fixed and relatively short, use of MLP instead of attention can be considered as another source of efficiency with a small increase of parameters.

Original: We evaluate a variant of CST with 2-layer MLP replacing the local attention, with 2 different hidden dimensions d hidden = 16 or 32 while L = 16 and α = 1 .
Revised: We evaluate a variant of CST with 2-layer MLP replacing the local attention, with 2 different hidden dimensions d hidden = 16 or 32 while L = 16 and α = 1 .

Original: MLP in each slice is equivalent to 1D convolution with the kernel size and the stride set to the same with the slice length.
Revised: MLP in each slice is equivalent to 1D convolution with the kernel size and the stride set to the same with the slice length.

Original: We observe that using attention for local modeling module is still preferred in terms of accuracy.
Revised: We observe that using attention for local modeling module is still preferred in terms of accuracy.

Original: Slice aggregation We further evaluate use of different aggregation methods from the simple mean pooling.
Revised: Slice aggregation We further evaluate use of different aggregation methods from the simple mean pooling.

Original: Similarly to (Rae et al., 2019), we try max pooling and linear projection.
Revised: Similarly to (Rae et al., 2019), we try max pooling and linear projection.

Original: Again, a linear projection is equivalent to 1D convolution with the kernel size and stride set to L .
Revised: Again, a linear projection is equivalent to 1D convolution with the kernel size and stride set to L .

Original: While showing comparable results, they lead to degraded performances in Image and Pathfinder tasks.
Revised: While showing comparable results, they lead to degraded performances in Image and Pathfinder tasks.

Original: One can expect better accuracy from the linear projection as it is a general form of other pooling methods, but this result can be interpreted as limitation of a simple linear projection for modeling sequence translations.
Revised: One can expect better accuracy from the linear projection as it is a general form of other pooling methods, but this result can be interpreted as limitation of a simple linear projection for modeling sequence translations.

Original: The mean pooling has already generalized well, and with better initialization, linear projection is expected to converge to the mean pooling or slightly better.
Revised: The mean pooling has already generalized well, and with better initialization, linear projection is expected to converge to the mean pooling or slightly better.

Original: A. M ORE I NFORMATION AND D ETAILS ON A UTOREGRESSIVE L ANGUAGE M ODELING A.7. E XPERIMENTAL D ETAIL OF A UTOREGRESSIVE L ANGUAGE M ODELING
Revised: A. M ORE I NFORMATION AND D ETAILS ON A UTOREGRESSIVE L ANGUAGE M ODELING A.7. E XPERIMENTAL D ETAIL OF A UTOREGRESSIVE L ANGUAGE M ODELING

Original: For the autoregressive language modeling experiment, we use an open-source language modeling experiment framework (Schlag et al., 2021) and plug in our CST implementation.
Revised: For the autoregressive language modeling experiment, we use an open-source language modeling experiment framework (Schlag et al., 2021) and plug in our CST implementation.

Original: We follow all training setup including dataset preparation and hyperparameters given in (Dai et al., 2019a), and evaluate with the small-sized network configuration: 16 layers, 8 attention heads, 128 model dimension, and 2,048 hidden dimension.
Revised: We follow all training setup including dataset preparation and hyperparameters given in (Dai et al., 2019a), and evaluate with the small-sized network configuration: 16 layers, 8 attention heads, 128 model dimension, and 2,048 hidden dimension.

Original: We train all models using the Adam optimizer (Kingma & Ba, 2014), the cosine annealing learning rate scheduler with 2000 warmup steps while the base learning rate is 2 .
Revised: We train all models using the Adam optimizer (Kingma & Ba, 2014), the cosine annealing learning rate scheduler with 2000 warmup steps while the base learning rate is 2 .

Original: 5 × 10 − 4 for 500K steps with the batch size of 96.
Revised: 5 × 10 − 4 for 500K steps with the batch size of 96.

Original: We also conduct autoregressive language modeling experiments with 1024 sequence length.
Revised: We also conduct autoregressive language modeling experiments with 1024 sequence length.

Original: Here, we roughly follow the experiment setup used in (Chen et al., 2021a).
Revised: Here, we roughly follow the experiment setup used in (Chen et al., 2021a).

Original: The larger model has 512 model dimension instead of 128, and we change the learning rate to 5 × 10 − 4 , number of steps to 90K, and batch size to 32, while keeping all the remaining hyperparameters same as that of the small model.
Revised: The larger model has 512 model dimension instead of 128, and we change the learning rate to 5 × 10 − 4 , number of steps to 90K, and batch size to 32, while keeping all the remaining hyperparameters same as that of the small model.

Original: Different choices of slice length L and extension ratio α results in the same local attention range.
Revised: Different choices of slice length L and extension ratio α results in the same local attention range.

Original: For instance, the combination of L = 8 , α = 3 has the same key/value lengths with those in the
Revised: For instance, the combination of L = 8 , α = 3 has the same key/value lengths with those in the

Original: L = 16 , α = 1 setting, i.e., L × (3 − 1) / 2 = L .
Revised: L = 16 , α = 1 setting, i.e., L × (3 − 1) / 2 = L .

Original: However, since the abstraction length and the resulting granularity of the global attention only depends on L , the choices of these hyperparameters can affect
Revised: However, since the abstraction length and the resulting granularity of the global attention only depends on L , the choices of these hyperparameters can affect

Original: [Table]
Revised: [Table]

Original: the performance differently.
Revised: the performance differently.

Original: We examine how the performance is affected by this configuration.
Revised: We examine how the performance is affected by this configuration.

Original: To this end
Revised: To this end

Original: [Table]
Revised: [Table]

Original: we evaluate CST with combinations of a set of slice lengths { 8 , 16 , 32 } and a set of extension ratios α { 1 , 2 , 3 } .
Revised: we evaluate CST with combinations of a set of slice lengths { 8 , 16 , 32 } and a set of extension ratios α { 1 , 2 , 3 } [Table]

Original: To ensure that the current query token has no interaction with the future key tokens
Revised: To ensure that the current query token has no interaction with the future key tokens

Original: [Table]
Revised: [Table]

Original: we design the causal mask and local attention extension scheme to better suit CSA as described in Section 4.
Revised: we design the causal mask and local attention extension scheme to better suit CSA as described in Section 4.

Original: [Table]
Revised: [Table]

Original: and use it throughout this experiment.
Revised: and use it throughout this experiment.

Original: A.7. A UTOREGRESSIVE L ANGUAGE M ODELING WITH M ATCHED A TTENTION C OMPLEXITY TO T RANSFORMERS ON PG- 
Revised: A.7. A UTOREGRESSIVE L ANGUAGE M ODELING WITH M ATCHED A TTENTION C OMPLEXITY TO T RANSFORMERS ON PG-

Original: We additionally perform evaluation of CST on PG-19 dataset (Rae et al., 2019) compare to Transformers where their attention complexities are matched, i.e., O ( N 2 /N 2 +0 .
Revised: We additionally perform evaluation of CST on PG-19 dataset (Rae et al., 2019) compare to Transformers where their attention complexities are matched, i.e., O ( N 2 /N 2 +0 .

Original: ( α +1) NL ) ≃ O ( N 2 ) .
Revised: ( α +1) NL ) ≃ O ( N 2 ) .

Original: We use GPT-2 (Radford et al., 2019) implementation in Huggingface transformers library (Wolf et al., 2020), and use the same architecture to implement CST equipped with CSA and MS-VIPE.
Revised: We use GPT-2 (Radford et al., 2019) implementation in Huggingface transformers library (Wolf et al., 2020), and use the same architecture to implement CST equipped with CSA and MS-VIPE.

Original: We train all models for 400 k training steps using the batch size 32 .
Revised: We train all models for 1 M training steps using the batch size 32 .

Original: We use AdamW optimizer with the learning rate = 1 e − 3 , β = [0 . 9 , 0 . 99] , and the weight decay = 0 .
Revised: We use AdamW optimizer with the learning rate = 1 e − 3 , β = [0 . 9 , 0 . 99] , and the weight decay = 0 .

Original: We report the validation and test perplexities in Table 9.
Revised: We report the validation and test perplexities in Table 9.

Original: Efficient attention in CST enables longer context length with the same complexity in attention computation, leading to significantly improved perplexity compared to the Transformer baseline.
Revised: Efficient attention in CST enables longer context length with the same complexity in attention computation, leading to significantly improved perplexity compared to the Transformer baseline.

Original: Note that, with the current architecture in these cases, the overall complexity can be higher in CST since there still are complexity increases in linear layers when they are processing longer inputs.
Revised: Note that, with the current architecture in these cases, the overall complexity can be higher in CST since there still are complexity increases in linear layers when they are processing longer inputs.

Original: We leave application of caching techniques as future work, that saves complexity in linear layers to match the overall network complexity while enjoying the long-range attention.
Revised: We leave application of caching techniques as future work, that saves complexity in linear layers to match the overall network complexity while enjoying the long-range attention.

Original: A. M ORE I NFORMATION AND D ETAILS ON B IDIRECTIONAL M ASKED L ANGUAGE M ODELING AND GLUE B ENCHMARK
Revised: A. M ORE I NFORMATION AND D ETAILS ON B IDIRECTIONAL M ASKED L ANGUAGE M ODELING AND GLUE B ENCHMARK

Original: In this section, we present more details about the experimental setup and results on the bidirectional masked language modeling (MLM) task Devlin et al.
Revised: In this section, we present more details about the experimental setup and results on the bidirectional masked language modeling (MLM) task Devlin et al.

Original: and the natural language understanding tasks in the GLUE benchmark Wang et al.
Revised: and the natural language understanding tasks in the GLUE benchmark Wang et al.

Original: (2018) with discussion.
Revised: (2018) with discussion.

Original: A.8. E XPERIMENTAL S ETUP
Revised: A.8. E XPERIMENTAL S ETUP

Original: We pre-train transformer models with the masked language modeling (MLM) objective on BookCorpus Zhu et al.
Revised: We pre-train transformer models with the masked language modeling (MLM) objective on BookCorpus Zhu et al.

Original: and English Wikipedia.
Revised: and English Wikipedia.

Original: After pre-training, the models are fine-tuned on downstream natural language understanding tasks from GLUE benchmark Wang et al.
Revised: After pre-training, the models are fine-tuned on downstream natural language understanding tasks from GLUE benchmark Wang et al.

Original: We follow the experimental setup of Devlin et al.
Revised: We follow the experimental setup of Devlin et al.

Original: (2018) for conducting both pre-training and fine-tuning experiment, including datasets, masking rate, batch size, and optimizer settings, with several exceptions.
Revised: (2018) for conducting both pre-training and fine-tuning experiment, including datasets, masking rate, batch size, and optimizer settings, with several exceptions.

Original: We pre-train the models for 900k steps with sequences of 128 tokens.
Revised: We pre-train the models for 900k steps with sequences of 128 tokens.

Original: This corresponds to the first phase of BERT pre-training and we consider it to be enough for evaluating on GLUE tasks while expediting the experiment.
Revised: This corresponds to the first phase of BERT pre-training and we consider it to be enough for evaluating on GLUE tasks while expediting the experiment.

Original: With the baseline network architecture of BERT base Devlin et al.
Revised: With the baseline network architecture of BERT base Devlin et al.

Original: (2018), we replace its self-attention layer with different types of efficient attentions such as Nystr¨omformer, Performer, Luna, and FNet.
Revised: (2018), we replace its self-attention layer with different types of efficient attentions such as Nystr¨omformer, Performer, Luna, and FNet.

Original: For downstream NLU tasks, we also follow the setup of Devlin et al.
Revised: For downstream NLU tasks, we also follow the setup of Devlin et al.

Original: (2018) with one minor change: instead of prepending [CLS] token in the input sequence, we use mean pooling to get the classifier input vector from the sequence output, as discussed in the following part of this section.
Revised: (2018) with one minor change: instead of prepending [CLS] token in the input sequence, we use mean pooling to get the classifier input vector from the sequence output, as discussed in the following part of this section.

Original: A.8. R ESULTS AND D ISCUSSION
Revised: A.8. R ESULTS AND D ISCUSSION

Original: As shown in Table 5, we can see that CST outperforms competing efficient transformers in terms of
Revised: As shown in Table 5, we can see that CST outperforms competing efficient transformers in terms of

Original: MLM perplexity.
Revised: MLM perplexity.

Original: It can be seen that, while CST s achieve consistently lower perplexity than most of the efficient transformers, its perplexity decreases as slice length and extension ratio increases.
Revised: It can be seen that, while CST s achieve consistently lower perplexity than most of the efficient transformers, its perplexity decreases as slice length and extension ratio increases.

Original: However, CST does not outperform BERT base on contrary to our results on the LRA benchmark.
Revised: However, CST does not outperform BERT base on contrary to our results on the LRA benchmark.

Original: We conjecture this is due to fine-grained masking strategy in the original MLM objective.
Revised: We conjecture this is due to fine-grained masking strategy in the original MLM objective.

Original: The MLM pre-training stage involves replacing random tokens in the input sequence with a dummy token that the model must predict.
Revised: The MLM pre-training stage involves replacing random tokens in the input sequence with a dummy token that the model must predict.

Original: After applying the positional embedding, replacing a standard token with a dummy token, i.e., [MASK], translates to injecting high-frequency noise into the input sequence.
Revised: After applying the positional embedding, replacing a standard token with a dummy token, i.e., [MASK], translates to injecting high-frequency noise into the input sequence.

Original: CST and other efficient attention alternatives are approximations to the full attention, which must truncate the high-frequency modes in exchange for faster evaluation.
Revised: CST and other efficient attention alternatives are approximations to the full attention, which must truncate the high-frequency modes in exchange for faster evaluation.

Original: In MLM pre-training, all of the input sequences have high-frequency content that attention approximations have difficulty capturing.
Revised: In MLM pre-training, all of the input sequences have high-frequency content that attention approximations have difficulty capturing.

Original: Meanwhile, the sequences in the LRA benchmark are long, low-frequency sequences which can be efficiently compressed.
Revised: Meanwhile, the sequences in the LRA benchmark are long, low-frequency sequences which can be efficiently compressed.

Original: An alternative pre-training method could eventually be developed specifically for efficient transformers that circumvents this issue.
Revised: An alternative pre-training method could eventually be developed specifically for efficient transformers that circumvents this issue.

Original: On downstream NLU classification tasks, CST also outperforms among the efficient transformers.
Revised: On downstream NLU classification tasks, CST also outperforms among the efficient transformers.

Original: But with under-performing pre-trained models compared to BERT, similar degradation trend is observed in evaluation on downstream tasks after fine-tuning them.
Revised: But with under-performing pre-trained models compared to BERT, similar degradation trend is observed in evaluation on downstream tasks after fine-tuning them.

Original: One drawback of the current version of CST in the original setting is that the use of [CLS] token is not trivial.
Revised: One drawback of the current version of CST in the original setting is that the use of [CLS] token is not trivial.

Original: [CLS] token can be seen as a global token that summarizes the sequence in the context of the given classification task, and it is often prepended in the input sequence to summarize information from the sequence for the classification purpose.
Revised: [CLS] token can be seen as a global token that summarizes the sequence in the context of the given classification task, and it is often prepended in the input sequence to summarize information from the sequence for the classification purpose.

Original: In CST , naively prepending it limits attention range in local attention part and also make the summarization in the global attention highly implicit.
Revised: In CST , naively prepending it limits attention range in local attention part and also make the summarization in the global attention highly implicit.

Original: We believe this can be addressed by an additional well-designed architecture that uses global tokens and input them to classifier.
Revised: We believe this can be addressed by an additional well-designed architecture that uses global tokens and input them to classifier.

Original: We leave this as a future work and we instead use mean pooling for the experiment for all models.
Revised: We leave this as a future work and we instead use mean pooling for the experiment for all models.

Original: While the efficiency gain is reduced due to the short sequence length, one can still expect reduction in total FLOPS from using CST .
Revised: While the efficiency gain is reduced due to the short sequence length, one can still expect reduction in total FLOPS from using CST .

Original: For instance, in the simple case of N = 128 , L = 64 and α = 1 , the complexity of CST is O (128 · 64+128 2 / 64 2 ) which is almost half of that of the standard transformer
Revised: For instance, in the simple case of N = 128 , L = 64 and α = 1 , the complexity of CST is O (128 · 64+128 2 / 64 2 ) which is almost half of that of the standard transformer

Original: O (128 2 ) .
Revised: O (128 2 ) .

Original: A. L IMITATIONS AND F UTURE W ORK A.9. L IMITATIONS
Revised: A. L IMITATIONS AND F UTURE W ORK A.9. L IMITATIONS

Original: We summarize limitations of CST discussed in the previous sections.
Revised: We summarize limitations of CST discussed in the previous sections.

Original: As discussed in A.8, there is a mismatch of using token-wise random masking for MLM pretraining.
Revised: As discussed in A.8, there is a mismatch of using token-wise random masking for MLM pretraining.

Original: In the abstraction of sequence slices for global attention, high-frequency modes are truncated, i.e., smoothed out, that leads to loss of information.
Revised: In the abstraction of sequence slices for global attention, high-frequency modes are truncated, i.e., smoothed out, that leads to loss of information.

Original: While similar arguments can be made to other transformers with abstractive attention, we plan to carefully analyze it and address the issue by designing an alternative pretraining method.
Revised: While similar arguments can be made to other transformers with abstractive attention, we plan to carefully analyze it and address the issue by designing an alternative pretraining method.

Original: There are some limitations on the casual mask and slice embedding introduced in Section 4.1, In the global attention, no information of tokens in current slice is taken into account because of the query shift and the attention mask excluding diagonal elements.
Revised: There are some limitations on the casual mask and slice embedding introduced in Section 4.1, In the global attention, no information of tokens in current slice is taken into account because of the query shift and the attention mask excluding diagonal elements.

Original: As a result, there is missing information from coarse-grained modeling in the current design.
Revised: As a result, there is missing information from coarse-grained modeling in the current design.

Original: This is only happening during the training phase, not in the testing, since the next token prediction with sliding window always has access to all tokens in the sequence up to the current time step.
Revised: This is only happening during the training phase, not in the testing, since the next token prediction with sliding window always has access to all tokens in the sequence up to the current time step.

Original: Therefore, this mismatch may prohibit the full modeling capability of CST .
Revised: Therefore, this mismatch may prohibit the full modeling capability of CST .

Original: To address the issue, we plan to design more suitable yet efficient training objective that allows full use of abstractions in the global attention during training.
Revised: To address the issue, we plan to design more suitable yet efficient training objective that allows full use of abstractions in the global attention during training.

Original: We leave design of more advanced autoregressive modeling as a future work.
Revised: We leave design of more advanced autoregressive modeling as a future work.

Original: CST requires additional computations of the complexity O ( N/L ) for the Q , K , and V transformation for the global attention, from the local attention output.
Revised: CST requires additional computations of the complexity O ( N/L ) for the Q , K , and V transformation for the global attention, from the local attention output.

Original: However, this increase in complexity is negligible compared to the overall computation, especially when N/L is small, as discussed in 4.3.
Revised: However, this increase in complexity is negligible compared to the overall computation, especially when N/L is small, as discussed in 4.3.

Original: A
Revised: A

Original:  F UTURE W ORKS
Revised:  F UTURE W ORKS

Original: There are several potential avenues for extending CST .
Revised: There are several potential avenues for extending CST .

Original: First, we can easily generalize our work to multi-level slicing with a higher level than two.
Revised: First, we can easily generalize our work to multi-level slicing with a higher level than two.

Original: This will lead to more improvement in expressiveness of the model while adding more degree of freedoms for better efficiency while enabling its application to much longer sequences.
Revised: This will lead to more improvement in expressiveness of the model while adding more degree of freedoms for better efficiency while enabling its application to much longer sequences.

Original: Second, as discussed in Section 4.1, more advanced design of autoregressive sequence modeling scheme will further improve the performance.
Revised: Second, as discussed in Section 4.1, more advanced design of autoregressive sequence modeling scheme will further improve the performance.

Original: Third, while CST is based on fixed-length slicing of the sequence, dynamic or semantic slicing would further improve applicability to non-stationary sequences.
Revised: Third, while CST is based on fixed-length slicing of the sequence, dynamic or semantic slicing would further improve applicability to non-stationary sequences.

Original: With end-to-end training, the model can learn the optimal slicing of the data for given tasks and provide layer-wise dynamic tokens.
Revised: With end-to-end training, the model can learn the optimal slicing of the data for given tasks and provide layer-wise dynamic tokens.

Original: To this end, efficient realization of the dynamic slicing would be the most challenging task.
Revised: To this end, efficient realization of the dynamic slicing would be the most challenging task.

Original: Finally, as a meaningful next step, modification of CST to better fit vision tasks or training a CST -based large language model would lead to more general and practical impacts.
Revised: Finally, as a meaningful next step, modification of CST to better fit vision tasks or training a CST -based large language model would lead to more general and practical impacts.