Paper: ConvFormer: Revisiting Token-mixers for Sequential User Modeling (ICLR-2024)

Reviewer comment (Experiments):
The most obscure part of the work is the experimental setting for devising the key three principles pertinent to the construction of effective and efficient sequential recommenders. Section 3 is devoted to this description and deserves more accurate wording and rich explanations.

Author response:
**[Q1.1] Causality. On the setting for key principles. Section 3 is devoted to this description and deserves more accurate wording and rich explanations. For some reason, there are no illustrations provided in Section 3 for the considered SAR-O, SAR-P, and SAR-R models. There is a footnote referring to the appendix A.4 section, which also doesn't contain the figures...Explanation probably requires more than a few sentences. Graphical representation along with mathematical formulation in this section would help to convince the reader and minimize confusion.**
- **More Explanations with Graphical Illustrations.** Thank you for your kind prompt. We have included comprehensive graphical illustrations and mathematical formulations in **Appendix C.2** for SAR and its variants. The corresponding footnote in the main text has been updated to direct readers to these contents.
- **Clarifying Order-Sensitivity in SAR and Its Variants:** Recognizing the importance of elucidating the order-sensitivity of SAR variants, **we have expanded our descriptions in the revised Appendix C.2**. 
The concept of order sensitivity in a token mixer can be defined as follows: for any item in the input sequence, changing the order of other items alters the representation outputted by the token mixer. To illustrate this, consider the representation of the $i$-th item in the input sequence, denoted as $r_i \in \mathbb{R}^\mathrm{D}$. 
In SAR and its variants, the output representation of $r_i$ is given by $s_i=\sum a_{i,j}r_j$ (we omit the value mapping here for brevity. It is reasonable since the value mapping is conducted on each step isolatedly, without impact on order sensitivity). 
If we switch the order of two items, $j_1$ and $j_2$, the output representation changes accordingly: $s_i^\prime=\sum_{j=1,j\neq{j_1},j\neq j_2}^{\mathrm{L}}a_{i,j}r_j + a_{i,j_1}r_{j_2}+a_{i,j_2}r_{j_1}$, reflecting the model's sensitivity to the item order. **In SAR**, $s_i$ does not exhibit order sensitivity.
    This is because the attention weights are dynamically determined based on the similarity between items, irrespective of their positions in the sequence. Consequently, swapping two items, $j_1$ and $j_2$, does not alter the output representation of the $i$-th item, i.e., $s^\prime_i=s_i$. 
    **In SAR-O and SAR-R**, the weights $a_{i,j}$ are fixed with respect to positions rather than dynamically calculated, which makes $s_i$ sensitive to the order of items, manifested as $s^\prime_i \neq s_i$.
    **In SAR-P**, the only difference from SAR-O is that the weights $a_i$ are dependent on $r_i$: a_i=[a_{i,j}]_{j=1:L}=\mathrm{MLP}(r_i). Since the generated weights remain dependent on the position index $j$, $s{^\prime}_i\neq s_i$ holds and thus SAR-P is order-sensitive.

Reviewer comment (Presentation):
For some reason, there are no illustrations provided in Section 3 for the considered SAR-O, SAR-P, and SAR-R models. There is a footnote referring to the appendix A.4 section, which also doesn't contain the figures.

Author response:
**[Q1.1] Causality. On the setting for key principles. Section 3 is devoted to this description and deserves more accurate wording and rich explanations. For some reason, there are no illustrations provided in Section 3 for the considered SAR-O, SAR-P, and SAR-R models. There is a footnote referring to the appendix A.4 section, which also doesn't contain the figures...Explanation probably requires more than a few sentences. Graphical representation along with mathematical formulation in this section would help to convince the reader and minimize confusion.**
- **More Explanations with Graphical Illustrations.** Thank you for your kind prompt. We have included comprehensive graphical illustrations and mathematical formulations in **Appendix C.2** for SAR and its variants. The corresponding footnote in the main text has been updated to direct readers to these contents.
- **Clarifying Order-Sensitivity in SAR and Its Variants:** Recognizing the importance of elucidating the order-sensitivity of SAR variants, **we have expanded our descriptions in the revised Appendix C.2**. 
The concept of order sensitivity in a token mixer can be defined as follows: for any item in the input sequence, changing the order of other items alters the representation outputted by the token mixer. To illustrate this, consider the representation of the $i$-th item in the input sequence, denoted as $r_i \in \mathbb{R}^\mathrm{D}$. 
In SAR and its variants, the output representation of $r_i$ is given by $s_i=\sum a_{i,j}r_j$ (we omit the value mapping here for brevity. It is reasonable since the value mapping is conducted on each step isolatedly, without impact on order sensitivity). 
If we switch the order of two items, $j_1$ and $j_2$, the output representation changes accordingly: $s_i^\prime=\sum_{j=1,j\neq{j_1},j\neq j_2}^{\mathrm{L}}a_{i,j}r_j + a_{i,j_1}r_{j_2}+a_{i,j_2}r_{j_1}$, reflecting the model's sensitivity to the item order. **In SAR**, $s_i$ does not exhibit order sensitivity.
    This is because the attention weights are dynamically determined based on the similarity between items, irrespective of their positions in the sequence. Consequently, swapping two items, $j_1$ and $j_2$, does not alter the output representation of the $i$-th item, i.e., $s^\prime_i=s_i$. 
    **In SAR-O and SAR-R**, the weights $a_{i,j}$ are fixed with respect to positions rather than dynamically calculated, which makes $s_i$ sensitive to the order of items, manifested as $s^\prime_i \neq s_i$.
    **In SAR-P**, the only difference from SAR-O is that the weights $a_i$ are dependent on $r_i$: a_i=[a_{i,j}]_{j=1:L}=\mathrm{MLP}(r_i). Since the generated weights remain dependent on the position index $j$, $s{^\prime}_i\neq s_i$ holds and thus SAR-P is order-sensitive.

Reviewer comment (Theory):
The most critical part that is missing in the explanation of these models is how the sequential order is taken into account and how the causal structure of learning is preserved (so that the leaks from future are avoided).

Author response:
**[Q1.1] Causality. On the setting for key principles. Section 3 is devoted to this description and deserves more accurate wording and rich explanations. For some reason, there are no illustrations provided in Section 3 for the considered SAR-O, SAR-P, and SAR-R models. There is a footnote referring to the appendix A.4 section, which also doesn't contain the figures...Explanation probably requires more than a few sentences. Graphical representation along with mathematical formulation in this section would help to convince the reader and minimize confusion.**
- **More Explanations with Graphical Illustrations.** Thank you for your kind prompt. We have included comprehensive graphical illustrations and mathematical formulations in **Appendix C.2** for SAR and its variants. The corresponding footnote in the main text has been updated to direct readers to these contents.
- **Clarifying Order-Sensitivity in SAR and Its Variants:** Recognizing the importance of elucidating the order-sensitivity of SAR variants, **we have expanded our descriptions in the revised Appendix C.2**. 
The concept of order sensitivity in a token mixer can be defined as follows: for any item in the input sequence, changing the order of other items alters the representation outputted by the token mixer. To illustrate this, consider the representation of the $i$-th item in the input sequence, denoted as $r_i \in \mathbb{R}^\mathrm{D}$. 
In SAR and its variants, the output representation of $r_i$ is given by $s_i=\sum a_{i,j}r_j$ (we omit the value mapping here for brevity. It is reasonable since the value mapping is conducted on each step isolatedly, without impact on order sensitivity). 
If we switch the order of two items, $j_1$ and $j_2$, the output representation changes accordingly: $s_i^\prime=\sum_{j=1,j\neq{j_1},j\neq j_2}^{\mathrm{L}}a_{i,j}r_j + a_{i,j_1}r_{j_2}+a_{i,j_2}r_{j_1}$, reflecting the model's sensitivity to the item order. **In SAR**, $s_i$ does not exhibit order sensitivity.
    This is because the attention weights are dynamically determined based on the similarity between items, irrespective of their positions in the sequence. Consequently, swapping two items, $j_1$ and $j_2$, does not alter the output representation of the $i$-th item, i.e., $s^\prime_i=s_i$. 
    **In SAR-O and SAR-R**, the weights $a_{i,j}$ are fixed with respect to positions rather than dynamically calculated, which makes $s_i$ sensitive to the order of items, manifested as $s^\prime_i \neq s_i$.
    **In SAR-P**, the only difference from SAR-O is that the weights $a_i$ are dependent on $r_i$: a_i=[a_{i,j}]_{j=1:L}=\mathrm{MLP}(r_i). Since the generated weights remain dependent on the position index $j$, $s{^\prime}_i\neq s_i$ holds and thus SAR-P is order-sensitive.

Reviewer comment (Experiments):
The authors use a simple leave-last-out holdout sampling scheme to hide test items, which is subject to "recommendations from future" type of leaks (see [Ji at el 2023]). Combined with non-causal item-to-item mixer this would create an unfair advantage to the proposed modifications of the SASRec, which employs strictly triangular mask and ensures causal learning. This could explain the observed superiority without addressing the sequential learning task. Modifying the experimentation scheme to exclude such leaks is important. One can also run a pre-check by lifting the triangular mask from SASRec, which would likely outperform the proposed modifications, revealing a general problem with the setup.

Author response:
**[Q1.2] Causality. Leave-last-out holdout sampling scheme combining with lacking causal mask would create an unfair advantage ... modifying the experimentation scheme that excludes the "recommendations from future" leaks is particularly important as well.**

Thank you for your valuable comment!  We highly value any non-clarity here and are pleased to give our response at a structral manner.

- **Preserving of Causality in Our Study.**  In our methodology, we have incorporated causality using an equivalent approach to a traditional causal mask. For an input sequence like [1,2,3,4,5], we employ autoregressive generation of sub-sequences ([1], [1,2], [1,2,3], [1,2,3,4]) to predict the next item ([2], [3], [4], [5]) respectively, minimizing the prediction error, as demonstrated in our code (lines 11-19 in datasets.py). It effectively mirrors the function of an explicit causal mask (i.e., multiplying the attention matrix with a causal mask and then minimize the error to predict [2,3,4,5] using the entire input sequence) and avoids the concerned leakage. Therefore, we do not confer any unfair advantage to the proposed SAR variants and ConvFormer in terms of causality. **We have highlighted it in the revised section 4.2.3.**

- After handling the raised concern "leave-last-out scheme coupling with non-causal item-to-item mixer would create an unfair advantage to SAR variants", we turn to discuss the leave-last-out scheme itself. In comparison to another popular scheme, the random split [1], the leave-last-out scheme has been more effective in mitigating leakage. We agree that in some instances, this scheme also suffers from data leakage. However, the critism [1] mainly focuses on collaborative filtering that recommends based on **user-wise similarity**. The root (see Figure 5 in [1]) is that the future data would be falsely involved when calculating the similarity between user A, B, C which interacted with the recommender chronologically with overlapping. In this case, the "future" interactions of C would be leaked to A by calculating the similarity between A-B and B-C.
In contrast, sequential recommendation is less prone to such root of leakage (i.e., user-wise similarity), since it models each user individually and avoids user-wise similarity calculations. 
Furthermore, we slightly note that in the realm of recommendation, **each split scheme has its trade-offs**; therefore, the key is to align the scheme with business demanding and respect it for fair comparison. The timepoint split scheme suggested by [1], while avoiding certain biases, leads to shorter and incomplete test sequences that misalign with real-world serving scenarios. In comparison, the leave-last-out scheme aligns more with the actual serving scenarios, where using the entire (long) user history for recommendations is common, especially in mature business. This decision was also influenced by our industrial experience and the need to stay consistent with established public benchmarks that sometimes lack timestamp data.

Reviewer comment (Experiments):
In the receptive field size experiments in Section 3.2, when K nearest neighbors are selected, the simple symmetric distance is used |i-j|<K. No causal mask is used here either, which reaffirms suspicions regarding possible leaks in the training.

Author response:
**[Q1.4] Causality. The related issue arises with receptive field size experiments in Section 3.2. When K nearest neighbors are selected, the simple symmetric distance is used |i-j|<K. So, no causal mask is used here either, which only reaffirms suspicions regarding the possible leaks in the training.**
- Thank you for your sincere comment. We believe **this concern could be resolved after rectifying Q1.2.** Specifically, causality is ensured by autoregressively generating sub-sequences for training beforehand, which is equivalent to using causal mask for the entire sequence.

Reviewer comment (Theory):
There seems to be a contradiction with how SASRec is represented in the text. In Section 4, the authors outline three key components of an effective sequential learner and state that SASRec does not correspond to order sensitivity and large receptive field. However, both properties are part of the sequential self-attention mechanism: it preserves order via successive learning of hidden states over the sequence using a triangular mask (positional embeddings have marginal contribution), and operates over all preceding items at any step within an input sequence. This requires careful wording and more clarification.

Author response:
**[Q1.5] Causality. There seems to be a contradiction. In section 4, the authors outline three key components and state that SASRec does not correspond to the two of them: order sensitivity and large receptive field. However, both properties are ingrained into the sequential self-attention mechanism...it operates over all preceding items at any step within an input sequence.**

- **Large Receptive Field.** We acknowledge the importance of a large receptive field in SASRec (rather than opposing it), and this is consistent with our perspective in the main text. We have highlighted the role of a large receptive field as a key factor in SASRec's performance: `By deconstructing vari
ous aspects of self-attentive token mixer, we identify two factors favouring its performance: a large
receptive field and a lightweight architecture.`, and `These findings underscore the importance of a large receptive field for SAR's performance.`
If there are any inaccuracies or potentially misleading statements in our manuscript regarding this point, we welcome specific references to them, and we are committed to rectifying any such issues.
- **Order Sensitivity.**  As previously discussed in response to Q1.2, we have ensured causality through autoregressive generation of training sequences, an approach that mirrors the function of a causal mask. In each generated sub-sequence, SASRec does not exhibit order sensitivity, as evidenced by the relationship $\mathbf{r}{^\prime}_i=\mathbf{r}_i$ (detailed in the resposne to Q1.1). As an equivalent (but kind of difficult to interpret) view, we can use causal mask to establish causality and then minimizing the prediction error from the 2nd to the L-th items in the sequence. It in essence devides the sequence [1,2,3,4,5] into L-1 sequences: [1], [1,2], [1,2,3], [1,2,3,4] for predicting the next item; in each subsequence SASRec does not exhibit order sensitivity.

Reviewer comment (Theory):
The proposed solution also raises concerns about information leakage. The depth-wise convolution operates within each hidden component, blending information across the entire sequence, potentially losing the causal structure and allowing information from future elements. A careful analysis of the leaks would help resolve these concerns.

Author response:
**[Q1.6] Causality. Continuing on the topic of information leakage, the proposed solution also raises concerns. Specifically, the depth-wise convolution that operates within each hidden component, meaning it blends in information across entire sequence, and therefore loses the causal structure and allows capturing information from future elements. A careful analysis of the leaks would help resolve the concerns.**
- Thank you for your sincere comment. We believe **this issue has been effectively addressed after rectifying Q1.2.** The **causality has been ensured by autoregressively generating training sequences beforehand**, which is equivalent to exerting causal attention mask and optimize the model outputs at multiple steps simultaneously. It also prevents depth-wise convolution from introducing leakage from future elements.

Reviewer comment (Experiments):
Even though the authors claim to only modify the attention mechanism, the proposed solution also deviates from the original SASRec implementation in terms of the loss function. Changing the loss function can significantly impact SASRec performance even without changing its architecture (see cited works). Therefore, comparison with SASRec is unfair unless the same loss is used in both models or SASRec's loss is also changed to the proposed pairwise objective. Otherwise, the claimed contribution of architectural changes is not convincing.

Author response:
**[Q2] Baselines. To make comparison fair, the same loss should used in the proposed model or the SASRec's loss function is also modified to employ the proposed pairwise objective.**

- **The loss functions of SASRec and ConvFormer are the same.** We acknowledge and appreciate your suggestion regarding the consistency of loss in model comparison. We apologize for an unexpected typo in the initial manuscript regarding the loss function formulation. In fact, they **share the same loss calculation**. As evidenced in our uploaded code, both SASRec and ConvFormer, along with all toy models, are trained using the same PairwiseTrainer (referenced in line 173 of src/trainers.py) and employ the same bce loss calculator (mentioned in line 152 of src/trainers.py).

- **Highlighting Consistency in Revised Documentation:** In the revised version of our paper, we have corrected the formulation of loss function. Then, we noted in the footnote 1 to highlight experimental fairness: `We use the experimental settings in Section 5.1, e.g., setting the embedding size to 64, the maximum
sequence length to 50. Experiments are repeated 5 times with different seeds. We
only modify the token mixer matrix A in (1), maintaining learning objectives and tricks identical to SASRec.`

Reviewer comment (Experiments):
The authors mention computational cost of long sequences, but most real recommender system applications have relatively short sequences (compared to NLP). Providing a table with dataset statistics, including average sequence lengths (as in the SASRec paper), would clarify this. Even sequence length 50 is rarely attainable in most datasets. Applying FFT to compute convolutions is standard practice and only loosely related to the problem of long sequences, so it should not be presented as a particular contribution by the authors.

Author response:
**[Q3.1] Other issues. Most of the real applications in recommender systems have relatively short sequences.**
- **Variability in Optimal Sequence Lengths:** The optimum of sequence length (L) varies based on business needs and dataset characteristics. For instance, movie-lens (used in our appendix A.3) exhibits optimal performance at L=200 as reported by BERT4Rec [8]. Additionally, in many practices, particularly in industry, behavior sequences can be considerably longer (L>1,000), since incorporating more user behaviors shows consistent performance improvements (referenced in Fig.1 in [9]). It highlights the efficiency and efficacy of handling long-term user sequences, sometimes extending to **extreme lengths (L>54,000) and life-long sequences** [9,10,11,12]. **Therefore, the complexity of processing long input sequences becomes a critical aspect when designing sequential user models.** Motivated by this, we extended our complexity discussion to long sequences, to avoid associated concerns regarding ConvFormer.


**[Q3.2] Other issues. Applying FFT to compute convolutions is a logical step but it is only remotely related to the problem of long sequences. It is ubiquitously used to compute convolutions and should not be presented as a particular contribution by the authors.**
- **Clarification of FFT's Role in the Study:** According to the response to Q3.1, the user sequence would be quite lengthy. It's important to clarify that **FFT is employed in our study not as a central novelty, but as a strategic component**  to manage the complexity of lengthy user sequences, to bypass unexpected concerns.  Despite these technical points, **our primary innovation and contribution is investigating and rethinking token mixer design in sequential user modeling**. It yields empirical rules that flourish a simple model achieving SOTA performance and could influence future innovations. To highlight the main contribution, we have only briefly mentioned the complexity issue and the FFT solution in the main text, allocating only a small section for this, while detailed descriptions, algorithms, and corresponding experiments are included in the appendix.

- **Comparison with Recent FFT-empowered Works:** 
Many recent studies [2-4] and contributions to ICLR 2024 [5-7] are featured by the use of Fourier analysis and Fourier convolution in specific problem-solving contexts. In contrast, our work does not highlight the application of Fourier analysis as its main novelty. Our technical novelty is the overall framework (as shown in the second contribution in our introduction), which integrates the lightTCN module and the meta-former architecture. The use of FFT for convolution is to support these components by addressing computational complexities, not as a standalone innovation.

### **Reference**

[1] Ji, Yitong, et al. "A critical study on data leakage in recommender system offline evaluation." ACM Transactions on Information Systems 41.3 (2023): 1-27.

[2] Li, Chongyi, et al. "Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement." The Eleventh International Conference on Learning Representations. 2022.

[3]Zhou, Man, et al. "Fourmer: an efficient global modeling paradigm for image restoration." International Conference on Machine Learning. PMLR, 2023.

[4]Huang, Zhipeng, et al. "Adaptive Frequency Filters As Efficient Global Token Mixers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] "Revitalizing Channel-dimension Fourier Transform for Image Enhancement." Submission to ICLR, Openreview, 2023.

[6] "HoloNets: Spectral Convolutions do extend to Directed Graphs." Submission to ICLR, Openreview, 2023.

[7] "On the Matrix Form of the Quaternion Fourier Transform and Quaternion Convolution." Submission to ICLR, Openreview, 2023.

[8] Sun, Fei, et al. "BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer." Proceedings of the 28th ACM international conference on information and knowledge management. 2019.

[9] Pi, Qi, et al. "Practice on long sequential user behavior modeling for click-through rate prediction." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019.

[10] Pi, Qi, et al. "Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction." Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.

[11] Zhang, Yuren, et al. "Clustering based behavior sampling with long sequential data for CTR prediction." Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022.

[12] Cao, Yue, et al. "Sampling is all you need on modeling long-term user behaviors for CTR prediction." Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 2022.

Reviewer comment (Experiments):
No online real experiment

Author response:
**[W1] No online real experiment. Only the ID feature was considered.**

- **Online Experiments.** We appreciate your emphasis on real-world application validity. This work follows the evaluation protocol of current arts in the field [1-4]. **Recognizing the importance of real-world applications, we have implemented ConvFormer on our private business dataset, which has surpassed most existing protocols [1-4]**. While our results show promise in both open benchmarks and industrial data, in our industry, new features typically undergo extensive offline testing before being integrated into online systems. As academic researchers collaborating with the business sector, we lack the authority to independently initiate online deployment of the model.

- **Non-ID Features:** We agree that incorporating non-id features is a good supplementary to this work. However, it's important to note that **the majority of academic benchmarks (and our industrial dataset) primarily focus on ID features.** 
Given that additional information can be integrated through supplementary representation extractors – a process not central to the core of sequential user modeling – most studies in this area concentrate solely on item IDs [1,2,3].
As a research paper, we adhere to this established norm of sequential user modeling, which typically involves using historical item IDs to predict future item IDs.

Reviewer comment (Experiments):
Only the ID feature was considered

Author response:
**[W1] No online real experiment. Only the ID feature was considered.**

- **Online Experiments.** We appreciate your emphasis on real-world application validity. This work follows the evaluation protocol of current arts in the field [1-4]. **Recognizing the importance of real-world applications, we have implemented ConvFormer on our private business dataset, which has surpassed most existing protocols [1-4]**. While our results show promise in both open benchmarks and industrial data, in our industry, new features typically undergo extensive offline testing before being integrated into online systems. As academic researchers collaborating with the business sector, we lack the authority to independently initiate online deployment of the model.

- **Non-ID Features:** We agree that incorporating non-id features is a good supplementary to this work. However, it's important to note that **the majority of academic benchmarks (and our industrial dataset) primarily focus on ID features.** 
Given that additional information can be integrated through supplementary representation extractors – a process not central to the core of sequential user modeling – most studies in this area concentrate solely on item IDs [1,2,3].
As a research paper, we adhere to this established norm of sequential user modeling, which typically involves using historical item IDs to predict future item IDs.

Reviewer comment (Experiments):
What's the relationship between the optimal K (kernel size) and the dataset sequence length?

Author response:
**[Q1] What's the relationship between the optimal K (kernel size) and the dataset sequence length (L)?**
- **Optimal K given different L.** Your query regarding the optimal kernel size (K) in relation to the sequence length (L) is insightful and directly pertains to the efficacy of the large receptive field criterion in sequential user modeling. Our experimental investigations on the Yelp dataset, conducted with five random seeds and adhering to the training and evaluation protocols specified in the main text, shed light on this relationship.  Due to changes in the hardware platform, experimental results may exhibit minor variations. 
The experiments indicate **a general trend: as the sequence length increases, so does the value of the optimal kernel size**. This trend suggests that longer sequences benefit from larger kernel sizes, which effectively capture the broader context within the sequence. For instance, on the Yelp dataset, when L is set to 10 and 20, the optimal values of K are observed to be 10 and 20, respectively. Further extending L to 100 results in optimal K values in the range of 70-100. Following your kind suggestion, the efficacy of large receptive field (the criterion 2) is further validated and underscored.

Reviewer comment (Experiments):
How would you choose the input sequence length?

Author response:
**[Q2] How would you choose the input sequence length?**

- **Common Setting in Academic Studies:** We appreciate your question regarding the selection of input sequence length. In academic contexts, the key is to ensure reproducibility and comparability with existing baselines. Therefore, adhering to the settings of current arts is crucial, i.e., L=50 as demonstrated by references [1,2,4]. We respected and adhered to this setting in our study.

- **Adjustment in Industrial Practice:** L is treated as a tunable hyperparameter in industry, optimized to balance model performance and business requirements for inference speed. Empirically, the optimal sequence length is closely related to the average sequence length of the dataset [2,3]. In our industrial practice, we set L=50, a choice that has been optimized over years and is in line with current online serving requirement.

Reviewer comment (Experiments):
Will adding more block improve performance?

Author response:
**[Q3] Will adding more block improve performance?**

- **Number of blocks (N).** This is an insightful comment. Our default setting follows the N=2 configuration from [1].  Nevertheless, we agree the importance of this hyperparameter, and add experiments to assess the impact of varying N. The results from the Yelp dataset, with block numbers ranging from 1 to 4 and two seeds are summarized as follows.

  - **N=1:** Suboptimal performance across all metrics.
  - **N=2:** Optimal performance, showing notable improvement over N=1.
  - **N>2:** Little improvement over N=2, suggesting a saturation point at N=2. Although it is expected to observe further improvement, such improvement often entails a finetuning in regularization strength, e.g., increasing dropout, to avoid overfitting. Besides, in sequential user modeling, simple and light models often exhibit superior performance. 

| HIT@1  | HIT@5  | HIT@10 | NDCG@5 | NDCG@10 | MRR   | Seed | Number of blocks (N) |
|--------|--------|--------|--------|---------|-------|------|----------------------|
| 0.2528 | 0.6037 | 0.7677 | 0.4343 | 0.4876  | 0.4128| 1    | 1                    |
| 0.2519 | 0.6058 | 0.7692 | 0.4348 | 0.4878  | 0.4124| 2    | 1                    |
| 0.2843 | 0.6316 | 0.7877 | 0.4655 | 0.5162  | 0.4423| 1    | 2                    |
| 0.2830 | 0.6357 | 0.7876 | 0.4670 | 0.5163  | 0.4424| 2    | 2                    |
| 0.2794 | 0.6332 | 0.7862 | 0.4633 | 0.5131  | 0.4387| 1    | 3                    |
| 0.2853 | 0.6385 | 0.7878 | 0.4692 | 0.5177  | 0.4441| 2    | 3                    |
| 0.2808 | 0.6320 | 0.7864 | 0.4635 | 0.5137  | 0.4394| 1    | 4                    |
| 0.2819 | 0.6347 | 0.7847 | 0.4657 | 0.5145  | 0.4408| 2    | 4                    |


  



- **Number of Hidden Units (D).** In addition to the number of blocks, we also explored the influence of varying D. Our findings can be summarized as follows.

  - **D=32:** Comparable performance to SOTA models with D=64, but slightly lower.
  - **D=64:** Balanced and optimal, yielding the best performance.
  - **D>64:** Increasing D beyond 64 doesn't significantly enhance performance, sometimes even showing a decrease.



| HIT@1  | HIT@5  | HIT@10 | NDCG@5 | NDCG@10 | MRR   | Seed | Number of hidden units (D) |
|--------|--------|--------|--------|---------|-------|------|----------------------------|
| 0.2672 | 0.6342 | 0.7988 | 0.4585 | 0.5120  | 0.4335| 1    | 32                         |
| 0.2581 | 0.6349 | 0.7970 | 0.4536 | 0.5064  | 0.4267| 2    | 32                         |
| 0.2843 | 0.6316 | 0.7877 | 0.4655 | 0.5162  | 0.4423| 1    | 64                         |
| 0.2830 | 0.6357 | 0.7876 | 0.4670 | 0.5163  | 0.4424| 2    | 64                         |
| 0.2834 | 0.6255 | 0.7910 | 0.4615 | 0.5120  | 0.4393| 1    | 96                         |
| 0.2857 | 0.6279 | 0.7762 | 0.4641 | 0.5123  | 0.4411| 2    | 96                         |
| 0.2816 | 0.6179 | 0.7675 | 0.4559 | 0.5045  | 0.4341| 1    | 128                        |
| 0.2808 | 0.6183 | 0.7668 | 0.4563 | 0.5045  | 0.4341| 2    | 128                        |
| 0.2814 | 0.6100 | 0.7525 | 0.4523 | 0.4986  | 0.4312| 1    | 196                        |
| 0.2828 | 0.6057 | 0.7492 | 0.4506 | 0.4972  | 0.4308| 2    | 196                        |
| 0.2746 | 0.5954 | 0.7405 | 0.4413 | 0.4885  | 0.4222| 1    | 256                        |
| 0.2732 | 0.5953 | 0.7448 | 0.4409 | 0.4894  | 0.4221| 2    | 256                        |


- These results reinforce our understanding of the ConvFormer model's behavior concerning these hyperparameters. The choice of N=2 blocks and D=64 hidden units seems to strike a balance, capturing the essential patterns in the dataset without unnecessarily increasing complexity.

Reviewer comment (Novelty):
The formulation of permutation-variant (or order-sensitive) self-attention is outdated. There are enormous ways to equip self-attention with the sensitivity to input sequence order. Just to name few of them: **Learnable absolute embedding**: Training data-efficient image transformers & distillation through attention, Touvron et al, ICML 2021; **Sinusoidal absolute embedding**: Attention is All You Need, Vaswani et al, NeuRIPS 2017; **Rotary relative embedding**: RoFormer: Enhanced Transformer with Rotary Position Embedding, Su et al, arXiv 2021; **Learnable relative embedding**: Swin Transformer V2: Scaling Up Capacity and Resolution, Liu et al, CVPR 2022. All these modelings have shown superior performance in NLP, CV, and Robotics.

Author response:
> The formulation of permutation-variant (or order-sensitive) self-attention is outdated.
- **We never discuss the concept `order-sensitive self-attention`.** The core of Section 3.1 is the order **IN**sensitivity of vanilla self-attention models.
- We focused on the order-sensitivity of **token-mixer**, rather than the role of positional embeddings (PEs). PEs do matter (and we also involved it) but their role in preserving order information is marginal, a perspective supported by extensive literature including TxzE's comment: `positional embeddings also count but their contribution is marginal`.

Reviewer comment (Novelty):
Eqn (2) should cite `Vasani et al.` and `Dosovitskiy et al` (An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale). Summing input tokens with position encodings is a common procedure in transformer models.

Author response:
> Eqn (2) should cite Vasani et al. and Dosovitskiy et al.
- **We cited `Zhou et al., 2022`, a recent literature that used PE in Eqn (2) in our field.** Recognizing the extensive literature on learnable PEs in diverse fields like NLP and computer vision, we opted for **recent literature in our primary area with similar settings**. The foundational role of the vanilla Transformer by `Vasani et al.` is also acknowledged in our related work section.

Reviewer comment (Theory):
The formulation of SAR-P seems not sensitive to the order of input sequence but the features of the input token $R[l]$.

Author response:
> The formulation of SAR-P seems not sensitive to the order of input sequence but the features of the input token
- **SAR-P is sensitive to the order of input sequence**. We provided analysis in Appendix C.2. 
- The order sensitivity can be defined as follows: for any item in the input sequence, changing the order of other items alters the representation outputted by the token mixer. To illustrate this, consider the representation of the $i$-th item in the input sequence, denoted as $r_i \in \mathbb{R}^\mathrm{D}$. 
In SAR and its variants, the output representation of $r_i$ is given by $s_i=\sum a_{i,j}r_j$ (we omit the value mapping here for brevity. It is reasonable since the value mapping is conducted on each step isolatedly, without impact on order sensitivity). 
If we switch the order of two items, $j_1$ and $j_2$, the output representation changes accordingly: $s_i^\prime=\sum_{j=1,j\neq{j_1},j\neq j_2}^{\mathrm{L}}a_{i,j}r_j + a_{i,j_1}r_{j_2}+a_{i,j_2}r_{j_1}$, reflecting the model's order sensitivity. **In SAR**, $s_i$ does not exhibit order sensitivity since $s^\prime_i=s_i$. **In SAR-O**, $a_{i,j}$ are fixed w.r.t. positions, which makes $s^\prime_i \neq s_i$. **In SAR-P**, the only difference from SAR-O is that the weights $a_i$ are dependent on $r_i$: a_i=[a_{i,j}]_{j=1:L}=\mathrm{MLP}(r_i). **Since the generated weights are dependent on the position index $j$, $s{^\prime}_i\neq s_i$ holds and thus SAR-P is order-sensitive.**
- If the statements above do not clarify your confusion, **could you elaborate on your perspective regarding SAR-P's order insensitivity?**

Reviewer comment (Theory):
The formulation of window-masked self-attention is odd. One key component in self-attention is to fix the numerical scale of aggregated features $S = A(RW^{(V)})$, where the summation of each row in $A$ is constrained to be $1$. To apply window masking, a common practice is to set the entries, outside the window, in $A$ to $-\inf$ (see Sec 3.2.3 in `Vasani et al.`). However, the formulation in Sec 3.2 of this paper does not guarantee scale invariance of feature aggregation. In extreme cases, where $A$ has very small attention within the window, the scales of aggregated features become $0$.

Author response:
> The formulation of window-masked self-attention is odd.
- Our practical implementation adhered to the scale-invariant approach you mentioned, as evidenced in our code (`lines 242-247 in src/modules.py`). The manuscript initially used a simplified description for clarity; but this simplification ignored the role of scale. We will revise the statement accordingly.

Reviewer comment (Experiments):
The ablation of receptive field in Fig 1 is weird. Existing works usually study the effect of receptive field with convolutional models `Liu et al.` (A ConvNet for the 2020s).

Author response:
> The ablation of receptive field in Fig 1 is weird. Existing works usually study the effect of receptive field with convolutional models.
- The criticism is not clear to us.  Our aim in Section 3 is to comprehensively ablate Transformer models to identify factors influencing performance, and thus we conduct experiments on Transformer. **Is it prohibitive to study the effect using different models with current works?**

Reviewer comment (Novelty):
This paper proposes to replace self-attention modules with convolutional layers. The idea is not novel, as the final model becomes Residual Network `He et al.` (Deep Residual Learning for Image Recognition). In fact, there are papers comparing transformer architectures with convolutional UNet for sequential modeling tasks, like `Chi et al.` (Diffusion Policy: Visuomotor Policy Learning via Action Diffusion). Moreover, there are also works integrating unify both sides in a single model `Rombach et al.` (High-Resolution Image Synthesis with Latent Diffusion Models).

Author response:
> This paper proposes to replace self-attention modules with convolutional layers. The idea is not novel, as the final model becomes Residual Network He et al. (ResNet).

- **The model significantly diverges from ResNet.** (1) ResNet used a limited receptive field, which contradicts Criterion 2 of our study and is shown to be ineffective in Section 5.3.1. (2) ResNet used conventional convolution operator, which contradicts Criteria 3 and is shown to be ineffective in Section 5.3.2. (3) ResNet does not conform to the meta-former architecture, which is central to our investigation focused on the token mixer design within this framework.
- It's noteworthy that if researchers used ResNet-like model to beat advanced Transformers in their specific field and explored the rational behind with theoretical or empirical evidence, **the work would be fairly valuable that should not be summarily dismissed**. See [1] for reference.

Reviewer comment (Novelty):
The idea of using convolution in sequential user modeling tasks is already explored in `Zhou et al., 2022`.

Author response:
> The idea of using convolution in sequential user modeling tasks is already explored in Zhou et al., 2022
- Our contribution transcends a specific architecture. It re-evaluates Transformer-like architectures and identifies three criteria for effective token mixers in sequential user modeling. ConvFormer serves as a **proof-of-concept**, illustrating that even simplified models adhering to our proposed criteria can excel over more complex solutions. This concept is further corroborated by experiments with **other simple models**, such as an affine token mixer detailed in Appendix F.