From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper seeks to develop an attention mechanism for sets that does not have any additional biases. The paper attempts to do this by splitting the points into subsets and using a specific attention pattern that forms a Hamiltonian path where each element is visited exactly once. There's a universal approximation theorem. The results look good relative to the methods that were compared to. Each of the reviewers sent me individual comments when I wrote them that moved me away from thinking this paper was borderline. The main criticism highlighted in the discussion centered around the motivation for the approach and whether the goals for the motivation were actually met. Some concrete issues raised were:

- Experiments on different kinds of data
- Precision on what is meant by "biases"
- Comparison with other methods like SetTransformer and other O(n) attention mechanisms
- Experiments on large-scale data with 10000 points

The reviewers, however, were positive about the general idea and think it warrants for clarification in motivation and a broader set of experimental results.
justification_for_why_not_higher_score: The main criticism highlighted in the discussion centered around the motivation for the approach and whether the goals for the motivation were actually met. Some concrete issues raised were:

- Experiments on different kinds of data
- Precision on what is meant by "biases"
- Comparison with other methods like SetTransformer and other O(n) attention mechanisms
- Experiments on large-scale data with 10000 points
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer V8rM: window-based transformer
comment: We thank the reviewer for appreciating that our work is efficient and well-supported. We appreciate the time you spent on the paper.

***Q: As the proposed method splits point sets into subsets, maybe it would be better if the authors could show the difference from the window-based transformer, like Stratified Transformer [1].***

***A:*** The window-based transformer is a very important branch of exploration of the transformer. Combined with the hierarchical backbone, it has been widely used in processing 2D images, languages, and 3D point clouds, such as the Swin Transformer and the Stratified Transformer. 

The window-based transformer is proposed to learn the cross-window relationships as well as the non-overlapping local relationship. However, our proposed sampled attention will not force the token to learn the relationships within subsets. The subset used in the proposed sampled attention in our Figure 1 (b) aims to link the permutation of tokens to the permutation of Cycle Attention (and as a result to the sampled attention). The sampled attention will not separate token-to-token relationships into local relationships and cross-window relationships: all token-to-token relationships (including tokens pairs in different windows) will be learned directly by Monte Carlo simulation. 

Thanks for this suggestion, and we have already included the comparison of the Stratified strategy (Figure 3 in Lai et al. 2022) in Appendix B.3. The Stratified transformer is an efficient design for learning token relationships in the hierarchical backbone but in the single-layer setting, learning $O(n^2)$ connections in the attention matrix directly may be a better solution as it could capture all global information. In contrast, our sampled attention estimates the $O(n^2)$ connections in the dense attention by implementing the Monte Carlo simulation.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer V8rM: more experiments
comment: We thank the reviewer for appreciating that our work is efficient and well-supported. We appreciate the time you spent on the paper.

***Q: From Tab.3, we can observe that the proposed method is efficient. I would like to see the performance on large-scale point cloud, such as S3DIS and SemanticKitti for segmentation. In addition, the performance on other datasets, like ScanObjectNN is also expected to show the effectiveness.***

***A:*** We have included the transfer learning results on ScanObjectNN in the BERT-based framework. Our sampled attention reached an accuracy (%) of 89.68 / 88.81 / 82.44 in OBJ-BG / OBJ-ONLY / PB-T50RS, respectively. Our sampled attention achieved a competitive performance in comparison with dense attention while reaching state-of-the-art performance. Please refer to Appendix D for more details.

In addition, we tried the point cloud segmentation experiment on the S3DIS dataset (mainly following the code provided by Point Transformer). However, due to limited time and computational resources, we could only run our implementation on sets of 2048 points. The performance of scalar attention + kNN attention mechanism (as per Point Cloud) is 62.9 / 70.6 / 88.1 with respect to mIoU / mAcc / OA, while the performance of scalar attention + sampled attention is 61.9 / 69.4 / 87.9. As we can see, our proposed sampled attention obtains a competitive performance with less computational complexity. 

We wish to emphasise that we tried our best in the limited time period to concretely address the reviewer's suggestions, and hope that the experiments are sufficient to support our proposed model, but welcome any further comments on this matter.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer AbVu: error bars
comment: We thank the reviewer for appreciating that our approach is novel and the writing is clear. We appreciate the time you spent on the paper.

***Q: For Table 1, 3, 5: do you have error bars?***

***A:*** Our implementation is mainly based on the Point Transformer (Zhao et al. 2021) and Point-BERT (Yu et al. 2022) which do not have error bars. The reason for this may be that in the implementations of Yu et al. (2022), points are sampled by furthest point sampling (FPS) in each batch (see line 121 of https://github.com/lulutang0608/Point-BERT/blob/master/tools/runner_BERT_pretrain.py). Moreover, as all points are used in training, no error bar is possible. On the other hand, Lee et al. (2019) sampled a subset of points in data pre-processing, so the point cloud is fixed in training and errors are available.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer AbVu: inductive bias
comment: We thank the reviewer for appreciating that our approach is novel and the writing is clear. We appreciate the time you spent on the paper.

***Q: Can you elaborate more on (1) why the inductive bias imposed by nearest neighbor search or learned inducing points is unneeded and (2) what evidence or intuition you have for your sampled attention having less of an inductive bias than both of these approaches?***

***A:*** We use inductive bias to refer to the prior knowledge and design built into a machine learning model. Loosely, more inductive bias may have better performance in specific tasks, while less inductive bias may have better generalisation ability (meaning, for example, wider applicability to different tasks and frameworks), and fewer hyperparameters that need to be tuned.

Our use of the phrase inductive bias is somewhat informal, and we thank the reviewer for drawing attention to it. We will revise our paper accordingly. Nonetheless, we now offer some more explanation of our intended meaning. As the reviewer mentioned, our initial research goal is to have an efficient and permutation invariant transformer for point sets / clouds. Both nearest neighbour search and inducing points are good designs as both models are efficient and permutation invariant. However, to implement the nearest neighbour search, one should introduce the hyperparameter of the number of neighbours, and the choice of definition of token-to-token distance. Further, the inducing points introduced additional parameters (inducing points themselves), which means additional backpropagation calculations. Close inspection by Lee et al. (2019) reveals a number of other non-trivial design choices.

It is difficult to compare the degrees to which different models introduce inductive bias, and we will rephrase our paper, however in contrast to e.g. the nearest neighbour based approaches, we find that our random permutation-based attention involves less intuition-guided assumptions and fewer additional hyper parameter choices. Instead, we lean on a formal proof of universal approximation, which suggests that our approach should be able to work well on a wide class of problems.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer AbVu: comparsion
comment: We thank the reviewer for appreciating that our approach is novel and the writing is clear. We appreciate the time you spent on the paper.

***Q: Could you also consider the learned inducing point method from Lee et al as a baseline?***

***A:*** We compared our proposed sampled attention with learning inducing points (Lee et al. 2019) in the basic classification setting (as per Section 5.2), by simply replacing our transformer block with the Induced Set Attention Block (ISAB) in Lee et al. (2019). The result indicates that our sampled attention outperformed Lee et al. (2019), for example, when the number of points equals 256, our sampled attention reaches an accuracy of 87.34% while the inducing points have an accuracy of 84.21%. The detailed results, implementation, and analysis are provided in our updated Appendix B.2.

Further, as our implementation of Table 3 is different from that of Lee et al. (2019) with respect to the data pre-processing step, also as the performance of the two implementations is different, we further compared the sampled attention and inducing points in the implementation provided by https://github.com/juho-lee/set_transformer, which is the official implementation of the set transformer (Lee et al. 2019). In this setting, we replaced the dense attention in Lee et al. (2019) with our proposed sampled attention, resulting in comparable performance but with less complexity. For example, when the number of points equals 5000, our sampled attention reaches an accuracy of 86.49% while the inducing points have an accuracy of 86.99%. This indicates that our sampled attention could also be applied to the inducing Set Attention Block (ISAB) introduced by Lee et al. (2019) to reduce the complexity from O(mn) to O(m). Details of the implementation may be found in our updated Appendix B.2. 

Finally, please note the numbers we report differ slightly from those of Lee et al. (2019). This may be explained by their stated use of a Tensorflow implementation in their paper while the released code is based on Pytorch. Some discrepancy is expected as documented by the authors at https://github.com/juho-lee/set_transformer/issues/2. We ran the official Pytorch implementation directly with the data pre-processing provided by Sinkformer (Sander et al. 2022) at https://github.com/michaelsdr/sinkformers.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer cTVx
comment: We thank the reviewer for appreciating our work. We thank the reviewer for the strong support and encouraging comments. We appreciate the time you spent on the paper.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer YJ8Q: contribution
comment: We thank the reviewer for appreciating our work. We appreciate the time you spent on the paper.

***Q: How do the authors think about the contributions of the proposed model in this context?***

***A:*** The vanilla transformer with a dense self-attention matrix has been the gold standard in recent years, and is based on learning $O(n^2)$ token-to-token relationships via shared query, key, and value parameters. However, it is impractical in many situations due to GPU memory limitations.  For example, our Tables 3 and 4 demonstrate a case in which the single layer of the vanilla transformer will run out of memory when dealing with point clouds with 3076 points / tokens without a tokenization step. 

The research sub-field of sparse transformers aims to address these issues while retaining the generalisation ability of the transformer. These works typically use a sparse attention pattern to estimate the original dense attention pattern. However, most of the existing sparse transformers could not be applied to set data directly as the sparse attention is not permutation invariant, which means the weighted sum of tokens in the self-attention layer will be changed when permuting tokens. We filled this research gap by proposing a permutation-invariant sparse transformer with linear complexity while retaining the good generalisation of the dense transformer.

Regarding generalisation performance, the method of Dropout (Srivastava et al. 2014) already demonstrates that sampled connections can effectively approximate and improve upon full dense connections by using Monte Carlo simulation. We also propose a form of sampled sparsity that in some cases outperforms the original dense model. For example, though the dense attention (vanilla transformer) achieves the best performance in the basic setting of a single-layer transformer (Table 3), the sampled attention outperforms the dense one when the neural network structure is relatively complex and the global minimum is difficult to attain, such as the case of the BERT-based framework (see our Table 2).

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer YJ8Q: experiments
comment: We thank the reviewer for appreciating our work. We appreciate the time you spent on the paper.

***Q: I wonder if the authors can provide more experiments on other set data.***

***A:*** We tested our proposed sampled attention with a 2D set data for the task of Amortized Clustering of Mixture Gaussians (Section 5.3 in Lee et al. 2019). The baseline is the set transformer in Lee et al. (2019), and our sampled attention is applied to Lee et al. (2019) by replacing the dense attention with the sampled one, so that the computation complexity drops to linear.

For the set transformer (Lee et al. 2019), we achieved the evaluation metric of LL0/data of -1.5034 with an error bar of 0.0072, and LL1/data of -1.4908 with an error bar of 0.0044. For the set transformer with sampled attention, we achieved the LL0/data of -1.5663 with an error bar of 0.0074, and LL1/data of -1.5272 with an error bar of 0.0052. In summary, our proposed sampled attention replaced the dense attention in the set transformer (Lee et al. 2019) to reduce the complexity from O(nm) to O(n) while retaining comparable performance, which is in line with the findings included in the submission version of our paper. The details of amortized clustering may be found in Appendix C of the updated version of our submission.

From: ICLR.cc/2023/Conference/Paper3815/Authors
title: For Reviewer YJ8Q: universal approximation
comment: We thank the reviewer for appreciating our work. We appreciate the time you spent on the paper.

***Q: What are the novel part exactly in the proof? I wonder if the authors can provide more details on this part.***

***A***: Yes, our proof of the universal approximation property is heavily based on Yun et al. (2019) and Yun et al. (2020). Specifically, the universal approximation property means that the proposed transformer with sampled attention in Figure 1(f) can approximate any function $f$ in the defined domain. 

The novel component of our proof comes from showing that our proposed sampled attention can implement contextual mappings – which is key to adapting the aforementioned proofs. 
Intuitively, a contextual map acts on sets of points to produce “labels” for those points. The important property of a contextual map is that the “labels” provided are unique not only among elements in a point cloud but are also unique depending on the point cloud (i.e. set of points) the points belong to (i.e. the “context”). For instance, an element $ x \in S_{1} \cap S_{2}$ will only have the same “label” via the contextual map if $ S_{1} $ and $ S_{2} $ are identical. This property could naively be viewed as mapping every point in a given (or different) point cloud to a unique scalar value, or ‘id-value’. 

Using contextual mapping, we have a simple procedure for approximating any set-to-set function. First, the contextual mapping maps input tokens to unique ‘id-values’ (implemented by multi-headed self-attention layers). Then these ‘id-values’ can be mapped to any desired function via the standard universal approximation properties of feed-forward layers. In this way, the transformer can estimate arbitrary set-to-set functions, and thus prove that our proposed sampled transformer block leads to a universal approximation of set-to-set functions.

Detailed proof concerning sampled attention implementing contextual maps can be found in Appendix E of our submission. Nevertheless, we highlight some of the main points here. We demonstrated that our sampled attention computes a contextual mapping by first showing that the permutation of attention is equivalent to the permutation of elements in a set (Figure 1e). We then show that sampled attention can also implement two special operators – selective shift operators and all-max-shift operators, as per Yun et al. (2020). Specifically, due to the selective shift operation, each token will first be represented by a weighted sum of its own features. Then, the first element of each token is updated according to the max value of the weighted sum step by step. As a result, this weighted sum w.r.t each token ends up creating the token’s unique ‘id-value’. The all-max-shift operator is further applied so that the interval of the unique ‘id-value’ is sufficiently large for each point cloud, which satisfies the requirements of contextual mapping.

From: ICLR.cc/2023/Conference/Paper3815/Reviewer_YJ8Q
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This work proposes a novel transformer model for dealing with point set data, with the main focus on achieving linear time complexity in the attention mechanism. By utilizing the idea of random sampling-based approximation to the vanilla attention from sampled Hamiltonian cycle attention, the authors show that the proposed model can be a universal function approximator of set-to-set functions. Experiments on point cloud data are studied with comparison to several baseline methods, showing that the proposed model can achieve comparable performance with better efficiency.
strength_and_weaknesses: - This paper is well written and easy to follow. The motivation is clear and the overall structure is well organized.
- It seems like the theoretical discussion on the universal approximation property is heavily based on the work of [1,2], while the authors also briefly touched on the role of contextual map. However, the definition 3.1 and the paragraph followed do not seem very self-explanatory to me and it's in fact a bit confusing about what are the novel part exactly in the proof. I wonder if the authors can provide more details on this part.
- One major contribution of this work focuses on achieving linear time complexity on set data. However, the experiments are only conducted on point cloud data, which is a special case of set data. I wonder if the authors can provide more experiments on other set data, such as graph data, to show the generalization of the proposed model. Also, there exists a lot of works on efficient attention in the past few years, but in recent benchmarks and scaling studies, the vanilla transformer is still the best performing one. In a similar spirit, how do the authors think about the contributions of the proposed model in this context?

[1] Yun, Chulhee, et al. "Are transformers universal approximators of sequence-to-sequence functions?." arXiv preprint arXiv:1912.10077 (2019).
[2] Yun, Chulhee, et al. "O (n) connections are expressive enough: Universal approximability of sparse transformers." Advances in Neural Information Processing Systems 33 (2020): 13783-13794.
clarity,_quality,_novelty_and_reproducibility: Please see above comments.
summary_of_the_review: I believe the problem studied in this work is interesting and this work made some contributions.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3815/Reviewer_cTVx
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposed an O(n) complexity sampled transformer that can process point set elements directly without additional inductive bias. The sampled transformer introduces random element sampling, which randomly splits point sets into subsets, followed by applying a shared Hamiltonian self-attention mechanism to each subset. The overall attention mechanism can be viewed as a Hamiltonian cycle in the complete attention graph, and the permutation of point set elements is equivalent to randomly sampling Hamiltonian cycles. This mechanism implements a Monte Carlo simulation of the dense attention connections. The paper shows that it is a universal approximator for continuous set-to-set functions.  Experimental results for classification and few-shot learning on point-clouds show comparable or better accuracy with significantly reduced computational complexity compared to the dense transformer or alternative sparse attention schemes.
strength_and_weaknesses: +: The sampled attention mechanism which maps the random permutation of set elements to the random sampling of Hamiltonian cycle attention matrices, permitting the direct processing of point sets.
+: It proves that the proposed sampled transformer is a universal approximator of continuous set-to-set functions.
+: Compared to previous transformer architectures, the empirical results show that our proposed sampled transformer achieves comparable (or better) performance with less inductive bias and complexity.
clarity,_quality,_novelty_and_reproducibility: The paper is rather theoretical and a little hard to read especially the proofs. Overall it does seem to be a good paper that is technically sound.
summary_of_the_review: This paper proposed an O(n) complexity sampled transformer that can process point set elements directly without additional inductive bias.  Experimental results for classification and few-shot learning on point-clouds show comparable or better accuracy with significantly reduced computational complexity compared to the dense transformer or alternative sparse attention schemes.

Overall I feel this is a good paper and hopefully can be very useful for a lot of follow up works.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3815/Reviewer_AbVu
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes an O(n) self-attention layer based on the sparse transformer that is permutation equivariant. The paper shows that their proposed architecture is a universal approximator, and present experiments on set-input datasets.
strength_and_weaknesses: The sparse transformer requires only O(n) computation but is not permutation invariant. This paper proposes a randomized linear-compute method that is permutation invariant in expectation and is a universal approximation of set-to-set functions.

The narrative around inductive bias is not clear to me. As the authors point out, existing works (Zhao et al, Lee et al) propose O(n) operations that are permutation invariance. Both papers prove the universal approximation property. The paper claims in the introduction that these methods “introduced additional inductive biases” and “a research question naturally arises to avoid introducing unneeded inductive bias”. Can you elaborate more on (1) why the inductive bias imposed by nearest neighbor search or learned inducing points is unneeded and (2) what evidence or intuition you have for your sampled attention having less of an inductive bias than both of these approaches?

I skimmed through the theory and proofs, and I did not find any major errors.

For some of the experiments such as Tables 3 and 5, you could also consider the learned inducing point method from Lee et al as a baseline, which I think can be better suited to this problem setting compared to kNN attention.

For Table 1, 3, 5: do you have error bars?
clarity,_quality,_novelty_and_reproducibility: The writing is clear. To my knowledge, the proposed architecture is new.
summary_of_the_review: The main narrative around "inductive biases" is unclear to me, and the experiments lack a comprehensive comparison to O(n) attention mechanisms.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3815/Reviewer_V8rM
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes an efficient sampled transformer with O(n) complexity. The developed method splits the point set into subsets, each one being processed by a shared Hamiltonian self-attention mechanism. The proposed strategy is well-supported theoretically. The method is evaluated on ModelNet40 under self-supervised learning and few-shot learning. 
strength_and_weaknesses: === strength ===

1. this paper is well-written and organized clearly. 

2. the proposed method is efficient, according to Tab.3. the proposed method is well supported theoretically.

=== weaknesses ===

1. From Tab.3, we can observe that the proposed method is efficient. I would like to see the performance on large-scale point cloud, such as S3DIS and SemanticKitti for segmentation. In addition, the performance on other datasets, like ScanObjectNN is also expected to show the effectiveness. 

2. As the proposed method splits point sets into subsets, maybe it would be better if the authors could show the difference from the window-based transformer, like Stratified Transformer [1].

[1] Lai et al., Stratified Transformer for 3D Point Cloud Segmentation.
clarity,_quality,_novelty_and_reproducibility: See above

Reproducibility: code is not available in the submission, but the implementation details might be useful for reproducibility.
summary_of_the_review: This paper is well-written and the proposed method is interesting. However, more experiments are needed.

== Post-Rebuttal ==

In the response, the authors respond that they do not conduct experiemnts on the large-scale data, and only on the 2048 points, due to the limited time and computational resources. Alothugh the reported results can show that the proposed method yields comparable performance in comparison with previous methods while requiring less resources, I still think it would be better to conduct comparison on large-scale data, which is important for the application of transformer on large-scale data. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold