Paper:
Many machine learning tasks such as multiple instance learning, 3D shape recognition and few-shot image classification are defined on sets of instances. Since solutions to such problems do not depend on the permutation of elements of the set, models used to address them should be permutation invariant. We present an attention-based neural network module, the Set Transformer, specifically designed to model interactions among elements in the input set. The model consists of an encoder and a decoder, both of which rely on attention mechanisms. In an effort to reduce computational complexity, we introduce an attention scheme inspired by inducing point methods from sparse Gaussian process literature. It reduces computation time of self-attention from quadratic to linear in the number of elements in the set. We show that our model is theoretically attractive and we evaluate it on a range of tasks, demonstrating increased performance compared to recent methods for set-structured data.

Review:
This paper presented an attention-based neural network, namely set transformer, a new neural model 
based on original transformer designed for set inputs. The basic idea is to introduce the attention
mechanism in both learning the feature embeddings of the set inputs during “encoding” and aggregating 
these embeddings during “decoding”. The paper is written clearly and well motivated. The extensive 
set of experiments were conducted to demonstrate the effectiveness of the proposed method. In general, 
I like reading this paper but there are some limitations or unclear parts I need authors to clarify
and explain. 
i) The proposed architecture is mainly adopted from the original transformer but it is highly related
to the baselines used in the experiments. For instance, it seems like that the current set 
transformer is a simple combination of Yang et al.(2018) and Mishra et al.(2018) (using Stack of
SABs) in encoder side and of Ilse et al.(2018) (using PMA and stack of SABs) in the decoder side. 
This simple combination makes the novelty of this paper unclear. I would like authors to clarify 
more on the originality w.r.t. these previous works. 
ii) Although authors proposed a variant of SABs - ISABs using landmark points to accelerate the 
computation, there are no any runtime comparisons between SABs and ISABs by fixing other components. 
It would be interesting to see that ISABs can approach the performance SABs and how it approaches it. 
For instance, shall we expect that ISABs approach the performance of SABs when increasing the number
of landmark points (inducing points)? Since in practice most of datasets are relatedly large, I think
understanding the behavior of ISABs is a more interesting problem. 
iii) After seeing the results in table 6, I have quite concerned about the practical performance of
set transformer on relatively large datasets (like 1000 points each class in the settings.) It looks
to me that not only set transformer may have computational issues to scale up, but more importantly
that when encoder learned really expressive embeddings with a relatively large number of the set 
inputs it might be little need to leverage attention in pooling anymore. I would like authors to 
conduct some other experiments on relatively large datasets to verify this hypothesis, which is 
important for the practical applications of the proposed model.