Paper:
We construct a general unified framework for learning representation of structured data, i.e. data which cannot be represented as the fixed-length vectors (e.g. sets, graphs, texts or images of varying sizes). The key factor is played by an intermediate network called SAN (Set Aggregating Network), which maps a structured object to a fixed length vector in a high dimensional latent space. Our main theoretical result shows that for sufficiently large dimension of the latent space, SAN is capable of learning a unique representation for every input example. Experiments demonstrate that replacing pooling operation by SAN in convolutional networks leads to better results in classifying images with different sizes. Moreover, its direct application to text and graph data allows to obtain results close to SOTA, by simpler networks with smaller number of parameters than competitive models.

Review:
Summary:
The paper argues that plain (fully connected) neural networks cannot represent structured data, e.g. sequences, graphs, etc. Specialized architectures have instead been invented for each such case, e.g. recurrent neural networks, graph networks etc. The paper then proposes to treat these structured data types as sets and propose a neural network method for converting a set into a fixed size vector, which can then be classified using plain neural networks. The method works by projecting each member of the set into M dimensions where they are passed through a ReLU and summed. The paper proves that given high enough dimensionality M, no information will be lost during this process. The paper performs experiments on graph data, text data and image data. The paper concludes "We proposed a general framework for processing structured data by neural network."
Quality:
The paper seems rushed. I think the paper has some language problems which unfortunately detract from the overall impression. "through its all one-dimensional projections". Should that be "through all its one-dimensional projections"? "Analogically". "can lead to the >lose< of meaningful information". "where (the?) Radon transform". 
Clarity:
The aim and purpose of the paper is fairly clear. I think the method explanation is overly complicated. If I understand correctly, the proposed method is simply to map each set element to an M dimensional space using a linear projection followed by a ReLU and then summing the set elements. The section on how to implement it in practice further complicates the paper unnecessarily in my opinion. It's not clear why or whether the ReLU is important.
Originality:
Transforming sets into fixed size vectors is not new. See e.g. [1] and [2], which the paper does not reference or compare to.
To the papers defence I think the main idea is to map any structured data into a set, and then use a (relatively) simple method on it.
Significance:
The paper acknowledges that methods do indeed exist for the various structured data types, but claim that they are complex, and that the proposed method is a simple general alternative. As such the significance of the paper hinges on whether the proposed method is indeed simpler, and how it compares when it comes to performance. The proposed method is simple and general, but it does require that the information lost when converting the structured data to the set is encoded as features in the set elements, e.g. the sequence index is added. For images, the normalized position must be added. For graphs, the edges should be added, which interestingly is not done in the single graph experiment (I'd be curious how the edges could be added). This detracts somewhat from the claim to generality, since each structured data type must still be handled differently.
It's clear to me that the structural priors built into the specialized networks, e.g. recurrent, graph, etc. should help for these data structures. For this reason I think the proposed method will have a hard time comparing head to head. That is OK, it becomes a tradeoff between generality versus performance. Unfortunately I'm not convinced of the performance by the experiments. Specifically the proposed method is not compared head to head against the methods it proposes to replace. 
 * On the graph data it is compared against various classifiers that use a fixed size, handcrafted representation. This is disingenuous in my opinion. For graph data, the natural method it proposed to replace is a graph neural network. 
 * On the sequence data it is compared against a 1D convolutional neural network. The canonical sequence model is a recurrent neural network. Also for the larger IMDB dataset the performance drop against a CNN is considerable.
 * The image experiments are probably the strongest case. Here the authors make a compelling case that the SAN pooling operator is better than max-pooling. Unfortunately the authors use a dataset which have already been size normalized during pre-processing, and then un-normalize it. It'd be more convincing if the authors showed superior performance on a dataset of images of varying sizes using SAN, compared to normalizing them during pre-processing (with comparable runtime and parameter counts).
Pros:
 - Ambitious aim
 - Simple method
Cons: 
 - Proposing to replace xyz methods, then not fairly comparing to them in their respective domains.
 - Not referencing relevant prior work on neural networks for sets.
 - Paper seems rushed/unfinished
 - Implicitly assumes classification/regression for entire structured data. In many cases, e.g. Named Entity Recognition, and many graph problems, the problem is to classify/regress on the individual elements, as part of the whole.
I could see the paper accepted, if sold less as the end-all solution for structured data and more as simply an improved pooling operator, with SOTA experiments to back up the claims. This would also allow the authors to use SOTA methods for pre-processing the data, e.g. RNNs, graph neural networks, etc., as they don't need to compete against every method. 
(btw. instead of zero padding the batches, the authors can probably maintain a list of indices of set elements and use https://www.tensorflow.org/api_docs/python/tf/math/segment_sum)
[1] - Vinyals, Oriol, Samy Bengio, and Manjunath Kudlur. "Order matters: Sequence to sequence for sets." arXiv preprint arXiv:1511.06391 (2015).
[2] - Zaheer, Manzil, et al. "Deep sets." Advances in Neural Information Processing Systems. 2017.