Paper:
We explore the idea of compositional set embeddings that can be used to infer not just a single class, but the set of classes associated with the input data (e.g., image, video, audio signal). This can be useful, for example, in multi-object detection in images, or multi-speaker diarization (one-shot learning) in audio. In particular, we devise and implement two novel models consisting of (1) an embedding function f trained jointly with a "composite" function g that computes set union operations between the classes encoded in two embedding vectors; and (2) embedding f trained jointly with a "query" function h that computes whether the classes encoded in one embedding subsume the classes encoded in another embedding. In contrast to prior work, these models must both perceive the classes associated with the input examples, and also encode the relationships between different class label sets. In experiments conducted on simulated data, OmniGlot, and COCO datasets, the proposed composite embedding models outperform baselines based on traditional embedding approaches.

Review:
This paper describes a way to train functions that are able to represent the union of classes as well as to query if the classes in an image subsume the classes in another image. This is done throughly jointly training embedding functions, a set union function and a query function. The paper reads well.
While the approach is reasonable, the experiments seem to be quite incomplete and no explanation is given why a trivial solution cannot be used instead of the learnt functions.
The paper argues for learning a set union function however much of the evaluation focuses on quite small sets of 2 or 3 items. On the evaluation that utilises larger sets, e.g. COCO, there isn't any analysis of how performance of the technique scales with the size of the set since that would be one of the defining characteristics of a set union function. The COCO experiment is also lacking in detail, for example, how many items are there in the positive and negative sets and how the test set is balanced. Finally, it seems that f, g and h could be trivial non-learnt functions. For example, f could be a function that maps an image to a binary representation of its classes (this could be a typical ResNet image classifier), g could be a function that does a binary OR of its two arguments and h could be a function that uses a binary AND and equality test on its two arguments. In this case, g and h don't need to be learnt at all. This may not be possible in the COCO experiment where the individual labels are not known but it seems quite unrealistic to have a dataset where only pairwise subset relationships are known.
It also seems that the f is always different between that used with g and that used with h, is this the case? SimRef also doesn't do data augmentation but there's no explanation why it is done for the proposed method and not for this baseline. The MF baseline in experiment 1 seems to be a straw man especially since the baselines in experiment 2 are much stronger.
================================================================================
Update after rebuttal:
Thanks for answering my questions and performing the additional experiments with a ResNet baseline and performing an additional analysis based on the number of subclasses in figure 5. I think these provide a substantially better analysis of the algorithm so I've increased my score correspondingly. For the final paper, I think it would be good to add TradEmb/ResNet to figure 5 as well to understand how those methods scale worse/better with the number of subclasses.