Paper:
We introduce a new deep learning technique that builds individual and class representations based on distance estimates to contextual dimensions for different modalities. Recent works have demonstrated advantages to creating representations from probability distributions over their contexts rather than single points in a low-dimensional Euclidean vector space. These methods, however, rely on preexisting features and are limited to textual information. In this work, we obtain generic template representations that are vectors containing the average distance between the distribution of a class and that of contextual information, the latter of which consists of a selection of a thresholded number of classes from the power set of classes. These representations have the benefit of being both interpretable and composable. They are initially learned by estimating the Wasserstein distance for different data subsets with deep neural networks. Individual samples or instances can then be compared to the generic class representations, which we call templates, to determine their similarity and thus class membership. We show that this technique, which we call WDVec, delivers good results for multi-label image classification. Additionally, we illustrate the benefit of templates and their composability by performing retrieval with complex queries where we modify the information content in the representations. Our method can be used in conjunction with any existing neural network and create theoretically infinitely large feature maps.

Review:
The paper defines a representation learning strategy based upon estimation
of a matrix of Wasserstein distances.
The idea is excellent.  The ability to "solve" IPMs reliably is a recent
development in deep learning whose ramifications are still being explored.
Intuitively this line of research could plausibly result in general
methods which are theoretically intelligible and broadly applicable.
Indexing at least one side of the matrix of estimated WDs with events
(rather than classes) has interpretability properties useful for
information retrieval and also conveys benefits reminiscent of learning
with privileged information.
However, the exposition could be greatly improved by using the
standard language of probability theory.  The discussion in 3.1
was particularly painful to read.  What is the difference between
"existing in an environment" and "conditioning on a measurable event"?
Phrases like "belonging to any random subset of the dataset" suggest
a non-deterministic method of selecting an element of the power set of
the training data, but it is unclear what to do if more training data
arrives in this case.  
Throughout the entire paper the word "random" is apparently used in the 
colloquial sense of "arbitrary".  *Correct every instance of this.*
If you actually are referring to generating samples from a distribution,
be explicit about the generative process.
Section 3.5 was more confusing than enlightening.  In general I understand 
that environments can be leveraged for intelligibility and admit manipulation 
for information retrieval.  The exact strategy remains somewhat opaque.  If 
you are under space constraints refer to an appendix with more explicit details.
In the experiments section phrases like "environments consist of random 
combinations of classes" is also not helpful.  Do you mean something like 
"uniformly selected from the set of all class pairs?"  Or something like 
"uniformly selected from the power set of all classes?"  How volatile
are the experimental results with respect to the non-deterministic choice 
of environments? 
I want to accept this paper if the exposition is improved, which I think
is possible during the response period.
My other comments are not blocking issues, but would either improve the
current paper or inform future directions of research.
The technique bears some resemblance to Wasserstein Discriminant
Analysis.[1]  That paper seeks a projection that maximizes the ratio of 
Wasserstein distance between classes vs. within classes.  Here, 
although the common representation is a nonlinear mapping 
analogous to a projection, we merely try to estimate all the
Wasserstein distances rather than maximize them, so it is not trained
to be discriminative per se.   That is ok since the representation is
designed to be used for a variety of tasks (modulo section 4.2), but it 
does leave open the question "what if the matrix of estimated 
Wasserstein distances isn't informative, e.g., due to poor choice of 
environments?"  There is no attempt to assess the representation 
except via utility in downstream tasks.
The common representation was justified computationally, but I suspect
is beneficial statistically.  It might facilitate safely including a
large number of environments and then spectrally compressing (i.e., SVD)
the resulting matrix without overfitting the data.  However clearly if
the capacity of this layer is too small, then all estimated WDs will
be close to zero.  If we posit a low Bayes error classifier for the
multi-class problem associated with the dataset, that might imply there
is some conditioning of the input under which the matrix of (actual) WDs
has rank equal to the number of classes, which would in turn provide a
useful diagnostic to guard against an insufficiently discriminative choice
of environments or insufficient capacity in the common representation. If
the matrix is full rank with a flat spectrum, however, that might indicate
the choice of environments is too granular and overfitting has occurred,
it's not immediately obvious to me how to guard against this.  
I am curious what the results in appendix A.1. look like relative to the spectral 
norm or the smallest eigenvalue of the estimated WD matrix (smallest 
eigenvalue assuming number of environments < number of classes,
otherwise the k-th eigenvalue where k = number of classes).
[1] https://arxiv.org/abs/1608.08063