Paper:
Learning Deep Disentangled Embeddings With the F-Statistic Loss
Deep-embedding methods aim to discover representations of a domain that make explicit the domain's class structure and thereby support few-shot learning. Disentangling methods aim to make explicit compositional or factorial structure. We combine these two active but independent lines of research and propose a new paradigm suitable for both goals. We propose and evaluate a novel loss function based on the F statistic, which describes the separation of two or more distributions. By ensuring that distinct classes are well separated on a subset of embedding dimensions, we obtain embeddings that are useful for few-shot learning. By not requiring separation on all dimensions, we encourage the discovery of disentangled representations. Our embedding method matches or beats state-of-the-art, as evaluated by performance on recall@k and few-shot learning tasks. Our method also obtains performance superior to a variety of alternatives on disentangling, as evaluated by two key properties of a disentangled representation: modularity and explicitness. The goal of our work is to obtain more interpretable, manipulable, and generalizable deep representations of concepts and categories.
The literature on deep embeddings (Chopra et al., 2005;Yi et al., 2014a;Schroff et al., 2015;Ustinova & Lempitsky, 2016;Song et al., 2016;Vinyals et al., 2016;Snell et al., 2017) addresses the problem of discovering representations of a domain that make explicit a particular property of the domain instances. We refer to this property as class or category or identity. For example, a set of animal images might be embedded such that animals of the same species lie closer to one another in the embedding space than to animals of a different species. Deep-embedding methods are trained using a class-aware oracle which can be queried to indicate whether two instances are of the same or different class. Because this paradigm can handle an arbitrary number of classes, and because the complete set of classes does not have to be specified in advance-as they would be in an ordinary classifier-deep embeddings are useful for few-shot learning. A small set of examples of novel classes can be projected into the embedding space, and an unknown instance can be classified by its proximity to the embeddings of the labeled examples.
Similar to deep embeddings, the literature on disentangling attempts to discover representations of a set of instances, but rather than making explicit a single property of the instances (class), the goal is to make explicit multiple, independent properties, which we refer to as factors. For example, a disentangled representation of animals might include factors indicating its size, length of its ears, and whether it has feet or fins. We will later be more rigorous in defining a disentangled representation, but for now we operate with the informal notion that the factors form a compositional or distributed representation such that with relatively few factors and relatively few values of each factor, the factor values can be recombined to span the set of instances. Disentangling has been explored using either a fully unsupervised procedure (Chen et al., 2016;Higgins et al., 2017) or a semi-supervised 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montréal, Canada.
procedure in which a factor-aware oracle can be queried to specify a factor along with sets of instances partitioned by factor value (Reed et al., 2014;Kulkarni et al., 2015;Karaletsos et al., 2015;Reed et al., 2015).
Despite their overlapping and related goals, surprisingly little effort has been made to connect research in deep embeddings and disentangling. There are two obvious ways to make the connection. First, a factor-aware oracle might be used to train deep embeddings (instead of a class-aware oracle), and hopefully disentangled representations would emerge. Second, a class-aware oracle might be used to train disentangled representations (instead of a factor-aware oracle), and hopefully an embedding suitable for few-shot learning would emerge. We primarily pursue the former approach, but briefly explore the latter as well.
In the next section, we propose a deep-embedding method that is suitable for both few-shot learning of novel classes and for disentangling factors. After describing the algorithm and showing that it obtains state-of-the-art results on the recall@k task that is ordinarily used to evaluate embeddings, we turn to analyzing how well the algorithm disentangles the factors that contribute to class identity. To perform a rigorous evaluation, we put forth formal, quantifiable criteria for disentanglement, and we show that our algorithm outperforms other state-of-the-art deep-embedding methods and disentanglement methods in achieving these criteria.

Review:
Summary  This work seeks to combine methods from deep embedding literature with the aims of disentanglement to create disentangled embeddings. The authors use the F statistic: a well known statistical measure of inter vs intra class variance, in order to introduce separation between classes.   To produce the embedding, convolutional features are used to train a fully connected embedding network with a loss designed around the F-statistic. The embeddings were evaluated against other state of the art deep metric losses such as triplet loss, structured softmax loss, and histogram loss. The F-statistic loss did not significantly outperform these other methods.  The authors then evaluated the degree of disentanglement between the produced embeddings using two main metrics: modularity and explicitness. Similar to InfoGAN and Beta-VAE, dimensions of a disentangled representations are required to have mutual information with at most one factor of variation - this is defined as modularity. Explicitness is defined as the existence of a linear transformation between a dimension of the embedding, and the value of the factor of variation it represents. Here this is calculated with a one-vs-rest linear classifier, where explicitness is predicated on the ability of the classifier to predict the correct class from a single dimension of the embedding.     The disentangled representations are trained in a weakly-supervised fashion, where an oracle conveys information about the similarities between batch samples, but not the name of the factor upon which they differ. The embeddings were compared against histogram loss (Ustinova & Lempitsky, 2016), triplet (Schroff et al., 2015), as well as the unsupervised Beta-VAE. Under the metrics of modularity and explicitness, the F-statistic based embeddings performed better than other methods.  Review  F-tests are often used to decompose the variance given in a modeled distribution. Using the F-statistic to encourage disentangled representations seems natural to me. Simply creating well structured embeddings was clearly not the point of this work, as the embeddings did not perform significantly better than other methods. However, under the two computed statistics of modularity and explicitness, the F-statistic based loss performed better than both fully supervised and unsupervised methods (even if comparing to unsupervised methods is unfair).   That being said, the paper was unclear for me in a number of places: 1) I didn’t understand the training mechanism for the disentangled representations. How was 5 fold validation used here, and why was it important? A graphic showing an example curated mini-batch would help here.  2) It would be helpful to explicitly state how this measure of modularity differs from the mutual information terms which InfoGAN use in their losses to encourage disentanglement.  3) Why not directly evaluate against a method which explicitly encourages disentanglement like the F-statistic loss, such as (Chen et al. 2016) which minimizes total correlation between dimensions.  4)  This could also be done in a fully supervised way, this is a stronger argument for the F-statistic in general if it is shown to beat state of the art as both a supervised and weakly-supervised method.  The main idea is simple if not appreciated: using the F-statistic to encourage separation between classes. I have not seen this before in disentanglement research, but I expect to see it again in the future. I think there could have been a stronger evaluation done by comparing against other methods. If already comparing to a unsupervised method like Beta-VAE, why not compare to a method like InfoGAN which explicitly maximizes mutual information between a small subset of the code, and the observation,  I believe this work was well thought-out, but it needs either a stronger evaluation, or a stronger argument for its current evaluation criterion.  ** After rebuttal:  I still don't like that they don't want to compare to InfoGAN. They used the metrics in the following paper (which delineates requirements for disentanglement)  https://openreview.net/pdf?id=By-7dz-AZ   And this paper scores InfoGAN higher than Beta-VAE on almost all metrics e.g. "InfoGAN achieves the highest average disentanglement",  "InfoGAN also achieves the highest average completeness",  "InfoGAN contains the most easily-extractable / explicit information about [embedding] z.  So I don't buy their assertion that Beta-VAE is better than InfoGAN thus comparing to Beta-VAE is all that matters. I still believe the evaluation is insufficient because of this.