S UPERCLASS -C ONDITIONAL G AUSSIAN M IXTURE M ODEL F OR L EARNING F INE -G RAINED E MBEDDINGS
A BSTRACT
Learning fine-grained embeddings is essential for extending the generalizability of models pre-trained on “coarse” labels ( e.g. , animals).
It is crucial to fields for which fine-grained labeling ( e.g. , breeds of animals) is expensive, but fine-grained prediction is desirable, such as medicine.
The dilemma necessitates adaptation of a “coarsely” pre-trained model to new tasks with a few “finer-grained” training labels.
However, coarsely supervised pre-training tends to suppress intra-class variation, which is vital for cross-granularity adaptation.
In this paper, we develop a training framework underlain by a novel superclass-conditional Gaussian mixture model (SCGM).
SCGM imitates the generative process of samples from hierarchies of classes through latent variable modeling of the fine-grained subclasses.
The framework is agnostic to the encoders and only adds a few distribution related parameters, thus is efficient, and flexible to different domains.
The model parameters are learned end-to-end by maximum-likelihood estimation via a principled Expectation-Maximization algorithm.
Extensive experiments on benchmark datasets and a real-life medical dataset indicate the effectiveness of our method.
I NTRODUCTION
Training deep models with sufficient generalizability is of fundamental importance, which demands immense training data with fine-grained annotations (Krizhevsky et al., 2012; Brown et al., 2020).
In many fields, however, data labeling requires domain-specific knowledge, such as medicine (Sohoni et al., 2020), thus is prohibitive, and infeasible to be exhaustive.
In this case, data for model training may only be “coarsely” labeled, while later the model is tested on a finer-grained classification task (Bukchin et al., 2021).
For example, consider an event prediction task for dialysis patients (Inaguma et al., 2019).
Hemodialysis is a major renal replacement therapy for patients with end-stage renal failure.
These patients have to take hemodialysis thrice a week, each lasts for 4-5 hours.
During the treatment, unexpected events, such as muscle cramp, perspiration, and dizziness, may happen as a result of lowering blood pressure, which need intensive medical care, thus should always be avoided.
It is therefore an important medical issue to predict such events before an initiation of hemodialysis.
In this task, binary labels, which mark the incidence of an event, can be collected.
In contrast, finergrained labels that annotate different subtypes of events are seldom recorded.
Since distinguishing different subtypes facilitates precise diagnoses, and helps physicians assess the risk for deciding whether to perform a hemodialysis (with certain precautions), it is desirable that a model trained with coarse (binary) labels can perform well on a finer-grained multi-class (subtypes) task.
To fill the gap of granularity between the training and testing scenarios, a practical way is to collect a few new records for a patient, with their fine-grained annotations.
These data constitute a support set for fine-tuning a pre-trained model to the specific data distribution induced by the annotations of the target patient, for whom the adapted model is used for future predictions.
Although massive finegrained annotation is impractical, annotating a few-shot set is feasible.
In this work, we are interested in such a Cross-Granularity Few-Shot (CGFS) learning problem, where a model pre-trained on a set of coarse classes (denoted as superclasses ), needs to adapt to an unseen set of fine-grained target classes (denoted as subclasses ).
The target subclasses could be descendants of the superclasses (as in the aforementioned example), or could descend from other superclasses that are unobserved during pre-training.
To be practical, the adaptation should only use a few samples from the subclasses.
The CGFS problem is not limited to the above application.
It occurs in a model’s lifespan when an application requires separating some subclasses from the superclasses yet when the training dataset was created these subclasses were unannotated.
For example, it could occur in detecting rare pathology or variants using medical images that were coarsely described (Oakden-Rayner et al., 2020), or personalizing a generic model trained on all historical users to a specific customer (Luo et al., 2020).
Despite its significance, CGFS cannot be trivially solved by regularly training models with coarse labels, because typical losses for supervised learning aim to maximize inter-class boundaries but neglect intra-class variation.
Thus, subclasses may arbitrarily and unevenly spread in every superclass.
Recently, Bukchin et al.
(2021) proposed to integrate coarse supervision and contrastive learning (within superclasses) for solving CGFS.
However, their approach cannot be readily used for medical records which typically contain static profiles and time series (Che et al., 2016).
This is due to the absence of a standard augmentation method for generating constrastive pairs on some data other than images.
Also, since contrastive learning does not model subclass explicitly, their solution could be suboptimal (as evaluated in Sec. 4).
Moreover, as their model was built upon MoCo (He et al., 2020), and maintains many dictionaries for data sampling, its computational costs are high (Sec. 4).
In this work, we propose a novel S uperclass-Conditional Gaussian Mixture model (SCGM) to learn fine-grained embeddings for the CGFS problem.
Our contributions are summarized as follows.
• SCGM is agnostic to the encoder, thus is flexible to different applications.
It models the generation of samples from hierarchical classes, and explicitly represents the unobserved subclasses by latent variables, without assuming their identities.
It dynamically computes a Gaussian mixture for every sample conditioned on its superclass, and the model forms a hierarchy of Gaussian mixtures.
• SCGM only adds a small overhead to an encoder, for parameterizing its distributions, thus is efficient.
The model parameters are learned end-to-end by maximum likelihood estimation via a principled Expectation-Maximization (EM) algorithm.
We also theoretically linked our loss function to InfoNCE (Oord et al., 2018), explaining its effectiveness from a contrastive perspective.
• In the experiments, we evaluated SCGM on both benchmark image datasets and a real-life medical dataset.
Since SCGM is compatible with contrastive learning, we also tested it with a momentum encoder (He et al., 2020).
The results demonstrate SCGM on generic encoders has already outperformed the state-of-the-art (SOTA) baselines, with less computational costs
and it achieves boosted performance in some cases with momentum contrast integrated.
R ELATED W ORK
To our best knowledge, this is the first work to develop an SCGM model for underlying a framework that enables tackling CGFS across domains.
The most relevant work (Bukchin et al., 2021) combines superclass-wise contrastive learning with coarse classification for preserving intra-class variation.
Another work (Yang et al., 2021) used a three-step approach that pseudo-labels the embeddings (pretrained by coarse classification and batch-wise contrastive learning) by clustering every superclass.
The pseudo-fine-labels were used to re-train the encoder.
A similar three-step method (Sohoni et al., 2020) used a different loss for maximizing the worst-case expected accuracy over the pseudo-labeled subclasses.
As discussed before, the former two methods require non-trivial searches for a suitable data augmentation method, which may be unavailable for some non-image data.
By splitting training steps, the latter two methods could lead to suboptimal pseudo-labeling, and misleading labels could confuse the downstream steps.
In contrast, our model is end-to-end.
It explicitly infers the posterior of the subclass during coarse training.
Moreover, it can achieve better performance without using as many computational resources as the model of (Bukchin et al., 2021)
(as evaluated in Sec. 4).
Learning with coarse supervision.
Several works deal with coarse/fine labels from the perspective of weakly supervised learning (Zhou, 2018) than to tackle the CGFS problem, including training methods that take advantage of a mixture of either balanced (Ristin et al., 2015; Guo et al., 2018;
Taherkhani et al., 2019) or unbalanced (Hsieh et al., 2019;
Liu et al., 2019; Robinson et al., 2020) coarse and fine labels.
Among them, Liu et al.
(2019) addressed a few-shot learning problem, but the model training assumes access to some fine labels and a graph of class hierarchy, which are unavailable in our problem.
Thus it cannot be adapted to solve our problem.
Few-shot learning.
Meta-learning has become a popular idea to handle the few-shot learning problem, which derives metric-based (Vinyals et al., 2016;
Snell et al., 2017) and optimization-based methods (Finn et al., 2017; Nichol et al., 2018).
The idea has been extended to semi-supervised (Ren et al., 2018), unsupervised (Hsu et al., 2018) and semantics-augmented (Xing et al., 2019) scenarios, when labels are scarce.
However, none of them explores coarse labels for cross-granularity learning.
Recently, many works observed that learning embeddings on all classes (without episodic training), followed by simple fine-tuning, is superior to SOTA meta-learning methods (Wang et al., 2019; Dhillon et al., 2020; Tian et al., 2020).
In this work, similar to (Bukchin et al., 2021), we focus on this paradigm to learn useful embeddings, and do not use meta-learning for pre-training.
Embedding methods.
The emerging self-supervised methods, such as contrastive methods (Oord et al., 2018; Chen et al., 2020a; He et al., 2020), are appealing in their ability to attain comparable embeddings to the supervised counterparts, and even surpass them when transferring to other tasks (He et al., 2020; Tian et al., 2020).
Beyond instance-wise contrast, recent methods have explored instance-cluster contrast to boost performance (Caron et al., 2020; Li et al., 2020).
In unsupervised methods, joint embedding and clustering has been found beneficial (Caron et al., 2018;
Asano et al., 2020).
Whereas, these methods never exploited coarse supervision.
Thus, their embeddings/clusters do not necessarily reflect intra-class variation, which is important to the CGFS task.
Vanilla Gaussian mixture (GM) model is unfit for the CGFS scenario.
A recent work (Manduchi et al., 2021) extended GM to constrained clustering by conditioning every sample with a prior clustering preference.
It is neither supervised by coarse classes nor hierarchically structured.
Conventional hierarchical GM is used for hierarchical clustering (Goldberger & Roweis, 2005; Olech &
Paradowski, 2016; Athey et al., 2019) by applying GM agglomeratively or divisively.
These unsupervised methods only infer clusters, but do not pre-train embedding models for task adaptation.
S UPERCLASS -C ONDITIONAL G AUSSIAN M IXTURE M ODEL
Firstly, a word about some notations.
Let D train =
( x i , y i ) } ni =1 be n sample-label training pairs, where y i ∈
Y super = { 1 , ..., c } is a superclass label.
Each x i is associated with a latent (unobserved) subclass label ˆ y i ∈ Y sub
= { 1 , ..., s } .
Y sub relates to Y super by a hierarchical structure, i.e. , Y sub can be partitioned into c disjoint sets Y sub- 1 , ...,
Y sub- c , such that
if ˆ y i ∈ Y sub- j , then y i =
j ( 1 ≤ j ≤ c ).
Let f θ be an encoder ( i.e. , backbone network) that is trained on D train (without knowing Y sub ).
It maps x i to a d -dimensional feature f θ ( x i ) .
At test time, given a k -shot support set for a subset
Y m sub ⊆
Y sub of m subclasses, i.e. , D support = { ( x i , ˆ y i )
| ˆ y i ∈
Y m sub } mki =1 , the task is to train a classifier
C : R d → Y m sub with optimal accuracy on a test set of Y m sub subclasses.
In our experiments, we also explored the case when the subclasses belong to superclasses that are not used for pre-training.
As discussed in Sec. 2, our focus is to train f θ for good embeddings without modifying f θ during the adaptation–a paradigm with SOTA few-shot performance (Dhillon et al., 2020; Tian et al., 2020).
Formally, let f θ ( x i )
= v i , our goal is to find the model parameter θ that maximizes the likelihood of the posterior distribution p
θ ( y i | v i ) on the observed data in D train for classification tasks (Grathwohl et al., 2019).
To model the unobserved subclasses, we associate every v
i with a latent variable z i to indicate to which subclass v i belongs.
Suppose there are r possible subclasses, the log-likelihood to maximize can be rewritten by marginalizing out the latent variables.
[Equation]
where the distribution p θ ( z i | v i ) specifies the subclass membership of v i , and p ( y i | z i ) associates
z i with a subclass partition Y sub-
y i .
Unlike some previous works (Sohoni et al., 2020), which searched the number of subclasses for every superclass using a quality metric, we only assume a total number of r subclasses, and seek to infer their relationship with superclasses, i.e. , p ( y i | z i ) , without any prior on subclass partitions, so that the model is more generalizable.
3. P ARAMETERIZATION OF P ROBABILITIES
In Eq.
(1), both p ( y i | z i ) and p θ
( z i | v i ) are not specified, and involve non-parametric variables.
To make it solvable, we introduce our SCGM that models the generative process of the embeddings.
̅" # ̅" $ " # " $ " %
Suppose & ' = 1 (a) * + ' & ' =
Conditional distribution of + '
Conditional distribution of + ' ̅" # ̅" $ " # " $ " %
Suppose & ' = 2 (b) ̅" # , ̅" $ are means of superclasses(i.e.
); " # , " $ , " % are means of subclasses (i.e., - ' ) .
& ' 1 ' - ' + '
SCGM .
/ - ' + '
Generic Gaussian mixture (c)
[Figure]
Let µ j and Σ j ( 1 ≤ j ≤ r ) be the mean and variance of the j -th mixture component, and π j be its corresponding mixture probability.
In a generic GM model, the generation of v i involves two steps: (1) draw a latent variable z i from a categorical distribution on all mixture components, and (2) draw v i from the Gaussian distribution N ( µ z i , Σ z i ) (Bishop, 2006), as illustrated by Fig.
1(c).
In step (1), the categorical distribution is usually defined on π =
[ π 1 , ..., π r ] ∈ R r , i.e. , the static mixture probabilities, which cannot reflect different samples’ superclasses.
In light of this, for each v i , we propose to define π i =
[ π i 1 , ..., π ir ] to be conditional on y i , which dynamically changes w.r.t.
different samples.
To this end, we introduce ¯ µ y i and ¯Σ y
i as the mean and variance of superclass
y i , and compute π ij = softmax ( − D y i (
µ j ) 2 / 2)
, where D y i ( · ) measures the Mahalanobis distance between a data point and the superclass distribution N ( ¯ µ y i , ¯Σ y i ) .
Basically, π ij evaluates the density ratio of the mixture component mean µ j , by regarding it as a sample from y
i ’s distribution.
[Equation]
where p N ( ·|· ) represents the density function of a multivariate Gaussian distribution.
Our generative process can be summarized as
for each sample index i : (a) draw a superclass label
y i ∼ Categorical ( p ( y i ))
(b) for j = 1 , ..., r : i. compute π ij = softmax ( − D y i ( µ j ) 2 / 2) using Mahalanobis distance D y i ( · ) (c) draw a latent subclass variable for the
i -th sample z i ∼ Categorical ([ π i 1 , ..., π ir ]) (d) draw a feature vector v i ∼ N ( µ z i ,
Σ z i )
where p ( y i ) is a prior on superclasses, which can be drawn from a Dirichlet distribution Dir ( α ) (the conjugate of categorical distribution), and α can be estimated by the ratios of different labels in
D train .
p ( y i ) can also be set as a uniform prior, for unknown datasets and better generalization.
As can be seen, steps (a)(b) specify a (static) mixture of superclasses, and steps (c)(d) specify a (dynamic) mixture of subclasses.
The whole process establishes a hierarchical GM distribution.
Fig.
1(a)(b) illustrate the process using 2 superclasses and 3 subclasses, where the embedding distribution p ( v i | y i ) dynamically changes w.r.t.
v i ’s superclass as a result of updated π i .
For example, when y i = 1 (Fig. 1(a)), v i is more likely to be generated from the first two subclasses.
This y
i -adjusted
Gaussian mixture distribution highlights the first two subclasses, and will be used to fit data with y i = 1 during model training.
Fig.
1(c) bottom illustrates the graphical model of SCGM, which distinguishes itself from the generic GM in Fig.
1(c) by (1) the condition y i , and (2) the variable π i .
Now, if we consider ¯ µ y i and µ z i as the surrogate representations of y i and z i in the embedding space, we can specify two probabilities p
( z i | y i ) and p ( v i |
z i ) using their corresponding densities:
[Equation]
from which, the probabilities p ( y i | z i ) and p ( z i | v i ) in Eq.
(1) can be derived by using Bayesian rules as (the detailed derivation is deferred to Appendix A.1)
[Equation]
where θ are the parameters of encoder f θ , ϕ includes our added distribution parameters.
To reduce complexity, in this work, we investigate the feasibility of using isotropic Gaussian with tied variance, i.e. , Σ z i
= I σ 2 and ¯Σ y
i
= I ¯ σ 2 , which turned out to be efficient in our experiments.
Here, I is the identity matrix, σ and ¯ σ are hyperparameters.
Empirically, it is sufficient to set ¯ σ = 1 and tune σ to adjust the relativity between super- and subclasses.
Also, tied variance is a commonly used trick in
Gaussian discriminate analysis (GDA) for generative classifiers (Lee et al., 2018).
Finally, substituting the factors in Eq.
(1) with Eq.
(4), whose probabilities are specified and parameterized, we get our SCGM induced loss ℓ SCGM ( D train ; θ , ϕ ) , where ϕ = {{ µ j } rj =1 , { ¯ µ j } cj =1 } .
3. M ODEL O PTIMIZATION VIA E XPECTATION -M AXIMIZATION
It is hard to directly optimize ℓ SCGM ( D train ; θ , ϕ ) , because exact posterior inference is intractable (due to an exponential searching space).
To solve it, we resort to variational methods, and introduce an approximated posterior q ( z i | v i , y i ) , where y i is included so that the inference is also conditioned on superclass.
Then, a lower-bound of Eq.
(1) is derived as (the details are in Appendix A.2)
[Equation]
which can be maximized by alternately inferring the posterior q ( z i | v i , y i ) and solving the model parameters { θ , ϕ } through an Expectation-Maximization (EM) algorithm.
E-step.
This step is to infer q ( z i | v i , y i ) when fixing model parameters.
A straightforward way is to apply k -means on the embeddings v i , similar to (Caron et al., 2018).
However, as discussed in (Asano et al., 2020), alternately applying k -means and learning embeddings without any constraints will lead to a degenerate solution, i.e. , all samples are assigned to a single (arbitrary) subclass.
Also, k -means is unaware of the superclass
y i as it doesn’t optimize Eq.
To address this problem, inspired by (Asano et al., 2020), let Q z i ,i =
q ( z i | v i , y i ) 1 n and P z i ,i = p ϕ
( y i |
z i ) p θ , ϕ
( z i | v i ) be two r × n matrices, we enforce an equal partition of subclasses by constraining Q to belong to the transportation polytope (Appendix A.3), and rewrite Eq.
(5) as (the derivation is in Appendix A.4)
[Equation]
where Q
[Equation]
where H ( Q ) = − (cid:80) ij Q ij log Q ij is the entropy function, 1 n is the vector of ones in dimension n , and λ ≥ 1 controls the regularization (it also further relaxes the lower-bound in Eq.
The problem in Eq.
(6) is a regularized instance of the optimal transport problem, whose minimizer can be written as Q ∗ =
Diag ( u ) P λ Diag ( v ) , where u ∈ R r and v ∈ R n are renormalization vectors.
These vectors can be efficiently solved by the iterative Sinkhorn-Knopp algorithm (Cuturi, 2013).
Once Q ∗ is found, rounding its values and using discrete codes adds more algorithmic stability.
M-step.
Fixing q
( z i | v i , y i ) , the model parameters { θ , ϕ } can be efficiently solved by stochastic gradient descent (SGD).
Combining Eq.
(4) and Eq.
(5), the loss to minimize is (Appendix A.5) ℓ ϕ , θ =
[Equation]
It is noteworthy that the two terms in the bracket have similar form to InfoNCE (Oord et al., 2018).
Therefore, Eq.
(7) can be interpreted from a contrastive perspective, where the positive pairs in both terms are specified by q ( z i | v i , y i ) from E-step, i.e. , ( z i , y i ) and ( z i , v i ) that induce the maximal q ( z i | v i , y i ) , and the variances σ 2 and ¯ σ 2 resemble temperature scaling.
Thus, the M-step seeks to push the embedding v i to its closest subclass µ z i , which should meanwhile be close to its superclass ¯ µ y i , in contrast to the distances of the negative pairs.
In the second term, π iz i (Eq.
(2)) serves as a weight to select µ z i that is close to ¯ µ y i .
As such, our problem can be seen as a generalized InfoNCE for subclass- and superclass-level contrastive learning.
In practice, we found it is beneficial to integrate Eq.
(7) with cross-entropy loss ℓ CE (on superclass)
[Equation]
where ℓ CE uses superclass means { ¯ µ j } cj =1 as the classification weights, γ is a trade-off parameter.
The detailed algorithm for training SCGM can be found in Algorithm 1 in Appendix A.6.
E XPERIMENTS
In this section, we first evaluate SCGM on benchmark datasets and compare it with SOTA methods.
Then we evaluate SCGM for a model personalization task on a real-life medical dataset.
The table below summarizes the benchmark datasets: (1) BREEDS (Santurkar et al., 2020) includes four datasets { Living17, Nonliving26, Entity13, Entity30 } derived from ImageNet with class hierarchy calibrated so that classes on the same level are of similar visual granularity ( i.e. , the granularity of images for distinguishing classes); (2) CIFAR-100 (Krizhevsky, 2009); and (3) tiered ImageNet (Ren et al., 2018), which has 34 superclasses as per the ImageNet hierarchy, and was divided into 20/6/8 splits for (disjoint) train/val/test sets.
For BREEDS and CIFAR-100, the val set is 10% of the train set.
These datasets have been used in (Bukchin et al., 2021).
[Table]
4. E XPERIMENTAL S ETUP
Baselines.
We compare SCGM with the most relevant SOTA models on embedding learning, including self-supervised models: (1) MoCo-v2 (Chen et al., 2020b) trained on each of the above dataset; (2) MoCo-v2-ImageNet pretrained on ImageNet by Chen et al.
(2020b); (3) SwAV-ImageNet is a pretrained model by (Caron et al., 2020), which takes advantage of cluster-level contrastive learning.
Note that pretrained models on full ImageNet saw more data during training than did SCGM.
ANCOR (Bukchin et al., 2021), as introduced in Sec.1 and 2; (5) GEORGE (Sohoni et al., 2020), a three-step framework that uses a separate clustering step to pseudo-fine-label the dataset; and (6) SeLa (Asano et al., 2020), an unsupervised algorithm for joint clustering and embedding.
The detailed setup of these models was deferred to Appendix C.1.
Similar to (Bukchin et al., 2021), we include natural baselines trained on superclasses: (7) “Coarse”, uses encoder f θ followed by a classifier C ; (8) “Coarse+”, uses f θ → E → C , which adds an embedder E .
Also, baselines trained on subclass labels represent performance upper-bounds: (9) “Fine”, uses f θ → C ; and (10) “Fine+”, uses f θ → E → C .
For our method, since it is a flexible framework, we tested it with a generic encoder, and a momentum-based encoder with superclasswise dictionaries similar to ANCOR (Bukchin et al., 2021).
We denote these two variants by SCGMG and SCGM-A, where the training loss ( i.e. , Eq.
(8)) of the latter adds an InfoNCE term with angular normalization.
The detailed implementation of SCGM-A can be found in Appendix C.2.
Implementation.
The encoder is ResNet-50 (He et al., 2016) for the 224 × 224 images in BREEDS, and ResNet-12 for the small resolution images in CIFAR-100 and tiered ImageNet, as is common in few-shot learning works.
The output dimension of these networks is 2048 and 640, respectively.
Similar to (Chen et al., 2020b), two types of embedder E were tested: (1) fc layer: d → e , and (2)
MLP: d → d → e , with e = 128 .
At test time, some models performed better with E dropped while some were better otherwise.
For each model, the better case was reported.
For training, we used cosine annealing with warm restarts schedule (Loshchilov & Hutter, 2017) with 20 epochs per cycle.
The batch size was 256 for BREEDS, 1024 for CIFAR-100, and 512 for tiered ImageNet.
The learning rate was 0.03 for BREEDs, and 0.12 for CIFAR-100 and tiered ImageNet.
The weight decay was 1 e − 4 .
All models were trained with 200 epochs.
The best training practices of (Tian et al., 2020) were also applied.
For SCGM, we set γ = 0 .
5 , σ 2 = 0 . 1 , and λ =
( λ follows (Asano et al., 2020)).
The number of latent variables r varies w.r.t.
different datasets, and was grid-searched from 100 to 500 with step size 50.
All hyperparameters were set according to validation sets.
4. E XPERIMENTAL R ESULTS
Following (Tian et al., 2020), we report the mean accuracy and the 95% confidence interval ofrandom episodes with 5-way/all-way k -shot, 15-query tests.
Unless otherwise stated, k = 1 .
Effects of more shots are in Appendix D.1.
For each episode, 5 augmented copies were created per support sample.
A logistic regression classifier was trained on the support set embedded by each model with
[Table]
[Table] [Table]
L 2 norm.
The model with the resulting classifier was used to classify the query samples.
The test classes of each episode were a random subset of all subclasses (or all of them for all-way tests).
Two cases were evaluated: (1) unseen subclasses of seen superclasses (on BREEDS, CIFAR-100); and (2) unseen subclasses of unseen superclasses (on tiered ImageNet).
On CIFAR-100 and tiered ImageNet,
ImageNet-pratrained models were excluded since pretrained ResNet-12 models are unavailable.
Evaluation case (1).
We first evaluate when test classes are unseen subclasses of the training superclasses.
Table 1 and 2 summarize the results.
For ANCOR, its variants when the embedder E is fc (with ‘- fc ’) or MLP (without ‘- fc ’) were included as either variant does not consistently outperform another.
The embedder setup of all models are in Appendix C.1.
From the tables, we have several observations.
First, models that cannot leverage superclasses, such as MoCo-v2 and SeLa, are inferior to Coarse(+).
Although full ImageNet-pretrained models used larger training datasets, they are unable to fill the gap of superclass supervision, compared with Coarse(+).
Second, the limitation of GEORGE indicates a separate clustering step may induce misleading pseudo-labels that degenerate embedding quality.
Third, ANCOR and SCGM consistently outperform Coarse(+), and SCGM significantly outperforms ANCOR(- fc ) in most all-way cases.
With a generic encoder, SCGM-G is suprior or competitive to ANCOR(- fc ).
Using momentum contrast, SCGM-A further gains on some datasets.
Note that 5-way has less room to improve than the all-way cases, because the random subclasses are more likely to be from different superclasses, which may degrade the evaluation to coarse classification.
The results suggest SCGM could perform better when more subclasses are from the same superclass.
To investigate it, we evaluated an “intra-class” case when all subclasses of a random superclass were sampled in each episode.
Table 3 indicates SCGM got more improvements in this challenging task, implying its embeddings distinguish subclasses better than other models
[Table]
Moreover, Table 4 compares the computational costs of ANCOR and SCGM-G, from which we can see training ANCOR requires much more resources and time than SCGM-G as it inherits
MoCo-v2 (Chen et al., 2020b)
[Table]
Also, its memory costs vary a lot
[Table]
[Figure]
by datasets since its dictionaries are superclass-wise whose size changes w.r
[Figure]
t. the number of superclasses.
In contrast, SCGM-G had lower and stable costs, with better performance, thus is more favorable in practical deployments.
Evaluation case (2).
Since tiered ImageNet has distinct train/val/test sets, it was used to evaluate the case when the testing subclasses is out of any training superclass.
All models were trained with the superclasses of the train set, and tested on the subclass labels of the test set.
Table 5 presents the results.
Another three-step method C2F-VSME
(Yang et al., 2021) was included as the same evaluation has been performed (but their code is unavailable).
First, from the smaller gaps between
Fine(+) and Coarse(+), this task is more difficult than the previous case.
Similar to Table 1, MoCov2 and SeLa are inferior to Coarse(+), underlining the benefit of superclass supervision, even when test subclasses are not their descendants.
Compared to GEORGE and C2F-VSME, ANCOR and
SCGM can better leverage superclasses, by using coherent learning frameworks.
Both SCGM-G and SCGM-A outperform ANCOR(- fc ) in most cases, demonstrating the effectiveness of explicit modeling of latent subclasses, in both evaluation case (1) and (2).
Visualization of embedding.
To understand how SCGM uncovers the subclass structures, we visualize a random batch of embeddings using tSNE (Maaten & Hinton, 2008).
Fig.
2 presents the visualization results of ANCOR and SCGM-G on Living17 dataset, where colors mark different superclasses, “+” markers are the learned subclass means by SCGM-G, and stars represent superclass means.
ANCOR’s embeddings resemble those of Coarse+ (Appendix D.5) as it distinguishes subclasses from an angular perspective, which induces intra-class variation that is suboptimal.
SCGM-G explicitly detects the boundaries of both super- and subclasses, properly positions Gaussian means, and associates subclasses with their corresponding superclasses, whereby establishing a hierarchical structure.
We evaluated SCGM-G’s detected subclasses using the subclass labels in the val set.
It got an NMI of 71.58% and purity accuracy of 74.79%, much higher than its unsupervised counterpart
SeLa (NMI 36.50%, purity 38.14%).
This validates SCGM’s effectiveness in leveraging superclass guidance for subclass inference.
Several other model’s visualization can be found in Appendix D.5.
Performance analysis.
We have evaluated SCGM in terms of the effects of more shots, the impacts of the number of latent variables r , the impacts of the variance σ 2 , and the convergence, which are in Appendix D.1, D.2, D.3, D.4.
4. C ASE S TUDY : P ERSONALIZED P REDICTION ON M EDICAL R ECORDS
Next, we evaluate SCGM on a real Dialysis-Event dataset collected by several hospitals in Japan.
This dataset consists of one-year medical records of 673 hemodialysis patients.
It contains 40 temporal features ( e.g. , blood flow, venous pressure, etc. )
that were monitored during each hemodialysis session (thrice/week).
Each session was annotated to indicate whether certain subtypes of events have occurred.
A subtype corresponds to certain unstable behaviors of blood pressure during the dialysis, as categorized by experts.
As introduced in Sec. 1, these events are risky and may develop complications, thus should be avoided.
Formally, our task is as follows.
Given x it : t + w − 1 ∈ R d × w , a length- w segment including the i -th patient’s records starting at time step t , where d = 40 is the dimensionality, the task outcome is a label y ∈ { 0 , 1 } indicating event occurs ( y = 1 ) or not ( y = 0 ), at time step t + w .
When y = 1 , a fine-grained label ˆ y ∈ { 1 , ..., s } is to be predicted to mark the subtype of the event.
[Figure]
Pretrained model ! ":$" !
%:$&"" ...... ! '
:'&$("" ! ":$% !
%:$&"% ...... ! '
:'&$("% ! ":$)(" !
%:$&")
(" ...... ! ':'&$(")(" ! ":$) !
%:$&") ...... ! ':'&$(") ......
New patients time ...
Support set for model personalization
Pretraining
[Figure]
Since annotating fine-grained labels is costly, training a model on them at a large scale is infeasible in practice.
To predict event subtypes, we use a model personalization framework as illustrated in
Fig. 3(a).
At the pre-training stage, only binary labels are used.
For every new patient, a short period of (a few) medical records are collected as the support set for the fine-grained labels w.r.t.
that specific patient.
It is used to train a personalized classifier based on the pre-trained model.
To evaluate how SCGM fits this paradigm, we randomly split the 673 patients into 50%/50% train/test set, and reserved 10% of the train set for validation.
For each testing patient, the first 50% records ( i.e. , half year) form the support set, and the remaining form the query set.
It is likely some event types didn’t appear in the support set will appear in the query set, which are out-ofdistribution (OOD) cases, thus is out of our scope.
Hence, we only selected patients without such cases, which constitute a major part ( > 65%) of the test set.
The dataset has two event subtypes, i.e. s = 2 .
The segments x ( i ) t : t + w − 1 were sampled with w = 5 , stride 1.
The encoder is a Dual-Channel
Combiner Network (DCCN), similar to (Che et al., 2016).
DCCN has two channels ( e.g. , RNNs and
MLP) for encoding temporal and static features, respectively.
The outputs of them are integrated by a classification head.
Its architecture can be found in Appendix C.3.
In this application, each patient has 28 static features such as demographic information and infrequent blood test results.
[Table]
Table 6 presents the average accuracy of all testing patients over 10 random train/test splits for the applicable methods.
As is consistent with Sec.
4.2, unsupervised method SeLa and three-step framework GEORGE cannot improve DCCN’s performance due to (1) ignorance of superclass supervision, and (2) a separate pseudolabeling step, respectively.
In contrast, SCGM outperforms the original DCCN significantly.
Fig.
3(b) compares different models by their distributions of personalized prediction accuracy of different testing patients in a random train/test split.
Each dot represents the mean accuracy of one patient’s personalized predictions in the query set (220 dots in total).
From the figure, SCGM has more dots close to the top ( e.g. , above 0.8) than other models, meaning more patients received fairly accurate predictions on the fine-grained labels.
Although current performance could be limited by the data size, the results demonstrate the flexibility of SCGM, and its potential in tackling the CGFS problem in real practice.
C ONCLUSION
In this paper, we proposed a new method, superclass-conditional Gaussian mixture model (SCGM), that underlies a general framework for tackling CGFS across domains.
SCGM imitates the generative process of samples from hierarchies of classes, and explicitly models unobserved subclasses by latent variables.
It dynamically computes a Gaussian mixture for every sample given its superclass, with a small overhead to an encoder.
We learned the model parameters by maximum likelihood estimation in a principled Expectation-Maximization framework, and also theoretically linked SCGM with contrastive learning.
The extensive experiments on various datasets demonstrated the effectiveness, efficiency, and flexibility of our proposed method.
E THICS S TATEMENT
In the case study in Sec. 4.
we evaluated our SCGM on datasets related to human subjects that are hemodialysis patients.
We clarify that we got the permission to perform the experiments on the dataset as described in Sec. 4.3.
There is no ethics issue nor harmful insights in our current experiments.
The dataset was properly used under the guidance of the data providers and domain experts.
Sensitive information about the patients was well protected.
The final description about the experiments was checked and approved by the data providers and relevant authorities.
R EPRODUCIBILITY S TATEMENT
The code of SCGM is available at https://github.com/nijingchao/SCGM for reproducibility study.
The benchmark datasets are large and cannot be upoloaded.
The repository includes an instruction on how to obtain the datasets and process them.
Relevant instructions were provided on how to use the code to train SCGM-G and SCGM-A, and evaluate their performance for CGFS task in 5-way and all-way cases (Table 1, 2, 5) and in the intra-class case (Table 3)
A A PPENDIX FOR D ETAILS OF D ERIVING SCGM A. D ERIVATION OF PARAMETERIZED PROBABILITIES
In this section, we provide the details of the probabilities in Eq.
First, using Bayesian rules and marginal property, we can derive
[Equation]
Since µ z i and ¯ µ y i are surrogate representations of z i and y i in the embedding space, substituting p ( z i | y i ) and p ( v i |
z i ) with the equality in Eq.
(3), Eq.
(4) can be achieved
[Equation]
A. T HE LOWER - BOUND OF THE LIKELIHOOD FUNCTION
In this section
[Equation]
we provide the details of the lower-bound in Eq.
By introducing the approximated posterior q ( z i | v i , y i ) , the likelihood in Eq.
(1) becomes
[Equation]
) where the third step uses Jensen’s inequality.
This completes the derivation of Eq.
A. D ETAILS ON TRANSPORTATION POLYTOPE
Let a be a probability vector in the simplex ∆ r = { x ∈ R r + | x ⊤ 1 r = 1 } and b be a probability vector in the simplex ∆ n = { x ∈ R n + | x ⊤ 1 n = 1 } , where 1 r and 1 n are the vectors of ones in dimension r and n , respectively.
The transportation polytope of a and b , i.e. , the polyhedral set of r × n matrices, is defined as (Cuturi, 2013)
[Equation]
which means Q is a set of non-negative r × n matrices with row and column sums a and b , respectively.
From a probabilistic perspective, any matrix Q ∈ Q represents a joint probability of two multinomial random variables X and Y that take values in { 1 , ..., r } and { 1 , ..., n } with marginal distributions a and b , respectively.
The transportation polytope defines a space of transport matrix Q for mapping a to b .
Given a cost matrix M ∈ R r × n + , the cost of the mapping can be quantified as Tr ( Q ⊤ M ) , and the problem
[Equation]
(12) is called an optimal transport (OT) problem between a and b , given cost M .
Since the traditional algorithms to solve the OT problem scale inefficiently to large datasets, (Cuturi, 2013) introduced a fast Sinkhorn-Knopp algorithm, which amounts to solve a regularized problem
[Equation]
(13) where H ( Q ) = − (cid:80) ij Q ij log Q ij is the entropy function, and λ is a trade-off parameter.
If λ is large, optimizing Eq.
(13) approximates optimizing Eq.
In our case of Eq.
(6), M = − log P , a = 1 r 1 r , and b = 1 n 1 n .
In addition, similar to (Asano et al., 2020)
, Q z i ,i =
q
( z i | v i , y i ) 1 n can be considered as a joint probability of z i and the i -th sample (note the condition ( v i , y i ) corresponds to the incidence of the i -th sample) with a uniform prior 1 n .
For the details of the Sinkhorn-Knopp algorithm, we refer the readers to (Cuturi, 2013).
It is noteworthy that our optimization problem in Eq.
(6) in the E-step is derived from our likelihood function in Eq.
(5), which turns out to be a form of the regularized OT problem in Eq.
(13), rather than directly formulated using Eq.
The derivation of the optimization problem in Eq.
(6) can be found in next section (Appendix A.4).
A. T HE OPTIMIZATION PROBLEM IN E- STEP
In this section, we elaborate the optimization problem for inferring the posterior q ( z i | v i , y i ) in
Eq.
Substituting q
( z i | v i , y i ) with n Q z
i ,i , and substituting p ϕ
( y i |
z i ) p θ , ϕ
( z i | v i ) with P
z i ,i , the third step in Eq.
(10) can be rewritten by
[Equation]
where Tr ( · ) is the trace function, H ( · ) is the entropy function, the last constant term is because
[Equation]
Q
[Equation]
,i = 1 , and the inequality stems from λ ≥ 1 , which is introduced to control the strength of the regularization (Cuturi, 2013).
This is the objective function in Eq.
(6), where the constant log n is omitted because it has no effects on the optimization.
The right hand side of Eq.
(14) is a valid lower bound of the likelihood function in Eq.
(5) according to Eq.
Therefore, optimizing it ( i.e. , the objective function in Eq.
(6)) provides a reasonably approximated solution to the likelihood function in Eq.
In addition, the transport polytope Q in
Eq.
(6) is introduced as an equal partition constraint on subclasses (Asano et al., 2020), so that the optimization problem is feasible.
It is noteworthy that although subclasses have the equal partition constraint, superclasses can have different sizes since they can have different numbers of subclasses.
A. T HE OPTIMIZATION PROBLEM IN M- STEP
In this section, we derive the optimization problem for learning model parameters in Eq.
Introducing the probabilities specification in Eq.
(4) to the negative of the lower-bounded likelihood in
Eq.
(5), and removing terms that are irrelevant to model parameters { θ , ϕ } , we have
[Equation]
Considering isotropic Gaussian with ¯Σ y
i
= I ¯ σ 2 , we have
[Equation]
where the second equivalence is because we apply L 2 norm to µ z i and ¯ µ y i , respectively.
Similarly, considering Σ z i = I σ 2 , and applying L 2 norm to v i , we have
[Equation]
Substituting the corresponding terms in Eq.
(15) with Eq.
(16) and Eq.
(17), and removing the constants exp (1 / ¯ σ 2 ) and exp (1 /σ 2 ) in the fractions, we obtain the loss function in Eq.
In
Eq.
(7), the prior p ( y i ) can be set as uniform, i.e. , p ( y i )
= 1 c .
If using a prior from Dirichlet distribution, p ( y i ) can be instantiated by estimating the parameters α (of the Dirichlet distribution) by the ratio of different superclass labels in D train , the training dataset.
Also, π iz i in the second term of Eq.
(7) can be calculated using Eq.
A. T HE TRAINING ALGORITHM OF SCGM
The training algorithm of SCGM is summarized in Algorithm 1.
Input: encoder f θ dataset
Output:
Initialization c
// initialize Gaussian means ϕ
{{ µ j } rj =1 , { ¯ µ j } cj i ← 1 to MaxEpoch do /* E-step */ for
Dataloader
D train ) do / * load a minibatch
) */ v = f θ (
) // get embeddings
end
// update model parameters end 14 end
B A PPENDIX FOR F URTHER D ISCUSSION B. D ISCUSSION ON THE EQUAL PARTITION CONSTRAINT
The equal partition constraint on subclasses in Eq.
(6) theoretically assumes subclasses are of equal or similar sizes, through using Q1 n = 1 r 1 r (i.e., a = 1 r 1 r in Eq.
as a prior distribution of the subclass sizes (superclasses can have different sizes because they can have different numbers of subclasses).
It is possible to replace 1 r 1 r by other prior distributions to reflect an uneven distribution of the subclasses sizes.
In this work, we used a uniform prior because we don’t assume such prior knowledge (since the subclasses are unobserved during training).
This prior is inspired by (Asano et al., 2020).
If we don’t have such a prior, the searching space of subclasses will be exponential and the optimization is intractable.
To make the optimization feasible, we included this constraint to help reduce the searching space.
In this way, we found the solution is efficient and effective to our
CGFS problem, which does not necessitate an exact identification of every subclass.
Compared to other baseline methods, the performance improvements indicate that reasonable boundaries between subclasses can be learned by SCGM with this constraint for model adaptation, though the boundaries may not be ideal.
However, it is worth to note that this uniform prior may not always be reasonable in different applications.
For example, if identifying rare subclass is important in an application (Sohoni et al., 2020), a more reasonable prior may be considered.
In our experiments, the evaluations on the benchmark datasets (which were used for a consistent setup with the existing works (Bukchin et al., 2021)) do not reflect the case of substantially skew distributions of subclass sizes.
We leave the extension of the proposed method to this scenario in our future work.
B. D ISCUSSION ON THE EVALUATION CASES IN S ECTION 4.
In Section 4.2, we evaluated two fundamental cases of (1) unseen subclasses of seen superclasses; and (2) unseen subclasses of unseen superclasses.
Another case that considers the mixture of the subclasses of seen and unseen superclasses is worth discussion because it is a practical case.
As discussed in (Xian et al., 2018; Oreshkin et al., 2020), a mixture of samples from seen and unseen classes during testing may result in an imbalance of prediction performance between seen and unseen classes, and the inductive bias may lead to misclassification of unseen classes towards the seen classes.
It is noteworthy that (Xian et al., 2018; Oreshkin et al., 2020) focus on a generalized zero-shot learning problem, which is not for cross-granularity adaptation and don’t consider class hierarchy.
Thus their task is remarkably different from our CGFS task.
Despite the difference, we are inspired to be aware of the challenge that there could be a similar imbalance in the prediction of the subclasses of seen and unseen superclasses if they are mixed.
This could be an open challenge in the CGFS task which is rarely discussed in the existing works.
To alleviate this problem, some specific technical extensions may be designed, as discussed in (Xian et al., 2018;
Oreshkin et al., 2020), which is out of our scope.
We leave this scenario in our future work.
C A PPENDIX FOR I MPLEMENTATION D ETAILS C. T HE SETUP OF THE COMPARED MODELS
For MoCo-v2 and MoCo-v2-ImageNet, following (Chen et al., 2020b), we set its queue size 65536, momentum 0.999 (0.99 for CIFAR-100), temperature 0.2, embedder E as MLP.
For SwAVImageNet, following (Caron et al., 2020), its number of prototypes was 3000, embedder E as MLP.
For ANCOR, following (Bukchin et al., 2021), we set its queue size 65536, momentum 0.999 (0. for CIFAR-100), temperature 0.2, embedder E as fc and MLP, with the multi-class queue, and angular norm on.
For GEORGE (Sohoni et al., 2020), we set its robust learning rate 1 e − 4 , E as fc , (dim 16, 10 neighbors, 0 distance) for its dimension reduction step, the number of clusters per superclasses was grid-searched within { 5, 10, 15, 20 } .
For SeLa, following (Asano et al., 2020), we set its λ = 25 , E as fc , and number of clusters was grid-searched from 100 to 500 with step size 50.
For Coarse+ and Fine+, we set E as MLP.
For our SCGM, we set γ = 0 .
5 , σ 2 = 0 . 1 , and λ =
( λ follows (Asano et al., 2020)), E as fc , the number of latent variables r was grid-searched fromto 500 with step size 50.
Remarks on embedder.
For the compared methods, adding an embedder follows the practice of the existing works such as MoCo-v2 (Chen et al., 2020b) and SimCLR
(Chen et al., 2020a), which was also used in the different compared methods in (Bukchin et al., 2021).
In these works, adding certain types of embedder (e.g., MLP) is empirically found helpful for improving performance.
We have the same observation in our experiments, and think the effectiveness could come from (1) more capacity; and (2) better generalization in some cases.
The latter is because after training, the embedder may be specific to the pre-training tasks, which enables more generalizable representations to be learned by the backbone networks.
Thus dropping the embedder at testing stage may help alleviate overfitting to some extent.
For comprehensiveness, we evaluated both of the cases when an embedder is added or not in our experiments, as described in Section 4.1.
C. T HE ARCHITECTURE OF SCGM-A
[Figure]
Fig.
4 illustrates the architecture of SCGM-A, which was implemented on a generic encoder ( e.g. ,
ResNet-50) with momentum encoders.
The query v and positive key k
+ are computed from two random augmentations of the input images through the encoder and embedder, and their momentum updates.
There is a superclass-wise dictionary maintains c queues for c different superclasses so that superclass-wise negative keys k − can be generated for within-superclass contrastive learning, which facilitates preserving intra-class variation.
At the SCGM classification head, v , k + and k − are applied with an angular normalization, followed by the InfoNCE loss, where the angular normalization is defined by (Bukchin et al., 2021)
[Equation]
which enables contrasts from an angular perspective, and induces better synergy with cross-entropy loss.
The top part of the SCGM classification head are similar to its counterpart using a generic encoder, where v interacts with superclass means { ¯ µ j } cj =1 and subclass means { µ j } rj =1 for computing our
SCGM loss ℓ θ , ϕ and a cross-entropy loss ℓ CE as in Eq.
Thus the loss for training SCGM-A is
[Equation]
where the model parameters are learned by our proposed EM-based algorithm that alternates between the SGD backpropagation (M-step) for computing model parameters { θ , ϕ } and the posterior inference (E-step) for computing subclass distributions.
C. T HE ARCHITECTURE OF DCCN
[Figure]
Fig.
5 illustrates the architecture of the Dual-Channel Combiner Network (DCCN).
DCCN is effective to process heterogeneous medical records data that usually consists of static profiles and time series ( i.e. , records) (Che et al., 2016).
It has a static channel realized by MLPs to encode static features ( e.g. , demographic information, infrequent blood test results, etc. ) and a temporal channel realized by RNNs ( e.g. , LSTM) to encode temporal feature ( e.g. , blood flow, venous pressure, etc. ).
The hidden representations output by the two channels are concatenated (or using certain pooling method) and further projected to a compact embedding that is used for prediction by the classification head, i.e. , the combination layer in Fig.
5, which is realized by MLPs.
Attention layer may be added before the combination layer for weighted combination of the hidden representations from different channels and different time steps.
D A PPENDIX FOR M ORE E XPERIMENTAL R ESULTS D. E FFECTS OF MORE SHOTS
[Table]
Table 7 and 8 summarize the performance of SCGM-G and SCGM-A, with the strongest baseline
ANCOR and ANCOR- fc , together with the coarse baseline Coarse+, on Living17 dataset.
From
[Table]
Table 7, we observe SCGM-A continuously improves performance as more shots are added, consistently outperforms other models, which validate the effectiveness of SCGM framework with varying number of shots.
SCGM-G’s performance becomes comparable to ANCOR(- fc ) as more shots are added, due to its simpler architecture, which however is more computationally efficient.
Thus it provides a good balance between performance and efficiency.
From Table
[Table]
we observe ANCOR and
SCGM perform similarly.
This is because the task is close to the regular superclass classification
[Table]
and its performance limit can be easily reached by adding a few more shots.
This can be seen from
Coarse+’s results, which become close to other models when there are 20 shots in the support set.
D. I MPACTS OF THE NUMBER OF SUBCLASSES
[Table]
Table 9 presents the results of SCGM-G on Living17 by varying the number of latent variables r from 50 to 500.
As can be seen, the performance of SCGM-G is generally stable and it consistently outperforms Coarse(+)
(Table 1)
w.r
[Table]
t. different r .
By looking into the details, we observe too small ( e.g. , 50) or too big ( e.g. , 500) r may degenerate the performance in the all-way case, which may be caused by slight underfitting and overfitting, respectively.
In the 5-way case, small r does not impact performance obviously, this is because the 5-way case is close to evaluate coarse classification, which does not need exact inference of subclasses.
Thus a small number of r is fine (which may not distinguish all possible subclasses).
D. I MPACTS OF THE VARIANCE
[Table]
Table 10 presents the results of SCGM-G on Living17 by varying the variance σ 2 of subclasses, which resembles the temperature scaling in Eq.
As described in Sec. 3.1, in our experiments, we set ¯ σ 2 = 1 and tune σ 2 to adjust the relativity between super- and subclasses, which we found is empirically effective.
From Table 10, we observe σ 2 is better to be selected from 0.1 to 0.2, and a too small ( e.g. , 0.05) or too big ( e.g. , 0.3) value may degenerate the performance in the all-way case.
This observation is similar to the evaluation of the temperature scaling parameter in MoCo-v (Chen et al., 2020b).
In the 5-way case, the performance is relatively stable w.r.t.
different values of σ 2 .
Similar to the observation in Appendix D.2, this is because the 5-way case is close to evaluate coarse classification, which is less impacted by the learning of subclasses.
D. C ONVERGENCE ANALYSIS
Fig.
6 presents the training loss ℓ ( D train ; θ , ϕ ) in Eq.
(8) of SCGM-G, and ℓ ( D train ; θ , ϕ ) in Eq.
(19) of SCGM-A, during the training process on the four BREEDS datasets.
The training alternates between E-step and M-step, where E-step runs every 5 epochs.
As can be seen, although there are some periodic fluctuations, the loss function values generally decrease and converge to a small value across different datasets.
The periodic fluctuations are not synchronized with the alternate E-step and M-step since its period is not 5 epoch as can be observed.
In fact, the fluctuations come from the
[Figure]
cosine annealing with warm restarts schedule for the learning rate (as discussed in Sec. 4.1), which has 20 epochs per cycle.
This is consistent with Fig.
In practice, we observed both E-step and
M-step can decrease the loss function values since both steps are theoretically optimizing the same loss function.
There is a rounding step (as discussed in Sec. 3.2) for generating discrete code from the posterior, which follows the E-step.
We found this rounding step is beneficial, since without it, too small values may be generated in the posterior inference, which is unstable in computation.
Generally, the current setup provides a stable training framework and we don’t need to interfere in the program during the training process on all of our experimental datasets.
D. V ISUALIZATION OF EMBEDDING
[Equation]
[Figure] [Equation] [Figure] In addition to the visualization results of ANCOR and SCGM-G provided in Fig. 2, Fig.
7 presents the tSNE visualization of the embeddings of several most relevant methods in comparison using Living17 dataset, including Coarse+, MoCo-v2, ANCOR- fc and SCGM-A.
In the figure, colors mark different superclasses, “+” markers are the learned subclass means by SCGM-G, and stars represent superclass means.
From Fig.
7(a)(c), we can observe the embeddings of both ANCOR and ANCORfc resemble Coarse+.
Although ANCOR- fc slightly distinguish several small groups of embeddings in some superclasses, their intra-class variation is suboptimal.
From Fig. 7(b), MoCo-v2 has the ability to detect superclasses to some extent, but cannot fill the gap of superclass supervision, by comparing with Coarse+, in this dataset.
The reason may be the Living17 dataset is not sufficiently large and comprehensive for MoCo-v2 to learn embeddings that are comparable to supervised methods.
In contrast to them, SCGM-A (and SCGM-G in Fig. 2) explicitly detects subclasses, with proper position of subclass means and superclass means.
It also associates every subclass to its corresponding superclass, which establishes a hierarchical class structure.
This explains the effectiveness of SCGM in cross-granularity learning tasks.