C LASS N ORMALIZATION FOR (C ONTINUAL )? G ENERALIZED Z ERO -S HOT L EARNING
A BSTRACT
Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime.
However, in the zero-shot learning (ZSL) world, these ideas have received only marginal attention.
This work studies normalization in ZSL scenario from both theoretical and practical perspectives.
First, we give a theoretical explanation to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass.
Next, we demonstrate that they are insufﬁcient to normalize a deep ZSL model and propose Class Normalization (CN) : a normalization scheme, which alleviates this issue both provably and in practice.
Third, we show that ZSL models typically have more irregular loss surface compared to traditional classiﬁers and that the proposed method partially remedies this problem.
Then, we test our approach on 4 standard ZSL datasets and outperform sophisticated modern SotA with a simple MLP optimized without any bells and whistles and having ≈
50 times faster training speed.
Finally, we generalize ZSL to a broader problem — continual ZSL, and introduce some principled metrics and rigorous baselines for this new setup.
The source code is available at https://github.com/universome/class-norm.
I NTRODUCTION
Zero-shot learning (ZSL) aims to understand new concepts based on their semantic descriptions instead of numerous input-output learning pairs.
It is a key element of human intelligence and our best machines still struggle to master it (Ferrari & Zisserman, 2008; Lampert et al., 2009; Xian et al., 2018a).
Normalization techniques like batch/layer/group normalization (Ioffe & Szegedy, 2015; Ba et al., 2016; Wu & He, 2018) are now a common and important practice of modern deep learning.
But despite their popularity in traditional supervised training, not much is explored in the realm of zero-shot learning, which motivated us to study and investigate normalization in ZSL models.
We start by analyzing two ubiquitous tricks employed by ZSL and representation learning practitioners: normalize+scale (NS) and attributes normalization (AN) (Bell et al., 2016;
Zhang et al., 2019; Guo et al., 2020; Chaudhry et al., 2019).
Their dramatic inﬂuence on performance can be observed from
Table 1.
When these two tricks are employed, a vanilla MLP model, described in Sec 3.1, can outperform some recent sophisticated ZSL methods.
Normalize+scale (NS) changes logits computation from usual dot-product to scaled cosine similarity:
[Equation]
where z is an image feature, p c is c -th class prototype and γ is a hyperparameter, usually picked from [5 , 10] interval (Li et al., 2019; Zhang et al., 2019).
Scaling by γ is equivalent to setting a high temperature of γ 2 in softmax.
In Sec. 3.2, we theoretically justify the need for this trick and explain why the value of γ must be so high.
When NS and AN are integrated into a basic ZSL model, its performance is boosted up to a level of some sophisticated SotA methods and additionally using CN allows to outperform them.
± NS and ±
AN denote if normalize+scale or attributes normalization are being used.
Bold/normal blue font denote best/second-best results.
Extended results are in Table 2, 5 and 8.
[Table]
Attributes Normalization (AN) technique simply divides class attributes by their L 2 norms:
[Equation]
While this may look inconsiderable, it is surprising to see it being preferred in practice (Li et al., 2019;
Narayan et al., 2020; Chaudhry et al., 2019) instead of the traditional zero-mean and unit-variance data standardization (Glorot & Bengio, 2010).
In Sec 3, we show that it helps in normalizing signal’s variance in and ablate its importance in Table 1 and Appx D.
These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear (see Figure 1), but they fail when we use a multi-layer architecture.
To remedy this issue, we introduce Class Normalization (CN) : a novel normalization scheme, which is based on a different initialization and a class-wise standardization transform.
Modern ZSL methods either utilize sophisticated architectural design like training generative models (Narayan et al., 2020; Felix et al., 2018)
or use heavy optimization schemes like episode-based training (Yu et al., 2020; Li et al., 2019).
In contrast, we show that simply adding Class Normalization on top of a vanilla MLP is enough to set new state-of-the-art results on several standard ZSL datasets (see Table 2).
Moreover, since it is optimized with plain gradient descent without any bells and whistles, training time for us takes 50-100 times less and runs in about 1 minute.
We also demonstrate that many ZSL models tend to have more irregular loss surface compared to traditional supervised learning classiﬁers and apply the results of Santurkar et al.
(2018) to show that our CN partially remedies the issue.
We discuss and empirically validate this in Sec 3.5 and Appx F.
Apart from the theoretical exposition and a new normalization scheme, we also propose a broader
ZSL setup: continual zero-shot learning (CZSL).
Continual learning (CL) is an ability to acquire new knowledge without forgetting (e.g. (Kirkpatrick et al., 2017)), which is scarcely investigated in ZSL.
We develop the ideas of lifelong learning with class attributes, originally proposed by Chaudhry et al.
(2019) and extended by Wei et al. (2020a), propose several principled metrics for it and test several classical CL methods in this new setup.
R ELATED WORK
Zero-shot learning .
Zero-shot learning (ZSL) aims at understand example of unseen classes from their language or semantic descriptions.
Earlier ZSL methods directly predict attribute conﬁdence from images to facilitate zero-shot recognition (e.g., Lampert et al.
; Farhadi et al.
Lampert et al.
(2013b)).
Recent ZSL methods for image classiﬁcation can be categorized into two groups: generative-based and embedding-based.
The main goal for generative-based approaches is to build a conditional generative model (e.g., GANs Goodfellow et al.
and VAEs (Kingma & Welling, 2014)) to synthesize visual generations conditioned on class descriptors (e.g., Xian et al.
(2018b); Zhu et al.
; Elhoseiny & Elfeki (2019); Guo et al.
; Guo et al.
Kumar Verma et al.
At test time, the trained generator is expected to produce synthetic/fake data of unseen classes given its semantic descriptor.
The fake data is then used to train a traditional classiﬁer or to perform a simple kNN-classiﬁcation on the test images.
Embedding-based approaches learn a mapping that projects semantic attributes and images into a common space where the distance between a class projection and the corresponding images is minimized (e.g, Romera-Paredes & Torr (2015); Frome et al.
; Lei Ba et al. (2015); Akata et al. (2016a); Zhang et al.
; Akata et al.
(2015; 2016b)).
One question that arises is what space to choose to project the attributes or images to.
Previous works projected images to the semantic space (Elhoseiny et al., 2013; Frome et al., 2013; Lampert et al., 2013a) or some common space (Zhang & Saligrama, 2015; Akata et al., 2015), but our approach follows the idea of Zhang et al.
; Li et al.
(2019) that shows that projecting attributes to the image space reduces the bias towards seen data.
Normalize+scale and attributes normalization .
It was observed both in ZSL (e.g., Li et al.
Zhang et al. ; Bell et al. and representation learning (e.g., Sohn (2016); Guo et al.
Ye et al. ﬁelds that normalize+scale (i.e. (1)) and attributes normalization (i.e. (2)) tend to signiﬁcantly improve the performance of a learning system.
In the literature, these two techniques lack rigorous motivation and are usually introduced as practical heuristics that aid training (Changpinyo et al., 2017; Zhang et al., 2019; 2021).
One of the earliest works that employ attributes normalization was done by (Norouzi et al., 2013), and in (Changpinyo et al., 2016a)
authors also ablate its importance.
The main consumers of normalize+scale trick had been similarity learning algorithms, which employ it to reﬁne the distance metric between the representations (Bellet et al., 2013; Guo et al., 2020; Shi et al., 2020).
Luo et al.
(2018) proposed to use cosine similarity in the ﬁnal output projection matrix as a normalization procedure, but didn’t incorporate any analysis on how it affects the variance.
They also didn’t use the scaling which our experiments in Tableshow to be crucial.
Gidaris & Komodakis (2018) demonstrated a greatly superior performance of an NS-enriched model compared to a dot-product based one in their setup where the classifying matrix is constructed dynamically.
Li et al.
(2019) motivated their usage of NS by variance reduction, but didn’t elaborate on this in their subsequent analysis.
Chen et al.
(2020) related the use of the normalized temperature-scaled cross entropy loss (NT-Xent) to different weighting of negative examples in contrastive learning framework.
Overall, to the best of our knowledge, there is no precise understanding of the inﬂuence of these two tricks on the optimization process and beneﬁts they provide.
Initialization schemes .
In the seminal work, Xavier’s init Glorot & Bengio (2010), the authors showed how to preserve the variance during a forward pass.
He et al.
(2015) applied a similar analysis but taking ReLU nonlinearities into account.
There is also a growing interest in two-step Jia et al.
(2014), data-dependent Krähenbühl et al.
(2015), and orthogonal Hu et al.
(2020) initialization schemes.
However, the importance of a good initialization for multi-modal embedding functions like attribute embedding is less studied and not well understood.
We propose a proper initialization scheme based on a different initialization variance and a dynamic standardization layer.
Our variance analyzis is similar in nature to Chang et al.
(2020) since attribute embedder may be seen as a hypernetwork (Ha et al., 2016) that outputs a linear classiﬁer.
But the exact embedding transformation is different from a hypernetwork since it has matrix-wise input and in our derivations we have to use more loose assumptions about attributes distribution (see Sec 3 and Appx H).
Normalization techniques .
A closely related branch of research is the development of normalization layers for deep neural networks (Ioffe & Szegedy, 2015) since they also inﬂuence a signal’s variance.
BatchNorm, being the most popular one, normalizes the location and scale of activations.
It is applied in a batch-wise fashion and that’s why its performance is highly dependent on batch size (Singh &
Krishnan, 2020).
That’s why several normalization techniques have been proposed to eliminate the batch-size dependecy (Wu & He, 2018; Ba et al., 2016; Singh & Krishnan, 2020).
The proposed class normalization is very similar to a standardization procedure which underlies BatchNorm, but it is applied class-wise in the attribute embedder.
This also makes it independent from the batch size.
Continual zero-shot learning .
We introduce continual zero-shot learning: a new benchmark for
ZSL agents that is inspired by continual learning literature (e.g., Kirkpatrick et al.
It is a development of the scenario proposed in Chaudhry et al.
, but authors there focused on ZSL performance only a single task ahead, while in our case we consider the performance on all seen (previous tasks) and all unseen data (future tasks).
This also contrasts our work to the very recent work by Wei et al.
(2020b), where a sequence of seen class splits of existing ZSL benchmsks is trained and the zero-shot performance is reported for every task individually at test time.
In contrast, for our setup, the label space is not restricted and covers the spectrum of all previous tasks (seen tasks so far), and future tasks (unseen tasks so far).
Due to this difference, we need to introduce a set of new metrics and benchmarks to measure this continual generalized ZSL skill over time.
From the lifelong learning perspective, the idea to consider all the processed data to evaluate the model is not new and was previously explored by Elhoseiny et al.
(2018); van de Ven & Tolias (2019).
It lies in contrast with the common practice of providing task identity at test time, which limits the prediction space for a model, making the problem easier (Kirkpatrick et al., 2017; Aljundi et al., 2017).
In Isele et al.
(2016); Lopez-Paz & Ranzato (2017) authors motivate the use of task descriptors for zero-shot knowledge transfer, but in our work we consider class descriptors instead.
We deﬁned CZSL as a continual version of generalized-ZSL which allows us to naturally extend all the existing ZSL metrics
Xian et al.
(2018a)
; Chao et al.
(2016) to our new continual setup.
N ORMALIZATION IN Z ERO -S HOT L EARNING
The goal of a good normalization scheme is to preserve a signal inside a model from severe ﬂuctuations and to keep it in the regions that are appropriate for subsequent transformations.
For example, for
ReLU activations, we aim that its input activations to be zero-centered and not scaled too much: otherwise, we risk to ﬁnd ourselves in all-zero or all-linear activation regimes, disrupting the model performance.
For logits , we aim them to have a close-to-unit variance since too small variance leads to poor gradients of the subsequent cross-entropy loss and too large variance is an indicator of poor scaling of the preceding weight matrix.
For linear layers, we aim their inputs to be zero-centered: in the opposite case, they would produce too biased outputs, which is undesirable.
In traditional supervised learning, we have different normalization and initialization techniques to control the signal ﬂow.
In zero-shot learning (ZSL), however, the set of tools is extremely limited.
In this section, we justify the popularity of Normalize+Scale (NS) and Attributes Normalization (AN) techniques by demonstrating that they just retain a signal variance.
We demonstrate that they are not enough to normalize a deep ZSL model and propose class normalization to regulate a signal inside a deep ZSL model.
We empirically evaluate our study in Sec. 5 and appendices A, B, D and F.
3. N OTATION
A ZSL setup considers access to datasets of seen and unseen images with the corresponding labels
D s = { x si , y si } N s i =1 and D u
= { x ui , y ui } N u
i =1 respectively.
Each class c is described by its class attribute vector a c ∈ R d a .
All attribute vectors are partitioned into non-overlapping seen and unseen sets as well: A s = { a i } K s i
=1 and A u = { a i } K u i =1 .
Here N s , N u , K s , K u are number of seen images, unseen images, seen classes, and unseen classes respectively.
In modern ZSL, all images are usually transformed via some standard feature extractor E : x (cid:55)→ z ∈ R d z (Xian et al., 2018a).
Then, a typical ZSL method trains attribute embedder P θ : a c → p c ∈ R d z which projects class attributes a sc onto feature space R d z in such a way that it lies closer to exemplar features z s of its class c .
This is done by solving a classiﬁcation task, where logits are computed using formula (1).
In such a way at test time we are able to classify unseen images by projecting unseen attribute vectors a uc into the feature space and computing similarity with the provided features z u .
Attribute embedder P θ is usually a very simple neural network (Li et al., 2019); in many cases even linear (Romera-Paredes & Torr, 2015; Elhoseiny et al., 2013), so it is the training procedure and different regularization schemes that carry the load.
We will denote the ﬁnal projection matrix and the body of P θ as V and H ϕ respectively, i.e. P θ ( a c ) =
V H ϕ ( a c ) .
During training, it receives matrix of class attributes
A =
[ a 1 , ..., a K s ] of size K s ×
d a and outputs matrix W = P θ ( A ) of size
K s × d z .
Then W is used to compute class logits with a batch of image feature vectors z 1 , ..., z N s .
3. U NDERSTANDING NORMALIZE + SCALE TRICK
One of the most popular tricks in ZSL and deep learning is using the scaled cosine similarity instead of a simple dot product in logits computation (Li et al., 2019; Zhang et al., 2019;
Ye et al., 2020):
[Equation]
[Equation] where hyperparameter γ is usually picked from [5 , 10] interval.
Both using the cosine similarity and scaling it afterwards by a large value is critical to obtain good performance; see Appendix D. To our knowledge, it has not been clear why exactly it has such big inﬂuence and why the value of γ must be so large.
The following statement provides an answer to these questions.
Statement 1 (informal) .
Normalize+scale trick forces the variance for ˆ y c to be approximately :
[Equation]
where d z is the dimensionality of the feature space .
See Appendix A for the assumptions, derivation and the empirical study.
Formula (4) demonstrates two things:
When we use cosine similarity, the variance of ˆ y c becomes independent from the variance of W = P
θ ( A ) , leading to better stability.
If one uses Eq. (3) without scaling (i e. γ = 1 ), then the Var [ˆ y c ] will be extremely low (especially for large d z ) and our model will always output uniform distribution and the training would stale.
That’s why we need very large values for γ .
Usually, the optimal value of γ is found via a hyperparameter search (Li et al., 2019), but our formula suggests another strategy: one can obtain any desired variance ν =
Var [ˆ y c ] by setting γ to:
[Equation]
For example, for Var [ˆ y c ] =
ν
= 1 and d z = 2048 we obtain γ ≈ 6 . 78 , which falls right in the middle of [5 , 10] — a usual search region for γ used by ZSL and representation learning practitioners Li et al.
; Zhang et al.
; Guo et al.
The above consideration not only gives a theoretical understanding of the trick, which we believe is important on its own right, but also allows to speed up the search by either picking the predicted “optimal” value for γ or by searching in its vicinity.
3. U NDERSTANDING ATTRIBUTES NORMALIZATION TRICK
We showed in the previous subsection that “normalize+scale” trick makes the variances of ˆ y c independent from variance of weights, features and attributes.
This may create an impression that it does not matter how we initialize the weights — normalization would undo any ﬂuctuations.
However it is not true, because it is still important how the signal ﬂows under the hood, i.e. for an unnormalized and unscaled logit value ˜
y c
= z (cid:62) p c .
Another common trick in ZSL is the normalization of attribute vectors to a unit norm a c (cid:55)−→ a c (cid:107) a c (cid:107) 2 .
We provide some theoretical underpinnings of its importance.
Let’s ﬁrst consider a linear case for P θ , i.e. H ϕ is an identity, thus ˜ y
c
= z (cid:62) p c =
z (cid:62) V a c .
Then, the way we initialize V is crucial since Var [˜ y ]
c depends on it.
To derive an initialization scheme people use 3 strong assumptions for the inputs Glorot & Bengio (2010)
; He et al.
; Chang et al.
(2020): 1) they are zero-centered 2) independent from each other; and 3) have the covariance matrix of the form σ 2 I .
But in ZSL setting, we have two sources of inputs: image features z and class attributes a c .
And these assumptions are safe to assume only for z but not for a c , because they do not hold for the standard datasets (see Appendix H).
To account for this, we derive the variance Var [ˆ y c ] without relying on these assumptions for a c (see Appendix B):
[Equation]
From equation (6) one can see that after giving up invalid assumptions for a c , pre-logits variance
Var [˜ y c ] now became dependent on (cid:107) a c (cid:107) 2 , which is not captured by traditional Glorot & Bengio (2010) and He et al.
(2015) initialization schemes and thus leads to poor variance control.
Attributes normalization trick rectiﬁes this limitation, which is summarized in the following statement.
Statement 2 (informal) .
Attributes normalization trick leads to the same pre-logits variance as we have with Xavier fan-out initialization.
(see Appendix B for the formal statement and the derivation).
Xavier fan-out initialization selects such a scale for a linear layer that the variance of backward pass representations is preserved across the model (in the absence of non-linearities).
The fact that attributes normalization results in the scaling of P θ equivalent to Xavier fan-out scaling and not some other one is a coincidence and shows what underlying meaning this procedure has.
[Figure] 3. C LASS N ORMALIZATION
What happens when P θ is not linear?
Let h c
= H ϕ ( a c ) be the output of H ϕ .
The analysis of this case is equivalent to the previous one but with plugging in h c everywhere instead of a c .
This lead to:
[Equation]
As a result, to obtain Var [˜ y c ]
=
Var [ z i ] property, we need to initialize Var [ V ij ] the following way:
[Equation]
This makes the initialization dependent on the magnitude of (cid:107) h c (cid:107) instead of (cid:107) a c (cid:107) , so normalizing attributes to a unit norm would not be sufﬁcient to preserve the variance.
To initialize the weights of
V using this formula, a two-step data-dependent initialization is required: ﬁrst initializing H ϕ , then computing average (cid:107) h c (cid:107) 22 , and then initializing V .
However, this is not reliable since (cid:107) h c (cid:107) 22 changes on each iteration, so we propose a more elegant solution to standardize h c
[Equation]
As one can note, this is similar to BatchNorm standardization without the subsequent afﬁne transform, but we apply it class-wise on top of attribute embeddings h c .
We plug it in right before V , i.e.
P θ ( a c ) =
V S ( H ϕ ( a c )) .
This does not add any parameters and has imperceptible computational overhead.
At test time, we use statistics accumulated during training similar to batch norm.
Standardization (9) makes inputs to V have constant norm, which now makes it trivial to pick a proper value to initialize V :
[Equation]
We coin the simultaneous use of (9) and (10) class normalization and highlight its inﬂuence in the following statement.
See Fig.
3 for the model diagram, Fig.
1 for empirical study of its impact, and
Appendix C for the assumptions, proof and additional details.
Statement 3 (informal) .
Standardization procedure (9) together with the proper variance formula (10) , preserves the variance between z and ˜ y for a mutli-layer attribute embedder P θ .
3. I MPROVED SMOOTHNESS
We also analyze the loss surface smoothness for P θ .
There are many ways to measure this notion (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016;
Dinh et al., 2017; Skorokhodov & Burtsev,
, but following Santurkar et al.
(2018), we deﬁne it in a “per-layer” fashion via Lipschitzness:
[Equation]
where (cid:96) is the layer index and X (cid:96) − 1 is its input data matrix.
This deﬁnition is intuitive: larger gradient magnitudes indicate that the loss surface is prone to abrupt changes.
We demonstrate two things:
For each example in a batch, parameters of a ZSL attribute embedder receive K more updates than a typical non-ZSL classiﬁer, where K is the number of classes.
This suggests a hypothesis that it has larger overall gradient magnitude, hence a more irregular loss surface.
Our standardization procedure (9) makes the surface more smooth.
We demonstrate it by simply applying Theorem 4.4 from (Santurkar et al., 2018).
Due to the space constraints, we defer the exposition on this to Appendix F.
C ONTINUAL Z ERO -S HOT L EARNING 4. P ROBLEM FORMULATION
In continual learning (CL), a model is being trained on a sequence of tasks that arrive one by one.
Each task is deﬁned by a dataset D t = { x ti , y ti } N t
i =1 of size N t .
The goal of the model is to learn all the tasks sequentially in such a way that at each task t it has good performance both on the current task and all the previously observed ones.
In this section we develop the ideas of Chaudhry et al.
and formulate a Continual Zero-Shot Learning (CZSL) problem.
Like in CL, CZSL also assumes a sequence of tasks, but now each task is a generalized zero-shot learning problem.
This means that apart from D t we also receive a set of corresponding class descriptions A t for each task t .
In this way, traditional zero-shot learning can be seen as a special case of CZSL with just two tasks.
In Chaudhry et al.
(2019), authors evaluate their zero-shot models on each task individually, without considering the classiﬁcation space across tasks; looking only one step ahead, which gives a limited picture of the model’s quality.
Instead, we borrow ideas from Generalized ZSL (Chao et al., 2016; Xian et al., 2018a), and propose to measure the performance on all the seen and all the unseen data for each task.
More formally, for timestep t we have the datasets:
[Equation]
which are the datasets of all seen data (learned tasks), all unseen data (future tasks), seen class attributes, and unseen class attributes respectively.
For our proposed CZSL, the model at timestep t has access to only data D t and attributes A t , but its goal is to have good performance on all seen data D ≤ t and all unseen data D >t with the corresponding attributes sets A ≤ t and A >t . For T = 2 , this would be equivalent to traditional generalized zero-shot learning. But for T > 2 , it is a novel and a much more challenging problem.
4. P ROPOSED EVALUATION METRICS
Our metrics for CZSL use GZSL metrics under the hood and are based on generalized accuracy (GA) (Chao et al., 2016; Xian et al., 2018a).
“Traditional” seen (unseen) accuracy computation discards unseen (seen) classes from the prediction space, thus making the problem easier, since the model has fewer classes to be distracted with.
For generalized accuracy, we always consider the joint space of both seen and unseen and this is how GZSL-S and GZSL-U are constructed.
We use this notion to construct mean seen (mS), mean unseen (mU) and mean harmonic (mH) accuracies.
We do that just by measuring GZSL-S/GZSL-U/GZSL-H at each timestep, considering all the past data as seen and all the future data as unseen.
Another set of CZSL metrics are mean joint accuracy (mJA) which measures the performance across all the classes and mean area under seen/unseen curve (mAUC) which is an adaptation of AUSUC measure by Xian et al.
(2018a).
A more rigorous formulation of these metrics is presented in Appendix G.2.
Apart from them, we also employ a popular forgetting measure (Lopez-Paz & Ranzato, 2017).
E XPERIMENTS 5. ZSL EXPERIMENTS
Experiment details .
We use 4 standard datasets: SUN (Patterson et al., 2014), CUB (Welinder et al., 2010), AwA1 and AwA2 and seen/unseen splits from Xian et al.
(2018a).
They have 645/72, 150/50, 40/10 and 40/10 seen/unseen classes respectively with d a being equal to 102, 312, 85 and 85 respectively.
Following standard practice, we use ResNet101 image features (with d z = 2048 ) from Xian et al.
(2018a).
Our attribute embedder P θ is a vanilla 3-layer MLP augmented with standardization procedure 9 and corrected output matrix initialization 10.
For all the datasets, we train the model with Adam optimizer for 50 epochs and evaluate it at the end of training.
We also employ
NS and AN techniques with γ = 5 for NS.
Additional hyperparameters are reported in Appx D. To perform cross-validation, we ﬁrst allocate 10% of seen classes for a validation unseen data (for AwA and AwA2 we allocated 15% since there are only 40 seen classes).
Then we allocate 10% out of the remaining 85% of the data for validation seen data.
This means that in total we allocate ≈ 30 % of all the seen data to perform validation.
It is known (Xian et al., 2018a; Min et al., 2020), that
GZSL-H score can be improved slightly by reducing the weight of seen class logits during accuracy computation since this would partially relieve the bias towards seen classes.
We also employ this trick by multiplying seen class logits by value s during evaluation and ﬁnd its optimal value using cross-validation together with the other hyperparameters.
On Figure 4 in Appendix D.4, we provide validation/test accuracy curves of how it inﬂuences the performance.
Evaluation and discussion .
We evaluate the model on the corresponding test sets using 3 metrics as proposed by Xian et al.
(2018a): seen generalized unseen accuracy (GZSL-U), generalized seen accuracy (GZSL-S) and GZSL-S/GZSL-U harmonic mean (GZSL-H), which is considered to the main metric for ZSL.
Table 2 shows that our model has the state-of-the-art in 3 out of 4 datasets.
Training speed results .
We conducted a survey and rerun several recent SotA methods from their ofﬁcial implementations to check their training speed, which details we report in Appx D. Tableshows the average training time for each of the methods.
Since our model is just a vanilla MLP and does not use any sophisticated training scheme, it trains from 30 to 500 times faster compared to other methods, while outperforming them in the ﬁnal performance
[Table]
5. CZSL EXPERIMENTS
Datasets .
We test our approach in CZSL scenario on two datasets: CUB Welinder et al.
(2010) and
SUN Patterson et al.
CUB contains 200 classes and is randomly split into 10 tasks with
[Table] [Table]
classes per task.
SUN contains 717 classes which is randomly split into 15 tasks, the ﬁrst 3 tasks have 47 classes and the rest of them have 48 classes each (717 classes are difﬁcult to separate evenly).
We use ofﬁcial train/test splits for training and testing the model.
Model and optimization .
We follow the proposed cross-validation procedure from Chaudhry et al.
Namely, for each run we allocate the ﬁrst 3 tasks for hyperparameter search, validating on the test data.
After that we reinitialize the model from scratch, discard the ﬁrst 3 tasks and train it on the rest of the data.
This reduces the effective number of tasks by 3, but provides a more fair way to perform cross-validation Chaudhry et al.
We use an ImageNet-pretrained ResNet-18 model as an image encoder E ( x ) which is optimized jointly with P θ .
For CZSL experiments, P θ is a 2-layer
MLP and we test the proposed CN procedure.
All the details can be found in Appendix G.
We test our approach on 3 continual learning methods: EWC Kirkpatrick et al.
(2017), MAS Aljundi et al.
(2017) and A-GEM Chaudhry et al.
(2019) and 2 benchmarks: Multi-Task model and Sequential model.
EWC and MAS ﬁght forgetting by regularizing the weight update for a new task in such a way that the important parameters are preserved.
A-GEM maintains a memory bank of previously encountered examples and performs a gradient step in such a manner that the loss does not increase on them.
Multi-Task is an “upper bound” baseline: a model which has an access to all the previously encountered data and trains on them jointly.
Sequential is a “lower bound” baseline: a model which does not employ any technique at all.
We give each model an equal number of update iterations on each task.
This makes the comparison of the Multi-Task baseline to other methods more fair: otherwise, since its dataset grows with time, it would make t times more updates inside task t than the other methods.
Evaluation and discussion .
Results for the proposed metrics mU, mS, mH, mAUC, mJA and forgetting measure from Lopez-Paz & Ranzato (2017) are reported in Table 3 and Appendix G.
As one can observe, class normalization boosts the performance of classical regularization-based and replay-based continual learning methods by up to 100% and leads to lesser forgetting.
However, we are still far behind traditional supervised classiﬁers as one can infer from mJA metric.
For example, some state-of-the-art approaches on CUB surpass 90% accuracy Ge et al.
(2019) which is drastically larger compared to what the considered approaches achieve.
C ONCLUSION
We investigated and developed normalization techniques for zero-shot learning.
We provided theoretical groundings for two popular tricks: normalize+scale and attributes normalization and showed both provably and in practice that they aid training by controlling a signal’s variance during a forward pass.
Next, we demonstrated that they are not enough to constrain a signal from ﬂuctuations for a deep ZSL model.
That motivated us to develop class normalization: a new normalization scheme that ﬁxes the problem and allows to obtain SotA performance on 4 standard ZSL datasets in terms of quantitative performance and training speed.
Next, we showed that ZSL attribute embedders tend to have more irregular loss landscape than traditional classiﬁers and that class normalization partially remedies this issue.
Finally, we generalized ZSL to a broader setting of continual zero-shot learning and proposed a set of principled metrics and baselines for it.
We believe that our work will spur the development of stronger zero-shot systems and motivate their deployment in real-world applications.
A “N ORMALIZE + SCALE ” TRICK
As being discussed in Section 3.2, “normalize+scale” trick changes the logits computation from a usual dot product to the scaled cosine similarity:
[Equation]
13) where ˆ y c is the logit value for class c ; z is an image feature vector; p c is the attribute embedding for class c : p c =
P θ ( a c [Equation] = V H ϕ [Equation]
a c [Equation] (14) and γ is the scaling hyperparameter.
Let’s denote a penultimate hidden representation of P θ as h c
= H ϕ ( a c ) .
We note that in case of linear P θ , we have h c = a c .
Let’s also denote the dimensionalities of z and h c by d z and d h .
A. A SSUMPTIONS
To derive the approximate variance formula for ˆ y c we will use the following assumptions and approximate identities:
(i) All weights in matrix V : • are independent from each other and from z k and h c,i (for all k, i ); • E
[ V ij ] =
for all i, j ; • Var [ V ij ]
= s v for all i, j .
(ii)
There exists (cid:15) > 0 s.t.
(2 + (cid:15) ) -th central moment exists for each of h c, 1 , ..., h c,d h .
We require this technical condition to be able apply the central limit theorem for variables with non-equal variances.
(iii)
All h c,i , h c,j are independent from each other for i (cid:54) =
j .
This is the least realistic assumption from the list, because in case of linear P θ it would be equivalent to independence of coordinates in attribute vector a c .
We are not going to use it in other statements.
As we show in Appendix A.3 it works well in practice.
(iv) All p c,i , p c,j are independent between each other.
This is also a nasty assumption, but more safe to assume in practice (for example, it is easy to demonstrate that Cov [ p c,i , p c,j ] = 0 for i (cid:54)
= j ).
We are going to use it only in normalize+scale approximate variance formula derivation.
(v) z ∼ N ( 0 , s z I ) .
This property is safe to assume since z is usually a hidden representation of a deep neural network and each coordinate is computed as a vector-vector product between independent vectors which results in the normal distribution (see the proof below for p c (cid:32) N ( 0 , s p I ) ).
(vi) For ξ ∈ { z , p c
we will use the approximations:
[Equation]
and E (cid:20)
[Equation]
i
[Equation]
j
[Equation]
This
approximation
is safe to use if the dimensionality of ξ is large enough (for neural networks it is deﬁnitely the case because the contribution of each individual ξ i in the norm (cid:107) ξ (cid:107) 2 becomes negligible.
Assumptions (i-v) are typical for such kind of analysis and can also be found in (Glorot & Bengio, 2010;
He et al., 2015; Chang et al., 2020).
Assumption (vi), as noted, holds only for large-dimensional inputs, but this is exactly our case and we validate that using leads to a decent approximation on Figure 2.
A. F ORMAL STATEMENT AND THE PROOF
Statement 1 (Normalize+scale trick) .
If conditions (i)-(vi) hold, then:
[Equation]
Proof.
First of all, we need to show that p c,i (cid:32) N (0 , s p ) for some constant s p .
Since
[Equation]
from assumption (i) we can easily compute its mean:
[Equation]
[Equation]
22) and the variance:
[Equation]
[Equation] [Equation]
[Equation]
Using E [ V i,j V i,k ] = 0 for k (cid:54)
=
j , we have:
[Equation]
[Equation]
Since s v =
Var [ V i,j ] = E (cid:2)
V 2 i,j (cid:3) − E
[ V i,j ] 2 = E (cid:2)
V 2 i,j (cid:3) , we have: =
d h (cid:88)
[Equation]
Now, from the assumptions (ii) and (iii) we can apply Lyapunov’s Central Limit theorem to p c,i , which gives us:
[Equation]
For ﬁnite d h , this allows us say that:
[Equation]
Now note that from (vi) we have:
[Equation]
Since ξ ∼ N ( 0 , s ξ ) for ξ ∈ { z , p c } , d ξ (cid:107) ξ (cid:107) 22 follows scaled inverse chi-squared distribution with inverse variance τ = 1 /s ξ , which has a known expression for expectation:
[Equation]
[Equation]
Now we are left with using approximation (vi) and plugging in the above expression into the variance formula:
[Equation]
[Equation]
A. E MPIRICAL VALIDATION
In this subsection, we validate the derived approximation empirically.
For empirical validation of the variance analyzis, see Appendix E.
For this, we perform two experiments:
• Synthetic data .
An experiment on a synthetic data.
We sample x ∼ N ( 0 , I d ) , y ∼ N ( 0 , I d ) for different dimensionalities d = 32 , 64 , 128 , ..., 8192 and compute the cosine similarity:
[Equation]
After that, we compute Var [ z ] and average it out across different samples.
The result is presented on ﬁgure 2a.
[Figure]
• Real data .
We take ImageNet-pretrained ResNet101 features and real class attributes (unnormalized) for SUN, CUB, AwA1, AwA2 and aPY datasets.
Then, we initialize a random 2-layer MLP withhidden units, and generate real logits (without scaling).
Then we compute mean empirical variance and the corresponding standard deviation over different batches of size 4096.
The resulted boxplots are presented on ﬁgure 2b.
In both experiments, we computed the logits with γ = 1 .
As one can see, even despite our demanding assumptions, our predicted variance formula is accurate for both synthetic and real-world data.
B A TTRIBUTES NORMALIZATION
We will use the same notation as in Appendix A. Attributes normalization trick normalizes attributes to the unit L 2 -norm:
[Equation]
[Equation]
We will show that it helps to preserve the variance for pre-logit computation when attribute embedder P θ is linear:
[Equation]
For a non-linear attribute embedder it is not true, that’s why we need the proposed initialization scheme.
B. A SSUMPTIONS
We will need the following assumptions:
(i)
Feature vector z has the properties: • E [ z ] = 0 ; • Var [ z i ] =
s z for all i = 1 , ..., d z .
• All z
i are independent from each other and from all p c,j . (ii)
Weight matrix V is initialized with Xavier fan-out mode,
i
e. Var [ V ij ] = 1 /d z and are independent from each other.
Note here, that we do not have any assumptions on a c .
This is the core difference from Chang et al.
and is an essential condition for ZSL (see Appendix H).
B. F ORMAL STATEMENT AND THE PROOF
Statement 2 (Attributes normalization for a linear embedder) .
If assumptions (i)-(ii) are satisﬁed and (cid:107) a c (cid:107) 2 = 1 , then:
[Equation]
Proof.
Now, note that:
[Equation]
Then the variance for ˜ y c has the following form:
[Equation]
since s v = 1 /d z [Equation] then:
[Equation]
since attributes are normalized, i [Equation] e. (cid:107) a c (cid:107) 2 = 1 , then:
[Equation]
C N ORMALIZATION FOR A DEEP ATTRIBUTE EMBEDDER C. F ORMAL STATEMENT AND THE PROOF
Using the same derivation as in B, one can show that for a deep attribute embedder:
[Equation]
[Equation] normalizing attributes is not enough to preserve the variance of Var [˜ y c ] , because
[Equation]
and h c
= H ϕ ( a c ) is not normalized to a unit norm.
To ﬁx the issue, we are going to use two mechanisms:
A different initialization scheme:
[Equation]
Using the standardization layer before the ﬁnal projection matrix:
[Equation]
µ x , σ x are the sample mean and variance and (cid:11) is the element-wise division.
C. A SSUMPTIONS
We’ll need the assumption:
(i)
Feature vector z has the
properties
: • E [ z ] = 0 ; • Var [ z i ] = s z for all i = 1 , ..., d z .
• All z
i are independent from each other and from all p c,j .
[Equation]
C. F ORMAL STATEMENT AND THE PROOF
Statement 3.
If the assumption (i) is satisﬁed, an attribute embedder has the form P θ =
V ◦ S ◦ H ϕ
and we initialize output matrix V s.t.
Var [ V ij ] = 1 d z d h , then the variance for ˜ y c is preserved:
Var [˜ y c ]
≈ Var [ z i ] =
s z (72)
Proof.
With some abuse of notation, let ¯ h c =
S ( h c ) (in practice, S receives a batch of h c instead of a single vector).
This leads to:
E [ h c ] =and
Var [ h c ] ≈(73)
Using the same reasoning as in Appendix B, one can show that:
[Equation]
(cid:107) [Equation] (cid:3) [Equation]
(cid:2) (cid:107) [Equation] (cid:107) 22 (cid:3) (74) So we are left to demonstrate that E (cid:2) (cid:107) ¯ h c (cid:107) 22 (cid:3) =
d h :
E (cid:2) (cid:107) ¯ h c (cid:107) 22 (cid:3) =
d h (cid:88) i =
E (cid:2) ¯ h 2 c,i (cid:3) =
d h (cid:88) i =
Var (cid:2) ¯ h c,i (cid:3)
≈ d h (75)
C. A DDITIONAL EMPIRICAL STUDIES D ZSL DETAILS D. E XPERIMENTS DETAILS
In this section, we cover hyperparameter and training details for our ZSL experiments and also provide the extended training speed comparisons with other methods.
We depict the architecture of our model on ﬁgure 3.
As being said, P θ is just a simple multi-layer MLP with standardization procedure (9) and adjusted output layer initialization (10).
Besides, we also found it useful to use entropy regularizer (the same one which is often used in policy gradient methods for exploration) for some datasets:
[Equation]
We train the model with Adam optimizer with default β 1 and β 2 hyperparams.
The list of hyperparameters is presented in Table 4.
In those ablation experiments where we do not attributes normalization (2), we apply simple standardization to convert attributes to zero-mean and unit-variance.
D. A DDITIONAL EXPERIMENTS AND ABLATIONS
In this section we present additional experiments and ablation studies for our approach (results are presented in Table 5).
We also run validate :
[Table]
[Table]
Kaiming fan-out + (9) 42.8 41.2 42.0 51.0 49.0 50.0 60.3 73.2 66.1 56.8 76.9 65.
• Dynamic normalization.
As one can see from formula (8), to achieve the desired variance it would be enough to initialize V s.t.
Var [ V ij ] = 1 /d z
(equivalent to Xavier fan-out) and use a dynamic normalization :
[Equation]
between V and H ϕ , i.e. P θ ( a c ) =
V DN ( H ϕ ( a c )) .
Expectation E (cid:2) (cid:107) h (cid:107) 22 (cid:3) is computed over a batch on each iteration.
A downside of such an approach is that if the dimensionality is large, than a lot of dimensions will get suppressed leading to bad signal propagation.
Besides, one has to compute the running statistics to use them at test time which is cumbersome.
• Traditional initializations + standardization procedure (9).
These experiments ablate the necessity of using the corrected variance formula (10).
• Performance of NS for different scaling values of γ and different number of layers.
D. M EASURING TRAINING SPEED
We conduct a survey and search for open-source implementations of classiﬁcation ZSL papers that were recently published on top conferences.
This is done by 1) checking the papers for code urls; 2) checking their supplementary; 3) searching for implementations on github.com and 4) searching authors by their names on github.com and checking their repositories list.
As a result we found 8 open-source implementations of the recent methods but one of them got discarded since the corresponding data was not provided.
We reran all these
[Table]
methods with the ofﬁcial hyperparameters on the corresponding datasets and report their training time in Table 6 in Appx D.
All runs are made with the ofﬁcial hyperparameters and training setups and on the same hardware: NVidia GeForce RTX 2080 Ti GPU, × 16 Intel Xeon Gold 6142 CPU and 128 GB RAM.
The results are depicted on
Table 6.
As one can see, the proposed method trains 50-100 faster than the recent SotA.
This is due to not using any sophisticated architectures employing generative models (Xian et al., 2018b; Narayan et al., 2020); or optimization schemes like episode-based training (Li et al., 2019; Yu et al., 2020).
D. C HOOSING SCALE s FOR SEEN CLASSES
As mentioned in Section [Table] we reweigh seen class logits by multiplying them on scale value s .
This is similar to a strategy considered by Xian et al.
(2018a)
; Min et al.
but we found that multiplying by a value instead of adding it by summation is more intuitive.
We ﬁnd the optimal scale value s by cross-validation together with all other hyperparameters on the grid [1 . 0 , 0 .
On Figure 4, we depict the curves of how s inﬂuences GZSL-U/GZSL-S/GZSL-U for each dataset.
D. I NCORPORATING CN FOR OTHER ATTRIBUTE EMBEDDERS
In this section, we employ our proposed class normalization for two other methods: RelationNet 1 (Sung et al., 2018) and CVC-ZSL 2 (Sung et al., 2018).
We build upon the ofﬁcially provided source code bases and use the ofﬁcial hyperparameters for all the setups.
For RelationNet, the authors provided the running commands.
For CVC-ZSL, we used those hyperparameters for each dataset, that were speciﬁed in their paper.
That included using different weight decay of 1 e − 4 , 1 e − 3 , 1 e − 3 and 1 e − 5 for AwA1, AwA2, CUB and SUN respectively, as stated in Section 4.2 of the paper (Li et al., 2019).
We incorporated our Class Normalization procedure to these attribute embedders and launched them on the corresponding datasets.
The results are reported in
Table 7.
For some reason, we couldn’t reproduce the ofﬁcial results for both these methods which we additionally report.
As one can see from the presented results, our method gives +2.0 and +1.8 of GZSL-H improvement on average for these two methods respectively which emphasizes once again its favorable inﬂuence on the learned representations
. D. A DDITIONAL ABLATION ON AN AND NS TRICKS
In Table 8 we provide additional ablations on attributes normalization and normalize+scale tricks.
As one can see, they greatly inﬂuence the performance of ZSL attribute embedders.
[Figure]
( s ∗ = 0 . 95 )
[Figure]
( s ∗ = 0 . 95 )
[Figure]
[Figure] [Figure] [Figure] the allocated the and
[Table]
RelationNet (ofﬁcial code) - - - 38.3 62.4 47.5 28.5 87.8 43.1 10.2 88.1 18.
RelationNet (ofﬁcial code) + CN - - - 40.1 62.8 48.9 29.8 88.4 44.6 12.7 88.8 22.
CVC-ZSL (ofﬁcial code) 20.
[Table] [Table]
[Figure]
-ZSL (ofﬁcial code)
[Figure]
NS
[Figure]
-ZSL (ofﬁcial code) -
[Figure]
ZSL
[Figure]
ofﬁcial code)
[Figure]
E A DDITIONAL VARIANCE ANALYZIS
In this section, we provide the extended variance analyzis for different setups and datasets.
The following models are used:
A linear ZSL model with/without normalize+scale (NS) and/or attributes normalization (AN).
A 3-layer ZSL model with/without NS and/or AN. 3.
A 3-layer ZSL model with class normalization , with/without NS and/or AN.
These models are trained on 4 standard ZSL datasets: SUN, CUB, AwA1 and AwA2 and their logits variance is calculated on each iteration and reported.
The same batch size, learning rate, number of epochs, hidden dimensionalities were used.
Results are presented on ﬁgures 5, 6, 7 and 8, which illustrates the same trend:
• A traditional linear model without NS and AN has poor variance.
Adding NS with a proper scaling of 5 and AN improves it and bounds to be close to 1 .
• After introducing new layers, NS and AN stop “working” and variance vanishes below unit.
• Incorporating class normalization allows to push it back to 1 .
F L OSS LANDSCAPE SMOOTHNESS ANALYSIS F. O VERVIEW
As being said, we demonstrate two things:
[Figure] [Figure]
[Figure]
For each example in a batch, parameters of a ZSL attribute embedder receive K more updates than a typical non-ZSL classiﬁer, where K is the number of classes.
This suggests a hypothesis that it has larger overall gradient magnitude, hence a more irregular loss surface. Our standardization procedure (9) makes the surface more smooth. We demonstrate it by simply applying Theorem 4.4 from (Santurkar et al. To see the ﬁrst point, one just needs to compute the derivative with respect to weight W ij for n -th data sample for loss surface L SL n of a traditional model and loss surface L ZSL n of a ZSL embedder:
[Equation]
Since the gradient has K more terms and these updates are not independent from each other (since the ﬁnal representations are used to construct a single logits vector after a dot-product with z n ), this may lead to an increased overall gradient magnitude.
We verify this empirically by computing the gradient magnitudes for our model and its non-ZSL “equivalent”: a model with the same number of layers and hidden dimensionalities, but trained to classify objects in non-ZSL fashion.
To show that our class standardization procedure (9) smoothes the landscape, we apply Theorem 4.4 from Santurkar et al.
(2018) that demonstrates that a model augmented with batch normalization (BN) has smaller
Lipschitz constant.
This is easily done after noticing that (9) is equivalent to BN, but without scaling/shifting and is applied in a class-wise instead of the batch-wise fashion.
We empirically validate the above observations on Figures 1 and 9.
F. F ORMAL REASONING
ZSL embedders are prone to have more irregular loss surface .
We demonstrate that the loss surface of attribute embedder P θ is more irregular compared to a traditional neural network.
The reason is that its output vectors p c
= P θ ( a c ) are not later used independently, but instead combined together in a single matrix W =
[ p 1 , ..., p K s ] to compute the logits vector y
=
W z .
Because of this, the gradient update for θ i receives K signals instead of just 1 like for a traditional model, where K is the number of classes.
Consider a classiﬁcation neural network F ψ ( x ) optimized with loss L and some its intermediate transformation y
= W (cid:96) x .
Then the gradient of L n on n -th training example with respect to W (cid:96)ij is computed as:
[Equation]
While for attribute embedder P θ ( a c ) , we have K times more terms in the above sum since we perform K forward passes for each individual class attribute vector a c .
The gradient on n -th training example for its inner transformation y =
W (cid:96) x
( a c ) is computed as:
[Equation]
From this, we can see that the average gradient for P θ is K times larger which may lead to the increased overall gradient magnitude and hence more irregular loss surface as deﬁned in Section 3.5.
CN smoothes the loss landscape .
In contrast to the previous point, we can prove this rigorously by applying The- orem 4.4 by Santurkar et al.
(2018), who showed that performing standardization across hidden representations smoothes the loss surface of neural networks.
Namely Santurkar et al. (2018) proved the following:
Theorem 4.4 from (Santurkar et al., 2018) .
For a network with BatchNorm with loss (cid:98) L and a network without BatchNorm with loss L if:
[Equation]
then:
[Equation]
where y (cid:96) , ˆ y (cid:96) are hidden representations at the (cid:96) -th layer, m is their dimensionality, σ (cid:96) is their standard deviation, µ g
=
m (cid:68) 1 , ∂ (cid:98) L /∂ z (cid:96) (cid:69) for z (cid:96) = γ y (cid:96) + β , is the average gradient norm, γ is the BN scaling parameter, X is the input data matrix at layer (cid:96) .
Now, it easy easy to see that our class standardization (9) is “equivalent” to BN (and thus the above theorem can be applied to our model): [Figure]
• First, set γ = 1 (i
e. remove scaling) and β =
(i.e. remove bias addition).
• Second, apply this modiﬁed BN inside attribute embedder P θ on top of attributes representations H
=
[ h 1 , ..., h K ] across K -axis
(class dimension) instead of objects representations X
=
[ x 1 , ..., x B ] across B -axis (batch dimension).
It is important to note here that there are no restricting assumptions on the loss function or what data X is being used.
Thus Theorem 4.4 of Santurkar et al.
(2018) is applicable to our model which means that CN smoothes its loss surface.
F. E MPIRICAL VALIDATION
To validate the above claim empirically, we approximate the quantity but computed for all the parameters of the model instead of a single layer on each iteration.
We do this by taking 10 random batches of sizefrom the dataset, adding (cid:15) ∼ N ( 0 , I ) noise to this batch, computing the gradient of the loss with respect to the parameters, then computing its norm scaled by 1 /n factor to account for a small difference in number of parameters ( ≈ 0 . 9 ) between a ZSL model and a non-ZSL one.
This approximates the quantity 11, but instead of approximating it around 0 , we approximate it around real data points since it is more practically relevant.
We run the described experiment for three models:
A vanilla MLP classiﬁer, i .e. without any
class attributes.
For
each dataset , it receives feature vector z and
produces logits. A vanilla MLP zero-shot classiﬁer, as described in
section
An MLP zero-shot classiﬁer with class normalization.
All three models were trained with cross-entropy loss with the same optimization hyperparameters: learning rate of 0.0001, batch size of 256, number of iterations of 2500.
They had the same numbers of layers, which was equal to 3.
The results are illustrated on ﬁgures 1 (left) and 9.
As one can see, traditional MLP models indeed have more ﬂat loss surface which is observed by a small gradient norm.
But class normalization helps to reduce the gap.
G C ONTINUAL Z ERO -S HOT L EARNING DETAILS G. CZSL EXPERIMENT DETAILS
As being said, we use the validation sequence approach from Chaudhry et al.
to ﬁnd the best hyperpa- rameters for each method.
We allocate the ﬁrst 3 tasks to perform grid search over a ﬁxed range.
After the best hyperparameters have been found, we train the model from scratch for the rest of the tasks.
The hyperparameter range for CZSL experiments are presented in Table 9 (we use the same range for all the experiments).
We train the model for 5 epochs on each task with SGD optimizer.
We also found it beneﬁcial to decrease learning rate after each task by a factor of 0.9.
This is equivalent to using step-wise learning rate schedule with the number of epochs equal to the number of epochs per task.
As being said, for CZSL experiments, we use an ImageNet-pretrained ResNet- [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Figure]
l [Figure] As one can clearly see, adding class normalization signiﬁcantly improves the results, at some timesteps even surpussing the multi-task baseline without ClassNorm.
G. A DDITIONAL CZSL METRICS
In this subsection, we describe our proposed CZSL metrics that are used to access a model’s performance.
Subscripts “tr”/“ts” denote train/test data.
Mean Seen Accuracy (mSA) .
We compute GZSL-S after tasks t [Equation] 1 , .., T and take the average:
[Equation]
Mean Unseen Accuracy (mUA) .
We compute GZSL-U after tasks t = 1 , ..., T − 1 (we do not compute it after task T since D >T = ∅ ) and take the average:
[Equation]
Mean Harmonic Seen/Unseen Accuracy (mH) .
We compute GZSL-H after tasks t = 1 , ..., T − 1 and take the average:
[Equation]
Mean Area Under Seen/Unseen Curve (mAUC) .
We compute AUSUC Chao et al.
after tasks t = 1 , ..., T − 1 and take the average:
[Equation]
AUSUC is a performance metric that allows to detect model’s bias towards seen or unseen data and in our case it measures this in a continual fashion.
Mean Joint Accuracy (mJA) .
On each task t we compute the generalized accuracy on all the test data we have for the entire problem:
[Equation]
This evaluation measure allows us to understand how far behind a model is from the traditional supervised classiﬁers.
A perfect model would be able to generalize on all the unseen classes from the very ﬁrst task and maintain the performance on par with normal classiﬁers.
H W HY CANNOT WE HAVE INDEPENDENCE , ZERO - MEAN AND SAME - VARIANCE ASSUMPTIONS FOR ATTRIBUTES IN ZSL?
Usually, when deriving an initialization scheme, people assume that their random vectors have zero mean, the same coordinate variance and the coordinates are independent from each other.
In the paper, we stated that these are unrealistic assumptions for class attributes in ZSL and in this section elaborate on it.
Attribute values for the common datasets need to be standardized to satisfy zero-mean and unit-variance (or any other same-variance) assumption.
But it is not a sensible thing to do, if your data does not follow normal distribution, because it makes it likely to encounter a skewed long-tail distribution like the one illustrated on Figure 14.
In reality, this does not break our theoretical derivations, but this creates an additional optimizational issue which hampers training and that we illustrate in Table 8.
This observation is also conﬁrmed by Changpinyo et al.
(2016b).
If we do not use these assumptions but rather use zero-mean and unit-variance one (and enforce it during training), than the formula (6) will transform into:
[Equation]
[Equation]
[Table]
[Figure]
This means, that we need to adjust the initialization by a value 1 /d a to preserve the variance.
This means, that we initialize the ﬁrst projection matrix with the variance:
[Equation]
In Table 10, we show what happens if we do count for this factor and if we don’t.
As one can see, just standardizing the attributes without accounting for 1 /d a factor leads to worse performance.
To show more rigorously that attributes do not follow normal distribution and are not independent from each other, we report two statistical results:
• Results of a normality test based on D’Agostino and Pearson’s tests, which comes with scipy python stats library.
We run it for each attribute dimension for each dataset and report the distribution of the resulted χ 2 -statistics with the corresponding p -values on Figure 12.
• Compute the distribution of absolute values of correlation coefﬁcients between attribute dimensions.
The results are presented on Figure 13 which demonstrates that attributes dimensions are not independent between each other in practice and thus we cannot use a common indepdence assumption when deriving
the
initialization sheme for a ZSL embedder.
We note, however, that attributes distribution is uni-modal and, in theory, it is possible to transform it into a normal one (by hacking it with log/inverse/sqrt/etc), but such an approach is far from being scalable.
It is not scalable because transforming a non-normal distribution into a normal one is tricky and is done either manually by ﬁnding a proper transformation or by solving an optimization task.
This is tedious to do for each dataset and thus scales poorly.
[Figure]
This ﬁgure shows that attributes are not independent that’s why it would be unreasonable to use such an assumption.
If attributes would be independent from each other, that would mean that, for example, that “having black stripes” is independent from “being orange”, which tigers would argue not to be a natural assumption to make.
[Figure]