C LASS N ORMALIZATION FOR Z ERO -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Normalization techniques have proved to be a crucial ingredient of successful training in a traditional supervised learning regime.
However, in zero-shot learning (ZSL) world, these ideas have recieved only marginal attention.
In this work, we study normalization in ZSL scenario from both theoretical and practical perspectives.
First, we give a theoretical explaination to two popular tricks used in zero-shot learning: normalize+scale and attributes normalization and show that they help training by preserving variance during a forward pass.
Next, we demonstrate that they are insufﬁcient to normalize a deep ZSL model and propose class normalization : a normalization scheme, which alleviates this issue both provably and in practice.
Third, we show that ZSL models typically have more irregular loss surface compared to traditional classiﬁers and that the proposed method partially remedies this problem.
Then, we test our approach on 4 standard ZSL datasets and outperform sophisticated modern SotA with a simple MLP optimized without any bells and whistles and having ≈
50 times faster training speed.
Finally, we generalize ZSL to a broader problem — continual ZSL, and introduce some principled metrics and rigorous baselines for this new setup.
The code will be made available.
I NTRODUCTION
Zero-shot learning (ZSL) aims to understand new concepts based on their semantic descriptions instead of numerous input-output learning pairs.
It is a key element of human intelligence and our best machines still struggle to master it (Ferrari & Zisserman, 2008; Lampert et al., 2009; Xian et al., 2018a).
Normalization techniques like batch/layer/group normalization (Ioffe & Szegedy, 2015; Ba et al., 2016; Wu & He, 2018) are now a common and important practice of modern deep learning.
But despite their popularity in traditional supervised training, not much is explored in the realm of zero-shot learning, which motivated us to study and investigate normalization in ZSL models.
We start by analyzing two ubiquitous tricks employed by ZSL and representation learning practitioners: normalize+scale (NS) and attributes normalization (AN) (Bell et al., 2016;
Zhang et al., 2019; Guo et al., 2020; Chaudhry et al., 2019).
Their dramatic inﬂuence on performance can be observed from
Table 1.
When these two tricks are employed, a vanilla MLP model, described in Sec 3.1, can outperform some recent sophisticated ZSL methods.
Normalize+scale (NS) changes logits computation from usual dot-product to scaled cosine similarity:
[Equation]
where z is an image feature, p c is c -th class prototype and γ is a hyperparameter, usually picked from [5 , 10] interval (Li et al., 2019; Zhang et al., 2019).
Scaling by γ is equivalent to setting a high temperature of γ 2 in softmax.
In Sec. 3.2, we theoretically justify the need for this trick and explain why the value of γ must be so high.
Attributes Normalization (AN) technique simply divides class attributes by their L 2 norms:
[Equation]
While this may look inconsiderable, it is surprising to see it being preferred in practice (Li et al., 2019;
Narayan et al., 2020; Chaudhry et al., 2019) instead of the traditional zero-mean and unit-variance data standardization (Glorot & Bengio, 2010).
In Sec 3, we show that it helps in normalizing signal’s variance in and ablate its importance in Table 1 and Appx D.
When NS and AN are integrated into a basic ZSL model, its performance is boosted up to a level of some sophisticated SotA methods and additionally using CN allows to outperform them.
± NS and ±
AN denote if normalize+scale or attributes normalization are being used.
Bold/normal blue font denote best/second-best results.
Extended results are in Table 2 and Table 5 of Appx D.
[Table]
These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear (see Figure 1), but they fail when we use a multi-layer architecture.
To remedy this issue, we introduce Class Normalization (CN) : a novel normalization scheme, which is based on a different initialization and a class-wise standardization transform.
Modern ZSL methods either utilize sophisticated architectural design like training generative models (Narayan et al., 2020; Felix et al., 2018)
or use heavy optimization schemes like episode-based training (Yu et al., 2020; Li et al., 2019).
In contrast, we show that simply incorporating the proposed Class Normalization into a vanilla
MLP model is enough to set new state-of-the-art results on 3/4 standard ZSL datasets; see Table 2.
We also perform an empirical study and rerun a bunch of recently proposed ZSL methods that have ofﬁcial open-source implementations and observe that it usually takes more than 1 hour to train a typical ZSL model from scratch.
Meanwhile, since our model is just a vanilla MLP with an additional standardization procedure and is optimized with plain gradient descent, training time for it takes 50-100 times less and runs in about 1 minute.
We also demonstrate that many ZSL models tend to have more irregular loss surface compared to traditional supervised learning classiﬁers.
This is due to using matrix-wise inputs instead of vector-wise ones which is equivalent to performing several forward passes without the subsequent averaging.
We discuss and empirically validate this in Sec 3.
and Appx F and demonstrate that CN partially remedies the problem.
Apart from the theoretical analysis and a new normalization scheme, we also propose a broader ZSL setup: continual zero-shot learning (CZSL).
Continual learning (CL) is an ability to acquire new knowledge without forgetting (e.g. (Kirkpatrick et al., 2017)), which is scarcely investigated in ZSL.
We develop the ideas of lifelong learning with class attributes, originally proposed by Chaudhry et al.
(2019) and extended by Wei et al. (2020a), propose several principled metrics for it and test several classical CL methods in this new setup.
R ELATED WORK
Zero-shot learning .
Zero-shot learning (ZSL) aims at understand example of unseen classes from their language or semantic descriptions.
Earlier ZSL methods directly predict attribute conﬁdence from images to facilitate zero-shot recognition (e.g., Lampert et al.
; Farhadi et al.
Lampert et al. (2013)).
Recent ZSL methods for image classiﬁcation can be categorized into two groups: generative-based and embedding-based.
The main goal for generative-based approaches is to build a conditional generative model (e.g., GANs Goodfellow et al.
and VAEs (Kingma & Welling, 2014)) to synthesize visual generations conditioned on class descriptors (e.g., Xian et al.
(2018b); Zhu et al.
; Elhoseiny & Elfeki (2019); Guo et al.
; Guo et al.
Kumar Verma et al.
At test time, the trained generator is expected to produce synthetic/fake data of unseen classes given its semantic descriptor.
The fake data is then used to train a traditional classiﬁer or to perform a simple kNN-classiﬁcation on the test images.
Embedding-based approaches learn a mapping that projects semantic attributes and images into a common space where the distance between a class projection and the corresponding images is minimized (e.g, Romera-Paredes & Torr (2015); Frome et al.
; Lei Ba et al. (2015); Akata et al. (2016a); Zhang et al.
; Akata et al.
(2015; 2016b)).
Normalize+scale and attributes normalization .
It was observed both in ZSL (e.g., (Li et al., 2019;
Zhang et al., 2019; Bell et al., 2016)) and representation learning literature
(e.g., (Guo et al., 2020;
Ye et al., 2020) research that normalize+scale (i.e. (1)) and attributes normalization (i.e. (2)) tend to signiﬁcantly improve the performance of a learning system.
In the literature, these two techniques lack rigorous motivation and are usually introduced as practical heuristics that aid training (Changpinyo et al., 2017; Zhang et al., 2019).
To the best of our knowledge, there is no precise understanding of their inﬂuence on the optimization process and the beneﬁt they provide.
Our work build upon a deep attribute embedder which is trained without any sophisticated regularization schemes or episode-based training (Li et al., 2019; Yu et al., 2020).
We theoretically and practically investigate how the mentioned tricks control a signal’s variance inside a model and show that they are not sufﬁcient to normalize a deep attribute embedder.
Initialization schemes .
In the seminal work, Xavier’s init Glorot & Bengio (2010), the authors showed how to preserve the variance during a forward pass.
He et al.
(2015) applied a similar analysis but taking ReLU nonlinearities into account.
There is also a growing interest in two-step Jia et al.
(2014), data-dependent Krähenbühl et al.
(2015), and orthogonal Hu et al.
(2020) initialization schemes.
However, the importance of a good initialization for multi-modal embedding functions like attribute embedding is less studied and not well understood.
We propose a proper initialization scheme based on a different initialization variance and a dynamic standardization layer.
Our variance analyzis is similar in nature to Chang et al.
(2020) since attribute embedder may be seen as a hypernetwork (Ha et al., 2016) that outputs a linear classiﬁer.
But the exact embedding transformation is different from a hypernetwork since it has matrix-wise input and in our derivations we have to use more loose assumptions about attributes distribution (see Sec 3 and Appx H).
Normalization techniques .
A closely related branch of research is the development of normalization layers for deep neural networks (Ioffe & Szegedy, 2015) since they also inﬂuence a signal’s variance.
BatchNorm, being the most popular one, normalizes the location and scale of activations.
It is applied in a batch-wise fashion and that’s why its performance is highly dependent on batch size (Singh &
Krishnan, 2020).
That’s why several normalization techniques have been proposed to eliminate the batch-size dependecy (Wu & He, 2018; Ba et al., 2016; Singh & Krishnan, 2020).
The proposed class normalization is very similar to a standardization procedure which underlies BatchNorm, but it is applied class-wise in the attribute embedder.
This also makes it independent from the batch size being used.
Continual zero-shot learning .
We introduce continual zero-shot learning: a new benchmark for
ZSL agents that is inspired by continual learning literature (e.g., Kirkpatrick et al.
It is a development of the scenario proposed in Chaudhry et al.
, but authors there focused on ZSL performance only a single task ahead, while in our case we consider the performance on all seen (previous tasks) and all unseen data (future tasks).
This also contrasts our work to the very recent work by Wei et al.
(2020b), where a sequence of seen class splits of existing ZSL benchmsks is trained and the zero-shot performance is reported for every task individually at test time.
In contrast, for our setup, the label space is not restricted and covers the spectrum of all previous tasks (seen tasks so far), and future tasks (unseen tasks so far).
Due to this difference, we need to introduce a set of new metrics and benchmarks to measure this continual generalized ZSL skill over time.
From the lifelong learning perspective, the idea to consider all the processed data to evaluate the model is not new and was previously explored by Elhoseiny et al.
(2018); van de Ven & Tolias (2019).
It lies in contrast with the common practice of providing task identity at test time, which limits the prediction space for a model, making the problem easier (Kirkpatrick et al., 2017; Aljundi et al., 2017).
In Isele et al.
(2016); Lopez-Paz & Ranzato (2017) authors motivate the use of task descriptors for zero-shot knowledge transfer, but in our work we consider class descriptors instead.
We deﬁned CZSL as a continual version of generalized-ZSL which allows us to naturally extend all the existing ZSL metrics
Xian et al.
(2018a)
; Chao et al.
(2016) to our new continual setup.
N ORMALIZATION IN Z ERO -S HOT L EARNING
The goal of a good normalization scheme is to preserve a signal inside a model from severe ﬂuctuations and to keep it in the regions that are appropriate for subsequent transformations.
For example, for
ReLU activations, we aim that its input activations to be zero-centered and not scaled too much: otherwise, we risk to ﬁnd ourselves in all-zero or all-linear activation regimes, disrupting the model performance.
For logits , we aim them to have a close-to-unit variance since too small variance leads to poor gradients of the subsequent cross-entropy loss and too large variance is an indicator of poor scaling of the preceding weight matrix.
For linear layers, we aim their inputs to be zero-centered: in the opposite case, they would produce too biased outputs, which is undesirable.
In traditional supervised learning, we have different normalization and initialization techniques to control the signal ﬂow.
In zero-shot learning (ZSL), however, the set of tools is extremely limited.
In this section, we justify the popularity of Normalize+Scale(NS) and Attributes Normalization(AN) by demonstrating that they just retain a signal variance.
We demonstrate that they are not enough to normalize a deep ZSL model and propose class normalization to regulate a signal inside a deep ZSL model.
We empirically evaluate our theoretical study in Sec. 5 and appendices A, B, D and F.
3. N OTATION
A ZSL setup considers an access to datasets of seen and unseen images with the corresponding labels D s = { x si , y si } N s i
=1 and D u
= { x ui , y ui } N u
i =1 respectively.
Each class c is described by its class attribute vector a c ∈ R d a .
All attribute vectors are partitioned into non-overlapping seen and unseen sets as well: A s = { a i } K s i
=1 and A u = { a i } K u i =1 .
Here N s , N u , K s , K u are number of seen images, unseen images, seen classes, and unseen classes respectively.
In modern ZSL, all images are usually transformed via some standard feature extractor E : x (cid:55)→ z ∈ R d z (Xian et al., 2018a).
Then, a typical ZSL method trains attribute embedder P θ : a c → p c ∈ R d z which projects class attributes a sc onto feature space R d z in such a way that it lies closer to exemplar features z s of its class c .
This is done by solving a classiﬁcation task, where logits are computed using formula (1).
In such a way at test time we are able to classify unseen images by projecting unseen attribute vectors a uc into the feature space and computing similarity with the provided features z u .
Attribute embedder P θ is usually a very simple neural network (Li et al., 2019); in many cases even linear (Romera-Paredes & Torr, 2015; Elhoseiny et al., 2013), so it is the training procedure and different regularization schemes that carry the load.
We will denote the ﬁnal projection matrix and the body of P θ as V and H ϕ respectively, i.e. P θ ( a c ) =
V H ϕ ( a c ) .
During training, it receives matrix of class attributes
A =
[ a 1 , ..., a K s ] of size K s ×
d a and outputs matrix W = P θ ( A ) of size
K s × d z .
Then W is used to compute class logits with a batch of image feature vectors z 1 , ..., z N s .
3. U NDERSTANDING NORMALIZE + SCALE TRICK
One of the most popular tricks in ZSL and deep learning is using the scaled cosine similarity instead of a simple dot product in logits computation (Li et al., 2019; Zhang et al., 2019;
Ye et al., 2020):
[Equation]
(3) where hyperparameter γ is usually picked from [5 , 10] interval.
Both using the cosine similarity and scaling it afterwards by a large value is critical to obtain good performance; see Appendix D. To our knowledge, it has not been clear why exactly it has such big inﬂuence and why the value of γ must be so large.
The following statement provides an answer to these questions.
Statement 1 (informal) .
Normalize+scale trick forces the variance for ˆ y c to be approximately :
[Equation]
[Equation]
where d z is the dimensionality of the feature space .
See Appendix A for the assumptions, derivation and the empirical study.
Formula (4) demonstrates two things:
When we use cosine similarity, the variance of ˆ y c becomes independent from the variance of W = P
θ ( A ) , leading to better stability.
If one uses (3) without scaling (i
e. γ = 1 ), then the Var [ˆ y c ] will be extremely low (especially for large d z ) and our model will always output uniform distribution and the training would stale.
That’s why we need very large values for γ .
Usually, the optimal value of γ is found via a hyperparameter search (Li et al., 2019), but our formula suggests another strategy: one can obtain any desired variance ν =
Var [ˆ y c ] by setting γ to:
[Equation]
For example, for Var [ˆ y c ] =
ν
= 1 and d z = 2048 we obtain γ ≈ 6 . 78 , which falls right in the middle of [5 , 10] — a usual search region for γ used by ZSL and representation learning practitioners Li et al.
; Zhang et al.
; Guo et al.
The above consideration not only gives a theoretical understanding of the trick, which we believe is important on its own right, but also allows to speed up the search by either picking the predicted “optimal” value for γ or by searching in its vicinity.
3. U NDERSTANDING ATTRIBUTES NORMALIZATION TRICK
We showed in the previous subsection that “normalize+scale” trick makes the variances of ˆ y c independent from variance of weights, features and attributes.
This may create an impression that it does not matter how we initialize the weights — normalization would undo any ﬂuctuations.
However it is not true, because it is still important how the signal ﬂows under the hood, i.e. for an unnormalized and unscaled logit value ˜
y c
= z (cid:62) p c .
Another common trick in ZSL is the normalization of attribute vectors to a unit norm a c (cid:55)−→ a c (cid:107) a c (cid:107) 2 .
We provide some theoretical underpinnings of its importance.
Let’s ﬁrst consider a linear case for P θ , i.e. H ϕ is an identity, thus ˜ y
c
= z (cid:62) p c =
z (cid:62) V a c .
Then, the way we initialize V is crucial since Var [˜ y ]
c depends on it.
To derive an initialization scheme people use 3 strong assumptions for the inputs Glorot & Bengio (2010)
; He et al.
; Chang et al.
(2020): 1) they are zero-centered 2) independent from each other; and 3) have the covariance matrix of the form σ 2 I .
But in ZSL setting, we have two sources of inputs: image features z and class attributes a c .
And these assumptions are safe to assume only for z but not for a c , because they do not hold for the standard datasets (see Appendix H).
To account for this, we derive the variance Var [ˆ y c ] without relying on these assumptions for a c (see Appendix B):
[Equation]
From equation (6) one can see that after giving up invalid assumptions for a c , pre-logits variance
Var [˜ y c ] now became dependent on (cid:107) a c (cid:107) 2 , which is not captured by traditional Glorot & Bengio (2010) and He et al.
(2015) initialization schemes and thus leads to poor variance control.
Attributes normalization trick rectiﬁes this limitation, which is summarized in the following statement.
Statement 2 (informal) .
Attributes normalization trick leads to the same pre-logits variance as we have with Xavier fan-out initialization.
(see Appendix B for the formal statement and the derivation).
3. N ORMALIZATION IN DEEPER MODELS
What happens when P θ is not linear?
Let h c
= H ϕ ( a c ) be the output of H ϕ .
The analysis of this case is equivalent to the previous one but with plugging in h c everywhere instead of a c .
This lead to:
[Equation]
As a result, to obtain Var [˜ y c ]
=
Var [ z i ] property, we need to initialize Var [ V ij ] the following way:
[Equation]
[Equation]
This makes the initialization dependent on the magnitude of (cid:107) h c (cid:107) instead of (cid:107) a c (cid:107) , so normalizing attributes to a unit norm would not be sufﬁcient to preserve the variance.
To initialize the weights of
V using this formula, a two-step data-dependent initialization is required: ﬁrst initializing H ϕ , then computing average (cid:107) h c (cid:107) 22 , and then initializing V .
However, this is not reliable since (cid:107) h c (cid:107) 22 changes on each iteration, so we propose a more elegant solution to standardize h c
[Equation]
As one can note, this is similar to BatchNorm standardization without the subsequent afﬁne transform, but we apply it class-wise on top of attribute embeddings h c .
We plug it in right before V , i.e.
P θ ( a c ) =
V S ( H ϕ ( a c )) .
This does not add any parameters and has imperceptible computational overhead.
At test time, we use statistics accumulated during training similar to batch norm.
Standardization (9) makes inputs to V have constant norm, which now makes it trivial to pick a proper value to initialize V :
[Equation]
[Figure]
We coin the simultaneous use of (9) and (10) class normalization and highlight its inﬂuence in the following statement.
See Fig.
3 for the model diagram, Fig.
1 for empirical study of its impact and
Appendix C for the assumptions, proof and additional details.
Statement 3 (informal) .
Standardization procedure (9) together with the proper variance formula (10) , preserves the variance between z and ˜ y for a mutli-layer attribute embedder P θ .
Improved smoothness Another analysis we do is on the loss surface smoothness of P θ .
In
Appendix F we demonstrate that ZSL models tend to have more irregular loss surface compared to traditional neural classiﬁers.
This suggests another beneﬁt from the proposed class normalization.
Santurkar et al.
(2018) showed that batch-wise standardization procedure decreases the Lipschitz constant of a model, which suggests that our class-wise standardization will provide the same impact.
Indeed, we observe the smoothing effect of the proposed class normalization in our empirical study as well, which is brieﬂy shown on ﬁgure.
1 and extensively studied in Appendix F.
C ONTINUAL Z ERO -S HOT L EARNING 4. P ROBLEM FORMULATION
In continual learning (CL), a model is being trained on a sequence of tasks that arrive one by one.
Each task is deﬁned by a dataset D t = { x ti , y ti } N t
i =1 of size N t .
The goal of the model is to learn all the tasks sequentially in such a way that at each task t it has good performance both on the current task and all the previously observed ones.
In this section we develop the ideas of Chaudhry et al.
and formulate a Continual Zero-Shot Learning (CZSL) problem.
Like in CL, CZSL also assumes a sequence of tasks, but now each task is a generalized zero-shot learning problem.
This means that apart from D t we also receive a set of the corresponding class descriptions A t for each task t .
In this way, traditional zero-shot learning can be seen as a special case of CZSL with just two tasks.
In Chaudhry et al.
(2019), authors evaluate their zero-shot models on each task individually, without considering the classiﬁcation space across tasks; looking only one step ahead, which gives a limited picture of the model’s quality.
Instead, we borrow ideas from Generalized ZSL (Chao et al., 2016; Xian et al., 2018a), and propose to measure the performance on all the seen and all the unseen data for each task.
More formally:
[Equation]
r =
D r
D
[Equation]
t =
T (cid:91) r = t + D r A ≤ t = t (cid:91) r =
A r
A >t =
T (cid:91) r
= t +
A r (11) which are the datasets of all seen data (trained tasks), all unseen data (future tasks), seen class attributes, and unseen class attributes respectively.
We use subscripts “tr”/“ts” to differentiate between train/test data.
4. P ROPOSED EVALUATION METRICS
Our metrics for CZSL use GZSL metrics under the hood and are based on generalized accuracy (GA) (Chao et al., 2016; Xian et al., 2018a).
“Traditional” seen (unseen) accuracy computation discards unseen (seen) classes from the prediction space, thus making the problem easier, since the model has fewer classes to be distracted with.
For generalized accuracy, we always consider the joint space of both seen and unseen and this is how GZSL-S and GZSL-U are constructed.
Using this notion of the generalized accuracy gives use 5 following metrics. Mean Seen Accuracy (mSA) . We compute GZSL-S after tasks t = 1 , .., T and take the average: [Equation]
Mean Unseen Accuracy (mUA) . We compute GZSL-U after tasks t = 1 , ..., T − 1 (we do not compute it after task T since D >T = ∅ ) and take the average: [Equation] Mean Harmonic Seen/Unseen Accuracy (mH) .
We compute GZSL-H after tasks t = 1 , ..., T −and take the average: [Equation] [Equation] Mean Area Under Seen/Unseen Curve (mAUC) .
We compute AUSUC Chao et al.
after tasks t = 1 , ..., T − 1 and take the average: [Equation] AUSUC is a performance metric that allows to detect model’s bias towards seen or unseen data and in our case it measures this in a continual fashion. Mean Joint Accuracy (mJA) .
On each task t we compute the generalized accuracy on all the test data we have for the entire problem: [Equation] This evaluation measure allows us to understand how far behind a model is from the traditional supervised classiﬁers. A perfect model would be able to generalize on all the unseen classes from the very ﬁrst task and maintain the performance on par with normal classiﬁers.
Apart from these metrics, we also employ a popular forgetting measure (Lopez-Paz & Ranzato, 2017) to see how well a model preserves its previously acquired knowledge.
E XPERIMENTS 5. ZSL EXPERIMENTS
Experiment details .
We use 4 standard datasets: SUN (Patterson et al., 2014), CUB (Welinder et al., 2010), AwA1 and AwA2 and seen/unseen splits from Xian et al.
(2018a).
They have 645/72, 150/50, 40/10 and 40/10 seen/unseen classes respectively with d a being equal to 102, 312, 85 and 85 respectively.
Following standard practice, we use ResNet101 image features (with d z = 2048 ) from Xian et al.
(2018a).
Our attribute embedder P θ is a vanilla 3-layer MLP augmented with standardization procedure 9 and corrected output matrix initialization 10.
For all the datasets, we train the model with Adam optimizer for 50 epochs and evaluate it at the end of training.
We also employ
NS and AN techniques with γ = 5 for NS.
Additional hyperparameters are reported in Appx D. To perform cross-validation, we ﬁrst allocate 10% of seen classes for a validation unseen data (for AwA and AwA2 we allocated 15% since there are only 40 seen classes).
Then we allocate 10% out of the remaining 85% of the data for validation seen data.
This means that in total we allocate ≈ 30 % of all the seen data to perform validation.
Evaluation and discussion .
We evaluate the model on the corresponding test sets using 3 metrics as proposed by Xian et al.
(2018a): seen generalized unseen accuracy (GZSL-U), generalized seen accuracy (GZSL-S) and GZSL-S/GZSL-U harmonic mean (GZSL-H), which is considered to the main metric for ZSL.
Table 2 shows that our model has the state-of-the-art in 3 out of 4 datasets.
Training speed results .
We conduct a survey and search for open-source implementations of classiﬁcation ZSL papers that were recently published on top conferences.
This is done by 1) checking the papers for code urls; 2) checking its supplementary; 3) searching for implementations on github.com and 4) searching authors by their names on github.com and checking their repositories list.
As a result, we found 7 open-source implementations of the recent methods, but one of them got discarded since the corresponding data was stored on a private cloud.
We reran all the found methods with the provided hyperparameters and training setups on the corresponding datasets and report their training time in Table 6 in Appx D. In Table 2, we report the average training time for each of the methods.
Since our model is just a vanilla MLP and does not use any specialized optimization schemes, it trains from 30 to 500 times faster compared to other methods, while outperforming them in the ﬁnal performance.
5. CZSL EXPERIMENTS
Datasets .
We test our approach in CZSL scenario on two datasets: CUB Welinder et al.
(2010) and
SUN Patterson et al.
CUB contains 200 classes and is randomly split into 10 tasks withclasses per task.
SUN contains 717 classes which is randomly split into 15 tasks, the ﬁrst 3 tasks have 47 classes and the rest of them have 48 classes each (717 classes are difﬁcult to separate evenly).
We use ofﬁcial train/test splits for training and testing the model.
[Table]
[Table] Model and optimization We follow the proposed cross-validation procedure from Chaudhry et al.
Namely, for each run we allocate the ﬁrst 3 tasks for hyperparameter search, validating on the test data.
After that we reinitialize the model from scratch, discard the ﬁrst 3 tasks and train it on the rest of the data.
This reduces the effective number of tasks by 3, but provides a more fair way to perform cross-validation Chaudhry et al.
We use an ImageNet-pretrained ResNet-18 model as an image encoder E ( x ) .
An important difference with the ZSL setup is that we do not keep it ﬁxed and train it jointly with attribute embedder P θ .
For CZSL experiments, P θ is a 2-layer MLP and we test the proposed class normalization scheme.
All the details can be found in Appendix G.
We test our approach on 3 continual learning methods: EWC Kirkpatrick et al. (2017), MAS Aljundi et al. (2017) and A-GEM Chaudhry et al. (2019) and 2 benchmarks: Multi-Task model and Sequential model.
EWC and MAS ﬁght forgetting by regularizing the weight update for a new task in such a way that the important parameters are preserved.
A-GEM maintains a memory bank of previously encountered examples and performs a gradient step in such a manner that the loss does not increase on them.
Multi-Task is an upper bound baseline: a model which has an access to all the previously encountered data and trains on them in a multi-task fashion Sequential baseline serves as a lower bound and is model which does not employ any of the techniques.
Evaluation and discussion Results for the proposed metrics mU, mS, mH, mAUC, mJA and forgetting measure from Lopez-Paz & Ranzato (2017) are reported in Table 3 and Appendix G.
As one can observe, class normalization boosts the performance of classical regularization-based and replay-based continual learning methods by up to 100% and leads to lesser forgetting.
However, we are still far behind traditional supervised classiﬁers as one can infer from mJA metric.
For example, some state-of-the-art approaches on CUB surpass 90% accuracy Ge et al.
(2019) which is drastically larger compared to what the considered approaches achieve.
C ONCLUSION
We investigated and developed normalization techniques for zero-shot learning.
We provided theoretical groundings for two popular tricks: normalize+scale and attributes normalization and showed both provably and in practice that they aid training by controlling a signal’s variance during a forward pass.
Next, we demonstrated that they are not enough to constrain a signal from ﬂuctuations for a deep ZSL model.
That motivated us to develop class normalization: a new normalization scheme that ﬁxes the problem and allows to obtain SotA results on 4 standard ZSL datasets both in terms of quantitative performance and training speed.
Next, we showed that ZSL models tend to have more irregular loss landscape than traditional classiﬁers and that class normalization remedies this issue partially.
Finally, we generalized ZSL to a broader setting of continual zero-shot learning and proposed a set of principled metrics and baselines for it.
We believe that our work will spur the development of stronger zero-shot systems and motivate their deployment in real-world applications.
A “N ORMALIZE + SCALE ” TRICK
As being discussed in Section 3.2, “normalize+scale” trick changes the logits computation from a usual dot product to the scaled cosine similarity:
[Equation]
(17) where ˆ y c is the logit value for class c ; z is an image feature vector; p c is the attribute embedding for class c : p c =
P θ ( a c ) =
V H ϕ ( a c ) (18) and γ is the scaling hyperparameter.
Let’s denote a penultimate hidden representation of P θ as h c
= H ϕ ( a c ) .
We note that in case of linear P θ , we have h c = a c .
Let’s also denote the dimensionalities of z and h c by d z and d h .
A. A SSUMPTIONS
To derive the approximate variance formula for ˆ y c we will use the following assumptions and approximate identities:
(i) All weights in matrix V : • are independent from each other and from z k and h c,i (for all k, i ); • E
[ V ij ] =
for all i, j ; • Var [ V ij ]
= s v for all i, j .
(ii)
There exists (cid:15) > 0 s.t.
(2 + (cid:15) ) -th central moment exists for each of h c, 1 , ..., h c,d h .
We require this technical condition to be able apply the central limit theorem for variables with non-equal variances.
(iii)
All h c,i , h c,j are independent from each other for i (cid:54) =
j .
This is the least realistic assumption from the list, because in case of linear P θ it would be equivalent to independence of coordinates in attribute vector a c .
We are not going to use it in other statements.
As we show in Appendix A.3 it works well in practice.
(iv)
All p c,i , p c,j are independent between each other.
This is also a nasty assumption, but more safe to assume in practice (for example, it is easy to demonstrate that Cov [ p c,i , p c,j ] = 0 for i (cid:54)
= j ).
We are going to use it only in normalize+scale approximate variance formula derivation.
(v) z ∼ N ( 0 , s z I ) .
This property is safe to assume since z is usually a hidden representation of a deep neural network and each coordinate is computed as a vector-vector product between independent vectors which results in the normal distribution (see the proof below for p c (cid:32) N (
, s p I ) ).
vi) For ξ ∈ { z , p c }
we will use the approximations:
[Equation]
and E (cid:20)
[Equation]
i
[Equation]
This approximation is safe to use if the dimensionality of ξ is large enough (for neural networks it is deﬁnitely the case) because the contribution of each individual ξ i in the norm (cid:107) ξ (cid:107) 2 becomes negligible.
A. F ORMAL STATEMENT AND THE PROOF
Statement 1 (Normalize+scale trick) .
If conditions (i)-(vi) hold, then:
[Equation]
Proof.
First of all, we need to show that p c,i (cid:32) N (0 , s p ) for some constant s p .
Since
[Equation]
from assumption (i) we can easily compute its mean:
[Equation] [Equation]
[Equation] [Equation]
(25) =
[Equation]
(26) and the variance:
Var
[Equation]
p
[Equation] [Equation]
[Equation] [Equation]
  (30)
Using E [ V i,j V i,k ] = 0 for k (cid:54)
=
j , we have: =
[Equation]
(cid:35) (31) =
[Equation]
(cid:88)
j
[Equation]
Since s v =
Var [ V i,j ] = E (cid:2)
V 2 i,j (cid:3) − E
[ V i,j ] 2 = E (cid:2)
V 2 i,j (cid:3) , we have: =
d h (cid:88)
[Equation]
Now, from the assumptions (ii) and (iii) we can apply Lyapunov’s Central Limit theorem to p c,i , which gives us:
[Equation]
For ﬁnite d h , this allows us say that:
[Equation]
Now note that from (vi) we have:
[Equation]
Since ξ ∼ N ( 0 , s ξ ) for ξ ∈ { z , p c } , d ξ (cid:107) ξ (cid:107) 22 follows scaled inverse chi-squared distribution with inverse variance τ = 1 /s ξ , which has a known expression for expectation:
[Equation]
Now we are left with using approximation (vi) and plugging in the above expression into the variance formula:
[Equation]
A. E MPIRICAL VALIDATION
In this subsection, we validate the derived approximation empirically.
For empirical validation of the variance analyzis, see Appendix E.
For this, we perform two experiments:
• Synthetic data .
An experiment on a synthetic data.
We sample x ∼ N ( 0 , I d ) , y ∼ N ( 0 , I d ) for different dimensionalities d = 32 , 64 , 128 , ..., 8192 and compute the cosine similarity:
[Equation]
After that, we compute Var [ z ] and average it out across different samples.
The result is presented on ﬁgure 2a.
[Figure]
• Real data .
We take ImageNet-pretrained ResNet101 features and real class attributes (unnormalized) for SUN, CUB, AwA1, AwA2 and aPY datasets.
Then, we initialize a random 2-layer MLP withhidden units, and generate real logits (without scaling).
Then we compute mean empirical variance and the corresponding standard deviation over different batches of size 4096.
The resulted boxplots are presented on ﬁgure 2b.
In both experiments, we computed the logits with γ = 1 .
As one can see, even despite our demanding assumptions, our predicted variance formula is accurate for both synthetic and real-world data.
B A TTRIBUTES NORMALIZATION
We will use the same notation as in Appendix A. Attributes normalization trick normalizes attributes to the unit L 2 -norm: [Equation]
[Equation]
We will show that it helps to preserve the variance for pre-logit computation when attribute embedder P θ is linear:
[Equation]
[Equation]
For a non-linear attribute embedder it is not true, that’s why we need the proposed initialization scheme.
B. A SSUMPTIONS
We will need the following assumptions:
(i)
Feature vector z has the properties: • E [ z ] = 0 ; • Var [ z i ] =
s z for all i = 1 , ..., d z .
• All z
i are independent from each other and from all p c,j . (ii)
Weight matrix V is initialized with Xavier fan-out mode,
i
e. Var [ V ij ] = 1 /d z and are independent from each other.
Note here, that we do not have any assumptions on a c .
This is the core difference from Chang et al.
and is an essential condition for ZSL (see Appendix H).
B. F ORMAL STATEMENT AND THE PROOF
Statement 2 (Attributes normalization for a linear embedder) .
If assumptions (i)-(ii) are satisﬁed and (cid:107) a c (cid:107) 2 = 1 , then:
[Equation]
Proof.
Now, note that:
[Equation]
Then the variance for ˜ y c has the following form:
[Equation]
since s v = 1 /d z , then:
[Equation]
since attributes are normalized, i.e. (cid:107) a c (cid:107) 2 = 1 , then:
[Equation]
C N ORMALIZATION FOR A DEEP ATTRIBUTE EMBEDDER C. F ORMAL STATEMENT AND THE PROOF
Using the same derivation as in B, one can show that for a deep attribute embedder:
[Equation]
) normalizing attributes is not enough to preserve the variance of Var [˜ y c ] , because
[Equation]
and h c
= H ϕ ( a c ) is not normalized to a unit norm.
To ﬁx the issue, we are going to use two mechanisms:
A different initialization scheme:
[Equation]
Using the standardization layer before the ﬁnal projection matrix:
[Equation]
µ x , σ x are the sample mean and variance and (cid:11) is the element-wise division.
C. A SSUMPTIONS
We’ll need the assumption:
(i)
Feature vector z has the
properties
: • E [ z ] = 0 ; • Var [ z i ] = s z for all i = 1 , ..., d z .
• All z
i are independent from each other and from all p c,j .
[Equation]
C. F ORMAL STATEMENT AND THE PROOF
Statement 3.
If the assumption (i) is satisﬁed, an attribute embedder has the form P θ =
V ◦ S ◦ H ϕ
and we initialize output matrix V s.t.
Var [ V ij ] = 1 d z d h , then the variance for ˜ y c is preserved:
Var [˜ y c ]
≈ Var [ z i ] =
s z (76)
Proof.
With some abuse of notation, let ¯ h c =
S ( h c ) (in practice, S receives a batch of h c instead of a single vector).
This leads to:
E [ h c ] =and
Var [ h c ] ≈(77)
Using the same reasoning as in Appendix B, one can show that:
[Equation]
(cid:3) (78)
So we are left to demonstrate that E (cid:2) (cid:107) ¯ h c (cid:107) 22 (cid:3) =
d h :
E (cid:2) (cid:107) ¯ h c (cid:107) 22 (cid:3) =
d h (cid:88) i =
E (cid:2) ¯ h 2 c,i (cid:3) =
d h (cid:88) i =
Var (cid:2) ¯ h c,i (cid:3)
≈ d h (79)
C. A DDITIONAL EMPIRICAL STUDIES D ZSL DETAILS D. E XPERIMENTS DETAILS
In this section, we cover hyperparameter and training details for our ZSL experiments and also provide the extended training speed comparisons with other methods.
We depict the architecture of our model on ﬁgure 3.
As being said, P θ is just a simple multi-layer MLP with standardization procedure (9) and adjusted output layer initialization (10).
Besides, we also found it useful to use entropy regularizer (the same one which is often used in policy gradient methods for exploration) for some datasets:
[Equation]
We train the model with Adam optimizer with default β 1 and β 2 hyperparams.
The list of hyperparameters is presented in Table 4.
In those ablation experiments where we do not attributes normalization (2), we apply simple standardization to convert attributes to zero-mean and unit-variance.
D. A DDITIONAL EXPERIMENTS AND ABLATIONS
In this section we present additional experiments and ablation studies for our approach (results are presented in Table 5).
We also run validate :
[Table]
[Table]
Kaiming fan-out + (9) 42.8 41.2 42.0 51.0 49.0 50.0 57.0 82.6 67.4 58.7 83.5 68.
• Dynamic normalization.
As one can see from formula (8), to achieve the desired variance it would be enough to initialize V s.t.
Var [ V ij ] = 1 /d z
(equivalent to Xavier fan-out) and use a dynamic normalization :
[Equation]
between V and H ϕ , i.e. P θ ( a c ) =
V DN ( H ϕ ( a c )) .
Expectation E (cid:2) (cid:107) h (cid:107) 22 (cid:3) is computed over a batch on each iteration.
A downside of such an approach is that if the dimensionality is large, than a lot of dimensions will get suppressed leading to bad signal propagation.
Besides, one has to compute the running statistics to use them at test time which is cumbersome.
• Traditional initializations + standardization procedure (9).
These experiments ablate the necessity of using the corrected variance formula (10).
• Performance of NS for different scaling values of γ and different number of layers.
D. M EASURING TRAINING SPEED
We perform an empirical study and rerun several ZSL methods from recent conferences that publicly released their source code.
All runs are made with the ofﬁcial hyperparameters and training setups and on the same hardware: NVidia GeForce RTX 2080 Ti GPU, × 16 Intel Xeon Gold 6142 CPU and 128 GB RAM.
The results are depicted on Table 6.
[Table]
[Figure]
As one can see, the proposed method trains 50-100 faster than the recent SotA.
This is due to not using any sophisticated architectures employing generative models (Xian et al., 2018b; Narayan et al., 2020); or optimization schemes like episode-based training (Li et al., 2019; Yu et al., 2020).
E A DDITIONAL VARIANCE ANALYZIS
In this section, we provide the extended variance analyzis for different setups and datasets.
The following models are used:
A linear ZSL model with/without normalize+scale (NS) and/or attributes normalization (AN).
A 3-layer ZSL model with/without NS and/or AN. 3.
A 3-layer ZSL model with class normalization , with/without NS and/or AN.
These models are trained on 4 standard ZSL datasets: SUN, CUB, AwA1 and AwA2 and their logits variance is calculated on each iteration and reported.
The same batch size, learning rate, number of epochs, hidden dimensionalities were used.
Results are presented on ﬁgures 4, 5, 6 and 7, which illustrates the same trend:
• A traditional linear model without NS and AN has poor variance.
Adding NS with a proper scaling of 5 and AN improves it and bounds to be close to 1 .
• After introducing new layers, NS and AN stop “working” and variance vanishes below unit.
• Incorporating class normalization allows to push it back to 1 .
F ZSL MODELS HAVE MORE IRREGULAR LOSS SURFACE F. F ORMAL REASONING
We demonstrate that the loss surface of attribute embedder P θ is more irregular compared to a traditional neural network.
The reason is that its output vectors p c
= P θ ( a c ) are not later used independently, but instead
[Figure] [Figure]
[Figure] [Figure]
combined together in a single matrix W =
[ p 1 , ..., p K s ] to compute the logits vector y
=
W z .
Because of this, the gradient update for θ i receives K signals instead of just 1 like for a traditional model, where K is the number of classes.
Consider a classiﬁcation neural network F ψ ( x ) optimized with loss L and some its intermediate transformation y
= W (cid:96) x .
Then the gradient of L n on n -th training example with respect to W (cid:96)ij is computed as:
[Equation]
While for attribute embedder P θ ( a c ) , we have K times more terms in the above sum since we perform K forward passes for each individual class attribute vector a c .
The gradient on n -th training example for its inner transformation y =
W (cid:96) x
( a c ) is computed as:
[Equation]
From this, we can see that the average gradient for P θ is much larger which is a sign of more irregular loss surface.
This gives additional motivation to incorporate the proposed class normalization technique.
Santurkar et al.
(2018) showed that batch-wise standardization procedure smoothes the loss surface of neural networks and we demonstrate that our class-wise standardization gives the same beneﬁt in our case as well.
F. E MPIRICAL VALIDATION
To validate the above claim empirically, we perform the following experiment.
We used three models:
A vanilla MLP classiﬁer i.e.
without any class attributes.
For each dataset, it receives feature vector z and produces logits.
A vanilla MLP zero-shot classiﬁer, as described in section 3. 3.
An MLP zero-shot classiﬁer with class normalization.
All three models were trained with cross-entropy loss with the same optimization hyperparameters: learning rate of 0.0001, batch size of 256, number of iterations of 2500.
They had the same numbers of layers, which was equal to 3.
The results are illustrated on ﬁgures 1 (left) and 8.
As one can see, traditional MLP models indeed have more ﬂat loss surface which is observed by a small gradient norm.
But class normalization helps to reduce the gap.
G C ONTINUAL Z ERO -S HOT L EARNING DETAILS
As being said, we use the validation sequence approach from Chaudhry et al.
to ﬁnd the best hyperpa- rameters for each method.
We allocate the ﬁrst 3 tasks to perform grid search over a ﬁxed range.
After the best hyperparameters have been found, we train the model from scratch for the rest of the tasks.
The hyperparameter range for CZSL experiments are presented in Table 7 (we use the same range for all the experiments).
We train the model for 5 epochs on each task with SGD optimizer.
We also found it beneﬁcial [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
with the number of epochs equal to the number of epochs per task.
As being said, for CZSL experiments, we use an ImageNet-pretrained ResNet-18 model as our image encoder.
In contrast with ZSL, we do not keep it ﬁxed during training.
The results for our mS, mU, mH, mJA, mAUC metrics, as well as the forgetting measure (Lopez-Paz & Ranzato, 2017) are presented on ﬁgures 9 and 10.
As one can clearly see, adding class normalization signiﬁcantly improves the results, at some timesteps even surpussing the multi-task baseline without ClassNorm.
H W HY CANNOT WE HAVE INDEPENDENCE , ZERO - MEAN AND SAME - VARIANCE ASSUMPTIONS FOR ATTRIBUTES IN ZSL?
Usually, when deriving an initialization scheme, people assume that their random vectors have zero mean, the same coordinate variance and the coordinates are independent from each other.
In the paper, we stated that these are unrealistic assumptions for class attributes in ZSL and in this section elaborate on it.
Attribute values for the common datasets need to be standardized to satisfy zero-mean and unit-variance (or any other same-variance) assumption.
But it is not a sensible thing to do, if your data does not follow normal distribution, which is the case for the attribute vectors for standard ZSL datasets, which is clearly seen from Figure 13.
To put it more rigorously, in this section we report two statistical results:
• Results of a normality test based on D’Agostino and Pearson’s tests, which comes with scipy python stats library.
We run it for each attribute dimension for each dataset and report the distribution of the resulted χ 2 -statistics with the corresponding p -values on Figure 11.
• Compute the distribution of absolute values of correlation coefﬁcients between attribute dimensions.
The results are presented on Figure 12 which demonstrates that attributes dimensions are not independent between each other in practice [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Figure]
This ﬁgure shows that attributes are not independent that’s why it would be unreasonable to use such an assumption.
If attributes would be independent from each other, that would mean that, for example, that “having black stripes” is independent from “being orange”, which tigers would argue not to be a natural assumption to make.
[Figure]
We note, however, that attributes distribution is uni-modal and, in theory, it is possible to transform it into a normal one (by hacking it with log/inverse/sqrt/etc), but such an approach is far from being scalable.
It is not scalable because transforming a non-normal distribution into a normal one is tricky and is done either manually by ﬁnding a proper transformation or by solving an optimization task.
This is tedious to do for each dataset and thus scales poorly.