Tight Lower Bounds on Worst-Case Guarantees for Zero-Shot Learning with Attributes
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We develop a rigorous mathematical analysis of zero-shot learning with attributes.
In this setting, the goal is to label novel classes with no training data, only detectorsfor attributes and a description of how those attributes are correlated with thetarget classes, called the class-attribute matrix.
We develop the ﬁrst non-triviallower bound on the worst-case error of the best map from attributes to classes forthis setting, even with perfect attribute detectors.
The lower bound characterizesthe theoretical intrinsic difﬁculty of the zero-shot problem based on the availableinformation—the class-attribute matrix—and the bound is practically computablefrom it.
Our lower bound is tight, as we show that we can always ﬁnd a randomizedmap from attributes to classes whose expected error is upper bounded by thevalue of the lower bound.
We show that our analysis can be predictive of howstandard zero-shot methods behave in practice, including which classes will likelybe confused with others.
Introduction
Labeled training data is often scarce or unavailable, and it can be very costly to obtain.
For thisreason, there is a growing interest in developing methods that can exploit source of informationother than labeled data, such as zero-shot learning (ZSL).
In ZSL, we want to recognize items ofunseen classes , for which labeled data is not available.
A ZSL model is trained on a disjoint set ofsimilar classes, called seen classes , for which labeled data is available instead.
The model is trainedto map examples to auxiliary information describing the seen classes.
Then, at test time, predictionscan be made using only descriptions of the unseen classes.
While ZSL is increasingly common inpractice from a theoretical perspective ZSL is a hard problem that deﬁes analysis, because in theworst case there can be an arbitrary shift between the distributions of the seen and unseen classes.
In this work we take a step towards a better theoretical understanding of ZSL.
We investigate thequestion: Given only auxiliary information in the form of attributes describing unseen classes, whatis the smallest worst-case error than any method can guarantee?
We provide the ﬁrst non-trivialanswer to this question by developing a framework based on adversarial optimization.
We also showthat this framework has practical application as a method for identifying when the predictions of ZSLmethods on certain unseen classes are more likely to be incorrect.
ZSL models have obtained impressive accuracy in practice, both for vision (Xian et al.,
2018a)and language domains (Sanh et al., 2022; Wei et al., 2022), but they come with no theoreticalcharacterization of their accuracy.
To address this gap, we analyze the attribute-based ZSL settingthat includes a large portion of the classic methods proposed in the literature (Romera-Paredes &
Torr, 2015; Lampert et al., 2014; Akata et al., 2015, 2016), as well as more recent end-to-end deeplearning approaches (Kodirov et al., 2017; Xian et al., 2018a; Huynh & Elhamifar, 2020).
While thissetting does not include all varieties of ZSL (discussed further in Section 2), we view this work as a critical ﬁrst step towards building up a broader theory of ZSL.
In attribute-based ZSL, an attributeis a property of a item to be classiﬁed.
Each item can either exhibit a given attribute or not.
Forexample, an image of a lion would often exhibit the attribute tail, while the image of a sheep wouldnot.
Attribute-based ZSL models are trained using attribute representations of the items of the seenclasses, and a class-attribute matrix that describes pairwise relations between the seen classes andeach attribute.
At test time, predictions are made for the unseen classes given the items’ attributerepresentation and a new class-attribute matrix.
Romera-Paredes & Torr (2015) is one of the few works to address theoretical questions related to
ZSL.
Studying attribute-based ZSL, they show a pair of basic bounds that characterize sufﬁcientconditions for either learning or impossibility: (1) if there is no shift from the seen to the unseenclasses, then learning is trivial, and (2) if the vectors of attributes of the seen classes are mutuallyorthogonal with those of the unseen classes, then the error can be arbitrarily large in the worst case.
In this paper, we provide the ﬁrst non-trivial lower bound for ZSL with attributes, addressing theopen problem posed by Romera-Paredes & Torr (2015).
We analyze ZSL with attributes by ﬁrst observing that it is a two stage process consisting of a trainingphase and an inference phase.
In the training phase, we learn a map from the items to the attributespace using the seen classes, while in the inference phase we use the class-attribute matrix to infer thecorrect class given the item-attribute representation.
Based on this two-stage decomposition, we canidentify two kinds of errors.
The ﬁrst kind, related to the training phase, is due to domain shift.
Themap from items to the attribute space that is trained on the seen classes might not generalize accuratelyto the unseen classes.
This contribution to the error can be arbitrarily large without introducingstrong assumptions, as no labeled data is available for the unseen classes.
Thus it is impossibleto characterize the domain shift between seen and unseen classes.
The second kind, related to theinference phase, is due to the fact that the class-attribute matrix might not fully differentiate amongthe unseen classes.
In particular, there can be an item of the unseen classes with a set of attributesthat according to the class-attribute matrix relation conforms with the description of two differentclasses.
The ﬁrst kind of error, domain shift, has been extensively studied both in the theory and theexperimental literature (Mansour et al., 2009;
Ben-David et al., 2010; Sener et al., 2016; Pinheiro,2018; Luo et al., 2019).
In this work, for the ﬁrst time, we theoretically characterize the contributionof the second kind of error.
It is important to understand and characterize this error for speciﬁc ZSLtasks because it corresponds to an inherent information gap in the problem setting that cannot becircumvented with a smarter algorithm.
We provide tight lower and upper bounds on the worst-case error of the best map from attributerepresentations to classes based on the class-attribute matrix.
Our analysis gives a lower bound inthe sense that it bounds from below the minimum error that any method can guarantee given onlythe information of the class-attribute matrix.
The class-attribute matrix speciﬁes the fraction ofitems in each class that exhibit each attribute.
There is a range of class-attribute distributions thatsatisfy the constrains deﬁned by a given matrix.
We give a lower bound on the error of the bestpossible method for the worst case distribution in that range.
This distribution represents a worst casecorrelation between attributes that satisﬁes the class-attribute matrix while maximizing the difﬁcultyto distinguish between attribute-representation of items belonging to distinct classes.
Our analysisalso gives an upper bound in the sense that we show a randomized classiﬁer that achieves at most theerror of the lower bound, assuming perfect item-to-attribute mapping.
This also shows that the lowerbound is tight.
Interestingly, the value of the lower bound can also be interpreted as the quality of theinformation provided by the class-attribute matrix.
To the best of our knowledge, this is the ﬁrst workto quantify such information.
Contributions.
Our main contributions are the following:
We show the ﬁrst non-trivial lower bound for attribute-based ZSL (Section 4).2.
We formulate the lower bound given a class-attribute matrix as a linear program (Section 4.1).3.
We show a closed form expression for the lower bound for binary classiﬁcation (Section 4.2).4.
We show that the lower bound is tight: we exhibit a randomized classiﬁer whose expected
error is upper bounded by the value of the lower bound (Section 4.3).
We run extensive experiments comparing the theoretical results with the error of popular
attribute-based ZSL methods, on benchmark datasets.
We show that information given by
the bound can be predictive of how standard methods behave, including which classes will
likely be confused with others (Section 5).
Much early work on ZSL focused on using logical descriptions of the classes as auxiliary information,including attributes (Chang et al., 2008; Lampert et al., 2009).
Since then, an increasing numberof ZSL methods have been proposed, which differ in methodology and the auxiliary informationthey use.
Examples of auxiliary information are symbolic descriptions of classes (Chang et al.
,2008; Lampert et al., 2009), pre-trained embedding of the classes (Frome et al., 2013), naturallanguage descriptions (Obeidat et al., 2019; Brown et al., 2020), and knowledge graphs (Wang et al.
,2018; Kampffmeyer et al., 2019; Nayak & Bach, 2020).
Recent ZSL methods can be grouped intotwo main categories: embedding-based and generation-based (Pourpanah et al., 2020).
Seminalembedding-based works used two-layer neural networks to link the image feature space to thesemantic one (Socher et al., 2013).
Later, they evolved into deep neural networks that either mapsemantic features into the visual space (Ba et al., 2015; Zhang et al., 2017; Changpinyo et al.,
2017)or project both the image and semantic features into the same space (Zhang & Saligrama, 2015;
Radford et al., 2021).
Generative-based approaches employ various kind of Generative Adversarial
Networks (GANs) (Mirza & Osindero, 2014) to synthesize the features of the unseen classes, and usethem to train a ZSL classiﬁer in a supervised fashion (Felix et al., 2018;
Li et al., 2019; Xian et al.
,2018b, 2019; Narayan et al., 2020).
ZSL with attributes generally consists of learning a linear map from the item to the attribute space, inthe ﬁrst stage.
Then, we use the class-attribute matrix to infer the correct class given the item-attributerepresentation (Xian et al., 2018a).
ZSL with attributes can be seen as a special case of embeddingbased ZSL, in which the class embeddings are the rows of the class-attribute matrix.
Analyzing moregeneral embedding-based or generation-based ZSL methods is challenging because they rely on deepneural networks for which relatively little theory is available.
Inspired by previous work to describe classes using error-correcting output codes (Dietterich & Bakiri,1994), Palatucci et al.
(2009) were the ﬁrst to propose a ZSL algorithm for which they can provide atheoretical analysis.
The algorithm learns linear classiﬁers individually for each binary attribute, andthe attributes are mapped to the closest class-attribute representation.
While they are able to providea PAC bound, their analysis relies on several strong assumptions that limit the problem setting.
First,they assume that they can learn each attribute independently, but attribute dependency is a widelyrecognized problem for attribute detection (Jakulin & Bratko, 2003).
Second, they assume that eachclass has a unique attribute representation, i.e. each attribute must be either present or not in all theitems of a given class.
Finally, they also assume that they are able to sample classes from a givendistribution, and they are able to generalize to the non-sampled classes.
That is, they do not separatebeforehand between seen and unseen classes, which is the common scenario observed in ZSL settings.
Conversely, our lower bound does not assume a unique binary representation for each class, as we aregiven a class-attribute matrix that provides the probabilities to observe an attribute given an item of aclass.
Also, our lower bound takes into account the possible correlation between attributes, and it iscomputed based on the information provided on the given unseen classes.
In more recent work, Romera-Paredes & Torr (2015) draw a connection between transfer learning(Ben-David et al., 2010) and ZSL to provide a novel theoretical result.
In particular, they show thattheir model is not able to generalize if the attribute representations of the seen classes are orthogonalto the one of the unseen classes.
Intuitively, if those representations are orthogonal, the attributemap learned for the seen classes would fail to provide information for the unseen classes.
Thisis an impossibility result, and it is not able to arbitrarily quantify the information given for theunseen classes.
Unfortunately, transfer learning or domain adaptation like-bounds are challengingto estimate in a ZSL setting.
In fact, a term of those bounds require access to labeled data for theunseen classes, which is unavailable in ZSL.
Another term, the discrepancy, depends on the differencebetween the attribute representations of the classes and the distribution of the items between seenand unseen.
While it would be theoretically possible to compute the discrepancy based on theinformation available, its computation is very challenging and it has been possible only in veryspeciﬁc cases (Mansour et al., 2009).
Our novel lower bound is developed using adversarial techniques that describe the worst-case scenariowith respect to the information available.
It is inspired by recent work on semi-supervised learning,where the goal is to use the information provided by weak supervision sources (Balsubramani &
Freund, 2015; Arachie & Huang, 2021; Mazzetto et al., 2021b,a).
The adversarial approach allows usto handle the possible dependencies between the attributes.
Preliminaries
We denote scalar and generic items using lowercase letters, vectors using lowercase bold letters,and matrices using bold uppercase letters.
Given two vectors v and v 0 , we denote with vv 0 theconcatenation of the two vectors.
For any n 2 N , we denote with [ n ] the set { 1 , . . . , n } .
Due to spaceconstraints, all proofs are deferred to the appendix.
Let D be a distribution deﬁned over the classiﬁcation domain X .
A multiclass classiﬁcation task isdeﬁned by labeling function y : X !
Y =
[ k ] that maps each item x 2 X to a class j in the labelspace Y , where k  2 .
We say that a multiclass classiﬁcation task is balanced if for each j 2 [ k ] ,it holds that P x ⇠ D [ y ( x ) =
j ] = 1 /k .
Unless otherwise stated, we assume that the classiﬁcationtask is balanced.
This assumption is not restrictive, and as we will observe later, it can be changedif a different prior is known on the class probabilities.
We will show that our lower bound holdseven if we do not assume balanced classes.
We also assume to have access to n attribute functions1 , . . . , n , where i : X ! { 0 , 1 } for i 2 [ n ] .
We say that a classiﬁcation item x 2 X has attribute 161 i 2 [ n ] if i ( x ) = 1 .
For ease of notation, we deﬁne ( x ) .
= ( 1 ( x ) , . . . , n ( x )) T .
The codomainof is { 0 , 1 } n , and it is referred to as attribute space .
All the information about the target unseenclasses available to the algorithm is encoded in a class-attribute matrix A 2 [0 , 1] k ⇥ n .
The matrixprovides information on the relations between classes and attributes.
In particular for a class j 2 [ k ] ,and an attribute i 2 [ n ] , A j,i is the probability that i ( x ) = 1 given that y ( x )
=
j , i.e.,
A
[Equation]
An attribute-class classiﬁer g is a map from vectors in the attribute space to classes, i.e., g : { 0 , 1 } n !
[ k ] .
The error of g is " ( g ) .
= P x ⇠ D [ g  ( x ) 6 = y ( x )] .
Let G be a collection of all the possibledeterministic maps { 0 , 1 } n !
[ k ] from the attribute space to the k classes.
We are interested inevaluating min g 2 G " ( g ) .
As we focus on the contribution of the information provided by the classattribute matrix, we assume access to the attribute functions 1 , . . . , n .
In practice, the map to theattribute space is learned on the available labeled data for the seen classes (Lampert et al., 2014;
Romera-Paredes & Torr, 2015), and it is likely noisy, and can be cause of additional error.
Let p ⇤ be the (unknown) probability mass function (PMF) of the random vector( 1 ( x ) , . . .
, n ( x ) , y ( x )) where x ⇠ D .
The support of p ⇤ is { 0 , 1 } n ⇥ [ k ] .
For v 2 { 0 , 1 } n ,and j 2 [ k ] , let p
⇤ ( v , j ) .
= P x ⇠ D [ ( x ) = v ^ y ( x ) = j ] .
The error of g is a function of p ⇤ :
[Equation]
A function g ⇤ 2 G that attains minimum error " ( g ⇤ ) =
min g 2 G " ( g ) is a Bayes optimal classiﬁer 177 with respect to p ⇤ , i.e. for each v 2 { 0 , 1 } n , we have that g ⇤ ( v ) =
arg max j 2 [ k ] p ⇤ ( v , j ) .
Thus,
[Equation]
We do not have access to labeled data for the unseen classes, so we cannot estimate p ⇤ .
Instead, weconstruct a lower bound with respect to the set of all distributions that ﬁt the available information.
Lower Bounds for Zero-Shot Learning with Attributes
In this section, we formally deﬁne our lower bound.
Consider a PMF p with support over { 0 , 1 } n ⇥ [ k ] .
We say that p satisﬁes the class-attribute matrix A if (as constraints (1)) for each
i 2 [ n ] and j 2 [ k ] ,
[Equation]
[Equation]
Recall that p is balanced if for each j 2 [ k ] , it holds that
P v p ( v , j ) = 1 /k .
Let P ( A ) be the setof all possible PMFs p with support over { 0 , 1 } n ⇥ [ k ] that satisfy (4) and are balanced.
Clearly,the unknown true distribution, p ⇤ 2 P ( A ) .
The set P ( A ) can be interpreted as the collectionof all the PMFs of the random vector ( 1 , . . . , n , y ) that satisfy the constraints imposed by theinformation available on the prediction task and on the attribute functions.
While the matrix A provides precise information on the correlation between any pair of attribute function and class,it fails to provide information on the correlation between attribute functions, i.e., it does not fullyspecify the distribution p
⇤ .
Without additional information, any PMF in P ( A ) could be equal to p ⇤ .
Similarly to (2), given a PMF p 2 P ( A ) and an attribute-class classiﬁer g 2 G , we can deﬁne theerror of g with respect to the distribution p as
[Equation]
Following the computation in (3), the error of the best map from attributes to classes with respect top 2 P ( A ) is computed as
[Equation]
We are interested in the quantity
[Equation]
i.e., Q is the maximum over all distributions p 2 P ( A ) of the error of the best algorithm fordistribution p .
In other words, Q is the worst Bayes error with respect to all the distributions thatsatisfy the constraints imposed by the class-attribute matrix and on the class probabilities.
Sincep ⇤ can be any vector in 2 P ( A ) , the value Q represents a lower bound to the best error rate thatan algorithm can guarantee.
In fact, without further information on the attribute functions or theprediction task, it is possible that p ⇤ attains the maximum of (7), that is in the worst-case we have that
[Equation]
In other words, the quantity Q reﬂects a worst-case scenario where the attribute functions arecorrelated in such a way that it is hard to distinguish between the classes, even if the attributefunctions still satisfy the constraints (1) given by the class-attribute matrix A .
In Section 4.3, weshow that this lower bound is tight.
In particular, we prove that there exists a randomized classiﬁerfrom the attribute space { 0 , 1 } n to the classes [ k ] whose expected error is at most Q with respect toany distribution p 2 P ( A ) .
Example.
Consider a balanced binary classiﬁcation task with two attributes.
The class-attributematrix A 2 R 2 ⇥ 2 is such that A i,j = 1 / 2 for i, j 2 { 1 , 2 } .
Based on this class-attribute matrix, weconsider two different scenario.
In the ﬁrst scenario ( best-case ), we have that items from the ﬁrstclass have either both attributes or none with probability 1 / 2 , and items from the second class haveonly either the ﬁrst attribute or the second attribute with probability 1 / 2 .
In this case, we can simplycount the number of attributes that an item has to assign it to the correct class.
In the second scenario( worst-case ), each item has either both attributes or none with probability 1 / 2 independently from theitem class.
In this case, any mapping from the attributes to the class is going to incur an error of 1 / 2 .
4.1 Computing the Lower Bound
In this subsection, we show how to compute Q as in (7) through a Linear Program (LP).
To describe ageneric PMF p , we introduce 2 n ⇥ k variables q v ,j with v 2 { 0 , 1 } n and j 2 [ k ] .
We use additional 219 2 n auxiliary variables  v , for v 2 { 0 , 1 } n , to denote the maximums of (6), i.e.  v = max j 2 [ k ] q v ,j .
The LP is formulated as follows.
[Equation] [Equation]
Theorem 4.1.
The optimal value of the LP (8) is equal to 1  Q , with Q is as in (6) .
By removing or modifying constraint ( b ) of the LP, it is possible to remove the assumption thatthe classes are balanced or provide different class weights.
All the previous results still hold bychanging the deﬁnition of P ( A ) accordingly.
It is important to point out that since we are computinga worst-case lower bound, the class weights provide signiﬁcant information.
Without constraintson the class weights, the worst-case distribution could concentrate all the probability mass on fewclasses that are hard to differentiate using the available class-attribute matrix A .
The LP has O ( k · 2 n ) variables and constraints, and therefore it is computationally expensive forlarge number of attributes.
The dependency on 2 n is required to describe all the possible correlationsbetween the output of the n attribute functions.
Nevertheless, we present an efﬁcient computation forthe binary case and an efﬁcient approximation for the general case.
4.2 Lower Bound for Binary Classiﬁcation
In this subsection, we show how to efﬁciently compute Q as in (7) in the case of a binary classiﬁcationtask, i.e. k = 2 and A = [0 , 1] 2 ⇥ n .
For ease of notation, let
[Equation] [Equation]
Theorem 4.2.
Consider a balanced binary classiﬁcation task and let A be as in (9) .
Let Q be as in(7) .
It holds that Q =
[Equation]
[ n ] |  i  ↵
[Equation]
|  .
Moreover, let g a be the attribute-class classiﬁer
[Equation]
for each v 2 { 0 , 1 } n , where i ⇤ = arg max i |  i  ↵ i | and v i is the i -th component of the vector v .
Then " ( g a , p ) =
Q for all p 2 P ( A ) , i.e. the lower bound Q is tight.
The theorem shows that in the worst-case, the attributes could be correlated in such a way that it isnot possible to do better than deciding solely based on the attribute with the largest gap between itsprobabilities in the two classes.
This result also formally proves that for binary classiﬁcation, theworst-case is determined by a single attribute, and there is no compounded beneﬁt in having multipleattributes.
This result is in line with other worst-case analyses in the context of weak supervision.
In
Mazzetto et al.
(2021b), it is noted that while combining the output of different weak supervisionsources to obtain a noisy label of a given input item, in the worst-case one cannot do better than justusing the most accurate weak supervision source without additional information.
In Appendix C, weshow how to approximate the lower bound in the multiclass setting by using Theorem 4.2.
4.3 Lower Bound is Tight
In this subsection, we prove that the worst-case lower bound (7) is tight.
We show a randomizedattribute-class classiﬁer whose expected error is upper bounded by Q with respect to any distributionp 2 P ( A ) .
This classiﬁer can be computed only based on the class-attribute matrix, and it providesan upper bound to the error of the best map from attributes to classes that matches the lower bound.
We consider the family G R of all randomized attribute-class classiﬁers, where each g 2 G R is a 254 random map from { 0 , 1 } n to [ k ] .
A attribute-class classiﬁer in G R is described with a right-stochasticmatrix W 2 [0 , 1] 2 n ⇥ k , where the rows are indexed by binary vectors v 2 { 0 , 1 } n , and the columnsare indexed by the classes j 2 [ k ] .
The entry W v ,j represents the probability of the randomizedclassiﬁer to output j given that the input is v .
We will use g W to denote the randomized classiﬁerin G R that is described with the right-stochastic matrix W .
Given a PMF p over { 0 , 1 } n ⇥ [ k ] , wedeﬁne the expected error of g W as
[Figure]
[Equation]
We can observe that the deﬁnition above extends deﬁnition (5), in fact (10) coincides with (5) if g Wis a deterministic classiﬁer, i.e. each row of W contains a 1 .
Theorem 4.3.
There exists a randomized attribute-class classiﬁer g a 2 G R such that its worst-caseexpected error is upper bounded by Q , i.e. max p 2 P ( A ) " ( g a , p )  Q , where Q is computed as in (7) .
Also, it holds max p 2 P ( A ) min g 2 G R " ( g, p ) =
Q , i.e. the lower bound Q also applies to the family of 265 randomized functions
G R .
It is possible to compute the randomized attribute-class classiﬁer g a that satisﬁes Theorem 4.3 solelybased on the matrix A through Linear Programming using O ( k · 2 n ) variables and constraints.
Dueto space constraints, we defer this computation to Appendix B.
Empirical Applications
In this section we compare our novel theory with the performance of popular attribute-based ZSLmethods.
Our lower bounds quantify the lowest error rate that any ZSL algorithm can guaranteebased on the information provided by the class-attribute matrix.
In practice, We show that the lowerbound is still predictive of the performance and the behaviour of attribute-based ZSL algorithms.
Werun two set of experiments.
Comparing the lower bound
and the empirical error (Section 5.2).
We
compare
the errorrates of ZSL models with attributes
with
the lower
bound on the error from Section
Pairwise misclassiﬁcation prediction (Section 5.3).
We measure the predictive power of
our lower
bounds
to
identify
pairs of
classes
that
ZSL models
are likely to misclassify.
This
hardness is measured using the lower bound on the error for a pair of classes (Section 4.2).
5.1 Experimental Setup
In this section, we brieﬂy describe the experimental setup.
Further details about the datasets andthe methods can be found in Appendix D.
We choose the following four datasets with attributesthat are widely used benchmarks in ZSL:
Animals with Attributes 2 ( AwA2 ) (Xian et al.
, 2018a),aPascal-aYahoo ( aPY ) (Farhadi et al., 2009), Caltech-UCSD Birds-200-2011 ( CUB )
(Wah et al.,2011), and SUN attribute database ( SUN ) (Patterson et al., 2014).
We focus on classic ZSL algorithmswith attributes that use at most the information in the class-attribute matrix for the unseen classes:
DAP (Lampert et al., 2014), ESZSL (Romera-Paredes & Torr, 2015), SAE (Kodirov et al., 2017),
ALE (Akata et al., 2016), SJE (Akata et al., 2015).
We choose these methods because they use theclass-attribute matrix that is the focus of our theoretical analysis.
Many other ZSL methods havebeen proposed in recent years (see Section 2), but their comparison with our lower bound would bevacuous as they often use other source of auxiliary information on the unseen classes, and thus donot ﬁt within our novel theoretical framework.
They are beyond the scope of this ﬁrst analysis of
ZSL.
However, we also run experiments on a more recent attribute-based method DAZLE (Huynh &
Elhamifar, 2020) which takes advantage of additional information, i.e., attribute semantic vectors.
5.2 Comparing Lower Bound and Empirical Error
In this section, we compare the lower bound presented in Section 4 with the actual error of the ZSLmodels.
To this end, we run two set of experiments: a ﬁrst set using the ZSL datasets mentioned inthe previous subsection, and a second set using adversarially generated synthetic data that conformwith the class-attribute matrices of those same ZSL datasets.
In the ﬁrst set of experiments, we follow the standard way to evaluate ZSL models.
We train ourmodel on the seen classes, and then compare our lower bound with the empirical error of the ZSLmodels on the unseen classes.
Since the computation of the lower bound is very expensive for alarge number of attributes (Section 4.1), we focus on a subsets of them.
We propose the followinggreedy strategy to ensure a selection of attributes that are informative with respect to the target classes.
Starting with no attributes, we iteratively add the attribute that decreases the most the value of thelower bound, up to 15 attributes.
Due to the large number of seen classes of SUN and CUB, werestrict them to a smaller random subset (see Appendix D).
In the ﬁrst row of Figure 1, we reportresults for aPY, AwA2, and CUB, due to space constraints.
The results for SUN are similar andin Figure 3 in Appendix E.
We observe that the value of the lower bound can be signiﬁcantly lower than the error rate of the ZSLmodels.
This gap is most probably due to the fact that the learned map from images to attributes doesnot generalize perfectly to the unseen classes.
In fact, in this setting we can identify two main sourceof error for the ZSL models: (1) the arbitrary error due to the domain shift, and (2) the error due tohow discriminative is the attribute space to differentiate between the different classes.
Our lower bound only addresses the latter, as no method can guarantee a smaller error than the lower bound tomap from attributes to classes given only the information of the class-attribute matrix.
Nonetheless,for CUB and SUN we observe that the empirical error of ZSL models roughly follow the trend ofthe lower bound.
This suggests that the lower bound can still be used as a tool to capture how theadditional information provided by an attribute leads to improvements of the ZSL models.
In the second set of experiments, we empirically demonstrate our theory by showing that even if weminimize the error due to domain shift, there exists data for which no method can do better than ourlower bound.
To this end, for each dataset we adversarially generate synthetic data with attributevalues satisfying the dataset’s class-attribute matrix.
Speciﬁcally, we use the same class-attributematrix with 15 attributes as in the previous set of experiments in order to compute the adversarialdistribution p over attributes and classes according to the linear program introduced in Section 4.1.
The data is generated by sampling attribute-class pairs from this distribution, and using the attributevector as the feature vector.
In order to minimize the error due to domain shift, this distribution is usedto generate data for both training and testing of the ZSL methods, and the same class-attribute matrixis used for both seen and unseen classes.
We report additional details on this experimental setup andsynthetic data generation in Appendix D.
We report the results of the experiments in the second rowof Figure 1, iterating over the same attributes greedily selected in the ﬁrst set of experiments for eachdataset.
(For this set of experiments, we do not report results for DAZLE as this method relies on theinput items being images, so it does not apply to our synthetic data.)
In this case, the methods are ableto achieve errors that are comparable with the lower bound as we minimized the error due to domainshift.
This experiment empirically validates that even in the absence of domain shift, there exists adistribution of the data that satisfy the constraints imposed by the class-attribute matrix for which nomethod can do better than the lower bound.
This adversarial distribution represent an intrinsic errorgap due to the quality of the information provided by the class-attribute matrix.
This is the ﬁrst workto quantify such information in ZSL.
5.3 Pairwise Misclassiﬁcation Prediction
Theorem 4.2 shows how to efﬁciently compute the lower bound on the error to distinguish betweena pair of classes given the class-attribute matrix.
In addition to the overall bound on error, it alsogives us ﬁne-grained information about which classes are harder to distinguish among.
We deﬁne thepairwise lower bound error matrix L , whose entry L j,j 0 is the lower bound on the error computedas in Section 4.2, for all classes j, j 0 2 [ k ] , j 6 = j 0 .
A large entry L j,j 0 between two classes j 6 = j 0 the with the by the discussed particular a misclassiﬁcation
P
[Equation]
) =
[Equation]
for misclassify item the j j versa
In Figure 2, we plot L together with the misclassiﬁcation matrices M of two ZSL methods: DAPand ESZSL, computed on the unseen classes of aPY.
The pairwise lower bound matrix L has largevalues within multiple groups of semantically similar classes, e.g., animals and transportation means.
This is in line with human intuition, as we expect visually similar classes to exhibit similar attributes.
Correspondingly, the misclassiﬁcation matrices of DAP and ESZSL highlight the presence of manyerrors for classes belonging to these groups.
We also note that ZSL models could misclassify otherpairs of classes due to other source of errors, such as an inaccurate map from image to attributes.
Wereport additional experimental analysis on all datasets in Appendix E.
Conclusions, Limitations, and Future Work
We present the ﬁrst non-trivial lower bound on the best error that an attribute-based ZSL method canguarantee given the information provided—the class attribute matrix.
While our method is limitedto class-attribute matrices, it constitutes a ﬁrst theoretical building block to quantify the auxiliaryinformation provided in ZSL.
In general, theoretical evaluation of the error of ZSL models remains ahard problem due to the arbitrary domain shift between seen and unseen classes, and the wide rangeof possible auxiliary information used.
As a future direction, it remains an open problem to be able toquantify this information for other families of ZSL methods.
However, our analysis readily extendsto other variants of ZSL, such as generalized ZSL, where we simply use the class-attribute matrix ofthe union of both seen and unseen classes while computing our lower bound.
Broader Societal Impacts Zero-shot learning is now a popular scenario in research, with potentialapplication to real-world language and vision tasks.
Worse-case guarantees have long been desiredin ZSL.
Any improvement in the rigor of claims about model performance has impact because itdemonstrates both what performance can be achieved and that some solutions are invalid.
However,such bounds do not cover many kinds of error, such as a generalization gap from domain shift orlabel errors.
Further, it is important that bounds are correctly interpreted such that no false claims orconﬁdences are drawn from our ﬁndings.
An educated interpretation of the effect of these boundsupon any particular machine learning application is still required.