W ITH recent advancements in image processing and computer vision, deep learning (DL) models have achieved extensive popularity due to their capability for providing an end-to-end solution from feature extraction to classification. Despite their success, training DL models with supervised learning requires labeled data, while it is a challenging issue to collect large-scale labelled samples. As an example, ImageNet [1] , which is a large data set, contains 14 million images with 21,814 classes in which many classes contain only few images. In addition, supervised learning can only recognize samples belonging to the classes that have been seen during the training phase, and they are not able to handle samples from unseen classes [2] . While in many real-world scenarios, there may not be a significant amount of labeled samples for all classes. On one hand, finegrained annotation of a large number of samples is laborious and it requires an expert domain knowledge.On the other hand, many categories are lack of sufficient labeled samples, e.g., endangered birds, or being observed in progress, e.g., COVID-19, or not covered during training but appear in the test phase [3] , [4] , [5] , [6] .
A number of techniques for various learning configurations have been developed. One-shot [7] and few-shot [8] learning techniques can learn from classes with a few learning samples. These techniques use the knowledge obtained from data samples of other classes and formulate a classifi-• F. Pourpanah versus generalized zero-shot learning (GZSL) (adapted from [14] ).
cation model for the classes with few samples. The open set recognition (OSR) [9] techniques are able to identify whether a test sample belongs to the unseen classes, but they are not able to predict an exact class label. The class incremental learning [10] techniques consider the classification problems where data samples from previously unseen classes incrementally appear after model learning. However, none of the above-mentioned techniques can classify samples from unseen classes. In contrast, human can recognize around 30,000 categories [11] in which they don't need to see all these categories in advance. For instance, a child can easily recognize zebra, if he/she has seen horses before and known that zebra looks like a horse with black and white strips. Zero-shot learning (ZSL) [12] , [13] techniques offer a good solution to address such challenge. ZSL aims to train a classification model that can classify objects of unseen classes (target domain) via transferring knowledge obtained from other seen classes (source domain) with the help of semantic information. The semantic information embeds the names of both seen and unseen classes in high-dimensional vectors. The most common semantic information includes manually defined attribute vectors [15] , automatically extracted word vectors [16] , context-based embedding [17] , or their combinations [18] , [19] . Note that the seen and unseen classes are disjoint from each other. In other words, ZSL attempts to use semantic information to bridge the gap between the seen and unseen classes. This learning paradigm can be compared to a human when recognizing a new object by measuring the likelihoods between its descriptions and the previously learned notions [20] . In conventional ZSL techniques, the test set only contains samples from the unseen classes, which is an unrealistic setting and it does not reflect the real-world recognition conditions. In practice, data samples of the seen classes are more common than those from the unseen ones, and it is important to recognize samples from both classes simultaneously rather than classifying only data samples of the unseen classes. This setting is called generalized zero-shot learning (GZSL) [21] . Fig. 1 presents a schematic diagram of GZSL and ZSL. Assume that the seen class set includes Lion and Otter, while the unseen class set contains Seal and T iger. During the training stage, both GZSL and ZSL have access to the samples and semantic representations of the seen class set. During the test stage, ZSL can only recognize samples from the unseen class set, e.g., Seal and T iger, while GZSL is able to recognize samples from both sets, e.g., Lion, Otter, Seal and T iger.
Scheirer et al. [8] was the first to introduce the concept of GZSL in 2013. However, GZSL did not gain traction until 2016, when Chao et al. [21] empirically showed that the techniques under ZSL setting cannot perform well under the GZSL setting. This is because ZSL is easy to overfit on the seen classes, i.e., classify test samples from unseen classes as a class from the seen classes. Later, Xian et al. [22] , [23] and Liu et al. [24] obtained similar findings on image and web-scale video data with ZSL, respectively. This is mainly because of the strong bias of the existing techniques towards the seen classes in which almost all test samples belonging to unseen classes are classified as one of the seen classes. To alleviate this issue, Chao et al. [21] introduced an effective calibration technique, called calibrated stacking, to balance the trade-off between recognizing samples from the seen and unseen classes, which allows learning knowledge about the unseen classes. Since then, the number of proposed techniques under the GZSL setting has been increased radically.
As stated earlier, GZSL is a realistic setting of ZSL, where the aim is to learn a model to classify samples from both seen and unseen classes. Since its introduction, GZSL has attracted the attention of many researchers. Although several comprehensive reviews of ZSL models can be found in the literature [3] , [23] , [25] , [26] , none of them include an in-depth survey and analysis of GZSL. To fill this gap, we aim to provide a comprehensive review of GZSL in this paper, including the problem formulation, challenging issues, hierarchical categorization, and evaluation protocol. We review published articles, conference papers, book chapters, and high-quality preprints (i.e. arXiv) related to GZSL commencing from its popularity in 2016 till September 2020. However, we may miss some of the recently published studies, which is not avoidable. In summary, the main contributions of this review paper include:
• comprehensive review of the GZSL methods, to the best of our knowledge, this is the first paper that attempts to provide an in-depth analysis of the GZSL methods;
• hierarchical categorization of the GZSL methods along with their corresponding representative models, real-world applications, and evaluation protocols;
• elucidation on the main research gaps and suggestions for future research directions.
This review paper contains seven sections. Section 2 gives an overview of GZSL, which includes the problem formulation, semantic information, and challenging issues. Section 3 reviews GZSL methods, in which a hierarchical categorization of the GZSL methods is provided. Each category is further divided into several constituents. Section 4 presents the applications of GZSL to various domains, including object/action recognition, image segmentation, text classification, and natural language processing (NLP). The evaluation settings including benchmark data sets, performance indicators, are discussed in Section 5. Section 6 identifies trends and offers suggestions for future research, while concluding remarks are presented in Section 7.
The training phase of GZSL methods can be divided into two broad settings: inductive learning and transductive learning [23] . The inductive setting uses only the seen class information to build a model for recognition. As such, the training set for inductive GZSL can be denoted by
features of the seen classes in the feature space X , which can be obtained using a pre-trained deep model such as ResNet [27] , VGG-19 [28] , GoogLeNet [29] ; a s i ∈ R K indicates the K-dimensional semantic representations of seen classes (i.e., attributes or word vectors) in the semantic space A; Y s = {y s 1 , ..., y s Cs } indicates the label set of seen classes in the label space Y and C s is the number of seen classes.
The transductive setting, in addition to the seen class information D tr , has access to visual X u and semantic representations A u of the unseen classes without knowing their target classes [23] , [30] . If only the semantic representations of unseen classes are available during the training phase, i.e., as the case in most of the generative-based methods, it is called semantic transductive GZSL. Fig. 2 illustrates the main difference of transductive GZSL, semantic transductive GZSL, and inductive GZSL [31] , [32] . As can be seen, in inductive GZSL setting only the visual features and semantic representations of seen classes (A and B) are available. Semantic transductive GZSL, in addition to the visual features and semantic representations of seen classes, has access to the semantic representations of unseen classes (C and D). Transductive GZSL has access to both visual and semantic representations of unseen classes. Although several frameworks have been developed under transductive learning [33] , [34] , [35] , [36] , [37] , [38] , this learning paradigm is impractical. On one hand, it violates the unseen assumption and reduces challenge. On the other hand, the unseen classes are usually rare, and it is not practical to assume that unlabeled data for all unseen classes are available. The label space of unseen classes is denoted by Y u = {y u 1 , ..., y u Cu }, where C u is the number of unseen classes, and Y = Y s ∪ Y u denotes the union of both seen and unseen classes, where Y s ∩Y u = ∅. Both inductive and transdactive GZSL settings aim to learn a model f GZSL : X → Y to classify N t test samples, i.e., D ts = {x m , y m } Nt m=1 where x m ∈ R D , and y m ∈ Y.
Semantic information is the key to GZSL. It is used to build a relationship between the seen and unseen classes, thus making it possible to perform generalized zero-shot recognition. To guarantee usability of the semantic information, it should include information of all unseen classes. Meanwhile, it must be related to the samples in the visual feature space. The most widely used semantic information for GZSL can be grouped into manually defined attributes [13] , word vectors [39] , or their combinations.
These attributes describe the high-level characteristics of a class (category), such as shape (i.e., circle) and color (i.e., blue), which enable the GZSL model to recognize classes in the world. The attributes are accurate, but require human efforts in annotation, which are not suitable for large-scale problems [40] . Several attribute-based data sets are available, e.g. AWA1, AWA2, aPY, SUN and CUB. Wu et al. [41] proposed a global semantic consistency network (GSC-Net) to exploit the semantic attributes for both seen and unseen classes. Lou et al. [42] developed a data-specific feature extractor according to the attribute label tree.
These vectors are automatically extracted from large text corpus (such as Wikipedia) to represent the similarities and differences between various words and describe the properties of each object. Word vectors require less human labor, therefore they are suitable for large-scale data sets. However, they contain noise which compromises the model performance. As an example, Wang et al. [43] applied Node2Vec to produce the conceptualized word vectors. The studies in [44] , [45] attempted to extract semantic representations from noisy text descriptions.
In GZSL, several challenging issues must be addressed. The hubness problem [46] , [47] is one of the challenging issues of early ZSL and GZSL methods that utilize the nearest neighbor search in a high-dimensional space. Hubness is an aspect of the curse of dimensionality that affects the nearest neighbors method, i.e., the number of times that a sample appears within the k-nearest neighbors of other samples [48] . Dinu et al. [49] observed that a large number of different map vectors are surrounded by many common items, in which the presence of such items causes problem in high-dimensional spaces. This problem is more common in early ZSL and GZSL methods, where they project visual features into the semantic space.
The domain shift problem is another challenging issues of ZSL and GZSL methods. These methods learn the projection functions between visual features and semantic representations of seen classes to recognize both seen and unseen classes. On one hand, both visual space and semantic space are two different spaces. On the other hand, data samples of seen and unseen classes are disjoint, unrelated for some classes and their distributions are different, resulting in a large domain gap. Thus, learning a projection function using data samples from the seen classes without any adaptation to the unseen classes causes the domain shift problem [14] , [51] , [52] . This problem is more challenging in GZSL, due to the existence of the seen classes during prediction. Fig. 3 shows some examples of ideal and practical mappings. This problem is more common in inductive-based methods, as they have no access to the unseen class data during training. To overcome this problem, inductive-based methods incorporate additional constraints or information from the seen classes. Besides that, several transdactive-based methods have been developed to alleviate the domain shift problem [34] , [35] , [36] , [37] . These methods use the manifold information of the unseen classes for learning.
Since GZSL methods use the data samples from the seen classes to learn a model to perform recognition for both seen and unseen classes, they are usually biased towards the seen classes, leading to misclassification of data from the unseen classes into the seen classes (see Fig. 4 ), in which most of the ZSL methods cannot effectively solve this problem [50] . To mitigate this issue, several strategies have been proposed, such as calibrated stacking [21] , [53] and novelty detector [16] , [54] , [55] , [56] . The calibrated stacking [21] method balances the trade-off between recognizing data samples from both seen and unseen classes using the following formulation.ŷ
where γ is a calibration factor and [·] ∈ {0, 1} indicates whether c is from seen classes or otherwise. In fact, γ can be interpreted as the prior likelihood of a sample from the unseen classes. When γ → −∞, the classifier will classify all data samples into one of the seen classes, and vice versa. Le et al. [57] proposed to find an optimal γ that balances the trade-off between accuracy of the seen and unseen classes. Later, several studies used the calibrated stacking technique to solve the GZSL problem [53] , [58] , [59] , [60] , [61] . Similar to the calibrated stacking, scaled calibration [62] , probabilistic representation [63] , [64] and parametric novelty detection [41] have been proposed to balance the trade-off between both seen and unseen classes. Detectors aim to identify whether a test sample belongs to the seen or unseen classes. This strategy limits the set of possible classes by providing information to which set (seen or unseen) a test sample belongs to. Socher et al. [16] considered that the unseen classes are projected to out-ofdistribution (OOD) with respect to the seen ones. Then, data samples from unseen classes are treated as outliers with respect to the distribution of the seen classes. Bhattaxharjee [54] developed an auto-encoder-based framework to identify the set of possible classes. To achieve this, additional information, i.e., the correct class information, is imposed into the decoder to reconstruct the input samples.
Later, entropy-based [56] and probabilistic-based [65] have been developed to detect OOD, i.e., the unseen classes. Li [66] introduced a semantic discriminator, which is a distance-based method, to identify the domain of instances, i.e., seen or unseen, in the semantic space. Felix et al. [67] learned a discriminative model using the latent space to identify whether a test sample belongs to a seen or unseen class. Liu et al. [2] made the unseen classes more confident and the seen classes less confident using temperature scaling [68] with an entropy-based regularizer. Geng et al. [69] decomposed GZSL into OSR and ZSL tasks. Specifically, OSR recognizes the seen classes and rejects the unseen ones, while ZSL identifies the unseen classes rejected by OSR. Adaptive confidence smoothing (COSMO) [55] , which is a probabilistic model, uses a gating mechanism to discriminate the seen classes from unseen ones.
The main idea of GZSL is to classify objects of both domains (seen and unseen classes) by transferring knowledge from the seen classes to the unseen ones through semantic representations. To achieve this, two key issues must be addressed: (i) how to transfer knowledge from the seen classes to unseen ones; (ii) how to learn a model to recognize images from both seen and unseen classes without having access to the labeled samples of unseen classes [2] . In this regard, many methods have been proposed, which can be categorized into embedding-based and generative-based methods. A hierarchical categorization of these methods is provided in Fig. 5 .
• Embedding-based methods: this category aims to learn a projection or embedding function to associate the low-level visual features of seen classes with their corresponding semantic vectors. The learned projection function is used to recognize novel classes by measuring the similarity level between the prototype representations and predicted representations of the data samples in the embedding space. These available methods can be further categorized into four groups: semantic embedding (Fig 6 (a) ), visual embedding ( Fig. 6 (b) ), latent space embedding (Fig. 6 (c)), and hybrid-embedding. As can be seen in Fig. 6 , both semantic and visual embedding models learn a projection/embedding function from the space of one modality, i.e., visual or semantic, to the space of other modality. However, it is a challenging issue to learn an explicit projection function between two spaces due to the distinctive properties of different modalities. The latent space embedding models project both visual features and semantic representations into a common space, i.e., latent space, to explore some common semantic properties across different modalities. The hybrid embedding models exploit both the semantic and visual embedding methods to construct better semantics. They alleviate the contractions between the visual features and semantic representations to solve the domain shift problem. We summarize the proposed embedding based methods for GZSL in sub-section 3.1.
• Generative-based methods: this category learns a model to generate images or visual features for the unseen classes based on the samples of seen classes and semantic representations of both classes. A supervised model is trained to produce a prediction for a given test sample. Although these methods perform recognition in the visual space and they can be categorized as visual embedding models, we separate them from the embedding based methods. Two common generative-based methods are generative adversarial networks (GANs) [71] and variational autoencoders (VAEs) [72] . However, these two methods are not able to directly generate visual features for the unseen classes, as there is no explicit constraints in the learning objectives. To alleviate this issue, several conditional generative methods have been developed. A review of generative-based methods for GZSL is presented in sub-section 3.2
In the following sub-sections, we summarize the aforementioned categories in detail, and discuss their corresponding representative models.
As stated earlier, embedding based methods attempt to establish a projection/embedding function between the visual features and semantic representations of the seen classes. The top-1 classification result can be retrieved through the nearest neighbor search to perform recognition. The projection function can be either a regression or ranking based model. As every entry of the attribute vector represents a description of the class, it is expected that the classes with similar descriptions contain a similar attribute vector in the semantic space. However, in the visual space, the classes with similar attributes may have large variations. Therefore, finding such an accurate projection is a challenging task, which may cause visual semantic ambiguity problems. In recent years, various approaches have been introduced to learn the projection function, which can be categorized into four groups, as in Fig. 6 . A review of these groups is summarized in Table 1 .
The semantic space can be divided into either Euclidean or non-Euclidean spaces. The Euclidean space is simpler, but it is subject to information loss. While the non-Euclidean space, which is commonly based on graph networks, manifold learning, or clusters, usually uses the geometrical relation between spaces to preserve the relationships among the data samples [26] .
This category of methods learns a (forward) projection function from the visual space to the semantic space using different constraints or loss functions, and perform classification in the semantic space. The aim is to force semantic embedding of all images belonging to a class to be mapped to some ground-truth label embedding [73] . The projection with a linear regression model can be formulated as follows:
where λ is regulation parameter and W ∈ R D×K indicates the projection function from the visual space to semantic space. Once the best projection function W * is obtained, the nearest neighbor search can be performed for recognition of a given test image:
In general, the semantic embedding models can be divided into two sub-categories. The first treats the semantic representations as an intermediate space to transfer knowledge between classes. It predicts the attributes of the images and then classifies a test image by computing the similarity of the predicted vectors with the attribute/word vector [74] , [75] , [76] , [77] , [78] . As such, it defines a prediction function to perform a classification task, where the output is the class, y, with the highest score. Then, the learned function and semantic representations of the unseen classes can be employed to estimate the image-class similarity level for recognition purposes [79] .
Daghaghi et al. [74] , mapped the visual features to the semantic space using a multi-layer perceptron (MLP) network. The similarity level between the true attributes and attribute representation of the input is computed. Then, the Softmax function is used to estimate the probability of all classes with respect to the computed similarity level. Jin et al. [75] used the cluster center to learn more discriminative visual features for different classes while minimizing the intra-class variations. To achieve this, the joint optimization of center loss and softmax loss is adopted. Besides that, the performance of GZSL can be improved by rectifying the model output. However, the human-defined attributes usually share the same attributes for different classes, resulting in less discriminative semantic vectors. To alleviate this problem, Jin et al. [76] projected the semantic vector of each class to a high-order attribute space by applying the Gaussian random projection. Liu et al. [77] used a graph attention network [78] to generate an attribute vector for each class. Since the attribute-based methods learn a classifier for each attribute separately, they are not able to handle largescale data sets.
The second sub-category of semantic embedding models learns a compatibility function between the image and semantic embeddings and then assigns the test image to the class that yields the highest score [80] . Rahman et al. [80] used the concept of class adapting principal direction (CAPD) to embed the visual features into a discriminative semantic space (Fig. 7) . A distance-based metric is defined for learning the relationship between both seen and unseen classes. Specifically, a linear mapping W s is used to define CADP for a given image belonging to the s-th seen classes p s (s = 1, ..., C s ), i.e.,:
To learn a more discriminative semantic space for the seen classes e s , the following loss function is minimized:
where k = Cs c=1 N s , λ s indicate the regularization weight and L is the cost of a specific input x i c , defined as follows:
The projected visual feature onto the CAPD gives the highest score to the semantic embedding of the true class. This procedure is repeated for each seen class, producing P S = [p 1 , ..., p cs ] ∈ R D×Cs . Then, a generalized version of CAPD for the s-th seen classes to solve the bias problem is developed p g s (s = 1, ..., C s ), i.e.,:
where γ s is a parameter vector for the seen classes, which is optimized by minimizing the following loss:
where λ u is a regularization parameter. The first term in (8) computes the reconstruction accuracy of the seen classes embedding a s and the second term computes the reconstruction accuracy of the unseen class embedding a u from the seen classes. Then, the learned CAPD for the seen classes is used to approximate the individual CAPD for the u-th unseen class p u (u = 1, ..., C u ), as follows: where θ u = [θ 1,u , ..., θ Cs,u ] T is the coefficient vector that maps knowledge of the seen classes to the unseen ones. To approximate θ u , a distance metric is learned for the seen CAPD, which groups data samples of similar classes together and maps other classes further apart. Then, the learned metric is used to formulate the relationship between both seen and unseen classes. During prediction, for a given image x, the approximated CAPD for the seen and unseen classes is used to predict a class, as follows:
where p l ∈ p u ∪ p g s and a l ∈ A. Deep unbiased embedding transfer (DUET) [14] simultaneously learns a visual feature extractor and a projection function. DUET consists of two modules: a deep embedding transfer (DET) and an unseen visual feature generation (UVG). DET extracts the visual features by a convolutional neural network (CNN)-based model and learns an embedding mapping function to connect the visual space to the semantic space by integrating the advantages of linear and nonlinear embedding functions. The UVG module generates the visual features for the unseen classes by a conditional generative adversarial network (GAN), i.e., Wasserstein GAN with gradient penalty [81] , to address the shift domain problem by adjusting the embedding mapping. In addition, several studies have attempted to find the most important image regions instead of projecting the whole image [63] , [82] . These methods divide an image into many regions and find the most important regions by applying an attention technique. Then, several mechanisms are designed to project these regions into the semantic space.
Unlike the above-mentioned methods that learn a projection from the visual space to the semantic space, studies that attempt to use other strategies have been proposed. Examples include SELAR [83] that projects the local features into the semantic space by using the localized attributes, and image-guided semantic classification (IGSC) [84] that learns from an image and searches for a combination of variables in the semantic space to separate a specific class from others.
This category of methods learns a (reverse) projection function to map the semantic representations (back) into the visual space, and perform classification in the visual space. The goal is to make the semantic representations close to their corresponding visual features [85] . A projection function that uses a ridge regression can be formulated, as follows:
where W ∈ R K×D indicates the projection function from the semantic space to visual space and λ is regulation parameter. Das and Lee [62] minimized the discrepancy between the semantic representations and visual features using the least square loss method (Fig. 8) . Rational matrices are constructed for each space, with the aim to minimize the inter-class pairwise relations between two spaces. To avoid the domain shift problem, a point-to-point correspondence between the semantic representations and test data samples is found. The co-representation network (CRnet) [50] decomposes the semantic vectors of the training set into K subsets using the k-means clustering algorithm. Each subset is projected to the visual space using a fully connected layer with a rectified linear unit (ReLU). This decomposition method allows the model to construct a uniform embedding space with a large local relative distance. The K projections are combined to form a cooperative module. In addition, a relation module [86] , which is a two-layer perceptron, is devised to compute the similarity metric between the output of the cooperation module and a sample feature vector. The learned similarity function is used for recognition.
Recently, the graph convolutional network (GCN) [87] has been applied to generate super-classes in the semantic space. The learned knowledge is transferred from the semantic space to the visual space with respect to the provided annotations [43] , [88] . To achieve this, the cosine distance between the visual features of the seen classes and corresponding semantic representations is minimized [43] . In addition, a triplet margin loss is optimized to avoid the hubness problem. The semantics-guided class imbalance learning model (SCILM) [89] considers the class imbalance problem in ZSL. It consists of two modules: (i) semantic embedding network (SEN), and (ii) semantic attention network (SAN). SEN uses a 3-layer neural network with ReLU to map the semantic representations into the visual space. SAN selects the same number of visual samples from each training class to ensure that each class contributes equally during training at each iteration. Self-focus [32] uses a deep embedding model [47] to map the semantic space into the visual space.
This category of methods projects both visual features and semantic representations into a common space W , i.e., a latent space [52] , [90] . The aim is to project x s and a s nearby into the latent space. This projection can be devised as φ(x s ) and θ(a s ), where φ(x s ) ∈ W and θ(a s ) ∈ W and φ(x s ) ≈ θ(a s ). An ideal latent space should fulfill two conditions: (i) intra-class compactness, and (ii) interclass separability [91] . Introduced by Zhang et al. [47] , this mapping aims to overcome the hubness problem of ZSL models. Many improvement studies have been conducted, e.g. use of the graph information to preserve the geometric structure of features in the latent space [52] , [92] . The shared the reconstruction graph (SRG) [52] , ], firstly, learns a relationship between the two spaces. Then, image prototypes for the unseen classes are synthesized for classification. SRG contains a sparsity constraint, which enables the model to divide the classes into many clusters of different subspaces. A regularization term is adopted to select fewer and relevant classes during the reconstruction process. The reconstruction coefficients are shared, in order to transfer knowledge from the semantic prototypes to image prototypes. In addition, the unseen semantic embedding is used to mitigate the domain shift problem, in which the graph of the seen image prototypes is adopted to alleviate the space shift problem. AGZSL [92] , i.e., asymmetric graph-based ZSL, combines the class-level semantic manifold with the instance-level visual manifold by constructing an asymmetric graph. In addition, a constraint is made to project the visual and attribute features orthogonally when they belong to different classes. Discriminative anchor generation and distribution alignment (DAGDA) [93] generates anchors using a diffusion-based GCN. A semantic relation regularization is derived to refine the distribution in the anchor space. To mitigate the hubness problem, two auto-encoders are employed to keep the original information of both features in the latent space. However, using the graph information increases the model complexity.
Cacheux et al. [94] formulated a framework based on three assumptions: (i) the classes are not equally different; (ii) the margin to separate seen classes are not fixed; (iii) all samples belonging to the same class are not equal. For the first assumption, a flexible semantic margin is devised to measure the similarity level among different classes. For the second assumption, a trade-off between the visual features and the unit normalized version is established by introducing a partial normalization. For the last assumption, a relevance weighting mechanism is introduced to quantify the importance of each data sample from the seen classes. These three mechanisms are integrated into a triplet loss, and tested under two settings: (i) semantic-embedding, and (ii) latent space embedding, using a linear projection. Exemplar synthesis (EXEM) [58] learns a function to predict the locations of visual features with respect to the unseen classes (Fig. 9) Fig. 9 : A schematic view of the exemplar synthesis (EXEM) (adopted from [58] ).
projection of the data samples belonging to a particular class. Then D support vector regressor (SVR) is learned with respect to the visual exemplar semantic representations. Finally, the similarity level between the test samples and bases is computed with regard to the predicted exemplars to produce a prediction. Zhang et al. [91] designed a loss function consisting of (i) a regression term to minimize the distance between the embeddings of visual sample and its semantic representation; and (ii) a classification term to force the embeddings of different classes to be discriminative.
Since the information obtained from the semantic representations is limited and less discriminative to recognize data samples from different classes in a specific domain, Jian et al. [95] proposed to obtain the class prototypes in both visual and semantic spaces to construct a class structure between both spaces (Fig. 10) . To achieve this, the class prototypes of seen classes in both spaces are learned. The visual-semantic structure is jointly aligned by introducing a coupled dictionary learning framework. The aim is to explore some bases in each space to represent each class. A domain adaptation is proposed to learn the prototypes from both seen and unseen classes in the visual space, in order to alleviate the domain shift problem. In addition, attributing label space (ALS) [96] defines a discriminative label space and projects the visual features via a linear projection matrix. Specifically, it fixes the labels of the seen classes and uses the attributes to map the label of the unseen classes into the label space. DDIP [97] , i.e., discriminative domain-invariant prototypes, projects both visual and semantic representations into a latent space, i.e., hyperspherical space. A dictionary for each projection, i.e., visual-latent and semantic-latent, is constructed by sparse coding. In addition, DDIP applies an orthogonal projection to make the model more discriminative.
Jiang et al. [98] developed a constructive network called the transferable constructive network (TCN). The TCN aims to judge the consistency of an image to a specific class. It consists of two parts: an information fusion and constructive learning. For information fusion, both visual features and semantic representations are encoded into a latent space using the CNN and MLP, respectively. Constructive learning checks how well an image is consistent with a class by considering two aspects. The first is whether learning is discriminative enough to recognize different classes, which uses semantic representations of the seen classes for supervision, while the second is whether learning is transferable Fig. 10 : A schematic view of the proposed method by Jian et al. [95] .
to the unseen classes based on the class similarity score between visual features of the seen and unseen classes. Deep calibration network (DCN) [2] maps both visual features and semantic representations into a common K dimensional latent space by maximizing the compatibility of the seen samples to the unseen classes. Specifically, the CNN is used to represent each image by feature embedding, while the MLP is employed to represent the semantic representation of each class in the same embedding space. In addition, empirical risk minimization and an uncertainty calibration are adopted to calibrate the DCN for undertaking the GZSL problems. During recognition, the nearest prototype classifier (NPC) is applied to assign a class label pertaining to semantic embedding that is the closest visual embedding of a given image.
On one hand, visual features obtained from the DNN are high-dimensional and are not semantically meaningful, limiting the learning model to formulate a proper relationship between the visual features and semantic representations. On the other hand, semantic representations of some classes are identical, which is challenging for the GZSL setting when one of these classes is among the unseen classes. To alleviate this problem, Zhu et al. [30] used a low-dimensional statistical embedding method to mirror the obtained information by semantic attributes visually. As such, the input is mapped into a finite set of feature vectors. Then, each feature vector is modeled as a C-dimensional Gaussian mixture model with isotropic components. In addition, a visual oracle is proposed for GZSL to reduce noise and provide information about the presence/absence of classes.
This category of methods applies several projections to construct a better relationship between both visual and semantic spaces and alleviate the domain shift problem. Joint inductive learning (JIL) [102] induces a bidirectional projection learning (including both visual-semantic and semantic-visual projections) and a shared subspace learning to learn better projections and adjust the seen and unseen class domains. Label activating framework (LAF) [70] defines the labels of both seen and unseen classes in the same space. It consists of two parts: (i) visual-label activating that learns a projection function from the visual space to the label space using regression; (ii) semantic-label activating that learns a projection function from the semantic space to the label space. In addition, a bidirectional reconstruction constraint among semantic and label is added to alleviate the projection shift problem. Zhang et al. [31] introduced a class level overfitting problem. It is related to parameter fitting during training without prior knowledge about the unseen classes. To solve this problem, a triple verification network (TVN) is used for addressing GZSL as a verification problem. TVN projects the seen classes into an orthogonal space to obtain a better performance and a faster convergence speed. Then, a dual regression (DR) method is proposed to regress both visual features and semantic representations to be compatible with the unseen classes. SABR-I [33] , i.e., semantically aligned bias reducing, is a two-step model that projects the visual features into a latent space by a two-layer fully connected network. Then, the learned latent space is used to simultaneously build a classifier and a regressor to learn the discriminative information and preserve the semantic relationships between different classes, respectively. In the second step, SABR-I uses a conditional generative method to form a more discriminative latent space.
Dual-triplet network (DTNet) [103] uses a mapping network to project the semantic representations into the visual space. Two triplet networks are used to construct a more discriminative metric space, i.e., one considers the attribute relationship between categories by learning a mapping from the attribute space to visual space, while another considers the visual features. Guo et al. [60] considered a dual-view ranking by introducing a loss function that jointly minimizes the image view label and label-view image rankings. This dual-view ranking enables the model to train a better image-label matching model. The image-view label ranking attempts to rank the correct label before any other labels for a training sample, while the label-view image ranking tries to rank the respective images to their correct classes before images from other classes. In addition, a density adaptive margin is used to set a margin based on the data samples, because the density of images varies in different feature spaces, and different images may have different similarity score.
Several auto-encoder-based frameworks have been proposed to solve the domain shift problem [104] , [105] , [106] , [107] , [109] . To achieve this, a decoder can be imposed by an additional constraint in learning different mappings. Annadani and Biswas [104] used an encoder to map the semantic space into the visual space using an MLP and a decoder to reconstruct the inputs. Specifically, the similarity level between different classes, which is measured by the cosine similarity measure, is integrated into an objective function to preserve the relations. A Low-rank embedded semantic auto-encoder (LESAE) [107] uses an encoder to map the low-rank visual features to the semantic space, while the decoder aims to reconstruct the original visual features. Latent space encoding (LSE) [105] explores some common semantic characteristics between different modalities and connects them together. For each modality, an encoderdecoder framework is exploited, in which an encoder is used to decompose the inputs as a latent representation while a decoder is employed to reconstruct the inputs. Product quantization ZSL (PQZSL) [106] defines an orthogonal common space, which learns a codebook, to project the visual features and semantic representations into a common space using a center loss [109] and an auto-encoder, respectively. The orthogonal common space enables the classes to be more discriminative, thus the model can achieve better performance. In addition, PQZSL compresses the visual features into compact codes using quantizers to approximate the nearest neighbors. The multi-channel Gaussian mixture variational auto-encoder (MCGM-VAE) [101] uses two VAE to learn the cross-modal latent features from the visual and semantic spaces, respectively. Cross alignment and distribution alignment strategies are devised to match the features from different spaces.
Generative-based methods are originally used to generate examples from the existing ones for training deep learning models [110] and compensating for imbalanced classification problems [111] . Recently, they have been adopted to generate samples (i.e., images or visual features) for unseen classes by leveraging their semantic representations. By generating data samples for unseen classes, a GZSL problem can be converted into a conventional supervised learning problem. Based on single homogenous process, a model can be trained to classify the test samples belonging to both seen and unseen classes and solve the bias problem. However, it is a difficult task to synthesize data samples from semantic representations, due to the overlap of common properties such as color and shape between many classes [108] .
The generated data samples are required to satisfy two conflicting conditions: (i) semantically related to the real samples, and (ii) discriminative so that the classification algorithm can classify the test samples easily. To satisfy the former condition, some underlying parametric representations can be used, while the classification loss function can be used to satisfy the second condition [112] . GANs [71] and VAEs [72] are two prominent members of generative models that have achieved good results in GZSL. A review of these groups is summarized in Table 2 .
GANs aim to synthesize new data samples by computing the joint distribution p(y, x) of samples utilizing the class conditional density p(x|y) and class prior probability p(y). GANs consist of a generator G SV : Z × A → X that uses semantic attributes A and a random uniform or Gaussian noise z ∈ Z, to generate visual featurex ∈ X , and a discriminator D V : X × A → [0, 1] that distinguishes real visual features from the generated ones. When a generator learns to synthesize data samples for the seen classes conditioned on their semantic representations A s , it can be used to generate data samples for the unseen classes through their semantic representations A u . However, the original GAN models are difficult to train, and there is a lack of variety in the generated samples. In addition, the mode collapse is a common issue in GANs, as there are no explicit constraints in the learning objective. To overcome this issue and stabilize the training procedure, many GAN models with alternative objective functions have been developed. Wasserstein GAN (WGAN) [113] mitigates the mode collapse issue using the Wasserstein distance as the objective function. It applies a weight clipping method on the discriminator to allow incorporation of the Lipschitz constraint. The improved WGAM model [81] reduces the negative effects of WGAN by utilizing gradient penalty instead of weight clipping.
Xian et al. [108] devised a conditional WGAN model with a classification loss to synthesize visual features for the unseen classes, which is known as f-CLSWGAN (see Fig 11) . To synthesize the related features, the semantic feature is integrated into both generator and discriminator by minimizing the following loss function:
wherex s = G(z, a s ),x = αx s + (1 − α)x s with α ∼ U (0, 1), λ is penalty factor and discriminator D : X × C → R is a multi-layer perceptron. In (12) , the first and second terms approximate the Wasserstein distance, while the last term is the gradient penalty. In addition, to generate discriminative features, the negative log-likelihood is used to minimize the classification loss, i.e.,:
where P (y s |x s ; θ) is the probability thatx s is predicted as class y s , which is estimated by a softmax classifier by θ and optimized by visual features of the seen classes. The final objective function can be written as:
where β is the weighting factor. TFGNSCS [114] is an extended version of f-CLSWGAN that considers transfer information. Compared with f-CLSWGAN, TFGNSCS contains two additional terms in its objective function. The first is the classification loss of generated features for the unseen classes,x u . As such, a transformation function T (P (θ)) is defined to transfer the classifier model P (θ) using the relationship between the seen and unseen classes in the semantic space, as follows:
wherex u = G(z, a u ), Q(y u |x u ; θ) is the probability thatx u is predicted as class y u using the classifier Q(θ) = T (P (θ)). This term aims to generate discriminative features for the unseen classes. The second term is formulated based on the discriminator to balance the discriminator bias by considering knowledge solicited from the unseen classes, i.e.
LisGAN [115] generates visual features using the conditional WGAN model [113] with supervised loss. To ensure that the synthesized samples are related to the real ones, the soul samples are introduced to consider the multi-view property of samples. As such, for each category, multiple soul samples, which are based on their features, are defined. Then the generated samples for each class are enforced to be close at least to one soul sample of the corresponding class using the following regularization term:
where s s j indicates jth soul sample for class s, n1 and k are the number of generated and soul samples, respectively. Simultaneously, the fake soul samples are enforced to be close to one of real soul samples as follows:
wheres s j is the generated soul sample for class c, and C is the number of classes.
AFC-GAN [116] uses a conditional WGAN [113] to generate synthetic features and introduces a boundary loss to maximize the decision boundary between the seen and unseen features. In addition, a multi-modal cycle consistency loss is introduced to preserve the semantic consistency of the generated visual features. Bucher et al. [112] used four different conditional GANs, i.e., generative moment matching network (GMMN) [117] , AC-GAN [118] , denoising autoencoder [119] , and adversarial auto-encoder [120] , to generate data samples for the unseen classes. The empirical results indicate that GMMN outperforms other methods. However, the semantic representations of similar classes are highly overlapped, as they are attributes defined by humans, thus they are prone to failure in prediction. To make the semantic representations more distinguishable, semantic rectifying GAN (SR-GAN) [121] rectifies features using the semantic rectifying network (SRN) [122] and then generated the visual features using semantic representations, rectified features R(s), and noise Z. Moreover, preand post-reconstruction modules were introduced to keep the consistency of both visual and semantic features. E-PGN [123] , which is an episode-based framework, generates class-level visual samples conditioned on semantic representations.
Inspired by the human creativity process [125] , CIZSL [126] introduces a signal learning strategy to explore the unseen space using a semantic representation t h ∼ p h text . A probability distribution p h text over the semantic description is defined, which is hard negative to the unseen classes. Then, t h is sampled as follows:
where α is a randomly selected value between 0.2 and 0.8, t s a and t s b are two randomly selected semantic representations from the seen classes. The loss based on G(t h , z) was defined as follows: (20) where L e is an entropy function over the seen classes, as produced by discriminator {D s,k (.)} k=1→S . The first and second terms in (20) enforce the model to generate the related features which are difficult to classify as the seen classes, respectively.
The generative-based models synthesize highly unconstrained visual features for the unseen classes that may produce synthetic samples far from the actual distribution of visual features. Several studies attempted to alleviate this issue and address the unpaired training issue during visual feature generation by enforcing the generated visual features to map back into their respective semantic space. To achieve this, Zhu et al. [128] used a cycle consistency loss term. Later, inspired by [128] , Felix et al. [129] proposed multi-modal cycle consistency loss to generate more constrained visual features for GZSL. The cycle-consistent Fig. 13 : A general view demonstrating the proposed Zero-VAE-GAN framework (adapted from [127] .
adversarial networks for ZSL (CANZSL) [130] synthesizes visual features from noisy text. Specifically, WGAN [113] and a classification network are employed to generate semantically related and discriminative features, respectively. In addition, an inverse adversarial network is adopted to convert the generated features into text to ensure that the synthesized visual features accurately reflect the semantic representations. DASCN [131] applies dual GANs to generate visual features and reconstruct their corresponding semantic features. The primal GAN includes a generator G SV , and a discriminator D V that differentiates visual features from the fake ones (generated by G SV ). The dual GAN model learns a reverse generator G V S : X → A, and a discriminator D V that differentiates semantic features from the fake ones (generated by G V S ). In addition, DASCN combines the classification loss with the semantics-consistency adversarial loss to ensure that the generated features are discriminative pertaining to different classes. SPGAN [132] uses WGAN [113] with a similarity preserving loss to generate related visual features for unseen classes. This loss attempts to reduce the distance between real samples and generated visual features. In addition, classification loss was used to make generated samples more discriminative. Wang et al. [133] proposed a framework based on optimal transport (OT) to address the domain shift and bias problems. A conditional generator is adopted to synthesize visual features for the unseen classes, while an attribute-based regularizer is used to discriminate the generated features.
VAEs aim to identify the relationship between data sample x and the distribution of the latent representation z. A parameterized distribution q Φ (x|x) is used to approximate the posterior probability p(z|x). Similar to GANs, VAEs consist of two components: (i) an encoder q Φ (z|x) with parameters Φ, and (ii) a decoder p θ (x|z) with parameters θ. The encoder maps sample space x to latent space z with respect to its class c, while a decoder maps the latent space to the sample. Conditional VAE (CVAE) [134] synthesizes samplex with certain properties via maximizing the lower bound of conditional likelihood p(x|x).
Mishra et al. [135] used neural network to model both encoder and decoder. During training, an encoder was used to estimate q(z i |x i , A yi ) = N (µ xi , Σ xi ) for each training sample x i , where A yi is the given conditional variable. Then, N (µ xi , Σ xi ) is applied to sample z. Next, decoder is employed to reconstruct x using z and A y . The following loss function was used:
where L 2 norm was used as the reconstruction loss.
In addition, several bi-directional CVAE-based frameworks have been developed. CADA-VAE [124] learns the shared cross-modal latent features of image and class attributes through distribution alignment (DA) and cross alignment (CA) objectives (see Fig. 12 ). As such, a VAE model [72] is used to learn the latent features of each data modality, i.e., image and semantic representations. Therefore, the VAE loss for M modalities can be written as: (23) where β weights the KL-Divergence. Then, the CA loss, which can be estimated via decoding the latent feature of a data sample from other modality of the same class, is used for reconstruction, i.e.,
where D j and E i are, respectively, the decoder and encoder of feature ith and jth modality. In addition, the DA loss is used to match generated image and class representations by minimizing their distance, i.e.,
where
Finally, the loss of the CADA-VAE can be written as follows:
where γ and σ are the weighting factors of the CA and DA, respectively. [46] is equipped with a discriminator-driven feedback mechanism to produce high-quality examples for the unseen classes. Specifically, a feedforward neural network is used as a regressor/discriminator to learn a mapping between examples and their corresponding semantic classes. The regressor/discriminator learns from the labeled and synthesized samples through supervised and unsupervised losses, respectively. GDAN [138] uses CVAE [134] to generate visual features conditioned on the class labels. A regressor network is devised to map the visual features back into their semantic features. These two models learn from each other using a consistency loss. In addition, GDAN contains a discriminator to estimate the similarity level among each visual-textual feature pair. The discriminator communicates with the other two networks using a dual adversarial loss. Li et al. [139] employed a multivariate regressor to map back the VAE decoder's output to the class attributes. This feedback helps the generator to generate more discriminative samples.
On one hand, VAEs generate blurry images due to the use of the element-wise distance in their structures [146] . On the other hand, the training process of GANs is not stable [147] . As a result, several studies have attempted to combine them to alleviate their shortcomings. Gao et al. [127] , [140] proposed a joint generative model (known as Zero-VAE-GAN) that combines CGAN and CVAE to generate high-quality visual features for the unseen classes (see Fig. 13 ). More specifically, CVAE conditioned on semantic attributes is combined with CGAN conditioned on both categories and attributes. Besides that, a perceptual reconstruction loss [148] is incorporated to generate high-quality features.
Apart from GANs and VAEs, several studies have attempted to generate visual features for the unseen classes using other approaches. For example, unseen visual data synthesis (UVDS) [141] and Class-specific synthesized dictionary (CSSD) [142] project both visual features and semantic representations into a latent space. UVDS proposes a diffusion regularisation (DR) to synthesize visual features for the unseen classes using embedding matrices. CSSD learns a class-specific encoding matrix in a latent space for each class and consequently a dictionary matrix within a dictionary framework. Then, the encoding matrices with the affinity seen classes, i.e., seen classes similar to the unseen ones, are used to generate visual features for the unseen classes. Feng and Zhao [150] extracted two types of knowledge, i.e., local rational knowledge and global rational knowledge, from the visual and semantic representations, respectively, to build a generative model. Li et al. [136] proposed to generate visual features for the unseen classes using the most similar seen classes to the unseen ones in the semantic space.
Shi and Wei [143] projected visual features into a discriminative embedding space and constructed an autoen- coder to generate visual features for the unseen classes. As such, the encoder maps visual features to the semantic embedding space. The regressor feedback is imposed into the decoder to reconstruct truthful visual features. Then, the learned decoder generates visual features for the unseen classes to train a classifier. Bi-adversarial auto-encoder (BAAE) [144] pairs an autoencoder with two adversarial networks. On one hand, the encoder, which operates as a generator, integrates the visual and synthesized features into an adversarial network to capture the real distribution. On the other hand, the decoder, which acts as semantic inference, formulates real class semantics for inference toward another adversarial network to enforce both real and synthesized visual features to be related to the semantic representation. Similar to BAAE, the multi-modality adversarial auto-encoder (MAAE) [145] pairs an auto-encoder with multi-modality adversarial networks. The encoder generates visual features while the decoder aims to relate both generated and real features to the class semantics. Both BAAE and MAAE integrate classification networks to ensure that the generated and real features are discriminative.
As stated earlier, the GZSL methods under the transductive learning setting are able to alleviate the domain shift problem by taking the unlabeled data samples from the unseen classes into consideration. Having access to the unlabeled data allows the model to know the distribution of the unseen classes and consequently learns a discriminative projection function or synthesizes the related visual features for the unseen classes by using embedding and generative based methods, respectively. In this sub-section, the transductive-based GZSL methods are reviewed. Since the number of publications is limited, we categorize them into embedding-based and generative-based methods. Table 3 summarizes the transductive-based GZSL methods.
These methods leverage the unlabeled data samples from the unseen classes to learn a projection function between the visual and semantic spaces. Using these unlabeled data samples, the geometric structure of the unseen classes can be estimated, and a more discriminative projection function in the common space, i.e., semantic, visual, or latent, can be formulated. The aim is to alleviate the domain shift problem.
To map the visual features of the seen classes to their corresponding semantic representations, a classification loss is derived and an additional constraint is needed to extract useful information from the data samples from the unseen classes.
Quasi-fully supervised learning (QFSL) [149] (Fig. 14) , which is a semantic-embedding based model, projects the visual features of the seen classes into a number of fixed points in a semantic space using a fully connected network with the ReLU activation function. The unlabeled visual features from the unseen classes are mapped onto other points using a bias loss.
where L p and Ω are the classification loss and regularization factor, and L b is the bias loss, i.e.,
where p u indicates the predicted probability of u-th class.
Other semantic embedding models have been proposed, e.g. [17] , [61] and [152] . Fu et al. [17] used a semantic manifold structure of the class prototypes distributed in the embedding space. The Dual visual semantic mapping paths (DMaP) [152] is formulated to learn the connection between semantic manifold structure and visual-semantic mapping of the seen classes. On the other hand, Adaptive embedding ZSL (AEZSL) [61] learns a visual-semantic mapping for each [17] Semantic embedding Nearest neighbor VGG-19 [28] , GoogLeNet [29] , AlexNet [151] Attributes, Word2Vec [39] AWA, CUB, aPY, ImageNet × AEZSL / DAEZSL [61] Semantic embedding Nearest neighbor VGG [28] , GoogLeNet [29] word vector CUB, SUN, Dogs, ImageNet × DMaP [152] Semantic embedding Nearest neighbor VGG [28] , GoogLeNet [29] , ResNet [27] Attributes, Word2Vec [39] AWA, CUB, Dogs, ImageNet × Several studies focus on the development of visual embedding-based models under the transductive setting. As an example, ERPL [36] optimizes the following function:
where γ controls the importance of the loss with respect to both seen and unseen classes.
Hu et al. [37] used superclass prototypes, instead of the seen/unseen class prototypes, to align with the seen and unseen class domains. The K-means clustering algorithm is used to group the semantic representations of all seen and unseen classes into r superclass. DTN [153] , which is a probabilistic-based approach, decomposes the training phase into two independent parts. One part uses the crossentropy loss to learn the labeled seen classes, while the other part applies a combination of cross-entropy loss and KL divergence to learn from the unlabeled unseen classes. In the second part, cross-entropy is adopted to avoid the falling unseen classes into seen classes.
MFMR [154] uses a matrix tri-factorization [157] framework to construct a projection function in the latent space. In addition, two manifold regularization terms are formulated to preserve the geometric structure in both spaces and a test time manifold structure to alleviate the shift problem. SABR-T [33] projects the visual features into a latent space by utilizing a two-layer fully connected network. Then, it uses WGAN to generate the latent space for the unseen classes via minimizing the marginal difference between the true latent space representation of the unlabeled unseen samples and the generated space. DMaP [152] extracts the interclass relation between the seen and unseen classes in both spaces. For a given test sample, if the inter-class relationship consistency is satisfied, a prediction is produced. In [38] , [91] , the pseudo labeling technique is used to solve the bias problem.
These methods use the unlabeled unseen data to generate the related visual features for the unseen classes. A generative model is usually formulated using the seen class data samples, while the unlabeled unseen class data samples are used to fine-tune the model. As an example Gao et al. [127] developed two self-training strategies based on pseudo-labeling procedures. The first strategy uses K-NN to provide pseudo-labels for the unseen visual features. As such, the semantic representations of the unseen classes are used to generate N fake unseen visual features from the Gaussian distribution. Then, for each unseen class, the average of N fake features is used as an anchor. At the same time, K-NN is used to update the anchor. Finally, the top M percent of unseen features are selected to finetune the generative model. The second strategy obtains the pseudo-labels directly through the classification probability. VAEGAN-D2 [155] learns the marginal feature distribution of the unlabeled samples using an additional unconditional discriminator. SDGN [156] integrates self-supervised learning into the generative model.
GZSL has been widely used in computer vision and natural language processing (NLP). In computer vision, GZSL is applied to solve problems related to both image and video. Human action/gesture recognition from both images and videos is one of the most challenging tasks in computer vision, due to a variety of actions that are not available among the seen action categories. In this regard, GZSLbased frameworks have been employed to recognize single label [24] , [135] , [159] and multi-label [158] human actions. As an example, a new multi-label ZSL (MZSL) framework using JLRE (Joint Latent Ranking Embedding) has been proposed in [158] . The relatedness score of various action labels is measured for the test video clips in the semantic embedding and joint latent visual spaces.In addition, a multimodal framework using audio, video, and text has been introduced in [160] , [161] .
In image classification, many GZSL studies have been conducted, which include object detection [162] , recognizing general categories ranging from single label [163] , [164] , [165] , [35] , [166] to multi-label [167] , tongue constitution recognition [168] , sense recognition [18] and fine-grained categories such as breeds of flowers [108] and breeds of birds [126] . GZSL also has been used for image segmentation [169] , image annotation [43] and image retrieval [170] , [171] .
NLP [172] , [173] and text analysis [174] [88] , [130] , [175] , multi-label [176] as well as noisy text description [44] , [45] .
In this section, the standard evaluation protocol to evaluate the performance of the GZSL methods including benchmark data sets, class and image embeddings, and performance indicators, is reviewed.
There exist a number of benchmark data sets for GZSL. ImageNet [1] , which is a WordNet hierarchy [177] data set, is one of the well-known image data sets that have been widely used in various studies with different settings in computer vision and image processing tasks. The original ImageNet includes over 15 million labeled images (highresolution) related to approximately 22,000 classes (categories). Several studies have used the full size of ImageNet such as [178] , [179] . Moreover, there exist several attribute-based data sets. The CUB-200-2011 data set (CUB) [180] is an extended form of the original CUB-200 [181] . It contains approximately two times more images in each category compared to the original CUB-200 data set. All images in the CUB-200-2011 data set are annotated with part locations, attribute labels, and bounding boxes. The aPascal-aYahoo data set [182] has 12,695 images of the original PASCAL VOC 2008 data set categorized into 20 different classes (aPascal). It also includes 2,644 images that were collected by using the Yahoo image search engine (aYahoo) related to 12 various classes. All images in the data set have been annotated by 64 binary attributes that specify the visible objects.
The Animal with Attribute (AWA) [15] is a data set with over 30,000 images (animal) in 50 classes. AWA1 [183] is a coarse-grained version of AWA including 30,475 images in 40 classes for training and 10 more classes for testing. AWA2 [22] is an extended version of AWA1 that includes more images. UNderstanding (SUN) [184] data set, which is a well-known scene-based data set, contains 130,519 images in 899 classes (categories). SUN attribute data set [185] , [186] is a subset of the original SUN data set for fine-grained scene classification (categorization). It has 14,340 images related to 717 categories. North America Birds (NAB) [187] is another data set of birds including 48,562 images belonging to 1,011 categories. Other data sets include Large-Scale Attribute Dataset (LADdataset) [188] , Dogs [189] , Oxford Flowers (FLO) [190] , Larger-scale DeepFashion [191] .
To evaluate the performance of the techniques under GZSL setting several performance indicators have been used. Accuracy of seen (Acc s ) and accuracy of unseen (Acc u ) classes are two common performance indicators. Chao et al. [21] introduced the area under seen-unseen accuracy curve (AUSUC) to balance the trade-off recognizing between seen and unseen classes for calibration-based techniques. This curve can be obtained by varying γ in (1). Techniques with higher AUSUC values perform in balance for GZSL tasks.
Harmonic mean (HM) is another performance indicator that is able to measure the inherent biasness of GZSL-based techniques towards seen classes:
Acc s * Acc u Acc s + Acc u .
If a GZSL technique is biased towards the seen classes, its Acc s will be higher than Acc u and consequently the HM drops down [80] . 
In this section, we discuss the main findings of our review. Several research gaps that lead to future research directions are highlighted. We categorize the GZSL methods into two groups: (i) embedding-based, and (ii) generative-based methods. Embedding-based methods learn a projection function, whether visual-semantic, semantic-visual, common/latent space, or a combination of them (hybrid), to link the visual space of the seen classes into their corresponding semantic space. They use the learned projection function to recognize data samples from both seen and unseen classes. Generative-based methods convert the GZSL problem into a conventional supervised learning problem by generating visual features for the unseen classes. Since the visual features of the unseen classes are not available during training, learning a projection function or a generative model using data samples from the seen classes cannot guarantee generalization pertaining to the unseen classes. Although the low-level attributes are useful for classification of the seen classes, it is challenging to extract useful information from structured data samples due to the difficulty in separating different classes. Thus, the main challenge of both methods is the lack of unseen visual samples at the training stage, leading to the bias and domain shift problems.
Embedding-based models have been proposed to address the ZSL and GZSL problems. These methods are less complex, and they are easy to implement. However, their capabilities in transferring knowledge are restricted by semantic loss, while the lack of visual samples for unseen classes causes the bias problem. This results in poor performances of the models under the GZSL setting. In semantic embedding models, the projection of visual features to a low dimensional semantic space shrinks their variance and restricts their discriminability [47] . The compatibility scores are unbounded and ranking may not be able to learn certain semantic structures because of the fixed margin. Moreover, they usually perform NN search in the shared space, which causes the hubness problem [41] , [70] , [73] .
Although visual embedding models are able to alleviate the hubness problem [70] , they suffer from several issues. Firstly, the visual features and semantic representations are obtained independently, and they are heterogeneous, e.g., they are from different spaces. As such, the data distributions of both spaces can be different, in which two close categories in one space can be located far away in other space [51] . In addition, regression-based methods can not explicitly discover the intrinsic topological structure between two spaces. In addition, regression-based methods cannot explicitly discover the intrinsic topological structures between two spaces. Therefore, learning a projection function directly from the space of one modality to the space of another modality may cause information loss, increased complexity, and overfitting toward the seen classes. To mitigate this issue, several studies have attempted to project visual features and semantic representations into a latent space. Such a projection can reconcile the structural differences between the two spaces. Besides that, hybrid models aim to learn a better projection and adjust the seenunseen class domains. However, these models still suffer from the domain shift and bias problems.
Generative-based methods are more effective in solving the bias problem, owing to synthesizing visual features for the unseen classes. Moreover, the availability of visual samples for the unseen classes allows the models to perform recognition of both seen and unseen classes in a single process. Nonetheless, they generate visual features through learning a model class conditioned on the seen classes. They do not learn a generic model for generalization toward both seen and unseen class generations [192] . They are complex in structure and difficult to train (owing to instability). Their performance is restricted either by obtaining the distribution of visual features using semantic representations or using the Euclidean distance as the constraint to retain the information between the generated visual features and real semantic representations. In addition, the unconstrained generation of data samples for the unseen classes may produce samples far from their actual distribution. These methods have access to the semantic representations (under semantic transductive setting) and unlabeled unseen data (under transductive setting), in which violate the GZSL setting. Table 4 summarizes the GZSL settings and their problems. The main properties of each method are also highlighted in Table 4 . As can be seen, the inductive setting is the most challenging task, while transductive semantic and transductive settings are less complex.
Despite considerable progresses of GZSL, particularly in methodologies, there are challenges pertaining to the unavailability of visual samples for the unseen classes. Based on our findings, the main research gaps that require further research are as follows:
• Methodology: Although many studies to solve the domain shift problem are available in the literature, it remains a key challenging issue with respect to the existing models. Most of the existing GZSL methods are based on ideal data sets, which is unrealistic in real-world scenarios. In practice, the ideal setting is affected by uncertain disturbances, e.g. a few samples in each category are different from other samples. Therefore, developing robust GZSL models is crucial. To achieve this, new frameworks that incorporate domain classification without relying on latent space learning are required. On the other hand, techniques that are capable of solving supervised classification problems can be adopted to solve the GZSL problem, e.g. ensemble models [193] and meta-learning strategy [192] . Ensemble models employ a number of individual classifiers to produce multiple predictions, and the final decision is reached by combining the predictions [194] , [195] . Recently, Felix et al. [53] introduced an ensemble of visual and semantic classifiers to explore the multi-modality aspect of GZSL. Metalearning aims to improve the learning ability of the model based on the experience of several learning episodes [77] , [192] . GZSL also can be combined with reinforcement learning [196] to better tackle new tasks. In addition, GZSL can be extended in several fronts, which include multi-modal learning [160] , weakly supervised learning to progressively incorporate training instances from easy to hard [105] , or online learning for few-shot learning where a small portion of labeled samples from some classes are available [46] .
• Data set: Semantic representations play an important role in bridging the gap between the seen and unseen classes. The existing data sets use human-defined attributes or word vectors. The former shares the same attributes for different classes, and human labor is required for annotation, which is not suitable for large-scale data sets. The latter automatically extracts information from a large test corpus, which is usually noisy. Therefore, extracting useful knowledge from such data sets is difficult, especially fine-grained data sets. In this regard, using other modalities such as audio can improve the quality of the data samples. As such, it is crucial to focus on developing new techniques to automatically generate discriminative semantic attribute vectors, exploring other semantic spaces or combinations of various semantic embeddings that can accurately formulate the relationship between seen and unseen classes, and consequently solve the domain shift and bias problems. In addition, there are many unforeseen scenarios in realworld applications, such as driver-less cars or action recognition that may not be included among the seen classes. While GZSL can be applied to tackle such problems, no suitable data sets for such applications are available at the moment. This constitutes a key focus area for future research.
Building models that can simultaneously perform recognition for both seen (source) and unseen (target) classes are vital for advancing intelligent data-based learning models. This paper, to the best of our knowledge, for the first time presents a comprehensive review of GZSL methods. Specifically, a hierarchical categorization of GZSL methods along with their representative models has been provided. In addition, evaluation protocols including benchmark problems and performance indicators, together with future research directions have been presented. This review aims to promote the perception related to GZSL.