C OORDINATE AND G ENERALIZE : A U NIFIED F RAMEWORK FOR A UDIO -V ISUAL Z ERO -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Audio-Visual Zero-Shot Learning (AV-ZSL) aims to train a model that can classify videos of unseen classes leveraging audio and visual data, which is achieved by transferring knowledge obtained from seen classes.
We identify two imperative issues needed to be addressed: (1) How to effectively exploit both the audio and visual information?
and (2) How to transfer the knowledge from seen classes to unseen classes?
In this paper, we ameliorate both of the issues in a unified framework by enhancing two ingredients that existing methods seldom consider.
(1) Multi-Modal Coordination : Different from existing methods simply fusing the audio and visual features by attention mechanism, we further perform knowledge distillation between the visual and audio branches.
This allows information interaction between the two branches and encourages them to learn from each other.
(2) Generalization Capacity : Existing methods only consider the alignment between the audio-visual features and semantic features on the seen classes, which ignores the generalization capacity.
Inspired by the interpretability methods of
Deep Neural Networks (DNNs), we propose a novel gradient-based approach to generate transferable masks for the visual and audio features, enforcing the model to focus on the most discriminative segments and benefiting knowledge transfer from seen to unseen classes.
Extensive experiments on three challenging benchmarks, ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL, demonstrate that our proposed approach can significantly outperform the state-of-the-art methods.
I NTRODUCTION
With the fast development of Deep Neural Networks (DNNs) (He et al., 2016; Huang et al., 2017), supervised learning has made significant progress over the past few decades.
However, its success is primarily attributed to the large amount of labeled data.
To this end, Zero-Shot Learning (ZSL) is proposed to classify samples belonging to unseen classes by transferring knowledge obtained from seen classes.
Due to its ability to alleviate the reliance of labeled data, ZSL has attracted extensive research attention and been applied in various fields such as image classification (Chen et al., 2017; Bansal et al., 2018;
Xian et al., 2018; Bucher et al., 2019; Ramesh et al., 2021), object detection (Bansal et al., 2018;
Rahman et al., 2019), semantic segmentation (Bucher et al., 2019; Zhang & Ding, 2021), image generation (Ramesh et al., 2021; Jain et al., 2022) and video classification (Brattoli et al., 2020; Gao et al., 2020).
Most existing zero-shot video classification methods are limited in the visual modality, i.e. , only the image sequences of a video are utilized for classification.
Considering the natural alignment between the audio and visual data in a video, Audio-Visual Zero-Shot Learning (AV-ZSL) (Parida et al., 2020) is recently introduced to leverage both of the audio and visual information for video classification.
Compared to AV-ZSL that only videos belonging to unseen classes appear in the test phase, Audio-Visual Generalized Zero-Shot Learning (AV-GZSL) is a more realistic and challenging variant of AV-ZSL where test videos can come from both the seen and unseen classes.
Given the audio and visual data of seen classes available for training, we identify two imperative issues that need to be addressed:
Issue 1: How to effectively exploit both the audio and visual information?
Existing methods (Mazumder et al., 2021; Mercea et al., 2022b;a) simply fuse the visual and audio features employing cross-modal attention, which is guided by either the cross-entropy loss or the reconstruction loss.
However, the audio and visual information should be complementary and benefit from each other when classifying a video.
For example, seeing a piano and hearing the music sound motivates us to classify a video as PlayingPiano .
Thus, the multi-modal coordination should be considered to effectively exploit both the audio and visual information.
Issue 2: How to transfer the knowledge obtained from seen classes to unseen classes?
Generalization capacity is at the core of ZSL.
Numerous efforts have been made to improve the generalization capacity, e.g. , relation modeling (Gao et al., 2019; 2020), uniformity-aware representation learning (Pu et al., 2022), and feature decomposition (Lin et al., 2022; Tong et al., 2019; Li et al., 2021).
Existing methods for AV-ZSL, however, only consider the alignment between the audio-visual features and the semantic features.
The improvement of the generalization capacity of AV-ZSL models remains to be explored.
In this paper, we tackle both of the issues by (1) introducing mutual knowledge distillation to encourage the multi-modal coordination, (2) proposing a gradient-based method to improve the generalization capacity of the model.
Specifically, given audio and visual features extracted from pre-trained models, we propose a two-branch framework to compute similarities with the semantic features of the labels.
To encourage the information interaction between the audio and visual branches, mutual knowledge distillation is performed to align the similarity distributions.
In addition, inspired by the interpretability methods of DNNs and feature decomposition approaches for ZSL, we elaborate a novel gradient-based approach to improve the generalization capacity.
Two transferable masks are generated for the original audio and visual features, which is guided by the gradients relative to the classification loss.
This enforces the model to focus on the segment that contributes most to classification, benefiting the knowledge transfer from seen to unseen classes.
Our key contributions can be summarized as follows:
• We identify two imperative issues that existing methods seldom considered for AV-ZSL, i.e. , multi-modal coordination and generalization capacity, and propose a unified framework to tackle both of them.
• We introduce mutual knowledge distillation between the audio and visual branches to make full use of the multi-modal information.
• We propose a gradient-based mask generation method to ensure the knowledge transfer from seen to unseen classes.
• Extensive experiments on three challenging benchmarks demonstrate the effectiveness of our proposed method.
R ELATED W ORK
Our work is relevant to zero-shot video classification, audio-visual zero-shot learning, knowledge distillation, and interpretability of deep neural networks.
We briefly review the relevant work below.
Zero-Shot Video Classification.
Zero-Shot Video Classification (ZSVC) (Socher et al., 2013) aims to classify videos of unseen classes by transferring knowledge obtained from seen classes, which has attracted increasing research attention over the last few years (Xu et al., 2015; Gao et al., 2019;
Brattoli et al., 2020; Lin et al., 2022; Pu et al., 2022).
TS-GCN (Gao et al., 2019) proposes a two-stream Graph Convolutional Network framework that can leverage knowledge graphs to model the relationships between action-attribute, action-action, and attribute-attribute.
ResT (Lin et al., 2022) presents an end-to-end cross-modal to associate both visual and semantic spaces and develop a semantic transfer scheme to composite unseen visual prototypes, which alleviates information loss and the hubness problem.
AURL (Pu et al., 2022) proposes to learn representation awareness of both alignment and uniformity properties for seen and unseen classes.
A supervised contrastive loss is introduced to jointly align visual-semantic features and encourage semantic clusters to distribute uniformly.
Despite great success, existing ZSVC works are limited in the visual modality and ignore the audio information which is naturally aligned with visual data.
Audio-Visual Zero-Shot Learning.
Recently, Audio-Visual Zero-Shot Learning (AV-ZSL) is proposed to employ both of audio and visual information for video classification.
AVGZSLNet (Mazumder et al., 2021) proposes cross-modal decoder and composite triplet to enforce the audio and video embeddings to move closer to the corresponding text embeddings.
AVCA (Mercea et al., 2022b) introduces three novel benchmarks for AV-ZSL and proposes a cross-modal attention model to fuse the audio and visual information.
TCAF (Mercea et al., 2022a) is built on AVCA and presents a cross-attention transformer framework to additionally leverage the temporal information.
Despite some progress, existing methods only consider the alignment between the audio-visual features and the semantic features.
Two imperative issues, i.e. , multi-modal coordination and generalization capacity, are rarely considered.
Knowledge Distillation.
Knowledge distillation (KD) (Hinton et al., 2015) was proposed to transfer information learned from one model to another, which has been actively studied and broadly applied in various fields, e.g. , model compression (Wang et al., 2020;
Hou et al., 2020), image classification (Xu et al., 2020a; Cheraghian et al., 2021), object detection (Chen et al., 2017; Zheng et al., 2022), and video understanding (Pan et al., 2020; Tang et al., 2021).
Existing KD methods can be roughly divided into two categories: (1) logit mimicking (Hinton et al., 2015; Zhou et al., 2018; Zheng et al., 2022), and (2) feature imitation (Wang et al., 2019;
Dai et al., 2021; Guo et al., 2021).
While logit mimicking methods supervise the output logit of the student classifier by those of the teacher classifier, feature imitation approaches mimic the intermediate-level features from the hidden layers of the teacher model.
In this paper, we perform mutual distillation to facilitate the multi-modal coordination between the audio and visual branches for zero-shot learning.
Interpretability of Deep Neural Networks.
DNNs have achieved remarkable success in the past decades.
However, the end-to-end learning strategy makes DNN representations a black box, facilitating researchers to delve into the interpretability of models.
Gradient-based methods (Selvaraju et al., 2017; Lundberg & Lee, 2017; Chattopadhay et al., 2018) are the mainstream for interpreting
DNN.
Grad-CAM (Selvaraju et al., 2017) uses the gradients of a target concept flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept.
Grad-CAM++ (Chattopadhay et al., 2018) introduces pixel-wise weighting of the gradients of the output w.r.t.
a particular spatial position in the final convolutional feature map.
Inspired by the interpretability methods of DNNs, we introduce a gradient-based mask generation method to ensure knowledge transfer from seen to unseen classes.
P RELIMINARY
In this section, we formulate the AV-ZSL problem.
Let S =
{ ( a si , v si , w si , y si ) N s i =1 | a si ∈
A s , v si ∈
V s , w si ∈ W s , y si ∈ Y s } represent the seen dataset and U
= { ( a uj , v uj , w uj , y uj )
N
u j =1 | a ui ∈
A u , v uj ∈ V u , w uj ∈ W u , y uj ∈ Y u } represent the unseen one, where a si , a uj ∈ R D A indicate the D A -dimensional audio feature in the audio space
A that can be obtained from pretrained audio model; v si , v uj ∈ R D V indicate D V -dimensional visual feature in the visual space
V ; w si , w uj ∈ R D W indicate D W -dimensional semantic features ( e.g. ,word vectors) in the semantic space W ;
Y s = { y s 1 , . . .
, y sC s } and Y u
= { y u 1 , . . .
, y uC u } is the label sets of the seen and unseen classes in the label space Y , where C s and C u are the number of seen and unseen classes,
Y s ∩ Y u = ∅ .
In AV-ZSL, the objective is to learn a model f ZSL :
( A , V )
→ Y u that can recognize a testing sample belonging to the unseen classes whose samples are not available during training.
AV-GZSL is a more realistic and challenging variant of AV-ZSL where a testing sample can come from both of the seen and unseen classes, i.e. , the aim of AV-GZSL is to learn a model f GZSL :
( A , V )
→ Y s ∪ Y u .
M ETHOD
Different from traditional zero-shot video classification where only the visual data is utilized, both the audio and visual data are available in AV-ZSL.
Therefore, we identify two imperative issues for AV-ZSL: (1) How to effectively exploit both the audio and visual information?
and (2) How to transfer the knowledge obtained from seen classes to unseen classes?
In this paper, we tackle both of the issues in a unified framework, as shown in Figure 1.
Given audio and visual features extracted from pre-trained models, two transferable masks are first generated for both of the features.
We then fuse the masked features by the attention network and embed them into a common space with the semantic features to obtain the prediction scores.
In addition, mutual knowledge distillation is performed to encourage information exchange and multi-modal coordination between the audio and visual branches.
[Figure]
Inspired by the interpretability methods, two transferable masks are generated by ψ V ( • ) and ψ A ( • ) for the visual and audio features, respectively.
We then fuse the masked features by an attention network and embed them into a common space with the semantic features to obtain the prediction scores.
In addition, mutual knowledge distillation is performed to encourage information interaction and multi-modal coordination between the audio and visual branches.
Three loss functions are employed for parameter optimization, namely: the classification loss L CLS , the knowledge distillation loss L KD , and the mask loss L M .
In the following, we elaborate the main components of our method: (1) mutual knowledge distillation, (2) transferable mask generation, and (3) training and inference.
4. M UTUAL K NOWLEDGE D ISTILLATION
For a video with visual data V and audio data A , we first extract their visual and audio features using pre-trained models ϕ V ( • ) and ϕ A ( • ) , i.e. , v = ϕ V ( V ) , a = ϕ
A ( A ) , where v and a are the visual and audio features, respectively.
Then, two transferable masks m v and m a are generated by ψ V ( • ) and ψ A ( • ) :
[Equation]
[Equation]
Next, we multiply the original features with the generated masks element by element and fuse the multi-modal features using an attention network:
[Equation] [Equation]
where c v and c a are the learned visual and audio features, respectively.
ATT is the attention network consisting of multi-head attention (Vaswani et al., 2017), feed-forward network, and
Like other zero-shot problems, semantic information is employed to bridge the gap between the seen and unseen classes in AV-ZSL.
Specifically, a pre-trained text model ϕ W ( • ) is utilized to extract the semantic features W = { w 1 , . . .
, w N s } from the names of classes, where w i is the semantic feature of the i -th class and N s is the number of seen classes.
The visual, audio, and semantic features are embedded into a common space to obtain similarity scores:
[Equation]
and
[Equation]
where p v and p a represent the score distributions of visual and audio branches, respectively.
f v ( • ) , f a ( • ) , and f w ( • ) are the embedding networks for the visual, audio, and semantic features, respectively.
τ is a temperature parameter to control the sharpness of the distribution.
In order to encourage information interaction between the visual and audio branches, we apply mutual knowledge distillation on the score distribution using the Kullback-Leibler Divergence loss:
[Equation]
where D KL is the Kullback–Leibler (KL) divergence, i.e. , D KL ( p 1 ∥ p 2 )
= p 1 log p 1 p 2 .
In this way, the visual and audio branches of the model can learn from each other.
4. T RANSFERABLE M ASK G ENERATION
After obtaining the p v and p a from the visual and audio branches, the final similarity score is obtained by a convex combination:
[Equation]
The classification loss is calculated as follows:
[Equation]
Inspired by the interpretability of DNNs, we propose a gradient-based mask generation method to facilitate the knowledge transfer from seen to unseen classes.
Specifically, the gradients of the classification loss relative to the visual and audio features are computed:
[Equation]
where g v and g a are the visual and audio gradients, respectively.
To enforce the model focus on the segments that contribute most to the classification, pseudo labels of masks are generated as follows:
[Equation]
where ˜ m v and ˜ m a represent the pseudo label of visual and audio masks, respectively.
S v and S a are the sets of top-k elements of the gradients g v
and g a , respectively.
The mask loss is employed to encourage the learned masks to be close with the pseudo ones, which can be formulated as:
[Equation]
4. T RAINING AND I NFERENCE
In summary, three loss functions are employed for parameter optimization, namely: the classification loss L CLS , the knowledge distillation loss L KD , and the mask loss L M .
The overall objective function can be formulated as:
[Equation]
where λ 1 and λ 2 are the weighting coefficients for the knowledge distillation loss and the mask loss, respectively.
The overall optimization is summarized in Algorithm 1.
During inference, we first obtain the similarity score p and the class prediction c is determined by the maximum value of p :
[Equation]
E XPERIMENTS
In this section, all datasets and evaluation protocol are first introduced in detail.
Then, we present the implementation details as well as the comparison of experimental results with other state-of-the-art methods.
Eventually, ablation study proves the effectiveness of our method.
5. D ATASETS
We evaluate our method on three recently-introduced benchmarks for AVZSL:
ActivityNet-GZSL (Caba Heilbron et al., 2015), UCF-GZSL (Soomro et al., 2012), and VGGSound-GZSL (Chen et al., 2020).
The ActivityNet-GZSL dataset covers 200 daily action classes, in which 99, 51, andclasses are seen for training, unseen for validation, and unseen for testing, respectively.
The UCFGZSL dataset consists of 30 seen classes, 12 unseen validation classes, and 9 unseen testing classes.
The VGGSound-GZSL dataset is a large-scale audio-visual dataset with 138 seen classes, 69 unseen validation classes, and 69 unseen testing classes.
The overall statistics of each dataset is reported in Table 1.
In order to avoid violating the zero-shot setting, we adopt the dataset splits proposed by
AVCA (Mercea et al., 2022b) so that the classes of test samples can be disjoint from those contained in Sports1M (Karpathy et al., 2014) which we used for pre-training the feature extractor
[Table] [Table]
5. E VALUATION M ETRICS
In addition to datasets setting, the evaluation metrics are shown as following:
• U : the average per-class accuracy on test videos from unseen classes for AV-GZSL.
• S : the average per-class accuracy on test images from seen classes for AV-GZSL.
[Table]
• H : the harmonic mean value for AV-GZSL, which is formulated as:
[Equation]
• ZSL : the average per-class accuracy on test videos from unseen classes for AV-ZSL.
5. I MPLEMENTATION D ETAILS
Following the previous method (Mercea et al., 2022b), we utilize a pre-trained SeLaVi (Asano et al., 2020) model to extract the audio and visual features.
The dimension of audio and visual features, i.e. , D A and D V , are set to 512.
The semantic features are obtained by word2vec embedding with the size D W = 300 .
The mask generation networks
ψ V ( • ) , ψ A ( • ) and embedding networks f v ( • ) , f a ( • ) , f w ( • ) are two-layer MLPs.
The size of the output of the attention network and the latent space are set to 300 and 64, respectively.
Our approach is implemented with PyTorch (Paszke et al., 2019) and optimized by ADAM (Kingma & Ba, 2014) optimizer with a learning rate of 0.001.
The weighting coefficients λ
and λ 2 in
Equation 11 are determined by the grad search and set to 1 and 0.5, respectively.
The proportions of masked locations in the mask generation are set to 0.1, 0.1, and 0.2 for the ActivityNet-GZSL, UCFGZSL, and VGGSound-GZSL datasets, respectively.
The temperature parameter τ in Equation 3, and β in Equation 6 are set to 1 and 0.5, respectively.
In addition, we set the batch size to 256.
Following previous methods (Mercea et al., 2022b;a), we adopt a two-stage training and evaluation protocol.
In the first stage, we train our models on the training set and evaluate on the validation set to determine the hyperparameters.
In the second stage, we re-train the models on the union set of training and validation sets using the hyperparameters from the first stage.
The final performances are evaluated on the testing set.
5. C OMPARISON WITH S TATE - OF - THE -A RT M ETHODS
Compared Methods .
We compare our proposed method with nine approaches adopted for the same task.
Among them, ALE (Akata et al., 2015a), SJE (Akata et al., 2015b), DeViSE (Frome et al., 2013), APN(Xu et al., 2020b), and f-VAEGAN-D2 (Xian et al., 2019) are originally proposed for zero-shot image classification.
We concatenate the audio and visual features to replace the image features for AV-ZSL.
CJME (Parida et al., 2020), AVGZSLNet (Mazumder et al., 2021), AVCA (Mercea et al., 2022b), and TCAF (Mercea et al., 2022a) are the state-of-the-art AV-ZSL approaches.
ALE, SJE, and DeViSE are classical embedding-based ZSL methods, which aim to learn a linear or nonlinear mapping function to measure the compatibility between the input features and label embeddings.
APN proposes an attribute prototype network to jointly learn global and local features.
While a visual-semantic embedding layer learns global features, local features are learned through an at-
[Figure]
ActivityNet
[Table]
S
U
[Table]
L CLS
[Table]
[Figure]
tribute prototype network that simultaneously regresses and decorrelates attributes from intermediate features.
f-VAEGAN-D2 develops a feature generating framework that synthesizes CNN image features from a class embedding, which circumvents the scarceness of the labeled training data issues and combines conditional VAE and GAN architectures to obtain a more robust generative model.
CJME introduces the task of AV-ZSL and proposes a modality attention-based method to indicate which modality is dominant for classification.
AVGZSLNet proposes cross-modal decoder and composite triplet to enforce the audio and video embeddings to move closer to the corresponding text embeddings.
AVCA proposes a cross-modal attention model to fuse the audio and visual information and align the audio-visual features with the textual label embeddings.
TCAF is built on AVCA and presents a cross-attention transformer framework to additionally leverage the temporal information.
Results .
Table 2 shows the results on the ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL datasets.
It is worth noting that TCAF uses the temporal features while others not, i.e. , temporally averaged features are used in other methods and non-averaged ones in TCAF.
Our proposed method can outperform previous approaches by a large margin, including TCAF with additional temporal information.
For example, compared with previous best method AVCA using the same averaged features, our method increases H and ZSL from 27.15% to 35.27% and from 20.01% to 28.22% respectively on the UCF-GZSL dataset.
5. A BLATION S TUDY
Effectiveness of mutual distillation and interpretability-inspired representation .
In order to evaluate the importance of the proposed mutual distillation and interpretability-inspired representation method, we conduct experiments using different models on the ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL datasets.
Three variants of our model are trained in this experiment: L CLS , + L KD , and + L KD + L M .
Specifically, L CLS represents the base model using the classification loss.
+ L KD is referred to the base model with the addition of mutual distillation.
+ L KD + L M
Perplexity=5 Perplexity=
Perplexity=
Perplexity=
Perplexity=5 Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity=
Perplexity=100 Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity= Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= Perplexity=
Perplexity= is our full model where both of the proposed interpretability-inspired representation and mutual knowledge distillation are utilized.
As shown in Table 3, mutual knowledge distillation can significantly outperform the base model with only the classification loss, which verifies the importance of multi-modal coordination.
Interpretability inspired representation can further increase the performance especially ZSL , demonstrating the effectiveness of the mask generation method for improving the generalization capacity.
Influence of the masked proportion in the mask generation .
To guide the learning of the transferable masks, pseudo labels are generated by setting the top-k elements of the gradients to 1 and the rest to 0 , enforcing the model focus on the segments that contribute most to the classification.
In this experiment, we investigate the influence of the masked proportion in mask generation processing.
To be specific, we vary the proportion of masked elements from 0 to 0.4 with step 0.1 and report the harmonic mean value H .
Figure 2 shows the results on the ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL datasets from left to right.
The best performances are obtained when the masked proportions are 0.1, 0.1, and 0.2 for the ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL datasets, respectively.
5. Q UALITATIVE R ESULTS
In order to further verify the effectiveness of our proposed method, we visualize the original and learned multi-modal features using t-SNE (Van der Maaten & Hinton, 2008).
Specifically, test samples belonging to five of the unseen classes on the UCF-GZSL are employed for visualization: BandMarching , CuttingInKitchen , PlayingSitar , ShavingBeard , and WritingOnBoard .
Figure 3 shows the results of the original audio features, the learned audio features, the original visual features and the learned visual features from left to right.
Compared with the original features extracted from pre-trained models, the learned features are better separated and clustered through none samples of unseen classes available during training.
C ONCLUSIONS
In this paper, we focus on the problem of AV-ZSL where both visual and audio data are available for classifying videos belonging to unseen classes.
We identify two imperative issues needed to be addressed but ignored by existing methods, i.e. , multi-modal coordination and generalization capacity, and propose a mutual distillation and interpretability-inspired representation method.
In order to make full use of the multi-modal information, we perform knowledge distillation between the visual and audio branches.
Moreover, to facilitate the knowledge transfer from seen to unseen classes, a gradient-based method is elaborated to generate two transferable masks for the visual and audio data, enforcing the model to focus on the segments that contribute most to the classification.
Extensive experiments on three benchmarks demonstrate that our proposed approach can significantly outperform the state-of-the-art methods.