From: ICLR.cc/2023/Conference/Paper806/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper addressed audio-visual zero-shot learning. The paper was reviewed by three expert reviewers that all argue for rejection (1 x marginally below the acceptance threshold, and 2 x reject, not good enough). While reviewers agree that the problem is interesting, they similarly agree that approach lacks novelty and theoretical or more through experimental analysis. In rebuttal authors have tried to alleviate these concerns, but ultimately the arguments did not convince the reviewers. AC concurs with recommendation of reviewers and believes that the paper is not ready for publication.  
justification_for_why_not_higher_score: Paper received weak scores from reviewers and is broadly considered not to be of sufficient novelty or quality to be published at ICLR in its current form. 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper806/Authors
title: Thanks for your valuable comments and hope for revaluating the contributions of our paper.
comment: Q1. It is unreasonable to claim the proposed method is a 'unified' one. It is difficult to say that the av-zsl is completed just because of the a-v fusion and network's generalization ability. 

A1. We identify two important issues (i.e., multi-modal coordination and generalization capacity) needed to be addressed yet seldom considered by existing methods. A unified framework means that we simultaneously tackle both of the issues in a single network. We agree with the reviewer that there is still a lot of room for AV-ZSL. We will improve the description in the revised version.

Q2. The generalization ability is close to seen class since the network is not overfitted to training data in the well-generalized case. For unseen classes, adaptation seems more crucial. 

A2. We propose a gradient-based mask generation method to enforce the model to focus on the most discriminative segments of audio and visual features, which is inspired by feature-decomposition methods for zero-shot learning methods [1,2,3]. As shown in the experiments, it can improve performances on both the seen and unseen classes.

[1] Cross-modal representation learning for zero-shot action recognition, CVPR2022.
[2] Generalized zero-shot learning via disentangled representation, AAAI2021.
[3] Hierarchical disentanglement of discriminative latent features for zero-shot learning. CVPR2019.

Q3. The authors point out the limit of cross-attention approaches in the complementary fusion of different modalities (Issue 1 in Intro). However, the attention weights indicate the importance of a modality. Then, it can desirably fuse the modalities conceptually. The authors' insistence is not acceptable in the current description. 

A3. Existing cross-attention approaches learn the attention simply guided by cross-entropy loss or reconstruction loss, which ignore the multi-modal coordination. We propose mutual knowledge distillation to enforce the audio and visual branches to learn from each other. We will improve the description.  

Q4. Which word embedding is utilized? Is the same with the baselines? 

A4. The semantic features are obtained by word2vec embedding, which is the same as the baselines.

Q5. To show the effectiveness of KD loss, the attention module should be removed in the ablation study, keeping the KD loss. 

A5. In the ablation study, our base model L_CLS and the variant +L_KD use the same attention module, which is a fair comparison and can verify the effectiveness of KD loss. Following the suggestion, we remove the attention module in both of  L_CLS and +L_KD. The H metric on the UCF-GZSL dataset are 22.14 and 30.55, respectively.  
  
Q6. In computing the cross-attention, which modality is value? 

A6. The cross-attention is computed in parallel by jointly optimizing the classification loss L_CLS, the knowledge distillation loss L_KD, and the masking loss L_M.

Q7. What is the motivation and expected effect of the masking? The current description is not enough. 

A7. Our gradient-based mask generation is inspired by recent feature-decomposition methods [1,2,3], aiming to encourage the model to focus on the most discriminative segments and benefit knowledge transfer from seen to unseen classes. We will improve the description in the revised version.

Q8. Why the pseudo labels are generated by gradient, rather than the predicted label? 

A8. Since we require per-element pseudo labels for the audio and visual features, they cannot be generated directly by the per-instance predicted label. 

Q9. [minor] typo: (line 3 of conclusions) mulit-modal -> multi-modal

A9. We will correct the typo in the revised version.

From: ICLR.cc/2023/Conference/Paper806/Authors
title: Thanks for your valuable comments and hope for revaluating the contributions of our paper.
comment: Q1. Neither distillation nor masking is the original contribution of this paper. The paper also fails to explain the challenges when applying these methods.

A1. We re-summarize the contributions of this paper, hoping for a better understanding of the novelty of our method.

(1) We identify two imperative issues that existing AV-ZSL methods seldom considered, i.e., multi-modal coordination and generalization capacity. Existing methods simply fuse the visual and audio features employing cross-modal attention guided by either the cross-entropy loss or the reconstruction loss, which ignores the multi-modal coordination. In addition, they only consider the alignment between the audio-visual features and the semantic feature. The core of zero-shot learning, i.e., the generalization capacity is totally ignored.

(2) For multi-modal coordination, we propose a two-branch framework to compute similarities with the semantic features of the labels and introduce mutual knowledge distillation between the two branches to make full use of the multi-modal information.

(3) For generalization capacity, we propose a novel gradient-based mask generation method to ensure the knowledge transfer from seen to unseen classes. To the best of our knowledge, we are the first to employ a gradient-based method for the generalization capacity of zero-shot learning.

Q2. The paper is lack of theoretical analysis of the proposed method. The ablation study is very rough and empirical. Only altering the loss function terms while observing the performance changes can only reflect the external behavior but fails to understand the internal properties of the proposed method. Also, tSNE is a qualitative method but cannot analyze the model with in-depth interpretation.

A2. Thanks for the valuable comments. Our proposed method is inspired by the recent feature-decomposition methods [1,2,3] and the interpretability of DNNs [4,5], which provide theoretical support. The ablation studies and qualitative results verify the effectiveness of our method. We will improve the analysis of our method. 

[1] Cross-modal representation learning for zero-shot action recognition, CVPR2022.
[2] Generalized zero-shot learning via disentangled representation, AAAI2021.
[3] Hierarchical disentanglement of discriminative latent features for zero-shot learning, CVPR2019.
[4] Grad-cam: Visual explanations from deep networks via gradient-based localization, ICCV2017.
[5] Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks, WACV2018.

From: ICLR.cc/2023/Conference/Paper806/Authors
title: Thanks for your valuable comments and hope for revaluating the contributions of our paper.
comment: Q1. Novelty is limited. The so-called mutual knowledge distillation is actually cross-modal contrastive learning, which has been widely used in existing audio-visual learning works.

A1. We cannot agree that mutual knowledge distillation (MKD) is the same as cross-modal contrastive learning (CCL). CCL aims to learn representations by pulling together an anchor with a positive sample and pushing away with negative samples. In contrast, our proposed MKD aligns the similarity distributions of the visual and audio branches to encourage information interaction between them. The differences can be summarized in three-fold: (1) CCL requires negative samples while MKD does not. (2) CCL performs multi-modal alignment at the feature level while MKD is at the logit level. (3) CCL can not leverage the semantic features, which is critical for zero-shot learning (ZSL), while MKD can.

Q2. The gradient-based mask generation is to keep the model focus on the important segments (or the informative segments), which is similar to the existing attention-based segment weights methods since they are both aimed at lower the weights of background (noisy) segments and try to put the model focus on the foreground (or informative) segments.

A2. Generalization capacity is at the core of ZSL but seldom considered by existing AV-ZSL approaches. Our gradient-based mask generation is inspired by recent feature-decomposition methods [1,2,3], aiming to ensure knowledge transfer from seen to unseen classes. It may look similar to attention-based segment weights methods, but is more interpretable and acts more like supervised attention.
[1] Cross-modal representation learning for zero-shot action recognition, CVPR2022.
[2] Generalized zero-shot learning via disentangled representation, AAAI2021.
[3] Hierarchical disentanglement of discriminative latent features for zero-shot learning. CVPR2019.

Q3. I did not see why these modules are proposed for generalized audio-visual zero-shot learning. The proposed method can be indeed applied to supervised/weak-supervised audio-visual learning. There is little connection to the zero-shot and especially generalized zero-shot setting.

A3. As mentioned in A1 and A2, our method is elaborated for audio-visual generalized zero-shot learning: (1) mutual knowledge distillation for leveraging the semantic information (the bridge between seen and unseen classes)  and multi-modal coordination, (2) gradient-based mask generation for knowledge transfer (at the core of ZSL) from seen to unseen classes. We believe some components of our method can be applied to other audio-visual learning problems.

Q4. Why L_KD improves a lot on UCF while not that much on the other two datasets?

A4.  L_KD can significantly improve the performances on all three datasets. For example, the H metric improves from 8.35 to 11.72 (40.4%), from 24.34 to 33.67 (38.3%), and from 5.87 to 6.76 (15.2%) on the ActivityNet-GZSL, UCF-GZSL, and VGGSound-GZSL, respectively. The higher scores on UCF-GZSL may be because UCF-GZSL has 51 classes while ActivityNet-GZSL and VGGSound-GZSL have 200 and 276, making UCF-GZSL a less challenging dataset than the other two.

From: ICLR.cc/2023/Conference/Paper806/Authors
title: Thanks to all reviewers
comment: Thanks for the valuable comments from all reviewers. We have tried our best to address the concerns of the reviewers and modify our paper, hoping the reviewers to revaluate the contributions of the proposed method and reconsider our paper.

From: ICLR.cc/2023/Conference/Paper806/Reviewer_hreg
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper proposed a new framework for Audio-Visual Zero-Shot Learning, which is a new setting to classify videos of both seen and unseen classes. The authors proposed a model with mutual knowledge distillation and pseudo-labeling mask selection. The proposed method achieves the state-of-the-art performance on this setting.
strength_and_weaknesses: Strength:
- A new setting, which seems to be fun to explore.
- High performance under the new setting

Weakness:
- Novelty is limited. The so-called mutual knowledge distillation is actually cross-modal contrastive learning, which has been widely used in existing audio-visual learning works.

- The gradient-based mask generation is to keep the model focus on the important segments (or the informative segments), which is similar to the existing attention-based segment weights methods since they are both aimed at lower the weights of background (noisy) segments and try to put the model focus on the foreground (or informative) segments.

- I did not see why these modules are proposed for generalized audio-visual zero-shot learning. The proposed method can be indeed applied to supervised/weak-supervised audio-visual learning. There is little connection to the zero-shot and especially generalized zero-shot setting. 

- Why L_KD improves a lot on UCF while not that much on the other two datasets? 

clarity,_quality,_novelty_and_reproducibility: Clarity is good, but the novelty is limited. 
summary_of_the_review: The main concern is the novelty. I can see the proposed method is similar to existing works. More importantly, these modules are not designed for generalized zero-shot learning. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper806/Reviewer_Zxhy
confidence: 1: You are unable to assess this paper and have alerted the ACs to seek an opinion from different reviewers.
summary_of_the_paper: This paper proposes multi-modal coordination with distillation and generalisation by masks. The method is evaluated against three benchmark datasets and achieved promising results, although the whole compared baselines are still quite low in performance.
strength_and_weaknesses: + Distillation is a good paradigm for ZSL tasks.
+ Masking is effective for generalisation.
- Neither distillation nor masking is the original contribution of this paper. The paper also fails to explain the challenges when applying these methods.
- The paper is lack of theoretical analysis of the proposed method.
- The ablation study is very rough and empirical. Only altering the loss function terms while observing the performance changes can only reflect the external behavior but fails to understand the internal properties of the proposed method.
- Also, tSNE is a qualitative method but cannot analyze the model with in-depth interpretation.
clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is fine. The methodology is more like a technical report that lack of theoretical analysis or proof. The proposed methods are well-motivated but not original.
summary_of_the_review: Overall, the proposed methods are interesting and intuitive. However, the novelty and theoretical contributions are the key weakness to be improved.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: NO
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper806/Reviewer_WqxW
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper addressed audio-visual zero-shot learning. To resolve the problem, the authors focused on multi-modal fusion and generalization capability of the model. 
strength_and_weaknesses: S1) the highest performance in various benchmarks

W1) It is unreasonable to claim the proposed method is a 'unified' one. It is difficult to say that the av-zsl is completed just because of the a-v fusion and network's generalization ability.
W2) The generalization ability is close to seen class since the network is not overfitted to training data in the well-generalized case. For unseen classes, adaptation seems more crucial. 
W3) the authors point out the limit of cross-attention approaches in the complementary fusion of different modalities (Issue 1 in Intro). However, the attention weights indicate the importance of a modality. Then, it can desirably fuse the modalities conceptually. The authors' insistence is not acceptable in the current description.
W4) Which word embedding is utilized? Is the same with the baselines? 
W5) To show the effectiveness of KD loss, the attention module should be removed in the ablation study, keeping the KD loss. 
W6) In computing the cross-attention, which modality is value? 
W7) What is the motivation and expected effect of the masking? The current description is not enough.
W8) Why the pseudo labels are generated by gradient, rather than the predicted label? 
W9) [minor] typo: (line 3 of conclusions) mulit-modal -> multi-modal 
clarity,_quality,_novelty_and_reproducibility: The writing is clear, and the topic is interesting. But, the contributions are not novel
summary_of_the_review: Please see the strength and weakness
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough