Paper: Learnable Counterfactual Attention for Singer Identification (ICLR-2024)

Reviewer comment (Experiments):
From the experiments in the original CAL method. It seems that the refinement of the counterfactual attention branch didn't improve the recognition performance. What is the motivation to refine the counterfactual attention branch? Did you find any insights from the preliminary experiments to support this enhancement path?

Author response:
Thank you for your comment. Yes, experiments (Table 1) show that CAL's random attention maps are ineffective for SID task. This is due to the distinct nature of music tasks compared to vision tasks. For instance, in the SID task, singing voices are distributed in most frequency bands in the spectrogram, whereas in the image classification task, the target object usually occupies only a small portion of the image. This leads to the ineffectiveness of using random attention maps as counterfactual interventions in CAL for SID task. We addressed this by developing the LCA mechanism, which uses learnable attention maps to target specific biases instead of CAL's random attention maps. This approach improves SID and music genre classification performance (see Table 3 in original manuscript and Table A1 in appendix).

Reviewer comment (Theory):
As shown in Figure 2 and Section 3.2, CAL applies the counterfactual "intervention" to cut off the causal relations between image and attention. The goal of this "intervention" is to obtain the independent effects and further benefit the calculation of the Total Direct Effect. Here, both the counterfactual and factual branches are learned from the image. From the perspective of causality, how can you guarantee the independence between counterfactual and factual attention for the causal inference?

Author response:
Our study, as shown in Table 1, reveals that using random attention maps as counterfactual interventions to cut off the causal link between feature maps and attention is ineffective for the SID task. We believe that forcibly maintaining such independence in model training is not always advantageous. We suggest that a data-driven approach, where counterfactual attentions could automatically learn from data, identify biased regions, could be more advantageous. This approach was validated in SID experiments (Table1 and 3) and further supported by music genre classification experiments (Table A1 in the appendix), proving that our LCA mechanism's strategy of linking feature maps and attention (i.e., dependence assumption) outperforms the CAL mechanism in music tasks.

Reviewer comment (Experiments):
For the extra loss terms, there are two contradictory losses, one is to make the counterfactual attention meaningful and the other is to make it not too meaningful. How can we balance these two losses? Is it robust for different scenarios? Do the hyper-parameters of loss rates matter for the performance?

Author response:
Thank you. We balance the two loss functions in our counterfactual attention branch by adjusting the hyperparameters $\lambda_{ce}^{cf}$ and $\lambda_{ent}^{cf}$ (see formula 1). In fact, in the SID task, we did not extensively experiment with these hyperparameters, setting them based on the original audio files and vocal-only tracks, without considering duration (3, 5, or 10 seconds) (see Table 3). For the new music genre classification task (see Table A1 in appendix), we also employed the same hyperparameters used in the SID task for original audio files, and achieved excellent performance, demonstrating our method's robustness. We think designing ablation experiments for various tasks to optimize hyperparameters for loss functions can enhance performance. However, from our experience, satisfactory results are achievable in our design as long as $\lambda_{ce}^{cf}$ is greater than $\lambda_{ent}^{cf}$.

Reviewer comment (Experiments):
The motivation of this paper is to improve counterfactual attention learning. Given this goal, it is better to evaluate the proposed method and the key baseline (CAL) in the same settings, such as CUB, Cars, Aircraft, and so on.

Author response:
This study focuses on evaluating the CAL mechanism's efficacy in music-related tasks. In our experiments (see Table 1), we have demonstrated the LCA mechanism's superior efficacy over CAL in the SID task (a fine-grained challenge (Kuo et al., 2021)) under identical experimental conditions. Additionally, we confirmed our LCA mechanism's superiority over CAL in music genre classification (see appendix).

Reviewer comment (Experiments):
Only one dataset is used. The evaluation process seems to be not solid. It is suggested to add more.

Author response:
We have included a music genre classification task in the appendix, which serves to further assess the performance of our method.

Reviewer comment (Experiments):
As shown in Table 1, which counterfactual attention is used as a baseline, random, mean, or shuffle?

Author response:
In our original manuscript, we have clearly described in the results section (section 4.2) for Table 1 that we adopted random attention in the CAL mechanism. This approach of using random attention is also recommended by the original authors of the CAL mechanism (Rao et al., 2021).

Reviewer comment (Reproducibility):
What are the loss rates used in the experiments?

Author response:
In the SID task, the weight of each loss function is as follows:

Original audio file setting:
$\lambda_{ce}^{main}$=  1.0,
$\lambda_{ce}^{effect}$= 1.0, 
$\lambda_{ce}^{cf}$= 0.3, 
$\lambda_{ent}^{cf}$= 0.25, 
$\lambda_{1}^{att}$= 1.0, 
$\lambda_{ent}^{main}$= 0.2

Vocal only setting:
$\lambda_{ce}^{main}$=  1.0,
$\lambda_{ce}^{effect}$= 1.0, 
$\lambda_{ce}^{cf}$= 0.8, 
$\lambda_{ent}^{cf}$= 0.025, 
$\lambda_{1}^{att}$= 1.0, 
$\lambda_{ent}^{main}$= 0.02

For the newly implemented music genre classification task, we have set the weight of each loss function to be the same as those used in the original audio file setting of the SID task. We have added the hyperparameter setting (weight) of each loss function in the appendix. Thank you.

Reviewer comment (Experiments):
My main concern is the generality of the proposed method. The proposed method is motivated by the limitations of the existing CAL method. However, this paper only evaluates the method on the SID task, which is a less popular and competitive task compared to the tasks considered in CAL. According to the analysis provided in Section 3.2, LCA didn't add assumptions on the data or task types over CAL. So it is not clear why the proposed method is only evaluated on the SID task. Considering ICLR is a machine learning conference, I think showing the generality of the proposed method is also helpful to make the paper more suitable for publishing on ICLR and interest the audience of the conference.

Author response:
For Weakness 1:

Thank you for your comment. Our study primarily investigates the efficacy of the CAL mechanism in music-related tasks, particularly in the SID task. Through experiments (as shown in Table 1), we found that the CAL mechanism is ineffective for the SID task. This is due to the distinct nature of music-related tasks compared to vision tasks. For instance, in the SID task, singing voices are distributed in most frequency bands in the spectrogram, whereas in the image classification task, the target object usually occupies only a small portion of the image. This leads to the ineffectiveness of using random attention maps as counterfactual interventions in CAL for SID task. To overcome this limitation, we developed the LCA mechanism. It replaces the random attention maps used as counterfactual interventions in CAL with attention maps that automatically focus on biased regions, generated by a learnable counterfactual attention branch. This improvement has been proven effective in enhancing the SID task in experiments (see Table 3).

On the other hand, as you pointed out, the LCA mechanism does not add assumptions about data or task types compared to CAL. This is because our initial design intention of the LCA mechanism was to allow it to learn from data, thereby generating counterfactual attention maps that benefit the model. We expect this design to be more easily applicable to different tasks, thereby enhancing its universality.

Finally, the SID task is a core topic in the field of Music Information Retrieval (MIR). However, following your valuable suggestion, we have added a task for music genre classification in the appendix to validate the universality and effectiveness of our proposed LCA mechanism in music-related tasks.

Reviewer comment (Evaluation):
The authors claim the method can "guide the main branch to deviate from those regions, thereby focusing attention on discriminative regions to learn singer-specific features in fine-grained vocals". I think it would be better to provide some quantitative evidence to support this claim.

Author response:
For Weakness 2:

Thank you for your comment. In Table 3 of our original manuscript, we have already demonstrated that integrating the LCA mechanism into the CRNN\_FGNL model (i.e., CRNN\_FGNL (w/ LCA)) leads to a comprehensive enhancement in performance. This is attributed to the counterfactual attention branch learned from the LCA mechanism, which effectively redirects the main branch from less discriminative regions (bias regions) to those with higher discriminative potential (singer-specific regions). Moreover, the t-SNE visualization in Figure 4 further bolsters our assertion that CRNN\_FGNL (w/ LCA) demonstrates a significantly stronger discriminative ability in the learned features (i.e., can distinguish between different singers more effectively), as compared to the original CRNN\_FGNL model. 

Additionally, in the appendix, we have included an experiment to further confirm that the counterfactual branch can focus on common data biases (see Figure A1), thus encouraging the main branch to deviate from these biases and enhancing the discriminative nature of its features.

Reviewer comment (Theory):
Regarding Characteristic 3 (Regions of focus should differ from the main branch’s attention), the manuscript appears to implicitly assume superior performance of the main branch. However, empirical observations suggest that the main branch’s performance is not as exemplary as presumed. A straightforward deviation from the main branch's attention map might inadvertently introduce inaccuracies. This aspect of the methodology warrants a more cautious approach and a thorough examination to validate its effectiveness and mitigate potential risks of error introduction.

Author response:
Thank you for your comment. During our model training process, we intentionally designed the counterfactual attention branch to perform less effectively than the main branch. This is to ensure that the counterfactual attention branch focuses on learning the biases present in the data, specifically the regions with weaker discriminative power, thereby compelling the main branch to avoid concentrating on these biased regions. The experimental results presented in Table 4 of our manuscript confirm our design approach: the performance of the counterfactual branch is significantly lower than that of the main branch, indicating that the regions it focuses on have lower discriminative capabilities. Based on this, we further emphasize maximizing the differences in the attention maps between the counterfactual and main branches, aiming to direct the main branch's focus to regions distinct from those the counterfactual branch concentrates on. As demonstrated in Table 3, incorporating the loss function from Characteristic 3, our proposed LCA mechanism, namely CRNN\_FGNL (w/LCA), outperforms the previous state-of-the-art CRNN\_FGNL in all aspects, whether it be on original audio files or vocal-only tracks. However, we agree with your perspective that maximizing the differences in the attention maps between the counterfactual and main branches could inadvertently introduce inaccuracies. Therefore, our future research will focus on how to further discern the impact on classification from the regions that the counterfactual branch concentrates on.