Paper: VBH-GNN: Variational Bayesian Heterogeneous Graph Neural Networks for Cross-subject Emotion Recognition (ICLR-2024)

Reviewer comment (Presentation):
In Table 1, there is a typo. I believe DEAT should read DEAP. Additionally, Mathod should be Method. Additionally, expanding Table 1 by including which modalities were used for each baseline would be more comprehensive.

Author response:
Thank you for carefully reading our manuscript and pointing out the typos. **We have revised these two typos**.

Following the reviewer's suggestion, **we have added the modality information in Table 1**. The Deap dataset contains EEG, EOG, EMG, and GSR modalities, and the Dreamer dataset contains EEG and ECG modalities. Since our method and baselines all utilize the full modalities available in each dataset, we do not specify the modalities used for individual baselines due to space constraints. Instead, we annotate the modalities included in each dataset in the table headers (see Table 1 on Page 8). Besides we have also added the description of modality information in the experiment setting section (see 4.1 EXPERIMENTAL SETTING).

Reviewer comment (Experiments):
Expanding Table 3 by seeing the effect of all modalities used would be more comprehensive.

Author response:
According to the reviewer's suggestion, **we have substantially expanded Table 3 to include experimental results for all individual and combined modalities**. The updated Table 3 with these additional results can be found on page 8 of the revised PDF paper.

Reviewer comment (Theory):
The paper assumes an infinite number of edges between nodes with probabilities $p_n​$ tending to zero in section 3.2.1. Could the authors clarify a bit more on the reasoning behind the assumption?

Author response:
We will explain this hypothesis through the physiological phenomena behind it as well as similar examples from other fields.

**The explanation of the infinity edges**: Physiological systems are highly complex and contain many biological processes that interact with each other in multiple ways. While some of these interactions are medically proven, such as heart rhythms altering skin temperature and unconscious eye movements due to brain activity, there are more complex and yet-to-be-fully explored interrelationships [3]. We, therefore, use edges that tend to be infinite to represent the complexity and diversity of signal relationships in physiological systems.

**The explanation of the probability of existence for edges close to 0**: Physiological systems often have multiple regulatory mechanisms, and the relationships between physiological signals may be multilevel and sublinear. Thus, most relationships between physiological signals are insignificant, and only a few relationships are meaningful in specific situations (e.g., emotions). Therefore, to reflect the sparsity of the relationships between physiological signals, we assumed that the probability of the existence of edges tends to 0 before determining the pattern of emotions. 

**Similar examples**：Similar assumptions are applied to the natural language field. For example, as mentioned in [1] and [2], there are relational structures between consecutive utterances, such as comments, analogies, and contrasts. Before producing complex relational structures among utterances, the listener unconsciously generates an infinite amount of similar data structures while the probability of relation existence is minimal (close to 0).

Reviewer comment (Reproducibility):
Could the authors clarify $\epsilon$ in equation 12 in the main paper. They mention it is a "very small hyperparameter." Does this mean that it is close to 0?

Author response:
The $\epsilon$ is a hyperparameter with a very small value 1e-5. **The $\epsilon$ had been listed in Table 5 in Appendix (A.5) of our original submission, but we inappropriately used a different symbol**.
We have updated the $\epsilon$ in Table 5 in the Appendix (A.5), and we have also added the description of $\epsilon$ in our revised submission for better clarification (see Line xxx).

| Hyper-parameter | Value |
| :--- | :---: |
| $\vdots$ | $\vdots$ |
| eps ($\epsilon$) | $1 \mathbf{e}-5$ |
| $\vdots$ | $\vdots$ |

Reviewer comment (Experiments):
Did the authors confirm that for Table 1, the comparable methods are all using the same experimental conditions as the authors (e.g., cropped experiment, experiment setup, modalities used).

Author response:
Yes, all the comparable methods use the same experimental conditions(i.e., cropped experiment, training strategy, and modalities). We have a shared Dataloader for all models.
Here are some more specific descriptions of our experiment settings:

**Modalities:**
In Table 1, **all baselines use the same modalities**: the EEG, EOG, EMG, and GSR modalities from DEAP and the EEG and ECG modalities from DREAMER.

**Training strategy:**
**We adopt the "leave-one-subject-out" (LOSO) paradigm to divide the dataset and implement a shared Dataloader**. We use one subject as the target domain and the remaining as the source domain. The training set consists of all source domain and some target domain data, and the remaining target domain data are divided into validation and testing sets.

**Cropping Experiment:**
**We employed a specific data cropping strategy for each trial**, segmenting the data into 4-second non-overlapping intervals for all methods. This data segmentation was conducted only after dividing the dataset into training and testing sets. Given the inherent time-series nature of physiological signals, adjacent segments are naturally correlated. If these segments were distributed before splitting training and testing sets, it could lead to data leakage, inflating model performance metrics like accuracy and F1 score. To ensure the reliability of our results, we applied the same cropping strategy across all models. This approach helps prevent abnormally high performance resulting from data leakage, ensuring a fair and accurate evaluation of model efficacy.

Reviewer comment (Presentation):
The quality Figure 2 needs to be improved.

Author response:
**We have redrawn Figure 3 (i.e., Figure 2 in the original version) to improve the readability of the RDA module**. Specifically, we use different background colors to denote different sub-modules and thus make the figure details more transparent.

Reviewer comment (Evaluation):
Statistical results in tables will be more convincing. In addition, the optimal results in Table 1 should all be highlighted.

Author response:
In the tables of our original submission, we only report two types of statistical results, i.e., Accuracy and F1 Score. Following the reviewer's suggestion, we have added a new type of statistical results in Table 1. **We have added the standard deviation of all Accuracy and F1 Score**. Please see Table 1 in our revised PDF version. We have been conducting experiments to calculate the standard deviation of accuracy results for other tables.

To increase the readability of the results in Table 1, we have highlighted the best, second-best, and third-best results in red, green, and blue, respectively.

Reviewer comment (Reproducibility):
Are the weights of each loss function in Formula 5 the same? Are the weights of each loss function considered?

Author response:
**Yes, the weights of all the loss functions in Formula 5 are set to be the same; specifically, the weights are equal to one**.

We have conducted additional experiments to find the best weight combination. We found that using a weight of 1 for all terms achieves the best experimental results. Therefore, we omitted the weights symbolic in the previous submission version. Based on the reviewer's comment, we have added the weight symbols in Formula 5 for a better understanding. The revised Formula 5 is as follows:

> **Loss of VBH-GNN** contains two types of loss: the RDA Loss for aligning the source domain and target domain and the prediction loss of the classifier. The final loss function is formulated as 
> $${\cal L} _{\text{VBH-GNN}} = \lambda _{1}{\cal L} _{\text{SRDA}}+\lambda _{2}{\cal L} _{\text{TRDA}}+ \lambda _{3}{\cal L} _{SBCE}+\lambda _{4}{\cal L} _{TBCE}$$ where ${\cal L} _{\text{SRDA}}$ and ${\cal L} _{\text{TRDA}}$ are loss of Spatial RDA and Temporal RDA (will be further explained in Section 3.2.3), ${\cal L} _{SBCE}$ and ${\cal L} _{TBCE}$ are Binary Cross-Entropy Loss for source and target domain classification. $\lambda _{1}$, $\lambda _{2}$, $\lambda _{3}$, and $\lambda _{4}$ are loss weights, which are all set to $1$ in the experiments.

Reviewer comment (Presentation):
The readability and visual appeal of the process diagram for RDA in Figure 2 could be improved.

Author response:
We have redrawn Figure 3 (i.e., Figure 2 in the original version) to improve the readability of the RDA module. Specifically, we use different background colors to denote different sub-modules and thus make the figure details more transparent. We have also added a global flow chart in Figure 2 to illustrate the role of each module in the overall process. Please check Figure 2 and Figure 3 in our revised PDF version.

Reviewer comment (Experiments):
The experimental setting of dividing source and target domains is unrealistic.

Author response:
We used the leave-one-subject-out (LOSO) paradigm to divide source and target domains. If we understand correctly, the reviewer means that the validation and testing sets should originate from different domains (subjects) and should not overlap. We explain why we use LOSO as follows.

**First, there are differences between domain adaptation (DA) and domain generalization (DG).** Our paper aims at the DA problem in a cross-subject emotion recognition (ER) task. The DA is similar to DG, and we speculate whether this might be causing the misunderstanding.

The main difference between DG and DA is whether the model can use the data from the target domain in the training process. For the DG, the subjects in the testing set are regarded as the target domain, and these subjects should be unseen to the trained model. To evaluate the trained model, the validation set should also be from the target domain but use a different subject from the testing set. In this case, using LOSO is unrealistic. However, for the DA, the training set typically contains data from both the source and target domains. Without access to target domain data, models cannot appropriately adapt to the new unknown domain during initial training. In other words, the DA implies using data from source subjects and the target subject for ER, so the LOSO can be used to select the target subject.


**Second, the LOSO is a commonly used experiment setup for domain adaptation (DA) in EEG-based cross-subject ER**. To support our point, we cite several typical papers and quote their relevant contents as follows:

1. Mentioned in ref [1]: 
> Specifically, only one experiment from each subject is involved in the **leave-one-subject-out** cross strategy to study the inter-subject variability.

2. Mentioned in ref [2]: 
> Based on the above EEG dataset, we adopt the **leave-one-out-cross-validation** method in the following experiments to evaluate the performance of the models in cross-subject scenarios.

3. Mentioned in ref [3]: 
> On the control experiments, we employ two transfer paradigms, i.e., 'one-to-one' and **'multi-to-one'**. ... . In the latter paradigm, when **one subject serves as the target, all the remaining subjects form the source**.

Despite their different paradigm names, they all randomly select one subject as the target domain and the rest of the subjects as the source domain. We follow their settings in our experiments.

**Third, the division of the training, testing and validation sets in our experiments is also common, and there is no data leakage**. We will explain this with the content of a published paper.

Mentioned in the baseline MMDA-VAE [4]:

> **The training set consisted of the source data and the labelled target data**, i.e., all the samples from the source session and samples from the first three or four trials (one trial per class) in the other target session. 

This means that the training set contains source and target domain data.

> We used **samples from the second three or four trials in the target session as the validation set**. 

This indicates that the validation set comprises part of the target domain data.

> The samples from **the rest of the twelve or sixteen trials in the target session were used to evaluate classification accuracy**.

This suggests that the testing set is also derived from the target domain but does not intersect with the training and validation sets. This experiment setting is the same as ours, where part of the target domain is included in the training set, while the training, validation, and testing sets do not overlap.   

**In summary, our experiment setting (i.e., LOSO paradigm) is commonly used in the cross-subject ER field and does not result in data leakage**. At the same time, we agree with the experiment setting that the reviewer suggested, and we will take the relatively more challenging DG task as our further research direction. We look forward to discussing this with you and thank you for noting the details of the experiment setup.

[1] Zhao, Li-Ming, Xu Yan, and Bao-Liang Lu. "Plug-and-play domain adaptation for cross-subject EEG-based ER." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 1. 2021.

[2] Gu, Rong-Fei, et al. "Cross-Subject Decision Confidence Estimation from EEG Signals Using Spectral-Spatial-Temporal Adaptive GCN with Domain Adaptation." 2023 International Joint Conference on Neural Networks (IJCNN). IEEE, 2023.

[3] Peng, Yong, et al. "Joint feature adaptation and graph adaptive label propagation for cross-subject ER from EEG signals." IEEE Transactions on Affective Computing 13.4 (2022): 1941-1958.

[4] Wang, Yixin, et al. "Multi-modal domain adaptation variational autoencoder for eeg-based ER." IEEE/CAA Journal of Automatica Sinica 9.9 (2022): 1612-1626.

Reviewer comment (Experiments):
After constructing the emotional graph, did the author only perform a single convolution layer by multiplying an adjacency matrix with node embeddings? Did the author consider using multiple convolution layers or using the more expressive graph neural network?

Author response:
If we understand the reviewer correctly, the reviewer asks whether we considered using a more expressive network layer to replace the RDA's Graph Attention (GA) module. We have tried using Graph Convolutional Network layers (GCNs) instead of GA, but it is unnecessary. The reasons are as follows.

**The output of EGT (EmoG) already adequately represents the relationships between nodes, so it does not need more expressive networks (such as GCNs) to represent such relationships again**. We found in our experiments that using GCNs does not improve performance compared to using GA. This is because the core idea of GCNs is to infer the relationship between nodes based on the input adjacency matrix and node embeddings and then update the node embeddings based on this relationship. However, the function of inferring the relationship between nodes is precisely accomplished by BGI and EGT. The EmoG has adequately represented the relationship between nodes under specific emotions, and the GA operation is also more efficient than the convolution layers or graph neural networks. Therefore, we simplify the GCNs to GA that retain only the function of updating node embeddings.

Reviewer comment (Evaluation):
Due to the necessity of employing Bayesian graph inference to construct multiple graphs in the author's method, I am concerned about whether the performance benefits outweigh the increased computational burden.

Author response:
If we understand the reviewer correctly, the reviewer is concerned that there is a step in BGI that couples $n$ relationships between multi-modal signals. This step constructs multiple graphs, thus incurring a substantial computational burden. However, this step does not construct multiple graphs and, therefore, does not increase the computational burden. 


**The BGI does not directly align the graphs between domains, but rather the edge existence probability distributions, so it doesn't construct multiple graphs**. In the paper, we mentioned:

> From this, we define the prior HetG edge distribution from the source domain as follows: 
> $$P(\text{HetG}|E_s) \sim \text{BIN}(n, p_s)$$

where $p_s \in \mathbb{R} ^{B \times N_{e} \times 1}$ is computed by the network and mathematically represents the probability of the existence of each of the $N_e$ edges. It is much smaller than the edge embedding matrix $E_s\in \mathbb{R}^{B \times N_{e} \times D_e}$.

To align domains by such probability distributions, we design the BGI Loss (essentially Kullback-Leibler Divergence (KLD)), which is used to minimize the divergence between the two probability distributions. However, as the reviewer is concerned, the direct computation of this KLD imposes a substantial computational burden due to the presence of $n$. To make this computation possible in the network, we propose Theorem 2, which mathematically computes an upper bound for this KLD:
> $$\mu_{lt} \log \frac{\mu_{lt}+\epsilon}{p_s+\epsilon} +(1-\mu_{lt}) \log \frac{1-\mu_{lt}+{\mu_{lt}}^2 / 2+\epsilon}{1-p_s+{p_s}^2 / 2+\epsilon}$$

which is not directly related to $n$. The computational complexity of this equation is $O(B \cdot N_e)$, which will not impose a significant computational burden.

Reviewer comment (Evaluation):
In Table 1, VBH-GNN achieves optimal accuracy, but the F1 scores consistently demonstrate poor performance. Does this observation imply the class imbalance issue in the predicted results of the method?

Author response:
Yes, the class imbalance issue is a common and challenging problem in cross-subject EEG ER. Detailed investigation of the imbalance issue will be part of our future effort. Actually, our method achieves the second-best results regarding the F1 score among all methods in most cases. For a better illustration, we updated Table 1 and highlighted the top 3 performances to show the advantages of our method.

Reviewer comment (Experiments):
The authors used leave-one-subject-out paradigm to divide the source and target domains is unrealistic. In this case, the target domain is actually a validation set, which is not the real domain adaptation setting.

Author response:
See the answer in Weakness 3.