Paper: EventRPG: Event Data Augmentation with Relevance Propagation Guidance (ICLR-2024)

Reviewer comment (Novelty):
The paper motivates the need for event data augmentation with the statement that "the lack of huge event-based datasets similar to Imagenet prevents us from improving the model performance on relatively small datasets using a Pretrain-Finetune paradigm". However, there is an event camera version of ImageNet available [1], and its paper shows that pre-training on N-ImageNet can greatly improve the accuracy on other datasets via transfer learning. Therefore, this statement in the Introduction is wrong.

Author response:
Thanks for pointing out this mistake. We've removed the misleading sentences accordingly in the revised version. Here's the revised content:

"In terms of classification tasks, a number of event-based datasets, such as N-MNIST, N-Caltech101 [1], and CIFAR10-DVS [2], have been used to evaluate the performance of artificial neural networks (ANNs) and SNNs. **However, the issue of overfitting still poses a significant challenge for event-based datasets.** Data augmentation is an efficient method for improving the generalization and performance of a model."

[1] Orchard, Garrick, et al. "Converting static image datasets to spiking neuromorphic datasets using saccades." Frontiers in neuroscience 9 (2015): 437.

[2] Li, Hongmin, et al. "Cifar10-dvs: an event-stream dataset for object classification." Frontiers in neuroscience 11 (2017): 309.

Reviewer comment (Experiments):
Since N-ImageNet is available, I would like to see results on this dataset. This is similar to conventional vision research on data augmentation, where ImageNet is the best testbed. If the authors are not able to train on N-ImageNet, the mini subset can be considered, though I do not think that is a comprehensive benchmark.

Author response:
Thanks for your advice. Mini N-ImageNet is a comprehensive datasets suitable for evaluating our method. We conduct experiments on this dataset and present them in the Appendix A.1. On this relatively difficult datasets, our method still achieves best results and is also shown to effectively alleviate the overfitting problem. As shown by the notable positive effects, augmentations significantly enhance the performance of the model. In this comparison, our approach outperforms the nearest competitor NDA, achieving a higher Top-1 accuracy by $5.06\\%$ and Top-5 accuracy by $4.1\\%$. Also the loss curves in Fig. 6 highlight the efficiency of our approach in boosting the model's performance and mitigating overfitting.

| Data Augmentation | Identity | EventDrop | NDA    | EventRPG |
| :------------------ | :------- | :-------- | :----- | :------- |
| Top-1 Accuracy      | 28\.16   | 34\.18    | 35\.84 | **40\.90**   |
| Top-5 Accuracy      | 52\.14   | 60\.94    | 63\.64 | **67\.74**   |

Reviewer comment (Experiments):
The authors compare EventRPG with conventional vision methods such as Grad-CAM in Table 1. Thus, I wonder if it is possible to compare with Puzzle Mix and SaliencyMix in Table 3?

Author response:
Thanks for your comments. Comparing our RPGMix method with Puzzle Mix and SaliencyMix is crucial for a thorough analysis. We compared various saliency-based mixing methods on N-Caltech101 and SL-Animals datasets, with results detailed in Appendix A.2. 

| Dataset          | Model        | Identity | Saliency Mix | PuzzleMix | PuzzleMix (mask only) | RPGMix |
| :--------------- | :----------- | -------: | -----------: | ---------------------------: | ---------------------------: | -----: |
| N-Caltech101     | Spiking-VGG11       | 75\.7    | 72\.63       | 79\.38                       | 78\.13                       | **81\.75** |
| SL-Animals-4Sets | SEW-Resnet18 | 85\.42   | 84           | 85\.34                       | 85\.68                       | **88\.67** |
| SL-Animals-3Sets | SEW-Resnet18 | 89\.09   | 89\.29       | 88\.39                       | 90\.18                       | **90\.45** |

Contrary to Saliency Mix and Puzzle Mix, which can sometimes reduce performance, our RPGMix consistently has a positive impact on both datasets. This proves the advantage of RPGMix compared with Puzzle Mix and Saliency Mix on event-based data.

Reviewer comment (Experiments):
NDA applies the method to unsupervised contrastive learning and shows promising results. Is it possible to conduct such experiments using EventRPG?

Author response:
In line with NDA, we apply SimSiam [1] for unsupervised pretraining on a Resnet-18 model, distinguishing between non-mixing and mixing augmentations. SimSiam employs non-mixing augmentations to create varied samples from a single one, which we adopt for EventDrop's unsupervised pretraining, as it exclusively uses non-mixing augmentations. For EventRPG and NDA, which incorporate both types of augmentations, we follow the Mixsiam [2] approach for unsupervised contrastive pretraining.

Within the SimSiam training framework, the predictor's output layer does not reflect classification results. Therefore, implementing CLRP at the output layer is not feasible. To address this, we initialize the relevance scores of the output layer with the neural network's output values.

During the finetuning phase, to ensure a fair comparison, we do not apply any augmentations to the data. As a result, our outcomes are expected to be lower than those presented in the original paper by NDA [3].

| Radom Initialized | ImageNet Pretrain | EventDrop | NDA    | EventRPG |
| :---------------- | :---------------- | :-------- | :----- | :------- |
| 47\.56            | 74\.32            | 74\.56    | 74\.56 | **76\.81**   |

The table shows that pretraining markedly improves model performance. Models pretrained using EventDrop and NDA perform comparably to those pretrained on ImageNet. However, models pretrained with our approach show a notable $2.49\\%$ improvement. This highlights the promise of applying EventRPG to unsupervised contrastive learning, an area we intend to investigate more in future work. Therefore, these results are not included in the current manuscript.

[1] Chen, Xinlei, and Kaiming He. "Exploring simple siamese representation learning." In CVPR 2021.

[2] Guo, Xiaoyang, et al. "Mixsiam: a mixture-based approach to self-supervised representation learning." arXiv preprint arXiv:2111.02679 (2021).

[3] Li, Yuhang, et al. "Neuromorphic data augmentation for training spiking neural networks." In ECCV 2022.

Reviewer comment (Presentation):
The description of RPGMix is quite simple and unclear. Section 4.3 fails to clearly illustrate the algorithm flow of RPGMix. Many operations in Fig 3(b) are not carefully analyzed, such as sample position in the Nonoverlapping Region, which makes it hard to understand. I think Fig3 is the main figure of this paper and Fig 3(b) accounts for the most part of Fig3, hence the authors need to spend more space to describe it. Otherwise, the readers may feel confused about RPGMix.

Author response:
We apologize for the earlier lack of clarity. We have added the motivation and more details of the implementation in the Section 4.3. Here's the content:

"**Event-based data, in contrast to image-based data, does not include color details, with the most crucial aspect being the texture information it contains. The overlapping of label-related objects will impair the texture details of these objects, which in turn further degrades the quality of features extracted in SNNs. Building upon this motivation, we propose Relevance Propagation Guided Event Mix (RPGMix).** The whole mixing strategy is illustrated in Fig. 2b. For two event-based data candidates, we utilize relevance propagation to localize the label-related regions and obtain two bounding boxes. **To mix two objects with clear texture features, we randomly select two positions ensuring minimal overlap of their bounding boxes. This involves initially positioning one box at a corner to maximize the nonoverlapping area for the other box's placement, then selecting positions for both boxes in order, maintaining minimal overlap and maximizing sampling options.** Finally, the two event streams are moved to the sampled positions. Although this linear translation part prevents the overlapping of label-related objects, the background of one object would still overlap with the other object. Moreover, in one single time step, the representation ability of the spiking neurons (which only output binary information) is much worse than that of the activation layer (usually ReLU) of ANNs, making them less capable of spatial resolution and more likely to fall into local optima. Therefore, to promise the presence of only events from a single event stream candidate per pixel, avoiding regions with mixed information from interfering with the SNN, we adopt a CutMix strategy to mask the two event streams based on the bounding box of the second event stream, as demonstrated in the left part in Fig. 2b. **[1] takes the sum of each sample's mask as the ratio of their corresponding labels. This ensures that the proportion of labels in the mixed label matches the proportion of pixels belonging to each sample. In our approach, we further aim to align the proportion of labels with the proportion of label-related pixels, which can be estimated using the bounding boxes. As a result, the labels of the two event streams are mixed as** 
$$L_{mix} = \frac{L_{1}(w_{1}h_{1} - S_{overlap}) + L_{2}w_{2}h_{2}}{w_{1}h_{1} + w_{2}h_{2} - S_{overlap}},$$
where $w_{i}$ and $h_{i}$ denote the width and height of the object in the event stream $i$. $L_1$ and $L_2$ are the one-hot labels of the two event streams and $S_{overlap}$ is the area of the overlapping region of the two bounding boxes."



[1] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local statistics for optimal mixup." In ICML 2020.

Reviewer comment (Theory):
Equation 15 lacks physical meaning and theoretical basis. It needs more explanations for researchers to make further progress.

Author response:
We apologize for the earlier lack of clarity. We have added the motivation and more details of the implementation in the Section 4.3. Here's the content:

"**Event-based data, in contrast to image-based data, does not include color details, with the most crucial aspect being the texture information it contains. The overlapping of label-related objects will impair the texture details of these objects, which in turn further degrades the quality of features extracted in SNNs. Building upon this motivation, we propose Relevance Propagation Guided Event Mix (RPGMix).** The whole mixing strategy is illustrated in Fig. 2b. For two event-based data candidates, we utilize relevance propagation to localize the label-related regions and obtain two bounding boxes. **To mix two objects with clear texture features, we randomly select two positions ensuring minimal overlap of their bounding boxes. This involves initially positioning one box at a corner to maximize the nonoverlapping area for the other box's placement, then selecting positions for both boxes in order, maintaining minimal overlap and maximizing sampling options.** Finally, the two event streams are moved to the sampled positions. Although this linear translation part prevents the overlapping of label-related objects, the background of one object would still overlap with the other object. Moreover, in one single time step, the representation ability of the spiking neurons (which only output binary information) is much worse than that of the activation layer (usually ReLU) of ANNs, making them less capable of spatial resolution and more likely to fall into local optima. Therefore, to promise the presence of only events from a single event stream candidate per pixel, avoiding regions with mixed information from interfering with the SNN, we adopt a CutMix strategy to mask the two event streams based on the bounding box of the second event stream, as demonstrated in the left part in Fig. 2b. **[1] takes the sum of each sample's mask as the ratio of their corresponding labels. This ensures that the proportion of labels in the mixed label matches the proportion of pixels belonging to each sample. In our approach, we further aim to align the proportion of labels with the proportion of label-related pixels, which can be estimated using the bounding boxes. As a result, the labels of the two event streams are mixed as** 
$$L_{mix} = \frac{L_{1}(w_{1}h_{1} - S_{overlap}) + L_{2}w_{2}h_{2}}{w_{1}h_{1} + w_{2}h_{2} - S_{overlap}},$$
where $w_{i}$ and $h_{i}$ denote the width and height of the object in the event stream $i$. $L_1$ and $L_2$ are the one-hot labels of the two event streams and $S_{overlap}$ is the area of the overlapping region of the two bounding boxes."



[1] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local statistics for optimal mixup." In ICML 2020.

Reviewer comment (Evaluation):
The objective faithfulness of SLRP and SLTRP is not outstanding enough. In N-Cars, DVSGesture, and SL-Animals datasets, the proposed methods have similar or even inferior performance compared to other algorithms.

Author response:
In this comparison, although traditional methods may perform well on specific datasets, our methods stand out for their overall superior performance. To provide a comprehensive comparison, we calculated the average improvement percentage for each method.

The A.I. or A.D. score of method $ i $ on dataset $j $ is denoted as $ v_{i, j} $. We compute each dataset's mean value as $ v_{mean,j}=\\frac{1}{N_{a}}\\sum_i v_{i,j} $, where $ N_a $ represents the count of augmentations. The augmentation improvement percentage is $ v'\_{i,j} =\\frac{(v_{i, j} - v_{mean, j})}{v_{mean, j}}$. Finally, we average the improvement for each method across datasets for A.I. or A.D. as $ I_{i}=\\frac{1}{N_{d}}\\sum_j v'_{i,j} $, with $ N_d $ being the number of datasets. Here's the averaged improvement of A.I. and A.D. In this comprehensive statistical analysis, SLTRP-Saliency Map and SLRP-Saliency Map surpass the performance of other methods, proving their efficiency.

| Method             | Average A.I. Improvement ↑ | Average A.D. Improvement ↓ |
| :----------------- | -------------------------: | -------------------------: |
| SAM                | -0\.52                     | -0\.04                     |
| Grad-CAM           | -0\.43                     | 1\.29                      |
| Grad-CAM++         | -0\.07                     | -0\.30                     |
| SLRP-RelCAM        | 0\.13                      | -0\.25                     |
| SLTRP-RelCAM       | 0\.13                      | -0\.25                     |
| SLRP-CAM           | -0\.05                     | 0\.18                      |
| SLTRP-CAM          | -0\.05                     | 0\.18                      |
| SLRP-Saliency Map  | **0\.44**                      | **-0\.41**                     |
| SLTRP-Saliency Map | 0\.42                      | -0\.40                     |                 |

Additionally, CAMs produced by previous methods highlight the model's focused regions at the feature map level, while saliency maps obtained from SLRP and SLTRP disclose the model's attention at a more specific pixel level.

Reviewer comment (Evaluation):
If anything, I would like to notice here that the improvements compared to competing methods are generally small. It would be interesting to see an apples-to-apples comparison in therms of compute overhead (which is mentioned, but I do not see numberical benchmark results); or, another compelling reason to use the presented methods vs e.g. second best.

Author response:
Thanks for pointing out this missing experiment. In the updated version of our paper, we've included time consumption experiments (refer to section 5.2.3). It can be seen that as batch size increases, the time cost of our method decreases in a near-linear manner. Particularly when the batch size per GPU exceeds 4, the augmentation time for EventRPG aligns closely with that of EventDrop and NDA. Additionally, we've conducted new experiments comparing various augmentation methods on the mini N-ImageNet dataset (see Appendix A.1) and in the context of unsupervised contrastive learning (refer to our responses to Reviewer rLwX's W4 query). Under these scenarios, our method surpasses both EventDrop and NDA without incurring extra computational costs, as all batch sizes per GPU are set above 4. Additionally, to the best of our knowledge, our results on N-Caltech101, CIFAR10-DVS, and SL-Animals represent the state-of-the-art (SOTA) in the direct training track for Spiking Neural Networks (SNNs).

Reviewer comment (Writing):
In abstract, expand SNN as 'Spiking Neural Networks', for the benefit of readers unfamiliar with the abbreviation.

Author response:
Thanks for the clarification. We expand "SNN" to "Spiking Neural Networks" in the Abstract and instead of Fig. 1, we include a brief citation to introduce the failure case, maintaining brevity in the text. This also helps to save more space for other valuable content.

Here's the revised version of the introduction to failure cases:

"Nevertheless, current mixing augmentation strategies in event-based field do not consider the size and location information of label-related objects, **and thus may produce events with incorrect labels and disrupt the training process [1, 2]. To address this problem in image processing field, They mix the label-related objects together based on the saliency information obtained from neural networks.** This paradigm achieves better results compared with conventional non-saliency augmentations."

[1] Uddin, A. F. M., et al. "Saliencymix: A saliency guided data augmentation strategy for better regularization." In ICLR 2020.

[2] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local statistics for optimal mixup." In ICML 2020.

Reviewer comment (Presentation):
Fig. 1 highlights a failure case of one of the augmentation methods; it would be great to see a citation that explains why this limitation is difficult to alleveiate (e.g. by considering correct label classes). The image also seems excessive, and a single-sentence explanation should be enough.

Author response:
Thanks for the clarification. We expand "SNN" to "Spiking Neural Networks" in the Abstract and instead of Fig. 1, we include a brief citation to introduce the failure case, maintaining brevity in the text. This also helps to save more space for other valuable content.

Here's the revised version of the introduction to failure cases:

"Nevertheless, current mixing augmentation strategies in event-based field do not consider the size and location information of label-related objects, **and thus may produce events with incorrect labels and disrupt the training process [1, 2]. To address this problem in image processing field, They mix the label-related objects together based on the saliency information obtained from neural networks.** This paradigm achieves better results compared with conventional non-saliency augmentations."

[1] Uddin, A. F. M., et al. "Saliencymix: A saliency guided data augmentation strategy for better regularization." In ICLR 2020.

[2] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local statistics for optimal mixup." In ICML 2020.

Reviewer comment (Evaluation):
I am curious if the better performance / saliency on action-related datasets is an artifact of the event camera itself - since the density of the events correlates with motion and SNN may get more cues on regions with faster motion.

Author response:
Your suggestion to validate our method's selectivity via qualitative comparison is excellent. We attempted to replicate leading optical flow estimation works like TMA [1] and E-RAFT [2], but their primary training on autonomous driving datasets somewhat restricts their effectiveness in object recognition tasks. As an alternative, we have visualized additional saliency map results using SLTRP in Appendix A.3. We found that saliency maps don't concentrate on high event density areas, but rather on objects or actions related to the label. For instance, in SL-Animals results (second row), the saliency maps focus on the person's hand raised to their head, assigning minimal attention to the lower area of the person, despite its high event density. Similarly, in the fourth row, the saliency maps show little interest in the person's right hand (from our perspective), focusing instead on the left hand, even though both hands have similar event densities. We believe that it's the label-related actions, rather than event density, that truly draw the focus of the saliency maps. This observation further confirms the selectivity of the SLTRP-generated saliency maps in both action recognition and object recognition datasets (see Fig. 8).

[1] Liu, Haotian, et al. "TMA: Temporal Motion Aggregation for Event-based Optical Flow." In ICCV 2023.

[2] Gehrig, Mathias, et al. "E-raft: Dense optical flow from event cameras." In 3DV 2021.