  Spiking Neural Networks (SNNs) compute and communicate with asynchronous
binary temporal events that can lead to significant energy savings with
neuromorphic hardware. Recent algorithmic efforts on training SNNs have shown
competitive performance on a variety of classification tasks. However, a
visualization tool for analysing and explaining the internal spike behavior of
such temporal deep SNNs has not been explored. In this paper, we propose a new
concept of bio-plausible visualization for SNNs, called Spike Activation Map
(SAM). The proposed SAM circumvents the non-differentiable characteristic of
spiking neurons by eliminating the need for calculating gradients to obtain
visual explanations. Instead, SAM calculates a temporal visualization map by
forward propagating input spikes over different time-steps. SAM yields an
attention map corresponding to each time-step of input data by highlighting
neurons with short inter-spike interval activity. Interestingly, without both
the backpropagation process and the class label, SAM highlights the
discriminative region of the image while capturing fine-grained details. With
SAM, for the first time, we provide a comprehensive analysis on how internal
spikes work in various SNN training configurations depending on optimization
types, leak behavior, as well as when faced with adversarial examples.
