  While Deep Learning (DL) is often considered the state-of-the art for
Artificial Intelligence-based medical decision support, it remains sparsely
implemented in clinical practice and poorly trusted by clinicians due to
insufficient interpretability of neural network models. We have tackled this
issue by developing interpretable DL models in the context of online detection
of epileptic seizure, based on EEG signal. This has conditioned the preparation
of the input signals, the network architecture, and the post-processing of the
output in line with the domain knowledge. Specifically, we focused the
discussion on three main aspects: 1) how to aggregate the classification
results on signal segments provided by the DL model into a larger time scale,
at the seizure-level; 2) what are the relevant frequency patterns learned in
the first convolutional layer of different models, and their relation with the
delta, theta, alpha, beta and gamma frequency bands on which the visual
interpretation of EEG is based; and 3) the identification of the signal
waveforms with larger contribution towards the ictal class, according to the
activation differences highlighted using the DeepLIFT method. Results show that
the kernel size in the first layer determines the interpretability of the
extracted features and the sensitivity of the trained models, even though the
final performance is very similar after post-processing. Also, we found that
amplitude is the main feature leading to an ictal prediction, suggesting that a
larger patient population would be required to learn more complex frequency
patterns. Still, our methodology was successfully able to generalize patient
inter-variability for the majority of the studied population with a
classification F1-score of 0.873 and detecting 90% of the seizures.
