While deep neural networks (DNNs) are being increasingly used to make predictions from high-dimensional, complex data , they are widely seen as uninterpretable "black boxes", since it can be difficult to discover what input information is used to make predictions. This ability is particularly important for applications in cognitive neuroscience and neuroinformatics . A saliency map is a common approach for producing interpretable visualizations of the relative importance of input features for a prediction. However, many methods for creating these maps fail due to  focusing too much on the input or being extremely sensitive to small input noise . It is also challenging to quantitatively evaluate how well saliency maps correspond to the truly relevant input information  . In this paper, we develop two quantitative evaluation procedures for saliency methods, using the fact that the Human Connectome Project (HCP) dataset contains functional magnetic resonance imaging (fMRI ) datafrom multiple tasks per subject to create ground truth saliency maps . We then introduce an adversarial training method that makes DNNs robust to small input noise, and demonstrate that it measurably improves interpretability .
Deep neural networks (DNNs) are being increasingly used to make predictions from functional magnetic resonance imaging (fMRI) data. However , they are widely seen as uninterpretable "black boxes", as it can be difficult to discover what input information is used by the DNN in the process, something important in both cognitive neuroscience and clinical applications . A saliency map is a common approach for producing interpretable visualizations of the relative importance of input features for a prediction. However, methods for creating maps often fail due to DNNs being sensitive to input noise, or by focusing too much on the input and too little on the model . It is also challenging to evaluate how well saliency maps correspond to the truly relevant input information , as ground truth is not always available . In this paper, we review a variety of methods for producing gradient-based saliency maps, and present a new adversarial training method we developed to make DNNs robust to input noise, with the goal of improving interpretability. We introduce two quantitative evaluation procedures for saliency map methods in fMRI, applicable whenever a DNN or linear model is being trained to decode some information from imaging data. We evaluate the procedures using a synthetic dataset where the complex activation structure is known, and on saliency maps produced for DNN and linear models for task decoding in the Human Connectome Project (HCP) dataset . Our key finding is that saliency maps produced with different methods vary widely in interpretability, in both in synthetic and HCP fMRI data. Strikingly, even when DNN and linear models decode at comparable levels of performance, DNN saliency maps score higher on interpretability than linear model saliency maps (derived via weights or gradient). Finally, saliency maps produced with our adversarial training method outperform those from other methods .
R
While deep
Deep
R
high-dimensional, complex data
functional magnetic resonance imaging (fMRI) data. However
R
since
as
R
to make predictions. This ability is particularly important for applications in
by the DNN in the process, something important in both
R
neuroinformatics
clinical applications
D
many
R
these maps
maps often
A
DNNs being sensitive to input noise, or by
R
or being extremely sensitive to small input noise
and too little on the model
D
quantitatively
A
, as ground truth is not always available
R
develop
review a variety of methods for producing gradient-based saliency maps, and present a new adversarial training method we developed to make DNNs robust to input noise, with the goal of improving interpretability. We introduce
R
methods, using
map methods in fMRI, applicable whenever a DNN or linear model is being trained to decode some information from imaging data. We evaluate the procedures using a synthetic dataset where the complex activation structure is known, and on saliency maps produced for DNN and linear models for task decoding in
D
fact that the
R
contains functional magnetic resonance imaging (fMRI ) datafrom multiple tasks per subject to create ground truth saliency maps . We then introduce an
. Our key finding is that saliency maps produced with different methods vary widely in interpretability, in both in synthetic and HCP fMRI data. Strikingly, even when DNN and linear models decode at comparable levels of performance, DNN saliency maps score higher on interpretability than linear model saliency maps (derived via weights or gradient). Finally, saliency maps produced with our
R
that makes DNNs robust to small input noise, and demonstrate that it measurably improves interpretability
outperform those from other methods