Towards trustworthy explanations with gradient-based attribution methods
Partners for the Future Program
Cold Spring Harbor Laboratory Friends Academy
Abstract
The low interpretability of deep neural networks (DNNs) remains a key barrier to their wide-spread adoption in the sciences.
Attribution methods offer a promising solution, providing feature importance scores that serve as ﬁrst-order model explanations for a given input.
In practice, gradient-based attribution methods, such as saliency maps, can yield noisy importance scores depending on model architecture and training procedure.
Here we explore how various regularization techniques affect model explanations with saliency maps using synthetic regulatory genomic data, which allows us to quantitatively assess the efﬁcacy of attribution maps.
Strikingly, we ﬁnd that generalization performance does not imply better saliency explanations; though unlike before, we do not observe a clear tradeoff.
Interestingly, we ﬁnd that conventional regularization strategies, when tuned appropriately, can yield high generalization and interpretability performance, similar to what can be achieved with more sophisticated techniques, such as manifold mixup .
Our work challenges the conventional knowledge that model selection should be based on test performance; another criterion is needed to sub-select models ideally suited for downstream post hoc interpretability for scientiﬁc discovery.
Introduction
Deep neural networks (DNNs) have demonstrated promising predictive performance across many scientiﬁc domains, including regulatory genomics, and post hoc model interpretations often reveal that they base their decisions on meaningful patterns, such as sequence motifs [1, 2, 3, 4].
However, it is well known that model explanations with attribution methods – such as saliency maps [5], integrated gradients [6] and DeepLIFT [7] – are fragile [8, 9], making it a challenge to know whether a feature is truly important or just an artifact of the interpretability method.
The question remains, how should we choose which DNN to interpret for scientiﬁc discovery?
The current model selection standard is based on the model that yields the best generalization performance.
Should test performance be the basis for model selection when the downstream application is model interpretability?
To yield higher quality post hoc explanations, there have been several approaches that have focused on improving the attribution method itself [10, 11] or imposing direct regularization on the attribution maps during training [12, 13].
Other approaches – such as adversarial training [14], manifold mixup [15], and randomized smoothing [16] – attempt to solve the crux of the issue by encouraging DNNs to learn more robust and smoother functions, which as a consequence, leads to better explanations with gradient-based attribution methods [17].
However, in many ﬁelds like computer vision, it remains a challenge to quantitatively compare the efﬁcacy of local explanations, because there often is no pixel-level ground truth and the relevant features are complex and hierarchical (eg. edges, textures, and shapes).
In genomics, it is possible to generate realistic data that are embedded with biologically meaningful patterns, called motifs, while the other positions are completely randomized – the complete ground truth can be established at a “pixel-level”.
Thus, analysis using regulatory genomic data is well-positioned to explore the aforementioned question in a quantitative manner.
Here, we perform a systematic comparison of different training hyperparameters – such as dropout rate, batch size, learning rate, and early stopping – to explore the interplay between a DNN’s prediction accuracy and the efﬁcacy of explanations with attribution methods, speciﬁcally saliency maps.
Strikingly, we ﬁnd that as training progresses, a DNN’s explanation with gradient-based attribution methods can signiﬁcantly be affected without noticeably inﬂuencing its generalization performance.
Although we found that advanced regularization techniques, like manifold mixup, can signiﬁcantly improve interpretability for DNNs that employ traditional hyperparameter choices in genomics, we also found that conventional regularization techniques, when tuned appropriately, can yield signiﬁcantly higher quality model explanations while maintaining a high generalization performance.
Together, our work suggests that prediction performance alone is not a reliable metric to choose which model should be used for downstream interpretability analysis, and standard regularization techniques can be effective at training models that generalize well and are highly interpretable with attribution methods.
Experiment overview
Dataset.
To quantify the efﬁcacy of local attribution maps, we analyzed synthetic data that recapitulates a billboard model of gene regulation from Ref.
Brieﬂy, positive sequences were embedded with 3 to 5 “core motifs” randomly selected with replacement from a pool of 5 known motifs.
Negative sequences were generated in a similar way with the exception that the pool of motifs also includes a large set of “background motifs.”
Background sequences can thus contain core motifs; however, it is statistically unlikely for these sequences to resemble a positive class.
20,000 sequences were randomly split into train (0.7), valid (0.1), and test (0.2) sets.
In this dataset, we have ground truth of which motifs were embedded in each sequence along with their locations.
Baseline model.
We employed a baseline convolutional neural network (CNN) from Ref.
[18], which consists of 5 convolutional layers, followed by a fully-connected hidden layer (see Appendix
A for additional details).
By default, dropout [19] is incorporated after each convolution with a rate of 0.1 and the dense layer with 0.5.
We uniformly trained each model by minimizing the binary cross-entropy loss with mini-batch stochastic gradient descent (128 sequences) for 40 epochs with
Adam updates using default parameters [20].
Each model was trained 5 times with different random initializations.
Variations of the baseline model and training hyperparameters are detailed when outlining speciﬁc experiments.
Quantifying interpretability.
After training, we computed a saliency map [5] for each positivelabel sequence and multiplied it by the inputs, i.e. grad-times-input.
We generated the distribution of saliency scores at positions where ground truth motifs were embedded and the distribution of saliency scores at other positions, as described previously [18].
We quantiﬁed the separation of these two distributions using the area under the receiver operating characteristic curve, which we call the interpretability AUROC.
Manifold mixup improves saliency-based explanations
It is hypothesized that the (un)reliability of gradient-based attribution methods, like saliency maps arises due to a DNN that learns a ‘noisy’ function, which can still yield good generalization performance but renders their gradients less reliable for model interpretability.
Manifold mixup is a regularization strategy that has demonstrated an ability to converge to solutions that exhibit smoother functions [15]; to our knowledge, manifold-mixup has not been previously applied to genomics.
Here, we ask the question - can manifold mixup improve local explanations from saliency maps?
[Table] We trained a baseline CNN model and different scaled versions – where parameters in each convolutional and dense layers were multiplied by a factor of 1, 4 and 8 – with and without manifold mixup (applied to random hidden layers throughout training) and assessed the efﬁcacy of local explanations given by saliency maps (Sec. 2).
Interestingly, we found that manifold mixup improves the classiﬁcation and interpretability AUROCs compared to a baseline CNN for larger model sizes when evaluation is performed at the end of training, i.e. 40 epochs (Table 1).
For a smaller model size, the poorer performance of manifold mixup may be due to a regularization that is too strong.
However, if we evaluate the model interpretability performance at the checkpoint speciﬁed by early stopping (based on maximum classiﬁcation AUROC on the validation set), which occurs around 5-15 epochs, the
CNNs with standard training now yield comparable performance relative to manifold mixup (Table 1).
Both training approaches improve their overall interpretability performance while their classiﬁcation performance is only improved marginally.
This suggests that CNN models with standard training and manifold mixup largely learn similar representations early in training.
Stochastic gradient descent learns functions of increased complexity [21], so it may be that earlier on, the function being learned by both methods is equally smooth.
As training progresses, the stronger regularization provided by manifold mixup maintains a smoother function, while the weaker regularization of a relatively low dropout rate of 0.1 is insufﬁcient, leading to a noisier function that degrades interpretability but largely maintains generalization performance.
The slight decrease observed in manifold mixup’s interpretability AUROC as training progresses suggests that, while it provides a stronger regularizer than the standard training, it is still not enough to maintain optimal performance here.
Better classiﬁcation performance does not imply better model interpretability
To explore how imposing a stronger regularization penalty could improve both generalization and interpretability performance, we performed a systematic exploration of the baseline CNN withscale factors (i.e. 1, 4, and 8) and using different combinations of conventional regularization tech-
[Table]
niques, i [Table] e. batch size and dropout rates after each convolutional layer, while keeping a dropout rate of 0.5 for the ﬁnal hidden dense layer (Appendix B).
Strikingly, when we plot the classiﬁcation AUROC versus the interpretability AUROC at early stopping [Table] we ﬁnd that models with the best classiﬁcation performance do not necessarily yield the best interpretability performance.
In addition [Table] the interpretability performance is more variable when evaluating the models at the end of training (i
[Table]
e. 40 epochs) without early stopping (Fig. 1a). This suggests that we should not base model selection solely on generalization performance when the downstream application is model interpretability.
Importantly, while conventional wisdom suggests that there is a trade-off between these two properties, we ﬁnd that models that yield high generalization performance can also yield the best interpretability performance.
However, it remains unclear whether there exists a measurable quantity that correlates with interpretability performance without requiring any a prior knowledge of ground truth.
Interestingly, we found that a combination of smaller batch sizes ( ≤ 64) and higher dropout (0.4) consistently provide better results across the 3 baseline CNNs for standard training and manifold mixup (Appendix B); typical batch sizes in genomics are relatively big (around 100) and dropout rates after convolutional layers are relatively small (0-0.2).
When regularization is stronger, the generalization and interpretability performance plateaus without a signiﬁcant decrease as training progresses (Fig.
1b & c), an observation that was only observed before for generalization performance [22].
Surprisingly, even a small decrease in classiﬁcation AUROC as training progresses leads to a large decrease in interpretability AUROC (Fig. 1d).
Thus, one way to assess which model to use for downstream model interpretability may be to assess the change in generalization performance beyond early stopping; a plateau should be observed with no signiﬁcant drop-off.
A new training regime for interpretable models for genomics
By sampling different learning rates, we found that training with a learning rate of 0.001 and no dropout for 2 epochs followed by a higher learning rate of 0.01 and dropout of 0.4 for the remaining epochs leads to a signiﬁcant improvement in the interpretability performance (Table 2), beyond what was achievable in our large scale hyperparameter search (Appendix B).
We hypothesize that the stochasticity from a high learning rate enables the navigation towards a region in parameter space that was otherwise inaccessible with lower learning rates (given our limited initialization strategy), within the ﬂat basin [23].
While previous methods have claimed observing better generalization performance within such regions [24], we only observe improved interpretability, and an ablation study shows that each hyperparameter choice is indeed synergistic.
However, the change in classiﬁcation performance is no longer indicative of better interpretability as was observed in Fig.
1d; nevertheless, this approach can still be informative to ﬁlter models from contention for downstream interpretability.
Conclusions
We performed a systematic evaluation of the generalization and interpretability performance for different model architectures and training procedures.
We found models that yield the best generaliza- tion performance do not necessarily lead to the best interpretability performance – though we did not observe a strict trade-off.
We could not ﬁnd a universal metric for model selection that consistently leads to best model interpretability, Interestingly, we observed that standard regularization techniques are often good enough.
Even still, it would be interesting to explore other regularization strategies, including adversarial training and Jacobian regularization [25], and to see how well all of these results generalize to in vivo data, despite the fact that evaluation remains tricky due to a lack of ground truth.  Code
Code for this paper is open source and can be found here: https://github.com/thelabelmaker/Regularization-Genomics-NeurIPS
A Model architecture
Our CNN is based on Ref.
Brieﬂy, it is given according to:
1. input (200 nucleotide sequence) 2.
convolution (24 × f ﬁlters, size 19, ReLU) dropout
(rate = r ) 3.
convolution (32 × f ﬁlters, size 7, ReLU) dropout (rate = r ) max-pooling (size 4) 4.
convolution (48 × f ﬁlters, size 5, ReLU) max-pooling (size 4) dropout
(rate = r ) 5.
convolution (64 × f ﬁlters, size 5, ReLU) max-pooling (size 4) dropout (rate = r ) 6.
fully-connected layer (96 × f units, ReLU) dropout (rate = 0 . 5 ) 7.
fully-connected output layer (1 units, sigmoid)
where f is a factor to change the width of the network, here f = 1, 4, and 8, and r is the dropout rate that was set to 0.1 by default, unless speciﬁed otherwise.
[Figure]