  Deep learning is a powerful tool whose applications in quantitative finance
are growing every day. Yet, artificial neural networks behave as black boxes
and this hinders validation and accountability processes. Being able to
interpret the inner functioning and the input-output relationship of these
networks has become key for the acceptance of such tools. In this paper we
focus on the calibration process of a stochastic volatility model, a subject
recently tackled by deep learning algorithms. We analyze the Heston model in
particular, as this model's properties are well known, resulting in an ideal
benchmark case. We investigate the capability of local strategies and global
strategies coming from cooperative game theory to explain the trained neural
networks, and we find that global strategies such as Shapley values can be
effectively used in practice. Our analysis also highlights that Shapley values
may help choose the network architecture, as we find that fully-connected
neural networks perform better than convolutional neural networks in predicting
and interpreting the Heston model prices to parameters relationship.
