  Machine Learning (ML) is a fundamental part of modern perception systems. In
the last decade, the performance of computer vision using trained deep neural
networks has outperformed previous approaches based on careful feature
engineering. However, the opaqueness of large ML models is a substantial
impediment for critical applications such as in the automotive context. As a
remedy, Gradient-weighted Class Activation Mapping (Grad-CAM) has been proposed
to provide visual explanations of model internals. In this paper, we
demonstrate how Grad-CAM heatmaps can be used to increase the explainability of
an image recognition model trained for a pedestrian underpass. We argue how the
heatmaps support compliance to the EU's seven key requirements for Trustworthy
AI. Finally, we propose adding automated heatmap analysis as a pipe segment in
an MLOps pipeline. We believe that such a building block can be used to
automatically detect if a trained ML-model is activated based on invalid pixels
in test images, suggesting biased models.
