  Interpretability in machine learning models is important in high-stakes
decisions, such as whether to order a biopsy based on a mammographic exam.
Mammography poses important challenges that are not present in other computer
vision tasks: datasets are small, confounding information is present, and it
can be difficult even for a radiologist to decide between watchful waiting and
biopsy based on a mammogram alone. In this work, we present a framework for
interpretable machine learning-based mammography. In addition to predicting
whether a lesion is malignant or benign, our work aims to follow the reasoning
processes of radiologists in detecting clinically relevant semantic features of
each image, such as the characteristics of the mass margins. The framework
includes a novel interpretable neural network algorithm that uses case-based
reasoning for mammography. Our algorithm can incorporate a combination of data
with whole image labelling and data with pixel-wise annotations, leading to
better accuracy and interpretability even with a small number of images. Our
interpretable models are able to highlight the classification-relevant parts of
the image, whereas other methods highlight healthy tissue and confounding
information. Our models are decision aids, rather than decision makers, aimed
at better overall human-machine collaboration. We do not observe a loss in mass
margin classification accuracy over a black box neural network trained on the
same data.
