  Learning semantically meaningful features is important for Deep Neural
Networks to win end-user trust. Attempts to generate post-hoc explanations fall
short in gaining user confidence as they do not improve the interpretability of
feature representations learned by the models. In this work, we propose
Semantic Convolutional Neural Network (SemCNN) that has an additional Concept
layer to learn the associations between visual features and word phrases.
SemCNN employs an objective function that optimizes for both the prediction
accuracy as well as the semantic meaningfulness of the learned feature
representations. Further, SemCNN makes its decisions as a weighted sum of the
contributions of these features leading to fully interpretable decisions.
Experiment results on multiple benchmark datasets demonstrate that SemCNN can
learn features with clear semantic meaning and their corresponding
contributions to the model decision without compromising prediction accuracy.
Furthermore, these learned concepts are transferrable and can be applied to new
classes of objects that have similar concepts.
