  Ongoing efforts to understand deep neural networks (DNN) have provided many
insights, but DNNs remain incompletely understood. Improving DNN's
interpretability has practical benefits, such as more accountable usage, better
algorithm maintenance and improvement. The complexity of dataset structure may
contribute to the difficulty in solving interpretability problem arising from
DNN's black-box mechanism. Thus, we propose to use pattern theory formulated by
Ulf Grenander, in which data can be described as configurations of fundamental
objects that allow us to investigate convolutional neural network's (CNN)
interpretability in a component-wise manner. Specifically, U-Net-like structure
is formed by attaching expansion blocks (EB) to ResNet, allowing it to perform
semantic segmentation-like tasks at its EB output channels designed to be
compatible with pattern theory's configurations. Through these modules, some
heatmap-based explainable artificial intelligence (XAI) methods will be shown
to extract explanations w.r.t individual generators that make up a single data
sample, potentially reducing the impact of dataset's complexity to
interpretability problem. The MNIST-equivalent dataset containing pattern
theory's elements is designed to facilitate smoother entry into this framework,
along which the theory's generative aspect is naturally presented.
