  Shapley values have become one of the most popular feature attribution
explanation methods. However, most prior work has focused on post-hoc Shapley
explanations, which can be computationally demanding due to its exponential
time complexity and preclude model regularization based on Shapley explanations
during training. Thus, we propose to incorporate Shapley values themselves as
latent representations in deep models thereby making Shapley explanations
first-class citizens in the modeling paradigm. This intrinsic explanation
approach enables layer-wise explanations, explanation regularization of the
model during training, and fast explanation computation at test time. We define
the Shapley transform that transforms the input into a Shapley representation
given a specific function. We operationalize the Shapley transform as a neural
network module and construct both shallow and deep networks, called ShapNets,
by composing Shapley modules. We prove that our Shallow ShapNets compute the
exact Shapley values and our Deep ShapNets maintain the missingness and
accuracy properties of Shapley values. We demonstrate on synthetic and
real-world datasets that our ShapNets enable layer-wise Shapley explanations,
novel Shapley regularizations during training, and fast computation while
maintaining reasonable performance. Code is available at
https://github.com/inouye-lab/ShapleyExplanationNetworks.
