  When developing AI systems that interact with humans, it is essential to
design both a system that can understand humans, and a system that humans can
understand. Most deep network based agent-modeling approaches are 1) not
interpretable and 2) only model external behavior, ignoring internal mental
states, which potentially limits their capability for assistance,
interventions, discovering false beliefs, etc. To this end, we develop an
interpretable modular neural framework for modeling the intentions of other
observed entities. We demonstrate the efficacy of our approach with experiments
on data from human participants on a search and rescue task in Minecraft, and
show that incorporating interpretability can significantly increase predictive
performance under the right conditions.
