  Providing explanations for deep neural networks (DNNs) is essential for their
use in domains wherein the interpretability of decisions is a critical
prerequisite. Despite the plethora of work on interpreting DNNs, most existing
solutions offer interpretability in an ad hoc, one-shot, and static manner,
without accounting for the perception, understanding, or response of end-users,
resulting in their poor usability in practice. In this paper, we argue that DNN
interpretability should be implemented as the interactions between users and
models. We present i-Algebra, a first-of-its-kind interactive framework for
interpreting DNNs. At its core is a library of atomic, composable operators,
which explain model behaviors at varying input granularity, during different
inference stages, and from distinct interpretation perspectives. Leveraging a
declarative query language, users are enabled to build various analysis tools
(e.g., "drill-down", "comparative", "what-if" analysis) via flexibly composing
such operators. We prototype i-Algebra and conduct user studies in a set of
representative analysis tasks, including inspecting adversarial inputs,
resolving model inconsistency, and cleansing contaminated data, all
demonstrating its promising usability.
