  Deep Neural Networks (DNNs) are rapidly gaining popularity in a variety of
important domains. Formally, DNNs are complicated vector-valued functions which
come in a variety of sizes and applications. Unfortunately, modern DNNs have
been shown to be vulnerable to a variety of attacks and buggy behavior. This
has motivated recent work in formally analyzing the properties of such DNNs.
This paper introduces SyReNN, a tool for understanding and analyzing a DNN by
computing its symbolic representation. The key insight is to decompose the DNN
into linear functions. Our tool is designed for analyses using low-dimensional
subsets of the input space, a unique design point in the space of DNN analysis
tools. We describe the tool and the underlying theory, then evaluate its use
and performance on three case studies: computing Integrated Gradients,
visualizing a DNN's decision boundaries, and patching a DNN.
