  In comparison to classical shallow representation learning techniques, deep
neural networks have achieved superior performance in nearly every application
benchmark. But despite their clear empirical advantages, it is still not well
understood what makes them so effective. To approach this question, we
introduce deep frame approximation, a unifying framework for representation
learning with structured overcomplete frames. While exact inference requires
iterative optimization, it may be approximated by the operations of a
feed-forward deep neural network. We then indirectly analyze how model capacity
relates to the frame structure induced by architectural hyperparameters such as
depth, width, and skip connections. We quantify these structural differences
with the deep frame potential, a data-independent measure of coherence linked
to representation uniqueness and stability. As a criterion for model selection,
we show correlation with generalization error on a variety of common deep
network architectures such as ResNets and DenseNets. We also demonstrate how
recurrent networks implementing iterative optimization algorithms achieve
performance comparable to their feed-forward approximations. This connection to
the established theory of overcomplete representations suggests promising new
directions for principled deep network architecture design with less reliance
on ad-hoc engineering.
