  The un-rectifying technique expresses a non-linear point-wise activation
function as a data-dependent variable, which means that the activation variable
along with its input and output can all be employed in optimization. The ReLU
network in this study was un-rectified means that the activation functions
could be replaced with data-dependent activation variables in the form of
equations and constraints. The discrete nature of activation variables
associated with un-rectifying ReLUs allows the reformulation of deep learning
problems as problems of combinatorial optimization. However, we demonstrate
that the optimal solution to a combinatorial optimization problem can be
preserved by relaxing the discrete domains of activation variables to closed
intervals. This makes it easier to learn a network using methods developed for
real-domain constrained optimization. We also demonstrate that by introducing
data-dependent slack variables as constraints, it is possible to optimize a
network based on the augmented Lagrangian approach. This means that our method
could theoretically achieve global convergence and all limit points are
critical points of the learning problem. In experiments, our novel approach to
solving the compressed sensing recovery problem achieved state-of-the-art
performance when applied to the MNIST database and natural images.
