  The training of deep residual neural networks (ResNets) with backpropagation
has a memory cost that increases linearly with respect to the depth of the
network. A simple way to circumvent this issue is to use reversible
architectures. In this paper, we propose to change the forward rule of a ResNet
by adding a momentum term. The resulting networks, momentum residual neural
networks (MomentumNets), are invertible. Unlike previous invertible
architectures, they can be used as a drop-in replacement for any existing
ResNet block. We show that MomentumNets can be interpreted in the infinitesimal
step size regime as second-order ordinary differential equations (ODEs) and
exactly characterize how adding momentum progressively increases the
representation capabilities of MomentumNets. Our analysis reveals that
MomentumNets can learn any linear mapping up to a multiplicative factor, while
ResNets cannot. In a learning to optimize setting, where convergence to a fixed
point is required, we show theoretically and empirically that our method
succeeds while existing invertible architectures fail. We show on CIFAR and
ImageNet that MomentumNets have the same accuracy as ResNets, while having a
much smaller memory footprint, and show that pre-trained MomentumNets are
promising for fine-tuning models.
