On the Neural Tangent Kernel of Equilibrium Models
Authors: Zhili Feng, J Zico Kolter
Keywords: deel learning, equilibrium model, neural tangent kernel
Abstract: Existing analyses of the neural tangent kernel (NTK) for infinite-depth networks show that the kernel typically becomes degenerate as the number of layers grows.  This raises the question of how to apply such methods to practical "infinite depth" architectures such as the recently-proposed deep equilibrium (DEQ) model, which directly computes the infinite-depth limit of a weight-tied network via root-finding.  In this work, we show that because of the input injection component of these networks, DEQ models have non-degenerate NTKs even in the infinite depth limit.  Furthermore, we show that these kernels themselves can be computed by an analogous root-finding problem as in traditional DEQs, and highlight methods for computing the NTK for both fully-connected and convolutional variants.  We evaluate these models empirically, showing they match or improve upon the performance of existing regularized NTK methods.