  Normalizing flows define a probability distribution by an explicit invertible
transformation $\boldsymbol{\mathbf{z}}=f(\boldsymbol{\mathbf{x}})$. In this
work, we present implicit normalizing flows (ImpFlows), which generalize
normalizing flows by allowing the mapping to be implicitly defined by the roots
of an equation $F(\boldsymbol{\mathbf{z}}, \boldsymbol{\mathbf{x}})=
\boldsymbol{\mathbf{0}}$. ImpFlows build on residual flows (ResFlows) with a
proper balance between expressiveness and tractability. Through theoretical
analysis, we show that the function space of ImpFlow is strictly richer than
that of ResFlows. Furthermore, for any ResFlow with a fixed number of blocks,
there exists some function that ResFlow has a non-negligible approximation
error. However, the function is exactly representable by a single-block
ImpFlow. We propose a scalable algorithm to train and draw samples from
ImpFlows. Empirically, we evaluate ImpFlow on several classification and
density modeling tasks, and ImpFlow outperforms ResFlow with a comparable
amount of parameters on all the benchmarks.
