  The cost volume, capturing the similarity of possible correspondences across
two input images, is a key ingredient in state-of-the-art optical flow
approaches. When sampling for correspondences to build the cost volume, a large
neighborhood radius is required to deal with large displacements, introducing a
significant computational burden. To address this, a sequential strategy is
usually adopted, where correspondence sampling in a local neighborhood with a
small radius suffices. However, such sequential approaches, instantiated by
either a pyramid structure over a deep neural network's feature hierarchy or by
a recurrent neural network, are slow due to the inherent need for sequential
processing of cost volumes. In this paper, we propose dilated cost volumes to
capture small and large displacements simultaneously, allowing optical flow
estimation without the need for the sequential estimation strategy. To process
the cost volume to get pixel-wise optical flow, existing approaches employ 2D
or separable 4D convolutions, which we show either suffer from high GPU memory
consumption, inferior accuracy, or large model size. Therefore, we propose
using 3D convolutions for cost volume filtering to address these issues. By
combining the dilated cost volumes and 3D convolutions, our proposed model
DCVNet not only exhibits real-time inference (71 fps on a mid-end 1080ti GPU)
but is also compact and obtains comparable accuracy to existing approaches.
