The estimation of the optical flow between two images is one of the key problems in low-level vision. According the optical flow evaluation site at http://vision.middlebury.edu/flow/, discontinuity preserving variational models based on Total Variation (TV) regularization and L 1 data terms are among the most accurate flow estimation techniques, but there is still room for improvements. This paper has two key contributions: 1. The isotropic Total Variation (TV) regularity is replaced with a penalty function initially proposed by Huber [2] in the field of robust statistics not favoring piecewise constant solutions. In addition we incorporate directional information yielding an anisotropic Huber regularity. 2. We propose a novel 3-frame spatio-temporal regularization that 'mirrors' the flow symmetric w.r.t. the central frame. TV regularization is an L 1 penalization of the flow gradient magnitudes, and due to the tendency of the L 1 norm to favor sparse solutions (i.e. lots of 'zeros'), the fill-in effect caused by the regularizer leads to piecewise constant solutions in weakly textured areas. This effect, known as 'staircasing' in a 1D setting, can be reduced significantly by using a quadratic penalization for small gradient magnitudes while sticking to linear penalization for larger magnitudes to maintain the discontinuity preserving properties known from TV. A comparison of isotropic TV and isotropic Huber regularity is shown in Fig. 1 by means of rendering the disparities u 1 of the Dimetrodon dataset. The color coded flow (cf. Fig. 1(a) ) is superimposed as texture. Based on the two observations that motion discontinuities often occur along object boundaries and that in turn object boundaries often coincide with large image gradients, Nagel and Enkelmann [3] proposed to adapt the regularization to the local image structure. Even for the quadratic regularizers used at that time, their anisotropic (image-driven) regularization decreased the well-known oversmoothing effects of the Horn & Schunck model [1] by impeding smoothing across image edges. Therefore, we propose to replace the isotropic TV regularization with an anisotropic (image-driven) Huber regularization term (cf. Fig. 2 ) which leads to the energy optimization problem The second contribution of this paper is motivated by an application: the restoration of historic video material via flow-based video interpolation. To cope with the gross outliers contained in the historic video material (cf . Fig. 3 ), we propose a novel spatio-temporal regularization approach and compare it against the well-known spatio-temporal regularization proposed in [4] . Instead of assuming gradual flow changes through time, we propose a 3-frame method that 'mirrors' the flow symmetric w.r.t. the central frame by extending the anisotropic flow (1) with one additional data fidelity term, one denoting the linearized brightness constancy constraints between the first and the central frame, and the other between the third and the central frame. This symmetry constraint outperforms methods using purely spatial regularization on sequences where single frames are degreded, e.g. with blobs and scratches (cf . Fig. 3 ).
For optical flow changing the weights using the image gradients is also proposed REF .
Anisotropic Huber-L1 Optical Flow
BMVC
Computer Science