This paper presents an algorithm for the estimation of the motion of textured objects undergoing non-rigid deformations over a sequence of images. An active mesh model, which is a finite element deformable membrane, is introduced in order to achieve efficient representation of global and local deformations. The mesh is constructed using an adaptive triangulation procedure that places more triangles over high detail areas. Through robust least squares techniques and modal analysis, efficient estimation of global object deformations is achieved, based on a set of sparse displacement measurements. A local warping procedure is then applied to minimize the intensity matching error between subsequent images, and thus estimate local deformations. Among the major contributions of this paper are novel techniques developed to acquire knowledge of the object dynamics and structure directly from the image sequence even in the absence of prior intelligence regarding the scene. Specifically, a coarse-to-fine estimation scheme is first developed, which adapts the model to locally deforming features. Subsequently, principal components modal analysis is used to accumulate knowledge of the object dynamics. This knowledge is finally exploited to constrain the object deformation. The problem of tracking the model over time is addressed, and a novel motion compensated prediction approach is proposed to facilitate this. A novel method for the determination of the dynamical principal axes of deformation is developed. The experimental results demonstrate the efficiency and robustness of the proposed scheme, which has many potential applications in the areas of image coding, image analysis, and computer graphics. In this paper we consider the problem of tracking a non-rigid object using a sequence of images. The need for non-rigid motion estimation arises in many applications of current interest in the fields of image understanding, image coding and computer graphics. Examples of such applications include: very low bit rate coding where a robust, artifact-free motion modeling system is required; deformation analysis of tissue images to aid medical diagnosis; and natural looking graphics animations based on image warping. The estimation of non-rigid or partially rigid motion is the subject of increasing attention in recent literature. In [1, 2] objects were modeled as partially rigid, while in [3, 4] object motion was portrayed as a perturbation of rigid motion. In order to accurately describe non-rigid deformations, physics-based models have also been proposed [5, 6] . These simulate the shape and dynamic behavior of physical objects. In [7] the physics-based paradigm was developed with active contours, simulating an elastic material deformed by forces corresponding to image intensity gradients. These results were extended to the 3D case by using finite-element models for tracking 3D feature point correspondences in [8, 9] . To obtain an analytic description of the shape and deformation of the objects, the finite-element approach was used in [9, 10, 11] . Finite element methods were shown to deal better with the sampling problem encountered with use of finite-difference methods. All above algorithms are used to detect and track image features, such as intensity edge boundaries, but they are unable to provide a so called canonical representation [12] , i.e. an analytic description of the deformation at each object point. Another shortcoming of physics-based modeling is that the models produced are coarse and prior assumptions about the model parameters are needed. A canonical representation was obtained in [11] where non-rigid motion modeling is performed via finite-elements analysis, by extracting the "deformation modes" of the object. These constitute a frequency-ordered orthonormal basis of the object's deformation space. Although the modal decomposition technique used decouples the degrees of freedom of the object's deformation, yielding advantages in fitting the model to data, the actual number of degrees of freedom is not known a priori. As a solution to this problem, [11] proposed to discard the high frequency modes. This results however in loss of local motion attributes. In addition, the deformation modes, corresponding to the generalized axes of symmetry of the shape, provide only a sub-optimal basis, in that they do not minimize the error of representing the actual deformations as a linear combination of elements of a minimal basis. This error is minimized when the "principal axes of deformation", described in section V of the present paper, are used. In [13] , an active contour is used to track the object boundary. Motion vectors in the interior of the object are interpolated from the boundary deformation vectors using a 2D triangular mesh. In this paper we use an active mesh to estimate the non-rigid motion of textured 2D objects. A 2D triangular mesh is placed over the object. Subsequent images are registered by estimating nodal deformations based on the minimization of an energy function. This function includes local intensity matching, under local as well as global smoothness constraints, in a coarse-to-fine approach. The proposed algorithm uses methodology similar to the warping-based motion estimation techniques introduced in [14, 15] and [16, 17, 18] . In [14, 15] a uniform mesh is placed over the image, and motion is estimated by locally warping the nodes of the mesh. The algorithm in [16] consists of a node point selection procedure, followed by triangulation using the selected node points. The mesh is refined in motion failure areas by the 2 introduction of new nodes and re-triangulation. The algorithm is computationally intensive and
In the area of predictive tracking, the work by Malassiotis REF resembles the structure of our approach (even though the underlying model is different): a triangular mesh is calculated to cover the 2-D image of a textured deformable object of interest.
Tracking textured deformable objects using a finite-element mesh
IEEE Trans. Circuits Syst. Video Techn.
IEEE Trans. Circuits Syst. Video Techn.
Computer Science