Mixtures of Gaussians, factor analyzers (probabilistic PCA) and hidden Markov models are staples of static and dynamic data modeling and image and video modeling in particular. We show how topographic trans formations in the input, such as translation and shearing in images, can be accounted for in these models by including a discrete transformation variable. The resulting mod els perform clustering, dimensionality reduc tion and time-series analysis in a way that is invariant to transformations in the input. Us ing the EM algorithm, these transformation invariant models can be fit to static data and time series. We give results on filtering mi croscopy images, face and facial pose cluster ing, handwritten digit modeling and recog nition, video clustering, object tracking, and removal of distractions from video sequences. 1 Graphical models and their dynamic variants are grow ing in popularity as a way to build appearance based models for computer vision.1 However, many appearance-based models-such as mixtures of Gaus sians, factor analyzers, mixtures of factor analyzers and hidden Markov models with Gaussian outputsare extremely sensitive to spatial or topographic trans formations of the input images. These transformations may include translation, rotation, shearing and warp ing. For example, if a mixture of Gaussians is fit to a set of images that include random translations, the centers will represent different transformations of es sentially the same data. Ta king face images as an ex ample, it would be more useful for the different clusters Imagine what happens to the point in the N dimensional space corresponding to anN-pixel image of an object, while the object is deformed by shear ing. A very small amount of shearing will move the point only slightly, so deforming the object by shear ing will trace a continuous curve in the space of pixel intensities. As illustrated in Fig. 1a , extensive levels of shearing will produce a highly nonlinear curve (con sider shearing a thin vertical line), although the curve can be approximated by a straight line locally. Linear approximations of the transformation manifold have been used to significantly improve the perfor mance of feedforward discriminative classifiers such as nearest neighbors and multilayer perceptrons (Simard et al., 1993) . Linear generative models (factor analyz ers, mixtures of factor analyzers) have also been modi fied using linear approximations of the transformation manifold to build in some degree of transformation in variance (Hinton et al., 1997) . In general, the linear approximation is accurate for transformations that couple neighboring pixels, but is inaccurate for transformations that couple nonneigh boring pixels. In some applications (e.g., handwritten digit recognition), the input can be blurred so that the linear approximation becomes more robust. For significant levels of transformation, the nonlinear manifold can be better modeled using a discrete ap proximation. For example, the curve in Fig. la can be represented by a set of points (filled discs). In this approach, a discrete set of possible transformations is specified beforehand and parameters are learned so that the model is invariant to the set of transforma tions. This approach has been used to design "convo lutional neural networks" that are invariant to trans lation (Le Cun et al., 1998) . We describe how invariance to a discrete set of known transformations (like translation) can be built into a
In a different context, Frey and Jojic REF show how to account for view transformations in a dynamic probabilistic model.
Learning Graphical Models of Images, Videos and Their Spatial Transformations
ArXiv
ArXiv
Computer Science Mathematics