: Overview of our approach to articulated 3D human pose estimation. Red boxes specify the selected component. Pictorial structure models are the de facto standard for 2D human pose estimation. Numerous refinements and improvements have been proposed such as discriminatively trained body part detectors, flexible body models and local and global mixtures. While these techniques allow to achieve the state-of-the-art performance for 2D pose estimation, they have not yet been extended to enable pose estimation in 3D, instead this problem is traditionally addressed using 3D body models and involves complex inference in a high-dimensional space of 3D body configurations. We formulate the articulated 3D human pose estimation problem as a joint inference over the set of 2D projections of the pose in each of the camera views. As a first contribution of this paper, we propose a 2D pose estimation approach that extends the state-of-the-art 2D pictorial structures model [6] with flexible parts, color features, multi-modal pairwise terms, and mixtures of pictorial structures. The second and main contribution is to extend this 2D pose estimation model to a multi-view model that performs joint reasoning over people poses seen from multiple viewpoints. The output of this novel model is then used to recover 3D pose. We evaluate our multi-view pictorial structures model on HumanEva-I [8] and MPII Cooking [7] dataset. In comparison to related work for 3D pose estimation our approach achieves similar or better results while operating on single-frames only and not relying on activity specific motion models or tracking. Notably, our approach outperforms state-of-the-art for activities with more complex motions. Single-view model: The pictorial structures model, originally introduced in [2, 3] , represents the human body as a configuration L = {l 1 , . . . , l N } of N rigid parts and a set of pairwise part relationships E. The image position and absolute orientation of each part is given by l i = (x i , y i , θ i ). We formulate the model as a conditional random field, and assume that the probability of the part configuration L given the image evidence I factorizes into a product of unary and pairwise terms: The part likelihood terms f n (l n ; I) are represented with boosted part detectors that rely on the encoding of the image using a densely computed grid of shape context descriptors [1] . We concatenate these shape context features with color features and learn a boosted part detector on top of this combined representation. Note that augumenting shape information with the color allows us to automatically learn the relative importance of both features at the part detection stage. The pairwise terms f i j (l i , l j ) which encode the spatial constraints between parts are traditionally modeled with Gaussian distribution in the transformed space of the joint between two parts. We extend our model by introducing mixture models at the level of these pairwise part dependencies. To that end we replace the unimodal Gaussian with the term that maximizes over multiple modes and represent each mode with a Gaussian. Following [4, 5] we extend our approach to a mixture of pictorial structures models. We obtain the mixture components by clustering the training data with k-means and learning a separate model for each cluster. The components typically correspond to major modes in the data, such as various viewpoints of the person with respect to the camera. The index of the component is treated as a latent variable to be inferred at test time. We select the best component with the minimal uncertainty in the marginal posterior distributions of the body parts. In our experiments this approach worked slightly better compared to a trained holistic classifier that distinguishes the mixture component based on the contents of the person bounding box. Multi-view model: To exploit the multi-view information we augument the model with appearance and spatial correspondence constraints across views. In order to estimate the 3D pose we proceed in two steps. In the first step we jointly infer the 2D projections of the 3D body joints across views exploiting multi-view constraints. In the second step, we recover the 3D pose by triangulation of the estimated 2D projections. For simplicity, we describe our multi-view model for the case of two views. For view m, let us denote the 2D body configuration as L m and image evidence as I m . According to Eq. 1 the single-view factors f (L 1 ; I 1 ) and f (L 2 ; I 2 ) representing the conditional posterior over body configurations decomposes into a product of unary and pairwise terms that define appearance and spatial constraints between parts independently for each view. The joint posterior over configurations in both views is given by p(L 1 , L 2 |I 1 ,
Amin et al. REF propose to estimate 2D poses by exploring the geometric relation between multi-view images.
Multi-view Pictorial Structures for 3D Human Pose Estimation
BMVC
Computer Science