pwc-archive/papers-with-abstracts huggingface dataset:
Regularizing Long Short Term Memory With 3D Human-Skeleton Sequences for Action Recognition
Authors: Behrooz Mahasseni, Sinisa Todorovic
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Recognition, Temporal Action Localization

This paper argues that large-scale action recognition in video can be greatly improved by providing an additional modality in training data -- namely, 3D human-skeleton sequences -- aimed at complementing poorly represented or missing features of human actions in the training videos. For recognition, we use Long Short Term Memory (LSTM) grounded via a deep Convolutional Neural Network (CNN) onto the video. Training of LSTM is regularized using the output of another encoder LSTM (eLSTM) grounded on 3D human-skeleton training data. For such regularized training of LSTM, we modify the standard backpropagation through time (BPTT) in order to address the well-known issues with gradient descent in constraint optimization. Our evaluation on three benchmark datasets -- Sports-1M, HMDB-51, and UCF101 -- shows accuracy improvements from 5.3% up to 17.4% relative to the state of the art.
End-To-End Learning of Deformable Mixture of Parts and Deep Convolutional Neural Networks for Human Pose Estimation
Authors: Wei Yang, Wanli Ouyang, Hongsheng Li, Xiaogang Wang
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Pose Estimation

Recently, Deep Convolutional Neural Networks (DCNNs) have been applied to the task of human pose estimation, and have shown its potential of learning better feature representations and capturing contextual relationships. However, it is difficult to incorporate domain prior knowledge such as geometric relationships among body parts into DCNNs. In addition, training DCNN-based body part detectors without consideration of global body joint consistency introduces ambiguities, which increases the complexity of training. In this paper, we propose a novel end-to-end framework for human pose estimation that combines DCNNs with the expressive deformable mixture of parts. We explicitly incorporate domain prior knowledge into the framework, which greatly regularizes the learning process and enables the flexibility of our framework for  loopy models or tree-structured models. The effectiveness of jointly learning a DCNN with a deformable mixture of parts model is evaluated through intensive experiments on several widely used benchmarks. The proposed approach significantly improves the performance compared with state-of-the-art approaches, especially on benchmarks with challenging articulations.
Temporal Action Localization With Pyramid of Score Distribution Features
Authors: Jun Yuan, Bingbing Ni, Xiaokang Yang, Ashraf A. Kassim
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Localization, Temporal Action Localization

We investigate the feature design and classification architectures in temporal action localization. This application focuses on detecting and labeling actions in untrimmed videos, which brings more challenge than classifying pre-segmented videos. The major difficulty for action localization is the uncertainty of action occurrence and utilization of information from different scales. Two innovations are proposed to address this issue. First, we propose a Pyramid of Score Distribution Feature (PSDF) to capture the motion information at multiple resolutions centered at each detection window. This novel feature mitigates the influence of unknown action position and duration, and shows significant performance gain over previous detection approaches. Second, inter-frame consistency is further explored by incorporating PSDF into the state-of-the-art Recurrent Neural Networks, which gives additional performance gain in detecting actions in temporally untrimmed videos. We tested our action localization framework on the THUMOS'15 and MPII Cooking Activities Dataset, both of which show a large performance improvement over previous attempts.
Harnessing Object and Scene Semantics for Large-Scale Video Understanding
Authors: Zuxuan Wu, Yanwei Fu, Yu-Gang Jiang, Leonid Sigal
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Recognition, Clustering, Object, Temporal Action Localization, Video Classification, Video Understanding

Large-scale action recognition and video categorization are important problems in computer vision. To address these problems, we propose a novel object- and scene-based semantic fusion network and representation. Our semantic fusion network combines three streams of information using a three-layer neural network: (i) frame-based low-level CNN features, (ii) object features from a state-of-the-art large-scale CNN object-detector trained to recognize 20K classes, and (iii) scene features from a state-of-the-art CNN scene-detector trained to recognize 205 scenes. The trained network achieves improvements in supervised activity and video categorization in two complex large-scale datasets -  ActivityNet and FCVID, respectively. Further, by examining and back propagating information through the fusion network, semantic relationships (correlations) between video classes and objects/scenes can be discovered. These video class-object/video class-scene relationships can in turn be used as semantic representation for the video classes themselves. We illustrate effectiveness of this semantic representation through experiments on zero-shot action/video classification and clustering.
Video-Story Composition via Plot Analysis
Authors: Jinsoo Choi, Tae-Hyun Oh, In So Kweon
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Optical Flow Estimation, Patch Matching

We address the problem of composing a story out of multiple short video clips taken by a person during an activity or experience. Inspired by plot analysis of written stories, our method generates a sequence of video clips ordered in such a way that it reflects plot dynamics and content coherency. That is, given a set of multiple video clips, our method composes a video which we call a video-story. We define metrics on scene dynamics and coherency by dense optical flow features and a patch matching algorithm. Using these metrics, we define an objective function for the video-story. To efficiently search for the best video-story, we introduce a novel Branch-and-Bound algorithm which guarantees the global optimum. We collect the dataset consisting of 23 video sets from the web, resulting in a total of 236 individual video clips. With the acquired dataset, we perform extensive user studies involving 30 human subjects by which the effectiveness of our approach is quantitatively and qualitatively verified.
Temporal Action Detection Using a Statistical Language Model
Authors: Alexander Richard, Juergen Gall
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Detection, Action Recognition, Language Modeling, Language Modelling, model, Temporal Action Localization

While current approaches to action recognition on pre-segmented video clips already achieve high accuracies, temporal action detection is still far from comparably good results. Automatically locating and classifying the relevant action segments in videos of varying lengths proves to be a challenging task. We propose a novel method for temporal action detection including statistical length and language modeling to represent temporal and contextual structure. Our approach aims at globally optimizing the joint probability of three components, a length and language model and a discriminative action model, without making intermediate decisions. The problem of finding the most likely action sequence and the corresponding segment boundaries in an exponentially large search space is addressed by dynamic programming.  We provide an extensive evaluation of each model component on Thumos 14, a large action detection dataset, and report state-of-the-art results on three datasets.
Multi-Scale Patch Aggregation (MPA) for Simultaneous Detection and Segmentation
Authors: Shu Liu, Xiaojuan Qi, Jianping Shi, Hong Zhang, Jiaya Jia
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Object, Object Proposal Generation, Segmentation

Aiming at simultaneous detection and segmentation (SDS), we propose a proposal-free framework, which detect and segment object instances via mid-level patches. We design a unified trainable network on patches, which is followed by a fast and effective patch aggregation algorithm to infer object instances. Our method benefits from end-to-end training. Without object proposal generation, computation time can also be reduced. In experiments, our method yields results 62.1% and 61.8% in terms of mAPr on VOC2012 segmentation val and VOC2012 SDS val, which are state-of-the-art at the time of submission. We also report results on Microsoft COCO test-std/test-dev dataset in this paper.
Feature Space Optimization for Semantic Video Segmentation
Authors: Abhijit Kundu, Vibhav Vineet, Vladlen Koltun
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Segmentation, Structured Prediction, Video Segmentation, Video Semantic Segmentation

We present an approach to long-range spatio-temporal regularization in semantic video segmentation. Temporal regularization in video is challenging because both the camera and the scene may be in motion. Thus Euclidean distance in the space-time volume is not a good proxy for correspondence. We optimize the mapping of pixels to a Euclidean feature space so as to minimize distances between corresponding points. Structured prediction is performed by a dense CRF that operates on the optimized features. Experimental results demonstrate that the presented approach increases the accuracy and temporal consistency of semantic video segmentation.
Large-Scale Semantic 3D Reconstruction: An Adaptive Multi-Resolution Model for Multi-Class Volumetric Labeling
Authors: Maros Blaha, Christoph Vogel, Audrey Richard, Jan D. Wegner, Thomas Pock, Konrad Schindler
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: 3D Reconstruction

We propose an adaptive multi-resolution formulation of semantic 3D reconstruction. Given a set of images of a scene, semantic 3D reconstruction aims to densely reconstruct both the 3D shape of the scene and a segmentation into semantic object classes. Jointly reasoning about shape and class allows one to take into account class-specific shape priors (e.g., building walls should be smooth and vertical, and vice versa smooth, vertical surfaces are likely to be building walls), leading to improved reconstruction results. So far, semantic 3D reconstruction methods have been limited to small scenes and low resolution, because of their large memory footprint and computational cost. To scale them up to large scenes, we propose a hierarchical scheme which refines the reconstruction only in regions that are likely to contain a surface, exploiting the fact that both high spatial resolution and high numerical precision are only required in those regions. Our scheme amounts to solving a sequence of convex optimizations while progressively removing constraints, in such a way that the energy, in each iteration, is the tightest possible approximation of the underlying energy at full resolution. In our experiments the method saves up to 98% memory and 95% computation time, without any loss of accuracy.
Gaussian Conditional Random Field Network for Semantic Segmentation
Authors: Raviteja Vemulapalli, Oncel Tuzel, Ming-Yu Liu, Rama Chellapa
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Segmentation, Semantic Segmentation

In contrast to the existing approaches that use discrete Conditional Random Field (CRF) models, we propose to use a Gaussian CRF model for the task of semantic segmentation. We propose a novel deep network, which we refer to as Gaussian Mean Field (GMF) network, whose layers perform mean field inference over a Gaussian CRF. The proposed GMF network has the desired property that each of its layers produces an output that is closer to the maximum a posteriori solution of the Gaussian CRF compared to its input. By combining the proposed GMF network with deep Convolutional Neural Networks (CNNs), we propose a new end-to-end trainable Gaussian conditional random field network. The proposed Gaussian CRF network is composed of three sub-networks: (i) a CNN-based unary network for generating unary potentials, (ii) a CNN-based pairwise network for generating pairwise potentials, and (iii) a GMF network for performing Gaussian CRF inference. When trained end-to-end in a discriminative fashion, and evaluated on the challenging PASCALVOC 2012 segmentation dataset, the proposed Gaussian CRF network outperforms various recent semantic segmentation approaches that combine CNNs with discrete CRF models.
The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
Authors: German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, Antonio M. Lopez
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Autonomous Driving, Segmentation, Semantic Segmentation

Vision-based semantic segmentation in urban scenarios is a key functionality for autonomous driving. Recent revolutionary results of deep convolutional neural networks (DCNNs) foreshadow the advent of reliable classifiers to perform such visual tasks. However, DCNNs require learning of many parameters from raw images; thus, having a sufficient amount of diverse images with class annotations is needed. These annotations are obtained via cumbersome, human labour which is particularly challenging for semantic segmentation since pixel-level annotations are required. In this paper, we propose to use a virtual world to automatically generate realistic synthetic images with pixel-level annotations. Then, we address the question of how useful such data can be for  semantic segmentation -- in particular, when using a DCNN paradigm. In order to answer this question we have generated a synthetic collection of diverse urban images, named SYNTHIA, with automatically generated class annotations. We use SYNTHIA in combination with publicly available real-world urban images with manually provided annotations. Then, we conduct experiments with DCNNs that show how the inclusion of SYNTHIA in the training stage significantly improves performance on the semantic segmentation task.
Progressive Prioritized Multi-View Stereo
Authors: Alex Locher, Michal Perdoch, Luc van Gool
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: CPU

This work proposes a progressive patch based multi-view stereo algorithm able to deliver a dense point cloud at any time.  This enables an immediate feedback on the reconstruction process in a user centric scenario.  With increasing processing time, the model is improved in terms of resolution and accuracy.  The algorithm explicitly handles input images with varying effective scale and creates visually pleasing point clouds.  A priority scheme assures that the limited computational power is invested in scene parts, where the user is most interested in or the overall error can be reduced the most.  The architecture of the proposed pipeline allows fast processing times in large scenes using a pure open-source CPU implementation.  We show the performance of our algorithm on challenging standard datasets as well as on real-world scenes and compare it to the baseline.
What Sparse Light Field Coding Reveals About Scene Structure
Authors: Ole Johannsen, Antonin Sulc, Bastian Goldluecke
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Depth Estimation

In this paper, we propose a novel method for depth estimation in light fields which employs a specifically designed sparse decomposition to leverage the depth-orientation relationship on its epipolar plane images. The proposed method learns the structure of the central view and uses this information to construct a light field dictionary for which groups of atoms correspond to unique disparities. This dictionary is then used to code a sparse representation of the light field. Analysing the coefficients of this representation with respect to the disparities of their corresponding atoms yields an accurate and robust estimate of depth. In addition, if the light field has multiple depth layers, such as for reflective or transparent surfaces, statistical analysis of the coefficients can be employed to infer the respective depth of the superimposed layers.
Online Reconstruction of Indoor Scenes From RGB-D Streams
Authors: Hao Wang, Jun Wang, Wang Liang
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

A system capable of performing robust online volumetric reconstruction of indoor scenes based on input from a handheld RGB-D camera is presented. Our system is powered by a two-pass reconstruction scheme. The first pass tracks camera poses at video rate and simultaneously constructs a pose graph on-the-fly. The tracker operates in real-time, which allows the reconstruction results to be visualized during the scanning process. Live visual feedbacks makes the scanning operation fast and intuitive. Upon termination of scanning, the second pass takes place to handle loop closures and reconstruct the final model using globally refined camera trajectories. The system is online with low delay and returns a dense model of sufficient accuracy. The beauty of this system lies in its speed, accuracy, simplicity and ease of implementation when compared to previous methods. We demonstrate the performance of our system on several real-world scenes and quantitatively assess the modeling accuracy with respect to ground truth models obtained from a LIDAR scanner.
Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction
Authors: Ali Osman Ulusoy, Michael J. Black, Andreas Geiger
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: 3D Reconstruction, Multi-View 3D Reconstruction

In this paper, we propose a non-local structured prior for volumetric multi-view 3D reconstruction. Towards this goal, we present a novel Markov random field model based on ray potentials in which assumptions about large 3D surface patches such as planarity or Manhattan world constraints can be efficiently encoded as probabilistic priors. We further derive an inference algorithm that reasons jointly about voxels, pixels and image segments, and estimates marginal distributions of appearance, occupancy, depth, normals and planarity. Key to tractable inference is a novel hybrid representation that spans both voxel and pixel space and that integrates non-local information from 2D image segmentations in a principled way. We compare our non-local prior to commonly employed local smoothness assumptions and a variety of state-of-the-art volumetric reconstruction baselines on challenging outdoor scenes with textureless and reflective surfaces. Our experiments indicate that regularizing over larger distances has the potential to resolve ambiguities where local regularizers fail.
Single Image Camera Calibration With Lenticular Arrays for Augmented Reality
Authors: Ian Schillebeeckx, Robert Pless
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Camera Calibration, Camera Pose Estimation, Object, Pose Estimation

We consider the problem of camera pose estimation for a scenario where the camera may have continuous and unknown changes in its focal length.  Understanding frame by frame changes in camera focal length is vital to accurately estimating camera pose and vital to accurately render virtual objects in a scene with the correct perspective. However, most approaches to camera calibration require geometric constraints from many frames or the observation of a 3D calibration object --- both of which may not be feasible in augmented reality settings. This paper introduces a calibration objects based on a flat lenticular array that creates a color coded light-field whose observed color changes depending on the angle from which it is viewed.  We derive an approach to estimate the focal length of the camera and the relative pose of an object from a single image.  We characterize the performance of camera calibration across various focal lengths and camera models, and we demonstrate the advantages of the focal length estimation in rendering a virtual object in a video with constant zooming.
Augmented Blendshapes for Real-Time Simultaneous 3D Head Modeling and Facial Motion Capture
Authors: Diego Thomas, Rin-ichiro Taniguchi
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

We propose a method to build in real-time animated 3D head models using a consumer-grade RGB-D camera. Our framework is the first one to provide simultaneously comprehensive facial motion tracking and a detailed 3D model of the user's head. Anyone's head can be instantly reconstructed and his facial motion captured without requiring any training or pre-scanning. The user starts facing the camera with a neutral expression in the first frame, but is free to move, talk and change his face expression as he wills otherwise. The facial motion is tracked using a blendshape representation while the fine geometric details are captured using a Bump image mapped over the template mesh. We propose an efficient algorithm to grow and refine the 3D model of the head on-the-fly and in real-time. We demonstrate robust and high-fidelity simultaneous facial motion tracking and 3D head modeling results on a wide range of subjects with various head poses and facial expressions. Our proposed method offers interesting possibilities for animation production and 3D video telecommunications.
Learned Binary Spectral Shape Descriptor for 3D Shape Correspondence
Authors: Jin Xie, Meng Wang, Yi Fang
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

Dense 3D shape correspondence is an important problem in computer vision and computer graphics. Recently, the local shape descriptor based 3D shape correspondence approaches have been widely studied, where the local shape descriptor is a real-valued vector to characterize the geometrical structure of the shape. Different from these real-valued local shape descriptors, in this paper, we propose to learn a novel binary spectral shape descriptor with the deep neural network for 3D shape correspondence. The binary spectral shape descriptor can require less storage space and enable fast matching. First, based on the eigenvectors of the Laplace-Beltrami operator, we construct a neural network to form a nonlinear spectral representation to characterize the shape. Then, for the defined positive and negative points on the shapes, we train the constructed neural network by minimizing the errors between the outputs and their corresponding binary descriptors, minimizing the variations of the outputs of the positive points and maximizing the variations of the outputs of the negative points, simultaneously. Finally, we binarize the output of the neural network to form the binary spectral shape descriptor for shape correspondence. The proposed binary spectral shape descriptor is evaluated on the  SCAPE and TOSCA 3D shape datasets for shape correspondence. The experimental results demonstrate the effectiveness of the proposed binary shape descriptor for the shape correspondence task.
Multiple Model Fitting as a Set Coverage Problem
Authors: Luca Magri, Andrea Fusiello
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Clustering, model

This paper deals with the extraction of multiple models from noisy or outlier-contaminated data.  We cast the multi-model fitting problem in terms of set covering, deriving a simple and effective method that  generalizes Ransac to multiple models and  deals with intersecting  structures and outliers in a straightforward and principled manner, while avoiding the typical shortcomings of sequential approaches and those of clustering. The method compares favourably against the state-of-the-art on simulated and publicly available real datasets.