pwc-archive/papers-with-abstracts huggingface dataset:
3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis
Authors: Xiaojuan Qi,  Zhengzhe Liu,  Qifeng Chen,  Jiaya Jia
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Video Prediction

A future video is the 2D projection of a 3D scene with predicted camera and object motion. Accurate future video prediction inherently requires understanding of 3D motion and geometry of a scene. In this paper, we propose a RGBD scene forecasting model with 3D motion decomposition. We predict ego-motion and foreground motion that are combined to generate a future 3D dynamic scene, which is then projected into a 2D image plane to synthesize future motion, RGB images and depth maps. Optional semantic maps can be integrated. Experimental results on KITTI and Driving datasets show that our model outperforms other state-of-the- arts in forecasting future RGBD dynamic scenes.
Bayesian Hierarchical Dynamic Model for Human Action Recognition
Authors: Rui Zhao,  Wanru Xu,  Hui Su,  Qiang Ji
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Recognition, Bayesian Inference, Missing Values, model, Multimodal Activity Recognition, Skeleton Based Action Recognition, Temporal Action Localization

Human action recognition remains as a challenging task partially due to the presence of large variations in the execution of action. To address this issue, we propose a probabilistic model called Hierarchical Dynamic Model (HDM). Leveraging on Bayesian framework, the model parameters are allowed to vary across different sequences of data, which increase the capacity of the model to adapt to intra-class variations on both spatial and temporal extent of actions. Meanwhile, the generative learning process allows the model to preserve the distinctive dynamic pattern for each action class. Through Bayesian inference, we are able to quantify the uncertainty of the classification, providing insight during the decision process. Compared to state-of-the-art methods, our method not only achieves competitive recognition performance within individual dataset but also shows better generalization capability across different datasets. Experiments conducted on data with missing values also show the robustness of the proposed method.
Mixed Effects Neural Networks (MeNets) With Applications to Gaze Estimation
Authors: Yunyang Xiong,  Hyunwoo J. Kim,  Vikas Singh
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Gaze Estimation

There is much interest in computer vision to utilize commodity hardware for gaze estimation. A number of papers have shown that algorithms based on deep convolutional architectures are approaching accuracies where streaming data from mass-market devices can offer good gaze tracking performance, although a gap still remains between what is possible and the performance users will expect in real deployments. We observe that one obvious avenue for improvement relates to a gap between some basic technical assumptions behind most existing approaches and the statistical properties of the data used for training. Specifically, most training datasets involve tens of users with a few hundreds (or more) repeated acquisitions per user. The non i.i.d. nature of this data suggests better estimation may be possible if the model explicitly made use of such "repeated measurements" from each user as is commonly done in classical statistical analysis using so-called mixed effects models. The goal of this paper is to adapt these "mixed effects" ideas from statistics within a deep neural network architecture for gaze estimation, based on eye images. Such a formulation seeks to specifically utilize information regarding the hierarchical structure of the training data -- each node in the hierarchy is a user who provides tens or hundreds of repeated samples. This modification yields an architecture that offers state of the art performance on various publicly available datasets improving results by 10-20%.
Unequal-Training for Deep Face Recognition With Long-Tailed Noisy Data
Authors: Yaoyao Zhong,  Weihong Deng,  Mei Wang,  Jiani Hu,  Jianteng Peng,  Xunqiang Tao,  Yaohai Huang
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Face Recognition, GPU

Large-scale face datasets usually exhibit a massive number of classes, a long-tailed distribution, and severe label noise, which undoubtedly aggravate the difficulty of training. In this paper, we propose a training strategy that treats the head data and the tail data in an unequal way, accompanying with noise-robust loss functions, to take full advantage of their respective characteristics. Specifically, the unequal-training framework provides two training data streams: the first stream applies the head data to learn discriminative face representation supervised by Noise Resistance loss; the second stream applies the tail data to learn auxiliary information by gradually mining the stable discriminative information from confusing tail classes. Consequently, both training streams offer complementary information to deep feature learning. Extensive experiments have demonstrated the effectiveness of the new unequal-training framework and loss functions. Better yet, our method could save a significant amount of GPU memory. With our method, we achieve the best result on MegaFace Challenge 2 (MF2) given a large-scale noisy training data set.
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Authors: Lele Chen,  Ross K. Maddox,  Zhiyao Duan,  Chenliang Xu
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Face Generation, Talking Face Generation

We devise a cascade GAN approach to generate talking face video, which is robust to different face shapes, view angles, facial characteristics, and noisy audio conditions. Instead of learning a direct mapping from audio to video frames, we propose first to transfer audio to high-level structure, i.e., the facial landmarks, and then to generate video frames conditioned on the landmarks. Compared to a direct audio-to-image approach, our cascade approach avoids fitting spurious correlations between audiovisual signals that are irrelevant to the speech content. We, humans, are sensitive to temporal discontinuities and subtle artifacts in video. To avoid those pixel jittering problems and to enforce the network to focus on audiovisual-correlated regions, we propose a novel dynamically adjustable pixel-wise loss with an attention mechanism. Furthermore, to generate a sharper image with well-synchronized facial movements, we propose a novel regression-based discriminator structure, which considers sequence-level information along with frame-level information. Thoughtful experiments on several datasets and real-world samples demonstrate significantly better results obtained by our method than the state-of-the-art methods in both quantitative and qualitative comparisons.
Collaborative Spatiotemporal Feature Learning for Video Action Recognition
Authors: Chao Li,  Qiaoyong Zhong,  Di Xie,  Shiliang Pu
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Classification, Action Recognition, Action Recognition In Videos, Temporal Action Localization, Video Recognition

Spatiotemporal feature learning is of central importance for action recognition in videos. Existing deep neural network models either learn spatial and temporal features independently (C2D) or jointly with unconstrained parameters (C3D). In this paper, we propose a novel neural operation which encodes spatiotemporal features collaboratively by imposing a weight-sharing constraint on the learnable parameters. In particular, we perform 2D convolution along three orthogonal views of volumetric video data, which learns spatial appearance and temporal motion cues respectively. By sharing the convolution kernels of different views, spatial and temporal features are collaboratively learned and thus benefit from each other. The complementary features are subsequently fused by a weighted summation whose coefficients are learned end-to-end. Our approach achieves state-of-the-art performance on large-scale benchmarks and won the 1st place in the Moments in Time Challenge 2018. Moreover, based on the learned coefficients of different views, we are able to quantify the contributions of spatial and temporal features. This analysis sheds light on interpretability of the model and may also guide the future design of algorithm for video recognition.
MARS: Motion-Augmented RGB Stream for Action Recognition
Authors: Nieves Crasto,  Philippe Weinzaepfel,  Karteek Alahari,  Cordelia Schmid
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Classification, Action Recognition, Optical Flow Estimation, Temporal Action Localization

Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearance stream for RGB frames and a motion stream for optical flow frames.  Although combining flow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-world applications requiring low latency. In this paper, we introduce two learning approaches to train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoids flow computation at test time.  First, by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity.  Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-based loss and the standard cross-entropy loss for action recognition. We denote the stream trained using this combined loss as Motion-Augmented RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics compared to 72.0% and 65.6% with RGB and Flow streams respectively.
Skeleton-Based Action Recognition With Directed Graph Neural Networks
Authors: Lei Shi,  Yifan Zhang,  Jian Cheng,  Hanqing Lu
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Recognition, Graph Neural Network, Skeleton Based Action Recognition, Temporal Action Localization

The skeleton data have been widely used for the action recognition tasks since they can robustly accommodate dynamic circumstances and complex backgrounds. In existing methods, both the joint and bone information in skeleton data have been proved to be of great help for action recognition tasks. However, how to incorporate these two types of data to best take advantage of the relationship between joints and bones remains a problem to be solved. In this work, we represent the skeleton data as a directed acyclic graph based on the kinematic dependency between the joints and bones in the natural human body. A novel directed graph neural network is designed specially to extract the information of joints, bones and their relations and make prediction based on the extracted features. In addition, to better fit the action recognition task, the topological structure of the graph is made adaptive based on the training process, which brings notable improvement. Moreover, the motion information of the skeleton sequence is exploited and combined with the spatial information to further enhance the performance in a two-stream framework. Our final model is tested on two large-scale datasets, NTU-RGBD and Skeleton-Kinetics, and exceeds state-of-the-art performance on both of them.
PA3D: Pose-Action 3D Machine for Video Recognition
Authors: An Yan,  Yali Wang,  Zhifeng Li,  Yu Qiao
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Action Recognition, Optical Flow Estimation, Skeleton Based Action Recognition, Temporal Action Localization, Video Recognition

Recent studies have witnessed the successes of using 3D CNNs for video action recognition. However, most 3D models are built upon RGB and optical flow streams, which may not fully exploit pose dynamics, i.e., an important cue of modeling human actions. To fill this gap, we propose a concise Pose-Action 3D Machine (PA3D), which can effectively encode multiple pose modalities within a unified 3D framework, and consequently learn spatio-temporal pose representations for action recognition.  More specifically, we introduce a novel temporal pose convolution to aggregate spatial poses over frames. Unlike the classical temporal convolution, our operation can explicitly learn the pose motions that are discriminative to recognize human actions. Extensive experiments on three popular benchmarks (i.e., JHMDB, HMDB, and Charades) show that, PA3D outperforms the recent pose-based approaches. Furthermore, PA3D is highly complementary to the recent 3D CNNs, e.g., I3D. Multi-stream fusion achieves the state-of-the-art performance on all evaluated data sets.
Depth-Attentional Features for Single-Image Rain Removal
Authors: Xiaowei Hu,  Chi-Wing Fu,  Lei Zhu,  Pheng-Ann Heng
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Rain Removal, Single Image Deraining

Rain is a common weather phenomenon, where object visibility varies with depth from the camera and objects faraway are visually blocked more by fog than by rain streaks. Existing methods and datasets for rain removal, however, ignore these physical properties, thereby limiting the rain removal efficiency on real photos. In this work, we first analyze the visual effects of rain subject to scene depth and formulate a rain imaging model collectively with rain streaks and fog; by then, we prepare a new dataset called RainCityscapes with rain streaks and fog on real outdoor photos. Furthermore, we design an end-to-end deep neural network, where we train it to learn depth-attentional features via a depth-guided attention mechanism, and regress a residual map to produce the rain-free image output. We performed various experiments to visually and quantitatively compare our method with several state-of-the-art methods to demonstrate its superiority over the others.
Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior
Authors: Lizhi Wang,  Chen Sun,  Ying Fu,  Min H. Kim,  Hua Huang
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Image Reconstruction

Regularization is a fundamental technique to solve an ill-posed optimization problem robustly and is essential to reconstruct compressive hyperspectral images. Various hand-crafted priors have been employed as a regularizer but are often insufficient to handle the wide variety of spectra of natural hyperspectral images, resulting in poor reconstruction quality. Moreover, the prior-regularized optimization requires manual tweaking of its weight parameters to achieve a balance between the spatial and spectral fidelity of result images. In this paper, we present a novel hyperspectral image reconstruction algorithm that substitutes the traditional hand-crafted prior with a data-driven prior, based on an optimization-inspired network. Our method consists of two main parts: First, we learn a novel data-driven prior that regularizes the optimization problem with a goal to boost the spatial-spectral fidelity. Our data-driven prior learns both local coherence and dynamic characteristics of natural hyperspectral images. Second, we combine our regularizer with an optimization-inspired network to overcome the heavy computation problem in the traditional iterative optimization methods. We learn the complete parameters in the network through end-to-end training, enabling robust performance with high accuracy. Extensive simulation and hardware experiments validate the superior performance of our method over the state-of-the-art methods.
UnOS: Unified Unsupervised Optical-Flow and Stereo-Depth Estimation by Watching Videos
Authors: Yang Wang,  Peng Wang,  Zhenheng Yang,  Chenxu Luo,  Yi Yang,  Wei Xu
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Depth Estimation, Motion Segmentation, Optical Flow Estimation, Stereo Depth Estimation, Visual Odometry

In this paper, we propose UnOS, an unified system for unsupervised optical flow and stereo depth estimation using convolutional neural network (CNN) by taking advantages of their inherent geometrical consistency based on the rigid-scene assumption. UnOS significantly outperforms other state-of-the-art (SOTA) unsupervised approaches that treated the two tasks independently. Specifically, given two consecutive stereo image pairs from a video, UnOS estimates per-pixel stereo depth images, camera ego-motion and optical flow with three parallel CNNs. Based on these quantities, UnOS computes rigid optical flow and compares it against the optical flow estimated from the FlowNet, yielding pixels satisfying the rigid-scene assumption. Then, we encourage geometrical consistency between the two estimated flows within rigid regions, from which we derive a rigid-aware direct visual odometry (RDVO) module. We also propose rigid and occlusion-aware flow-consistency losses for the learning of UnOS. We evaluated our results on the popular KITTI dataset over 4 related tasks, i.e. stereo depth, optical flow, visual odometry and motion segmentation.
D2-Net: A Trainable CNN for Joint Description and Detection of Local Features
Authors: Mihai Dusmanu,  Ignacio Rocco,  Tomas Pajdla,  Marc Pollefeys,  Josef Sivic,  Akihiko Torii,  Torsten Sattler
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: 3D Reconstruction, Indoor Localization

In this work we address the problem of finding reliable pixel-level correspondences under difficult imaging conditions. We propose an approach where a single convolutional neural network plays a dual role: It is simultaneously a dense feature descriptor and a feature detector. By postponing the detection to a later stage, the obtained keypoints are more stable than their traditional counterparts based on early detection of low-level structures. We show that this model can be trained using pixel correspondences extracted from readily available large-scale SfM reconstructions, without any further annotations. The proposed method obtains state-of-the-art performance on both the difficult Aachen Day-Night localization dataset and the InLoc indoor localization benchmark, as well as competitive performance on other benchmarks for image matching and 3D reconstruction.
Recurrent Neural Networks With Intra-Frame Iterations for Video Deblurring
Authors: Seungjun Nah,  Sanghyun Son,  Kyoung Mu Lee
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Deblurring, Video Deblurring

Recurrent neural networks (RNNs) are widely used for sequential data processing. Recent state-of-the-art video deblurring methods bank on convolutional recurrent neural network architectures to exploit the temporal relationship between neighboring frames. In this work, we aim to improve the accuracy of recurrent models by adapting the hidden states transferred from past frames to the frame being processed so that the relations between video frames could be better used. We iteratively update the hidden state via re-using RNN cell parameters before predicting an output deblurred frame. Since we use existing parameters to update the hidden state, our method improves accuracy without additional modules. As the architecture remains the same regardless of iteration number, fewer iteration models can be considered as a partial computational path of the models with more iterations. To take advantage of this property, we employ a stochastic method to optimize our iterative models better. At training time, we randomly choose the iteration number on the fly and apply a regularization loss that favors less computation unless there are considerable reconstruction gains. We show that our method exhibits state-of-the-art video deblurring performance while operating in real-time speed.
Learning to Extract Flawless Slow Motion From Blurry Videos
Authors: Meiguang Jin,  Zhe Hu,  Paolo Favaro
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6

In this paper, we introduce the task of generating a sharp slow-motion video given a low frame rate blurry video.  We propose a data-driven approach, where the training data is captured with a high frame rate camera and blurry images are simulated through an averaging process.  While it is possible to train a neural network to recover the sharp frames from their average, there is no guarantee of the temporal smoothness for the formed video, as the frames are estimated independently. To address the temporal smoothness requirement we propose a system with two networks: One, DeblurNet, to predict sharp keyframes  and the second, InterpNet, to predict intermediate frames between the generated keyframes. A smooth transition is ensured by interpolating between consecutive keyframes using InterpNet.  Moreover, the proposed scheme enables further increase in frame rate without retraining the network, by applying InterpNet recursively between pairs of sharp frames. We evaluate the proposed method on several datasets, including a novel dataset captured with a Sony RX V camera. We also demonstrate its performance of increasing the frame rate up to 20 times on real blurry videos.
A Mutual Learning Method for Salient Object Detection With Intertwined Multi-Supervision
Authors: Runmin Wu,  Mengyang Feng,  Wenlong Guan,  Dong Wang,  Huchuan Lu,  Errui Ding
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Contour Detection, Edge Detection, Object, object-detection, Object Detection, RGB Salient Object Detection, Saliency Detection, Salient Object Detection

Though deep learning techniques have made great progress in salient object detection recently, the predicted saliency maps still suffer from incomplete predictions due to the internal complexity of objects and inaccurate boundaries caused by strides in convolution and pooling operations. To alleviate these issues, we propose to train saliency detection networks by exploiting the supervision from not only salient object detection, but also foreground contour detection and edge detection. First, we leverage salient object detection and foreground contour detection tasks in an intertwined manner to generate saliency maps with uniform highlight. Second, the foreground contour and edge detection tasks guide each other simultaneously, thereby leading to preciser foreground contour prediction and reducing the local noises for edge prediction. In addition, we develop a novel mutual learning module (MLM) which serves as the building block of our method. Each MLM consists of multiple network branches trained in a mutual learning manner, which improves the performance by a large margin. Extensive experiments on seven challenging datasets demonstrate that the proposed method has delivered state-of-the-art results in both salient object detection and edge detection.
Enhanced Pix2pix Dehazing Network
Authors: Yanyun Qu,  Yizi Chen,  Jingying Huang,  Yuan Xie
Date: 2019-06-01 00:00:00
Proceeding: CVPR 2019 6
Tasks: Generative Adversarial Network, Image Dehazing, Image-to-Image Translation, SSIM, Translation

In this paper,  we reduce the image dehazing problem to an image-to-image translation problem, and propose  Enhanced Pix2pix Dehazing Network (EPDN),  which generates a haze-free image without relying on the physical scattering model.  EPDN is embedded by a generative adversarial network,  which is followed by a well-designed enhancer.  Inspired by visual perception global-first theory,  the discriminator guides the generator to create a pseudo realistic image on a coarse scale,  while the enhancer following the generator is required to produce a realistic dehazing image on the fine scale.  The enhancer contains two enhancing blocks based on the receptive field model,  which reinforces the dehazing effect in both color and details. The embedded GAN is jointly trained with the enhancer. Extensive experiment results on synthetic datasets and real-world datasets show that the proposed EPDN is superior to the state-of-the-art methods in terms of PSNR,  SSIM,   PI, and subjective visual effect.