pwc-archive/papers-with-abstracts huggingface dataset:
Video Rain Streak Removal by Multiscale Convolutional Sparse Coding
Authors: Minghan Li, Qi Xie, Qian Zhao, Wei Wei, Shuhang Gu, Jing Tao, Deyu Meng
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Rain Removal, Video deraining

Videos captured by outdoor surveillance equipments sometimes contain unexpected rain streaks, which brings difficulty in subsequent video processing tasks. Rain streak removal from a video is thus an important topic in recent computer vision research.   In this paper, we raise two intrinsic characteristics specifically possessed by rain streaks.   Firstly, the rain streaks in a video contain repetitive local patterns sparsely scattered over different positions of the video.   Secondly, the rain streaks are with multiscale configurations due to their occurrence on positions with different distances to the cameras.   Based on such understanding, we specifically formulate both characteristics into a multiscale convolutional sparse coding (MS-CSC) model for the video rain streak removal task.   Specifically, we use multiple convolutional filters convolved on the sparse feature maps to deliver the former characteristic, and further use multiscale filters to represent different scales of rain streaks.   Such a new encoding manner makes the proposed method capable of properly extracting rain streaks from videos, thus getting fine video deraining effects. Experiments implemented on synthetic and real videos verify the superiority of the proposed method, as compared with the state-of-the-art ones along this research line, both visually and quantitatively.
A Revised Underwater Image Formation Model
Authors: Derya Akkaynak, Tali treibitz
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: model, valid

The current underwater image formation model descends from atmospheric dehazing equations where attenuation is a weak function of wavelength. We recently showed that this model introduces significant errors and dependencies in the estimation of the direct transmission signal because underwater, light attenuates in a wavelength-dependent manner. Here, we show that the backscattered signal derived from the current model also suffers from dependencies that were previously unaccounted for. In doing so, we use oceanographic measurements to derive the physically valid space of backscatter, and further show that the wideband coefficients that govern backscatter are different than those that govern direct transmission, even though the current model treats them to be the same. We propose a revised equation for underwater image formation that takes these differences into account, and validate it through in situ experiments underwater. This revised model might explain frequent instabilities of current underwater color reconstruction models, and calls for the development of new methods.
Temporal Deformable Residual Networks for Action Segmentation in Videos
Authors: Peng Lei, Sinisa Todorovic
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Action Segmentation, Segmentation

This paper is about temporal segmentation of human actions in videos. We introduce a new model -- temporal deformable residual network (TDRN) -- aimed at  analyzing video intervals at multiple temporal scales for labeling video frames.  Our TDRN computes two parallel temporal streams: i) Residual stream that analyzes video information at its full temporal  resolution, and ii) Pooling/unpooling stream that captures long-range video information at different scales. The former  facilitates local, fine-scale action segmentation, and the latter uses multiscale context for improving accuracy of frame classification.  These two streams are computed by a set of temporal residual modules with deformable convolutions, and fused by temporal residuals at the full video resolution. Our evaluation on the University of Dundee 50 Salads, Georgia Tech  Egocentric Activities, and JHU-ISI Gesture and Skill Assessment Working Set demonstrates that TDRN outperforms the state of the art  in frame-wise segmentation accuracy, segmental edit score, and segmental overlap F1 score.
PoseFlow: A Deep Motion Representation for Understanding Human Behaviors in Videos
Authors: Dingwen Zhang, Guangyu Guo, Dong Huang, Junwei Han
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Action Recognition, Optical Flow Estimation, Pose Estimation, Pose Tracking, Temporal Action Localization

Motion of the human body is the critical cue for understanding and characterizing human behavior in videos. Most existing approaches explore the motion cue using optical flows. However, optical flow usually contains motion on both the interested human bodies and the undesired background. This "noisy" motion representation makes it very challenging for pose estimation and action recognition in real scenarios. To address this issue, this paper presents a novel deep motion representation, called PoseFlow, which reveals human motion in videos while suppressing background and motion blur, and being robust to occlusion. For learning PoseFlow with mild computational cost, we propose a functionally structured spatial-temporal deep network, PoseFlow Net (PFN), to jointly solve the skeleton localization and matching problems of PoseFlow. Comprehensive experiments show that PFN outperforms the state-of-the-art deep flow estimation models in generating PoseFlow. Moreover, PoseFlow demonstrates its potential on  improving two challenging tasks in human video analysis: pose estimation and action recognition.
Where and Why Are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
Authors: Ping Wei, Yang Liu, Tianmin Shu, Nanning Zheng, Song-Chun Zhu
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6

This paper addresses a new problem - jointly inferring human attention, intentions, and tasks from videos. Given an RGB-D video where a human performs a task, we answer three questions simultaneously: 1) where the human is looking - attention prediction; 2) why the human is looking there - intention prediction; and 3) what task the human is performing - task recognition. We propose a hierarchical model of human-attention-object (HAO) which represents tasks, intentions, and attention under a unified framework. A task is represented as sequential intentions which transition to each other. An intention is composed of the human pose, attention, and objects. A beam search algorithm is adopted for inference on the HAO graph to output the attention, intention, and task results. We built a new video dataset of tasks, intentions, and attention. It contains 14 task classes, 70 intention categories, 28 object classes, 809 videos, and approximately 330,000 frames. Experiments show that our approach outperforms existing approaches.
Interpretable Video Captioning via Trajectory Structured Localization
Authors: Xian Wu, Guanbin Li, Qingxing Cao, Qingge Ji, Liang Lin
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Decoder, Image Captioning, Sentence, Video Captioning, Video Description

Automatically describing open-domain videos with natural language are attracting increasing interest in the field of artificial intelligence. Most existing methods simply borrow ideas from image captioning and obtain a compact video representation from an ensemble of global image feature before feeding to an RNN decoder which outputs a sentence of variable length. However, it is not only arduous for the generator to focus on specific salient objects at different time given the global video representation, it is more formidable to capture the fine-grained motion information and the relation between moving instances for more subtle linguistic descriptions. In this paper, we propose a Trajectory Structured Attentional Encoder-Decoder (TSA-ED) neural network framework for more elaborate video captioning which works by integrating local spatial-temporal representation at trajectory level through structured attention mechanism. Our proposed method is based on a LSTM-based encoder-decoder framework, which incorporates an attention modeling scheme to adaptively learn the correlation between sentence structure and the moving objects in videos, and consequently generates more accurate and meticulous statement description in the decoding stage. Experimental results demonstrate that the feature representation and structured attention mechanism based on the trajectory cluster can efficiently obtain the local motion information in the video to help generate a more fine-grained video description, and achieve the state-of-the-art performance on the well-known Charades and MSVD datasets.
Deep Hashing via Discrepancy Minimization
Authors: Zhixiang Chen, Xin Yuan, Jiwen Lu, Qi Tian, Jie zhou
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Deep Hashing

This paper presents a discrepancy minimizing model to address the discrete optimization problem in hashing learning.  The discrete optimization introduced by binary constraint is an NP-hard mixed integer programming problem. It is usually addressed by relaxing the binary variables into continuous variables to adapt to the gradient based learning of hashing functions, especially the training of deep neural networks. To deal with the objective discrepancy caused by relaxation, we transform the original binary optimization into differentiable optimization problem over hash functions through series expansion. This transformation decouples the binary constraint and the similarity preserving hashing function optimization. The transformed objective is optimized in a tractable alternating optimization framework with gradual discrepancy minimization. Extensive experimental results on three benchmark datasets validate the efficacy of the proposed discrepancy minimizing hashing.
Deep Reinforcement Learning of Region Proposal Networks for Object Detection
Authors: Aleksis Pirinen, Cristian Sminchisescu
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Deep Reinforcement Learning, Object, object-detection, Object Detection, Region Proposal, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We propose drl-RPN, a deep reinforcement learning-based visual recognition model consisting of a sequential region proposal network (RPN) and an object detector. In contrast to typical RPNs, where candidate object regions (RoIs) are selected greedily via class-agnostic NMS, drl-RPN optimizes an objective closer to the final detection task. This is achieved by replacing the greedy RoI selection process with a sequential attention mechanism which is trained via deep reinforcement learning (RL). Our model is capable of accumulating class-specific evidence over time, potentially affecting subsequent proposals and classification scores, and we show that such context integration significantly boosts detection accuracy. Moreover, drl-RPN automatically decides when to stop the search process and has the benefit of being able to jointly learn the parameters of the policy and the detector, both represented as deep networks. Our model can further learn to search over a wide range of exploration-accuracy trade-offs making it possible to specify or adapt the exploration extent at test time. The resulting search trajectories are image- and category-dependent, yet rely only on a single policy over all object categories. Results on the MS COCO and PASCAL VOC challenges show that our approach outperforms established, typical state-of-the-art object detection pipelines.
Occluded Pedestrian Detection Through Guided Attention in CNNs
Authors: Shanshan Zhang, Jian Yang, Bernt Schiele
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Pedestrian Detection

Pedestrian detection has progressed significantly in the last years. However, occluded people are notoriously hard to detect, as their appearance varies substantially depending on a wide range of partial occlusions. In this paper, we aim to propose a simple and compact method based on the FasterRCNN architecture for occluded pedestrian detection.  We start with interpreting CNN channel features of a pedestrian detector, and we find that different channels activate responses for different body parts respectively. These findings strongly motivate us to employ an attention mechanism across channels to represent various occlusion patterns in one single model, as each occlusion pattern can be formulated as some specific combination of body parts. Therefore, an attention network with self or external guidances is proposed as an add-on to the baseline FasterRCNN detector. When evaluating on the heavy occlusion subset, we achieve a significant improvement of 8pp to the baseline FasterRCNN detector on CityPersons and on Caltech we outperform the state-of-the-art method by 4pp.
Weakly-Supervised Semantic Segmentation Network With Deep Seeded Region Growing
Authors: Zilong Huang, Xinggang Wang, Jiasi Wang, Wenyu Liu, Jingdong Wang
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Image Segmentation, Segmentation, Semantic Segmentation, Weakly supervised Semantic Segmentation, Weakly-Supervised Semantic Segmentation

This paper studies the problem of learning image semantic segmentation networks only using image-level labels as supervision, which is important since it can significantly reduce human annotation efforts. Recent state-of-the-art methods on this problem first infer the sparse and discriminative regions for each object class using a deep classification network, then train semantic a segmentation network using the discriminative regions as supervision. Inspired by the traditional image segmentation methods of seeded region growing, we propose to train a semantic segmentation network starting from the discriminative regions and progressively increase the pixel-level supervision using by seeded region growing. The seeded region growing module is integrated in a deep segmentation network and can benefit from deep features. Different from conventional deep networks which have fixed/static labels, the proposed weakly-supervised network generates new labels using the contextual information within an image. The proposed method significantly outperforms the weakly-supervised semantic segmentation methods using static labels, and obtains the state-of-the-art performance, which are 63.2% mIoU score on the PASCAL VOC 2012 test set and 26.0% mIoU score on the COCO dataset.
PoTion: Pose MoTion Representation for Action Recognition
Authors: Vasileios Choutas, Philippe Weinzaepfel, JÃ©rÃ´me Revaud, Cordelia Schmid
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Action Recognition, Skeleton Based Action Recognition, Temporal Action Localization

Most state-of-the-art methods for action recognition rely on a two-stream architecture that processes appearance and motion independently. In this paper, we claim that considering them jointly offers rich information for action recognition. We introduce a novel representation that gracefully encodes the movement of some semantic keypoints. We use the human joints as these keypoints and term our Pose moTion representation PoTion. Specifically, we first run a state-of-the-art human pose estimator and extract heatmaps for the human joints in each frame. We obtain our PoTion representation by temporally aggregating these probability maps. This is achieved by colorizing each of them depending on the relative time of the frames in the video clip and summing them. This fixed-size representation for an entire video clip is suitable to classify actions using a shallow convolutional neural network. Our experimental evaluation shows that PoTion outperforms other state-of-the-art pose representations. Furthermore, it is complementary to standard appearance and motion streams. When combining PoTion with the recent two-stream I3D approach [5], we obtain state-of-the-art performance on the JHMDB, HMDB and UCF101 datasets.
Bilateral Ordinal Relevance Multi-Instance Regression for Facial Action Unit Intensity Estimation
Authors: Yong Zhang, Rui Zhao, Wei-Ming Dong, Bao-Gang Hu, Qiang Ji
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: regression

Automatic intensity estimation of facial action units (AUs) is challenging in two aspects. First, capturing subtle changes of facial appearance is quiet difficult. Second, the annotation of AU intensity is scarce and expensive. Intensity annotation requires strong domain knowledge thus only experts are qualified. The majority of methods directly apply supervised learning techniques to AU intensity estimation while few methods exploit unlabeled samples to improve the performance. In this paper, we propose a novel weakly supervised regression model-Bilateral Ordinal Relevance Multi-instance Regression (BORMIR), which learns a frame-level intensity estimator with weakly labeled sequences. From a new perspective, we introduce relevance to model sequential data and consider two bag labels for each bag. The AU intensity estimation is formulated as a joint regressor and relevance learning problem. Temporal dynamics of both relevance and AU intensity are leveraged to build connections among labeled and unlabeled image frames to provide weak supervision. We also develop an efficient algorithm for optimization based on the alternating minimization framework. Evaluations on three expression databases demonstrate the effectiveness of the proposed model.
Pulling Actions out of Context: Explicit Separation for Effective Combination
Authors: Yang Wang, Minh Hoai
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Action Recognition, Temporal Action Localization

The ability to recognize human actions in video has many potential applications. Human action recognition, however, is tremendously challenging for computers due to the complexity of video data and the subtlety of human actions. Most current recognition systems flounder on the inability to separate human actions from co-occurring factors that usually dominate subtle human actions.   In this paper, we propose a novel approach for training a human action recognizer, one that can: (1) explicitly factorize human actions from the co-occurring factors; (2) deliberately build a model for human actions and a separate model for all correlated contextual elements; and (3) effectively combine the models for human action recognition. Our approach exploits the benefits of conjugate samples of human actions, which are video clips that are contextually similar to human action samples, but do not contain the action. Experiments on ActionThread, PASCAL VOC, UCF101, and Hollywood2 datasets demonstrate the ability to separate action from context of the proposed approach.
Dynamic Feature Learning for Partial Face Recognition
Authors: Lingxiao He, Haiqing Li, Qi Zhang, Zhenan Sun
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Face Recognition

Partial face recognition (PFR) in unconstrained environment is a very important task, especially in video surveillance, mobile devices, etc. However, a few studies have tackled how to recognize an arbitrary patch of a face image. This study combines Fully Convolutional Network (FCN) with Sparse Representation Classification (SRC) to propose a novel partial face recognition approach, called Dynamic Feature Matching (DFM), to address partial face images regardless of sizes. Based on DFM, we propose a sliding loss to optimize FCN by reducing the intra-variation between a face patch and face images of a subject, which further improves the performance of DFM. The proposed DFM is evaluated on several partial face databases, including LFW, YTF and CASIA-NIR-Distance databases. Experimental results demonstrate the effectiveness and advantages of DFM in comparison with state-of-the-art PFR methods.
Exploiting Transitivity for Learning Person Re-Identification Models on a Budget
Authors: Sourya Roy, Sujoy Paul, Neal E. Young, Amit K. Roy-Chowdhury
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Person Re-Identification

Minimization of labeling effort for person re-identification in camera networks is an important problem as most of the existing popular methods are supervised and they require large amount of manual annotations, acquiring which is a tedious job. In this work, we focus on this labeling effort minimization problem and approach it as a subset selection task where the objective is to select an optimal subset of image-pairs for labeling without compromising performance.  Towards this goal, our proposed scheme first represents any camera network (with k number of cameras) as an edge weighted complete k-partite graph where each vertex denotes a person and similarity scores between persons are used as edge-weights. Then in the second stage, our algorithm selects an optimal subset of pairs by solving a triangle free subgraph maximization problem on the k-partite graph. This sub-graph weight maximization problem is NP-hard (at least for k > = 4) which means for large datasets the optimization problem becomes intractable. In order to make our framework scalable, we propose two polynomial time approximately-optimal algorithms. The first algorithm is a 1/2-approximation algorithm which runs in linear time in the number of edges. The second algorithm is a greedy algorithm with sub-quadratic (in number of edges) time-complexity.  Experiments on three state-of-the-art datasets depict that the proposed approach requires on an average only  8-15 % manually labeled pairs in order to achieve the performance when all the pairs are manually annotated.
Webly Supervised Learning Meets Zero-Shot Learning: A Hybrid Approach for Fine-Grained Classification
Authors: Li Niu, Ashok Veeraraghavan, Ashutosh Sabharwal
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Fine-Grained Image Classification, General Classification, image-classification, Image Classification, Zero-Shot Learning

Fine-grained image classification, which targets at distinguishing subtle distinctions among various subordinate categories, remains a very difficult task due to the high annotation cost of enormous fine-grained categories. To cope with the scarcity of well-labeled training images, existing works mainly follow two research directions: 1) utilize freely available web images without human annotation; 2) only annotate some fine-grained categories and transfer the knowledge to other fine-grained categories, which falls into the scope of zero-shot learning (ZSL). However, the above two directions have their own drawbacks. For the first direction, the labels of web images are very noisy and the data distribution between web images and test images are considerably different. For the second direction, the performance gap between ZSL and traditional supervised learning is still very large. The drawbacks of the above two directions motivate us to design a new framework which can jointly leverage both web data and auxiliary labeled categories to predict the test categories that are not associated with any well-labeled training images. Comprehensive experiments on three benchmark datasets demonstrate the effectiveness of our proposed framework.
CVM-Net: Cross-View Matching Network for Image-Based Ground-to-Aerial Geo-Localization
Authors: Sixing Hu, Mengdan Feng, Rang M. H. Nguyen, Gim Hee Lee
Date: 2018-06-01 00:00:00
Proceeding: CVPR 2018 6
Tasks: Benchmarking, geo-localization, Metric Learning

The problem of localization on a geo-referenced aerial/satellite map given a query ground view image remains challenging due to the drastic change in viewpoint that causes traditional image descriptors based matching to fail. We leverage on the recent success of deep learning to propose the CVM-Net for the cross-view image-based ground-to-aerial geo-localization task. Specifically, our network is based on the Siamese architecture to do metric learning for the matching task. We first use the fully convolutional layers to extract local image features, which are then encoded into global image descriptors using the powerful NetVLAD. As part of the training procedure, we also introduce a simple yet effective weighted soft margin ranking loss function that not only speeds up the training convergence but also improves the final matching accuracy. Experimental results show that our proposed network significantly outperforms the state-of-the-art approaches on two existing benchmarking datasets.