pwc-archive/papers-with-abstracts huggingface dataset:
Image Style Transfer Using Convolutional Neural Networks
Authors: Leon A. Gatys, Alexander S. Ecker, Matthias Bethge
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Image Generation, Object Recognition, Style Transfer

Rendering the semantic content of an image in different styles is a difficult image processing task. Arguably, a major limiting factor for previous approaches has been the lack of image representations that explicitly represent semantic information and, thus, allow to separate image content from style. Here we use image representations derived from Convolutional Neural Networks optimised for object recognition, which make high level image information explicit. We introduce A Neural Algorithm of Artistic Style that can separate and recombine the image content and style of natural images. The algorithm allows us to produce new images of high perceptual quality that combine the content of an arbitrary photograph with the appearance of numerous well-known artworks. Our results provide new insights into the deep image representations learned by Convolutional Neural Networks and demonstrate their potential for high level image synthesis and manipulation.
Hedgehog Shape Priors for Multi-Object Segmentation
Authors: Hossam Isack, Olga Veksler, Milan Sonka, Yuri Boykov
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Descriptive, Object, Segmentation, Semantic Segmentation

Star-convexity prior is popular for interactive single object segmentation due to its simplicity and amenability to binary graph cut optimization. We propose a more general multi-object segmentation approach. Moreover, each object can be constrained by a more descriptive shape prior, "hedgehog". Each hedgehog shape has its surface normals locally constrained by an arbitrary given vector field, e.g. gradient of the user-scribble distance transform. In contrast to star-convexity, the tightness of our normal constraint can be changed giving better control over allowed shapes. For example, looser constraints, i.e. wider cones of allowed normals, give more relaxed hedgehog shapes. On the other hand, the tightest constraint enforces skeleton consistency with the scribbles. In general, hedgehog shapes are more descriptive than a star, which is only a special case corresponding to a radial vector field and weakest tightness. Our approach has significantly more applications than standard single star-convex segmentation, e.g. in medical data we can separate multiple non-star organs with similar appearances and weak edges. Optimization is done by our modified a-expansion moves shown to be submodular for multi-hedgehog shapes.
Latent Variable Graphical Model Selection Using Harmonic Analysis: Applications to the Human Connectome Project (HCP)
Authors: Won Hwa Kim, Hyunwoo J. Kim, Nagesh Adluru, Vikas Singh
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Model Selection

A major goal of imaging studies such as the (ongoing) Human Connectome Project (HCP) is to characterize the structural network map of the human brain and identify its associations with covariates such as genotype, risk factors, and so on that correspond to an individual. But the set of image derived measures and the set of covariates are both large, so we must first estimate a 'parsimonious' set of relations between the measurements. For instance, a Gaussian graphical model will show conditional independences between the random variables, which can then be used to setup specific hypothesis based analyses downstream. But most such data involve a large list of 'latent' variables that remain unobserved, yet affect the 'observed' variables sustantially. Accounting for such latent variables falls outside the scope of standard inverse covariance matrix estimation, and is tackled via highly specialized optimization methods. This paper offers a unique harmonic analysis view of this problem. By casting the estimation of the precision matrix in terms of a composition of low-frequency latent variables and high-frequency sparse terms, we show how the problem can be formulated using a new wavelet-type expansion in non-Euclidean spaces. Our formalization poses the estimation problem entirely in the frequency space and shows how it can be solved by a simple sub-gradient scheme (involving a single variable). We provide a compelling set of scientific results on   500 scans from the recently released HCP data where our algorithm recovers highly interpretable and sparse conditional dependencies between brain connectivity pathways and well-known covariates.
Simultaneous Estimation of Near IR BRDF and Fine-Scale Surface Geometry
Authors: Gyeongmin Choe, Srinivasa G. Narasimhan, In So Kweon
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Depth Estimation, Intrinsic Image Decomposition, Inverse Rendering

Near-Infrared (NIR) images of most materials exhibit less texture or albedo variations making them beneficial for vision tasks such as intrinsic image decomposition and structured light depth estimation. Understanding the reflectance properties (BRDF) of materials in the NIR wavelength range can be further useful for many photometric methods including shape from shading and inverse rendering. However, even with less albedo variation, many materials e.g. fabrics, leaves, etc. exhibit complex fine-scale surface detail making it hard to accurately estimate BRDF. In this paper, we present an approach to simultaneously estimate NIR BRDF and fine-scale surface details by imaging materials under different IR lighting and viewing directions. This is achieved by an iterative scheme that alternately estimates surface detail and NIR BRDF of materials. Our setup does not require complicated gantries or calibration and we present the first NIR dataset of 100 materials including a variety of fabrics (knits, weaves, cotton, satin, leather), and organic (skin, leaves, jute, trunk, fur) and inorganic materials (plastic, concrete, carpet). The NIR BRDFs measured from material samples are used with a shape-from-shading algorithm to demonstrate fine-scale reconstruction of objects from a single NIR image.
Do It Yourself Hyperspectral Imaging With Everyday Digital Cameras
Authors: Seoung Wug Oh, Michael S. Brown, Marc Pollefeys, Seon Joo Kim
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

Capturing hyperspectral images requires expensive and specialized hardware that is not readily accessible to most users. Digital cameras, on the other hand, are significantly cheaper in comparison and can be easily purchased and used. In this paper, we present a framework for reconstructing hyperspectral images by using multiple consumer-level digital cameras. Our approach works by exploiting the different spectral sensitivities of different camera sensors. In particular, due to the differences in spectral sensitivities of the cameras, different cameras yield different RGB measurements for the same spectral signal. We  introduce an algorithm that is able to combine and convert these different RGB measurements into a single hyperspectral image for both indoor and outdoor scenes. This camera-based approach allows hyperspectral imaging at a fraction of the cost of most existing hyperspectral hardware. We validate the accuracy of our reconstruction against ground truth hyperspectral images (using both synthetic and real cases) and show its usage on relighting applications.
Conformal Surface Alignment With Optimal Mobius Search
Authors: Huu Le, Tat-Jun Chin, David Suter
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

Deformations of surfaces with the same intrinsic shape can often be described accurately by a conformal model. A major focus of computational conformal geometry is the estimation of the conformal mapping that aligns a given pair of object surfaces. The uniformization theorem en- ables this task to be acccomplished in a canonical 2D do- main, wherein the surfaces can be aligned using a M obius transformation. Current algorithms for estimating M obius transformations, however, often cannot provide satisfactory alignment or are computationally too costly. This paper in- troduces a novel globally optimal algorithm for estimating Mobius transformations to align surfaces that are topologi- cal discs. Unlike previous methods, the proposed algorithm deterministically calculates the best transformation, with- out requiring good initializations. Further, our algorithm is also much faster than previous techniques in practice. We demonstrate the efficacy of our algorithm on data commonly used in computational conformal geometry.
Coupled Harmonic Bases for Longitudinal Characterization of Brain Networks
Authors: Seong Jae Hwang, Nagesh Adluru, Maxwell D. Collins, Sathya N. Ravi, Barbara B. Bendlin, Sterling C. Johnson, Vikas Singh
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6

There is a great deal of interest in using large scale brain imaging studies to understand how brain connectivity evolves over time for an individual and how it varies over different levels/quantiles of cognitive function. To do so, one typically performs so-called tractography procedures on diffusion MR brain images and derives measures of brain connectivity expressed as graphs. The nodes correspond to distinct brain regions and the edges encode the strength of the connection. The scientific interest is in characterizing the evolution of these graphs over time or from healthy individuals to diseased. We pose this important question in terms of the Laplacian of the connectivity graphs derived from various longitudinal or disease time points - quantifying its progression is then expressed in terms of coupling the harmonic bases of a full set of Laplacians. We derive a coupled system of generalized eigenvalue problems (and corresponding numerical optimization schemes) whose solution helps characterize the full life cycle of brain connectivity evolution in a given dataset. Finally, we show a set of results on a diffusion MR imaging dataset of middle aged people at risk for Alzheimer's disease (AD), who are cognitively healthy. In such asymptomatic adults, we find that a framework for characterizing brain connectivity evolution provides the ability to predict cognitive scores for individual subjects, and for estimating the progression of participant's brain connectivity into the future.
Learning to Co-Generate Object Proposals With a Deep Structured Network
Authors: Zeeshan Hayder, Xuming He, Mathieu Salzmann
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Object, object-detection, Object Detection, Rolling Shutter Correction

Generating object proposals has become a key component of modern object detection pipelines. However, most existing methods generate the object candidates independently of each other. In this paper, we present an approach to co-generating object proposals in multiple images, thus leveraging the collective power of multiple object candidates. In particular, we introduce a deep structured network that jointly predicts the objectness scores and the bounding box locations of multiple object candidates. Our deep structured network consists of a fully-connected Conditional Random Field built on top of a set of deep Convolutional Neural Networks, which learn features to model both the individual object candidate and the similarity between multiple candidates. To train our deep structured network, we develop an end-to-end learning algorithm that, by unrolling the CRF inference procedure, lets us backpropagate the loss gradient throughout the entire structured network. We demonstrate the effectiveness of our approach on two benchmark datasets, showing significant improvement over state-of-the-art object proposal algorithms.
MDL-CW: A Multimodal Deep Learning Framework With Cross Weights
Authors: Sarah Rastegar, Mahdieh Soleymani, Hamid R. Rabiee, Seyed Mohsen Shojaee
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Attribute, Deep Learning, Information Retrieval, Multimodal Deep Learning, Representation Learning, Retrieval, Sentence

Deep learning has received much attention as of the most powerful approaches for multimodal representation learning in recent years. An ideal model for multimodal data can reason about missing modalities using the available ones, and usually provides more information when multiple modalities are being considered. All the previous deep models contain separate modality-specific networks and find a shared representation on top of those networks. Therefore, they only consider high level interactions between modalities to find a joint representation for them. In this paper, we propose a multimodal deep learning framework (MDL-CW) that exploits the cross weights between representation of modalities, and try to gradually learn interactions of the modalities in a deep network manner (from low to high level interactions). Moreover, we theoretically show that considering these interactions provide more intra-modality information, and introduce a multi-stage pre-training method that is based on the properties of multi-modal data. In the proposed framework, as opposed to the existing deep methods for multi-modal data, we try to reconstruct the representation of each modality at a given level, with representation of other modalities in the previous layer. Extensive experimental results show that the proposed model outperforms state-of-the-art information retrieval methods for both image and text queries on the PASCAL-sentence and SUN-Attribute databases.
First Person Action Recognition Using Deep Learned Descriptors
Authors: Suriya Singh, Chetan Arora, C. V. Jawahar
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Recognition, Activity Recognition, Temporal Action Localization

We focus on the problem of wearer's action recognition in first person a.k.a. egocentric videos. This problem is more challenging than third person activity recognition due to unavailability of wearer's pose and sharp movements in the videos caused by the natural head motion of the wearer. Carefully crafted features based on hands and objects cues for the problem have been shown to be successful for limited targeted datasets. We propose convolutional neural networks (CNNs) for end to end learning and classification of wearer's actions. The proposed network makes use of egocentric cues by capturing hand pose, head motion and saliency map. It is compact. It can also be trained from relatively small number of labeled egocentric videos that are available. We show that the proposed network can generalize and give state of the art performance on various disparate egocentric action datasets.
Recognizing Micro-Actions and Reactions From Paired Egocentric Videos
Authors: Ryo Yonetani, Kris M. Kitani, Yoichi Sato
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Video Summarization

We aim to understand the dynamics of social interactions between two people by recognizing their actions and reactions using a head-mounted camera. Our work will impact several first-person vision tasks that need the detailed understanding of social interactions, such as automatic video summarization of group events and assistive systems. To recognize micro-level actions and reactions, such as slight shifts in attention, subtle nodding, or small hand actions, where only subtle body motion is apparent, we propose to use paired egocentric videos recorded by two interacting people. We show that the first-person and second-person points-of-view features of two people, enabled by paired egocentric videos, are complementary and essential for reliably recognizing micro-actions and reactions. We also build a new dataset of dyadic (two-persons) interactions that comprises more than 1000 pairs of egocentric videos to enable systematic evaluations on the task of micro-action and reaction recognition.
Mining 3D Key-Pose-Motifs for Action Recognition
Authors: Chunyu Wang, Yizhou Wang, Alan L. Yuille
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Recognition, Quantization, Temporal Action Localization

Recognizing an action from a sequence of 3D skeletal poses is a challenging task. First, different actors may perform the same action in various performing styles. Second, the estimated poses are sometimes inaccurate due to sensory noises. These challenges can cause large variations between instances of the same class. Third, the datasets are usually small, with only a few actors performing few repetitions of each action. Hence training complex classifiers risks over-fitting the data. We address this task by mining a set of key-pose-motifs for each action class. A key-pose-motif contains a set of ordered  poses or  action units(a short sequence of poses), which are required to be close but not necessarily adjacent in the action sequences. The representation is robust to style variations and outlier poses. The key-pose-motifs are represented in terms of a dictionary using soft-quantization (probabilities) to deal with inaccuracies caused by quantization. We propose an efficient algorithm to mine key-pose-motifs taking into account these probabilities. We classify a sequence by matching it to the motifs of each class and select the class that maximizes the matching score. This simple classifier obtains state-of-the-art performance on two benchmark datasets and outperforms a deep network approach.
Predicting the Where and What of Actors and Actions Through Online Action Localization
Authors: Khurram Soomro, Haroon Idrees, Mubarak Shah
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Action Localization, Superpixels

This paper proposes a novel approach to tackle the challenging problem of 'online action localization' which entails predicting actions and their locations as they happen in a video. Typically, action localization or recognition is performed in an offline manner where all the frames in the video are processed together and action labels are not predicted for the future. This dis-allows timely localization of actions - an important consideration for surveillance tasks. In our approach, given a batch of frames from the immediate past in a video, we estimate pose and over- segment the current frame into superpixels. Next, we discriminatively train an actor foreground model on the superpixels using the pose bounding boxes. A Conditional Random Field with superpixels as nodes, and edges connecting spatio-temporal neighbors is used to obtain action segments. The action confidence is predicted using dynamic programming on SVM scores obtained on short segments of the video, thereby capturing sequential information of the actions. The issue of visual drift is handled by updating the appearance model and pose refinement in an online manner. Lastly, we introduce a new measure to quantify the performance of action prediction (i.e. online action localization), which analyzes how the prediction accuracy varies as a function of observed portion of the video. Our experiments suggest that despite using only a few frames to localize actions at each time instant, we are able to predict the action and obtain competitive results to state-of-the-art offline methods.
Visual Path Prediction in Complex Scenes With Crowded Moving Objects
Authors: YoungJoon Yoo, Kimin Yun, Sangdoo Yun, JongHee Hong, Hawook Jeong, Jin Young Choi
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Prediction

This paper proposes a novel path prediction algorithm for progressing one step further than the existing works focusing on single target path prediction. In this paper, we consider moving dynamics of co-occurring objects for path prediction in a scene that includes crowded moving objects. To solve this problem, we first suggest a two-layered probabilistic model to find major movement patterns and their co-occurrence tendency. By utilizing the unsupervised learning results from the model, we present an algorithm to find the future location of any target object. Through extensive qualitative/quantitative experiments, we show that our algorithm can find a plausible future path in complex scenes with a large number of moving objects.
Laplacian Patch-Based Image Synthesis
Authors: Joo Ho Lee, Inchang Choi, Min H. Kim
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: global-optimization, Image Generation

Patch-based image synthesis has been enriched with global optimization on the image pyramid. Successively, the gradient-based synthesis has improved structural coherence and details. However, the gradient operator is directional and inconsistent and requires computing multiple operators. It also introduces a significantly heavy computational burden to solve the Poisson equation that often accompanies artifacts in non-integrable gradient fields. In this paper, we propose a patch-based synthesis using a Laplacian pyramid to improve searching correspondence with enhanced awareness of edge structures. Contrary to the gradient operators, the Laplacian pyramid has the advantage of being isotropic in detecting changes to provide more consistent performance in decomposing the base structure and the detailed localization. Furthermore, it does not require heavy computation as it employs approximation by the differences of Gaussians. We examine the potentials of the Laplacian pyramid for enhanced edge-aware correspondence search. We demonstrate the effectiveness of the Laplacian-based approach over the state-of-the-art patch-based image synthesis methods.
Rain Streak Removal Using Layer Priors
Authors: Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, Michael S. Brown
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Dictionary Learning

This paper addresses the problem of rain streak removal from a single image. Rain streaks impair visibility of an image and introduce undesirable interference that can severely affect the performance of computer vision algorithms.   Rain streak removal can be formulated as a layer decomposition problem, with a rain streak layer superimposed on a background layer containing the true scene content. Existing decomposition methods that address this problem employ either dictionary learning methods or impose a low rank structure on the appearance of the rain streaks.   While these methods can improve the overall visibility, they tend to leave too many rain streaks in the background image or over-smooth the background image.   In this paper, we propose an effective method that uses simple patch-based priors for both the background and rain layers.  These priors are based on Gaussian mixture models and can accommodate multiple orientations and scales of the rain streaks. This simple approach removes rain streaks better than the existing methods qualitatively and quantitatively.  We overview our method and demonstrate its effectiveness over prior work on a number of examples.
Gradient-Domain Image Reconstruction Framework With Intensity-Range and Base-Structure Constraints
Authors: Takashi Shibata, Masayuki Tanaka, Masatoshi Okutomi
Date: 2016-06-01 00:00:00
Proceeding: CVPR 2016 6
Tasks: Image Reconstruction, Image Restoration, Tone Mapping

This paper presents a novel unified gradient-domain image reconstruction framework with intensity-range constraint and base-structure constraint. The existing method for manipulating base structures and detailed textures are classifiable into two major approaches: i) gradient-domain and ii) layer-decomposition. To generate detail-preserving and artifact-free output images, we combine the benefits of the two approaches into the proposed framework by introducing the intensity-range constraint and the base-structure constraint. To preserve details of the input image, the proposed method takes advantage of reconstructing the output image in the gradient domain, while the output intensity is guaranteed to lie within the specified intensity range, e.g. 0-to-255, by the intensity-range constraint. In addition, the reconstructed image lies close to the base structure by the base-structure constraint, which is effective for restraining artifacts. Experimental results show that the proposed framework is effective for various applications such as tone mapping, seamless image cloning, detail enhancement, and image restoration.