pwc-archive/papers-with-abstracts huggingface dataset:
Neural Scene Decomposition for Multi-Person Motion Capture
Authors: Helge Rhodin, Victor Constantin, Isinsu Katircioglu, Mathieu Salzmann, Pascal Fua
Date: 2019-03-13 00:00:00
Proceeding: CVPR 2019 6
Tasks: 3D geometry, 3D Pose Estimation, Instance Segmentation, Pose Estimation, Semantic Segmentation

Learning general image representations has proven key to the success of many
computer vision tasks. For example, many approaches to image understanding
problems rely on deep networks that were initially trained on ImageNet, mostly
because the learned features are a valuable starting point to learn from
limited labeled data. However, when it comes to 3D motion capture of multiple
people, these features are only of limited use.
  In this paper, we therefore propose an approach to learning features that are
useful for this purpose. To this end, we introduce a self-supervised approach
to learning what we call a neural scene decomposition (NSD) that can be
exploited for 3D pose estimation. NSD comprises three layers of abstraction to
represent human subjects: spatial layout in terms of bounding-boxes and
relative depth; a 2D shape representation in terms of an instance segmentation
mask; and subject-specific appearance and 3D pose information. By exploiting
self-supervision coming from multiview data, our NSD model can be trained
end-to-end without any 2D or 3D supervision. In contrast to previous
approaches, it works for multiple persons and full-frame images. Because it
encodes 3D geometry, NSD can then be effectively leveraged to train a 3D pose
estimation network from small amounts of annotated data.
Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments
Authors: Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, Jan Kautz
Date: 2019-03-13 00:00:00
Proceeding: CVPR 2019 6
Tasks: valid

Affordance modeling plays an important role in visual understanding. In this
paper, we aim to predict affordances of 3D indoor scenes, specifically what
human poses are afforded by a given indoor environment, such as sitting on a
chair or standing on the floor. In order to predict valid affordances and learn
possible 3D human poses in indoor scenes, we need to understand the semantic
and geometric structure of a scene as well as its potential interactions with a
human. To learn such a model, a large-scale dataset of 3D indoor affordances is
required. In this work, we build a fully automatic 3D pose synthesizer that
fuses semantic knowledge from a large number of 2D poses extracted from TV
shows as well as 3D geometric knowledge from voxel representations of indoor
scenes. With the data created by the synthesizer, we introduce a 3D pose
generative model to predict semantically plausible and physically feasible
human poses within a given scene (provided as a single RGB, RGB-D, or depth
image). We demonstrate that our human affordance prediction method consistently
outperforms existing state-of-the-art methods.
PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
Authors: Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, Simon Lucey
Date: 2019-03-13 00:00:00
Proceeding: CVPR 2019 6
Tasks: Computational Efficiency, Point Cloud Registration, Rolling Shutter Correction

PointNet has revolutionized how we think about representing point clouds. For
classification and segmentation tasks, the approach and its subsequent
extensions are state-of-the-art. To date, the successful application of
PointNet to point cloud registration has remained elusive. In this paper we
argue that PointNet itself can be thought of as a learnable "imaging" function.
As a consequence, classical vision algorithms for image alignment can be
applied on the problem - namely the Lucas & Kanade (LK) algorithm. Our central
innovations stem from: (i) how to modify the LK algorithm to accommodate the
PointNet imaging function, and (ii) unrolling PointNet and the LK algorithm
into a single trainable recurrent deep neural network. We describe the
architecture, and compare its performance against state-of-the-art in common
registration scenarios. The architecture offers some remarkable properties
including: generalization across shape categories and computational efficiency
- opening up new paths of exploration for the application of deep learning to
point cloud registration. Code and videos are available at
https://github.com/hmgoforth/PointNetLK.
LPM: Learnable Pooling Module for Efficient Full-Face Gaze Estimation
Authors: Reo Ogusu, Takao Yamanaka
Date: 2019-03-13 00:00:00
Tasks: Gaze Estimation

Gaze tracking is an important technology in many domains. Techniques such as
Convolutional Neural Networks (CNN) has allowed the invention of gaze tracking
method that relies only on commodity hardware such as the camera on a personal
computer. It has been shown that the full-face region for gaze estimation can
provide better performance than from an eye image alone. However, a problem
with using the full-face image is the heavy computation due to the larger image
size. This study tackles this problem through compression of the input
full-face image by removing redundant information using a novel learnable
pooling module. The module can be trained end-to-end by backpropagation to
learn the size of the grid in the pooling filter. The learnable pooling module
keeps the resolution of valuable regions high and vice versa. This proposed
method preserved the gaze estimation accuracy at a certain level when the image
was reduced to a smaller size.
Learning Parallax Attention for Stereo Image Super-Resolution
Authors: Longguang Wang, Yingqian Wang, Zhengfa Liang, Zaiping Lin, Jungang Yang, Wei An, Yulan Guo
Date: 2019-03-14 00:00:00
Proceeding: CVPR 2019 6
Tasks: Image Super-Resolution, Stereo Image Super-Resolution, Super-Resolution

Stereo image pairs can be used to improve the performance of super-resolution
(SR) since additional information is provided from a second viewpoint. However,
it is challenging to incorporate this information for SR since disparities
between stereo images vary significantly. In this paper, we propose a
parallax-attention stereo superresolution network (PASSRnet) to integrate the
information from a stereo image pair for SR. Specifically, we introduce a
parallax-attention mechanism with a global receptive field along the epipolar
line to handle different stereo images with large disparity variations. We also
propose a new and the largest dataset for stereo image SR (namely, Flickr1024).
Extensive experiments demonstrate that the parallax-attention mechanism can
capture correspondence between stereo images to improve SR performance with a
small computational and memory cost. Comparative results show that our PASSRnet
achieves the state-of-the-art performance on the Middlebury, KITTI 2012 and
KITTI 2015 datasets.
Neural Style Transfer for Point Clouds
Authors: Xu Cao, Weimin WANG, Katashi Nagao
Date: 2019-03-14 00:00:00
Tasks: Style Transfer

How can we edit or transform the geometric or color property of a point
cloud? In this study, we propose a neural style transfer method for point
clouds which allows us to transfer the style of geometry or color from one
point cloud either independently or simultaneously to another. This transfer is
achieved by manipulating the content representations and Gram-based style
representations extracted from a pre-trained PointNet-based classification
network for colored point clouds. As Gram-based style representation is
invariant to the number or the order of points, the same method can be extended
to transfer the style extracted from an image to the color expression of a
point cloud by merely treating the image as a set of pixels. Experimental
results demonstrate the capability of the proposed method for transferring
style from either an image or a point cloud to another point cloud of a single
object or even an indoor scene.
Purifying Naturalistic Images through a Real-time Style Transfer Semantics Network
Authors: Tongtong Zhao, Yuxiao Yan, Ibrahim Shehi Shehu, Xianping Fu, Huibing Wang
Date: 2019-03-14 00:00:00
Tasks: Gaze Estimation, Style Transfer

Recently, the progress of learning-by-synthesis has proposed a training model
for synthetic images, which can effectively reduce the cost of human and
material resources. However, due to the different distribution of synthetic
images compared to real images, the desired performance cannot still be
achieved. Real images consist of multiple forms of light orientation, while
synthetic images consist of a uniform light orientation. These features are
considered to be characteristic of outdoor and indoor scenes, respectively. To
solve this problem, the previous method learned a model to improve the realism
of the synthetic image. Different from the previous methods, this paper takes
the first step to purify real images. Through the style transfer task, the
distribution of outdoor real images is converted into indoor synthetic images,
thereby reducing the influence of light. Therefore, this paper proposes a
real-time style transfer network that preserves image content information (eg,
gaze direction, pupil center position) of an input image (real image) while
inferring style information (eg, image color structure, semantic features) of
style image (synthetic image). In addition, the network accelerates the
convergence speed of the model and adapts to multi-scale images. Experiments
were performed using mixed studies (qualitative and quantitative) methods to
demonstrate the possibility of purifying real images in complex directions.
Qualitatively, it compares the proposed method with the available methods in a
series of indoor and outdoor scenarios of the LPW dataset. In quantitative
terms, it evaluates the purified image by training a gaze estimation model on
the cross data set. The results show a significant improvement over the
baseline method compared to the raw real image.
Learning Orientation-Estimation Convolutional Neural Network for Building Detection in Optical Remote Sensing Image
Authors: Yongliang Chen
Date: 2019-03-14 00:00:00
Tasks: GPU, object-detection, Object Detection, regression

Benefiting from the great success of deep learning in computer vision,
CNN-based object detection methods have drawn significant attentions. Various
frameworks have been proposed which show awesome and robust performance for a
large range of datasets. However, for building detection in remote sensing
images, buildings always pose a diversity of orientations which makes it a
challenge for the application of off-the-shelf methods to building detection.
In this work, we aim to integrate orientation regression into the popular
axis-aligned bounding-box detection method to tackle this problem. To adapt the
axis-aligned bounding boxes to arbitrarily orientated ones, we also develop an
algorithm to estimate the Intersection over Union (IoU) overlap between any two
arbitrarily oriented boxes which is convenient to implement in Graphics
Processing Unit (GPU) for accelerating computation. The proposed method
utilizes CNN for both robust feature extraction and rotated bounding box
regression. We present our modelin an end-to-end fashion making it easy to
train. The model is formulated and trained to predict orientation, location and
extent simultaneously obtaining tighter bounding box and hence, higher mean
average precision (mAP). Experiments on remote sensing images of different
scales shows a promising performance over the conventional one.
Learning to Reconstruct People in Clothing from a Single RGB Camera
Authors: Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll
Date: 2019-03-14 00:00:00
Proceeding: CVPR 2019 6

We present a learning-based model to infer the personalized 3D shape of
people from a few frames (1-8) of a monocular video in which the person is
moving, in less than 10 seconds with a reconstruction accuracy of 5mm. Our
model learns to predict the parameters of a statistical body model and instance
displacements that add clothing and hair to the shape. The model achieves fast
and accurate predictions based on two key design choices. First, by predicting
shape in a canonical T-pose space, the network learns to encode the images of
the person into pose-invariant latent codes, where the information is fused.
Second, based on the observation that feed-forward predictions are fast but do
not always align with the input images, we predict using both, bottom-up and
top-down streams (one per view) allowing information to flow in both
directions. Learning relies only on synthetic 3D data. Once learned, the model
can take a variable number of frames as input, and is able to reconstruct
shapes even from a single image with an accuracy of 6mm. Results on 3 different
datasets demonstrate the efficacy and accuracy of our approach.
Scalable Facial Image Compression with Deep Feature Reconstruction
Authors: Shurun Wang, Shiqi Wang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Wen Gao
Date: 2019-03-14 00:00:00
Tasks: Image Compression

In this paper, we propose a scalable image compression scheme, including the base layer for feature representation and enhancement layer for texture representation. More specifically, the base layer is designed as the deep learning feature for analysis purpose, and it can also be converted to the fine structure with deep feature reconstruction. The enhancement layer, which serves to compress the residuals between the input image and the signals generated from the base layer, aims to faithfully reconstruct the input texture. The proposed scheme can feasibly inherit the advantages of both compress-then-analyze and analyze-then-compress schemes in surveillance applications. The performance of this framework is validated with facial images, and the conducted experiments provide useful evidences to show that the proposed framework can achieve better rate-accuracy and rate-distortion performance over conventional image compression schemes.
Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning
Authors: Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, In So Kweon
Date: 2019-03-14 00:00:00
Proceeding: CVPR 2019 6
Tasks: Diversity, Image Captioning, POS, Relational Captioning

Our goal in this work is to train an image captioning model that generates more dense and informative captions. We introduce "relational captioning," a novel image captioning task which aims to generate multiple captions with respect to relational information between objects in an image. Relational captioning is a framework that is advantageous in both diversity and amount of information, leading to image understanding based on relationships. Part-of speech (POS, i.e. subject-object-predicate categories) tags can be assigned to every English word. We leverage the POS as a prior to guide the correct sequence of words in a caption. To this end, we propose a multi-task triple-stream network (MTTSNet) which consists of three recurrent units for the respective POS and jointly performs POS prediction and captioning. We demonstrate more diverse and richer representations generated by the proposed model against several baselines and competing methods.
Superpixel-based Color Transfer
Authors: Rémi Giraud, Vinh-Thong Ta, Nicolas Papadakis
Date: 2019-03-14 00:00:00
Tasks: Diversity, Superpixels

In this work, we propose a fast superpixel-based color transfer method (SCT)
between two images. Superpixels enable to decrease the image dimension and to
extract a reduced set of color candidates. We propose to use a fast approximate
nearest neighbor matching algorithm in which we enforce the match diversity by
limiting the selection of the same superpixels. A fusion framework is designed
to transfer the matched colors, and we demonstrate the improvement obtained
over exact matching results. Finally, we show that SCT is visually competitive
compared to state-of-the-art methods.
Audiovisual Speaker Tracking using Nonlinear Dynamical Systems with Dynamic Stream Weights
Authors: Christopher Schymura, Dorothea Kolossa
Date: 2019-03-14 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

Data fusion plays an important role in many technical applications that
require efficient processing of multimodal sensory observations. A prominent
example is audiovisual signal processing, which has gained increasing attention
in automatic speech recognition, speaker localization and related tasks. If
appropriately combined with acoustic information, additional visual cues can
help to improve the performance in these applications, especially under adverse
acoustic conditions. A dynamic weighting of acoustic and visual streams based
on instantaneous sensor reliability measures is an efficient approach to data
fusion in this context. This paper presents a framework that extends the
well-established theory of nonlinear dynamical systems with the notion of
dynamic stream weights for an arbitrary number of sensory observations. It
comprises a recursive state estimator based on the Gaussian filtering paradigm,
which incorporates dynamic stream weights into a framework closely related to
the extended Kalman filter. Additionally, a convex optimization approach to
estimate oracle dynamic stream weights in fully observed dynamical systems
utilizing a Dirichlet prior is presented. This serves as a basis for a generic
parameter learning framework of dynamic stream weight estimators. The proposed
system is application-independent and can be easily adapted to specific tasks
and requirements. A study using audiovisual speaker tracking tasks is
considered as an exemplary application in this work. An improved tracking
performance of the dynamic stream weight-based estimation framework over
state-of-the-art methods is demonstrated in the experiments.
Deep learning enabled multi-wavelength spatial coherence microscope for the classification of malaria-infected stages with limited labelled data size
Authors: Neeru Singla, Vishal Srivastava
Date: 2019-03-14 00:00:00
Tasks: General Classification

Malaria is a life-threatening mosquito-borne blood disease, hence early
detection is very crucial for health. The conventional method for the detection
is a microscopic examination of Giemsa-stained blood smears, which needs a
highly trained skilled technician. Automated classifications of different
stages of malaria still a challenging task, especially having poor sensitivity
in detecting the early trophozoite and late trophozoite or schizont stage with
limited labelled datasize. The study aims to develop a fast, robust and fully
automated system for the classification of different stages of malaria with
limited data size by using the pre-trained convolutional neural networks (CNNs)
as a classifier and multi-wavelength to increase the sample size. We also
compare our customized CNN with other well-known CNNs and shows that our
network have a comparable performance with less computational time. We believe
that our proposed method can be applied to other limited labelled biological
datasets.
Deep Residual Autoencoder for quality independent JPEG restoration
Authors: Simone Zini, Simone Bianco, Raimondo Schettini
Date: 2019-03-14 00:00:00
Tasks: SSIM

In this paper we propose a deep residual autoencoder exploiting
Residual-in-Residual Dense Blocks (RRDB) to remove artifacts in JPEG compressed
images that is independent from the Quality Factor (QF) used. The proposed
approach leverages both the learning capacity of deep residual networks and
prior knowledge of the JPEG compression pipeline. The proposed model operates
in the YCbCr color space and performs JPEG artifact restoration in two phases
using two different autoencoders: the first one restores the luma channel
exploiting 2D convolutions; the second one, using the restored luma channel as
a guide, restores the chroma channels explotining 3D convolutions. Extensive
experimental results on three widely used benchmark datasets (i.e. LIVE1,
BDS500, and CLASSIC-5) show that our model is able to outperform the state of
the art with respect to all the evaluation metrics considered (i.e. PSNR,
PSNR-B, and SSIM). This results is remarkable since the approaches in the state
of the art use a different set of weights for each compression quality, while
the proposed model uses the same weights for all of them, making it applicable
to images in the wild where the QF used for compression is unkwnown.
Furthermore, the proposed model shows a greater robustness than
state-of-the-art methods when applied to compression qualities not seen during
training.
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition
Authors: Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, Jiebo Luo
Date: 2019-03-14 00:00:00
Proceeding: CVPR 2019 6
Tasks: Fine-Grained Image Classification, Fine-Grained Image Recognition

Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps by modeling the inter-channel relationships, 2) an attention-based sampler which highlights attended parts with high resolution, and 3) a feature distiller, which distills part features into a global one by weight sharing and feature preserving strategies. Extensive experiments verify that TASN yields the best performance under the same settings with the most competitive approaches, in iNaturalist-2017, CUB-Bird, and Stanford-Cars datasets.
Aesthetics of Neural Network Art
Authors: Aaron Hertzmann
Date: 2019-03-13 00:00:00
Tasks: Image Stylization

This paper proposes a way to understand neural network artworks as
juxtapositions of natural image cues. It is hypothesized that images with
unusual combinations of realistic visual cues are interesting, and, neural
models trained to model natural images are well-suited to creating interesting
images. Art using neural models produces new images similar to those of natural
images, but with weird and intriguing variations. This analysis is applied to
neural art based on Generative Adversarial Networks, image stylization, Deep
Dreams, and Perception Engines.
Inferring 3D Shapes of Unknown Rigid Objects in Clutter through Inverse Physics Reasoning
Authors: Changkyu Song, Abdeslam Boularias
Date: 2019-03-13 00:00:00

We present a probabilistic approach for building, on the fly, 3-D models of
unknown objects while being manipulated by a robot. We specifically consider
manipulation tasks in piles of clutter that contain previously unseen objects.
Most manipulation algorithms for performing such tasks require known geometric
models of the objects in order to grasp or rearrange them robustly. One of the
novel aspects of this work is the utilization of a physics engine for verifying
hypothesized geometries in simulation. The evidence provided by physics
simulations is used in a probabilistic framework that accounts for the fact
that mechanical properties of the objects are uncertain. We present an
efficient algorithm for inferring occluded parts of objects based on their
observed motions and mutual interactions. Experiments using a robot show that
this approach is efficient for constructing physically realistic 3-D models,
which can be useful for manipulation planning. Experiments also show that the
proposed approach significantly outperforms alternative approaches in terms of
shape accuracy.