pwc-archive/papers-with-abstracts huggingface dataset:
Recurrent Multimodal Interaction for Referring Image Segmentation
Authors: Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Alan Yuille
Date: 2017-03-23 00:00:00
Proceeding: ICCV 2017 10
Tasks: Image Segmentation, multimodal interaction, Segmentation, Semantic Segmentation

In this paper we are interested in the problem of image segmentation given
natural language descriptions, i.e. referring expressions. Existing works
tackle this problem by first modeling images and sentences independently and
then segment images by combining these two types of representations. We argue
that learning word-to-image interaction is more native in the sense of jointly
modeling two modalities for the image segmentation task, and we propose
convolutional multimodal LSTM to encode the sequential interactions between
individual words, visual information, and spatial information. We show that our
proposed model outperforms the baseline model on benchmark datasets. In
addition, we analyze the intermediate output of the proposed multimodal LSTM
approach and empirically explain how this approach enforces a more effective
word-to-image interaction.
Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting
Authors: Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, Matthias Nießner
Date: 2017-08-04 00:00:00
Proceeding: ICCV 2017 10
Tasks: 3D Reconstruction, Surface Reconstruction

We introduce a novel method to obtain high-quality 3D reconstructions from
consumer RGB-D sensors. Our core idea is to simultaneously optimize for
geometry encoded in a signed distance field (SDF), textures from
automatically-selected keyframes, and their camera poses along with material
and scene lighting. To this end, we propose a joint surface reconstruction
approach that is based on Shape-from-Shading (SfS) techniques and utilizes the
estimation of spatially-varying spherical harmonics (SVSH) from subvolumes of
the reconstructed scene. Through extensive examples and evaluations, we
demonstrate that our method dramatically increases the level of detail in the
reconstructed scene geometry and contributes highly to consistent surface
texture recovery.
Better Together: Joint Reasoning for Non-rigid 3D Reconstruction with Specularities and Shading
Authors: Qi Liu-Yin, Rui Yu, Lourdes Agapito, Andrew Fitzgibbon, Chris Russell
Date: 2017-08-04 00:00:00
Tasks: 3D Reconstruction, Object Tracking

We demonstrate the use of shape-from-shading (SfS) to improve both the
quality and the robustness of 3D reconstruction of dynamic objects captured by
a single camera. Unlike previous approaches that made use of SfS as a
post-processing step, we offer a principled integrated approach that solves
dynamic object tracking and reconstruction and SfS as a single unified cost
function. Moving beyond Lambertian S f S , we propose a general approach that
models both specularities and shading while simultaneously tracking and
reconstructing general dynamic objects. Solving these problems jointly prevents
the kinds of tracking failures which can not be recovered from by pipeline
approaches. We show state-of-the-art results both qualitatively and
quantitatively.
Explaining Recurrent Neural Network Predictions in Sentiment Analysis
Authors: Leila Arras, Grégoire Montavon, Klaus-Robert Müller, Wojciech Samek
Date: 2017-06-22 00:00:00
Proceeding: WS 2017 9
Tasks: General Classification, Interpretable Machine Learning, Sentiment Analysis

Recently, a technique called Layer-wise Relevance Propagation (LRP) was shown
to deliver insightful explanations in the form of input space relevances for
understanding feed-forward neural network classification decisions. In the
present work, we extend the usage of LRP to recurrent neural networks. We
propose a specific propagation rule applicable to multiplicative connections as
they arise in recurrent network architectures such as LSTMs and GRUs. We apply
our technique to a word-based bi-directional LSTM model on a five-class
sentiment prediction task, and evaluate the resulting LRP relevances both
qualitatively and quantitatively, obtaining better results than a
gradient-based related method which was used in previous work.
3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks
Authors: Chuhang Zou, Ersin Yumer, Jimei Yang, Duygu Ceylan, Derek Hoiem
Date: 2017-08-04 00:00:00
Proceeding: ICCV 2017 10
Tasks: Retrieval

The success of various applications including robotics, digital content
creation, and visualization demand a structured and abstract representation of
the 3D world from limited sensor data. Inspired by the nature of human
perception of 3D shapes as a collection of simple parts, we explore such an
abstract shape representation based on primitives. Given a single depth image
of an object, we present 3D-PRNN, a generative recurrent neural network that
synthesizes multiple plausible shapes composed of a set of primitives. Our
generative model encodes symmetry characteristics of common man-made objects,
preserves long-range structural coherence, and describes objects of varying
complexity with a compact representation. We also propose a method based on
Gaussian Fields to generate a large scale dataset of primitive-based shape
representations to train our network. We evaluate our approach on a wide range
of examples and show that it outperforms nearest-neighbor based shape retrieval
methods and is on-par with voxel-based generative models while using a
significantly reduced parameter space.
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Authors: Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, Ram Nevatia
Date: 2017-03-17 00:00:00
Proceeding: ICCV 2017 10
Tasks: Action Localization, GPU, regression, Temporal Action Localization

Temporal Action Proposal (TAP) generation is an important problem, as fast
and accurate extraction of semantically important (e.g. human actions) segments
from untrimmed videos is an important step for large-scale video analysis. We
propose a novel Temporal Unit Regression Network (TURN) model. There are two
salient aspects of TURN: (1) TURN jointly predicts action proposals and refines
the temporal boundaries by temporal coordinate regression; (2) Fast computation
is enabled by unit feature reuse: a long untrimmed video is decomposed into
video units, which are reused as basic building blocks of temporal proposals.
TURN outperforms the state-of-the-art methods under average recall (AR) by a
large margin on THUMOS-14 and ActivityNet datasets, and runs at over 880 frames
per second (FPS) on a TITAN X GPU. We further apply TURN as a proposal
generation stage for existing temporal action localization pipelines, it
outperforms state-of-the-art performance on THUMOS-14 and ActivityNet.
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection
Authors: Debidatta Dwibedi, Ishan Misra, Martial Hebert
Date: 2017-08-04 00:00:00
Proceeding: ICCV 2017 10
Tasks: Object, object-detection, Object Detection

A major impediment in rapidly deploying object detection models for instance
detection is the lack of large annotated datasets. For example, finding a large
labeled dataset containing instances in a particular kitchen is unlikely. Each
new environment with new instances requires expensive data collection and
annotation. In this paper, we propose a simple approach to generate large
annotated instance datasets with minimal effort. Our key insight is that
ensuring only patch-level realism provides enough training signal for current
object detector models. We automatically `cut' object instances and `paste'
them on random backgrounds. A naive way to do this results in pixel artifacts
which result in poor performance for trained models. We show how to make
detectors ignore these artifacts during training and generate data that gives
competitive performance on real data. Our method outperforms existing synthesis
approaches and when combined with real images improves relative performance by
more than 21% on benchmark datasets. In a cross-domain setting, our synthetic
data combined with just 10% real data outperforms models trained on all real
data.
Localizing Moments in Video with Natural Language
Authors: Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, Bryan Russell
Date: 2017-08-04 00:00:00
Proceeding: ICCV 2017 10
Tasks: Natural Language Queries

We consider retrieving a specific temporal segment, or moment, from a video
given a natural language text description. Methods designed to retrieve whole
video clips with natural language determine what occurs in a video but not
when. To address this issue, we propose the Moment Context Network (MCN) which
effectively localizes natural language queries in videos by integrating local
and global video features over time. A key obstacle to training our MCN model
is that current video datasets do not include pairs of localized video segments
and referring expressions, or text descriptions which uniquely identify a
corresponding moment. Therefore, we collect the Distinct Describable Moments
(DiDeMo) dataset which consists of over 10,000 unedited, personal videos in
diverse visual settings with pairs of localized video segments and referring
expressions. We demonstrate that MCN outperforms several baseline methods and
believe that our initial results together with the release of DiDeMo will
inspire further research on localizing video moments with natural language.
Learning Approximately Objective Priors
Authors: Eric Nalisnick, Padhraic Smyth
Date: 2017-04-04 00:00:00

Informative Bayesian priors are often difficult to elicit, and when this is
the case, modelers usually turn to noninformative or objective priors. However,
objective priors such as the Jeffreys and reference priors are not tractable to
derive for many models of interest. We address this issue by proposing
techniques for learning reference prior approximations: we select a parametric
family and optimize a black-box lower bound on the reference prior objective to
find the member of the family that serves as a good approximation. We
experimentally demonstrate the method's effectiveness by recovering Jeffreys
priors and learning the Variational Autoencoder's reference prior.
Game theory models for communication between agents: a review
Authors: Aisha D. Farooqui, Muaz A. Niazi
Date: 2017-08-04 00:00:00

In the real world, agents or entities are in a continuous state of
interactions. These inter- actions lead to various types of complexity
dynamics. One key difficulty in the study of complex agent interactions is the
difficulty of modeling agent communication on the basis of rewards. Game theory
offers a perspective of analysis and modeling these interactions. Previously,
while a large amount of literature is available on game theory, most of it is
from specific domains and does not cater for the concepts from an agent- based
perspective. Here in this paper, we present a comprehensive multidisciplinary
state-of-the-art review and taxonomy of game theory models of complex
interactions between agents.
A simple yet effective baseline for 3d human pose estimation
Authors: Julieta Martinez, Rayat Hossain, Javier Romero, James J. Little
Date: 2017-05-08 00:00:00
Proceeding: ICCV 2017 10
Tasks: 3D Human Pose Estimation, 3D Pose Estimation, Monocular 3D Human Pose Estimation, Pose Estimation

Following the success of deep convolutional networks, state-of-the-art
methods for 3d human pose estimation have focused on deep end-to-end systems
that predict 3d joint locations given raw image pixels. Despite their excellent
performance, it is often not easy to understand whether their remaining error
stems from a limited 2d pose (visual) understanding, or from a failure to map
2d poses into 3-dimensional positions. With the goal of understanding these
sources of error, we set out to build a system that given 2d joint locations
predicts 3d positions. Much to our surprise, we have found that, with current
technology, "lifting" ground truth 2d joint locations to 3d space is a task
that can be solved with a remarkably low error rate: a relatively simple deep
feed-forward network outperforms the best reported result by about 30\% on
Human3.6M, the largest publicly available 3d pose estimation benchmark.
Furthermore, training our system on the output of an off-the-shelf
state-of-the-art 2d detector (\ie, using images as input) yields state of the
art results -- this includes an array of systems that have been trained
end-to-end specifically for this task. Our results indicate that a large
portion of the error of modern deep 3d pose estimation systems stems from their
visual analysis, and suggests directions to further advance the state of the
art in 3d human pose estimation.
FAST: A Framework to Accelerate Super-Resolution Processing on Compressed Videos
Authors: Zhengdong Zhang, Vivienne Sze
Date: 2016-03-29 00:00:00
Tasks: Super-Resolution

State-of-the-art super-resolution (SR) algorithms require significant
computational resources to achieve real-time throughput (e.g., 60Mpixels/s for
HD video). This paper introduces FAST (Free Adaptive Super-resolution via
Transfer), a framework to accelerate any SR algorithm applied to compressed
videos. FAST exploits the temporal correlation between adjacent frames such
that SR is only applied to a subset of frames; SR pixels are then transferred
to the other frames. The transferring process has negligible computation cost
as it uses information already embedded in the compressed video (e.g., motion
vectors and residual). Adaptive processing is used to retain accuracy when the
temporal correlation is not present (e.g., occlusions). FAST accelerates
state-of-the-art SR algorithms by up to 15x with a visual quality loss of
0.2dB. FAST is an important step towards real-time SR algorithms for ultra-HD
displays and energy constrained devices (e.g., phones and tablets).
Validation of Enhanced Emotion Enabled Cognitive Agent Using Virtual Overlay Multi-Agent System Approach
Authors: Faisal Riaz, Muaz A. Niazi
Date: 2017-08-04 00:00:00
Tasks: Autonomous Vehicles, Collision Avoidance

Making roads safer by avoiding road collisions is one of the main reasons for
inventing Autonomous vehicles (AVs). In this context, designing agent-based
collision avoidance components of AVs which truly represent human cognition and
emotions look is a more feasible approach as agents can replace human drivers.
However, to the best of our knowledge, very few human emotion and
cognition-inspired agent-based studies have previously been conducted in this
domain. Furthermore, these agent-based solutions have not been validated using
any key validation technique. Keeping in view this lack of validation
practices, we have selected state-of-the-art Emotion Enabled Cognitive Agent
(EEC_Agent), which was proposed to avoid lateral collisions between semi-AVs.
The architecture of EEC_Agent has been revised using Exploratory Agent Based
Modeling (EABM) level of the Cognitive Agent Based Computing (CABC) framework
and real-time fear emotion generation mechanism using the Ortony, Clore &
Collins (OCC) model has also been introduced. Then the proposed fear generation
mechanism has been validated using the Validated Agent Based Modeling level of
CABC framework using a Virtual Overlay MultiAgent System (VOMAS). Extensive
simulation and practical experiments demonstrate that the Enhanced EEC_Agent
exhibits the capability to feel different levels of fear, according to
different traffic situations and also needs a smaller Stopping Sight Distance
(SSD) and Overtaking Sight Distance (OSD) as compared to human drivers.
Agent based Tools for Modeling and Simulation of Self-Organization in Peer-to-Peer, Ad-Hoc and other Complex Networks
Authors: Muaz A. Niazi, Amir Hussain
Date: 2017-08-04 00:00:00

Agent-based modeling and simulation tools provide a mature platform for
development of complex simulations. They however, have not been applied much in
the domain of mainstream modeling and simulation of computer networks. In this
article, we evaluate how and if these tools can offer any value-addition in the
modeling & simulation of complex networks such as pervasive computing,
large-scale peer-to-peer systems, and networks involving considerable
environment and human/animal/habitat interaction. Specifically, we demonstrate
the effectiveness of NetLogo - a tool that has been widely used in the area of
agent-based social simulation.
Recurrent Models for Situation Recognition
Authors: Arun Mallya, Svetlana Lazebnik
Date: 2017-03-18 00:00:00
Proceeding: ICCV 2017 10
Tasks: Grounded Situation Recognition, Human-Object Interaction Detection, Image Captioning, Prediction, Situation Recognition

This work proposes Recurrent Neural Network (RNN) models to predict
structured 'image situations' -- actions and noun entities fulfilling semantic
roles related to the action. In contrast to prior work relying on Conditional
Random Fields (CRFs), we use a specialized action prediction network followed
by an RNN for noun prediction. Our system obtains state-of-the-art accuracy on
the challenging recent imSitu dataset, beating CRF-based models, including ones
trained with additional data. Further, we show that specialized features
learned from situation prediction can be transferred to the task of image
captioning to more accurately describe human-object interactions.
Region-Based Multiscale Spatiotemporal Saliency for Video
Authors: Trung-Nghia Le, Akihiro Sugimoto
Date: 2017-08-04 00:00:00
Tasks: Saliency Detection

Detecting salient objects from a video requires exploiting both spatial and
temporal knowledge included in the video. We propose a novel region-based
multiscale spatiotemporal saliency detection method for videos, where static
features and dynamic features computed from the low and middle levels are
combined together. Our method utilizes such combined features spatially over
each frame and, at the same time, temporally across frames using consistency
between consecutive frames. Saliency cues in our method are analyzed through a
multiscale segmentation model, and fused across scale levels, yielding to
exploring regions efficiently. An adaptive temporal window using motion
information is also developed to combine saliency values of consecutive frames
in order to keep temporal consistency across frames. Performance evaluation on
several popular benchmark datasets validates that our method outperforms
existing state-of-the-arts.
Sensing Urban Land-Use Patterns By Integrating Google Tensorflow And Scene-Classification Models
Authors: Yao Yao, Haolin Liang, Xia Li, Jinbao Zhang, Jialv He
Date: 2017-08-04 00:00:00
Tasks: General Classification, Scene Classification, Transfer Learning

With the rapid progress of China's urbanization, research on the automatic
detection of land-use patterns in Chinese cities is of substantial importance.
Deep learning is an effective method to extract image features. To take
advantage of the deep-learning method in detecting urban land-use patterns, we
applied a transfer-learning-based remote-sensing image approach to extract and
classify features. Using the Google Tensorflow framework, a powerful
convolution neural network (CNN) library was created. First, the transferred
model was previously trained on ImageNet, one of the largest object-image data
sets, to fully develop the model's ability to generate feature vectors of
standard remote-sensing land-cover data sets (UC Merced and WHU-SIRI). Then, a
random-forest-based classifier was constructed and trained on these generated
vectors to classify the actual urban land-use pattern on the scale of traffic
analysis zones (TAZs). To avoid the multi-scale effect of remote-sensing
imagery, a large random patch (LRP) method was used. The proposed method could
efficiently obtain acceptable accuracy (OA = 0.794, Kappa = 0.737) for the
study area. In addition, the results show that the proposed method can
effectively overcome the multi-scale effect that occurs in urban land-use
classification at the irregular land-parcel level. The proposed method can help
planners monitor dynamic urban land use and evaluate the impact of
urban-planning schemes.
Identification of Probabilities
Authors: Paul M. B. Vitanyi, Nick Chater
Date: 2017-08-04 00:00:00

Within psychology, neuroscience and artificial intelligence, there has been
increasing interest in the proposal that the brain builds probabilistic models
of sensory and linguistic input: that is, to infer a probabilistic model from a
sample. The practical problems of such inference are substantial: the brain has
limited data and restricted computational resources. But there is a more
fundamental question: is the problem of inferring a probabilistic model from a
sample possible even in principle? We explore this question and find some
surprisingly positive and general results. First, for a broad class of
probability distributions characterised by computability restrictions, we
specify a learning algorithm that will almost surely identify a probability
distribution in the limit given a finite i.i.d. sample of sufficient but
unknown length. This is similarly shown to hold for sequences generated by a
broad class of Markov chains, subject to computability assumptions. The
technical tool is the strong law of large numbers. Second, for a large class of
dependent sequences, we specify an algorithm which identifies in the limit a
computable measure for which the sequence is typical, in the sense of
Martin-Lof (there may be more than one such measure). The technical tool is the
theory of Kolmogorov complexity. We analyse the associated predictions in both
cases. We also briefly consider special cases, including language learning, and
wider theoretical implications for psychology.
Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes
Authors: Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, Carsten Rother
Date: 2017-08-04 00:00:00
Tasks: Instance Segmentation, Object, object-detection, Object Detection, Semantic Segmentation

The success of deep learning in computer vision is based on availability of
large annotated datasets. To lower the need for hand labeled images, virtually
rendered 3D worlds have recently gained popularity. Creating realistic 3D
content is challenging on its own and requires significant human effort. In
this work, we propose an alternative paradigm which combines real and synthetic
data for learning semantic instance segmentation and object detection models.
Exploiting the fact that not all aspects of the scene are equally important for
this task, we propose to augment real-world imagery with virtual objects of the
target category. Capturing real-world images at large scale is easy and cheap,
and directly provides real background appearances without the need for creating
complex 3D models of the environment. We present an efficient procedure to
augment real images with virtual objects. This allows us to create realistic
composite images which exhibit both realistic background appearance and a large
number of complex object arrangements. In contrast to modeling complete 3D
environments, our augmentation approach requires only a few user interactions
in combination with 3D shapes of the target object. Through extensive
experimentation, we conclude the right set of parameters to produce augmented
data which can maximally enhance the performance of instance segmentation
models. Further, we demonstrate the utility of our approach on training
standard deep models for semantic instance segmentation and object detection of
cars in outdoor driving scenes. We test the models trained on our augmented
data on the KITTI 2015 dataset, which we have annotated with pixel-accurate
ground truth, and on Cityscapes dataset. Our experiments demonstrate that
models trained on augmented imagery generalize better than those trained on
synthetic data or models trained on limited amount of annotated real data.