pwc-archive/papers-with-abstracts huggingface dataset:
Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents
Authors: Bo Wang, Youjiang Xu, Yahong Han, Richang Hong
Date: 2018-04-25 00:00:00
Tasks: Question Answering, Video Question Answering

Movies provide us with a mass of visual content as well as attracting
stories. Existing methods have illustrated that understanding movie stories
through only visual content is still a hard problem. In this paper, for
answering questions about movies, we put forward a Layered Memory Network (LMN)
that represents frame-level and clip-level movie content by the Static Word
Memory module and the Dynamic Subtitle Memory module, respectively.
Particularly, we firstly extract words and sentences from the training movie
subtitles. Then the hierarchically formed movie representations, which are
learned from LMN, not only encode the correspondence between words and visual
content inside frames, but also encode the temporal alignment between sentences
and frames inside movie clips. We also extend our LMN model into three variant
frameworks to illustrate the good extendable capabilities. We conduct extensive
experiments on the MovieQA dataset. With only visual content as inputs, LMN
with frame-level representation obtains a large performance improvement. When
incorporating subtitles into LMN to form the clip-level representation, we
achieve the state-of-the-art performance on the online evaluation task of
'Video+Subtitles'. The good performance successfully demonstrates that the
proposed framework of LMN is effective and the hierarchically formed movie
representations have good potential for the applications of movie question
answering.
Probabilistic Plant Modeling via Multi-View Image-to-Image Translation
Authors: Takahiro Isokane, Fumio Okura, Ayaka Ide, Yasuyuki Matsushita, Yasushi Yagi
Date: 2018-04-25 00:00:00
Proceeding: CVPR 2018 6
Tasks: Image-to-Image Translation, Translation

This paper describes a method for inferring three-dimensional (3D) plant
branch structures that are hidden under leaves from multi-view observations.
Unlike previous geometric approaches that heavily rely on the visibility of the
branches or use parametric branching models, our method makes statistical
inferences of branch structures in a probabilistic framework. By inferring the
probability of branch existence using a Bayesian extension of image-to-image
translation applied to each of multi-view images, our method generates a
probabilistic plant 3D model, which represents the 3D branching pattern that
cannot be directly observed. Experiments demonstrate the usefulness of the
proposed approach in generating convincing branch structures in comparison to
prior approaches.
Generative Temporal Models with Spatial Memory for Partially Observed Environments
Authors: Marco Fraccaro, Danilo Jimenez Rezende, Yori Zwols, Alexander Pritzel, S. M. Ali Eslami, Fabio Viola
Date: 2018-04-25 00:00:00
Proceeding: ICML 2018 7
Tasks: Model-based Reinforcement Learning, Reinforcement Learning

In model-based reinforcement learning, generative and temporal models of
environments can be leveraged to boost agent performance, either by tuning the
agent's representations during training or via use as part of an explicit
planning mechanism. However, their application in practice has been limited to
simplistic environments, due to the difficulty of training such models in
larger, potentially partially-observed and 3D environments. In this work we
introduce a novel action-conditioned generative model of such challenging
environments. The model features a non-parametric spatial memory system in
which we store learned, disentangled representations of the environment.
Low-dimensional spatial updates are computed using a state-space model that
makes use of knowledge on the prior dynamics of the moving agent, and
high-dimensional visual observations are modelled with a Variational
Auto-Encoder. The result is a scalable architecture capable of performing
coherent predictions over hundreds of time steps across a range of partially
observed 2D and 3D environments.
3D Consistent & Robust Segmentation of Cardiac Images by Deep Learning with Spatial Propagation
Authors: Qiao Zheng, Hervé Delingette, Nicolas Duchateau, Nicholas Ayache
Date: 2018-04-25 00:00:00
Tasks: Cardiac Segmentation, Segmentation

We propose a method based on deep learning to perform cardiac segmentation on
short axis MRI image stacks iteratively from the top slice (around the base) to
the bottom slice (around the apex). At each iteration, a novel variant of U-net
is applied to propagate the segmentation of a slice to the adjacent slice below
it. In other words, the prediction of a segmentation of a slice is dependent
upon the already existing segmentation of an adjacent slice. 3D-consistency is
hence explicitly enforced. The method is trained on a large database of 3078
cases from UK Biobank. It is then tested on 756 different cases from UK Biobank
and three other state-of-the-art cohorts (ACDC with 100 cases, Sunnybrook with
30 cases, RVSC with 16 cases). Results comparable or even better than the
state-of-the-art in terms of distance measures are achieved. They also
emphasize the assets of our method, namely enhanced spatial consistency
(currently neither considered nor achieved by the state-of-the-art), and the
generalization ability to unseen cases even from other databases.
Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
Authors: Hao-Wen Dong, Yi-Hsuan Yang
Date: 2018-04-25 00:00:00
Tasks: Music Generation

It has been shown recently that deep convolutional generative adversarial
networks (GANs) can learn to generate music in the form of piano-rolls, which
represent music by binary-valued time-pitch matrices. However, existing models
can only generate real-valued piano-rolls and require further post-processing,
such as hard thresholding (HT) or Bernoulli sampling (BS), to obtain the final
binary-valued results. In this paper, we study whether we can have a
convolutional GAN model that directly creates binary-valued piano-rolls by
using binary neurons. Specifically, we propose to append to the generator an
additional refiner network, which uses binary neurons at the output layer. The
whole network is trained in two stages. Firstly, the generator and the
discriminator are pretrained. Then, the refiner network is trained along with
the discriminator to learn to binarize the real-valued piano-rolls the
pretrained generator creates. Experimental results show that using binary
neurons instead of HT or BS indeed leads to better results in a number of
objective measures. Moreover, deterministic binary neurons perform better than
stochastic ones in both objective measures and a subjective test. The source
code, training data and audio examples of the generated results can be found at
https://salu133445.github.io/bmusegan/ .
Learnable Histogram: Statistical Context Features for Deep Neural Networks
Authors: Zhe Wang, Hongsheng Li, Wanli Ouyang, Xiaogang Wang
Date: 2018-04-25 00:00:00
Tasks: General Classification, object-detection, Object Detection, Semantic Segmentation

Statistical features, such as histogram, Bag-of-Words (BoW) and Fisher
Vector, were commonly used with hand-crafted features in conventional
classification methods, but attract less attention since the popularity of deep
learning methods. In this paper, we propose a learnable histogram layer, which
learns histogram features within deep neural networks in end-to-end training.
Such a layer is able to back-propagate (BP) errors, learn optimal bin centers
and bin widths, and be jointly optimized with other layers in deep networks
during training. Two vision problems, semantic segmentation and object
detection, are explored by integrating the learnable histogram layer into deep
networks, which show that the proposed layer could be well generalized to
different applications. In-depth investigations are conducted to provide
insights on the newly introduced layer.
Quantitative Susceptibility Map Reconstruction Using Annihilating Filter-based Low-Rank Hankel Matrix Approach
Authors: Hyun-Seo Ahn, Sung-Hong Park, Jong Chul Ye
Date: 2018-04-25 00:00:00

Quantitative susceptibility mapping (QSM) inevitably suffers from streaking artifacts caused by zeros on the conical surface of the dipole kernel in k-space. This work proposes a novel and accurate QSM reconstruction method based on a direct k-space interpolation approach, avoiding problems of over smoothing and streaking artifacts. Inspired by the recent theory of annihilating filter-based low-rank Hankel matrix approach (ALOHA), QSM reconstruction problem is formulated as deconvolution problem under low-rank Hankel matrix constraint in the k-space. To reduce the computational complexity and the memory requirement, the problem is formulated as successive reconstruction of 2-D planes along three independent axes of the 3-D phase image in Fourier domain. Extensive experiments were performed to verify and compare the proposed method with existing QSM reconstruction methods. The proposed ALOHA-QSM effectively reduced streaking artifacts and accurately estimated susceptibility values in deep gray matter structures, compared to the existing QSM methods. Our suggested ALOHA-QSM algorithm successfully solves the three-dimensional QSM dipole inversion problem without additional anatomical information or prior assumption and provides good image quality and quantitative accuracy.
Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
Authors: Bei Liu, Jianlong Fu, Makoto P. Kato, Masatoshi Yoshikawa
Date: 2018-04-23 00:00:00
Tasks: 8k

Automatic generation of natural language from images has attracted extensive
attention. In this paper, we take one step further to investigate generation of
poetic language (with multiple lines) to an image for automatic poetry
creation. This task involves multiple challenges, including discovering poetic
clues from the image (e.g., hope from green), and generating poems to satisfy
both relevance to the image and poeticness in language level. To solve the
above challenges, we formulate the task of poem generation into two correlated
sub-tasks by multi-adversarial training via policy gradient, through which the
cross-modal relevance and poetic language style can be ensured. To extract
poetic clues from images, we propose to learn a deep coupled visual-poetic
embedding, in which the poetic representation from objects, sentiments and
scenes in an image can be jointly learned. Two discriminative networks are
further introduced to guide the poem generation, including a multi-modal
discriminator and a poem-style discriminator. To facilitate the research, we
have released two poem datasets by human annotators with two distinct
properties: 1) the first human annotated image-to-poem pair dataset (with 8,292
pairs in total), and 2) to-date the largest public English poem corpus dataset
(with 92,265 different poems in total). Extensive experiments are conducted
with 8K images, among which 1.5K image are randomly picked for evaluation. Both
objective and subjective evaluations show the superior performances against the
state-of-the-art methods for poem generation from images. Turing test carried
out with over 500 human subjects, among which 30 evaluators are poetry experts,
demonstrates the effectiveness of our approach.
Driving Policy Transfer via Modularity and Abstraction
Authors: Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, Vladlen Koltun
Date: 2018-04-25 00:00:00
Tasks: Autonomous Driving

End-to-end approaches to autonomous driving have high sample complexity and
are difficult to scale to realistic urban driving. Simulation can help
end-to-end driving systems by providing a cheap, safe, and diverse training
environment. Yet training driving policies in simulation brings up the problem
of transferring such policies to the real world. We present an approach to
transferring driving policies from simulation to reality via modularity and
abstraction. Our approach is inspired by classic driving systems and aims to
combine the benefits of modular architectures and end-to-end deep learning
approaches. The key idea is to encapsulate the driving policy such that it is
not directly exposed to raw perceptual input or low-level vehicle dynamics. We
evaluate the presented approach in simulated urban environments and in the real
world. In particular, we transfer a driving policy trained in simulation to a
1/5-scale robotic truck that is deployed in a variety of conditions, with no
finetuning, on two continents. The supplementary video can be viewed at
https://youtu.be/BrMDJqI6H5U
Pose-Normalized Image Generation for Person Re-identification
Authors: Xuelin Qian, Yanwei Fu, Tao Xiang, Wenxuan Wang, Jie Qiu, Yang Wu, Yu-Gang Jiang, xiangyang xue
Date: 2017-12-06 00:00:00
Proceeding: ECCV 2018 9
Tasks: Generative Adversarial Network, Image Generation, Person Re-Identification, Transfer Learning

Person Re-identification (re-id) faces two major challenges: the lack of
cross-view paired training data and learning discriminative identity-sensitive
and view-invariant features in the presence of large pose variations. In this
work, we address both problems by proposing a novel deep person image
generation model for synthesizing realistic person images conditional on the
pose. The model is based on a generative adversarial network (GAN) designed
specifically for pose normalization in re-id, thus termed pose-normalization
GAN (PN-GAN). With the synthesized images, we can learn a new type of deep
re-id feature free of the influence of pose variations. We show that this
feature is strong on its own and complementary to features learned with the
original images. Importantly, under the transfer learning setting, we show that
our model generalizes well to any new re-id dataset without the need for
collecting any training data for model fine-tuning. The model thus has the
potential to make re-id model truly scalable.
Multi-level Semantic Feature Augmentation for One-shot Learning
Authors: Zitian Chen, Yanwei Fu, yinda zhang, Yu-Gang Jiang, xiangyang xue, Leonid Sigal
Date: 2018-04-15 00:00:00
Tasks: Decoder, Novel Concepts, One-Shot Learning

The ability to quickly recognize and learn new visual concepts from limited
samples enables humans to swiftly adapt to new environments. This ability is
enabled by semantic associations of novel concepts with those that have already
been learned and stored in memory. Computers can start to ascertain similar
abilities by utilizing a semantic concept space. A concept space is a
high-dimensional semantic space in which similar abstract concepts appear close
and dissimilar ones far apart. In this paper, we propose a novel approach to
one-shot learning that builds on this idea. Our approach learns to map a novel
sample instance to a concept, relates that concept to the existing ones in the
concept space and generates new instances, by interpolating among the concepts,
to help learning. Instead of synthesizing new image instance, we propose to
directly synthesize instance features by leveraging semantics using a novel
auto-encoder network we call dual TriNet. The encoder part of the TriNet learns
to map multi-layer visual features of deep CNNs, that is, multi-level concepts,
to a semantic vector. In semantic space, we search for related concepts, which
are then projected back into the image feature spaces by the decoder portion of
the TriNet. Two strategies in the semantic space are explored. Notably, this
seemingly simple strategy results in complex augmented feature distributions in
the image feature space, leading to substantially better performance.
GP-GAN: Gender Preserving GAN for Synthesizing Faces from Landmarks
Authors: Xing Di, Vishwanath A. Sindagi, Vishal M. Patel
Date: 2017-10-03 00:00:00
Tasks: Face Generation, Generative Adversarial Network

Facial landmarks constitute the most compressed representation of faces and
are known to preserve information such as pose, gender and facial structure
present in the faces. Several works exist that attempt to perform high-level
face-related analysis tasks based on landmarks. In contrast, in this work, an
attempt is made to tackle the inverse problem of synthesizing faces from their
respective landmarks. The primary aim of this work is to demonstrate that
information preserved by landmarks (gender in particular) can be further
accentuated by leveraging generative models to synthesize corresponding faces.
Though the problem is particularly challenging due to its ill-posed nature, we
believe that successful synthesis will enable several applications such as
boosting performance of high-level face related tasks using landmark points and
performing dataset augmentation. To this end, a novel face-synthesis method
known as Gender Preserving Generative Adversarial Network (GP-GAN) that is
guided by adversarial loss, perceptual loss and a gender preserving loss is
presented. Further, we propose a novel generator sub-network UDeNet for GP-GAN
that leverages advantages of U-Net and DenseNet architectures. Extensive
experiments and comparison with recent methods are performed to verify the
effectiveness of the proposed method.
Gaussian variational approximation for high-dimensional state space models
Authors: Matias Quiroz, David J. Nott, Robert Kohn
Date: 2018-01-24 00:00:00
Tasks: State Space Models, Vocal Bursts Intensity Prediction

Our article considers a Gaussian variational approximation of the posterior density in a high-dimensional state space model. The variational parameters to be optimized are the mean vector and the covariance matrix of the approximation. The number of parameters in the covariance matrix grows as the square of the number of model parameters, so it is necessary to find simple yet effective parameterizations of the covariance structure when the number of model parameters is large. We approximate the joint posterior distribution over the high-dimensional state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. The usefulness of the approach is illustrated for prediction in two high-dimensional applications that are challenging for Markov chain Monte Carlo sampling. The first is a spatio-temporal model for the spread of the Eurasian Collared-Dove across North America; the second is a Wishart-based multivariate stochastic volatility model for financial returns.
Generalized Gaussian Kernel Adaptive Filtering
Authors: Tomoya Wada, Kosuke Fukumori, Toshihisa Tanaka, Simone Fiori
Date: 2018-04-25 00:00:00

The present paper proposes generalized Gaussian kernel adaptive filtering,
where the kernel parameters are adaptive and data-driven. The Gaussian kernel
is parametrized by a center vector and a symmetric positive definite (SPD)
precision matrix, which is regarded as a generalization of the scalar width
parameter. These parameters are adaptively updated on the basis of a proposed
least-square-type rule to minimize the estimation error. The main contribution
of this paper is to establish update rules for precision matrices on the SPD
manifold in order to keep their symmetric positive-definiteness. Different from
conventional kernel adaptive filters, the proposed regressor is a superposition
of Gaussian kernels with all different parameters, which makes such regressor
more flexible. The kernel adaptive filtering algorithm is established together
with a l1-regularized least squares to avoid overfitting and the increase of
dimensionality of the dictionary. Experimental results confirm the validity of
the proposed method.
Adaptation and Re-Identification Network: An Unsupervised Deep Transfer Learning Approach to Person Re-Identification
Authors: Yu-Jhe Li, Fu-En Yang, Yen-Cheng Liu, Yu-Ying Yeh, Xiaofei Du, Yu-Chiang Frank Wang
Date: 2018-04-25 00:00:00
Tasks: Domain Adaptation, Person Re-Identification, Transfer Learning, Unsupervised Domain Adaptation

Person re-identification (Re-ID) aims at recognizing the same person from
images taken across different cameras. To address this task, one typically
requires a large amount labeled data for training an effective Re-ID model,
which might not be practical for real-world applications. To alleviate this
limitation, we choose to exploit a sufficient amount of pre-existing labeled
data from a different (auxiliary) dataset. By jointly considering such an
auxiliary dataset and the dataset of interest (but without label information),
our proposed adaptation and re-identification network (ARN) performs
unsupervised domain adaptation, which leverages information across datasets and
derives domain-invariant features for Re-ID purposes. In our experiments, we
verify that our network performs favorably against state-of-the-art
unsupervised Re-ID approaches, and even outperforms a number of baseline Re-ID
methods which require fully supervised data for training.
Reusing Weights in Subword-aware Neural Language Models
Authors: Zhenisbek Assylbekov, Rustem Takhanov
Date: 2018-02-23 00:00:00
Proceeding: NAACL 2018 6

We propose several ways of reusing subword embeddings and other weights in
subword-aware neural language models. The proposed techniques do not benefit a
competitive character-aware model, but some of them improve the performance of
syllable- and morpheme-aware models while showing significant reductions in
model sizes. We discover a simple hands-on principle: in a multi-layer input
embedding model, layers should be tied consecutively bottom-up if reused at
output. Our best morpheme-aware model with properly reused weights beats the
competitive word-level model by a large margin across multiple languages and
has 20%-87% fewer parameters.
Learning a Discriminative Feature Network for Semantic Segmentation
Authors: Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang
Date: 2018-04-25 00:00:00
Proceeding: CVPR 2018 6
Tasks: Semantic Segmentation, Thermal Image Segmentation

Most existing methods of semantic segmentation still suffer from two aspects
of challenges: intra-class inconsistency and inter-class indistinction. To
tackle these two problems, we propose a Discriminative Feature Network (DFN),
which contains two sub-networks: Smooth Network and Border Network.
Specifically, to handle the intra-class inconsistency problem, we specially
design a Smooth Network with Channel Attention Block and global average pooling
to select the more discriminative features. Furthermore, we propose a Border
Network to make the bilateral features of boundary distinguishable with deep
semantic boundary supervision. Based on our proposed DFN, we achieve
state-of-the-art performance 86.2% mean IOU on PASCAL VOC 2012 and 80.3% mean
IOU on Cityscapes dataset.
Multi-focus Noisy Image Fusion using Low-Rank Representation
Authors: Hui Li, Xiao-Jun Wu, Tariq Durrani
Date: 2018-04-25 00:00:00
Tasks: Representation Learning

Multi-focus noisy image fusion represents an important task in the field of image fusion which generates a single, clear and focused image from all source images. In this paper, we propose a novel multi-focus noisy image fusion method based on low-rank representation (LRR) which is a powerful tool in representation learning. A multi-scale transform framework is adopted in which source images are decomposed into low frequency and high frequency coefficients, respectively. For low frequency coefficients, the fused low frequency coefficients are determined by a spatial frequency strategy, while the high frequency coefficients are fused by the LRR-based fusion strategy. Finally, the fused image is reconstructed by inverse multi-scale transforms with fused coefficients. Experimental results demonstrate that the proposed algorithm offers state-of-the-art performance even when the source images contain noise. The Code of our fusion method is available at https://github.com/hli1221/imagefusion_noisy_lrr