pwc-archive/papers-with-abstracts huggingface dataset:
Advancing Acoustic-to-Word CTC Model
Authors: Jinyu Li, Guoli Ye, Amit Das, Rui Zhao, Yifan Gong
Date: 2018-03-15 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, model

The acoustic-to-word model based on the connectionist temporal classification
(CTC) criterion was shown as a natural end-to-end (E2E) model directly
targeting words as output units. However, the word-based CTC model suffers from
the out-of-vocabulary (OOV) issue as it can only model limited number of words
in the output layer and maps all the remaining words into an OOV output node.
Hence, such a word-based CTC model can only recognize the frequent words
modeled by the network output nodes. Our first attempt to improve the
acoustic-to-word model is a hybrid CTC model which consults a letter-based CTC
when the word-based CTC model emits OOV tokens during testing time. Then, we
propose a much better solution by training a mixed-unit CTC model which
decomposes all the OOV words into sequences of frequent words and multi-letter
units. Evaluated on a 3400 hours Microsoft Cortana voice assistant task, the
final acoustic-to-word solution improves the baseline word-based CTC by
relative 12.09% word error rate (WER) reduction when combined with our proposed
attention CTC. Such an E2E model without using any language model (LM) or
complex decoder outperforms the traditional context-dependent phoneme CTC which
has strong LM and decoder by relative 6.79%.
Bidirectional Conditional Generative Adversarial Networks
Authors: Ayush Jaiswal, Wael Abd-Almageed, Yue Wu, Premkumar Natarajan
Date: 2017-11-20 00:00:00

Conditional Generative Adversarial Networks (cGANs) are generative models
that can produce data samples ($x$) conditioned on both latent variables ($z$)
and known auxiliary information ($c$). We propose the Bidirectional cGAN
(BiCoGAN), which effectively disentangles $z$ and $c$ in the generation process
and provides an encoder that learns inverse mappings from $x$ to both $z$ and
$c$, trained jointly with the generator and the discriminator. We present
crucial techniques for training BiCoGANs, which involve an extrinsic factor
loss along with an associated dynamically-tuned importance weight. As compared
to other encoder-based cGANs, BiCoGANs encode $c$ more accurately, and utilize
$z$ and $c$ more effectively and in a more disentangled way to generate
samples.
Advancing Connectionist Temporal Classification With Attention Modeling
Authors: Amit Das, Jinyu Li, Rui Zhao, Yifan Gong
Date: 2018-03-15 00:00:00
Tasks: Classification, General Classification, Language Modeling, Language Modelling, speech-recognition, Speech Recognition

In this study, we propose advancing all-neural speech recognition by directly
incorporating attention modeling within the Connectionist Temporal
Classification (CTC) framework. In particular, we derive new context vectors
using time convolution features to model attention as part of the CTC network.
To further improve attention modeling, we utilize content information extracted
from a network representing an implicit language model. Finally, we introduce
vector based attention weights that are applied on context vectors across both
time and their individual components. We evaluate our system on a 3400 hours
Microsoft Cortana voice assistant task and demonstrate that our proposed model
consistently outperforms the baseline model achieving about 20% relative
reduction in word error rates.
Operator Variational Inference
Authors: Rajesh Ranganath, Jaan Altosaar, Dustin Tran, David M. Blei
Date: 2016-10-27 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Bayesian Inference, Variational Inference

Variational inference is an umbrella term for algorithms which cast Bayesian
inference as optimization. Classically, variational inference uses the
Kullback-Leibler divergence to define the optimization. Though this divergence
has been widely used, the resultant posterior approximation can suffer from
undesirable statistical properties. To address this, we reexamine variational
inference from its roots as an optimization problem. We use operators, or
functions of functions, to design variational objectives. As one example, we
design a variational objective with a Langevin-Stein operator. We develop a
black box algorithm, operator variational inference (OPVI), for optimizing any
operator objective. Importantly, operators enable us to make explicit the
statistical and computational tradeoffs for variational inference. We can
characterize different properties of variational objectives, such as objectives
that admit data subsampling---allowing inference to scale to massive data---as
well as objectives that admit variational programs---a rich class of posterior
approximations that does not require a tractable density. We illustrate the
benefits of OPVI on a mixture model and a generative model of images.
Video Object Detection with an Aligned Spatial-Temporal Memory
Authors: Fanyi Xiao, Yong Jae Lee
Date: 2017-12-18 00:00:00
Proceeding: ECCV 2018 9
Tasks: Object, object-detection, Object Detection, Video Object Detection

We introduce Spatial-Temporal Memory Networks for video object detection. At
its core, a novel Spatial-Temporal Memory module (STMM) serves as the recurrent
computation unit to model long-term temporal appearance and motion dynamics.
The STMM's design enables full integration of pretrained backbone CNN weights,
which we find to be critical for accurate detection. Furthermore, in order to
tackle object motion in videos, we propose a novel MatchTrans module to align
the spatial-temporal memory from frame to frame. Our method produces
state-of-the-art results on the benchmark ImageNet VID dataset, and our
ablative studies clearly demonstrate the contribution of our different design
choices. We release our code and models at
http://fanyix.cs.ucdavis.edu/project/stmn/project.html.
Recurrent Segmentation for Variable Computational Budgets
Authors: Lane McIntosh, Niru Maheswaranathan, David Sussillo, Jonathon Shlens
Date: 2017-11-28 00:00:00
Tasks: Image Segmentation, Segmentation, Semantic Segmentation, Video Segmentation, Video Semantic Segmentation

State-of-the-art systems for semantic image segmentation use feed-forward
pipelines with fixed computational costs. Building an image segmentation system
that works across a range of computational budgets is challenging and
time-intensive as new architectures must be designed and trained for every
computational setting. To address this problem we develop a recurrent neural
network that successively improves prediction quality with each iteration.
Importantly, the RNN may be deployed across a range of computational budgets by
merely running the model for a variable number of iterations. We find that this
architecture is uniquely suited for efficiently segmenting videos. By
exploiting the segmentation of past frames, the RNN can perform video
segmentation at similar quality but reduced computational cost compared to
state-of-the-art image segmentation methods. When applied to static images in
the PASCAL VOC 2012 and Cityscapes segmentation datasets, the RNN traces out a
speed-accuracy curve that saturates near the performance of state-of-the-art
segmentation methods.
Object Detection in Video with Spatiotemporal Sampling Networks
Authors: Gedas Bertasius, Lorenzo Torresani, Jianbo Shi
Date: 2018-03-15 00:00:00
Proceeding: ECCV 2018 9
Tasks: Object, object-detection, Object Detection, Optical Flow Estimation, Video Object Detection

We propose a Spatiotemporal Sampling Network (STSN) that uses deformable
convolutions across time for object detection in videos. Our STSN performs
object detection in a video frame by learning to spatially sample features from
the adjacent frames. This naturally renders the approach robust to occlusion or
motion blur in individual frames. Our framework does not require additional
supervision, as it optimizes sampling locations directly with respect to object
detection performance. Our STSN outperforms the state-of-the-art on the
ImageNet VID dataset and compared to prior video object detection methods it
uses a simpler design, and does not require optical flow data for training.
A Simple and Effective Approach to the Story Cloze Test
Authors: Siddarth Srinivasan, Richa Arora, Mark Riedl
Date: 2018-03-15 00:00:00
Proceeding: NAACL 2018 6
Tasks: Cloze Test, Feature Engineering, Question Answering, Sentence

In the Story Cloze Test, a system is presented with a 4-sentence prompt to a
story, and must determine which one of two potential endings is the 'right'
ending to the story. Previous work has shown that ignoring the training set and
training a model on the validation set can achieve high accuracy on this task
due to stylistic differences between the story endings in the training set and
validation and test sets. Following this approach, we present a simpler
fully-neural approach to the Story Cloze Test using skip-thought embeddings of
the stories in a feed-forward network that achieves close to state-of-the-art
performance on this task without any feature engineering. We also find that
considering just the last sentence of the prompt instead of the whole prompt
yields higher accuracy with our approach.
Context-Aware Mixed Reality: A Framework for Ubiquitous Interaction
Authors: Long Chen, Wen Tang, Nigel John, Tao Ruan Wan, Jian Jun Zhang
Date: 2018-03-14 00:00:00
Tasks: Mixed Reality

Mixed Reality (MR) is a powerful interactive technology that yields new types
of user experience. We present a semantic based interactive MR framework that
exceeds the current geometry level approaches, a step change in generating
high-level context-aware interactions. Our key insight is to build semantic
understanding in MR that not only can greatly enhance user experience through
object-specific behaviours, but also pave the way for solving complex
interaction design challenges. The framework generates semantic properties of
the real world environment through dense scene reconstruction and deep image
understanding. We demonstrate our approach with a material-aware prototype
system for generating context-aware physical interactions between the real and
the virtual objects. Quantitative and qualitative evaluations are carried out
and the results show that the framework delivers accurate and fast semantic
information in interactive MR environment, providing effective semantic level
interactions.
Maintaining Natural Image Statistics with the Contextual Loss
Authors: Roey Mechrez, Itamar Talmi, Firas Shama, Lihi Zelnik-Manor
Date: 2018-03-13 00:00:00
Tasks: Image Super-Resolution, Super-Resolution, Surface Normal Estimation

Maintaining natural image statistics is a crucial factor in restoration and
generation of realistic looking images. When training CNNs, photorealism is
usually attempted by adversarial training (GAN), that pushes the output images
to lie on the manifold of natural images. GANs are very powerful, but not
perfect. They are hard to train and the results still often suffer from
artifacts. In this paper we propose a complementary approach, that could be
applied with or without GAN, whose goal is to train a feed-forward CNN to
maintain natural internal statistics. We look explicitly at the distribution of
features in an image and train the network to generate images with natural
feature distributions. Our approach reduces by orders of magnitude the number
of images required for training and achieves state-of-the-art results on both
single-image super-resolution, and high-resolution surface normal estimation.
Self-Supervised Monocular Image Depth Learning and Confidence Estimation
Authors: Long Chen, Wen Tang, Nigel John
Date: 2018-03-14 00:00:00
Tasks: Depth Estimation

Convolutional Neural Networks (CNNs) need large amounts of data with ground
truth annotation, which is a challenging problem that has limited the
development and fast deployment of CNNs for many computer vision tasks. We
propose a novel framework for depth estimation from monocular images with
corresponding confidence in a self-supervised manner. A fully differential
patch-based cost function is proposed by using the Zero-Mean Normalized Cross
Correlation (ZNCC) that takes multi-scale patches as a matching strategy. This
approach greatly increases the accuracy and robustness of the depth learning.
In addition, the proposed patch-based cost function can provide a 0 to 1
confidence, which is then used to supervise the training of a parallel network
for confidence map learning and estimation. Evaluation on KITTI dataset shows
that our method outperforms the state-of-the-art results.
Unpaired Image Captioning by Language Pivoting
Authors: Jiuxiang Gu, Shafiq Joty, Jianfei Cai, Gang Wang
Date: 2018-03-14 00:00:00
Proceeding: ECCV 2018 9
Tasks: Image Captioning, Sentence

Image captioning is a multimodal task involving computer vision and natural
language processing, where the goal is to learn a mapping from the image to its
natural language description. In general, the mapping function is learned from
a training set of image-caption pairs. However, for some language, large scale
image-caption paired corpus might not be available. We present an approach to
this unpaired image captioning problem by language pivoting. Our method can
effectively capture the characteristics of an image captioner from the pivot
language (Chinese) and align it to the target language (English) using another
pivot-target (Chinese-English) sentence parallel corpus. We evaluate our method
on two image-to-English benchmark datasets: MSCOCO and Flickr30K. Quantitative
comparisons against several baseline approaches demonstrate the effectiveness
of our method.
Neural Task Programming: Learning to Generalize Across Hierarchical Tasks
Authors: Danfei Xu, Suraj Nair, Yuke Zhu, Julian Gao, Animesh Garg, Li Fei-Fei, Silvio Savarese
Date: 2017-10-04 00:00:00
Tasks: Few-Shot Learning, Program induction, Robot Manipulation

In this work, we propose a novel robot learning framework called Neural Task
Programming (NTP), which bridges the idea of few-shot learning from
demonstration and neural program induction. NTP takes as input a task
specification (e.g., video demonstration of a task) and recursively decomposes
it into finer sub-task specifications. These specifications are fed to a
hierarchical neural program, where bottom-level programs are callable
subroutines that interact with the environment. We validate our method in three
robot manipulation tasks. NTP achieves strong generalization across sequential
tasks that exhibit hierarchal and compositional structures. The experimental
results show that NTP learns to generalize well to- wards unseen tasks with
increasing lengths, variable topologies, and changing objectives.
Emergent Complexity via Multi-Agent Competition
Authors: Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch
Date: 2017-10-10 00:00:00
Proceeding: ICLR 2018 1
Tasks: Blocking, Reinforcement Learning

Reinforcement learning algorithms can train agents that solve problems in
complex, interesting environments. Normally, the complexity of the trained
agent is closely related to the complexity of the environment. This suggests
that a highly capable agent requires a complex environment for training. In
this paper, we point out that a competitive multi-agent environment trained
with self-play can produce behaviors that are far more complex than the
environment itself. We also point out that such environments come with a
natural curriculum, because for any skill level, an environment full of agents
of this level will have the right level of difficulty. This work introduces
several competitive multi-agent environments where agents compete in a 3D world
with simulated physics. The trained agents learn a wide variety of complex and
interesting skills, even though the environment themselves are relatively
simple. The skills include behaviors such as running, blocking, ducking,
tackling, fooling opponents, kicking, and defending using both arms and legs. A
highlight of the learned behaviors can be found here: https://goo.gl/eR7fbX
Improving Video Generation for Multi-functional Applications
Authors: Bernhard Kratzwald, Zhiwu Huang, Danda Pani Paudel, Acharya Dinesh, Luc van Gool
Date: 2017-11-30 00:00:00
Tasks: Colorization, Future prediction, Video Generation, Video Inpainting

In this paper, we aim to improve the state-of-the-art video generative
adversarial networks (GANs) with a view towards multi-functional applications.
Our improved video GAN model does not separate foreground from background nor
dynamic from static patterns, but learns to generate the entire video clip
conjointly. Our model can thus be trained to generate - and learn from - a
broad set of videos with no restriction. This is achieved by designing a robust
one-stream video generation architecture with an extension of the
state-of-the-art Wasserstein GAN framework that allows for better convergence.
The experimental results show that our improved video GAN model outperforms
state-of-theart video generative models on multiple challenging datasets.
Furthermore, we demonstrate the superiority of our model by successfully
extending it to three challenging problems: video colorization, video
inpainting, and future prediction. To the best of our knowledge, this is the
first work using GANs to colorize and inpaint video clips.
Challenges in Discriminating Profanity from Hate Speech
Authors: Shervin Malmasi, Marcos Zampieri
Date: 2018-03-14 00:00:00
Tasks: Clustering, General Classification

In this study we approach the problem of distinguishing general profanity
from hate speech in social media, something which has not been widely
considered. Using a new dataset annotated specifically for this task, we employ
supervised classification along with a set of features that includes n-grams,
skip-grams and clustering-based word representations. We apply approaches based
on single classifiers as well as more advanced ensemble classifiers and stacked
generalization, achieving the best result of 80% accuracy for this 3-class
classification task. Analysis of the results reveals that discriminating hate
speech and profanity is not a simple task, which may require features that
capture a deeper understanding of the text not always possible with surface
n-grams. The variability of gold labels in the annotated data, due to
differences in the subjective adjudications of the annotators, is also an
issue. Other directions for future work are discussed.
A-NICE-MC: Adversarial Training for MCMC
Authors: Jiaming Song, Shengjia Zhao, Stefano Ermon
Date: 2017-06-23 00:00:00
Proceeding: NeurIPS 2017 12

Existing Markov Chain Monte Carlo (MCMC) methods are either based on
general-purpose and domain-agnostic schemes which can lead to slow convergence,
or hand-crafting of problem-specific proposals by an expert. We propose
A-NICE-MC, a novel method to train flexible parametric Markov chain kernels to
produce samples with desired properties. First, we propose an efficient
likelihood-free adversarial training method to train a Markov chain and mimic a
given data distribution. Then, we leverage flexible volume preserving flows to
obtain parametric kernels for MCMC. Using a bootstrap approach, we show how to
train efficient Markov chains to sample from a prescribed posterior
distribution by iteratively improving the quality of both the model and the
samples. A-NICE-MC provides the first framework to automatically design
efficient domain-specific MCMC proposals. Empirical results demonstrate that
A-NICE-MC combines the strong guarantees of MCMC with the expressiveness of
deep neural networks, and is able to significantly outperform competing methods
such as Hamiltonian Monte Carlo.
Driving Digital Rock towards Machine Learning: predicting permeability with Gradient Boosting and Deep Neural Networks
Authors: Oleg Sudakov, Evgeny Burnaev, Dmitry Koroteev
Date: 2018-03-02 00:00:00
Tasks: BIG-bench Machine Learning, Prediction

We present a research study aimed at testing of applicability of machine
learning techniques for prediction of permeability of digitized rock samples.
We prepare a training set containing 3D images of sandstone samples imaged with
X-ray microtomography and corresponding permeability values simulated with Pore
Network approach. We also use Minkowski functionals and Deep Learning-based
descriptors of 3D images and 2D slices as input features for predictive model
training and prediction. We compare predictive power of various feature sets
and methods. The later include Gradient Boosting and various architectures of
Deep Neural Networks (DNN). The results demonstrate applicability of machine
learning for image-based permeability prediction and open a new area of Digital
Rock research.
Targeted change detection in remote sensing images
Authors: Vladimir Ignatiev, Alexey Trekin, Viktor Lobachev, Georgy Potapov, Evgeny Burnaev
Date: 2018-03-14 00:00:00
Tasks: Change Detection, Deep Learning, General Classification

Recent developments in the remote sensing systems and image processing made
it possible to propose a new method of the object classification and detection
of the specific changes in the series of satellite Earth images (so called
targeted change detection). In this paper we propose a formal problem statement
that allows to use effectively the deep learning approach to analyze
time-dependent series of remote sensing images. We also introduce a new
framework for the development of deep learning models for targeted change
detection and demonstrate some cases of business applications it can be used
for.
SUSTain: Scalable Unsupervised Scoring for Tensors and its Application to Phenotyping
Authors: Ioakeim Perros, Evangelos E. Papalexakis, Haesun Park, Richard Vuduc, Xiaowei Yan, Christopher deFilippi, Walter F. Stewart, Jimeng Sun
Date: 2018-03-14 00:00:00
Tasks: Feature Importance

This paper presents a new method, which we call SUSTain, that extends
real-valued matrix and tensor factorizations to data where values are integers.
Such data are common when the values correspond to event counts or ordinal
measures. The conventional approach is to treat integer data as real, and then
apply real-valued factorizations. However, doing so fails to preserve important
characteristics of the original data, thereby making it hard to interpret the
results. Instead, our approach extracts factor values from integer datasets as
scores that are constrained to take values from a small integer set. These
scores are easy to interpret: a score of zero indicates no feature contribution
and higher scores indicate distinct levels of feature importance.
  At its core, SUSTain relies on: a) a problem partitioning into
integer-constrained subproblems, so that they can be optimally solved in an
efficient manner; and b) organizing the order of the subproblems' solution, to
promote reuse of shared intermediate results. We propose two variants,
SUSTain_M and SUSTain_T, to handle both matrix and tensor inputs, respectively.
We evaluate SUSTain against several state-of-the-art baselines on both
synthetic and real Electronic Health Record (EHR) datasets. Comparing to those
baselines, SUSTain shows either significantly better fit or orders of magnitude
speedups that achieve a comparable fit (up to 425X faster). We apply SUSTain to
EHR datasets to extract patient phenotypes (i.e., clinically meaningful patient
clusters). Furthermore, 87% of them were validated as clinically meaningful
phenotypes related to heart failure by a cardiologist.