pwc-archive/papers-with-abstracts huggingface dataset:
Data-Dependent Stability of Stochastic Gradient Descent
Authors: Ilja Kuzborskij, Christoph H. Lampert
Date: 2017-03-05 00:00:00
Proceeding: ICML 2018 7
Tasks: Generalization Bounds

We establish a data-dependent notion of algorithmic stability for Stochastic
Gradient Descent (SGD), and employ it to develop novel generalization bounds.
This is in contrast to previous distribution-free algorithmic stability results
for SGD which depend on the worst-case constants. By virtue of the
data-dependent argument, our bounds provide new insights into learning with SGD
on convex and non-convex problems. In the convex case, we show that the bound
on the generalization error depends on the risk at the initialization point. In
the non-convex case, we prove that the expected curvature of the objective
function around the initialization point has crucial influence on the
generalization error. In both cases, our results suggest a simple data-driven
strategy to stabilize SGD by pre-screening its initialization. As a corollary,
our results allow us to show optimistic generalization bounds that exhibit fast
convergence rates for SGD subject to a vanishing empirical risk and low noise
of stochastic gradient.
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Authors: Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, Marcus Rohrbach
Date: 2018-02-15 00:00:00
Proceeding: CVPR 2018 6
Tasks: Activity Recognition, Explainable Models, Question Answering, Visual Question Answering, Visual Question Answering (VQA)

Deep models that are both effective and explainable are desirable in many
settings; prior explainable models have been unimodal, offering either
image-based visualization of attention weights or text-based generation of
post-hoc justifications. We propose a multimodal approach to explanation, and
argue that the two modalities provide complementary explanatory strengths. We
collect two new datasets to define and evaluate this task, and propose a novel
model which can provide joint textual rationale generation and attention
visualization. Our datasets define visual and textual justifications of a
classification decision for activity recognition tasks (ACT-X) and for visual
question answering tasks (VQA-X). We quantitatively show that training with the
textual explanations not only yields better textual justification models, but
also better localizes the evidence that supports the decision. We also
qualitatively show cases where visual explanation is more insightful than
textual explanation, and vice versa, supporting our thesis that multimodal
explanation models offer significant benefits over unimodal approaches.
A Study of All-Convolutional Encoders for Connectionist Temporal Classification
Authors: Kalpesh Krishna, Liang Lu, Kevin Gimpel, Karen Livescu
Date: 2017-10-28 00:00:00
Tasks: All, Automatic Speech Recognition, Automatic Speech Recognition (ASR), Classification, General Classification, speech-recognition, Speech Recognition

Connectionist temporal classification (CTC) is a popular sequence prediction
approach for automatic speech recognition that is typically used with models
based on recurrent neural networks (RNNs). We explore whether deep
convolutional neural networks (CNNs) can be used effectively instead of RNNs as
the "encoder" in CTC. CNNs lack an explicit representation of the entire
sequence, but have the advantage that they are much faster to train. We present
an exploration of CNNs as encoders for CTC models, in the context of
character-based (lexicon-free) automatic speech recognition. In particular, we
explore a range of one-dimensional convolutional layers, which are particularly
efficient. We compare the performance of our CNN-based models against typical
RNNbased models in terms of training time, decoding time, model size and word
error rate (WER) on the Switchboard Eval2000 corpus. We find that our CNN-based
models are close in performance to LSTMs, while not matching them, and are much
faster to train and decode.
Inverting The Generator Of A Generative Adversarial Network (II)
Authors: Antonia Creswell, Anil A. Bharath
Date: 2018-02-15 00:00:00
Tasks: Generative Adversarial Network, Retrieval

Generative adversarial networks (GANs) learn a deep generative model that is
able to synthesise novel, high-dimensional data samples. New data samples are
synthesised by passing latent samples, drawn from a chosen prior distribution,
through the generative model. Once trained, the latent space exhibits
interesting properties, that may be useful for down stream tasks such as
classification or retrieval. Unfortunately, GANs do not offer an "inverse
model", a mapping from data space back to latent space, making it difficult to
infer a latent representation for a given data sample. In this paper, we
introduce a technique, inversion, to project data samples, specifically images,
to the latent space using a pre-trained GAN. Using our proposed inversion
technique, we are able to identify which attributes of a dataset a trained GAN
is able to model and quantify GAN performance, based on a reconstruction loss.
We demonstrate how our proposed inversion technique may be used to
quantitatively compare performance of various GAN models trained on three image
datasets. We provide code for all of our experiments,
https://github.com/ToniCreswell/InvertingGAN.
A Machine Learning Approach for Virtual Flow Metering and Forecasting
Authors: Nikolai Andrianov
Date: 2018-02-15 00:00:00
Tasks: BIG-bench Machine Learning

We are concerned with robust and accurate forecasting of multiphase flow
rates in wells and pipelines during oil and gas production. In practice, the
possibility to physically measure the rates is often limited; besides, it is
desirable to estimate future values of multiphase rates based on the previous
behavior of the system. In this work, we demonstrate that a Long Short-Term
Memory (LSTM) recurrent artificial network is able not only to accurately
estimate the multiphase rates at current time (i.e., act as a virtual flow
meter), but also to forecast the rates for a sequence of future time instants.
For a synthetic severe slugging case, LSTM forecasts compare favorably with the
results of hydrodynamical modeling. LSTM results for a realistic noizy dataset
of a variable rate well test show that the model can also successfully forecast
multiphase rates for a system with changing flow patterns.
Multinomial Adversarial Networks for Multi-Domain Text Classification
Authors: Xilun Chen, Claire Cardie
Date: 2018-02-15 00:00:00
Proceeding: NAACL 2018 6
Tasks: Classification, Cross-Domain Text Classification, Domain Adaptation, General Classification, text-classification, Text Classification, Unsupervised Domain Adaptation

Many text classification tasks are known to be highly domain-dependent.
Unfortunately, the availability of training data can vary drastically across
domains. Worse still, for some domains there may not be any annotated data at
all. In this work, we propose a multinomial adversarial network (MAN) to tackle
the text classification problem in this real-world multidomain setting (MDTC).
We provide theoretical justifications for the MAN framework, proving that
different instances of MANs are essentially minimizers of various f-divergence
metrics (Ali and Silvey, 1966) among multiple probability distributions. MANs
are thus a theoretically sound generalization of traditional adversarial
networks that discriminate over two distributions. More specifically, for the
MDTC task, MAN learns features that are invariant across multiple domains by
resorting to its ability to reduce the divergence among the feature
distributions of each domain. We present experimental results showing that MANs
significantly outperform the prior art on the MDTC task. We also show that MANs
achieve state-of-the-art performance for domains with no labeled data.
A Bio-inspired Redundant Sensing Architecture
Authors: Anh Tuan Nguyen, Jian Xu, Zhi Yang
Date: 2018-02-15 00:00:00
Proceeding: NeurIPS 2016 12

Sensing is the process of deriving signals from the environment that allows
artificial systems to interact with the physical world. The Shannon theorem
specifies the maximum rate at which information can be acquired. However, this
upper bound is hard to achieve in many man-made systems. The biological visual
systems, on the other hand, have highly efficient signal representation and
processing mechanisms that allow precise sensing. In this work, we argue that
redundancy is one of the critical characteristics for such superior
performance. We show architectural advantages by utilizing redundant sensing,
including correction of mismatch error and significant precision enhancement.
For a proof-of-concept demonstration, we have designed a heuristic-based
analog-to-digital converter - a zero-dimensional quantizer. Through Monte Carlo
simulation with the error probabilistic distribution as a priori, the
performance approaching the Shannon limit is feasible. In actual measurements
without knowing the error distribution, we observe at least 2-bit extra
precision. The results may also help explain biological processes including the
dominance of binocular vision, the functional roles of the fixational eye
movements, and the structural mechanisms allowing hyperacuity.
Event Nugget Detection with Forward-Backward Recurrent Neural Networks
Authors: Reza Ghaeini, Xiaoli Z. Fern, Liang Huang, Prasad Tadepalli
Date: 2018-02-15 00:00:00
Proceeding: ACL 2016 8
Tasks: Event Detection, Feature Engineering

Traditional event detection methods heavily rely on manually engineered rich
features. Recent deep learning approaches alleviate this problem by automatic
feature engineering. But such efforts, like tradition methods, have so far only
focused on single-token event mentions, whereas in practice events can also be
a phrase. We instead use forward-backward recurrent neural networks (FBRNNs) to
detect events that can be either words or phrases. To the best our knowledge,
this is one of the first efforts to handle multi-word events and also the first
attempt to use RNNs for event detection. Experimental results demonstrate that
FBRNN is competitive with the state-of-the-art methods on the ACE 2005 and the
Rich ERE 2015 event detection tasks.
Model compression via distillation and quantization
Authors: Antonio Polino, Razvan Pascanu, Dan Alistarh
Date: 2018-02-15 00:00:00
Proceeding: ICLR 2018 1
Tasks: image-classification, model, Model Compression, Quantization, Reinforcement Learning

Deep neural networks (DNNs) continue to make significant advances, solving
tasks from image classification to translation or reinforcement learning. One
aspect of the field receiving considerable attention is efficiently executing
deep models in resource-constrained environments, such as mobile or embedded
devices. This paper focuses on this problem, and proposes two new compression
methods, which jointly leverage weight quantization and distillation of larger
teacher networks into smaller student networks. The first method we propose is
called quantized distillation and leverages distillation during the training
process, by incorporating distillation loss, expressed with respect to the
teacher, into the training of a student network whose weights are quantized to
a limited set of levels. The second method, differentiable quantization,
optimizes the location of quantization points through stochastic gradient
descent, to better fit the behavior of the teacher model. We validate both
methods through experiments on convolutional and recurrent architectures. We
show that quantized shallow students can reach similar accuracy levels to
full-precision teacher models, while providing order of magnitude compression,
and inference speedup that is linear in the depth reduction. In sum, our
results enable DNNs for resource-constrained environments to leverage
architecture and accuracy advances developed on more powerful devices.
DeepMatch: Balancing Deep Covariate Representations for Causal Inference Using Adversarial Training
Authors: Nathan Kallus
Date: 2018-02-15 00:00:00
Proceeding: ICML 2020 1
Tasks: Causal Inference

We study optimal covariate balance for causal inferences from observational
data when rich covariates and complex relationships necessitate flexible
modeling with neural networks. Standard approaches such as propensity weighting
and matching/balancing fail in such settings due to miscalibrated propensity
nets and inappropriate covariate representations, respectively. We propose a
new method based on adversarial training of a weighting and a discriminator
network that effectively addresses this methodological gap. This is
demonstrated through new theoretical characterizations of the method as well as
empirical results using both fully connected architectures to learn complex
relationships and convolutional architectures to handle image confounders,
showing how this new method can enable strong causal analyses in these
challenging settings.
On Quadratic Convergence of DC Proximal Newton Algorithm for Nonconvex Sparse Learning in High Dimensions
Authors: Xingguo Li, Lin F. Yang, Jason Ge, Jarvis Haupt, Tong Zhang, Tuo Zhao
Date: 2017-06-19 00:00:00
Tasks: Sparse Learning

We propose a DC proximal Newton algorithm for solving nonconvex regularized
sparse learning problems in high dimensions. Our proposed algorithm integrates
the proximal Newton algorithm with multi-stage convex relaxation based on the
difference of convex (DC) programming, and enjoys both strong computational and
statistical guarantees. Specifically, by leveraging a sophisticated
characterization of sparse modeling structures/assumptions (i.e., local
restricted strong convexity and Hessian smoothness), we prove that within each
stage of convex relaxation, our proposed algorithm achieves (local) quadratic
convergence, and eventually obtains a sparse approximate local optimum with
optimal statistical properties after only a few convex relaxations. Numerical
experiments are provided to support our theory.
Reliable Uncertain Evidence Modeling in Bayesian Networks by Credal Networks
Authors: Sabina Marchetti, Alessandro Antonucci
Date: 2018-02-15 00:00:00

A reliable modeling of uncertain evidence in Bayesian networks based on a
set-valued quantification is proposed. Both soft and virtual evidences are
considered. We show that evidence propagation in this setup can be reduced to
standard updating in an augmented credal network, equivalent to a set of
consistent Bayesian networks. A characterization of the computational
complexity for this task is derived together with an efficient exact procedure
for a subclass of instances. In the case of multiple uncertain evidences over
the same variable, the proposed procedure can provide a set-valued version of
the geometric approach to opinion pooling.
cGANs with Projection Discriminator
Authors: Takeru Miyato, Masanori Koyama
Date: 2018-02-15 00:00:00
Proceeding: ICLR 2018 1
Tasks: Conditional Image Generation, Image Generation, Super-Resolution

We propose a novel, projection based way to incorporate the conditional
information into the discriminator of GANs that respects the role of the
conditional information in the underlining probabilistic model. This approach
is in contrast with most frameworks of conditional GANs used in application
today, which use the conditional information by concatenating the (embedded)
conditional vector to the feature vectors. With this modification, we were able
to significantly improve the quality of the class conditional image generation
on ILSVRC2012 (ImageNet) 1000-class image dataset from the current
state-of-the-art result, and we achieved this with a single pair of a
discriminator and a generator. We were also able to extend the application to
super-resolution and succeeded in producing highly discriminative
super-resolution images. This new structure also enabled high quality category
transformation based on parametric functional transformation of conditional
batch normalization layers in the generator.
A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization
Authors: Tianyi Liu, Zhehui Chen, Enlu Zhou, Tuo Zhao
Date: 2018-02-14 00:00:00
Tasks: Bayesian Inference, Dimensionality Reduction, Stochastic Optimization

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points, but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.
CNN+LSTM Architecture for Speech Emotion Recognition with Data Augmentation
Authors: Caroline Etienne, Guillaume Fidanza, Andrei Petrovskii, Laurence Devillers, Benoit Schmauch
Date: 2018-02-15 00:00:00
Tasks: Data Augmentation, Emotion Recognition, Speech Emotion Recognition

In this work we design a neural network for recognizing emotions in speech,
using the IEMOCAP dataset. Following the latest advances in audio analysis, we
use an architecture involving both convolutional layers, for extracting
high-level features from raw spectrograms, and recurrent ones for aggregating
long-term dependencies. We examine the techniques of data augmentation with
vocal track length perturbation, layer-wise optimizer adjustment, batch
normalization of recurrent layers and obtain highly competitive results of
64.5% for weighted accuracy and 61.7% for unweighted accuracy on four emotions.
Noisy Networks for Exploration
Authors: Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, Shane Legg
Date: 2017-06-30 00:00:00
Proceeding: ICLR 2018 1
Tasks: Atari Games, Deep Reinforcement Learning, Efficient Exploration, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We introduce NoisyNet, a deep reinforcement learning agent with parametric noise added to its weights, and show that the induced stochasticity of the agent's policy can be used to aid efficient exploration. The parameters of the noise are learned with gradient descent along with the remaining network weights. NoisyNet is straightforward to implement and adds little computational overhead. We find that replacing the conventional exploration heuristics for A3C, DQN and dueling agents (entropy reward and $\epsilon$-greedy respectively) with NoisyNet yields substantially higher scores for a wide range of Atari games, in some cases advancing the agent from sub to super-human performance.
Conditioning of three-dimensional generative adversarial networks for pore and reservoir-scale models
Authors: Lukas Mosser, Olivier Dubrule, Martin J. Blunt
Date: 2018-02-15 00:00:00

Geostatistical modeling of petrophysical properties is a key step in modern
integrated oil and gas reservoir studies. Recently, generative adversarial
networks (GAN) have been shown to be a successful method for generating
unconditional simulations of pore- and reservoir-scale models. This
contribution leverages the differentiable nature of neural networks to extend
GANs to the conditional simulation of three-dimensional pore- and
reservoir-scale models. Based on the previous work of Yeh et al. (2016), we use
a content loss to constrain to the conditioning data and a perceptual loss
obtained from the evaluation of the GAN discriminator network. The technique is
tested on the generation of three-dimensional micro-CT images of a Ketton
limestone constrained by two-dimensional cross-sections, and on the simulation
of the Maules Creek alluvial aquifer constrained by one-dimensional sections.
Our results show that GANs represent a powerful method for sampling conditioned
pore and reservoir samples for stochastic reservoir evaluation workflows.
Unsupervised Image-to-Image Translation Networks
Authors: Ming-Yu Liu, Thomas Breuel, Jan Kautz
Date: 2017-03-02 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Domain Adaptation, Image-to-Image Translation, Multimodal Unsupervised Image-To-Image Translation, Translation, Unsupervised Image-To-Image Translation

Unsupervised image-to-image translation aims at learning a joint distribution
of images in different domains by using images from the marginal distributions
in individual domains. Since there exists an infinite set of joint
distributions that can arrive the given marginal distributions, one could infer
nothing about the joint distribution from the marginal distributions without
additional assumptions. To address the problem, we make a shared-latent space
assumption and propose an unsupervised image-to-image translation framework
based on Coupled GANs. We compare the proposed framework with competing
approaches and present high quality image translation results on various
challenging unsupervised image translation tasks, including street scene image
translation, animal image translation, and face image translation. We also
apply the proposed framework to domain adaptation and achieve state-of-the-art
performance on benchmark datasets. Code and additional results are available in
https://github.com/mingyuliutw/unit .
Prioritized Sweeping Neural DynaQ with Multiple Predecessors, and Hippocampal Replays
Authors: Lise Aubin, Mehdi Khamassi, Benoît Girard
Date: 2018-02-15 00:00:00
Tasks: Hippocampus, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

During sleep and awake rest, the hippocampus replays sequences of place cells
that have been activated during prior experiences. These have been interpreted
as a memory consolidation process, but recent results suggest a possible
interpretation in terms of reinforcement learning. The Dyna reinforcement
learning algorithms use off-line replays to improve learning. Under limited
replay budget, a prioritized sweeping approach, which requires a model of the
transitions to the predecessors, can be used to improve performance. We
investigate whether such algorithms can explain the experimentally observed
replays. We propose a neural network version of prioritized sweeping
Q-learning, for which we developed a growing multiple expert algorithm, able to
cope with multiple predecessors. The resulting architecture is able to improve
the learning of simulated agents confronted to a navigation task. We predict
that, in animals, learning the world model should occur during rest periods,
and that the corresponding replays should be shuffled.