pwc-archive/papers-with-abstracts huggingface dataset:
Relational recurrent neural networks
Authors: Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, Timothy Lillicrap
Date: 2018-06-05 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Language Modeling, Language Modelling, Relational Reasoning

Memory-based neural networks model temporal data by leveraging an ability to
remember information for long periods. It is unclear, however, whether they
also have an ability to perform complex relational reasoning with the
information they remember. Here, we first confirm our intuitions that standard
memory architectures may struggle at tasks that heavily involve an
understanding of the ways in which entities are connected -- i.e., tasks
involving relational reasoning. We then improve upon these deficits by using a
new memory module -- a \textit{Relational Memory Core} (RMC) -- which employs
multi-head dot product attention to allow memories to interact. Finally, we
test the RMC on a suite of tasks that may profit from more capable relational
reasoning across sequential information, and show large gains in RL domains
(e.g. Mini PacMan), program evaluation, and language modeling, achieving
state-of-the-art results on the WikiText-103, Project Gutenberg, and GigaWord
datasets.
Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks
Authors: Yarin Gal, Lewis Smith
Date: 2018-06-02 00:00:00
Tasks: image-classification, Image Classification

We prove, under two sufficient conditions, that idealised models can have no
adversarial examples. We discuss which idealised models satisfy our conditions,
and show that idealised Bayesian neural networks (BNNs) satisfy these. We
continue by studying near-idealised BNNs using HMC inference, demonstrating the
theoretical ideas in practice. We experiment with HMC on synthetic data derived
from MNIST for which we know the ground-truth image density, showing that
near-perfect epistemic uncertainty correlates to density under image manifold,
and that adversarial images lie off the manifold in our setting. This suggests
why MC dropout, which can be seen as performing approximate inference, has been
observed to be an effective defence against adversarial examples in practice;
We highlight failure-cases of non-idealised BNNs relying on dropout, suggesting
a new attack for dropout models and a new defence as well. Lastly, we
demonstrate the defence on a cats-vs-dogs image classification task with a
VGG13 variant.
LSTM Benchmarks for Deep Learning Frameworks
Authors: Stefan Braun
Date: 2018-06-05 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Deep Learning, speech-recognition, Speech Recognition

This study provides benchmarks for different implementations of LSTM units
between the deep learning frameworks PyTorch, TensorFlow, Lasagne and Keras.
The comparison includes cuDNN LSTMs, fused LSTM variants and less optimized,
but more flexible LSTM implementations. The benchmarks reflect two typical
scenarios for automatic speech recognition, notably continuous speech
recognition and isolated digit recognition. These scenarios cover input
sequences of fixed and variable length as well as the loss functions CTC and
cross entropy. Additionally, a comparison between four different PyTorch
versions is included. The code is available online
https://github.com/stefbraun/rnn_benchmarks.
Perturbative Neural Networks
Authors: Felix Juefei-Xu, Vishnu Naresh Boddeti, Marios Savvides
Date: 2018-06-05 00:00:00
Proceeding: CVPR 2018 6

Convolutional neural networks are witnessing wide adoption in computer vision
systems with numerous applications across a range of visual recognition tasks.
Much of this progress is fueled through advances in convolutional neural
network architectures and learning algorithms even as the basic premise of a
convolutional layer has remained unchanged. In this paper, we seek to revisit
the convolutional layer that has been the workhorse of state-of-the-art visual
recognition models. We introduce a very simple, yet effective, module called a
perturbation layer as an alternative to a convolutional layer. The perturbation
layer does away with convolution in the traditional sense and instead computes
its response as a weighted linear combination of non-linearly activated
additive noise perturbed inputs. We demonstrate both analytically and
empirically that this perturbation layer can be an effective replacement for a
standard convolutional layer. Empirically, deep neural networks with
perturbation layers, called Perturbative Neural Networks (PNNs), in lieu of
convolutional layers perform comparably with standard CNNs on a range of visual
datasets (MNIST, CIFAR-10, PASCAL VOC, and ImageNet) with fewer parameters.
A Review of Challenges and Opportunities in Machine Learning for Health
Authors: Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L. Beam, Irene Y. Chen, Rajesh Ranganath
Date: 2018-06-01 00:00:00
Tasks: BIG-bench Machine Learning

Modern electronic health records (EHRs) provide data to answer clinically meaningful questions. The growing data in EHRs makes healthcare ripe for the use of machine learning. However, learning in a clinical setting presents unique challenges that complicate the use of common machine learning methodologies. For example, diseases in EHRs are poorly labeled, conditions can encompass multiple underlying endotypes, and healthy individuals are underrepresented. This article serves as a primer to illuminate these challenges and highlights opportunities for members of the machine learning community to contribute to healthcare.
Videos as Space-Time Region Graphs
Authors: Xiaolong Wang, Abhinav Gupta
Date: 2018-06-05 00:00:00
Proceeding: ECCV 2018 9
Tasks: Action Classification, Action Recognition

How do humans recognize the action "opening a book" ? We argue that there are
two important cues: modeling temporal shape dynamics and modeling functional
relationships between humans and objects. In this paper, we propose to
represent videos as space-time region graphs which capture these two important
cues. Our graph nodes are defined by the object region proposals from different
frames in a long range video. These nodes are connected by two types of
relations: (i) similarity relations capturing the long range dependencies
between correlated objects and (ii) spatial-temporal relations capturing the
interactions between nearby objects. We perform reasoning on this graph
representation via Graph Convolutional Networks. We achieve state-of-the-art
results on both Charades and Something-Something datasets. Especially for
Charades, we obtain a huge 4.4% gain when our model is applied in complex
environments.
Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
Authors: Mor Shpigel Nacson, Nathan Srebro, Daniel Soudry
Date: 2018-06-05 00:00:00

Stochastic Gradient Descent (SGD) is a central tool in machine learning. We prove that SGD converges to zero loss, even with a fixed (non-vanishing) learning rate - in the special case of homogeneous linear classifiers with smooth monotone loss functions, optimized on linearly separable data. Previous works assumed either a vanishing learning rate, iterate averaging, or loss assumptions that do not hold for monotone loss functions used for classification, such as the logistic loss. We prove our result on a fixed dataset, both for sampling with or without replacement. Furthermore, for logistic loss (and similar exponentially-tailed losses), we prove that with SGD the weight vector converges in direction to the $L_2$ max margin vector as $O(1/\log(t))$ for almost all separable datasets, and the loss converges as $O(1/t)$ - similarly to gradient descent. Lastly, we examine the case of a fixed learning rate proportional to the minibatch size. We prove that in this case, the asymptotic convergence rate of SGD (with replacement) does not depend on the minibatch size in terms of epochs, if the support vectors span the data. These results may suggest an explanation to similar behaviors observed in deep networks, when trained with SGD.
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
Authors: Adam R. Kosiorek, Hyunjik Kim, Ingmar Posner, Yee Whye Teh
Date: 2018-06-05 00:00:00
Proceeding: NeurIPS 2018 12

We present Sequential Attend, Infer, Repeat (SQAIR), an interpretable deep
generative model for videos of moving objects. It can reliably discover and
track objects throughout the sequence of frames, and can also generate future
frames conditioning on the current frame, thereby simulating expected motion of
objects. This is achieved by explicitly encoding object presence, locations and
appearances in the latent variables of the model. SQAIR retains all strengths
of its predecessor, Attend, Infer, Repeat (AIR, Eslami et. al., 2016),
including learning in an unsupervised manner, and addresses its shortcomings.
We use a moving multi-MNIST dataset to show limitations of AIR in detecting
overlapping or partially occluded objects, and show how SQAIR overcomes them by
leveraging temporal consistency of objects. Finally, we also apply SQAIR to
real-world pedestrian CCTV data, where it learns to reliably detect, track and
generate walking pedestrians with no supervision.
Machines hear better when they have ears
Authors: Deepak Baby, Sarah Verhulst
Date: 2018-06-01 00:00:00

Deep-neural-network (DNN) based noise suppression systems yield significant
improvements over conventional approaches such as spectral subtraction and
non-negative matrix factorization, but do not generalize well to noise
conditions they were not trained for. In comparison to DNNs, humans show
remarkable noise suppression capabilities that yield successful speech
intelligibility under various adverse listening conditions and negative
signal-to-noise ratios (SNRs). Motivated by the excellent human performance,
this paper explores whether numerical models that simulate human cochlear
signal processing can be combined with DNNs to improve the robustness of DNN
based noise suppression systems. Five cochlear models were coupled to
fully-connected and recurrent NN-based noise suppression systems and were
trained and evaluated for a variety of noise conditions using objective
metrics: perceptual speech quality (PESQ), segmental SNR and cepstral distance.
The simulations show that biophysically-inspired cochlear models improve the
generalizability of DNN-based noise suppression systems for unseen noise and
negative SNRs. This approach thus leads to robust noise suppression systems
that are less sensitive to the noise type and noise level. Because cochlear
models capture the intrinsic nonlinearities and dynamics of peripheral auditory
processing, it is shown here that accounting for their deterministic signal
processing improves machine hearing and avoids overtraining of multi-layer
DNNs. We hence conclude that machines hear better when realistic cochlear
models are used at the input of DNNs.
Mix&Match - Agent Curricula for Reinforcement Learning
Authors: Wojciech Marian Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Simon Osindero, Nicolas Heess, Razvan Pascanu
Date: 2018-06-05 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We introduce Mix&Match (M&M) - a training framework designed to facilitate
rapid and effective learning in RL agents, especially those that would be too
slow or too challenging to train otherwise. The key innovation is a procedure
that allows us to automatically form a curriculum over agents. Through such a
curriculum we can progressively train more complex agents by, effectively,
bootstrapping from solutions found by simpler agents. In contradistinction to
typical curriculum learning approaches, we do not gradually modify the tasks or
environments presented, but instead use a process to gradually alter how the
policy is represented internally. We show the broad applicability of our method
by demonstrating significant performance gains in three different experimental
setups: (1) We train an agent able to control more than 700 actions in a
challenging 3D first-person task; using our method to progress through an
action-space curriculum we achieve both faster training and better final
performance than one obtains using traditional methods. (2) We further show
that M&M can be used successfully to progress through a curriculum of
architectural variants defining an agents internal state. (3) Finally, we
illustrate how a variant of our method can be used to improve agent performance
in a multitask setting.
Contextual Slot Carryover for Disparate Schemas
Authors: Chetan Naik, Arpit Gupta, Hancheng Ge, Lambert Mathias, Ruhi Sarikaya
Date: 2018-06-05 00:00:00
Tasks: slot-filling, Slot Filling

In the slot-filling paradigm, where a user can refer back to slots in the
context during a conversation, the goal of the contextual understanding system
is to resolve the referring expressions to the appropriate slots in the
context. In large-scale multi-domain systems, this presents two challenges -
scaling to a very large and potentially unbounded set of slot values, and
dealing with diverse schemas. We present a neural network architecture that
addresses the slot value scalability challenge by reformulating the contextual
interpretation as a decision to carryover a slot from a set of possible
candidates. To deal with heterogenous schemas, we introduce a simple
data-driven method for trans- forming the candidate slots. Our experiments show
that our approach can scale to multiple domains and provides competitive
results over a strong baseline.
Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
Authors: Louis C. Tiao, Edwin V. Bonilla, Fabio Ramos
Date: 2018-06-05 00:00:00
Tasks: Bayesian Inference, Variational Inference

We formalize the problem of learning interdomain correspondences in the
absence of paired data as Bayesian inference in a latent variable model (LVM),
where one seeks the underlying hidden representations of entities from one
domain as entities from the other domain. First, we introduce implicit latent
variable models, where the prior over hidden representations can be specified
flexibly as an implicit distribution. Next, we develop a new variational
inference (VI) algorithm for this model based on minimization of the symmetric
Kullback-Leibler (KL) divergence between a variational joint and the exact
joint distribution. Lastly, we demonstrate that the state-of-the-art
cycle-consistent adversarial learning (CYCLEGAN) models can be derived as a
special case within our proposed VI framework, thus establishing its connection
to approximate Bayesian inference methods.
How Does Batch Normalization Help Optimization?
Authors: Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry
Date: 2018-05-29 00:00:00
Proceeding: NeurIPS 2018 12

Batch Normalization (BatchNorm) is a widely adopted technique that enables
faster and more stable training of deep neural networks (DNNs). Despite its
pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly
understood. The popular belief is that this effectiveness stems from
controlling the change of the layers' input distributions during training to
reduce the so-called "internal covariate shift". In this work, we demonstrate
that such distributional stability of layer inputs has little to do with the
success of BatchNorm. Instead, we uncover a more fundamental impact of
BatchNorm on the training process: it makes the optimization landscape
significantly smoother. This smoothness induces a more predictive and stable
behavior of the gradients, allowing for faster training.
Graph Saliency Maps through Spectral Convolutional Networks: Application to Sex Classification with Brain Connectivity
Authors: Salim Arslan, Sofia Ira Ktena, Ben Glocker, Daniel Rueckert
Date: 2018-06-05 00:00:00
Tasks: General Classification

Graph convolutional networks (GCNs) allow to apply traditional convolution
operations in non-Euclidean domains, where data are commonly modelled as
irregular graphs. Medical imaging and, in particular, neuroscience studies
often rely on such graph representations, with brain connectivity networks
being a characteristic example, while ultimately seeking the locus of
phenotypic or disease-related differences in the brain. These regions of
interest (ROIs) are, then, considered to be closely associated with function
and/or behaviour. Driven by this, we explore GCNs for the task of ROI
identification and propose a visual attribution method based on class
activation mapping. By undertaking a sex classification task as proof of
concept, we show that this method can be used to identify salient nodes (brain
regions) without prior node labels. Based on experiments conducted on
neuroimaging data of more than 5000 participants from UK Biobank, we
demonstrate the robustness of the proposed method in highlighting reproducible
regions across individuals. We further evaluate the neurobiological relevance
of the identified regions based on evidence from large-scale UK Biobank
studies.
Local Optimality and Generalization Guarantees for the Langevin Algorithm via Empirical Metastability
Authors: Belinda Tzen, Tengyuan Liang, Maxim Raginsky
Date: 2018-02-18 00:00:00

We study the detailed path-wise behavior of the discrete-time Langevin
algorithm for non-convex Empirical Risk Minimization (ERM) through the lens of
metastability, adopting some techniques from Berglund and Gentz (2003.
  For a particular local optimum of the empirical risk, with an arbitrary
initialization, we show that, with high probability, at least one of the
following two events will occur: (1) the Langevin trajectory ends up somewhere
outside the $\varepsilon$-neighborhood of this particular optimum within a
short recurrence time; (2) it enters this $\varepsilon$-neighborhood by the
recurrence time and stays there until a potentially exponentially long escape
time. We call this phenomenon empirical metastability.
  This two-timescale characterization aligns nicely with the existing
literature in the following two senses. First, the effective recurrence time
(i.e., number of iterations multiplied by stepsize) is dimension-independent,
and resembles the convergence time of continuous-time deterministic Gradient
Descent (GD). However unlike GD, the Langevin algorithm does not require strong
conditions on local initialization, and has the possibility of eventually
visiting all optima. Second, the scaling of the escape time is consistent with
the Eyring-Kramers law, which states that the Langevin scheme will eventually
visit all local minima, but it will take an exponentially long time to transit
among them. We apply this path-wise concentration result in the context of
statistical learning to examine local notions of generalization and optimality.
Monte Carlo Convolution for Learning on Non-Uniformly Sampled Point Clouds
Authors: Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Àlvar Vinacua, Timo Ropinski
Date: 2018-06-05 00:00:00
Tasks: GPU, Point Cloud Segmentation

Deep learning systems extensively use convolution operations to process input
data. Though convolution is clearly defined for structured data such as 2D
images or 3D volumes, this is not true for other data types such as sparse
point clouds. Previous techniques have developed approximations to convolutions
for restricted conditions. Unfortunately, their applicability is limited and
cannot be used for general point clouds. We propose an efficient and effective
method to learn convolutions for non-uniformly sampled point clouds, as they
are obtained with modern acquisition techniques. Learning is enabled by four
key novelties: first, representing the convolution kernel itself as a
multilayer perceptron; second, phrasing convolution as a Monte Carlo
integration problem, third, using this notion to combine information from
multiple samplings at different levels; and fourth using Poisson disk sampling
as a scalable means of hierarchical point cloud learning. The key idea across
all these contributions is to guarantee adequate consideration of the
underlying non-uniform sample distribution function from a Monte Carlo
perspective. To make the proposed concepts applicable to real-world tasks, we
furthermore propose an efficient implementation which significantly reduces the
GPU memory required during the training process. By employing our method in
hierarchical network architectures we can outperform most of the
state-of-the-art networks on established point cloud segmentation,
classification and normal estimation benchmarks. Furthermore, in contrast to
most existing approaches, we also demonstrate the robustness of our method with
respect to sampling variations, even when training with uniformly sampled data
only. To support the direct application of these concepts, we provide a
ready-to-use TensorFlow implementation of these layers at
https://github.com/viscom-ulm/MCCNN
New Hybrid Neuro-Evolutionary Algorithms for Renewable Energy and Facilities Management Problems
Authors: L. Cornejo-Bueno
Date: 2018-06-05 00:00:00
Tasks: Evolutionary Algorithms, feature selection, Management

This Ph.D. thesis deals with the optimization of several renewable energy
resources development as well as the improvement of facilities management in
oceanic engineering and airports, using computational hybrid methods belonging
to AI to this end. Energy is essential to our society in order to ensure a good
quality of life. This means that predictions over the characteristics on which
renewable energies depend are necessary, in order to know the amount of energy
that will be obtained at any time. The second topic tackled in this thesis is
related to the basic parameters that influence in different marine activities
and airports, whose knowledge is necessary to develop a proper facilities
management in these environments. Within this work, a study of the
state-of-the-art Machine Learning have been performed to solve the problems
associated with the topics above-mentioned, and several contributions have been
proposed: One of the pillars of this work is focused on the estimation of the
most important parameters in the exploitation of renewable resources. The
second contribution of this thesis is related to feature selection problems.
The proposed methodologies are applied to multiple problems: the prediction of
$H_s$, relevant for marine energy applications and marine activities, the
estimation of WPREs, undesirable variations in the electric power produced by a
wind farm, the prediction of global solar radiation in areas from Spain and
Australia, really important in terms of solar energy, and the prediction of
low-visibility events at airports. All of these practical issues are developed
with the consequent previous data analysis, normally, in terms of
meteorological variables.
Breaking the $1/\sqrt{n}$ Barrier: Faster Rates for Permutation-based Models in Polynomial Time
Authors: Cheng Mao, Ashwin Pananjady, Martin J. Wainwright
Date: 2018-02-27 00:00:00

Many applications, including rank aggregation and crowd-labeling, can be
modeled in terms of a bivariate isotonic matrix with unknown permutations
acting on its rows and columns. We consider the problem of estimating such a
matrix based on noisy observations of a subset of its entries, and design and
analyze a polynomial-time algorithm that improves upon the state of the art. In
particular, our results imply that any such $n \times n$ matrix can be
estimated efficiently in the normalized Frobenius norm at rate
$\widetilde{\mathcal O}(n^{-3/4})$, thus narrowing the gap between
$\widetilde{\mathcal O}(n^{-1})$ and $\widetilde{\mathcal O}(n^{-1/2})$, which
were hitherto the rates of the most statistically and computationally efficient
methods, respectively.