pwc-archive/papers-with-abstracts huggingface dataset:
Instance Normalization: The Missing Ingredient for Fast Stylization
Authors: Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
Date: 2016-07-27 00:00:00
Tasks: Image Generation, Image Stylization, Style Transfer

It this paper we revisit the fast stylization method introduced in Ulyanov
et. al. (2016). We show how a small change in the stylization architecture
results in a significant qualitative improvement in the generated images. The
change is limited to swapping batch normalization with instance normalization,
and to apply the latter both at training and testing times. The resulting
method can be used to train high-performance architectures for real-time image
generation. The code will is made available on github at
https://github.com/DmitryUlyanov/texture_nets. Full paper can be found at
arXiv:1701.02096.
Revenue Optimization with Approximate Bid Predictions
Authors: Andrés Muñoz Medina, Sergei Vassilvitskii
Date: 2017-06-15 00:00:00
Proceeding: NeurIPS 2017 12

In the context of advertising auctions, finding good reserve prices is a
notoriously challenging learning problem. This is due to the heterogeneity of
ad opportunity types and the non-convexity of the objective function. In this
work, we show how to reduce reserve price optimization to the standard setting
of prediction under squared loss, a well understood problem in the learning
community. We further bound the gap between the expected bid and revenue in
terms of the average loss of the predictor. This is the first result that
formally relates the revenue gained to the quality of a standard machine
learned model.
Causal Effect Inference with Deep Latent-Variable Models
Authors: Christos Louizos, Uri Shalit, Joris Mooij, David Sontag, Richard Zemel, Max Welling
Date: 2017-05-24 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Causal Inference

Learning individual-level causal effects from observational data, such as
inferring the most effective medication for a specific patient, is a problem of
growing importance for policy makers. The most important aspect of inferring
causal effects from observational data is the handling of confounders, factors
that affect both an intervention and its outcome. A carefully designed
observational study attempts to measure all important confounders. However,
even if one does not have direct access to all confounders, there may exist
noisy and uncertain measurement of proxies for confounders. We build on recent
advances in latent variable modeling to simultaneously estimate the unknown
latent space summarizing the confounders and the causal effect. Our method is
based on Variational Autoencoders (VAE) which follow the causal structure of
inference with proxies. We show our method is significantly more robust than
existing methods, and matches the state-of-the-art on previous benchmarks
focused on individual treatment effects.
Interpretable Feature Recommendation for Signal Analytics
Authors: Snehasis Banerjee, Tanushyam Chattopadhyay, Ayan Mukherjee
Date: 2017-11-06 00:00:00

This paper presents an automated approach for interpretable feature
recommendation for solving signal data analytics problems. The method has been
tested by performing experiments on datasets in the domain of prognostics where
interpretation of features is considered very important. The proposed approach
is based on Wide Learning architecture and provides means for interpretation of
the recommended features. It is to be noted that such an interpretation is not
available with feature learning approaches like Deep Learning (such as
Convolutional Neural Network) or feature transformation approaches like
Principal Component Analysis. Results show that the feature recommendation and
interpretation techniques are quite effective for the problems at hand in terms
of performance and drastic reduction in time to develop a solution. It is
further shown by an example, how this human-in-loop interpretation system can
be used as a prescriptive system.
Distributed deep learning on edge-devices: feasibility via adaptive compression
Authors: Corentin Hardy, Erwan Le Merrer, Bruno Sericola
Date: 2017-02-15 00:00:00
Tasks: BIG-bench Machine Learning, CPU, Deep Learning

A large portion of data mining and analytic services use modern machine
learning techniques, such as deep learning. The state-of-the-art results by
deep learning come at the price of an intensive use of computing resources. The
leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end
servers in datacenters. On the other end, there is a proliferation of personal
devices with possibly free CPU cycles; this can enable services to run in
users' homes, embedding machine learning operations. In this paper, we ask the
following question: Is distributed deep learning computation on WAN connected
devices feasible, in spite of the traffic caused by learning tasks? We show
that such a setup rises some important challenges, most notably the ingress
traffic that the servers hosting the up-to-date model have to sustain.
  In order to reduce this stress, we propose adaComp, a novel algorithm for
compressing worker updates to the model on the server. Applicable to stochastic
gradient descent based approaches, it combines efficient gradient selection and
learning rate modulation. We then experiment and measure the impact of
compression, device heterogeneity and reliability on the accuracy of learned
models, with an emulator platform that embeds TensorFlow into Linux containers.
We report a reduction of the total amount of data sent by workers to the server
by two order of magnitude (e.g., 191-fold reduction for a convolutional network
on the MNIST dataset), when compared to a standard asynchronous stochastic
gradient descent, while preserving model accuracy.
Trimmed Density Ratio Estimation
Authors: Song Liu, Akiko Takeda, Taiji Suzuki, Kenji Fukumizu
Date: 2017-03-09 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: BIG-bench Machine Learning, Density Ratio Estimation, parameter estimation

Density ratio estimation is a vital tool in both machine learning and
statistical community. However, due to the unbounded nature of density ratio,
the estimation procedure can be vulnerable to corrupted data points, which
often pushes the estimated ratio toward infinity. In this paper, we present a
robust estimator which automatically identifies and trims outliers. The
proposed estimator has a convex formulation, and the global optimum can be
obtained via subgradient descent. We analyze the parameter estimation error of
this estimator under high-dimensional settings. Experiments are conducted to
verify the effectiveness of the estimator.
Bayesian Compression for Deep Learning
Authors: Christos Louizos, Karen Ullrich, Max Welling
Date: 2017-05-24 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Computational Efficiency, Deep Learning

Compression and computational efficiency in deep learning have become a
problem of great significance. In this work, we argue that the most principled
and effective way to attack this problem is by adopting a Bayesian point of
view, where through sparsity inducing priors we prune large parts of the
network. We introduce two novelties in this paper: 1) we use hierarchical
priors to prune nodes instead of individual weights, and 2) we use the
posterior uncertainties to determine the optimal fixed point precision to
encode the weights. Both factors significantly contribute to achieving the
state of the art in terms of compression rates, while still staying competitive
with methods designed to optimize for speed or energy efficiency.
Flexible statistical inference for mechanistic models of neural dynamics
Authors: Jan-Matthis Lueckmann, Pedro J. Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, Jakob H. Macke
Date: 2017-11-06 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Bayesian Inference

Mechanistic models of single-neuron dynamics have been extensively studied in
computational neuroscience. However, identifying which models can
quantitatively reproduce empirically measured data has been challenging. We
propose to overcome this limitation by using likelihood-free inference
approaches (also known as Approximate Bayesian Computation, ABC) to perform
full Bayesian inference on single-neuron models. Our approach builds on recent
advances in ABC by learning a neural network which maps features of the
observed data to the posterior distribution over parameters. We learn a
Bayesian mixture-density network approximating the posterior over multiple
rounds of adaptively chosen simulations. Furthermore, we propose an efficient
approach for handling missing features and parameter settings for which the
simulator fails, as well as a strategy for automatically learning relevant
features using recurrent neural networks. On synthetic data, our approach
efficiently estimates posterior distributions and recovers ground-truth
parameters. On in-vitro recordings of membrane voltages, we recover
multivariate posteriors over biophysical parameters, which yield
model-predicted voltage traces that accurately match empirical data. Our
approach will enable neuroscientists to perform Bayesian inference on complex
neuron models without having to design model-specific algorithms, closing the
gap between mechanistic and statistical approaches to single-neuron modelling.
Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations
Authors: Marcel Nonnenmacher, Srinivas C. Turaga, Jakob H. Macke
Date: 2017-11-06 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Dimensionality Reduction, parameter estimation

A powerful approach for understanding neural population dynamics is to
extract low-dimensional trajectories from population recordings using
dimensionality reduction methods. Current approaches for dimensionality
reduction on neural data are limited to single population recordings, and can
not identify dynamics embedded across multiple measurements. We propose an
approach for extracting low-dimensional dynamics from multiple, sequential
recordings. Our algorithm scales to data comprising millions of observed
dimensions, making it possible to access dynamics distributed across large
populations or multiple brain areas. Building on subspace-identification
approaches for dynamical systems, we perform parameter estimation by minimizing
a moment-matching objective using a scalable stochastic gradient descent
algorithm: The model is optimized to predict temporal covariations across
neurons and across time. We show how this approach naturally handles missing
data and multiple partial recordings, and can identify dynamics and predict
correlations even in the presence of severe subsampling and small overlap
between recordings. We demonstrate the effectiveness of the approach both on
simulated data and a whole-brain larval zebrafish imaging dataset.
Fast amortized inference of neural activity from calcium imaging data with variational autoencoders
Authors: Artur Speiser, Jinyao Yan, Evan Archer, Lars Buesing, Srinivas C. Turaga, Jakob H. Macke
Date: 2017-11-06 00:00:00
Proceeding: NeurIPS 2017 12

Calcium imaging permits optical measurement of neural activity. Since
intracellular calcium concentration is an indirect measurement of neural
activity, computational tools are necessary to infer the true underlying
spiking activity from fluorescence measurements. Bayesian model inversion can
be used to solve this problem, but typically requires either computationally
expensive MCMC sampling, or faster but approximate maximum-a-posteriori
optimization. Here, we introduce a flexible algorithmic framework for fast,
efficient and accurate extraction of neural spikes from imaging data. Using the
framework of variational autoencoders, we propose to amortize inference by
training a deep neural network to perform model inversion efficiently. The
recognition network is trained to produce samples from the posterior
distribution over spike trains. Once trained, performing inference amounts to a
fast single forward pass through the network, without the need for iterative
optimization or sampling. We show that amortization can be applied flexibly to
a wide range of nonlinear generative models and significantly improves upon the
state of the art in computation time, while achieving competitive accuracy. Our
framework is also able to represent posterior distributions over spike-trains.
We demonstrate the generality of our method by proposing the first
probabilistic approach for separating backpropagating action potentials from
putative synaptic inputs in calcium imaging of dendritic spines.
Computer activity learning from system call time series
Authors: Curt Hastings, Ronnie Mainieri
Date: 2017-11-06 00:00:00
Tasks: General Classification, Malware Classification, Time Series, Time Series Analysis

Using a previously introduced similarity function for the stream of system
calls generated by a computer, we engineer a program-in-execution classifier
using deep learning methods. Tested on malware classification, it significantly
outperforms current state of the art. We provide a series of performance
measures and tests to demonstrate the capabilities, including measurements from
production use. We show how the system scales linearly with the number of
endpoints. With the system we estimate the total number of malware families
created over the last 10 years as 3450, in line with reasonable economic
constraints. The more limited rate for new malware families than previously
acknowledged implies that machine learning malware classifiers risk being
tested on their training set; we achieve F1 = 0.995 in a test carefully
designed to mitigate this risk.
Online Tool Condition Monitoring Based on Parsimonious Ensemble+
Authors: Mahardhika Pratama, Eric Dimla, Edwin Lughofer, Witold Pedrycz, Tegoeh Tjahjowidowo
Date: 2017-11-06 00:00:00
Tasks: Active Learning, Ensemble Learning, feature selection

Accurate diagnosis of tool wear in metal turning process remains an open challenge for both scientists and industrial practitioners because of inhomogeneities in workpiece material, nonstationary machining settings to suit production requirements, and nonlinear relations between measured variables and tool wear. Common methodologies for tool condition monitoring still rely on batch approaches which cannot cope with a fast sampling rate of metal cutting process. Furthermore they require a retraining process to be completed from scratch when dealing with a new set of machining parameters. This paper presents an online tool condition monitoring approach based on Parsimonious Ensemble+, pENsemble+. The unique feature of pENsemble+ lies in its highly flexible principle where both ensemble structure and base-classifier structure can automatically grow and shrink on the fly based on the characteristics of data streams. Moreover, the online feature selection scenario is integrated to actively sample relevant input attributes. The paper presents advancement of a newly developed ensemble learning algorithm, pENsemble+, where online active learning scenario is incorporated to reduce operator labelling effort. The ensemble merging scenario is proposed which allows reduction of ensemble complexity while retaining its diversity. Experimental studies utilising real-world manufacturing data streams and comparisons with well known algorithms were carried out. Furthermore, the efficacy of pENsemble was examined using benchmark concept drift data streams. It has been found that pENsemble+ incurs low structural complexity and results in a significant reduction of operator labelling effort.
Tapping the sensorimotor trajectory
Authors: Oswald Berthold, Verena Hafner
Date: 2017-04-25 00:00:00
Tasks: Relation Linking

In this paper, we propose the concept of sensorimotor tappings, a new
graphical technique that explicitly represents relations between the time steps
of an agent's sensorimotor loop and a single training step of an adaptive
internal model. In the simplest case this is a relation linking two time steps.
In realistic cases these relations can extend over several time steps and over
different sensory channels. The aim is to capture the footprint of information
intake relative to the agent's current time step. We argue that this view
allows us to make prior considerations explicit and then use them in
implementations without modification once they are established.
  Here we explain the basic idea, provide example tappings for standard
configurations used in developmental models, and show how tappings can be
applied to problems in related fields.
Sum-Product Networks for Hybrid Domains
Authors: Alejandro Molina, Antonio Vergari, Nicola Di Mauro, Sriraam Natarajan, Floriana Esposito, Kristian Kersting
Date: 2017-10-09 00:00:00

While all kinds of mixed data -from personal data, over panel and scientific
data, to public and commercial data- are collected and stored, building
probabilistic graphical models for these hybrid domains becomes more difficult.
Users spend significant amounts of time in identifying the parametric form of
the random variables (Gaussian, Poisson, Logit, etc.) involved and learning the
mixed models. To make this difficult task easier, we propose the first
trainable probabilistic deep architecture for hybrid domains that features
tractable queries. It is based on Sum-Product Networks (SPNs) with piecewise
polynomial leave distributions together with novel nonparametric decomposition
and conditioning steps using the Hirschfeld-Gebelein-R\'enyi Maximum
Correlation Coefficient. This relieves the user from deciding a-priori the
parametric form of the random variables but is still expressive enough to
effectively approximate any continuous distribution and permits efficient
learning and inference. Our empirical evidence shows that the architecture,
called Mixed SPNs, can indeed capture complex distributions across a wide range
of hybrid domains.
On Bayesian index policies for sequential resource allocation
Authors: Emilie Kaufmann
Date: 2016-01-06 00:00:00

This paper is about index policies for minimizing (frequentist) regret in a
stochastic multi-armed bandit model, inspired by a Bayesian view on the
problem. Our main contribution is to prove that the Bayes-UCB algorithm, which
relies on quantiles of posterior distributions, is asymptotically optimal when
the reward distributions belong to a one-dimensional exponential family, for a
large class of prior distributions. We also show that the Bayesian literature
gives new insight on what kind of exploration rates could be used in
frequentist, UCB-type algorithms. Indeed, approximations of the Bayesian
optimal solution or the Finite Horizon Gittins indices provide a justification
for the kl-UCB+ and kl-UCB-H+ algorithms, whose asymptotic optimality is also
established.
HyperNetworks with statistical filtering for defending adversarial examples
Authors: Zhun Sun, Mete Ozay, Takayuki Okatani
Date: 2017-11-06 00:00:00
Tasks: General Classification, image-classification, Image Classification

Deep learning algorithms have been known to be vulnerable to adversarial
perturbations in various tasks such as image classification. This problem was
addressed by employing several defense methods for detection and rejection of
particular types of attacks. However, training and manipulating networks
according to particular defense schemes increases computational complexity of
the learning algorithms. In this work, we propose a simple yet effective method
to improve robustness of convolutional neural networks (CNNs) to adversarial
attacks by using data dependent adaptive convolution kernels. To this end, we
propose a new type of HyperNetwork in order to employ statistical properties of
input data and features for computation of statistical adaptive maps. Then, we
filter convolution weights of CNNs with the learned statistical maps to compute
dynamic kernels. Thereby, weights and kernels are collectively optimized for
learning of image classification models robust to adversarial attacks without
employment of additional target detection and rejection algorithms. We
empirically demonstrate that the proposed method enables CNNs to spontaneously
defend against different types of attacks, e.g. attacks generated by Gaussian
noise, fast gradient sign methods (Goodfellow et al., 2014) and a black-box
attack(Narodytska & Kasiviswanathan, 2016).
Simultaneous Block-Sparse Signal Recovery Using Pattern-Coupled Sparse Bayesian Learning
Authors: Hang Xiao, Zhengli Xing, Linxiao Yang, Jun Fang, Yanlun Wu
Date: 2017-11-06 00:00:00

In this paper, we consider the block-sparse signals recovery problem in the
context of multiple measurement vectors (MMV) with common row sparsity
patterns. We develop a new method for recovery of common row sparsity MMV
signals, where a pattern-coupled hierarchical Gaussian prior model is
introduced to characterize both the block-sparsity of the coefficients and the
statistical dependency between neighboring coefficients of the common row
sparsity MMV signals. Unlike many other methods, the proposed method is able to
automatically capture the block sparse structure of the unknown signal. Our
method is developed using an expectation-maximization (EM) framework.
Simulation results show that our proposed method offers competitive performance
in recovering block-sparse common row sparsity pattern MMV signals.
Submultiplicative Glivenko-Cantelli and Uniform Convergence of Revenues
Authors: Noga Alon, Moshe Babaioff, Yannai A. Gonczarowski, Yishay Mansour, Shay Moran, Amir Yehudayoff
Date: 2017-05-23 00:00:00
Proceeding: NeurIPS 2017 12

In this work we derive a variant of the classic Glivenko-Cantelli Theorem,
which asserts uniform convergence of the empirical Cumulative Distribution
Function (CDF) to the CDF of the underlying distribution. Our variant allows
for tighter convergence bounds for extreme values of the CDF.
  We apply our bound in the context of revenue learning, which is a
well-studied problem in economics and algorithmic game theory. We derive
sample-complexity bounds on the uniform convergence rate of the empirical
revenues to the true revenues, assuming a bound on the $k$th moment of the
valuations, for any (possibly fractional) $k>1$.
  For uniform convergence in the limit, we give a complete characterization and
a zero-one law: if the first moment of the valuations is finite, then uniform
convergence almost surely occurs; conversely, if the first moment is infinite,
then uniform convergence almost never occurs.
AdaBatch: Efficient Gradient Aggregation Rules for Sequential and Parallel Stochastic Gradient Methods
Authors: Alexandre Défossez, Francis Bach
Date: 2017-11-06 00:00:00

We study a new aggregation operator for gradients coming from a mini-batch
for stochastic gradient (SG) methods that allows a significant speed-up in the
case of sparse optimization problems. We call this method AdaBatch and it only
requires a few lines of code change compared to regular mini-batch SGD
algorithms. We provide a theoretical insight to understand how this new class
of algorithms is performing and show that it is equivalent to an implicit
per-coordinate rescaling of the gradients, similarly to what Adagrad methods
can do. In theory and in practice, this new aggregation allows to keep the same
sample efficiency of SG methods while increasing the batch size.
Experimentally, we also show that in the case of smooth convex optimization,
our procedure can even obtain a better loss when increasing the batch size for
a fixed number of samples. We then apply this new algorithm to obtain a
parallelizable stochastic gradient method that is synchronous but allows
speed-up on par with Hogwild! methods as convergence does not deteriorate with
the increase of the batch size. The same approach can be used to make
mini-batch provably efficient for variance-reduced SG methods such as SVRG.