pwc-archive/papers-with-abstracts huggingface dataset:
Reliability and Sharpness in Border Crossing Traffic Interval Prediction
Authors: Lei Lin, John Handley, Adel Sadek
Date: 2017-11-13 00:00:00
Tasks: Management, Prediction, Prediction Intervals, Value prediction

Short-term traffic volume prediction models have been extensively studied in
the past few decades. However, most of the previous studies only focus on
single-value prediction. Considering the uncertain and chaotic nature of the
transportation system, an accurate and reliable prediction interval with upper
and lower bounds may be better than a single point value for transportation
management. In this paper, we introduce a neural network model called Extreme
Learning Machine (ELM) for interval prediction of short-term traffic volume and
improve it with the heuristic particle swarm optimization algorithm (PSO). The
hybrid PSO-ELM model can generate the prediction intervals under different
confidence levels and guarantee the quality by minimizing a multi-objective
function which considers two criteria reliability and interval sharpness. The
PSO-ELM models are built based on an hourly traffic dataset and compared with
ARMA and Kalman Filter models. The results show that ARMA models are the worst
for all confidence levels, and the PSO-ELM models are comparable with Kalman
Filter from the aspects of reliability and narrowness of the intervals,
although the parameters of PSO-ELM are fixed once the training is done while
Kalman Filter is updated in an online approach. Additionally, only the PSO-ELMs
are able to produce intervals with coverage probabilities higher than or equal
to the confidence levels. For the points outside of the prediction levels given
by PSO-ELMs, they lie very close to the bounds.
Invariances and Data Augmentation for Supervised Music Transcription
Authors: John Thickstun, Zaid Harchaoui, Dean Foster, Sham M. Kakade
Date: 2017-11-13 00:00:00
Tasks: Data Augmentation, Music Transcription, Translation

This paper explores a variety of models for frame-based music transcription,
with an emphasis on the methods needed to reach state-of-the-art on human
recordings. The translation-invariant network discussed in this paper, which
combines a traditional filterbank with a convolutional neural network, was the
top-performing model in the 2017 MIREX Multiple Fundamental Frequency
Estimation evaluation. This class of models shares parameters in the
log-frequency domain, which exploits the frequency invariance of music to
reduce the number of model parameters and avoid overfitting to the training
data. All models in this paper were trained with supervision by labeled data
from the MusicNet dataset, augmented by random label-preserving pitch-shift
transformations.
A Deep Reinforced Model for Abstractive Summarization
Authors: Romain Paulus, Caiming Xiong, Richard Socher
Date: 2017-05-11 00:00:00
Proceeding: ICLR 2018 1
Tasks: Abstractive Text Summarization, Decoder, model, Prediction, Reinforcement Learning, Reinforcement Learning (RL), Text Summarization

Attentional, RNN-based encoder-decoder models for abstractive summarization
have achieved good performance on short input and output sequences. For longer
documents and summaries however these models often include repetitive and
incoherent phrases. We introduce a neural network model with a novel
intra-attention that attends over the input and continuously generated output
separately, and a new training method that combines standard supervised word
prediction and reinforcement learning (RL). Models trained only with supervised
learning often exhibit "exposure bias" - they assume ground truth is provided
at each step during training. However, when standard word prediction is
combined with the global sequence prediction training of RL the resulting
summaries become more readable. We evaluate this model on the CNN/Daily Mail
and New York Times datasets. Our model obtains a 41.16 ROUGE-1 score on the
CNN/Daily Mail dataset, an improvement over previous state-of-the-art models.
Human evaluation also shows that our model produces higher quality summaries.
Towards the Limit of Network Quantization
Authors: Yoojin Choi, Mostafa El-Khamy, Jungwon Lee
Date: 2016-12-05 00:00:00
Tasks: Clustering, Quantization

Network quantization is one of network compression techniques to reduce the
redundancy of deep neural networks. It reduces the number of distinct network
parameter values by quantization in order to save the storage for them. In this
paper, we design network quantization schemes that minimize the performance
loss due to quantization given a compression ratio constraint. We analyze the
quantitative relation of quantization errors to the neural network loss
function and identify that the Hessian-weighted distortion measure is locally
the right objective function for the optimization of network quantization. As a
result, Hessian-weighted k-means clustering is proposed for clustering network
parameters to quantize. When optimal variable-length binary codes, e.g.,
Huffman codes, are employed for further compression, we derive that the network
quantization problem can be related to the entropy-constrained scalar
quantization (ECSQ) problem in information theory and consequently propose two
solutions of ECSQ for network quantization, i.e., uniform quantization and an
iterative solution similar to Lloyd's algorithm. Finally, using the simple
uniform quantization followed by Huffman coding, we show from our experiments
that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet,
32-layer ResNet and AlexNet, respectively.
Armstrong's Axioms and Navigation Strategies
Authors: Kaya Deuser, Pavel Naumov
Date: 2017-07-13 00:00:00
Tasks: valid

The paper investigates navigability with imperfect information. It shows that
the properties of navigability with perfect recall are exactly those captured
by Armstrong's axioms from the database theory. If the assumption of perfect
recall is omitted, then Armstrong's transitivity axiom is not valid, but it can
be replaced by two new weaker principles. The main technical results are
soundness and completeness theorems for the logical systems describing
properties of navigability with and without perfect recall.
Calibrated Boosting-Forest
Authors: Haozhen Wu
Date: 2017-10-16 00:00:00
Tasks: Deep Learning, General Classification, regression

Excellent ranking power along with well calibrated probability estimates are
needed in many classification tasks. In this paper, we introduce a technique,
Calibrated Boosting-Forest that captures both. This novel technique is an
ensemble of gradient boosting machines that can support both continuous and
binary labels. While offering superior ranking power over any individual
regression or classification model, Calibrated Boosting-Forest is able to
preserve well calibrated posterior probabilities. Along with these benefits, we
provide an alternative to the tedious step of tuning gradient boosting
machines. We demonstrate that tuning Calibrated Boosting-Forest can be reduced
to a simple hyper-parameter selection. We further establish that increasing
this hyper-parameter improves the ranking performance under a diminishing
return. We examine the effectiveness of Calibrated Boosting-Forest on
ligand-based virtual screening where both continuous and binary labels are
available and compare the performance of Calibrated Boosting-Forest with
logistic regression, gradient boosting machine and deep learning. Calibrated
Boosting-Forest achieved an approximately 48% improvement compared to a
state-of-art deep learning model. Moreover, it achieved around 95% improvement
on probability quality measurement compared to the best individual gradient
boosting machine. Calibrated Boosting-Forest offers a benchmark demonstration
that in the field of ligand-based virtual screening, deep learning is not the
universally dominant machine learning model and good calibrated probabilities
can better facilitate virtual screening process.
Neural Networks Architecture Evaluation in a Quantum Computer
Authors: Adenilton José da Silva, Rodolfo Luan F. de Oliveira
Date: 2017-11-13 00:00:00

In this work, we propose a quantum algorithm to evaluate neural networks
architectures named Quantum Neural Network Architecture Evaluation (QNNAE). The
proposed algorithm is based on a quantum associative memory and the learning
algorithm for artificial neural networks. Unlike conventional algorithms for
evaluating neural network architectures, QNNAE does not depend on
initialization of weights. The proposed algorithm has a binary output and
results in 0 with probability proportional to the performance of the network.
And its computational cost is equal to the computational cost to train a neural
network.
Safe Model-based Reinforcement Learning with Stability Guarantees
Authors: Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, Andreas Krause
Date: 2017-05-23 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: model, Model-based Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Reinforcement learning is a powerful paradigm for learning optimal policies
from experimental data. However, to find optimal policies, most reinforcement
learning algorithms explore all possible actions, which may be harmful for
real-world systems. As a consequence, learning algorithms are rarely applied on
safety-critical systems in the real world. In this paper, we present a learning
algorithm that explicitly considers safety, defined in terms of stability
guarantees. Specifically, we extend control-theoretic results on Lyapunov
stability verification and show how to use statistical models of the dynamics
to obtain high-performance control policies with provable stability
certificates. Moreover, under additional regularity assumptions in terms of a
Gaussian process prior, we prove that one can effectively and safely collect
data in order to learn about the dynamics and thus both improve control
performance and expand the safe region of the state space. In our experiments,
we show how the resulting algorithm can safely optimize a neural network policy
on a simulated inverted pendulum, without the pendulum ever falling down.
ACtuAL: Actor-Critic Under Adversarial Learning
Authors: Anirudh Goyal, Nan Rosemary Ke, Alex Lamb, R. Devon Hjelm, Chris Pal, Joelle Pineau, Yoshua Bengio
Date: 2017-11-13 00:00:00
Tasks: Language Modeling, Language Modelling, Reinforcement Learning

Generative Adversarial Networks (GANs) are a powerful framework for deep
generative modeling. Posed as a two-player minimax problem, GANs are typically
trained end-to-end on real-valued data and can be used to train a generator of
high-dimensional and realistic images. However, a major limitation of GANs is
that training relies on passing gradients from the discriminator through the
generator via back-propagation. This makes it fundamentally difficult to train
GANs with discrete data, as generation in this case typically involves a
non-differentiable function. These difficulties extend to the reinforcement
learning setting when the action space is composed of discrete decisions. We
address these issues by reframing the GAN framework so that the generator is no
longer trained using gradients through the discriminator, but is instead
trained using a learned critic in the actor-critic framework with a Temporal
Difference (TD) objective. This is a natural fit for sequence modeling and we
use it to achieve improvements on language modeling tasks over the standard
Teacher-Forcing methods.
To prune, or not to prune: exploring the efficacy of pruning for model compression
Authors: Michael Zhu, Suyog Gupta
Date: 2017-10-05 00:00:00
Proceeding: ICLR 2018 1
Tasks: Model Compression

Model pruning seeks to induce sparsity in a deep neural network's various
connection matrices, thereby reducing the number of nonzero-valued parameters
in the model. Recent reports (Han et al., 2015; Narang et al., 2017) prune deep
networks at the cost of only a marginal loss in accuracy and achieve a sizable
reduction in model size. This hints at the possibility that the baseline models
in these experiments are perhaps severely over-parameterized at the outset and
a viable alternative for model compression might be to simply reduce the number
of hidden units while maintaining the model's dense connection structure,
exposing a similar trade-off in model size and accuracy. We investigate these
two distinct paths for model compression within the context of energy-efficient
inference in resource-constrained environments and propose a new gradual
pruning technique that is simple and straightforward to apply across a variety
of models/datasets with minimal tuning and can be seamlessly incorporated
within the training process. We compare the accuracy of large, but pruned
models (large-sparse) and their smaller, but dense (small-dense) counterparts
with identical memory footprint. Across a broad range of neural network
architectures (deep CNNs, stacked LSTM, and seq2seq LSTM models), we find
large-sparse models to consistently outperform small-dense models and achieve
up to 10x reduction in number of non-zero parameters with minimal loss in
accuracy.
Compression-aware Training of Deep Networks
Authors: Jose M. Alvarez, Mathieu Salzmann
Date: 2017-11-07 00:00:00
Proceeding: NeurIPS 2017 12

In recent years, great progress has been made in a variety of application
domains thanks to the development of increasingly deeper neural networks.
Unfortunately, the huge number of units of these networks makes them expensive
both computationally and memory-wise. To overcome this, exploiting the fact
that deep networks are over-parametrized, several compression strategies have
been proposed. These methods, however, typically start from a network that has
been trained in a standard manner, without considering such a future
compression. In this paper, we propose to explicitly account for compression in
the training process. To this end, we introduce a regularizer that encourages
the parameter matrix of each layer to have low rank during training. We show
that accounting for compression during training allows us to learn much more
compact, yet at least as effective, models than state-of-the-art compression
techniques.
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
Authors: Jeffrey Pennington, Samuel S. Schoenholz, Surya Ganguli
Date: 2017-11-13 00:00:00
Proceeding: NeurIPS 2017 12

It is well known that the initialization of weights in deep neural networks
can have a dramatic impact on learning speed. For example, ensuring the mean
squared singular value of a network's input-output Jacobian is $O(1)$ is
essential for avoiding the exponential vanishing or explosion of gradients. The
stronger condition that all singular values of the Jacobian concentrate near
$1$ is a property known as dynamical isometry. For deep linear networks,
dynamical isometry can be achieved through orthogonal weight initialization and
has been shown to dramatically speed up learning; however, it has remained
unclear how to extend these results to the nonlinear setting. We address this
question by employing powerful tools from free probability theory to compute
analytically the entire singular value distribution of a deep network's
input-output Jacobian. We explore the dependence of the singular value
distribution on the depth of the network, the weight initialization, and the
choice of nonlinearity. Intriguingly, we find that ReLU networks are incapable
of dynamical isometry. On the other hand, sigmoidal networks can achieve
isometry, but only with orthogonal weight initialization. Moreover, we
demonstrate empirically that deep nonlinear networks achieving dynamical
isometry learn orders of magnitude faster than networks that do not. Indeed, we
show that properly-initialized deep sigmoidal networks consistently outperform
deep ReLU networks. Overall, our analysis reveals that controlling the entire
distribution of Jacobian singular values is an important design consideration
in deep learning.
Evaluating prose style transfer with the Bible
Authors: Keith Carlson, Allen Riddell, Daniel Rockmore
Date: 2017-11-13 00:00:00
Tasks: Style Transfer

In the prose style transfer task a system, provided with text input and a
target prose style, produces output which preserves the meaning of the input
text but alters the style. These systems require parallel data for evaluation
of results and usually make use of parallel data for training. Currently, there
are few publicly available corpora for this task. In this work, we identify a
high-quality source of aligned, stylistically distinct text in different
versions of the Bible. We provide a standardized split, into training,
development and testing data, of the public domain versions in our corpus. This
corpus is highly parallel since many Bible versions are included. Sentences are
aligned due to the presence of chapter and verse numbers within all versions of
the text. In addition to the corpus, we present the results, as measured by the
BLEU and PINC metrics, of several models trained on our data which can serve as
baselines for future research. While we present these data as a style transfer
corpus, we believe that it is of unmatched quality and may be useful for other
natural language tasks as well.
Learning Disentangled Representations with Semi-Supervised Deep Generative Models
Authors: N. Siddharth, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, Philip H. S. Torr
Date: 2017-06-01 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Decoder, Representation Learning

Variational autoencoders (VAEs) learn representations of data by jointly
training a probabilistic encoder and decoder network. Typically these models
encode all features of the data into a single variable. Here we are interested
in learning disentangled representations that encode distinct aspects of the
data into separate variables. We propose to learn such representations using
model architectures that generalise from standard VAEs, employing a general
graphical model structure in the encoder and decoder. This allows us to train
partially-specified models that make relatively strong assumptions about a
subset of interpretable variables and rely on the flexibility of neural
networks to learn representations for the remaining variables. We further
define a general objective for semi-supervised learning in this model class,
which can be approximated using an importance sampling procedure. We evaluate
our framework's ability to learn disentangled representations, both by
qualitative exploration of its generative capacity, and quantitative evaluation
of its discriminative ability on a variety of models and datasets.
Theory-guided Data Science: A New Paradigm for Scientific Discovery from Data
Authors: Anuj Karpatne, Gowtham Atluri, James Faghmous, Michael Steinbach, Arindam Banerjee, Auroop Ganguly, Shashi Shekhar, Nagiza Samatova, Vipin Kumar
Date: 2016-12-27 00:00:00
Tasks: scientific discovery

Data science models, although successful in a number of commercial domains,
have had limited applicability in scientific problems involving complex
physical phenomena. Theory-guided data science (TGDS) is an emerging paradigm
that aims to leverage the wealth of scientific knowledge for improving the
effectiveness of data science models in enabling scientific discovery. The
overarching vision of TGDS is to introduce scientific consistency as an
essential component for learning generalizable models. Further, by producing
scientifically interpretable models, TGDS aims to advance our scientific
understanding by discovering novel domain insights. Indeed, the paradigm of
TGDS has started to gain prominence in a number of scientific disciplines such
as turbulence modeling, material discovery, quantum chemistry, bio-medical
science, bio-marker discovery, climate science, and hydrology. In this paper,
we formally conceptualize the paradigm of TGDS and present a taxonomy of
research themes in TGDS. We describe several approaches for integrating domain
knowledge in different research themes using illustrative examples from
different disciplines. We also highlight some of the promising avenues of novel
research for realizing the full potential of theory-guided data science.
ADaPTION: Toolbox and Benchmark for Training Convolutional Neural Networks with Reduced Numerical Precision Weights and Activation
Authors: Moritz B. Milde, Daniel Neil, Alessandro Aimar, Tobi Delbruck, Giacomo Indiveri
Date: 2017-11-13 00:00:00
Tasks: Quantization

Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) are
useful for many practical tasks in machine learning. Synaptic weights, as well
as neuron activation functions within the deep network are typically stored
with high-precision formats, e.g. 32 bit floating point. However, since storage
capacity is limited and each memory access consumes power, both storage
capacity and memory access are two crucial factors in these networks. Here we
present a method and present the ADaPTION toolbox to extend the popular deep
learning library Caffe to support training of deep CNNs with reduced numerical
precision of weights and activations using fixed point notation. ADaPTION
includes tools to measure the dynamic range of weights and activations. Using
the ADaPTION tools, we quantized several CNNs including VGG16 down to 16-bit
weights and activations with only 0.8% drop in Top-1 accuracy. The
quantization, especially of the activations, leads to increase of up to 50% of
sparsity especially in early and intermediate layers, which we exploit to skip
multiplications with zero, thus performing faster and computationally cheaper
inference.
Randomized Near Neighbor Graphs, Giant Components, and Applications in Data Science
Authors: George C. Linderman, Gal Mishne, Yuval Kluger, Stefan Steinerberger
Date: 2017-11-13 00:00:00

If we pick $n$ random points uniformly in $[0,1]^d$ and connect each point to
its $k-$nearest neighbors, then it is well known that there exists a giant
connected component with high probability. We prove that in $[0,1]^d$ it
suffices to connect every point to $ c_{d,1} \log{\log{n}}$ points chosen
randomly among its $ c_{d,2} \log{n}-$nearest neighbors to ensure a giant
component of size $n - o(n)$ with high probability. This construction yields a
much sparser random graph with $\sim n \log\log{n}$ instead of $\sim n \log{n}$
edges that has comparable connectivity properties. This result has nontrivial
implications for problems in data science where an affinity matrix is
constructed: instead of picking the $k-$nearest neighbors, one can often pick
$k' \ll k$ random points out of the $k-$nearest neighbors without sacrificing
efficiency. This can massively simplify and accelerate computation, we
illustrate this with several numerical examples.