pwc-archive/papers-with-abstracts huggingface dataset:
On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
Authors: Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen
Date: 2019-02-05 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, Sequence-To-Sequence Speech Recognition, speech-recognition, Speech Recognition

In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.
Meta-Amortized Variational Inference and Learning
Authors: Mike Wu, Kristy Choi, Noah Goodman, Stefano Ermon
Date: 2019-02-05 00:00:00
Tasks: Clustering, Density Estimation, image-classification, Image Classification, Variational Inference

Despite the recent success in probabilistic modeling and their applications, generative models trained using traditional inference techniques struggle to adapt to new distributions, even when the target distribution may be closely related to the ones seen during training. In this work, we present a doubly-amortized variational inference procedure as a way to address this challenge. By sharing computation across not only a set of query inputs, but also a set of different, related probabilistic models, we learn transferable latent representations that generalize across several related distributions. In particular, given a set of distributions over images, we find the learned representations to transfer to different data transformations. We empirically demonstrate the effectiveness of our method by introducing the MetaVAE, and show that it significantly outperforms baselines on downstream image classification tasks on MNIST (10-50%) and NORB (10-35%).
Active Learning for High-Dimensional Binary Features
Authors: Ali Vahdat, Mouloud Belbahri, Vahid Partovi Nia
Date: 2019-02-05 00:00:00
Tasks: Active Learning, Management, Vocal Bursts Intensity Prediction

Erbium-doped fiber amplifier (EDFA) is an optical amplifier/repeater device used to boost the intensity of optical signals being carried through a fiber optic communication system. A highly accurate EDFA model is important because of its crucial role in optical network management and optimization. The input channels of an EDFA device are treated as either on or off, hence the input features are binary. Labeled training data is very expensive to collect for EDFA devices, therefore we devise an active learning strategy suitable for binary variables to overcome this issue. We propose to take advantage of sparse linear models to simplify the predictive model. This approach simultaneously improves prediction and accelerates active learning query generation. We show the performance of our proposed active learning strategies on simulated data and real EDFA data.
CMS Sematrix: A Tool to Aid the Development of Clinical Quality Measures (CQMs)
Authors: Michael A. Schwemmer, Po-Hsu Chen, Mithun Balakrishna, Amy Leibrand, Aaron Leonard, Nancy J. McMillan, Jeffrey J. Geppert
Date: 2019-02-05 00:00:00

As part of the effort to improve quality and to reduce national healthcare
costs, the Centers for Medicare and Medicaid Services (CMS) are responsible for
creating and maintaining an array of clinical quality measures (CQMs) for
assessing healthcare structure, process, outcome, and patient experience across
various conditions, clinical specialties, and settings. The development and
maintenance of CQMs involves substantial and ongoing evaluation of the evidence
on the measure's properties: importance, reliability, validity, feasibility,
and usability. As such, CMS conducts monthly environmental scans of the
published clinical and health service literature. Conducting time consuming,
exhaustive evaluations of the ever-changing healthcare literature presents one
of the largest challenges to an evidence-based approach to healthcare quality
improvement. Thus, it is imperative to leverage automated techniques to aid CMS
in the identification of clinical and health services literature relevant to
CQMs. Additionally, the estimated labor hours and related cost savings of using
CMS Sematrix compared to a traditional literature review are roughly 818 hours
and 122,000 dollars for a single monthly environmental scan.
Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization
Authors: Eldad Meller, Alexander Finkelstein, Uri Almog, Mark Grobman
Date: 2019-02-05 00:00:00
Tasks: Network Pruning, Quantization

Quantization of neural networks has become common practice, driven by the
need for efficient implementations of deep neural networks on embedded devices.
In this paper, we exploit an oft-overlooked degree of freedom in most networks
- for a given layer, individual output channels can be scaled by any factor
provided that the corresponding weights of the next layer are inversely scaled.
Therefore, a given network has many factorizations which change the weights of
the network without changing its function. We present a conceptually simple and
easy to implement method that uses this property and show that proper
factorizations significantly decrease the degradation caused by quantization.
We show improvement on a wide variety of networks and achieve state-of-the-art
degradation results for MobileNets. While our focus is on quantization, this
type of factorization is applicable to other domains such as network-pruning,
neural nets regularization and network interpretability.
Uniform concentration and symmetrization for weak interactions
Authors: Andreas Maurer, Massimiliano Pontil
Date: 2019-02-05 00:00:00

The method to derive uniform bounds with Gaussian and Rademacher complexities is extended to the case where the sample average is replaced by a nonlinear statistic. Tight bounds are obtained for U-statistics, smoothened L-statistics and error functionals of l2-regularized algorithms.
Adaptive Stress Testing for Autonomous Vehicles
Authors: Mark Koren, Saud Alsaif, Ritchie Lee, Mykel J. Kochenderfer
Date: 2019-02-05 00:00:00
Tasks: Autonomous Vehicles, Decision Making, Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This paper presents a method for testing the decision making systems of
autonomous vehicles. Our approach involves perturbing stochastic elements in
the vehicle's environment until the vehicle is involved in a collision. Instead
of applying direct Monte Carlo sampling to find collision scenarios, we
formulate the problem as a Markov decision process and use reinforcement
learning algorithms to find the most likely failure scenarios. This paper
presents Monte Carlo Tree Search (MCTS) and Deep Reinforcement Learning (DRL)
solutions that can scale to large environments. We show that DRL can find more
likely failure scenarios than MCTS with fewer calls to the simulator. A
simulation scenario involving a vehicle approaching a crosswalk is used to
validate the framework. Our proposed approach is very general and can be easily
applied to other scenarios given the appropriate models of the vehicle and the
environment.
Exponentiated Gradient Meets Gradient Descent
Authors: Udaya Ghai, Elad Hazan, Yoram Singer
Date: 2019-02-05 00:00:00

The (stochastic) gradient descent and the multiplicative update method are
probably the most popular algorithms in machine learning. We introduce and
study a new regularization which provides a unification of the additive and
multiplicative updates. This regularization is derived from an hyperbolic
analogue of the entropy function, which we call hypentropy. It is motivated by
a natural extension of the multiplicative update to negative numbers. The
hypentropy has a natural spectral counterpart which we use to derive a family
of matrix-based updates that bridge gradient methods and the multiplicative
method for matrices. While the latter is only applicable to positive
semi-definite matrices, the spectral hypentropy method can naturally be used
with general rectangular matrices. We analyze the new family of updates by
deriving tight regret bounds. We study empirically the applicability of the new
update for settings such as multiclass learning, in which the parameters
constitute a general rectangular matrix.
Asymptotic Consistency of $α-$Rényi-Approximate Posteriors
Authors: Prateek Jaiswal, Vinayak A. Rao, Harsha Honnappa
Date: 2019-02-05 00:00:00

We study the asymptotic consistency properties of $\alpha$-R\'enyi approximate posteriors, a class of variational Bayesian methods that approximate an intractable Bayesian posterior with a member of a tractable family of distributions, the member chosen to minimize the $\alpha$-R\'enyi divergence from the true posterior. Unique to our work is that we consider settings with $\alpha > 1$, resulting in approximations that upperbound the log-likelihood, and consequently have wider spread than traditional variational approaches that minimize the Kullback-Liebler (KL) divergence from the posterior. Our primary result identifies sufficient conditions under which consistency holds, centering around the existence of a 'good' sequence of distributions in the approximating family that possesses, among other properties, the right rate of convergence to a limit distribution. We further characterize the good sequence by demonstrating that a sequence of distributions that converges too quickly cannot be a good sequence. We also extend our analysis to the setting where $\alpha$ equals one, corresponding to the minimizer of the reverse KL divergence, and to models with local latent variables. We also illustrate the existence of good sequence with a number of examples. Our results complement a growing body of work focused on the frequentist properties of variational Bayesian methods.
Reinforcement Learning for Optimal Load Distribution Sequencing in Resource-Sharing System
Authors: Fei Wu, Yang Cao, Thomas Robertazzi
Date: 2019-02-05 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Scheduling

Divisible Load Theory (DLT) is a powerful tool for modeling divisible load
problems in data-intensive systems. This paper studied an optimal divisible
load distribution sequencing problem using a machine learning framework. The
problem is to decide the optimal sequence to distribute divisible load to
processors in order to achieve minimum finishing time. The scheduling is
performed in a resource-sharing system where each physical processor is
virtualized to multiple virtual processors. A reinforcement learning method
called Multi-armed bandit (MAB) is used for our problem. We first provide a
naive solution using the MAB algorithm and then several optimizations are
performed. Various numerical tests are conducted. Our algorithm shows an
increasing performance during the training progress and the global optimum will
be acheived when the sample size is large enough.
A Generalized Framework for Population Based Training
Authors: Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, Pramod Gupta
Date: 2019-02-05 00:00:00

Population Based Training (PBT) is a recent approach that jointly optimizes
neural network weights and hyperparameters which periodically copies weights of
the best performers and mutates hyperparameters during training. Previous PBT
implementations have been synchronized glass-box systems. We propose a general,
black-box PBT framework that distributes many asynchronous "trials" (a small
number of training steps with warm-starting) across a cluster, coordinated by
the PBT controller. The black-box design does not make assumptions on model
architectures, loss functions or training procedures. Our system supports
dynamic hyperparameter schedules to optimize both differentiable and
non-differentiable metrics. We apply our system to train a state-of-the-art
WaveNet generative model for human voice synthesis. We show that our PBT system
achieves better accuracy, less sensitivity and faster convergence compared to
existing methods, given the same computational resource.
TzK: Flow-Based Conditional Generative Model
Authors: Micha Livne, David Fleet
Date: 2019-02-05 00:00:00
Tasks: Attribute, model

We formulate a new class of conditional generative models based on
probability flows. Trained with maximum likelihood, it provides efficient
inference and sampling from class-conditionals or the joint distribution, and
does not require a priori knowledge of the number of classes or the
relationships between classes. This allows one to train generative models from
multiple, heterogeneous datasets, while retaining strong prior models over
subsets of the data (e.g., from a single dataset, class label, or attribute).
In this paper, in addition to end-to-end learning, we show how one can learn a
single model from multiple datasets with a relatively weak Glow architecture,
and then extend it by conditioning on different knowledge types (e.g., a single
dataset). This yields log likelihood comparable to state-of-the-art, compelling
samples from conditional priors.
Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
Authors: Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton
Date: 2019-02-05 00:00:00
Tasks: Classification

We explore and expand the $\textit{Soft Nearest Neighbor Loss}$ to measure
the $\textit{entanglement}$ of class manifolds in representation space: i.e.,
how close pairs of points from the same class are relative to pairs of points
from different classes. We demonstrate several use cases of the loss. As an
analytical tool, it provides insights into the evolution of class similarity
structures during learning. Surprisingly, we find that $\textit{maximizing}$
the entanglement of representations of different classes in the hidden layers
is beneficial for discrimination in the final layer, possibly because it
encourages representations to identify class-independent similarity structures.
Maximizing the soft nearest neighbor loss in the hidden layers leads not only
to improved generalization but also to better-calibrated estimates of
uncertainty on outlier data. Data that is not from the training distribution
can be recognized by observing that in the hidden layers, it has fewer than the
normal number of neighbors from the predicted class.
Situational Grounding within Multimodal Simulations
Authors: James Pustejovsky, Nikhil Krishnaswamy
Date: 2019-02-05 00:00:00
Tasks: Novel Concepts, Spatial Reasoning

In this paper, we argue that simulation platforms enable a novel type of
embodied spatial reasoning, one facilitated by a formal model of object and
event semantics that renders the continuous quantitative search space of an
open-world, real-time environment tractable. We provide examples for how a
semantically-informed AI system can exploit the precise, numerical information
provided by a game engine to perform qualitative reasoning about objects and
events, facilitate learning novel concepts from data, and communicate with a
human to improve its models and demonstrate its understanding. We argue that
simulation environments, and game engines in particular, bring together many
different notions of "simulation" and many different technologies to provide a
highly-effective platform for developing both AI systems and tools to
experiment in both machine and human intelligence.
Separating value functions across time-scales
Authors: Joshua Romoff, Peter Henderson, Ahmed Touati, Emma Brunskill, Joelle Pineau, Yann Ollivier
Date: 2019-02-05 00:00:00
Tasks: Reinforcement Learning, Reinforcement Learning (RL)

In many finite horizon episodic reinforcement learning (RL) settings, it is desirable to optimize for the undiscounted return - in settings like Atari, for instance, the goal is to collect the most points while staying alive in the long run. Yet, it may be difficult (or even intractable) mathematically to learn with this target. As such, temporal discounting is often applied to optimize over a shorter effective planning horizon. This comes at the risk of potentially biasing the optimization target away from the undiscounted goal. In settings where this bias is unacceptable - where the system must optimize for longer horizons at higher discounts - the target of the value function approximator may increase in variance leading to difficulties in learning. We present an extension of temporal difference (TD) learning, which we call TD($\Delta$), that breaks down a value function into a series of components based on the differences between value functions with smaller discount factors. The separation of a longer horizon value function into these components has useful properties in scalability and performance. We discuss these properties and show theoretic and empirical improvements over standard TD learning in certain settings.
Quantum Sparse Support Vector Machines
Authors: Seyran Saeedi, Tom Arodz
Date: 2019-02-05 00:00:00

We analyze the computational complexity of Quantum Sparse Support Vector Machine, a linear classifier that minimizes the hinge loss and the $L_1$ norm of the feature weights vector and relies on a quantum linear programming solver instead of a classical solver. Sparse SVM leads to sparse models that use only a small fraction of the input features in making decisions, and is especially useful when the total number of features, $p$, approaches or exceeds the number of training samples, $m$. We prove a $\Omega(m)$ worst-case lower bound for computational complexity of any quantum training algorithm relying on black-box access to training samples; quantum sparse SVM has at least linear worst-case complexity. However, we prove that there are realistic scenarios in which a sparse linear classifier is expected to have high accuracy, and can be trained in sublinear time in terms of both the number of training samples and the number of features.
Disguised-Nets: Image Disguising for Privacy-preserving Outsourced Deep Learning
Authors: Sagar Sharma, Keke Chen
Date: 2019-02-05 00:00:00
Tasks: Deep Learning, GPU, image-classification, Image Classification, Privacy Preserving, Privacy Preserving Deep Learning

Deep learning model developers often use cloud GPU resources to experiment
with large data and models that need expensive setups. However, this practice
raises privacy concerns. Adversaries may be interested in: 1) personally
identifiable information or objects encoded in the training images, and 2) the
models trained with sensitive data to launch model-based attacks. Learning deep
neural networks (DNN) from encrypted data is still impractical due to the large
training data and the expensive learning process. A few recent studies have
tried to provide efficient, practical solutions to protect data privacy in
outsourced deep-learning. However, we find out that they are vulnerable under
certain attacks. In this paper, we specifically identify two types of unique
attacks on outsourced deep-learning: 1) the visual re-identification attack on
the training data, and 2) the class membership attack on the learned models,
which can break existing privacy-preserving solutions. We develop an image
disguising approach to address these attacks and design a suite of methods to
evaluate the levels of attack resilience for a privacy-preserving solution for
outsourced deep learning. The experimental results show that our
image-disguising mechanisms can provide a high level of protection against the
two attacks while still generating high-quality DNN models for image
classification.
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI
Authors: Shane T. Mueller, Robert R. Hoffman, William Clancey, Abigail Emrey, Gary Klein
Date: 2019-02-05 00:00:00
Tasks: Articles, Explainable Artificial Intelligence (XAI)

This is an integrative review that address the question, "What makes for a
good explanation?" with reference to AI systems. Pertinent literatures are
vast. Thus, this review is necessarily selective. That said, most of the key
concepts and issues are expressed in this Report. The Report encapsulates the
history of computer science efforts to create systems that explain and instruct
(intelligent tutoring systems and expert systems). The Report expresses the
explainability issues and challenges in modern AI, and presents capsule views
of the leading psychological theories of explanation. Certain articles stand
out by virtue of their particular relevance to XAI, and their methods, results,
and key points are highlighted. It is recommended that AI/XAI researchers be
encouraged to include in their research reports fuller details on their
empirical or experimental methods, in the fashion of experimental psychology
research reports: details on Participants, Instructions, Procedures, Tasks,
Dependent Variables (operational definitions of the measures and metrics),
Independent Variables (conditions), and Control Conditions.
Stabilizing Inputs to Approximated Nonlinear Functions for Inference with Homomorphic Encryption in Deep Neural Networks
Authors: Moustafa AboulAtta, Matthias Ossadnik, Seyed-Ahmad Ahmadi
Date: 2019-02-05 00:00:00

Leveled Homomorphic Encryption (LHE) offers a potential solution that could
allow sectors with sensitive data to utilize the cloud and securely deploy
their models for remote inference with Deep Neural Networks (DNN). However,
this application faces several obstacles due to the limitations of LHE. One of
the main problems is the incompatibility of commonly used nonlinear functions
in DNN with the operations supported by LHE, i.e. addition and multiplication.
As common in LHE approaches, we train a model with a nonlinear function, and
replace it with a low-degree polynomial approximation at inference time on
private data. While this typically leads to approximation errors and loss in
prediction accuracy, we propose a method that reduces this loss to small values
or eliminates it entirely, depending on simple hyper-parameters. This is
achieved by the introduction of a novel and elegantly simple Min-Max
normalization scheme, which scales inputs to nonlinear functions into ranges
with low approximation error. While being intuitive in its concept and trivial
to implement, we empirically show that it offers a stable and effective
approximation solution to nonlinear functions in DNN. In return, this can
enable deeper networks with LHE, and facilitate the development of security-
and privacy-aware analytics applications.