pwc-archive/papers-with-abstracts huggingface dataset:
What Really is Deep Learning Doing?
Authors: Chuyu Xiong
Date: 2017-11-06 00:00:00
Tasks: Deep Learning, Open-Ended Question Answering

Deep learning has achieved a great success in many areas, from computer
vision to natural language processing, to game playing, and much more. Yet,
what deep learning is really doing is still an open question. There are a lot
of works in this direction. For example, [5] tried to explain deep learning by
group renormalization, and [6] tried to explain deep learning from the view of
functional approximation. In order to address this very crucial question, here
we see deep learning from perspective of mechanical learning and learning
machine (see [1], [2]). From this particular angle, we can see deep learning
much better and answer with confidence: What deep learning is really doing? why
it works well, how it works, and how much data is necessary for learning. We
also will discuss advantages and disadvantages of deep learning at the end of
this work.
Improved training for online end-to-end speech recognition systems
Authors: Suyoun Kim, Michael L. Seltzer, Jinyu Li, Rui Zhao
Date: 2017-11-06 00:00:00
Tasks: speech-recognition, Speech Recognition

Achieving high accuracy with end-to-end speech recognizers requires careful
parameter initialization prior to training. Otherwise, the networks may fail to
find a good local optimum. This is particularly true for online networks, such
as unidirectional LSTMs. Currently, the best strategy to train such systems is
to bootstrap the training from a tied-triphone system. However, this is time
consuming, and more importantly, is impossible for languages without a
high-quality pronunciation lexicon. In this work, we propose an initialization
strategy that uses teacher-student learning to transfer knowledge from a large,
well-trained, offline end-to-end speech recognition model to an online
end-to-end model, eliminating the need for a lexicon or any other linguistic
resources. We also explore curriculum learning and label smoothing and show how
they can be combined with the proposed teacher-student learning for further
improvements. We evaluate our methods on a Microsoft Cortana personal assistant
task and show that the proposed method results in a 19 % relative improvement
in word error rate compared to a randomly-initialized baseline system.
Social welfare and profit maximization from revealed preferences
Authors: Ziwei Ji, Ruta Mehta, Matus Telgarsky
Date: 2017-11-06 00:00:00

Consider the seller's problem of finding optimal prices for her $n$
(divisible) goods when faced with a set of $m$ consumers, given that she can
only observe their purchased bundles at posted prices, i.e., revealed
preferences. We study both social welfare and profit maximization with revealed
preferences. Although social welfare maximization is a seemingly non-convex
optimization problem in prices, we show that (i) it can be reduced to a dual
convex optimization problem in prices, and (ii) the revealed preferences can be
interpreted as supergradients of the concave conjugate of valuation, with which
subgradients of the dual function can be computed. We thereby obtain a simple
subgradient-based algorithm for strongly concave valuations and convex cost,
with query complexity $O(m^2/\epsilon^2)$, where $\epsilon$ is the additive
difference between the social welfare induced by our algorithm and the optimum
social welfare. We also study social welfare maximization under the online
setting, specifically the random permutation model, where consumers arrive
one-by-one in a random order. For the case where consumer valuations can be
arbitrary continuous functions, we propose a price posting mechanism that
achieves an expected social welfare up to an additive factor of $O(\sqrt{mn})$
from the maximum social welfare. Finally, for profit maximization (which may be
non-convex in simple cases), we give nearly matching upper and lower bounds on
the query complexity for separable valuations and cost (i.e., each good can be
treated independently).
Off-policy evaluation for slate recommendation
Authors: Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miroslav Dudík, John Langford, Damien Jose, Imed Zitouni
Date: 2016-05-16 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Learning-To-Rank, Off-policy evaluation

This paper studies the evaluation of policies that recommend an ordered set
of items (e.g., a ranking) based on some context---a common scenario in web
search, ads, and recommendation. We build on techniques from combinatorial
bandits to introduce a new practical estimator that uses logged data to
estimate a policy's performance. A thorough empirical evaluation on real-world
data reveals that our estimator is accurate in a variety of settings, including
as a subroutine in a learning-to-rank task, where it achieves competitive
performance. We derive conditions under which our estimator is unbiased---these
conditions are weaker than prior heuristics for slate evaluation---and
experimentally demonstrate a smaller bias than parametric approaches, even when
these conditions are violated. Finally, our theory and experiments also show
exponential savings in the amount of required data compared with general
unbiased estimators.
Unsupervised Learning of Semantic Audio Representations
Authors: Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel P. W. Ellis, Shawn Hershey, Jiayang Liu, R. Channing Moore, Rif A. Saurous
Date: 2017-11-06 00:00:00
Tasks: Audio Classification, Classification, General Classification, Retrieval, Triplet

Even in the absence of any explicit semantic annotation, vast collections of
audio recordings provide valuable information for learning the categorical
structure of sounds. We consider several class-agnostic semantic constraints
that apply to unlabeled nonspeech audio: (i) noise and translations in time do
not change the underlying sound category, (ii) a mixture of two sound events
inherits the categories of the constituents, and (iii) the categories of events
in close temporal proximity are likely to be the same or related. Without
labels to ground them, these constraints are incompatible with classification
loss functions. However, they may still be leveraged to identify geometric
inequalities needed for triplet loss-based training of convolutional neural
networks. The result is low-dimensional embeddings of the input spectrograms
that recover 41% and 84% of the performance of their fully-supervised
counterparts when applied to downstream query-by-example sound retrieval and
sound event classification tasks, respectively. Moreover, in
limited-supervision settings, our unsupervised embeddings double the
state-of-the-art classification performance.
Towards Language-Universal End-to-End Speech Recognition
Authors: Suyoun Kim, Michael L. Seltzer
Date: 2017-11-06 00:00:00
Tasks: Multi-Task Learning, speech-recognition, Speech Recognition

Building speech recognizers in multiple languages typically involves
replicating a monolingual training recipe for each language, or utilizing a
multi-task learning approach where models for different languages have separate
output labels but share some internal parameters. In this work, we exploit
recent progress in end-to-end speech recognition to create a single
multilingual speech recognition system capable of recognizing any of the
languages seen in training. To do so, we propose the use of a universal
character set that is shared among all languages. We also create a
language-specific gating mechanism within the network that can modulate the
network's internal representations in a language-specific way. We evaluate our
proposed approach on the Microsoft Cortana task across three languages and show
that our system outperforms both the individual monolingual systems and systems
built with a multi-task learning approach. We also show that this model can be
used to initialize a monolingual speech recognizer, and can be used to create a
bilingual model for use in code-switching scenarios.
Regret Bounds and Regimes of Optimality for User-User and Item-Item Collaborative Filtering
Authors: Guy Bresler, Mina Karzand
Date: 2017-11-06 00:00:00
Tasks: Collaborative Filtering, Recommendation Systems, Vocal Bursts Type Prediction

We consider an online model for recommendation systems, with each user being recommended an item at each time-step and providing 'like' or 'dislike' feedback. Each user may be recommended a given item at most once. A latent variable model specifies the user preferences: both users and items are clustered into types. All users of a given type have identical preferences for the items, and similarly, items of a given type are either all liked or all disliked by a given user. We assume that the matrix encoding the preferences of each user type for each item type is randomly generated; in this way, the model captures structure in both the item and user spaces, the amount of structure depending on the number of each of the types. The measure of performance of the recommendation system is the expected number of disliked recommendations per user, defined as expected regret. We propose two algorithms inspired by user-user and item-item collaborative filtering (CF), modified to explicitly make exploratory recommendations, and prove performance guarantees in terms of their expected regret. For two regimes of model parameters, with structure only in item space or only in user space, we prove information-theoretic lower bounds on regret that match our upper bounds up to logarithmic factors. Our analysis elucidates system operating regimes in which existing CF algorithms are nearly optimal.
Exploring Sparsity in Recurrent Neural Networks
Authors: Sharan Narang, Erich Elsen, Gregory Diamos, Shubho Sengupta
Date: 2017-04-17 00:00:00

Recurrent Neural Networks (RNN) are widely used to solve a variety of
problems and as the quantity of data and the amount of available compute have
increased, so have model sizes. The number of parameters in recent
state-of-the-art networks makes them hard to deploy, especially on mobile
phones and embedded devices. The challenge is due to both the size of the model
and the time it takes to evaluate it. In order to deploy these RNNs
efficiently, we propose a technique to reduce the parameters of a network by
pruning weights during the initial training of the network. At the end of
training, the parameters of the network are sparse while accuracy is still
close to the original dense neural network. The network size is reduced by 8x
and the time required to train the model remains constant. Additionally, we can
prune a larger dense network to achieve better than baseline performance while
still reducing the total number of parameters significantly. Pruning RNNs
reduces the size of the model and can also help achieve significant inference
time speed-up using sparse matrix multiply. Benchmarks show that using our
technique model size can be reduced by 90% and speed-up is around 2x to 7x.
Deep Neural Networks
Authors: Randall Balestriero, Richard Baraniuk
Date: 2017-10-25 00:00:00
Tasks: image-classification, Image Classification, Object Tracking, speech-recognition, Speech Recognition

Deep Neural Networks (DNNs) are universal function approximators providing
state-of- the-art solutions on wide range of applications. Common perceptual
tasks such as speech recognition, image classification, and object tracking are
now commonly tackled via DNNs. Some fundamental problems remain: (1) the lack
of a mathematical framework providing an explicit and interpretable
input-output formula for any topology, (2) quantification of DNNs stability
regarding adversarial examples (i.e. modified inputs fooling DNN predictions
whilst undetectable to humans), (3) absence of generalization guarantees and
controllable behaviors for ambiguous patterns, (4) leverage unlabeled data to
apply DNNs to domains where expert labeling is scarce as in the medical field.
Answering those points would provide theoretical perspectives for further
developments based on a common ground. Furthermore, DNNs are now deployed in
tremendous societal applications, pushing the need to fill this theoretical gap
to ensure control, reliability, and interpretability.
Nonconvex penalties with analytical solutions for one-bit compressive sensing
Authors: Xiaolin Huang, Ming Yan
Date: 2017-06-04 00:00:00
Tasks: Compressive Sensing, Learning Theory

One-bit measurements widely exist in the real world, and they can be used to
recover sparse signals. This task is known as the problem of learning
halfspaces in learning theory and one-bit compressive sensing (1bit-CS) in
signal processing. In this paper, we propose novel algorithms based on both
convex and nonconvex sparsity-inducing penalties for robust 1bit-CS. We provide
a sufficient condition to verify whether a solution is globally optimal or not.
Then we show that the globally optimal solution for positive homogeneous
penalties can be obtained in two steps: a proximal operator and a normalization
step. For several nonconvex penalties, including minimax concave penalty (MCP),
$\ell_0$ norm, and sorted $\ell_1$ penalty, we provide fast algorithms for
finding the analytical solutions by solving the dual problem. Specifically, our
algorithm is more than $200$ times faster than the existing algorithm for MCP.
Its efficiency is comparable to the algorithm for the $\ell_1$ penalty in time,
while its performance is much better. Among these penalties, the sorted
$\ell_1$ penalty is most robust to noise in different settings.
Learned D-AMP: Principled Neural Network based Compressive Image Recovery
Authors: Christopher A. Metzler, Ali Mousavi, Richard G. Baraniuk
Date: 2017-04-21 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: Denoising, GPU

Compressive image recovery is a challenging problem that requires fast and
accurate algorithms. Recently, neural networks have been applied to this
problem with promising results. By exploiting massively parallel GPU processing
architectures and oodles of training data, they can run orders of magnitude
faster than existing techniques. However, these methods are largely
unprincipled black boxes that are difficult to train and often-times specific
to a single measurement matrix.
  It was recently demonstrated that iterative sparse-signal-recovery algorithms
can be "unrolled" to form interpretable deep networks. Taking inspiration from
this work, we develop a novel neural network architecture that mimics the
behavior of the denoising-based approximate message passing (D-AMP) algorithm.
We call this new network Learned D-AMP (LDAMP).
  The LDAMP network is easy to train, can be applied to a variety of different
measurement matrices, and comes with a state-evolution heuristic that
accurately predicts its performance. Most importantly, it outperforms the
state-of-the-art BM3D-AMP and NLR-CS algorithms in terms of both accuracy and
run time. At high resolutions, and when used with sensing matrices that have
fast implementations, LDAMP runs over $50\times$ faster than BM3D-AMP and
hundreds of times faster than NLR-CS.
VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
Authors: Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, Charles Sutton
Date: 2017-05-22 00:00:00
Proceeding: NeurIPS 2017 12

Deep generative models provide powerful tools for distributions over
complicated manifolds, such as those of natural images. But many of these
methods, including generative adversarial networks (GANs), can be difficult to
train, in part because they are prone to mode collapse, which means that they
characterize only a few modes of the true distribution. To address this, we
introduce VEEGAN, which features a reconstructor network, reversing the action
of the generator by mapping from data to noise. Our training objective retains
the original asymptotic consistency guarantee of GANs, and can be interpreted
as a novel autoencoder loss over the noise. In sharp contrast to a traditional
autoencoder over data points, VEEGAN does not require specifying a loss
function over the data, but rather only over the representations, which are
standard normal by assumption. On an extensive set of synthetic and real world
image datasets, VEEGAN indeed resists mode collapsing to a far greater extent
than other recent GAN variants, and produces more realistic samples.
User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
Authors: Arnak S. Dalalyan, Avetik G. Karagulyan
Date: 2017-09-29 00:00:00

In this paper, we study the problem of sampling from a given probability density function that is known to be smooth and strongly log-concave. We analyze several methods of approximate sampling based on discretizations of the (highly overdamped) Langevin diffusion and establish guarantees on its error measured in the Wasserstein-2 distance. Our guarantees improve or extend the state-of-the-art results in three directions. First, we provide an upper bound on the error of the first-order Langevin Monte Carlo (LMC) algorithm with optimized varying step-size. This result has the advantage of being horizon free (we do not need to know in advance the target precision) and to improve by a logarithmic factor the corresponding result for the constant step-size. Second, we study the case where accurate evaluations of the gradient of the log-density are unavailable, but one can have access to approximations of the aforementioned gradient. In such a situation, we consider both deterministic and stochastic approximations of the gradient and provide an upper bound on the sampling error of the first-order LMC that quantifies the impact of the gradient evaluation inaccuracies. Third, we establish upper bounds for two versions of the second-order LMC, which leverage the Hessian of the log-density. We provide nonasymptotic guarantees on the sampling error of these second-order LMCs. These guarantees reveal that the second-order LMC algorithms improve on the first-order LMC in ill-conditioned settings.
Deep Function Machines: Generalized Neural Networks for Topological Layer Expression
Authors: William H. Guss
Date: 2016-12-14 00:00:00
Proceeding: ICLR 2018 1

In this paper we propose a generalization of deep neural networks called deep
function machines (DFMs). DFMs act on vector spaces of arbitrary (possibly
infinite) dimension and we show that a family of DFMs are invariant to the
dimension of input data; that is, the parameterization of the model does not
directly hinge on the quality of the input (eg. high resolution images). Using
this generalization we provide a new theory of universal approximation of
bounded non-linear operators between function spaces. We then suggest that DFMs
provide an expressive framework for designing new neural network layer types
with topological considerations in mind. Finally, we introduce a novel
architecture, RippLeNet, for resolution invariant computer vision, which
empirically achieves state of the art invariance.
Schema Independent Relational Learning
Authors: Jose Picado, Arash Termehchy, Alan Fern, Parisa Ataei
Date: 2015-08-16 00:00:00
Tasks: Novel Concepts, Relational Reasoning

Learning novel concepts and relations from relational databases is an
important problem with many applications in database systems and machine
learning. Relational learning algorithms learn the definition of a new relation
in terms of existing relations in the database. Nevertheless, the same data set
may be represented under different schemas for various reasons, such as
efficiency, data quality, and usability. Unfortunately, the output of current
relational learning algorithms tends to vary quite substantially over the
choice of schema, both in terms of learning accuracy and efficiency. This
variation complicates their off-the-shelf application. In this paper, we
introduce and formalize the property of schema independence of relational
learning algorithms, and study both the theoretical and empirical dependence of
existing algorithms on the common class of (de) composition schema
transformations. We study both sample-based learning algorithms, which learn
from sets of labeled examples, and query-based algorithms, which learn by
asking queries to an oracle. We prove that current relational learning
algorithms are generally not schema independent. For query-based learning
algorithms we show that the (de) composition transformations influence their
query complexity. We propose Castor, a sample-based relational learning
algorithm that achieves schema independence by leveraging data dependencies. We
support the theoretical results with an empirical study that demonstrates the
schema dependence/independence of several algorithms on existing benchmark and
real-world datasets under (de) compositions.
Adaptive Bayesian Sampling with Monte Carlo EM
Authors: Anirban Roychowdhury, Srinivasan Parthasarathy
Date: 2017-11-06 00:00:00
Proceeding: NeurIPS 2017 12

We present a novel technique for learning the mass matrices in samplers
obtained from discretized dynamics that preserve some energy function. Existing
adaptive samplers use Riemannian preconditioning techniques, where the mass
matrices are functions of the parameters being sampled. This leads to
significant complexities in the energy reformulations and resultant dynamics,
often leading to implicit systems of equations and requiring inversion of
high-dimensional matrices in the leapfrog steps. Our approach provides a
simpler alternative, by using existing dynamics in the sampling step of a Monte
Carlo EM framework, and learning the mass matrices in the M step with a novel
online technique. We also propose a way to adaptively set the number of samples
gathered in the E step, using sampling error estimates from the leapfrog
dynamics. Along with a novel stochastic sampler based on Nos\'{e}-Poincar\'{e}
dynamics, we use this framework with standard Hamiltonian Monte Carlo (HMC) as
well as newer stochastic algorithms such as SGHMC and SGNHT, and show strong
performance on synthetic and real high-dimensional sampling scenarios; we
achieve sampling accuracies comparable to Riemannian samplers while being
significantly faster.
Weighted Transformer Network for Machine Translation
Authors: Karim Ahmed, Nitish Shirish Keskar, Richard Socher
Date: 2017-11-06 00:00:00
Proceeding: ICLR 2018 1
Tasks: Machine Translation, Translation

State-of-the-art results on neural machine translation often use attentional
sequence-to-sequence models with some form of convolution or recursion. Vaswani
et al. (2017) propose a new architecture that avoids recurrence and convolution
completely. Instead, it uses only self-attention and feed-forward layers. While
the proposed architecture achieves state-of-the-art results on several machine
translation tasks, it requires a large number of parameters and training
iterations to converge. We propose Weighted Transformer, a Transformer with
modified attention layers, that not only outperforms the baseline network in
BLEU score but also converges 15-40% faster. Specifically, we replace the
multi-head attention by multiple self-attention branches that the model learns
to combine during the training process. Our model improves the state-of-the-art
performance by 0.5 BLEU points on the WMT 2014 English-to-German translation
task and by 0.4 on the English-to-French translation task.