pwc-archive/papers-with-abstracts huggingface dataset:
Global Fitting of the Response Surface via Estimating Multiple Contours of a Simulator
Authors: Feng Yang, C. Devon Lin, Pritam Ranjan
Date: 2019-02-04 00:00:00
Tasks: Decision Making, Experimental Design

Computer simulators are nowadays widely used to understand complex physical
systems in many areas such as aerospace, renewable energy, climate modeling,
and manufacturing. One fundamental issue in the study of computer simulators is
known as experimental design, that is, how to select the input settings where
the computer simulator is run and the corresponding response is collected.
Extra care should be taken in the selection process because computer simulators
can be computationally expensive to run. The selection shall acknowledge and
achieve the goal of the analysis. This article focuses on the goal of producing
more accurate prediction which is important for risk assessment and decision
making. We propose two new methods of design approaches that sequentially
select input settings to achieve this goal. The approaches make novel
applications of simultaneous and sequential contour estimations. Numerical
examples are employed to demonstrate the effectiveness of the proposed
approaches.
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
Authors: R. Thomas McCoy, Ellie Pavlick, Tal Linzen
Date: 2019-02-04 00:00:00
Proceeding: ACL 2019 7
Tasks: Natural Language Inference, Sentence

A machine learning system can score well on a given test set by relying on heuristics that are effective for frequent example types but break down in more challenging cases. We study this issue within natural language inference (NLI), the task of determining whether one sentence entails another. We hypothesize that statistical NLI models may adopt three fallible syntactic heuristics: the lexical overlap heuristic, the subsequence heuristic, and the constituent heuristic. To determine whether models have adopted these heuristics, we introduce a controlled evaluation set called HANS (Heuristic Analysis for NLI Systems), which contains many examples where the heuristics fail. We find that models trained on MNLI, including BERT, a state-of-the-art model, perform very poorly on HANS, suggesting that they have indeed adopted these heuristics. We conclude that there is substantial room for improvement in NLI systems, and that the HANS dataset can motivate and measure progress in this area
Study of Robust Distributed Diffusion RLS Algorithms with Side Information for Adaptive Networks
Authors: Y. Yu, H. Zhao, R. C. de Lamare, Y. Zakharov, L. Lu
Date: 2019-02-04 00:00:00

This work develops robust diffusion recursive least squares algorithms to
mitigate the performance degradation often experienced in networks of agents in
the presence of impulsive noise. The first algorithm minimizes an exponentially
weighted least-squares cost function subject to a time-dependent constraint on
the squared norm of the intermediate update at each node. A recursive strategy
for computing the constraint is proposed using side information from the
neighboring nodes to further improve the robustness. We also analyze the
mean-square convergence behavior of the proposed algorithm. The second proposed
algorithm is a modification of the first one based on the dichotomous
coordinate descent iterations. It has a performance similar to that of the
former, however its complexity is significantly lower especially when input
regressors of agents have a shift structure and it is well suited to practical
implementation. Simulations show the superiority of the proposed algorithms
over previously reported techniques in various impulsive noise scenarios.
BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services
Authors: Amir Erfan Eshratifar, Amirhossein Esmaili, Massoud Pedram
Date: 2019-02-04 00:00:00
Tasks: Cloud Computing

Recent studies have shown the latency and energy consumption of deep neural
networks can be significantly improved by splitting the network between the
mobile device and cloud. This paper introduces a new deep learning
architecture, called BottleNet, for reducing the feature size needed to be sent
to the cloud. Furthermore, we propose a training method for compensating for
the potential accuracy loss due to the lossy compression of features before
transmitting them to the cloud. BottleNet achieves on average 30x improvement
in end-to-end latency and 40x improvement in mobile energy consumption compared
to the cloud-only approach with negligible accuracy loss.
Is There an Analog of Nesterov Acceleration for MCMC?
Authors: Yi-An Ma, Niladri Chatterji, Xiang Cheng, Nicolas Flammarion, Peter Bartlett, Michael. I. Jordan
Date: 2019-02-04 00:00:00

We formulate gradient-based Markov chain Monte Carlo (MCMC) sampling as optimization on the space of probability measures, with Kullback-Leibler (KL) divergence as the objective functional. We show that an underdamped form of the Langevin algorithm performs accelerated gradient descent in this metric. To characterize the convergence of the algorithm, we construct a Lyapunov functional and exploit hypocoercivity of the underdamped Langevin algorithm. As an application, we show that accelerated rates can be obtained for a class of nonconvex functions with the Langevin algorithm.
Minimax experimental design: Bridging the gap between statistical and worst-case approaches to least squares regression
Authors: Michał Dereziński, Kenneth L. Clarkson, Michael W. Mahoney, Manfred K. Warmuth
Date: 2019-02-04 00:00:00
Tasks: Experimental Design, regression

In experimental design, we are given a large collection of vectors, each with
a hidden response value that we assume derives from an underlying linear model,
and we wish to pick a small subset of the vectors such that querying the
corresponding responses will lead to a good estimator of the model. A classical
approach in statistics is to assume the responses are linear, plus zero-mean
i.i.d. Gaussian noise, in which case the goal is to provide an unbiased
estimator with smallest mean squared error (A-optimal design). A related
approach, more common in computer science, is to assume the responses are
arbitrary but fixed, in which case the goal is to estimate the least squares
solution using few responses, as quickly as possible, for worst-case inputs.
Despite many attempts, characterizing the relationship between these two
approaches has proven elusive. We address this by proposing a framework for
experimental design where the responses are produced by an arbitrary unknown
distribution. We show that there is an efficient randomized experimental design
procedure that achieves strong variance bounds for an unbiased estimator using
few responses in this general model. Nearly tight bounds for the classical
A-optimality criterion, as well as improved bounds for worst-case responses,
emerge as special cases of this result. In the process, we develop a new
algorithm for a joint sampling distribution called volume sampling, and we
propose a new i.i.d. importance sampling method: inverse score sampling. A key
novelty of our analysis is in developing new expected error bounds for
worst-case regression by controlling the tail behavior of i.i.d. sampling via
the jointness of volume sampling. Our result motivates a new minimax-optimality
criterion for experimental design which can be viewed as an extension of both
A-optimal design and sampling for worst-case regression.
Improving Question Answering with External Knowledge
Authors: Xiaoman Pan, Kai Sun, Dian Yu, Jianshu Chen, Heng Ji, Claire Cardie, Dong Yu
Date: 2019-02-03 00:00:00
Proceeding: WS 2019 11
Tasks: ARC, Multiple-choice, Question Answering

We focus on multiple-choice question answering (QA) tasks in subject areas such as science, where we require both broad background knowledge and the facts from the given subject-area reference corpus. In this work, we explore simple yet effective methods for exploiting two sources of external knowledge for subject-area QA. The first enriches the original subject-area reference corpus with relevant text snippets extracted from an open-domain resource (i.e., Wikipedia) that cover potentially ambiguous concepts in the question and answer options. As in other QA research, the second method simply increases the amount of training data by appending additional in-domain subject-area instances. Experiments on three challenging multiple-choice science QA tasks (i.e., ARC-Easy, ARC-Challenge, and OpenBookQA) demonstrate the effectiveness of our methods: in comparison to the previous state-of-the-art, we obtain absolute gains in accuracy of up to 8.1%, 13.0%, and 12.8%, respectively. While we observe consistent gains when we introduce knowledge from Wikipedia, we find that employing additional QA training instances is not uniformly helpful: performance degrades when the added instances exhibit a higher level of difficulty than the original training data. As one of the first studies on exploiting unstructured external knowledge for subject-area QA, we hope our methods, observations, and discussion of the exposed limitations may shed light on further developments in the area.
Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
Authors: Hisham Husain, Richard Nock, Robert C. Williamson
Date: 2019-02-03 00:00:00
Tasks: Generalization Bounds

Since the introduction of Generative Adversarial Networks (GANs) and
Variational Autoencoders (VAE), the literature on generative modelling has
witnessed an overwhelming resurgence. The impressive, yet elusive empirical
performance of GANs has lead to the rise of many GAN-VAE hybrids, with the
hopes of GAN level performance and additional benefits of VAE, such as an
encoder for feature reduction, which is not offered by GANs. Recently, the
Wasserstein Autoencoder (WAE) was proposed, achieving performance similar to
that of GANs, yet it is still unclear whether the two are fundamentally
different or can be further improved into a unified model. In this work, we
study the $f$-GAN and WAE models and make two main discoveries. First, we find
that the $f$-GAN and WAE objectives partake in a primal-dual relationship and
are equivalent under some assumptions, which then allows us to explicate the
success of WAE. Second, the equivalence result allows us to, for the first
time, prove generalization bounds for Autoencoder models, which is a pertinent
problem when it comes to theoretical analyses of generative models.
Furthermore, we show that the WAE objective is related to other statistical
quantities such as the $f$-divergence and in particular, upper bounded by the
Wasserstein distance, which then allows us to tap into existing efficient
(regularized) optimal transport solvers. Our findings thus present the first
primal-dual relationship between GANs and Autoencoder models, comment on
generalization abilities and make a step towards unifying these models.
A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
Authors: Yifang Chen, Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei
Date: 2019-02-03 00:00:00
Tasks: Multi-Armed Bandits

We propose the first contextual bandit algorithm that is parameter-free, efficient, and optimal in terms of dynamic regret. Specifically, our algorithm achieves dynamic regret $\mathcal{O}(\min\{\sqrt{ST}, \Delta^{\frac{1}{3}}T^{\frac{2}{3}}\})$ for a contextual bandit problem with $T$ rounds, $S$ switches and $\Delta$ total variation in data distributions. Importantly, our algorithm is adaptive and does not need to know $S$ or $\Delta$ ahead of time, and can be implemented efficiently assuming access to an ERM oracle. Our results strictly improve the $\mathcal{O}(\min \{S^{\frac{1}{4}}T^{\frac{3}{4}}, \Delta^{\frac{1}{5}}T^{\frac{4}{5}}\})$ bound of (Luo et al., 2018), and greatly generalize and improve the $\mathcal{O}(\sqrt{ST})$ result of (Auer et al, 2018) that holds only for the two-armed bandit problem without contextual information. The key novelty of our algorithm is to introduce replay phases, in which the algorithm acts according to its previous decisions for a certain amount of time in order to detect non-stationarity while maintaining a good balance between exploration and exploitation.
Universal Lemmatizer: A Sequence to Sequence Model for Lemmatizing Universal Dependencies Treebanks
Authors: Jenna Kanerva, Filip Ginter, Tapio Salakoski
Date: 2019-02-03 00:00:00
Tasks: Data Augmentation, LEMMA, Lemmatization

In this paper we present a novel lemmatization method based on a sequence-to-sequence neural network architecture and morphosyntactic context representation. In the proposed method, our context-sensitive lemmatizer generates the lemma one character at a time based on the surface form characters and its morphosyntactic features obtained from a morphological tagger. We argue that a sliding window context representation suffers from sparseness, while in majority of cases the morphosyntactic features of a word bring enough information to resolve lemma ambiguities while keeping the context representation dense and more practical for machine learning systems. Additionally, we study two different data augmentation methods utilizing autoencoder training and morphological transducers especially beneficial for low resource languages. We evaluate our lemmatizer on 52 different languages and 76 different treebanks, showing that our system outperforms all latest baseline systems. Compared to the best overall baseline, UDPipe Future, our system outperforms it on 62 out of 76 treebanks reducing errors on average by 19% relative. The lemmatizer together with all trained models is made available as a part of the Turku-neural-parsing-pipeline under the Apache 2.0 license.
Deep Autotuner: A Data-Driven Approach to Natural-Sounding Pitch Correction for Singing Voice in Karaoke Performances
Authors: Sanna Wager, George Tzanetakis, Cheng-i Wang, Lijiang Guo, Aswin Sivaraman, Minje Kim
Date: 2019-02-03 00:00:00

We describe a machine-learning approach to pitch correcting a solo singing
performance in a karaoke setting, where the solo voice and accompaniment are on
separate tracks. The proposed approach addresses the situation where no musical
score of the vocals nor the accompaniment exists: It predicts the amount of
correction from the relationship between the spectral contents of the vocal and
accompaniment tracks. Hence, the pitch shift in cents suggested by the model
can be used to make the voice sound in tune with the accompaniment. This
approach differs from commercially used automatic pitch correction systems,
where notes in the vocal tracks are shifted to be centered around notes in a
user-defined score or mapped to the closest pitch among the twelve
equal-tempered scale degrees. We train the model using a dataset of 4,702
amateur karaoke performances selected for good intonation. We present a
Convolutional Gated Recurrent Unit (CGRU) model to accomplish this task. This
method can be extended into unsupervised pitch correction of a vocal
performance, popularly referred to as autotuning.
Stochastic first-order methods: non-asymptotic and computer-aided analyses via potential functions
Authors: Adrien Taylor, Francis Bach
Date: 2019-02-03 00:00:00

We provide a novel computer-assisted technique for systematically analyzing first-order methods for optimization. In contrast with previous works, the approach is particularly suited for handling sublinear convergence rates and stochastic oracles. The technique relies on semidefinite programming and potential functions. It allows simultaneously obtaining worst-case guarantees on the behavior of those algorithms, and assisting in choosing appropriate parameters for tuning their worst-case performances. The technique also benefits from comfortable tightness guarantees, meaning that unsatisfactory results can be improved only by changing the setting. We use the approach for analyzing deterministic and stochastic first-order methods under different assumptions on the nature of the stochastic noise. Among others, we treat unstructured noise with bounded variance, different noise models arising in over-parametrized expectation minimization problems, and randomized block-coordinate descent schemes.
Bayesian optimization in ab initio nuclear physics
Authors: A. Ekström, C. Forssén, C. Dimitrakakis, D. Dubhashi, H. T. Johansson, A. S. Muhammad, H. Salomonsson, A. Schliep
Date: 2019-02-03 00:00:00
Tasks: Bayesian Optimization

Theoretical models of the strong nuclear interaction contain unknown coupling
constants (parameters) that must be determined using a pool of calibration
data. In cases where the models are complex, leading to time consuming
calculations, it is particularly challenging to systematically search the
corresponding parameter domain for the best fit to the data. In this paper, we
explore the prospect of applying Bayesian optimization to constrain the
coupling constants in chiral effective field theory descriptions of the nuclear
interaction. We find that Bayesian optimization performs rather well with
low-dimensional parameter domains and foresee that it can be particularly
useful for optimization of a smaller set of coupling constants. A specific
example could be the determination of leading three-nucleon forces using data
from finite nuclei or three-nucleon scattering experiments.
Depthwise Convolution is All You Need for Learning Multiple Visual Domains
Authors: Yunhui Guo, Yandong Li, Rogerio Feris, Liqiang Wang, Tajana Rosing
Date: 2019-02-03 00:00:00
Tasks: All, Continual Learning

There is a growing interest in designing models that can deal with images
from different visual domains. If there exists a universal structure in
different visual domains that can be captured via a common parameterization,
then we can use a single model for all domains rather than one model per
domain. A model aware of the relationships between different domains can also
be trained to work on new domains with less resources. However, to identify the
reusable structure in a model is not easy. In this paper, we propose a
multi-domain learning architecture based on depthwise separable convolution.
The proposed approach is based on the assumption that images from different
domains share cross-channel correlations but have domain-specific spatial
correlations. The proposed model is compact and has minimal overhead when being
applied to new domains. Additionally, we introduce a gating mechanism to
promote soft sharing between different domains. We evaluate our approach on
Visual Decathlon Challenge, a benchmark for testing the ability of multi-domain
models. The experiments show that our approach can achieve the highest score
while only requiring 50% of the parameters compared with the state-of-the-art
approaches.
Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
Authors: Yao Zhang, Alpha A. Lee
Date: 2019-02-03 00:00:00
Tasks: Active Learning, Drug Discovery, Representation Learning, Uncertainty Quantification

Predicting bioactivity and physical properties of small molecules is a central challenge in drug discovery. Deep learning is becoming the method of choice but studies to date focus on mean accuracy as the main metric. However, to replace costly and mission-critical experiments by models, a high mean accuracy is not enough: Outliers can derail a discovery campaign, thus models need reliably predict when it will fail, even when the training data is biased; experiments are expensive, thus models need to be data-efficient and suggest informative training sets using active learning. We show that uncertainty quantification and active learning can be achieved by Bayesian semi-supervised graph convolutional neural networks. The Bayesian approach estimates uncertainty in a statistically principled way through sampling from the posterior distribution. Semi-supervised learning disentangles representation learning and regression, keeping uncertainty estimates accurate in the low data limit and allowing the model to start active learning from a small initial pool of training data. Our study highlights the promise of Bayesian deep learning for chemistry.
Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learning
Authors: R. Srikant, Lei Ying
Date: 2019-02-03 00:00:00

We consider the dynamics of a linear stochastic approximation algorithm
driven by Markovian noise, and derive finite-time bounds on the moments of the
error, i.e., deviation of the output of the algorithm from the equilibrium
point of an associated ordinary differential equation (ODE). We obtain
finite-time bounds on the mean-square error in the case of constant step-size
algorithms by considering the drift of an appropriately chosen Lyapunov
function. The Lyapunov function can be interpreted either in terms of Stein's
method to obtain bounds on steady-state performance or in terms of Lyapunov
stability theory for linear ODEs. We also provide a comprehensive treatment of
the moments of the square of the 2-norm of the approximation error. Our
analysis yields the following results: (i) for a given step-size, we show that
the lower-order moments can be made small as a function of the step-size and
can be upper-bounded by the moments of a Gaussian random variable; (ii) we show
that the higher-order moments beyond a threshold may be infinite in
steady-state; and (iii) we characterize the number of samples needed for the
finite-time bounds to be of the same order as the steady-state bounds. As a
by-product of our analysis, we also solve the open problem of obtaining
finite-time bounds for the performance of temporal difference learning
algorithms with linear function approximation and a constant step-size, without
requiring a projection step or an i.i.d. noise assumption.
MICIK: MIning Cross-Layer Inherent Similarity Knowledge for Deep Model Compression
Authors: Jie Zhang, Xiaolong Wang, Dawei Li, Shalini Ghosh, Abhishek Kolagunda, Yalin Wang
Date: 2019-02-03 00:00:00
Tasks: Knowledge Distillation, Model Compression

State-of-the-art deep model compression methods exploit the low-rank
approximation and sparsity pruning to remove redundant parameters from a
learned hidden layer. However, they process each hidden layer individually
while neglecting the common components across layers, and thus are not able to
fully exploit the potential redundancy space for compression. To solve the
above problem and enable further compression of a model, removing the
cross-layer redundancy and mining the layer-wise inheritance knowledge is
necessary. In this paper, we introduce a holistic model compression framework,
namely MIning Cross-layer Inherent similarity Knowledge (MICIK), to fully
excavate the potential redundancy space. The proposed MICIK framework
simultaneously, (1) learns the common and unique weight components across deep
neural network layers to increase compression rate; (2) preserves the inherent
similarity knowledge of nearby layers and distant layers to minimize the
accuracy loss and (3) can be complementary to other existing compression
techniques such as knowledge distillation. Extensive experiments on large-scale
convolutional neural networks demonstrate that MICIK is superior over
state-of-the-art model compression approaches with 16X parameter reduction on
VGG-16 and 6X on GoogLeNet, all without accuracy loss.