pwc-archive/papers-with-abstracts huggingface dataset:
ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
Authors: Amir Gholami, Kurt Keutzer, George Biros
Date: 2019-02-27 00:00:00
Tasks: Clustering Multivariate Time Series, Multivariate Time Series Imputation

Residual neural networks can be viewed as the forward Euler discretization of an Ordinary Differential Equation (ODE) with a unit time step. This has recently motivated researchers to explore other discretization approaches and train ODE based networks. However, an important challenge of neural ODEs is their prohibitive memory cost during gradient backpropogation. Recently a method proposed in [8], claimed that this memory overhead can be reduced from O(LN_t), where N_t is the number of time steps, down to O(L) by solving forward ODE backwards in time, where L is the depth of the network. However, we will show that this approach may lead to several problems: (i) it may be numerically unstable for ReLU/non-ReLU activations and general convolution operators, and (ii) the proposed optimize-then-discretize approach may lead to divergent training due to inconsistent gradients for small time step sizes. We discuss the underlying problems, and to address them we propose ANODE, an Adjoint based Neural ODE framework which avoids the numerical instability related problems noted above, and provides unconditionally accurate gradients. ANODE has a memory footprint of O(L) + O(N_t), with the same computational cost as reversing ODE solve. We furthermore, discuss a memory efficient algorithm which can further reduce this footprint with a trade-off of additional computational cost. We show results on Cifar-10/100 datasets using ResNet and SqueezeNext neural networks.
Representing Formal Languages: A Comparison Between Finite Automata and Recurrent Neural Networks
Authors: Joshua J. Michalenko, Ameesh Shah, Abhinav Verma, Richard G. Baraniuk, Swarat Chaudhuri, Ankit B. Patel
Date: 2019-02-27 00:00:00
Tasks: Clustering

We investigate the internal representations that a recurrent neural network
(RNN) uses while learning to recognize a regular formal language. Specifically,
we train a RNN on positive and negative examples from a regular language, and
ask if there is a simple decoding function that maps states of this RNN to
states of the minimal deterministic finite automaton (MDFA) for the language.
Our experiments show that such a decoding function indeed exists, and that it
maps states of the RNN not to MDFA states, but to states of an {\em
abstraction} obtained by clustering small sets of MDFA states into
"superstates". A qualitative analysis reveals that the abstraction often has a
simple interpretation. Overall, the results suggest a strong structural
relationship between internal representations used by RNNs and finite automata,
and explain the well-known ability of RNNs to recognize formal grammatical
structure.
A Framework for Decoding Event-Related Potentials from Text
Authors: Shaorong Yan, Aaron Steven White
Date: 2019-02-27 00:00:00
Proceeding: WS 2019 6
Tasks: Language Modeling, Language Modelling, Sentence, Word Embeddings

We propose a novel framework for modeling event-related potentials (ERPs)
collected during reading that couples pre-trained convolutional decoders with a
language model. Using this framework, we compare the abilities of a variety of
existing and novel sentence processing models to reconstruct ERPs. We find that
modern contextual word embeddings underperform surprisal-based models but that,
combined, the two outperform either on its own.
Training Variational Autoencoders with Buffered Stochastic Variational Inference
Authors: Rui Shu, Hung H. Bui, Jay Whang, Stefano Ermon
Date: 2019-02-27 00:00:00
Tasks: Variational Inference

The recognition network in deep latent variable models such as variational
autoencoders (VAEs) relies on amortized inference for efficient posterior
approximation that can scale up to large datasets. However, this technique has
also been demonstrated to select suboptimal variational parameters, often
resulting in considerable additional error called the amortization gap. To
close the amortization gap and improve the training of the generative model,
recent works have introduced an additional refinement step that applies
stochastic variational inference (SVI) to improve upon the variational
parameters returned by the amortized inference model. In this paper, we propose
the Buffered Stochastic Variational Inference (BSVI), a new refinement
procedure that makes use of SVI's sequence of intermediate variational proposal
distributions and their corresponding importance weights to construct a new
generalized importance-weighted lower bound. We demonstrate empirically that
training the variational autoencoders with BSVI consistently out-performs SVI,
yielding an improved training procedure for VAEs.
On Multi-Cause Causal Inference with Unobserved Confounding: Counterexamples, Impossibility, and Alternatives
Authors: Alexander D'Amour
Date: 2019-02-27 00:00:00
Tasks: Causal Inference, Sensitivity

Unobserved confounding is a central barrier to drawing causal inferences from
observational data. Several authors have recently proposed that this barrier
can be overcome in the case where one attempts to infer the effects of several
variables simultaneously. In this paper, we present two simple, analytical
counterexamples that challenge the general claims that are central to these
approaches. In addition, we show that nonparametric identification is
impossible in this setting. We discuss practical implications, and suggest
alternatives to the methods that have been proposed so far in this line of
work: using proxy variables and shifting focus to sensitivity analysis.
Ordinal Distance Metric Learning with MDS for Image Ranking
Authors: Panpan Yu, Qingna Li
Date: 2019-02-27 00:00:00
Tasks: Dimensionality Reduction, Metric Learning

Image ranking is to rank images based on some known ranked images. In this
paper, we propose an improved linear ordinal distance metric learning approach
based on the linear distance metric learning model. By decomposing the distance
metric $A$ as $L^TL$, the problem can be cast as looking for a linear map
between two sets of points in different spaces, meanwhile maintaining some data
structures. The ordinal relation of the labels can be maintained via classical
multidimensional scaling, a popular tool for dimension reduction in statistics.
A least squares fitting term is then introduced to the cost function, which can
also maintain the local data structure. The resulting model is an unconstrained
problem, and can better fit the data structure. Extensive numerical results
demonstrate the improvement of the new approach over the linear distance metric
learning model both in speed and ranking performance.
Towards Efficient Data Valuation Based on the Shapley Value
Authors: Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gurel, Bo Li, Ce Zhang, Dawn Song, Costas Spanos
Date: 2019-02-27 00:00:00
Tasks: Data Valuation

"How much is my data worth?" is an increasingly common question posed by organizations and individuals alike. An answer to this question could allow, for instance, fairly distributing profits among multiple data contributors and determining prospective compensation when data breaches happen. In this paper, we study the problem of data valuation by utilizing the Shapley value, a popular notion of value which originated in cooperative game theory. The Shapley value defines a unique payoff scheme that satisfies many desiderata for the notion of data value. However, the Shapley value often requires exponential time to compute. To meet this challenge, we propose a repertoire of efficient algorithms for approximating the Shapley value. We also demonstrate the value of each training instance for various benchmark datasets.
Non-Rigid Structure from Motion: Prior-Free Factorization Method Revisited
Authors: Suryansh Kumar
Date: 2019-02-27 00:00:00

A simple prior free factorization algorithm \cite{dai2014simple} is quite often cited work in the field of Non-Rigid Structure from Motion (NRSfM). The benefit of this work lies in its simplicity of implementation, strong theoretical justification to the motion and structure estimation, and its invincible originality. Despite this, the prevailing view is, that it performs exceedingly inferior to other methods on several benchmark datasets \cite{jensen2018benchmark,akhter2009nonrigid}. However, our subtle investigation provides some empirical statistics which made us think against such views. The statistical results we obtained supersedes Dai {\it{et al.}}\cite{dai2014simple} originally reported results on the benchmark datasets by a significant margin under some elementary changes in their core algorithmic idea \cite{dai2014simple}. Now, these results not only exposes some unrevealed areas for research in NRSfM but also give rise to new mathematical challenges for NRSfM researchers. We argue that by \textbf{properly} utilizing the well-established assumptions about a non-rigidly deforming shape i.e, it deforms smoothly over frames \cite{rabaud2008re} and it spans a low-rank space, the simple prior-free idea can provide results which is comparable to the best available algorithms. In this paper, we explore some of the hidden intricacies missed by Dai {\it{et. al.}} work \cite{dai2014simple} and how some elementary measures and modifications can enhance its performance, as high as approx. 18\% on the benchmark dataset. The improved performance is justified and empirically verified by extensive experiments on several datasets. We believe our work has both practical and theoretical importance for the development of better NRSfM algorithms.
Zero-shot Learning of 3D Point Cloud Objects
Authors: Ali Cheraghian, Shafin Rahman, Lars Petersson
Date: 2019-02-27 00:00:00
Tasks: Attribute, Zero-Shot Learning

Recent deep learning architectures can recognize instances of 3D point cloud
objects of previously seen classes quite well. At the same time, current 3D
depth camera technology allows generating/segmenting a large amount of 3D point
cloud objects from an arbitrary scene, for which there is no previously seen
training data. A challenge for a 3D point cloud recognition system is, then, to
classify objects from new, unseen, classes. This issue can be resolved by
adopting a zero-shot learning (ZSL) approach for 3D data, similar to the 2D
image version of the same problem. ZSL attempts to classify unseen objects by
comparing semantic information (attribute/word vector) of seen and unseen
classes. Here, we adapt several recent 3D point cloud recognition systems to
the ZSL setting with some changes to their architectures. To the best of our
knowledge, this is the first attempt to classify unseen 3D point cloud objects
in the ZSL setting. A standard protocol (which includes the choice of datasets
and the seen/unseen split) to evaluate such systems is also proposed. Baseline
performances are reported using the new protocol on the investigated models.
This investigation throws a new challenge to the 3D point cloud recognition
community that may instigate numerous future works.
On the well-posedness of Bayesian inverse problems
Authors: Jonas Latz
Date: 2019-02-26 00:00:00

The subject of this article is the introduction of a new concept of well-posedness of Bayesian inverse problems. The conventional concept of (Lipschitz, Hellinger) well-posedness in [Stuart 2010, Acta Numerica 19, pp. 451-559] is difficult to verify in practice and may be inappropriate in some contexts. Our concept simply replaces the Lipschitz continuity of the posterior measure in the Hellinger distance by continuity in an appropriate distance between probability measures. Aside from the Hellinger distance, we investigate well-posedness with respect to weak convergence, the total variation distance, the Wasserstein distance, and also the Kullback--Leibler divergence. We demonstrate that the weakening to continuity is tolerable and that the generalisation to other distances is important. The main results of this article are proofs of well-posedness with respect to some of the aforementioned distances for large classes of Bayesian inverse problems. Here, little or no information about the underlying model is necessary; making these results particularly interesting for practitioners using black-box models. We illustrate our findings with numerical examples motivated from machine learning and image processing.
Diagnosing Bottlenecks in Deep Q-learning Algorithms
Authors: Justin Fu, Aviral Kumar, Matthew Soh, Sergey Levine
Date: 2019-02-26 00:00:00
Tasks: continuous-control, Continuous Control, Deep Reinforcement Learning, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Q-learning methods represent a commonly used class of algorithms in
reinforcement learning: they are generally efficient and simple, and can be
combined readily with function approximators for deep reinforcement learning
(RL). However, the behavior of Q-learning methods with function approximation
is poorly understood, both theoretically and empirically. In this work, we aim
to experimentally investigate potential issues in Q-learning, by means of a
"unit testing" framework where we can utilize oracles to disentangle sources of
error. Specifically, we investigate questions related to function
approximation, sampling error and nonstationarity, and where available, verify
if trends found in oracle settings hold true with modern deep RL methods. We
find that large neural network architectures have many benefits with regards to
learning stability; offer several practical compensations for overfitting; and
develop a novel sampling method based on explicitly compensating for function
approximation error that yields fair improvement on high-dimensional continuous
control domains.
TensorMap: Lidar-Based Topological Mapping and Localization via Tensor Decompositions
Authors: Sirisha Rambhatla, Nikos D. Sidiropoulos, Jarvis Haupt
Date: 2019-02-26 00:00:00
Tasks: Tensor Decomposition

We propose a technique to develop (and localize in) topological maps from
light detection and ranging (Lidar) data. Localizing an autonomous vehicle with
respect to a reference map in real-time is crucial for its safe operation.
Owing to the rich information provided by Lidar sensors, these are emerging as
a promising choice for this task. However, since a Lidar outputs a large amount
of data every fraction of a second, it is progressively harder to process the
information in real-time. Consequently, current systems have migrated towards
faster alternatives at the expense of accuracy. To overcome this inherent
trade-off between latency and accuracy, we propose a technique to develop
topological maps from Lidar data using the orthogonal Tucker3 tensor
decomposition. Our experimental evaluations demonstrate that in addition to
achieving a high compression ratio as compared to full data, the proposed
technique, $\textit{TensorMap}$, also accurately detects the position of the
vehicle in a graph-based representation of a map. We also analyze the
robustness of the proposed technique to Gaussian and translational noise, thus
initiating explorations into potential applications of tensor decompositions in
Lidar data analysis.
A Dictionary-Based Generalization of Robust PCA Part II: Applications to Hyperspectral Demixing
Authors: Sirisha Rambhatla, Xingguo Li, Jineng Ren, Jarvis Haupt
Date: 2019-02-26 00:00:00

We consider the task of localizing targets of interest in a hyperspectral
(HS) image based on their spectral signature(s), by posing the problem as two
distinct convex demixing task(s). With applications ranging from remote sensing
to surveillance, this task of target detection leverages the fact that each
material/object possesses its own characteristic spectral response, depending
upon its composition. However, since $\textit{signatures}$ of different
materials are often correlated, matched filtering-based approaches may not be
apply here. To this end, we model a HS image as a superposition of a low-rank
component and a dictionary sparse component, wherein the dictionary consists of
the $\textit{a priori}$ known characteristic spectral responses of the target
we wish to localize, and develop techniques for two different sparsity
structures, resulting from different model assumptions. We also present the
corresponding recovery guarantees, leveraging our recent theoretical results
from a companion paper. Finally, we analyze the performance of the proposed
approach via experimental evaluations on real HS datasets for a classification
task, and compare its performance with related techniques.
Learning When Not to Answer: A Ternary Reward Structure for Reinforcement Learning based Question Answering
Authors: Fréderic Godin, Anjishnu Kumar, Arpit Mittal
Date: 2019-02-26 00:00:00
Proceeding: NAACL 2019 6
Tasks: Knowledge Graphs, Question Answering, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we investigate the challenges of using reinforcement learning
agents for question-answering over knowledge graphs for real-world
applications. We examine the performance metrics used by state-of-the-art
systems and determine that they are inadequate for such settings. More
specifically, they do not evaluate the systems correctly for situations when
there is no answer available and thus agents optimized for these metrics are
poor at modeling confidence. We introduce a simple new performance metric for
evaluating question-answering agents that is more representative of practical
usage conditions, and optimize for this metric by extending the binary reward
structure used in prior work to a ternary reward structure which also rewards
an agent for not answering a question rather than giving an incorrect answer.
We show that this can drastically improve the precision of answered questions
while only not answering a limited number of previously correctly answered
questions. Employing a supervised learning strategy using depth-first-search
paths to bootstrap the reinforcement learning algorithm further improves
performance.
Continual Prediction from EHR Data for Inpatient Acute Kidney Injury
Authors: Rohit J. Kate, Noah Pearce, Debesh Mazumdar, Vani Nilakantan
Date: 2019-02-26 00:00:00
Tasks: Prediction

Acute kidney injury (AKI) commonly occurs in hospitalized patients and can
lead to serious medical complications. In order to optimally predict AKI before
it develops at any time during a hospital stay, we present a novel framework in
which AKI is continually predicted automatically from EHR data over the entire
hospital stay instead of at only one particular time. The continual model
predicts AKI every time a patients AKI-relevant variable changes in the EHR.
Thus the model is not only independent of a particular time for making
predictions, but it can also leverage the latest values of all the AKI-relevant
patient variables for making predictions. Using data of 44,691 hospital stays
of duration longer than 24 hours we evaluated our continual prediction model
and compared it with the traditional one-time prediction models. Excluding
hospitals stays in which AKI occurred within 24 hours from admission, the
one-time prediction model predicting at 24 hours from admission obtained area
under ROC curve (AUC) of 0.653 while the continual prediction model obtained
AUC of 0.724. The one-time prediction model that predicts at 24 hours obviously
cannot predict AKI incidences that occur within 24 hours of admission which
when included in the evaluation reduced its AUC to 0.57. In comparison, the
continual prediction model had AUC of 0.709. The continual prediction model
also did better than all other one-time prediction models predicting at other
fixed times. By being able to take into account the latest values of
AKI-relevant patient variables and by not being limited to a particular time of
prediction, the continual prediction model out-performed one-time prediction
models in predicting AKI.
Implicit Kernel Learning
Authors: Chun-Liang Li, Wei-Cheng Chang, Youssef Mroueh, Yiming Yang, Barnabás Póczos
Date: 2019-02-26 00:00:00
Tasks: Text Generation

Kernels are powerful and versatile tools in machine learning and statistics.
Although the notion of universal kernels and characteristic kernels has been
studied, kernel selection still greatly influences the empirical performance.
While learning the kernel in a data driven way has been investigated, in this
paper we explore learning the spectral distribution of kernel via implicit
generative models parametrized by deep neural networks. We called our method
Implicit Kernel Learning (IKL). The proposed framework is simple to train and
inference is performed via sampling random Fourier features. We investigate two
applications of the proposed IKL as examples, including generative adversarial
networks with MMD (MMD GAN) and standard supervised learning. Empirically, MMD
GAN with IKL outperforms vanilla predefined kernels on both image and text
generation benchmarks; using IKL with Random Kitchen Sinks also leads to
substantial improvement over existing state-of-the-art kernel learning
algorithms on popular supervised learning benchmarks. Theory and conditions for
using IKL in both applications are also studied as well as connections to
previous state-of-the-art methods.
Reliable Deep Grade Prediction with Uncertainty Estimation
Authors: Qian Hu, Huzefa Rangwala
Date: 2019-02-26 00:00:00
Tasks: Decision Making, Prediction

Currently, college-going students are taking longer to graduate than their
parental generations. Further, in the United States, the six-year graduation
rate has been 59% for decades. Improving the educational quality by training
better-prepared students who can successfully graduate in a timely manner is
critical. Accurately predicting students' grades in future courses has
attracted much attention as it can help identify at-risk students early so that
personalized feedback can be provided to them on time by advisors. Prior
research on students' grade prediction include shallow linear models; however,
students' learning is a highly complex process that involves the accumulation
of knowledge across a sequence of courses that can not be sufficiently modeled
by these linear models. In addition to that, prior approaches focus on
prediction accuracy without considering prediction uncertainty, which is
essential for advising and decision making. In this work, we present two types
of Bayesian deep learning models for grade prediction. The MLP ignores the
temporal dynamics of students' knowledge evolution. Hence, we propose RNN for
students' performance prediction. To evaluate the performance of the proposed
models, we performed extensive experiments on data collected from a large
public university. The experimental results show that the proposed models
achieve better performance than prior state-of-the-art approaches. Besides more
accurate results, Bayesian deep learning models estimate uncertainty associated
with the predictions. We explore how uncertainty estimation can be applied
towards developing a reliable educational early warning system. In addition to
uncertainty, we also develop an approach to explain the prediction results,
which is useful for advisors to provide personalized feedback to students.