pwc-archive/papers-with-abstracts huggingface dataset:
Cross-validation based Nonlinear Shrinkage
Authors: Daniel Bartz
Date: 2016-11-02 00:00:00

Many machine learning algorithms require precise estimates of covariance
matrices. The sample covariance matrix performs poorly in high-dimensional
settings, which has stimulated the development of alternative methods, the
majority based on factor models and shrinkage. Recent work of Ledoit and Wolf
has extended the shrinkage framework to Nonlinear Shrinkage (NLS), a more
powerful covariance estimator based on Random Matrix Theory.
  Our contribution shows that, contrary to claims in the literature,
cross-validation based covariance matrix estimation (CVC) yields comparable
performance at strongly reduced complexity and runtime. On two real world data
sets, we show that the CVC estimator yields superior results than competing
shrinkage and factor based methods.
Predicting Domain Generation Algorithms with Long Short-Term Memory Networks
Authors: Jonathan Woodbridge, Hyrum S. Anderson, Anjum Ahuja, Daniel Grant
Date: 2016-11-02 00:00:00
Tasks: Binary Classification

Various families of malware use domain generation algorithms (DGAs) to
generate a large number of pseudo-random domain names to connect to a command
and control (C&C) server. In order to block DGA C&C traffic, security
organizations must first discover the algorithm by reverse engineering malware
samples, then generating a list of domains for a given seed. The domains are
then either preregistered or published in a DNS blacklist. This process is not
only tedious, but can be readily circumvented by malware authors using a large
number of seeds in algorithms with multivariate recurrence properties (e.g.,
banjori) or by using a dynamic list of seeds (e.g., bedep). Another technique
to stop malware from using DGAs is to intercept DNS queries on a network and
predict whether domains are DGA generated. Such a technique will alert network
administrators to the presence of malware on their networks. In addition, if
the predictor can also accurately predict the family of DGAs, then network
administrators can also be alerted to the type of malware that is on their
networks. This paper presents a DGA classifier that leverages long short-term
memory (LSTM) networks to predict DGAs and their respective families without
the need for a priori feature extraction. Results are significantly better than
state-of-the-art techniques, providing 0.9993 area under the receiver operating
characteristic curve for binary classification and a micro-averaged F1 score of
0.9906. In other terms, the LSTM technique can provide a 90% detection rate
with a 1:10000 false positive (FP) rate---a twenty times FP improvement over
comparable methods. Experiments in this paper are run on open datasets and code
snippets are provided to reproduce the results.
Rationalizing Neural Predictions
Authors: Tao Lei, Regina Barzilay, Tommi Jaakkola
Date: 2016-06-13 00:00:00
Proceeding: EMNLP 2016 11
Tasks: Prediction, Retrieval, Sentiment Analysis

Prediction without justification has limited applicability. As a remedy, we
learn to extract pieces of input text as justifications -- rationales -- that
are tailored to be short and coherent, yet sufficient for making the same
prediction. Our approach combines two modular components, generator and
encoder, which are trained to operate well together. The generator specifies a
distribution over text fragments as candidate rationales and these are passed
through the encoder for prediction. Rationales are never given during training.
Instead, the model is regularized by desiderata for rationales. We evaluate the
approach on multi-aspect sentiment analysis against manually annotated test
cases. Our approach outperforms attention-based baseline by a significant
margin. We also successfully illustrate the method on the question retrieval
task.
Learning Leading Indicators for Time Series Predictions
Authors: Magda Gregorova, Alexandros Kalousis, Stéphane Marchand-Maillet
Date: 2015-07-07 00:00:00
Tasks: Time Series, Time Series Analysis

We consider the problem of learning models for forecasting multiple
time-series systems together with discovering the leading indicators that serve
as good predictors for the system. We model the systems by linear vector
autoregressive models (VAR) and link the discovery of leading indicators to
inferring sparse graphs of Granger-causality. We propose new problem
formulations and develop two new methods to learn such models, gradually
increasing the complexity of assumptions and approaches. While the first method
assumes common structures across the whole system, our second method uncovers
model clusters based on the Granger-causality and leading indicators together
with learning the model parameters. We study the performance of our methods on
a comprehensive set of experiments and confirm their efficacy and their
advantages over state-of-the-art sparse VAR and graphical Granger learning
methods.
Learning Sensor Multiplexing Design through Back-propagation
Authors: Ayan Chakrabarti
Date: 2016-05-23 00:00:00
Proceeding: NeurIPS 2016 12

Recent progress on many imaging and vision tasks has been driven by the use
of deep feed-forward neural networks, which are trained by propagating
gradients of a loss defined on the final output, back through the network up to
the first layer that operates directly on the image. We propose
back-propagating one step further---to learn camera sensor designs jointly with
networks that carry out inference on the images they capture. In this paper, we
specifically consider the design and inference problems in a typical color
camera---where the sensor is able to measure only one color channel at each
pixel location, and computational inference is required to reconstruct a full
color image. We learn the camera sensor's color multiplexing pattern by
encoding it as layer whose learnable weights determine which color channel,
from among a fixed set, will be measured at each location. These weights are
jointly trained with those of a reconstruction network that operates on the
corresponding sensor measurements to produce a full color image. Our network
achieves significant improvements in accuracy over the traditional Bayer
pattern used in most color cameras. It automatically learns to employ a sparse
color measurement approach similar to that of a recent design, and moreover,
improves upon that design by learning an optimal layout for these measurements.
Scalable Semi-Supervised Learning over Networks using Nonsmooth Convex Optimization
Authors: Alexander Jung, Alfred O. Hero III, Alexandru Mara, Sabeur Aridhi
Date: 2016-11-02 00:00:00
Tasks: Transductive Learning

We propose a scalable method for semi-supervised (transductive) learning from
massive network-structured datasets. Our approach to semi-supervised learning
is based on representing the underlying hypothesis as a graph signal with small
total variation. Requiring a small total variation of the graph signal
representing the underlying hypothesis corresponds to the central smoothness
assumption that forms the basis for semi-supervised learning, i.e., input
points forming clusters have similar output values or labels. We formulate the
learning problem as a nonsmooth convex optimization problem which we solve by
appealing to Nesterovs optimal first-order method for nonsmooth optimization.
We also provide a message passing formulation of the learning method which
allows for a highly scalable implementation in big data frameworks.
Deep counter networks for asynchronous event-based processing
Authors: Jonathan Binas, Giacomo Indiveri, Michael Pfeiffer
Date: 2016-11-02 00:00:00

Despite their advantages in terms of computational resources, latency, and
power consumption, event-based implementations of neural networks have not been
able to achieve the same performance figures as their equivalent
state-of-the-art deep network models. We propose counter neurons as minimal
spiking neuron models which only require addition and comparison operations,
thus avoiding costly multiplications. We show how inference carried out in deep
counter networks converges to the same accuracy levels as are achieved with
state-of-the-art conventional networks. As their event-based style of
computation leads to reduced latency and sparse updates, counter networks are
ideally suited for efficient compact and low-power hardware implementation. We
present theory and training methods for counter networks, and demonstrate on
the MNIST benchmark that counter networks converge quickly, both in terms of
time and number of operations required, to state-of-the-art classification
accuracy.
Stochastic Variational Deep Kernel Learning
Authors: Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing
Date: 2016-11-01 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Gaussian Processes, General Classification, Multi-Task Learning, Variational Inference

Deep kernel learning combines the non-parametric flexibility of kernel
methods with the inductive biases of deep learning architectures. We propose a
novel deep kernel learning model and stochastic variational inference procedure
which generalizes deep kernel learning approaches to enable classification,
multi-task learning, additive covariance structures, and stochastic gradient
training. Specifically, we apply additive base kernels to subsets of output
features from deep neural architectures, and jointly learn the parameters of
the base kernels and deep network through a Gaussian process marginal
likelihood objective. Within this framework, we derive an efficient form of
stochastic variational inference which leverages local kernel interpolation,
inducing points, and structure exploiting algebra. We show improved performance
over stand alone deep networks, SVMs, and state of the art scalable Gaussian
processes on several classification benchmarks, including an airline delay
dataset containing 6 million training points, CIFAR, and ImageNet.
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension
Authors: Yang Yu, Wei zhang, Kazi Hasan, Mo Yu, Bing Xiang, Bo-Wen Zhou
Date: 2016-10-31 00:00:00
Tasks: Question Answering, Reading Comprehension

This paper proposes dynamic chunk reader (DCR), an end-to-end neural reading
comprehension (RC) model that is able to extract and rank a set of answer
candidates from a given document to answer questions. DCR is able to predict
answers of variable lengths, whereas previous neural RC models primarily
focused on predicting single tokens or entities. DCR encodes a document and an
input question with recurrent neural networks, and then applies a word-by-word
attention mechanism to acquire question-aware representations for the document,
followed by the generation of chunk representations and a ranking module to
propose the top-ranked chunk as the answer. Experimental results show that DCR
achieves state-of-the-art exact match and F1 scores on the SQuAD dataset.
Correlated-PCA: Principal Components' Analysis when Data and Noise are Correlated
Authors: Namrata Vaswani, Han Guo
Date: 2016-08-15 00:00:00
Proceeding: NeurIPS 2016
Tasks: valid

Given a matrix of observed data, Principal Components Analysis (PCA) computes
a small number of orthogonal directions that contain most of its variability.
Provably accurate solutions for PCA have been in use for a long time. However,
to the best of our knowledge, all existing theoretical guarantees for it assume
that the data and the corrupting noise are mutually independent, or at least
uncorrelated. This is valid in practice often, but not always. In this paper,
we study the PCA problem in the setting where the data and noise can be
correlated. Such noise is often also referred to as "data-dependent noise". We
obtain a correctness result for the standard eigenvalue decomposition (EVD)
based solution to PCA under simple assumptions on the data-noise correlation.
We also develop and analyze a generalization of EVD, cluster-EVD, that improves
upon EVD in certain regimes.
Building Machines That Learn and Think Like People
Authors: Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, Samuel J. Gershman
Date: 2016-04-01 00:00:00
Tasks: Board Games, Object Recognition

Recent progress in artificial intelligence (AI) has renewed interest in
building systems that learn and think like people. Many advances have come from
using deep neural networks trained end-to-end in tasks such as object
recognition, video games, and board games, achieving performance that equals or
even beats humans in some respects. Despite their biological inspiration and
performance achievements, these systems differ from human intelligence in
crucial ways. We review progress in cognitive science suggesting that truly
human-like learning and thinking machines will have to reach beyond current
engineering trends in both what they learn, and how they learn it.
Specifically, we argue that these machines should (a) build causal models of
the world that support explanation and understanding, rather than merely
solving pattern recognition problems; (b) ground learning in intuitive theories
of physics and psychology, to support and enrich the knowledge that is learned;
and (c) harness compositionality and learning-to-learn to rapidly acquire and
generalize knowledge to new tasks and situations. We suggest concrete
challenges and promising routes towards these goals that can combine the
strengths of recent neural network advances with more structured cognitive
models.
A Framework for Searching for General Artificial Intelligence
Authors: Marek Rosa, Jan Feyereisl, The GoodAI Collective
Date: 2016-11-02 00:00:00

There is a significant lack of unified approaches to building generally
intelligent machines. The majority of current artificial intelligence research
operates within a very narrow field of focus, frequently without considering
the importance of the 'big picture'. In this document, we seek to describe and
unify principles that guide the basis of our development of general artificial
intelligence. These principles revolve around the idea that intelligence is a
tool for searching for general solutions to problems. We define intelligence as
the ability to acquire skills that narrow this search, diversify it and help
steer it to more promising areas. We also provide suggestions for studying,
measuring, and testing the various skills and abilities that a human-level
intelligent machine needs to acquire. The document aims to be both
implementation agnostic, and to provide an analytic, systematic, and scalable
way to generate hypotheses that we believe are needed to meet the necessary
conditions in the search for general artificial intelligence. We believe that
such a framework is an important stepping stone for bringing together
definitions, highlighting open problems, connecting researchers willing to
collaborate, and for unifying the arguably most significant search of this
century.
Wearable Vision Detection of Environmental Fall Risks using Convolutional Neural Networks
Authors: Mina Nouredanesh, Andrew McCormick, Sunil L. Kukreja, James Tung
Date: 2016-11-02 00:00:00
Tasks: Classification, General Classification

In this paper, a method to detect environmental hazards related to a fall
risk using a mobile vision system is proposed. First-person perspective videos
are proposed to provide objective evidence on cause and circumstances of
perturbed balance during activities of daily living, targeted to seniors. A
classification problem was defined with 12 total classes of potential fall
risks, including slope changes (e.g., stairs, curbs, ramps) and surfaces (e.g.,
gravel, grass, concrete). Data was collected using a chest-mounted GoPro
camera. We developed a convolutional neural network for automatic feature
extraction, reduction, and classification of frames. Initial results, with a
mean square error of 8%, are promising.
Improving variational methods via pairwise linear response identities
Authors: Jack Raymond, Federico Ricci-Tersenghi
Date: 2016-11-02 00:00:00

Inference methods are often formulated as variational approximations: these
approximations allow easy evaluation of statistics by marginalization or linear
response, but these estimates can be inconsistent. We show that by introducing
constraints on covariance, one can ensure consistency of linear response with
the variational parameters, and in so doing inference of marginal probability
distributions is improved. For the Bethe approximation and its generalizations,
improvements are achieved with simple choices of the constraints. The
approximations are presented as variational frameworks; iterative procedures
related to message passing are provided for finding the minima.
Deep Model Compression: Distilling Knowledge from Noisy Teachers
Authors: Bharat Bhusan Sau, Vineeth N. Balasubramanian
Date: 2016-10-30 00:00:00
Tasks: Model Compression

The remarkable successes of deep learning models across various applications
have resulted in the design of deeper networks that can solve complex problems.
However, the increasing depth of such models also results in a higher storage
and runtime complexity, which restricts the deployability of such very deep
models on mobile and portable devices, which have limited storage and battery
capacity. While many methods have been proposed for deep model compression in
recent years, almost all of them have focused on reducing storage complexity.
In this work, we extend the teacher-student framework for deep model
compression, since it has the potential to address runtime and train time
complexity too. We propose a simple methodology to include a noise-based
regularizer while training the student from the teacher, which provides a
healthy improvement in the performance of the student network. Our experiments
on the CIFAR-10, SVHN and MNIST datasets show promising improvement, with the
best performance on the CIFAR-10 dataset. We also conduct a comprehensive
empirical evaluation of the proposed method under related settings on the
CIFAR-10 dataset to show the promise of the proposed approach.
Stochastic Structured Prediction under Bandit Feedback
Authors: Artem Sokolov, Julia Kreutzer, Christopher Lo, Stefan Riezler
Date: 2016-06-02 00:00:00
Proceeding: NeurIPS 2016 12
Tasks: Prediction, Structured Prediction

Stochastic structured prediction under bandit feedback follows a learning
protocol where on each of a sequence of iterations, the learner receives an
input, predicts an output structure, and receives partial feedback in form of a
task loss evaluation of the predicted structure. We present applications of
this learning scenario to convex and non-convex objectives for structured
prediction and analyze them as stochastic first-order methods. We present an
experimental evaluation on problems of natural language processing over
exponential output spaces, and compare convergence speed across different
objectives under the practical criterion of optimal task performance on
development data and the optimization-theoretic criterion of minimal squared
gradient norm. Best results under both criteria are obtained for a non-convex
objective for pairwise preference learning under bandit feedback.
A tentative model for dimensionless phoneme distance from binary distinctive features
Authors: Tiago Tresoldi
Date: 2016-10-05 00:00:00

This work proposes a tentative model for the calculation of dimensionless
distances between phonemes; sounds are described with binary distinctive
features and distances show linear consistency in terms of such features. The
model can be used as a scoring function for local and global pairwise alignment
of phoneme sequences, and the distances can be used as prior probabilities for
Bayesian analyses on the phylogenetic relationship between languages,
particularly for cognate identification in cases where no empirical prior
probability is available.
Operator-valued Kernels for Learning from Functional Response Data
Authors: Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, Julien Audiffren
Date: 2015-10-28 00:00:00
Tasks: Audio Signal Processing, General Classification

In this paper we consider the problems of supervised classification and
regression in the case where attributes and labels are functions: a data is
represented by a set of functions, and the label is also a function. We focus
on the use of reproducing kernel Hilbert space theory to learn from such
functional data. Basic concepts and properties of kernel-based learning are
extended to include the estimation of function-valued functions. In this
setting, the representer theorem is restated, a set of rigorously defined
infinite-dimensional operator-valued kernels that can be valuably applied when
the data are functions is described, and a learning algorithm for nonlinear
functional data analysis is introduced. The methodology is illustrated through
speech and audio signal processing experiments.
Streaming regularization parameter selection via stochastic gradient descent
Authors: Ricardo Pio Monti, Romy Lorenz, Robert Leech, Christoforos Anagnostopoulos, Giovanni Montana
Date: 2015-11-06 00:00:00
Tasks: regression

We propose a framework to perform streaming covariance selection. Our
approach employs regularization constraints where a time-varying sparsity
parameter is iteratively estimated via stochastic gradient descent. This allows
for the regularization parameter to be efficiently learnt in an online manner.
The proposed framework is developed for linear regression models and extended
to graphical models via neighbourhood selection. Under mild assumptions, we are
able to obtain convergence results in a non-stochastic setting. The
capabilities of such an approach are demonstrated using both synthetic data as
well as neuroimaging data.
Sensitivity Maps of the Hilbert-Schmidt Independence Criterion
Authors: Adrián Pérez-Suay, Gustau Camps-Valls
Date: 2016-11-02 00:00:00
Tasks: Causal Inference, feature selection, Sensitivity

Kernel dependence measures yield accurate estimates of nonlinear relations
between random variables, and they are also endorsed with solid theoretical
properties and convergence rates. Besides, the empirical estimates are easy to
compute in closed form just involving linear algebra operations. However, they
are hampered by two important problems: the high computational cost involved,
as two kernel matrices of the sample size have to be computed and stored, and
the interpretability of the measure, which remains hidden behind the implicit
feature map. We here address these two issues. We introduce the Sensitivity
Maps (SMs) for the Hilbert-Schmidt independence criterion (HSIC). Sensitivity
maps allow us to explicitly analyze and visualize the relative relevance of
both examples and features on the dependence measure. We also present the
randomized HSIC (RHSIC) and its corresponding sensitivity maps to cope with
large scale problems. We build upon the framework of random features and the
Bochner's theorem to approximate the involved kernels in the canonical HSIC.
The power of the RHSIC measure scales favourably with the number of samples,
and it approximates HSIC and the sensitivity maps efficiently. Convergence
bounds of both the measure and the sensitivity map are also provided. Our
proposal is illustrated in synthetic examples, and challenging real problems of
dependence estimation, feature selection, and causal inference from empirical
data.