pwc-archive/papers-with-abstracts huggingface dataset:
Tunability: Importance of Hyperparameters of Machine Learning Algorithms
Authors: Philipp Probst, Bernd Bischl, Anne-Laure Boulesteix
Date: 2018-02-26 00:00:00
Tasks: Benchmarking, BIG-bench Machine Learning

Modern supervised machine learning algorithms involve hyperparameters that
have to be set before running them. Options for setting hyperparameters are
default values from the software package, manual configuration by the user or
configuring them for optimal predictive performance by a tuning procedure. The
goal of this paper is two-fold. Firstly, we formalize the problem of tuning
from a statistical point of view, define data-based defaults and suggest
general measures quantifying the tunability of hyperparameters of algorithms.
Secondly, we conduct a large-scale benchmarking study based on 38 datasets from
the OpenML platform and six common machine learning algorithms. We apply our
measures to assess the tunability of their parameters. Our results yield
default values for hyperparameters and enable users to decide whether it is
worth conducting a possibly time consuming tuning strategy, to focus on the
most important hyperparameters and to chose adequate hyperparameter spaces for
tuning.
Data-dependent PAC-Bayes priors via differential privacy
Authors: Gintare Karolina Dziugaite, Daniel M. Roy
Date: 2018-02-26 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Generalization Bounds, valid

The Probably Approximately Correct (PAC) Bayes framework (McAllester, 1999)
can incorporate knowledge about the learning algorithm and (data) distribution
through the use of distribution-dependent priors, yielding tighter
generalization bounds on data-dependent posteriors. Using this flexibility,
however, is difficult, especially when the data distribution is presumed to be
unknown. We show how an {\epsilon}-differentially private data-dependent prior
yields a valid PAC-Bayes bound, and then show how non-private mechanisms for
choosing priors can also yield generalization bounds. As an application of this
result, we show that a Gaussian prior mean chosen via stochastic gradient
Langevin dynamics (SGLD; Welling and Teh, 2011) leads to a valid PAC-Bayes
bound given control of the 2-Wasserstein distance to an
{\epsilon}-differentially private stationary distribution. We study our
data-dependent bounds empirically, and show that they can be nonvacuous even
when other distribution-dependent bounds are vacuous.
Near-Linear Time Local Polynomial Nonparametric Estimation with Box Kernels
Authors: Yining Wang, Yi Wu, Simon S. Du
Date: 2018-02-26 00:00:00
Tasks: Density Estimation, regression

Local polynomial regression (Fan and Gijbels 1996) is an important class of methods for nonparametric density estimation and regression problems. However, straightforward implementation of local polynomial regression has quadratic time complexity which hinders its applicability in large-scale data analysis. In this paper, we significantly accelerate the computation of local polynomial estimates by novel applications of multi-dimensional binary indexed trees (Fenwick 1994). Both time and space complexity of our proposed algorithm is nearly linear in the number of input data points. Simulation results confirm the efficiency and effectiveness of our proposed approach.
i3PosNet: Instrument Pose Estimation from X-Ray in temporal bone surgery
Authors: David Kügler, Jannik Sehring, Andrei Stefanov, Igor Stenin, Julia Kristin, Thomas Klenzner, Jörg Schipper, Anirban Mukhopadhyay
Date: 2018-02-26 00:00:00
Tasks: Image Registration, Pose Estimation, Position, Translation

Purpose: Accurate estimation of the position and orientation (pose) of surgical instruments is crucial for delicate minimally invasive temporal bone surgery. Current techniques lack in accuracy and/or line-of-sight constraints (conventional tracking systems) or expose the patient to prohibitive ionizing radiation (intra-operative CT). A possible solution is to capture the instrument with a c-arm at irregular intervals and recover the pose from the image. Methods: i3PosNet infers the position and orientation of instruments from images using a pose estimation network. Said framework considers localized patches and outputs pseudo-landmarks. The pose is reconstructed from pseudo-landmarks by geometric considerations. Results: We show i3PosNet reaches errors less than 0.05mm. It outperforms conventional image registration-based approaches reducing average and maximum errors by at least two thirds. i3PosNet trained on synthetic images generalizes to real x-rays without any further adaptation. Conclusion: The translation of Deep Learning based methods to surgical applications is difficult, because large representative datasets for training and testing are not available. This work empirically shows sub-millimeter pose estimation trained solely based on synthetic training data.
A Differential Privacy Mechanism Design Under Matrix-Valued Query
Authors: Thee Chanyaswad, Alex Dytso, H. Vincent Poor, Prateek Mittal
Date: 2018-02-26 00:00:00

Traditionally, differential privacy mechanism design has been tailored for a
scalar-valued query function. Although many mechanisms such as the Laplace and
Gaussian mechanisms can be extended to a matrix-valued query function by adding
i.i.d. noise to each element of the matrix, this method is often sub-optimal as
it forfeits an opportunity to exploit the structural characteristics typically
associated with matrix analysis. In this work, we consider the design of
differential privacy mechanism specifically for a matrix-valued query function.
The proposed solution is to utilize a matrix-variate noise, as opposed to the
traditional scalar-valued noise. Particularly, we propose a novel differential
privacy mechanism called the Matrix-Variate Gaussian (MVG) mechanism, which
adds a matrix-valued noise drawn from a matrix-variate Gaussian distribution.
We prove that the MVG mechanism preserves $(\epsilon,\delta)$-differential
privacy, and show that it allows the structural characteristics of the
matrix-valued query function to naturally be exploited. Furthermore, due to the
multi-dimensional nature of the MVG mechanism and the matrix-valued query, we
introduce the concept of directional noise, which can be utilized to mitigate
the impact the noise has on the utility of the query. Finally, we demonstrate
the performance of the MVG mechanism and the advantages of directional noise
using three matrix-valued queries on three privacy-sensitive datasets. We find
that the MVG mechanism notably outperforms four previous state-of-the-art
approaches, and provides comparable utility to the non-private baseline. Our
work thus presents a promising prospect for both future research and
implementation of differential privacy for matrix-valued query functions.
Dynamic Fusion Networks for Machine Reading Comprehension
Authors: Yichong Xu, Jingjing Liu, Jianfeng Gao, Yelong Shen, Xiaodong Liu
Date: 2017-11-14 00:00:00
Tasks: Machine Reading Comprehension, Reading Comprehension, Reinforcement Learning

This paper presents a novel neural model - Dynamic Fusion Network (DFN), for
machine reading comprehension (MRC). DFNs differ from most state-of-the-art
models in their use of a dynamic multi-strategy attention process, in which
passages, questions and answer candidates are jointly fused into attention
vectors, along with a dynamic multi-step reasoning module for generating
answers. With the use of reinforcement learning, for each input sample that
consists of a question, a passage and a list of candidate answers, an instance
of DFN with a sample-specific network architecture can be dynamically
constructed by determining what attention strategy to apply and how many
reasoning steps to take. Experiments show that DFNs achieve the best result
reported on RACE, a challenging MRC dataset that contains real human reading
questions in a wide variety of types. A detailed empirical analysis also
demonstrates that DFNs can produce attention vectors that summarize information
from questions, passages and answer candidates more effectively than other
popular MRC models.
Adversarial Training for Probabilistic Spiking Neural Networks
Authors: Alireza Bagheri, Osvaldo Simeone, Bipin Rajendran
Date: 2018-02-22 00:00:00
Tasks: Sensitivity

Classifiers trained using conventional empirical risk minimization or maximum
likelihood methods are known to suffer dramatic performance degradations when
tested over examples adversarially selected based on knowledge of the
classifier's decision rule. Due to the prominence of Artificial Neural Networks
(ANNs) as classifiers, their sensitivity to adversarial examples, as well as
robust training schemes, have been recently the subject of intense
investigation. In this paper, for the first time, the sensitivity of spiking
neural networks (SNNs), or third-generation neural networks, to adversarial
examples is studied. The study considers rate and time encoding, as well as
rate and first-to-spike decoding. Furthermore, a robust training mechanism is
proposed that is demonstrated to enhance the performance of SNNs under
white-box attacks.
Re3 : Real-Time Recurrent Regression Networks for Visual Tracking of Generic Objects
Authors: Daniel Gordon, Ali Farhadi, Dieter Fox
Date: 2017-05-17 00:00:00
Tasks: Object, Object Tracking, regression, Visual Tracking

Robust object tracking requires knowledge and understanding of the object
being tracked: its appearance, its motion, and how it changes over time. A
tracker must be able to modify its underlying model and adapt to new
observations. We present Re3, a real-time deep object tracker capable of
incorporating temporal information into its model. Rather than focusing on a
limited set of objects or training a model at test-time to track a specific
instance, we pretrain our generic tracker on a large variety of objects and
efficiently update on the fly; Re3 simultaneously tracks and updates the
appearance model with a single forward pass. This lightweight model is capable
of tracking objects at 150 FPS, while attaining competitive results on
challenging benchmarks. We also show that our method handles temporary
occlusion better than other comparable trackers using experiments that directly
measure performance on sequences with occlusion.
Human Perceptions of Fairness in Algorithmic Decision Making: A Case Study of Criminal Risk Prediction
Authors: Nina Grgić-Hlača, Elissa M. Redmiles, Krishna P. Gummadi, Adrian Weller
Date: 2018-02-26 00:00:00
Tasks: Decision Making, Fairness

As algorithms are increasingly used to make important decisions that affect
human lives, ranging from social benefit assignment to predicting risk of
criminal recidivism, concerns have been raised about the fairness of
algorithmic decision making. Most prior works on algorithmic fairness
normatively prescribe how fair decisions ought to be made. In contrast, here,
we descriptively survey users for how they perceive and reason about fairness
in algorithmic decision making.
  A key contribution of this work is the framework we propose to understand why
people perceive certain features as fair or unfair to be used in algorithms.
Our framework identifies eight properties of features, such as relevance,
volitionality and reliability, as latent considerations that inform people's
moral judgments about the fairness of feature use in decision-making
algorithms. We validate our framework through a series of scenario-based
surveys with 576 people. We find that, based on a person's assessment of the
eight latent properties of a feature in our exemplar scenario, we can
accurately (> 85%) predict if the person will judge the use of the feature as
fair.
  Our findings have important implications. At a high-level, we show that
people's unfairness concerns are multi-dimensional and argue that future
studies need to address unfairness concerns beyond discrimination. At a
low-level, we find considerable disagreements in people's fairness judgments.
We identify root causes of the disagreements, and note possible pathways to
resolve them.
Learning Weighted Representations for Generalization Across Designs
Authors: Fredrik D. Johansson, Nathan Kallus, Uri Shalit, David Sontag
Date: 2018-02-23 00:00:00
Proceeding: ICLR 2018 1
Tasks: Causal Inference, Domain Adaptation, Representation Learning, Unsupervised Domain Adaptation

Predictive models that generalize well under distributional shift are often
desirable and sometimes crucial to building robust and reliable machine
learning applications. We focus on distributional shift that arises in causal
inference from observational data and in unsupervised domain adaptation. We
pose both of these problems as prediction under a shift in design. Popular
methods for overcoming distributional shift make unrealistic assumptions such
as having a well-specified model or knowing the policy that gave rise to the
observed data. Other methods are hindered by their need for a pre-specified
metric for comparing observations, or by poor asymptotic properties. We devise
a bound on the generalization error under design shift, incorporating both
representation learning and sample re-weighting. Based on the bound, we propose
an algorithmic framework that does not require any of the above assumptions and
which is asymptotically consistent. We empirically study the new framework
using two synthetic datasets, and demonstrate its effectiveness compared to
previous methods.
Missing Data in Sparse Transition Matrix Estimation for Sub-Gaussian Vector Autoregressive Processes
Authors: Amin Jalali, Rebecca Willett
Date: 2018-02-26 00:00:00
Tasks: Time Series, Time Series Analysis

High-dimensional time series data exist in numerous areas such as finance,
genomics, healthcare, and neuroscience. An unavoidable aspect of all such
datasets is missing data, and dealing with this issue has been an important
focus in statistics, control, and machine learning. In this work, we consider a
high-dimensional estimation problem where a dynamical system, governed by a
stable vector autoregressive model, is randomly and only partially observed at
each time point. Our task amounts to estimating the transition matrix, which is
assumed to be sparse. In such a scenario, where covariates are highly
interdependent and partially missing, new theoretical challenges arise. While
transition matrix estimation in vector autoregressive models has been studied
previously, the missing data scenario requires separate efforts. Moreover,
while transition matrix estimation can be studied from a high-dimensional
sparse linear regression perspective, the covariates are highly dependent and
existing results on regularized estimation with missing data from
i.i.d.~covariates are not applicable. At the heart of our analysis lies 1) a
novel concentration result when the innovation noise satisfies the convex
concentration property, as well as 2) a new quantity for characterizing the
interactions of the time-varying observation process with the underlying
dynamical system.
Lasso Regularization Paths for NARMAX Models via Coordinate Descent
Authors: Antônio H. Ribeiro, Luis A. Aguirre
Date: 2017-10-02 00:00:00
Tasks: Computational Efficiency

We propose a new algorithm for estimating NARMAX models with $L_1$
regularization for models represented as a linear combination of basis
functions. Due to the $L_1$-norm penalty the Lasso estimation tends to produce
some coefficients that are exactly zero and hence gives interpretable models.
The novelty of the contribution is the inclusion of error regressors in the
Lasso estimation (which yields a nonlinear regression problem). The proposed
algorithm uses cyclical coordinate descent to compute the parameters of the
NARMAX models for the entire regularization path. It deals with the error terms
by updating the regressor matrix along with the parameter vector. In
comparative timings we find that the modification does not reduce the
computational efficiency of the original algorithm and can provide the most
important regressors in very few inexpensive iterations. The method is
illustrated for linear and polynomial models by means of two examples.
Retrieval-Augmented Convolutional Neural Networks for Improved Robustness against Adversarial Examples
Authors: Jake Zhao, Kyunghyun Cho
Date: 2018-02-26 00:00:00
Tasks: Retrieval

We propose a retrieval-augmented convolutional network and propose to train
it with local mixup, a novel variant of the recently proposed mixup algorithm.
The proposed hybrid architecture combining a convolutional network and an
off-the-shelf retrieval engine was designed to mitigate the adverse effect of
off-manifold adversarial examples, while the proposed local mixup addresses
on-manifold ones by explicitly encouraging the classifier to locally behave
linearly on the data manifold. Our evaluation of the proposed approach against
five readily-available adversarial attacks on three datasets--CIFAR-10, SVHN
and ImageNet--demonstrate the improved robustness compared to the vanilla
convolutional network.
Disentangling the independently controllable factors of variation by interacting with the world
Authors: Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, Yoshua Bengio
Date: 2018-02-26 00:00:00
Tasks: Open-Ended Question Answering

It has been postulated that a good representation is one that disentangles
the underlying explanatory factors of variation. However, it remains an open
question what kind of training framework could potentially achieve that.
Whereas most previous work focuses on the static setting (e.g., with images),
we postulate that some of the causal factors could be discovered if the learner
is allowed to interact with its environment. The agent can experiment with
different actions and observe their effects. More specifically, we hypothesize
that some of these factors correspond to aspects of the environment which are
independently controllable, i.e., that there exists a policy and a learnable
feature for each such aspect of the environment, such that this policy can
yield changes in that feature with minimal changes to other features that
explain the statistical variations in the observed data. We propose a specific
objective function to find such factors, and verify experimentally that it can
indeed disentangle independently controllable aspects of the environment
without any extrinsic reward signal.
Topic Compositional Neural Language Model
Authors: Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, Lawrence Carin
Date: 2017-12-28 00:00:00
Tasks: Language Modeling, Language Modelling, Mixture-of-Experts, model

We propose a Topic Compositional Neural Language Model (TCNLM), a novel
method designed to simultaneously capture both the global semantic meaning and
the local word ordering structure in a document. The TCNLM learns the global
semantic coherence of a document via a neural topic model, and the probability
of each learned latent topic is further used to build a Mixture-of-Experts
(MoE) language model, where each expert (corresponding to one topic) is a
recurrent neural network (RNN) that accounts for learning the local structure
of a word sequence. In order to train the MoE model efficiently, a matrix
factorization method is applied, by extending each weight matrix of the RNN to
be an ensemble of topic-dependent weight matrices. The degree to which each
member of the ensemble is used is tied to the document-dependent probability of
the corresponding topics. Experimental results on several corpora show that the
proposed approach outperforms both a pure RNN-based model and other
topic-guided language models. Further, our model yields sensible topics, and
also has the capacity to generate meaningful sentences conditioned on given
topics.
Generalization Guarantees for Multi-item Profit Maximization: Pricing, Auctions, and Randomized Mechanisms
Authors: Maria-Florina Balcan, Tuomas Sandholm, Ellen Vitercik
Date: 2017-04-29 00:00:00

We study multi-item profit maximization when there is an underlying distribution over buyers' values. In practice, a full description of the distribution is typically unavailable, so we study the setting where the mechanism designer only has samples from the distribution. If the designer uses the samples to optimize over a complex mechanism class -- such as the set of all multi-item, multi-buyer mechanisms -- a mechanism may have high average profit over the samples but low expected profit. This raises the central question of this paper: how many samples are sufficient to ensure that a mechanism's average profit is close to its expected profit? To answer this question, we uncover structure shared by many pricing, auction, and lottery mechanisms: for any set of buyers' values, profit is piecewise linear in the mechanism's parameters. Using this structure, we prove new bounds for mechanism classes not yet studied in the sample-based mechanism design literature and match or improve over the best-known guarantees for many classes.
Understanding image motion with group representations
Authors: Andrew Jaegle, Stephen Phillips, Daphne Ippolito, Kostas Daniilidis
Date: 2016-12-01 00:00:00
Proceeding: ICLR 2018 1

Motion is an important signal for agents in dynamic environments, but
learning to represent motion from unlabeled video is a difficult and
underconstrained problem. We propose a model of motion based on elementary
group properties of transformations and use it to train a representation of
image motion. While most methods of estimating motion are based on pixel-level
constraints, we use these group properties to constrain the abstract
representation of motion itself. We demonstrate that a deep neural network
trained using this method captures motion in both synthetic 2D sequences and
real-world sequences of vehicle motion, without requiring any labels. Networks
trained to respect these constraints implicitly identify the image
characteristic of motion in different sequence types. In the context of vehicle
motion, this method extracts information useful for localization, tracking, and
odometry. Our results demonstrate that this representation is useful for
learning motion in the general setting where explicit labels are difficult to
obtain.
On b-bit min-wise hashing for large-scale regression and classification with sparse data
Authors: Rajen D. Shah, Nicolai Meinshausen
Date: 2013-08-06 00:00:00
Tasks: Dimensionality Reduction, General Classification, regression

Large-scale regression problems where both the number of variables, $p$, and
the number of observations, $n$, may be large and in the order of millions or
more, are becoming increasingly more common. Typically the data are sparse:
only a fraction of a percent of the entries in the design matrix are non-zero.
Nevertheless, often the only computationally feasible approach is to perform
dimension reduction to obtain a new design matrix with far fewer columns and
then work with this compressed data.
  $b$-bit min-wise hashing (Li and Konig, 2011) is a promising dimension
reduction scheme for sparse matrices which produces a set of random features
such that regression on the resulting design matrix approximates a kernel
regression with the resemblance kernel. In this work, we derive bounds on the
prediction error of such regressions. For both linear and logistic models we
show that the average prediction error vanishes asymptotically as long as $q
\|\beta^*\|_2^2 /n \rightarrow 0$, where $q$ is the average number of non-zero
entries in each row of the design matrix and $\beta^*$ is the coefficient of
the linear predictor.
  We also show that ordinary least squares or ridge regression applied to the
reduced data can in fact allow us fit more flexible models. We obtain
non-asymptotic prediction error bounds for interaction models and for models
where an unknown row normalisation must be applied in order for the signal to
be linear in the predictors.