pwc-archive/papers-with-abstracts huggingface dataset:
PAC-learning in the presence of evasion adversaries
Authors: Daniel Cullina, Arjun Nitin Bhagoji, Prateek Mittal
Date: 2018-06-05 00:00:00
Tasks: Open-Ended Question Answering, PAC learning

The existence of evasion attacks during the test phase of machine learning
algorithms represents a significant challenge to both their deployment and
understanding. These attacks can be carried out by adding imperceptible
perturbations to inputs to generate adversarial examples and finding effective
defenses and detectors has proven to be difficult. In this paper, we step away
from the attack-defense arms race and seek to understand the limits of what can
be learned in the presence of an evasion adversary. In particular, we extend
the Probably Approximately Correct (PAC)-learning framework to account for the
presence of an adversary. We first define corrupted hypothesis classes which
arise from standard binary hypothesis classes in the presence of an evasion
adversary and derive the Vapnik-Chervonenkis (VC)-dimension for these, denoted
as the adversarial VC-dimension. We then show that sample complexity upper
bounds from the Fundamental Theorem of Statistical learning can be extended to
the case of evasion adversaries, where the sample complexity is controlled by
the adversarial VC-dimension. We then explicitly derive the adversarial
VC-dimension for halfspace classifiers in the presence of a sample-wise
norm-constrained adversary of the type commonly studied for evasion attacks and
show that it is the same as the standard VC-dimension, closing an open
question. Finally, we prove that the adversarial VC-dimension can be either
larger or smaller than the standard VC-dimension depending on the hypothesis
class and adversary, making it an interesting object of study in its own right.
Surveillance Face Recognition Challenge
Authors: Zhiyi Cheng, Xiatian Zhu, Shaogang Gong
Date: 2018-04-25 00:00:00
Tasks: Face Recognition

Face recognition (FR) is one of the most extensively investigated problems in
computer vision. Significant progress in FR has been made due to the recent
introduction of the larger scale FR challenges, particularly with constrained
social media web images, e.g. high-resolution photos of celebrity faces taken
by professional photo-journalists. However, the more challenging FR in
unconstrained and low-resolution surveillance images remains largely
under-studied. To facilitate more studies on developing FR models that are
effective and robust for low-resolution surveillance facial images, we
introduce a new Surveillance Face Recognition Challenge, which we call the
QMUL-SurvFace benchmark. This new benchmark is the largest and more importantly
the only true surveillance FR benchmark to our best knowledge, where
low-resolution images are not synthesised by artificial down-sampling of native
high-resolution images. This challenge contains 463,507 face images of 15,573
distinct identities captured in real-world uncooperative surveillance scenes
over wide space and time. As a consequence, it presents an extremely
challenging FR benchmark. We benchmark the FR performance on this challenge
using five representative deep learning face recognition models, in comparison
to existing benchmarks. We show that the current state of the arts are still
far from being satisfactory to tackle the under-investigated surveillance FR
problem in practical forensic scenarios. Face recognition is generally more
difficult in an open-set setting which is typical for surveillance scenarios,
owing to a large number of non-target people (distractors) appearing open
spaced scenes. This is evidently so that on the new Surveillance FR Challenge,
the top-performing CentreFace deep learning FR model on the MegaFace benchmark
can now only achieve 13.2% success rate (at Rank-20) at a 10% false alarm rate.
Spectral Inference Networks: Unifying Deep and Spectral Learning
Authors: David Pfau, Stig Petersen, Ashish Agarwal, David G. T. Barrett, Kimberly L. Stachenfeld
Date: 2018-06-06 00:00:00
Proceeding: ICLR 2019 5
Tasks: Atari Games, Bilevel Optimization, Representation Learning, Stochastic Optimization, Variational Monte Carlo

We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization. Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics. As such, they can be a powerful tool for unsupervised representation learning from video or graph-structured data. We cast training Spectral Inference Networks as a bilevel optimization problem, which allows for online learning of multiple eigenfunctions. We show results of training Spectral Inference Networks on problems in quantum mechanics and feature learning for videos on synthetic datasets. Our results demonstrate that Spectral Inference Networks accurately recover eigenfunctions of linear operators and can discover interpretable representations from video in a fully unsupervised manner.
Deep Neural Networks with Multi-Branch Architectures Are Less Non-Convex
Authors: Hongyang Zhang, Junru Shao, Ruslan Salakhutdinov
Date: 2018-06-06 00:00:00

Several recently proposed architectures of neural networks such as ResNeXt,
Inception, Xception, SqueezeNet and Wide ResNet are based on the designing idea
of having multiple branches and have demonstrated improved performance in many
applications. We show that one cause for such success is due to the fact that
the multi-branch architecture is less non-convex in terms of duality gap. The
duality gap measures the degree of intrinsic non-convexity of an optimization
problem: smaller gap in relative value implies lower degree of intrinsic
non-convexity. The challenge is to quantitatively measure the duality gap of
highly non-convex problems such as deep neural networks. In this work, we
provide strong guarantees of this quantity for two classes of network
architectures. For the neural networks with arbitrary activation functions,
multi-branch architecture and a variant of hinge loss, we show that the duality
gap of both population and empirical risks shrinks to zero as the number of
branches increases. This result sheds light on better understanding the power
of over-parametrization where increasing the network width tends to make the
loss surface less non-convex. For the neural networks with linear activation
function and $\ell_2$ loss, we show that the duality gap of empirical risk is
zero. Our two results work for arbitrary depths and adversarial data, while the
analytical techniques might be of independent interest to non-convex
optimization more broadly. Experiments on both synthetic and real-world
datasets validate our results.
SOM-VAE: Interpretable Discrete Representation Learning on Time Series
Authors: Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, Gunnar Rätsch
Date: 2018-06-06 00:00:00
Proceeding: ICLR 2019 5
Tasks: Clustering, Dimensionality Reduction, Representation Learning, Time Series, Time Series Analysis, Time Series Clustering

High-dimensional time series are common in many domains. Since human
cognition is not optimized to work well in high-dimensional spaces, these areas
could benefit from interpretable low-dimensional representations. However, most
representation learning algorithms for time series data are difficult to
interpret. This is due to non-intuitive mappings from data features to salient
properties of the representation and non-smoothness over time. To address this
problem, we propose a new representation learning framework building on ideas
from interpretable discrete dimensionality reduction and deep generative
modeling. This framework allows us to learn discrete representations of time
series, which give rise to smooth and interpretable embeddings with superior
clustering performance. We introduce a new way to overcome the
non-differentiability in discrete representation learning and present a
gradient-based version of the traditional self-organizing map algorithm that is
more performant than the original. Furthermore, to allow for a probabilistic
interpretation of our method, we integrate a Markov model in the representation
space. This model uncovers the temporal transition structure, improves
clustering performance even further and provides additional explanatory
insights as well as a natural representation of uncertainty. We evaluate our
model in terms of clustering performance and interpretability on static
(Fashion-)MNIST data, a time series of linearly interpolated (Fashion-)MNIST
images, a chaotic Lorenz attractor system with two macro states, as well as on
a challenging real world medical time series application on the eICU data set.
Our learned representations compare favorably with competitor methods and
facilitate downstream tasks on the real world data.
GraKeL: A Graph Kernel Library in Python
Authors: Giannis Siglidis, Giannis Nikolentzos, Stratis Limnios, Christos Giatsidis, Konstantinos Skianis, Michalis Vazirgiannis
Date: 2018-06-06 00:00:00
Tasks: Clustering, General Classification, Graph Classification

The problem of accurately measuring the similarity between graphs is at the core of many applications in a variety of disciplines. Graph kernels have recently emerged as a promising approach to this problem. There are now many kernels, each focusing on different structural aspects of graphs. Here, we present GraKeL, a library that unifies several graph kernels into a common framework. The library is written in Python and adheres to the scikit-learn interface. It is simple to use and can be naturally combined with scikit-learn's modules to build a complete machine learning pipeline for tasks such as graph classification and clustering. The code is BSD licensed and is available at: https://github.com/ysig/GraKeL .
Neural Relational Inference for Interacting Systems
Authors: Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, Richard Zemel
Date: 2018-02-13 00:00:00
Proceeding: ICML 2018 7

Interacting systems are prevalent in nature, from dynamical systems in
physics to complex societal dynamics. The interplay of components can give rise
to complex behavior, which can often be explained using a simple model of the
system's constituent parts. In this work, we introduce the neural relational
inference (NRI) model: an unsupervised model that learns to infer interactions
while simultaneously learning the dynamics purely from observational data. Our
model takes the form of a variational auto-encoder, in which the latent code
represents the underlying interaction graph and the reconstruction is based on
graph neural networks. In experiments on simulated physical systems, we show
that our NRI model can accurately recover ground-truth interactions in an
unsupervised manner. We further demonstrate that we can find an interpretable
structure and predict complex dynamics in real motion capture and sports
tracking data.
Incorporating Features Learned by an Enhanced Deep Knowledge Tracing Model for STEM/Non-STEM Job Prediction
Authors: Chun-kit Yeung, Zizheng Lin, Kai Yang, Dit-yan Yeung
Date: 2018-06-06 00:00:00
Tasks: Job Prediction, Knowledge Tracing

The 2017 ASSISTments Data Mining competition aims to use data from a
longitudinal study for predicting a brand-new outcome of students which had
never been studied before by the educational data mining research community.
Specifically, it facilitates research in developing predictive models that
predict whether the first job of a student out of college belongs to a STEM
(the acronym for science, technology, engineering, and mathematics) field. This
is based on the student's learning history on the ASSISTments blended learning
platform in the form of extensive clickstream data gathered during the middle
school years. To tackle this challenge, we first estimate the expected
knowledge state of students with respect to different mathematical skills using
a deep knowledge tracing (DKT) model and an enhanced DKT (DKT+) model. We then
combine the features corresponding to the DKT/DKT+ expected knowledge state
with other features extracted directly from the student profile in the dataset
to train several machine learning models for the STEM/non-STEM job prediction.
Our experiments show that models trained with the combined features generally
perform better than the models trained with the student profile alone. Detailed
analysis of the student's knowledge state reveals that, when compared with
non-STEM students, STEM students generally show a higher mastery level and a
higher learning gain in mathematics.
Online Variance Reduction for Stochastic Optimization
Authors: Zalán Borsos, Andreas Krause, Kfir. Y. Levy
Date: 2018-02-13 00:00:00
Tasks: Stochastic Optimization

Modern stochastic optimization methods often rely on uniform sampling which
is agnostic to the underlying characteristics of the data. This might degrade
the convergence by yielding estimates that suffer from a high variance. A
possible remedy is to employ non-uniform importance sampling techniques, which
take the structure of the dataset into account. In this work, we investigate a
recently proposed setting which poses variance reduction as an online
optimization problem with bandit feedback. We devise a novel and efficient
algorithm for this setting that finds a sequence of importance sampling
distributions competitive with the best fixed distribution in hindsight, the
first result of this kind. While we present our method for sampling datapoints,
it naturally extends to selecting coordinates or even blocks of thereof.
Empirical validations underline the benefits of our method in several settings.
Asymptotic Bayesian Generalization Error in Latent Dirichlet Allocation and Stochastic Matrix Factorization
Authors: Naoki Hayashi, Sumio Watanabe
Date: 2017-09-13 00:00:00
Tasks: Bayesian Inference, Topic Models

Latent Dirichlet allocation (LDA) is useful in document analysis, image processing, and many information systems; however, its generalization performance has been left unknown because it is a singular learning machine to which regular statistical theory can not be applied. Stochastic matrix factorization (SMF) is a restricted matrix factorization in which matrix factors are stochastic; the column of the matrix is in a simplex. SMF is being applied to image recognition and text mining. We can understand SMF as a statistical model by which a stochastic matrix of given data is represented by a product of two stochastic matrices, whose generalization performance has also been left unknown because of non-regularity. In this paper, by using an algebraic and geometric method, we show the analytic equivalence of LDA and SMF, both of which have the same real log canonical threshold (RLCT), resulting in that they asymptotically have the same Bayesian generalization error and the same log marginal likelihood. Moreover, we derive the upper bound of the RLCT and prove that it is smaller than the dimension of the parameter divided by two, hence the Bayesian generalization errors of them are smaller than those of regular statistical models.
Addressing Two Problems in Deep Knowledge Tracing via Prediction-Consistent Regularization
Authors: Chun-kit Yeung, Dit-yan Yeung
Date: 2018-06-06 00:00:00
Tasks: Knowledge Tracing, Vocal Bursts Valence Prediction

Knowledge tracing is one of the key research areas for empowering
personalized education. It is a task to model students' mastery level of a
knowledge component (KC) based on their historical learning trajectories. In
recent years, a recurrent neural network model called deep knowledge tracing
(DKT) has been proposed to handle the knowledge tracing task and literature has
shown that DKT generally outperforms traditional methods. However, through our
extensive experimentation, we have noticed two major problems in the DKT model.
The first problem is that the model fails to reconstruct the observed input. As
a result, even when a student performs well on a KC, the prediction of that
KC's mastery level decreases instead, and vice versa. Second, the predicted
performance for KCs across time-steps is not consistent. This is undesirable
and unreasonable because student's performance is expected to transit gradually
over time. To address these problems, we introduce regularization terms that
correspond to reconstruction and waviness to the loss function of the original
DKT model to enhance the consistency in prediction. Experiments show that the
regularized loss function effectively alleviates the two problems without
degrading the original task of DKT.
Open Domain Suggestion Mining: Problem Definition and Datasets
Authors: Sapna Negi, Maarten de Rijke, Paul Buitelaar
Date: 2018-06-06 00:00:00
Tasks: Suggestion mining

We propose a formal definition for the task of suggestion mining in the
context of a wide range of open domain applications. Human perception of the
term \emph{suggestion} is subjective and this effects the preparation of hand
labeled datasets for the task of suggestion mining. Existing work either lacks
a formal problem definition and annotation procedure, or provides domain and
application specific definitions. Moreover, many previously used manually
labeled datasets remain proprietary. We first present an annotation study, and
based on our observations propose a formal task definition and annotation
procedure for creating benchmark datasets for suggestion mining. With this
study, we also provide publicly available labeled datasets for suggestion
mining in multiple domains.
On the Long-Term Memory of Deep Recurrent Networks
Authors: Yoav Levine, Or Sharir, Alon Ziv, Amnon Shashua
Date: 2017-10-25 00:00:00
Tasks: Attribute, Tensor Networks

A key attribute that drives the unprecedented success of modern Recurrent
Neural Networks (RNNs) on learning tasks which involve sequential data, is
their ability to model intricate long-term temporal dependencies. However, a
well established measure of RNNs long-term memory capacity is lacking, and thus
formal understanding of the effect of depth on their ability to correlate data
throughout time is limited. Specifically, existing depth efficiency results on
convolutional networks do not suffice in order to account for the success of
deep RNNs on data of varying lengths. In order to address this, we introduce a
measure of the network's ability to support information flow across time,
referred to as the Start-End separation rank, which reflects the distance of
the function realized by the recurrent network from modeling no dependency
between the beginning and end of the input sequence. We prove that deep
recurrent networks support Start-End separation ranks which are combinatorially
higher than those supported by their shallow counterparts. Thus, we establish
that depth brings forth an overwhelming advantage in the ability of recurrent
networks to model long-term dependencies, and provide an exemplar of
quantifying this key attribute which may be readily extended to other RNN
architectures of interest, e.g. variants of LSTM networks. We obtain our
results by considering a class of recurrent networks referred to as Recurrent
Arithmetic Circuits, which merge the hidden state with the input via the
Multiplicative Integration operation, and empirically demonstrate the discussed
phenomena on common RNNs. Finally, we employ the tool of quantum Tensor
Networks to gain additional graphic insight regarding the complexity brought
forth by depth in recurrent networks.
SBAF: A New Activation Function for Artificial Neural Net based Habitability Classification
Authors: Snehanshu Saha, Archana Mathur, Kakoli Bora, Surbhi Agrawal, Suryoday Basak
Date: 2018-06-06 00:00:00
Tasks: BIG-bench Machine Learning, General Classification

We explore the efficacy of using a novel activation function in Artificial
Neural Networks (ANN) in characterizing exoplanets into different classes. We
call this Saha-Bora Activation Function (SBAF) as the motivation is derived
from long standing understanding of using advanced calculus in modeling
habitability score of Exoplanets. The function is demonstrated to possess nice
analytical properties and doesn't seem to suffer from local oscillation
problems. The manuscript presents the analytical properties of the activation
function and the architecture implemented on the function. Keywords:
Astroinformatics, Machine Learning, Exoplanets, ANN, Activation Function.
The Mechanics of n-Player Differentiable Games
Authors: David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, Thore Graepel
Date: 2018-02-15 00:00:00
Proceeding: ICML 2018 7

The cornerstone underpinning deep learning is the guarantee that gradient
descent on an objective converges to local minima. Unfortunately, this
guarantee fails in settings, such as generative adversarial nets, where there
are multiple interacting losses. The behavior of gradient-based methods in
games is not well understood -- and is becoming increasingly important as
adversarial and multi-objective architectures proliferate. In this paper, we
develop new techniques to understand and control the dynamics in general games.
The key result is to decompose the second-order dynamics into two components.
The first is related to potential games, which reduce to gradient descent on an
implicit function; the second relates to Hamiltonian games, a new class of
games that obey a conservation law, akin to conservation laws in classical
mechanical systems. The decomposition motivates Symplectic Gradient Adjustment
(SGA), a new algorithm for finding stable fixed points in general games. Basic
experiments show SGA is competitive with recently proposed algorithms for
finding stable fixed points in GANs -- whilst at the same time being applicable
to -- and having guarantees in -- much more general games.
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
Authors: Boris Hanin
Date: 2018-01-11 00:00:00
Proceeding: NeurIPS 2018 12

We give a rigorous analysis of the statistical behavior of gradients in a
randomly initialized fully connected network N with ReLU activations. Our
results show that the empirical variance of the squares of the entries in the
input-output Jacobian of N is exponential in a simple architecture-dependent
constant beta, given by the sum of the reciprocals of the hidden layer widths.
When beta is large, the gradients computed by N at initialization vary wildly.
Our approach complements the mean field theory analysis of random networks.
From this point of view, we rigorously compute finite width corrections to the
statistics of gradients at the edge of chaos.
PointFlowNet: Learning Representations for Rigid Motion Estimation from Point Clouds
Authors: Aseem Behl, Despoina Paschalidou, Simon Donné, Andreas Geiger
Date: 2018-06-06 00:00:00
Proceeding: CVPR 2019 6
Tasks: Motion Estimation, Scene Flow Estimation, Self-Driving Cars, Translation

Despite significant progress in image-based 3D scene flow estimation, the
performance of such approaches has not yet reached the fidelity required by
many applications. Simultaneously, these applications are often not restricted
to image-based estimation: laser scanners provide a popular alternative to
traditional cameras, for example in the context of self-driving cars, as they
directly yield a 3D point cloud. In this paper, we propose to estimate 3D
motion from such unstructured point clouds using a deep neural network. In a
single forward pass, our model jointly predicts 3D scene flow as well as the 3D
bounding box and rigid body motion of objects in the scene. While the prospect
of estimating 3D scene flow from unstructured point clouds is promising, it is
also a challenging task. We show that the traditional global representation of
rigid body motion prohibits inference by CNNs, and propose a translation
equivariant representation to circumvent this problem. For training our deep
network, a large dataset is required. Because of this, we augment real scans
from KITTI with virtual objects, realistically modeling occlusions and
simulating sensor noise. A thorough comparison with classic and learning-based
techniques highlights the robustness of the proposed approach.