pwc-archive/papers-with-abstracts huggingface dataset:
Detecting Dead Weights and Units in Neural Networks
Authors: Utku Evci
Date: 2018-06-15 00:00:00
Tasks: Quantization

Deep Neural Networks are highly over-parameterized and the size of the neural
networks can be reduced significantly after training without any decrease in
performance. One can clearly see this phenomenon in a wide range of
architectures trained for various problems. Weight/channel pruning,
distillation, quantization, matrix factorization are some of the main methods
one can use to remove the redundancy to come up with smaller and faster models.
  This work starts with a short informative chapter, where we motivate the
pruning idea and provide the necessary notation. In the second chapter, we
compare various saliency scores in the context of parameter pruning. Using the
insights obtained from this comparison and stating the problems it brings we
motivate why pruning units instead of the individual parameters might be a
better idea. We propose some set of definitions to quantify and analyze units
that don't learn and create any useful information. We propose an efficient way
for detecting dead units and use it to select which units to prune. We get 5x
model size reduction through unit-wise pruning on MNIST.
Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Authors: L. Elisa Celis, Lingxiao Huang, Vijay Keswani, Nisheeth K. Vishnoi
Date: 2018-06-15 00:00:00
Tasks: Classification, Fairness, General Classification

Developing classification algorithms that are fair with respect to sensitive attributes of the data has become an important problem due to the growing deployment of classification algorithms in various social contexts. Several recent works have focused on fairness with respect to a specific metric, modeled the corresponding fair classification problem as a constrained optimization problem, and developed tailored algorithms to solve them. Despite this, there still remain important metrics for which we do not have fair classifiers and many of the aforementioned algorithms do not come with theoretical guarantees; perhaps because the resulting optimization problem is non-convex. The main contribution of this paper is a new meta-algorithm for classification that takes as input a large class of fairness constraints, with respect to multiple non-disjoint sensitive attributes, and which comes with provable guarantees. This is achieved by first developing a meta-algorithm for a large family of classification problems with convex constraints, and then showing that classification problems with general types of fairness constraints can be reduced to those in this family. We present empirical results that show that our algorithm can achieve near-perfect fairness with respect to various fairness metrics, and that the loss in accuracy due to the imposed fairness constraints is often small. Overall, this work unifies several prior works on fair classification, presents a practical algorithm with theoretical guarantees, and can handle fairness metrics that were previously not possible.
An Online Prediction Algorithm for Reinforcement Learning with Linear Function Approximation using Cross Entropy Method
Authors: Ajin George Joseph, Shalabh Bhatnagar
Date: 2018-06-15 00:00:00
Tasks: Computational Efficiency, Reinforcement Learning, Reinforcement Learning (RL)

In this paper, we provide two new stable online algorithms for the problem of
prediction in reinforcement learning, \emph{i.e.}, estimating the value
function of a model-free Markov reward process using the linear function
approximation architecture and with memory and computation costs scaling
quadratically in the size of the feature set. The algorithms employ the
multi-timescale stochastic approximation variant of the very popular cross
entropy (CE) optimization method which is a model based search method to find
the global optimum of a real-valued function. A proof of convergence of the
algorithms using the ODE method is provided. We supplement our theoretical
results with experimental comparisons. The algorithms achieve good performance
fairly consistently on many RL benchmark problems with regards to computational
efficiency, accuracy and stability.
Deep Lip Reading: a comparison of models and an online application
Authors: Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman
Date: 2018-06-15 00:00:00
Tasks: Language Modeling, Language Modelling, Lip Reading, speech-recognition, Speech Recognition, Visual Speech Recognition

The goal of this paper is to develop state-of-the-art models for lip reading
-- visual speech recognition. We develop three architectures and compare their
accuracy and training times: (i) a recurrent model using LSTMs; (ii) a fully
convolutional model; and (iii) the recently proposed transformer model. The
recurrent and fully convolutional models are trained with a Connectionist
Temporal Classification loss and use an explicit language model for decoding,
the transformer is a sequence-to-sequence model. Our best performing model
improves the state-of-the-art word error rate on the challenging BBC-Oxford Lip
Reading Sentences 2 (LRS2) benchmark dataset by over 20 percent.
  As a further contribution we investigate the fully convolutional model when
used for online (real time) lip reading of continuous speech, and show that it
achieves high performance with low latency.
Versatile Auxiliary Classifier with Generative Adversarial Network (VAC+GAN)
Authors: Shabab Bazrafkan, Hossein Javidnia, Peter Corcoran
Date: 2018-05-01 00:00:00
Tasks: General Classification, Generative Adversarial Network

One of the most interesting challenges in Artificial Intelligence is to train
conditional generators which are able to provide labeled adversarial samples
drawn from a specific distribution. In this work, a new framework is presented
to train a deep conditional generator by placing a classifier in parallel with
the discriminator and back propagate the classification error through the
generator network. The method is versatile and is applicable to any variations
of Generative Adversarial Network (GAN) implementation, and also gives superior
results compared to similar methods.
Computationally Efficient Estimation of the Spectral Gap of a Markov Chain
Authors: Richard Combes, Mikael Touati
Date: 2018-06-15 00:00:00

We consider the problem of estimating from sample paths the absolute spectral
gap $\gamma_*$ of a reversible, irreducible and aperiodic Markov chain
$(X_t)_{t \in \mathbb{N}}$ over a finite state space $\Omega$. We propose the
${\tt UCPI}$ (Upper Confidence Power Iteration) algorithm for this problem, a
low-complexity algorithm which estimates the spectral gap in time ${\cal O}(n)$
and memory space ${\cal O}((\ln n)^2)$ given $n$ samples. This is in stark
contrast with most known methods which require at least memory space ${\cal
O}(|\Omega|)$, so that they cannot be applied to large state spaces.
Furthermore, ${\tt UCPI}$ is amenable to parallel implementation.
High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
Authors: Tim Pearce, Mohamed Zaki, Alexandra Brintrup, Andy Neely
Date: 2018-02-20 00:00:00
Proceeding: ICML 2018 7
Tasks: Form, Prediction Intervals, regression, Uncertainty Quantification

This paper considers the generation of prediction intervals (PIs) by neural
networks for quantifying uncertainty in regression tasks. It is axiomatic that
high-quality PIs should be as narrow as possible, whilst capturing a specified
portion of data. We derive a loss function directly from this axiom that
requires no distributional assumption. We show how its form derives from a
likelihood principle, that it can be used with gradient descent, and that model
uncertainty is accounted for in ensembled form. Benchmark experiments show the
method outperforms current state-of-the-art uncertainty quantification methods,
reducing average PI width by over 10%.
Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
Authors: Tim Pearce, Nicolas Anastassacos, Mohamed Zaki, Andy Neely
Date: 2018-05-29 00:00:00
Tasks: Bayesian Inference, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The use of ensembles of neural networks (NNs) for the quantification of
predictive uncertainty is widespread. However, the current justification is
intuitive rather than analytical. This work proposes one minor modification to
the normal ensembling methodology, which we prove allows the ensemble to
perform Bayesian inference, hence converging to the corresponding Gaussian
Process as both the total number of NNs, and the size of each, tend to
infinity. This working paper provides early-stage results in a reinforcement
learning setting, analysing the practicality of the technique for an ensemble
of small, finite number. Using the uncertainty estimates produced by anchored
ensembles to govern the exploration-exploitation process results in steadier,
more stable learning.
Three Factors Influencing Minima in SGD
Authors: Stanisław Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos Storkey
Date: 2017-11-13 00:00:00
Proceeding: ICLR 2018 1
Tasks: Memorization, Open-Ended Question Answering

We investigate the dynamical and convergent properties of stochastic gradient
descent (SGD) applied to Deep Neural Networks (DNNs). Characterizing the
relation between learning rate, batch size and the properties of the final
minima, such as width or generalization, remains an open question. In order to
tackle this problem we investigate the previously proposed approximation of SGD
by a stochastic differential equation (SDE). We theoretically argue that three
factors - learning rate, batch size and gradient covariance - influence the
minima found by SGD. In particular we find that the ratio of learning rate to
batch size is a key determinant of SGD dynamics and of the width of the final
minima, and that higher values of the ratio lead to wider minima and often
better generalization. We confirm these findings experimentally. Further, we
include experiments which show that learning rate schedules can be replaced
with batch size schedules and that the ratio of learning rate to batch size is
an important factor influencing the memorization process.
The Toybox Dataset of Egocentric Visual Object Transformations
Authors: Xiaohan Wang, Tengyu Ma, James Ainooson, Seunghwan Cha, Xiaotian Wang, Azhar Molla, Maithilee Kunda
Date: 2018-06-15 00:00:00
Tasks: Object, Object Recognition, Translation

In object recognition research, many commonly used datasets (e.g., ImageNet
and similar) contain relatively sparse distributions of object instances and
views, e.g., one might see a thousand different pictures of a thousand
different giraffes, mostly taken from a few conventionally photographed angles.
These distributional properties constrain the types of computational
experiments that are able to be conducted with such datasets, and also do not
reflect naturalistic patterns of embodied visual experience. As a contribution
to the small (but growing) number of multi-view object datasets that have been
created to bridge this gap, we introduce a new video dataset called Toybox that
contains egocentric (i.e., first-person perspective) videos of common household
objects and toys being manually manipulated to undergo structured
transformations, such as rotation, translation, and zooming. To illustrate
potential uses of Toybox, we also present initial neural network experiments
that examine 1) how training on different distributions of object instances and
views affects recognition performance, and 2) how viewpoint-dependent object
concepts are represented within the hidden layers of a trained network.
Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow
Authors: Steffen Wiewel, Moritz Becher, Nils Thuerey
Date: 2018-02-27 00:00:00
Tasks: Dimensionality Reduction

We propose a method for the data-driven inference of temporal evolutions of
physical functions with deep learning. More specifically, we target fluid
flows, i.e. Navier-Stokes problems, and we propose a novel LSTM-based approach
to predict the changes of pressure fields over time. The central challenge in
this context is the high dimensionality of Eulerian space-time data sets. We
demonstrate for the first time that dense 3D+time functions of physics system
can be predicted within the latent spaces of neural networks, and we arrive at
a neural-network based simulation algorithm with significant practical
speed-ups. We highlight the capabilities of our method with a series of complex
liquid simulations, and with a set of single-phase buoyancy simulations. With a
set of trained networks, our method is more than two orders of magnitudes
faster than a traditional pressure solver. Additionally, we present and discuss
a series of detailed evaluations for the different components of our algorithm.
One-Shot Unsupervised Cross Domain Translation
Authors: Sagie Benaim, Lior Wolf
Date: 2018-06-15 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Translation, Unsupervised Image-To-Image Translation, Zero-Shot Learning

Given a single image x from domain A and a set of images from domain B, our
task is to generate the analogous of x in B. We argue that this task could be a
key AI capability that underlines the ability of cognitive agents to act in the
world and present empirical evidence that the existing unsupervised domain
translation methods fail on this task. Our method follows a two step process.
First, a variational autoencoder for domain B is trained. Then, given the new
sample x, we create a variational autoencoder for domain A by adapting the
layers that are close to the image in order to directly fit x, and only
indirectly adapt the other layers. Our experiments indicate that the new method
does as well, when trained on one sample x, as the existing domain transfer
methods, when these enjoy a multitude of training samples from domain A. Our
code is made publicly available at
https://github.com/sagiebenaim/OneShotTranslation
Variational Attention for Sequence-to-Sequence Models
Authors: Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, Pascal Poupart
Date: 2017-12-21 00:00:00
Proceeding: COLING 2018 8
Tasks: Decoder, Diversity

The variational encoder-decoder (VED) encodes source information as a set of
random variables using a neural network, which in turn is decoded into target
data using another neural network. In natural language processing,
sequence-to-sequence (Seq2Seq) models typically serve as encoder-decoder
networks. When combined with a traditional (deterministic) attention mechanism,
the variational latent space may be bypassed by the attention model, and thus
becomes ineffective. In this paper, we propose a variational attention
mechanism for VED, where the attention vector is also modeled as Gaussian
distributed random variables. Results on two experiments show that, without
loss of quality, our proposed method alleviates the bypassing phenomenon as it
increases the diversity of generated sentences.
Low-shot learning with large-scale diffusion
Authors: Matthijs Douze, Arthur Szlam, Bharath Hariharan, Hervé Jégou
Date: 2017-06-07 00:00:00
Proceeding: CVPR 2018 6
Tasks: Few-Shot Image Classification, graph construction

This paper considers the problem of inferring image labels from images when
only a few annotated examples are available at training time. This setup is
often referred to as low-shot learning, where a standard approach is to
re-train the last few layers of a convolutional neural network learned on
separate classes for which training examples are abundant. We consider a
semi-supervised setting based on a large collection of images to support label
propagation. This is possible by leveraging the recent advances on large-scale
similarity graph construction.
  We show that despite its conceptual simplicity, scaling label propagation up
to hundred millions of images leads to state of the art accuracy in the
low-shot learning regime.
Homonym Detection in Curated Bibliographies: Learning from dblp's Experience (full version)
Authors: Marcel R. Ackermann, Florian Reitz
Date: 2018-06-15 00:00:00
Tasks: BIG-bench Machine Learning

Identifying (and fixing) homonymous and synonymous author profiles is one of
the major tasks of curating personalized bibliographic metadata repositories
like the dblp computer science bibliography. In this paper, we present and
evaluate a machine learning approach to identify homonymous author
bibliographies using a simple multilayer perceptron setup. We train our model
on a novel gold-standard data set derived from the past years of active, manual
curation at the dblp computer science bibliography.
Detecting abnormal events in video using Narrowed Normality Clusters
Authors: Radu Tudor Ionescu, Sorina Smeureanu, Marius Popescu, Bogdan Alexe
Date: 2018-01-12 00:00:00
Tasks: Anomaly Detection, Clustering, CPU, Event Detection, Outlier Detection

We formulate the abnormal event detection problem as an outlier detection
task and we propose a two-stage algorithm based on k-means clustering and
one-class Support Vector Machines (SVM) to eliminate outliers. In the feature
extraction stage, we propose to augment spatio-temporal cubes with deep
appearance features extracted from the last convolutional layer of a
pre-trained neural network. After extracting motion and appearance features
from the training video containing only normal events, we apply k-means
clustering to find clusters representing different types of normal motion and
appearance features. In the first stage, we consider that clusters with fewer
samples (with respect to a given threshold) contain mostly outliers, and we
eliminate these clusters altogether. In the second stage, we shrink the borders
of the remaining clusters by training a one-class SVM model on each cluster. To
detected abnormal events in the test video, we analyze each test sample and
consider its maximum normality score provided by the trained one-class SVM
models, based on the intuition that a test sample can belong to only one
cluster of normality. If the test sample does not fit well in any narrowed
normality cluster, then it is labeled as abnormal. We compare our method with
several state-of-the-art methods on three benchmark data sets. The empirical
results indicate that our abnormal event detection framework can achieve better
results in most cases, while processing the test video in real-time at 24
frames per second on a single CPU.
Real-Time Deep Learning Method for Abandoned Luggage Detection in Video
Authors: Sorina Smeureanu, Radu Tudor Ionescu
Date: 2018-03-03 00:00:00
Tasks: Deep Learning, Motion Estimation, object-detection, Object Detection

Recent terrorist attacks in major cities around the world have brought many
casualties among innocent citizens. One potential threat is represented by
abandoned luggage items (that could contain bombs or biological warfare) in
public areas. In this paper, we describe an approach for real-time automatic
detection of abandoned luggage in video captured by surveillance cameras. The
approach is comprised of two stages: (i) static object detection based on
background subtraction and motion estimation and (ii) abandoned luggage
recognition based on a cascade of convolutional neural networks (CNN). To train
our neural networks we provide two types of examples: images collected from the
Internet and realistic examples generated by imposing various suitcases and
bags over the scene's background. We present empirical results demonstrating
that our approach yields better performance than a strong CNN baseline method.
Optimizing the Trade-off between Single-Stage and Two-Stage Object Detectors using Image Difficulty Prediction
Authors: Petru Soviany, Radu Tudor Ionescu
Date: 2018-03-23 00:00:00
Tasks: Object, object-detection, Object Detection, Region Proposal, regression

There are mainly two types of state-of-the-art object detectors. On one hand,
we have two-stage detectors, such as Faster R-CNN (Region-based Convolutional
Neural Networks) or Mask R-CNN, that (i) use a Region Proposal Network to
generate regions of interests in the first stage and (ii) send the region
proposals down the pipeline for object classification and bounding-box
regression. Such models reach the highest accuracy rates, but are typically
slower. On the other hand, we have single-stage detectors, such as YOLO (You
Only Look Once) and SSD (Singe Shot MultiBox Detector), that treat object
detection as a simple regression problem by taking an input image and learning
the class probabilities and bounding box coordinates. Such models reach lower
accuracy rates, but are much faster than two-stage object detectors. In this
paper, we propose to use an image difficulty predictor to achieve an optimal
trade-off between accuracy and speed in object detection. The image difficulty
predictor is applied on the test images to split them into easy versus hard
images. Once separated, the easy images are sent to the faster single-stage
detector, while the hard images are sent to the more accurate two-stage
detector. Our experiments on PASCAL VOC 2007 show that using image difficulty
compares favorably to a random split of the images. Our method is flexible, in
that it allows to choose a desired threshold for splitting the images into easy
versus hard.
Partially-Supervised Image Captioning
Authors: Peter Anderson, Stephen Gould, Mark Johnson
Date: 2018-06-15 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Image Captioning, Object, object-detection, Object Detection, Sentence

Image captioning models are becoming increasingly successful at describing
the content of images in restricted domains. However, if these models are to
function in the wild - for example, as assistants for people with impaired
vision - a much larger number and variety of visual concepts must be
understood. To address this problem, we teach image captioning models new
visual concepts from labeled images and object detection datasets. Since image
labels and object classes can be interpreted as partial captions, we formulate
this problem as learning from partially-specified sequence data. We then
propose a novel algorithm for training sequence models, such as recurrent
neural networks, on partially-specified sequences which we represent using
finite state automata. In the context of image captioning, our method lifts the
restriction that previously required image captioning models to be trained on
paired image-sentence corpora only, or otherwise required specialized model
architectures to take advantage of alternative data modalities. Applying our
approach to an existing neural captioning model, we achieve state of the art
results on the novel object captioning task using the COCO dataset. We further
show that we can train a captioning model to describe new visual concepts from
the Open Images dataset while maintaining competitive COCO evaluation scores.
On Machine Learning and Structure for Mobile Robots
Authors: Markus Wulfmeier
Date: 2018-06-15 00:00:00
Tasks: BIG-bench Machine Learning

Due to recent advances - compute, data, models - the role of learning in
autonomous systems has expanded significantly, rendering new applications
possible for the first time. While some of the most significant benefits are
obtained in the perception modules of the software stack, other aspects
continue to rely on known manual procedures based on prior knowledge on
geometry, dynamics, kinematics etc. Nonetheless, learning gains relevance in
these modules when data collection and curation become easier than manual rule
design. Building on this coarse and broad survey of current research, the final
sections aim to provide insights into future potentials and challenges as well
as the necessity of structure in current practical applications.