pwc-archive/papers-with-abstracts huggingface dataset:
Dropout as data augmentation
Authors: Xavier Bouthillier, Kishore Konda, Pascal Vincent, Roland Memisevic
Date: 2015-06-29 00:00:00
Tasks: Data Augmentation

Dropout is typically interpreted as bagging a large number of models sharing
parameters. We show that using dropout in a network can also be interpreted as
a kind of data augmentation in the input space without domain knowledge. We
present an approach to projecting the dropout noise within a network back into
the input space, thereby generating augmented versions of the training data,
and we show that training a deterministic network on the augmented samples
yields similar results. Finally, we propose a new dropout noise scheme based on
our observations and show that it improves dropout results without adding
significant computational cost.
Learning Dense Convolutional Embeddings for Semantic Segmentation
Authors: Adam W. Harley, Konstantinos G. Derpanis, Iasonas Kokkinos
Date: 2015-11-13 00:00:00
Tasks: General Classification, Object, Semantic Segmentation

This paper proposes a new deep convolutional neural network (DCNN)
architecture that learns pixel embeddings, such that pairwise distances between
the embeddings can be used to infer whether or not the pixels lie on the same
region. That is, for any two pixels on the same object, the embeddings are
trained to be similar; for any pair that straddles an object boundary, the
embeddings are trained to be dissimilar. Experimental results show that when
this embedding network is used in conjunction with a DCNN trained on semantic
segmentation, there is a systematic improvement in per-pixel classification
accuracy. Our contributions are integrated in the popular Caffe deep learning
framework, and consist in straightforward modifications to convolution
routines. As such, they can be exploited for any task involving convolution
layers.
A convnet for non-maximum suppression
Authors: Jan Hosang, Rodrigo Benenson, Bernt Schiele
Date: 2015-11-19 00:00:00
Tasks: Clustering, Object, object-detection, Object Detection, Pedestrian Detection

Non-maximum suppression (NMS) is used in virtually all state-of-the-art
object detection pipelines. While essential object detection ingredients such
as features, classifiers, and proposal methods have been extensively researched
surprisingly little work has aimed to systematically address NMS. The de-facto
standard for NMS is based on greedy clustering with a fixed distance threshold,
which forces to trade-off recall versus precision. We propose a convnet
designed to perform NMS of a given set of detections. We report experiments on
a synthetic setup, and results on crowded pedestrian detection scenes. Our
approach overcomes the intrinsic limitations of greedy NMS, obtaining better
recall and precision.
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
Authors: Alec Radford, Luke Metz, Soumith Chintala
Date: 2015-11-19 00:00:00
Tasks: Conditional Image Generation, Image Clustering, Representation Learning

In recent years, supervised learning with convolutional networks (CNNs) has
seen huge adoption in computer vision applications. Comparatively, unsupervised
learning with CNNs has received less attention. In this work we hope to help
bridge the gap between the success of CNNs for supervised learning and
unsupervised learning. We introduce a class of CNNs called deep convolutional
generative adversarial networks (DCGANs), that have certain architectural
constraints, and demonstrate that they are a strong candidate for unsupervised
learning. Training on various image datasets, we show convincing evidence that
our deep convolutional adversarial pair learns a hierarchy of representations
from object parts to scenes in both the generator and discriminator.
Additionally, we use the learned features for novel tasks - demonstrating their
applicability as general image representations.
An Information Retrieval Approach to Finding Dependent Subspaces of Multiple Views
Authors: Ziyuan Lin, Jaakko Peltonen
Date: 2015-11-19 00:00:00
Tasks: Information Retrieval, Retrieval

Finding relationships between multiple views of data is essential both for
exploratory analysis and as pre-processing for predictive tasks. A prominent
approach is to apply variants of Canonical Correlation Analysis (CCA), a
classical method seeking correlated components between views. The basic CCA is
restricted to maximizing a simple dependency criterion, correlation, measured
directly between data coordinates. We introduce a new method that finds
dependent subspaces of views directly optimized for the data analysis task of
\textit{neighbor retrieval between multiple views}. We optimize mappings for
each view such as linear transformations to maximize cross-view similarity
between neighborhoods of data samples. The criterion arises directly from the
well-defined retrieval task, detects nonlinear and local similarities, is able
to measure dependency of data relationships rather than only individual data
coordinates, and is related to well understood measures of information
retrieval quality. In experiments we show the proposed method outperforms
alternatives in preserving cross-view neighborhood similarities, and yields
insights into local dependencies between multiple views.
Conditional Computation in Neural Networks for faster models
Authors: Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, Doina Precup
Date: 2015-11-19 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Deep learning has become the state-of-art tool in many applications, but the
evaluation and training of deep models can be time-consuming and
computationally expensive. The conditional computation approach has been
proposed to tackle this problem (Bengio et al., 2013; Davis & Arel, 2013). It
operates by selectively activating only parts of the network at a time. In this
paper, we use reinforcement learning as a tool to optimize conditional
computation policies. More specifically, we cast the problem of learning
activation-dependent policies for dropping out blocks of units as a
reinforcement learning problem. We propose a learning scheme motivated by
computation speed, capturing the idea of wanting to have parsimonious
activations while maintaining prediction accuracy. We apply a policy gradient
algorithm for learning policies that optimize this loss function and propose a
regularization mechanism that encourages diversification of the dropout policy.
We present encouraging empirical results showing that this approach improves
the speed of computation without impacting the quality of the approximation.
Recurrent Models for Auditory Attention in Multi-Microphone Distance Speech Recognition
Authors: Suyoun Kim, Ian Lane
Date: 2015-11-19 00:00:00
Tasks: Robust Speech Recognition, Speech Enhancement, speech-recognition, Speech Recognition

Integration of multiple microphone data is one of the key ways to achieve
robust speech recognition in noisy environments or when the speaker is located
at some distance from the input device. Signal processing techniques such as
beamforming are widely used to extract a speech signal of interest from
background noise. These techniques, however, are highly dependent on prior
spatial information about the microphones and the environment in which the
system is being used. In this work, we present a neural attention network that
directly combines multi-channel audio to generate phonetic states without
requiring any prior knowledge of the microphone layout or any explicit signal
preprocessing for speech enhancement. We embed an attention mechanism within a
Recurrent Neural Network (RNN) based acoustic model to automatically tune its
attention to a more reliable input source. Unlike traditional multi-channel
preprocessing, our system can be optimized towards the desired output in one
step. Although attention-based models have recently achieved impressive results
on sequence-to-sequence learning, no attention mechanisms have previously been
applied to learn potentially asynchronous and non-stationary multiple inputs.
We evaluate our neural attention model on the CHiME-3 challenge task, and show
that the model achieves comparable performance to beamforming using a purely
data-driven method.
Manifold Regularized Discriminative Neural Networks
Authors: Shuangfei Zhai, Zhongfei Zhang
Date: 2015-11-19 00:00:00

Unregularized deep neural networks (DNNs) can be easily overfit with a
limited sample size. We argue that this is mostly due to the disriminative
nature of DNNs which directly model the conditional probability (or score) of
labels given the input. The ignorance of input distribution makes DNNs
difficult to generalize to unseen data. Recent advances in regularization
techniques, such as pretraining and dropout, indicate that modeling input data
distribution (either explicitly or implicitly) greatly improves the
generalization ability of a DNN. In this work, we explore the manifold
hypothesis which assumes that instances within the same class lie in a smooth
manifold. We accordingly propose two simple regularizers to a standard
discriminative DNN. The first one, named Label-Aware Manifold Regularization,
assumes the availability of labels and penalizes large norms of the loss
function w.r.t. data points. The second one, named Label-Independent Manifold
Regularization, does not use label information and instead penalizes the
Frobenius norm of the Jacobian matrix of prediction scores w.r.t. data points,
which makes semi-supervised learning possible. We perform extensive control
experiments on fully supervised and semi-supervised tasks using the MNIST,
CIFAR10 and SVHN datasets and achieve excellent results.
How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
Authors: Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, Synho Do
Date: 2015-11-19 00:00:00
Tasks: Classification, Computed Tomography (CT), General Classification, image-classification, Image Classification, Medical Image Classification

The use of Convolutional Neural Networks (CNN) in natural image
classification systems has produced very impressive results. Combined with the
inherent nature of medical images that make them ideal for deep-learning,
further application of such systems to medical image classification holds much
promise. However, the usefulness and potential impact of such a system can be
completely negated if it does not reach a target accuracy. In this paper, we
present a study on determining the optimum size of the training data set
necessary to achieve high classification accuracy with low variance in medical
image classification systems. The CNN was applied to classify axial Computed
Tomography (CT) images into six anatomical classes. We trained the CNN using
six different sizes of training data set (5, 10, 20, 50, 100, and 200) and then
tested the resulting system with a total of 6000 CT images. All images were
acquired from the Massachusetts General Hospital (MGH) Picture Archiving and
Communication System (PACS). Using this data, we employ the learning curve
approach to predict classification accuracy at a given training sample size.
Our research will present a general methodology for determining the training
data set size necessary to achieve a certain target classification accuracy
that can be easily applied to other problems within such systems.
An Automaton Learning Approach to Solving Safety Games over Infinite Graphs
Authors: Daniel Neider, Ufuk Topcu
Date: 2016-01-07 00:00:00
Tasks: Motion Planning

We propose a method to construct finite-state reactive controllers for
systems whose interactions with their adversarial environment are modeled by
infinite-duration two-player games over (possibly) infinite graphs. The
proposed method targets safety games with infinitely many states or with such a
large number of states that it would be impractical---if not impossible---for
conventional synthesis techniques that work on the entire state space. We
resort to constructing finite-state controllers for such systems through an
automata learning approach, utilizing a symbolic representation of the
underlying game that is based on finite automata. Throughout the learning
process, the learner maintains an approximation of the winning region
(represented as a finite automaton) and refines it using different types of
counterexamples provided by the teacher until a satisfactory controller can be
derived (if one exists). We present a symbolic representation of safety games
(inspired by regular model checking), propose implementations of the learner
and teacher, and evaluate their performance on examples motivated by robotic
motion planning in dynamic environments.
Large Collection of Diverse Gene Set Search Queries Recapitulate Known Protein-Protein Interactions and Gene-Gene Functional Associations
Authors: Avi Ma'ayan, Neil R. Clark
Date: 2016-01-07 00:00:00

Popular online enrichment analysis tools from the field of molecular systems
biology provide users with the ability to submit their experimental results as
gene sets for individual analysis. Such queries are kept private, and have
never before been considered as a resource for integrative analysis. By
harnessing gene set query submissions from thousands of users, we aim to
discover biological knowledge beyond the scope of an individual study. In this
work, we investigated a large collection of gene sets submitted to the tool
Enrichr by thousands of users. Based on co-occurrence, we constructed a global
gene-gene association network. We interpret this inferred network as providing
a summary of the structure present in this crowdsourced gene set library, and
show that this network recapitulates known protein-protein interactions and
functional associations between genes. This finding implies that this network
also offers predictive value. Furthermore, we visualize this gene-gene
association network using a new edge-pruning algorithm that retains both the
local and global structures of large-scale networks. Our ability to make
predictions for currently unknown gene associations, that may not be captured
by individual researchers and data sources, is a demonstration of the potential
of harnessing collective knowledge from users of popular tools in the field of
molecular systems biology.
Policy Distillation
Authors: Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, Raia Hadsell
Date: 2015-11-19 00:00:00
Tasks: Deep Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Policies for complex visual tasks have been successfully learned with deep
reinforcement learning, using an approach called deep Q-networks (DQN), but
relatively large (task-specific) networks and extensive training are needed to
achieve good performance. In this work, we present a novel method called policy
distillation that can be used to extract the policy of a reinforcement learning
agent and train a new network that performs at the expert level while being
dramatically smaller and more efficient. Furthermore, the same method can be
used to consolidate multiple task-specific policies into a single policy. We
demonstrate these claims using the Atari domain and show that the multi-task
distilled agent outperforms the single-task teachers as well as a
jointly-trained DQN agent.
MazeBase: A Sandbox for Learning from Games
Authors: Sainbayar Sukhbaatar, Arthur Szlam, Gabriel Synnaeve, Soumith Chintala, Rob Fergus
Date: 2015-11-23 00:00:00
Tasks: Negation, Reinforcement Learning, Reinforcement Learning (RL), Starcraft

This paper introduces MazeBase: an environment for simple 2D games, designed
as a sandbox for machine learning approaches to reasoning and planning. Within
it, we create 10 simple games embodying a range of algorithmic tasks (e.g.
if-then statements or set negation). A variety of neural models (fully
connected, convolutional network, memory network) are deployed via
reinforcement learning on these games, with and without a procedurally
generated curriculum. Despite the tasks' simplicity, the performance of the
models is far from optimal, suggesting directions for future development. We
also demonstrate the versatility of MazeBase by using it to emulate small
combat scenarios from StarCraft. Models trained on the MazeBase version can be
directly applied to StarCraft, where they consistently beat the in-game AI.
Grid Long Short-Term Memory
Authors: Nal Kalchbrenner, Ivo Danihelka, Alex Graves
Date: 2015-07-06 00:00:00
Tasks: Language Modelling, Memorization, Translation

This paper introduces Grid Long Short-Term Memory, a network of LSTM cells
arranged in a multidimensional grid that can be applied to vectors, sequences
or higher dimensional data such as images. The network differs from existing
deep LSTM architectures in that the cells are connected between network layers
as well as along the spatiotemporal dimensions of the data. The network
provides a unified way of using LSTM for both deep and sequential computation.
We apply the model to algorithmic tasks such as 15-digit integer addition and
sequence memorization, where it is able to significantly outperform the
standard LSTM. We then give results for two empirical tasks. We find that 2D
Grid LSTM achieves 1.47 bits per character on the Wikipedia character
prediction benchmark, which is state-of-the-art among neural approaches. In
addition, we use the Grid LSTM to define a novel two-dimensional translation
model, the Reencoder, and show that it outperforms a phrase-based reference
system on a Chinese-to-English translation task.
Efficient and Parsimonious Agnostic Active Learning
Authors: Tzu-Kuo Huang, Alekh Agarwal, Daniel J. Hsu, John Langford, Robert E. Schapire
Date: 2015-06-29 00:00:00
Proceeding: NeurIPS 2015 12
Tasks: Active Learning, General Classification

We develop a new active learning algorithm for the streaming setting
satisfying three important properties: 1) It provably works for any classifier
representation and classification problem including those with severe noise. 2)
It is efficiently implementable with an ERM oracle. 3) It is more aggressive
than all previous approaches satisfying 1 and 2. To do this we create an
algorithm based on a newly defined optimization problem and analyze it. We also
conduct the first experimental analysis of all efficient agnostic active
learning algorithms, evaluating their strengths and weaknesses in different
settings.
Toward Organic Computing Approach for Cybernetic Responsive Environment
Authors: Duhart Clément, Bertelle Cyrille
Date: 2016-01-07 00:00:00

The developpment of the Internet of Things (IoT) concept revives Responsive
Environments (RE) technologies. Nowadays, the idea of a permanent connection
between physical and digital world is technologically possible. The capillar
Internet relates to the Internet extension into daily appliances such as they
become actors of Internet like any hu-man. The parallel development of
Machine-to-Machine communications and Arti cial Intelligence (AI) technics
start a new area of cybernetic. This paper presents an approach for Cybernetic
Organism (Cyborg) for RE based on Organic Computing (OC). In such approach,
each appli-ance is a part of an autonomic system in order to control a physical
environment. The underlying idea is that such systems must have self-x
properties in order to adapt their behavior to external disturbances with a
high-degree of autonomy.
NodIO, a JavaScript framework for volunteer-based evolutionary algorithms : first results
Authors: Juan-J. Merelo, Mario García-Valdez, Pedro A. Castillo, Pablo García-Sánchez, P. de las Cuevas, Nuria Rico
Date: 2016-01-07 00:00:00
Tasks: Distributed Computing, Evolutionary Algorithms

JavaScript is an interpreted language mainly known for its inclusion in web
browsers, making them a container for rich Internet based applications. This
has inspired its use, for a long time, as a tool for evolutionary algorithms,
mainly so in browser-based volunteer computing environments. Several libraries
have also been published so far and are in use. However, the last years have
seen a resurgence of interest in the language, becoming one of the most popular
and thus spawning the improvement of its implementations, which are now the
foundation of many new client-server applications. We present such an
application for running distributed volunteer-based evolutionary algorithm
experiments, and we make a series of measurements to establish the speed of
JavaScript in evolutionary algorithms that can serve as a baseline for
comparison with other distributed computing experiments. These experiments use
different integer and floating point problems, and prove that the speed of
JavaScript is actually competitive with other languages commonly used by the
evolutionary algorithm practitioner.
Deep Feature Learning for EEG Recordings
Authors: Sebastian Stober, Avital Sternin, Adrian M. Owen, Jessica A. Grahn
Date: 2015-11-13 00:00:00
Tasks: EEG, Electroencephalogram (EEG)

We introduce and compare several strategies for learning discriminative
features from electroencephalography (EEG) recordings using deep learning
techniques. EEG data are generally only available in small quantities, they are
high-dimensional with a poor signal-to-noise ratio, and there is considerable
variability between individual subjects and recording sessions. Our proposed
techniques specifically address these challenges for feature learning.
Cross-trial encoding forces auto-encoders to focus on features that are stable
across trials. Similarity-constraint encoders learn features that allow to
distinguish between classes by demanding that two trials from the same class
are more similar to each other than to trials from other classes. This
tuple-based training approach is especially suitable for small datasets.
Hydra-nets allow for separate processing pathways adapting to subsets of a
dataset and thus combine the advantages of individual feature learning (better
adaptation of early, low-level processing) with group model training (better
generalization of higher-level processing in deeper layers). This way, models
can, for instance, adapt to each subject individually to compensate for
differences in spatial patterns due to anatomical differences or variance in
electrode positions. The different techniques are evaluated using the publicly
available OpenMIIR dataset of EEG recordings taken while participants listened
to and imagined music.
Robust EM kernel-based methods for linear system identification
Authors: Giulio Bottegal, Aleksandr Y. Aravkin, Håkan Hjalmarsson, Gianluigi Pillonetto
Date: 2014-11-21 00:00:00

Recent developments in system identification have brought attention to
regularized kernel-based methods. This type of approach has been proven to
compare favorably with classic parametric methods. However, current
formulations are not robust with respect to outliers. In this paper, we
introduce a novel method to robustify kernel-based system identification
methods. To this end, we model the output measurement noise using random
variables with heavy-tailed probability density functions (pdfs), focusing on
the Laplacian and the Student's t distributions. Exploiting the representation
of these pdfs as scale mixtures of Gaussians, we cast our system identification
problem into a Gaussian process regression framework, which requires estimating
a number of hyperparameters of the data size order. To overcome this
difficulty, we design a new maximum a posteriori (MAP) estimator of the
hyperparameters, and solve the related optimization problem with a novel
iterative scheme based on the Expectation-Maximization (EM) method. In presence
of outliers, tests on simulated data and on a real system show a substantial
performance improvement compared to currently used kernel-based methods for
linear system identification.