pwc-archive/papers-with-abstracts huggingface dataset:
Efficient softmax approximation for GPUs
Authors: Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou
Date: 2016-09-14 00:00:00
Proceeding: ICML 2017 8

We propose an approximate strategy to efficiently train neural network based
language models over very large vocabularies. Our approach, called adaptive
softmax, circumvents the linear dependency on the vocabulary size by exploiting
the unbalanced word distribution to form clusters that explicitly minimize the
expectation of computation time. Our approach further reduces the computational
time by exploiting the specificities of modern architectures and matrix-matrix
vector operations, making it particularly suited for graphical processing
units. Our experiments carried out on standard benchmarks, such as EuroParl and
One Billion Word, show that our approach brings a large gain in efficiency over
standard approximations while achieving an accuracy close to that of the full
softmax. The code of our method is available at
https://github.com/facebookresearch/adaptive-softmax.
Evaluating 35 Methods to Generate Structural Connectomes Using Pairwise Classification
Authors: Dmitry Petrov, Alexander Ivanov, Joshua Faskowitz, Boris Gutman, Daniel Moyer, Julio Villalon, Neda Jahanshad, Paul Thompson
Date: 2017-06-19 00:00:00
Tasks: Classification, Diffusion  MRI, General Classification

There is no consensus on how to construct structural brain networks from
diffusion MRI. How variations in pre-processing steps affect network
reliability and its ability to distinguish subjects remains opaque. In this
work, we address this issue by comparing 35 structural connectome-building
pipelines. We vary diffusion reconstruction models, tractography algorithms and
parcellations. Next, we classify structural connectome pairs as either
belonging to the same individual or not. Connectome weights and eight
topological derivative measures form our feature set. For experiments, we use
three test-retest datasets from the Consortium for Reliability and
Reproducibility (CoRR) comprised of a total of 105 individuals. We also compare
pairwise classification results to a commonly used parametric test-retest
measure, Intraclass Correlation Coefficient (ICC).
The $\mathcal{E}$-Average Common Submatrix: Approximate Searching in a Restricted Neighborhood
Authors: Alessia Amelio, Darko Brodić
Date: 2017-06-19 00:00:00
Tasks: Information Retrieval, Retrieval

This paper introduces a new (dis)similarity measure for 2D arrays, extending
the Average Common Submatrix measure. This is accomplished by: (i) considering
the frequency of matching patterns, (ii) restricting the pattern matching to a
fixed-size neighborhood, and (iii) computing a distance-based approximate
matching. This will achieve better performances with low execution time and
larger information retrieval.
Optimising the topological information of the $A_\infty$-persistence groups
Authors: Francisco Belchí
Date: 2017-06-19 00:00:00

Persistent homology typically studies the evolution of homology groups
$H_p(X)$ (with coefficients in a field) along a filtration of topological
spaces. $A_\infty$-persistence extends this theory by analysing the evolution
of subspaces such as $V := \text{Ker}\, {\Delta_n}_{| H_p(X)} \subseteq
H_p(X)$, where $\{\Delta_m\}_{m\geq1}$ denotes a structure of
$A_\infty$-coalgebra on $H_*(X)$. In this paper we illustrate how
$A_\infty$-persistence can be useful beyond persistent homology by discussing
the topological meaning of $V$, which is the most basic form of
$A_\infty$-persistence group. In addition, we explore how to choose
$A_\infty$-coalgebras along a filtration to make the $A_\infty$-persistence
groups carry more faithful information.
Classifying Options for Deep Reinforcement Learning
Authors: Kai Arulkumaran, Nat Dilokthanakul, Murray Shanahan, Anil Anthony Bharath
Date: 2016-04-27 00:00:00
Tasks: Deep Reinforcement Learning, Hierarchical Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

In this paper we combine one method for hierarchical reinforcement learning -
the options framework - with deep Q-networks (DQNs) through the use of
different "option heads" on the policy network, and a supervisory network for
choosing between the different options. We utilise our setup to investigate the
effects of architectural constraints in subtasks with positive and negative
transfer, across a range of network capacities. We empirically show that our
augmented DQN has lower sample complexity when simultaneously learning subtasks
with negative transfer, without degrading performance when learning subtasks
with positive transfer.
The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
Authors: Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, Ce Zhang
Date: 2016-11-16 00:00:00
Tasks: Quantization

Recently there has been significant interest in training machine-learning
models at low precision: by reducing precision, one can reduce computation and
communication by one order of magnitude. We examine training at reduced
precision, both from a theoretical and practical perspective, and ask: is it
possible to train models at end-to-end low precision with provable guarantees?
Can this lead to consistent order-of-magnitude speedups? We present a framework
called ZipML to answer these questions. For linear models, the answer is yes.
We develop a simple framework based on one simple but novel strategy called
double sampling. Our framework is able to execute training at low precision
with no bias, guaranteeing convergence, whereas naive quantization would
introduce significant bias. We validate our framework across a range of
applications, and show that it enables an FPGA prototype that is up to 6.5x
faster than an implementation using full 32-bit precision. We further develop a
variance-optimal stochastic quantization strategy and show that it can make a
significant difference in a variety of settings. When applied to linear models
together with double sampling, we save up to another 1.7x in data movement
compared with uniform quantization. When training deep networks with quantized
models, we achieve higher accuracy than the state-of-the-art XNOR-Net. Finally,
we extend our framework through approximation to non-linear models, such as
SVM. We show that, although using low-precision data induces bias, we can
appropriately bound and control the bias. We find in practice 8-bit precision
is often sufficient to converge to the correct solution. Interestingly,
however, in practice we notice that our framework does not always outperform
the naive rounding approach. We discuss this negative result in detail.
Deep Counterfactual Networks with Propensity-Dropout
Authors: Ahmed M. Alaa, Michael Weisz, Mihaela van der Schaar
Date: 2017-06-19 00:00:00
Tasks: Causal Inference, counterfactual, Selection bias

We propose a novel approach for inferring the individualized causal effects
of a treatment (intervention) from observational data. Our approach
conceptualizes causal inference as a multitask learning problem; we model a
subject's potential outcomes using a deep multitask network with a set of
shared layers among the factual and counterfactual outcomes, and a set of
outcome-specific layers. The impact of selection bias in the observational data
is alleviated via a propensity-dropout regularization scheme, in which the
network is thinned for every training example via a dropout probability that
depends on the associated propensity score. The network is trained in
alternating phases, where in each phase we use the training examples of one of
the two potential outcomes (treated and control populations) to update the
weights of the shared layers and the respective outcome-specific layers.
Experiments conducted on data based on a real-world observational study show
that our algorithm outperforms the state-of-the-art.
Bayesian Joint Modelling for Object Localisation in Weakly Labelled Images
Authors: Zhiyuan Shi, Timothy M. Hospedales, Tao Xiang
Date: 2017-06-19 00:00:00
Tasks: Domain Adaptation, Object, Transfer Learning

We address the problem of localisation of objects as bounding boxes in images
and videos with weak labels. This weakly supervised object localisation problem
has been tackled in the past using discriminative models where each object
class is localised independently from other classes. In this paper, a novel
framework based on Bayesian joint topic modelling is proposed, which differs
significantly from the existing ones in that: (1) All foreground object classes
are modelled jointly in a single generative model that encodes multiple object
co-existence so that "explaining away" inference can resolve ambiguity and lead
to better learning and localisation. (2) Image backgrounds are shared across
classes to better learn varying surroundings and "push out" objects of
interest. (3) Our model can be learned with a mixture of weakly labelled and
unlabelled data, allowing the large volume of unlabelled images on the Internet
to be exploited for learning. Moreover, the Bayesian formulation enables the
exploitation of various types of prior knowledge to compensate for the limited
supervision offered by weakly labelled data, as well as Bayesian domain
adaptation for transfer learning. Extensive experiments on the PASCAL VOC,
ImageNet and YouTube-Object videos datasets demonstrate the effectiveness of
our Bayesian joint model for weakly supervised object localisation.
Recognizing Multi-talker Speech with Permutation Invariant Training
Authors: Dong Yu, Xuankai Chang, Yanmin Qian
Date: 2017-03-22 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition

In this paper, we propose a novel technique for direct recognition of
multiple speech streams given the single channel of mixed speech, without first
separating them. Our technique is based on permutation invariant training (PIT)
for automatic speech recognition (ASR). In PIT-ASR, we compute the average
cross entropy (CE) over all frames in the whole utterance for each possible
output-target assignment, pick the one with the minimum CE, and optimize for
that assignment. PIT-ASR forces all the frames of the same speaker to be
aligned with the same output layer. This strategy elegantly solves the label
permutation problem and speaker tracing problem in one shot. Our experiments on
artificially mixed AMI data showed that the proposed approach is very
promising.
Deep learning with spatiotemporal consistency for nerve segmentation in ultrasound images
Authors: Adel Hafiane, Pierre Vieyres, Alain Delbos
Date: 2017-06-19 00:00:00

Ultrasound-Guided Regional Anesthesia (UGRA) has been gaining importance in
the last few years, offering numerous advantages over alternative methods of
nerve localization (neurostimulation or paraesthesia). However, nerve detection
is one of the most tasks that anaesthetists can encounter in the UGRA
procedure. Computer aided system that can detect automatically region of nerve,
would help practitioner to concentrate more in anaesthetic delivery. In this
paper we propose a new method based on deep learning combined with
spatiotemporal information to robustly segment the nerve region. The proposed
method is based on two phases, localisation and segmentation. The first phase,
consists in using convolutional neural network combined with spatial and
temporal consistency to detect the nerve zone. The second phase utilises active
contour model to delineate the region of interest. Obtained results show the
validity of the proposed approach and its robustness.
Histograms of Gaussian normal distribution for feature matching in clutter scenes
Authors: Wei Zhou, Caiwen Ma, Arjan Kuijper
Date: 2017-06-19 00:00:00

3D feature descriptor provide information between corresponding models and
scenes. 3D objection recognition in cluttered scenes, however, remains a
largely unsolved problem. Practical applications impose several challenges
which are not fully addressed by existing methods. Especially in cluttered
scenes there are many feature mismatches between scenes and models. We
therefore propose Histograms of Gaussian Normal Distribution (HGND) for
extracting salient features on a local reference frame (LRF) that enables us to
solve this problem. We propose a LRF on each local surface patches using the
scatter matrix's eigenvectors. Then the HGND information of each salient point
is calculated on the LRF, for which we use both the mesh and point data of the
depth image. Experiments on 45 cluttered scenes of the Bologna Dataset and 50
cluttered scenes of the UWA Dataset are made to evaluate the robustness and
descriptiveness of our HGND. Experiments carried out by us demonstrate that
HGND obtains a more reliable matching rate than state-of-the-art approaches in
cluttered situations.
Rapid Probabilistic Interest Learning from Domain-Specific Pairwise Image Comparisons
Authors: Michael Burke, Siyabonga Mbonambi, Purity Molala, Raesetje Sefala
Date: 2017-06-19 00:00:00
Tasks: Information Retrieval, Retrieval

A great deal of work aims to discover large general purpose models of image interest or memorability for visual search and information retrieval. This paper argues that image interest is often domain and user specific, and that efficient mechanisms for learning about this domain-specific image interest as quickly as possible, while limiting the amount of data-labelling required, are often more useful to end-users. This work uses pairwise image comparisons to reduce the labelling burden on these users, and introduces an image interest estimation approach that performs similarly to recent data hungry deep learning approaches trained using pairwise ranking losses. Here, we use a Gaussian process model to interpolate image interest inferred using a Bayesian ranking approach over image features extracted using a pre-trained convolutional neural network. Results show that fitting a Gaussian process in high-dimensional image feature space is not only computationally feasible, but also effective across a broad range of domains. The proposed probabilistic interest estimation approach produces image interests paired with uncertainties that can be used to identify images for which additional labelling is required and measure inference convergence, allowing for sample efficient active model training. Importantly, the probabilistic formulation allows for effective visual search and information retrieval when limited labelling data is available.
Weakly supervised learning of actions from transcripts
Authors: Hilde Kuehne, Alexander Richard, Juergen Gall
Date: 2016-10-07 00:00:00
Tasks: Weakly-supervised Learning

We present an approach for weakly supervised learning of human actions from
video transcriptions. Our system is based on the idea that, given a sequence of
input data and a transcript, i.e. a list of the order the actions occur in the
video, it is possible to infer the actions within the video stream, and thus,
learn the related action models without the need for any frame-based
annotation. Starting from the transcript information at hand, we split the
given data sequences uniformly based on the number of expected actions. We then
learn action models for each class by maximizing the probability that the
training video sequences are generated by the action models given the sequence
order as defined by the transcripts. The learned model can be used to
temporally segment an unseen video with or without transcript. We evaluate our
approach on four distinct activity datasets, namely Hollywood Extended, MPII
Cooking, Breakfast and CRIM13. We show that our system is able to align the
scripted actions with the video data and that the learned models localize and
classify actions competitively in comparison to models trained with full
supervision, i.e. with frame level annotations, and that they outperform any
current state-of-the-art approach for aligning transcripts with video data.
Efficiently applying attention to sequential data with the Recurrent Discounted Attention unit
Authors: Brendan Maginnis, Pierre H. Richemond
Date: 2017-05-23 00:00:00
Proceeding: ICLR 2018 1

Recurrent Neural Networks architectures excel at processing sequences by
modelling dependencies over different timescales. The recently introduced
Recurrent Weighted Average (RWA) unit captures long term dependencies far
better than an LSTM on several challenging tasks. The RWA achieves this by
applying attention to each input and computing a weighted average over the full
history of its computations. Unfortunately, the RWA cannot change the attention
it has assigned to previous timesteps, and so struggles with carrying out
consecutive tasks or tasks with changing requirements. We present the Recurrent
Discounted Attention (RDA) unit that builds on the RWA by additionally allowing
the discounting of the past.
  We empirically compare our model to RWA, LSTM and GRU units on several
challenging tasks. On tasks with a single output the RWA, RDA and GRU units
learn much quicker than the LSTM and with better performance. On the multiple
sequence copy task our RDA unit learns the task three times as quickly as the
LSTM or GRU units while the RWA fails to learn at all. On the Wikipedia
character prediction task the LSTM performs best but it followed closely by our
RDA unit. Overall our RDA unit performs well and is sample efficient on a large
variety of sequence tasks.
An a Priori Exponential Tail Bound for k-Folds Cross-Validation
Authors: Karim Abou-Moustafa, Csaba Szepesvari
Date: 2017-06-19 00:00:00
Tasks: Generalization Bounds, valid

We consider a priori generalization bounds developed in terms of
cross-validation estimates and the stability of learners. In particular, we
first derive an exponential Efron-Stein type tail inequality for the
concentration of a general function of n independent random variables. Next,
under some reasonable notion of stability, we use this exponential tail bound
to analyze the concentration of the k-fold cross-validation (KFCV) estimate
around the true risk of a hypothesis generated by a general learning rule.
While the accumulated literature has often attributed this concentration to the
bias and variance of the estimator, our bound attributes this concentration to
the stability of the learning rule and the number of folds k. This insight
raises valid concerns related to the practical use of KFCV and suggests
research directions to obtain reliable empirical estimates of the actual risk.
Structured Best Arm Identification with Fixed Confidence
Authors: Ruitong Huang, Mohammad M. Ajallooeian, Csaba Szepesvári, Martin Müller
Date: 2017-06-16 00:00:00

We study the problem of identifying the best action among a set of possible
options when the value of each action is given by a mapping from a number of
noisy micro-observables in the so-called fixed confidence setting. Our main
motivation is the application to the minimax game search, which has been a
major topic of interest in artificial intelligence. In this paper we introduce
an abstract setting to clearly describe the essential properties of the
problem. While previous work only considered a two-move game tree search
problem, our abstract setting can be applied to the general minimax games where
the depth can be non-uniform and arbitrary, and transpositions are allowed. We
introduce a new algorithm (LUCB-micro) for the abstract setting, and give its
lower and upper sample complexity results. Our bounds recover some previous
results, which were only available in more limited settings, while they also
shed further light on how the structure of minimax problems influence sample
complexity.
An Entropy-based Pruning Method for CNN Compression
Authors: Jian-Hao Luo, Jianxin Wu
Date: 2017-06-19 00:00:00

This paper aims to simultaneously accelerate and compress off-the-shelf CNN
models via filter pruning strategy. The importance of each filter is evaluated
by the proposed entropy-based method first. Then several unimportant filters
are discarded to get a smaller CNN model. Finally, fine-tuning is adopted to
recover its generalization ability which is damaged during filter pruning. Our
method can reduce the size of intermediate activations, which would dominate
most memory footprint during model training stage but is less concerned in
previous compression methods. Experiments on the ILSVRC-12 benchmark
demonstrate the effectiveness of our method. Compared with previous filter
importance evaluation criteria, our entropy-based method obtains better
performance. We achieve 3.3x speed-up and 16.64x compression on VGG-16, 1.54x
acceleration and 1.47x compression on ResNet-50, both with about 1% top-5
accuracy decrease.
ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras
Authors: Raul Mur-Artal, Juan D. Tardos
Date: 2016-10-20 00:00:00
Tasks: Simultaneous Localization and Mapping, Visual Odometry

We present ORB-SLAM2 a complete SLAM system for monocular, stereo and RGB-D
cameras, including map reuse, loop closing and relocalization capabilities. The
system works in real-time on standard CPUs in a wide variety of environments
from small hand-held indoors sequences, to drones flying in industrial
environments and cars driving around a city. Our back-end based on bundle
adjustment with monocular and stereo observations allows for accurate
trajectory estimation with metric scale. Our system includes a lightweight
localization mode that leverages visual odometry tracks for unmapped regions
and matches to map points that allow for zero-drift localization. The
evaluation on 29 popular public sequences shows that our method achieves
state-of-the-art accuracy, being in most cases the most accurate SLAM solution.
We publish the source code, not only for the benefit of the SLAM community, but
with the aim of being an out-of-the-box SLAM solution for researchers in other
fields.
Kapre: On-GPU Audio Preprocessing Layers for a Quick Implementation of Deep Neural Network Models with Keras
Authors: Keunwoo Choi, Deokjin Joo, Ju-ho Kim
Date: 2017-06-19 00:00:00
Tasks: Data Augmentation, GPU

We introduce Kapre, Keras layers for audio and music signal preprocessing.
Music research using deep neural networks requires a heavy and tedious
preprocessing stage, for which audio processing parameters are often ignored in
parameter optimisation. To solve this problem, Kapre implements time-frequency
conversions, normalisation, and data augmentation as Keras layers. We report
simple benchmark results, showing real-time on-GPU preprocessing adds a
reasonable amount of computation.
An Empirical Study of Mini-Batch Creation Strategies for Neural Machine Translation
Authors: Makoto Morishita, Yusuke Oda, Graham Neubig, Koichiro Yoshino, Katsuhito Sudoh, Satoshi Nakamura
Date: 2017-06-19 00:00:00
Proceeding: WS 2017 8
Tasks: Machine Translation, NMT, Sentence, Translation

Training of neural machine translation (NMT) models usually uses mini-batches
for efficiency purposes. During the mini-batched training process, it is
necessary to pad shorter sentences in a mini-batch to be equal in length to the
longest sentence therein for efficient computation. Previous work has noted
that sorting the corpus based on the sentence length before making mini-batches
reduces the amount of padding and increases the processing speed. However,
despite the fact that mini-batch creation is an essential step in NMT training,
widely used NMT toolkits implement disparate strategies for doing so, which
have not been empirically validated or compared. This work investigates
mini-batch creation strategies with experiments over two different datasets.
Our results suggest that the choice of a mini-batch creation strategy has a
large effect on NMT training and some length-based sorting strategies do not
always work well compared with simple shuffling.