pwc-archive/papers-with-abstracts huggingface dataset:
Using the Output Embedding to Improve Language Models
Authors: Ofir Press, Lior Wolf
Date: 2016-08-20 00:00:00
Proceeding: EACL 2017 4
Tasks: Translation, valid

We study the topmost weight matrix of neural network language models. We show
that this matrix constitutes a valid word embedding. When training language
models, we recommend tying the input embedding and this output embedding. We
analyze the resulting update rules and show that the tied embedding evolves in
a more similar way to the output embedding than to the input embedding in the
untied model. We also offer a new method of regularizing the output embedding.
Our methods lead to a significant reduction in perplexity, as we are able to
show on a variety of neural network language models. Finally, we show that
weight tying can reduce the size of neural translation models to less than half
of their original size without harming their performance.
Systèmes du LIA à DEFT'13
Authors: Xavier Bost, Ilaria Brunetti, Luis Adrián Cabrera-Diego, Jean-Valère Cossu, Andréa Linhares, Mohamed Morchid, Juan-Manuel Torres-Moreno, Marc El-Bèze, Richard Dufour
Date: 2017-02-21 00:00:00
Tasks: Document Classification, General Classification

The 2013 D\'efi de Fouille de Textes (DEFT) campaign is interested in two
types of language analysis tasks, the document classification and the
information extraction in the specialized domain of cuisine recipes. We present
the systems that the LIA has used in DEFT 2013. Our systems show interesting
results, even though the complexity of the proposed tasks.
Linearized GMM Kernels and Normalized Random Fourier Features
Authors: Ping Li
Date: 2016-05-18 00:00:00

The method of "random Fourier features (RFF)" has become a popular tool for
approximating the "radial basis function (RBF)" kernel. The variance of RFF is
actually large. Interestingly, the variance can be substantially reduced by a
simple normalization step as we theoretically demonstrate. We name the improved
scheme as the "normalized RFF (NRFF)".
  We also propose the "generalized min-max (GMM)" kernel as a measure of data
similarity. GMM is positive definite as there is an associated hashing method
named "generalized consistent weighted sampling (GCWS)" which linearizes this
nonlinear kernel. We provide an extensive empirical evaluation of the RBF
kernel and the GMM kernel on more than 50 publicly available datasets. For a
majority of the datasets, the (tuning-free) GMM kernel outperforms the
best-tuned RBF kernel.
  We conduct extensive experiments for comparing the linearized RBF kernel
using NRFF with the linearized GMM kernel using GCWS. We observe that, to reach
a comparable classification accuracy, GCWS typically requires substantially
fewer samples than NRFF, even on datasets where the original RBF kernel
outperforms the original GMM kernel. The empirical success of GCWS (compared to
NRFF) can also be explained from a theoretical perspective. Firstly, the
relative variance (normalized by the squared expectation) of GCWS is
substantially smaller than that of NRFF, except for the very high similarity
region (where the variances of both methods are close to zero). Secondly, if we
make a model assumption on the data, we can show analytically that GCWS
exhibits much smaller variance than NRFF for estimating the same object (e.g.,
the RBF kernel), except for the very high similarity region.
Efficient Social Network Multilingual Classification using Character, POS n-grams and Dynamic Normalization
Authors: Carlos-Emiliano González-Gallardo, Juan-Manuel Torres-Moreno, Azucena Montes Rendón, Gerardo Sierra
Date: 2017-02-21 00:00:00
Tasks: Author Profiling, General Classification, POS

In this paper we describe a dynamic normalization process applied to social
network multilingual documents (Facebook and Twitter) to improve the
performance of the Author profiling task for short texts. After the
normalization process, $n$-grams of characters and n-grams of POS tags are
obtained to extract all the possible stylistic information encoded in the
documents (emoticons, character flooding, capital letters, references to other
users, hyperlinks, hashtags, etc.). Experiments with SVM showed up to 90% of
performance.
Extended Object Tracking: Introduction, Overview and Applications
Authors: Karl Granstrom, Marcus Baum, Stephan Reuter
Date: 2016-03-14 00:00:00
Tasks: Object, Object Tracking

This article provides an elaborate overview of current research in extended
object tracking. We provide a clear definition of the extended object tracking
problem and discuss its delimitation to other types of object tracking. Next,
different aspects of extended object modelling are extensively discussed.
Subsequently, we give a tutorial introduction to two basic and well used
extended object tracking approaches - the random matrix approach and the Kalman
filter-based approach for star-convex shapes. The next part treats the tracking
of multiple extended objects and elaborates how the large number of feasible
association hypotheses can be tackled using both Random Finite Set (RFS) and
Non-RFS multi-object trackers. The article concludes with a summary of current
applications, where four example applications involving camera, X-band radar,
light detection and ranging (lidar), red-green-blue-depth (RGB-D) sensors are
highlighted.
Adaptive Computation Time for Recurrent Neural Networks
Authors: Alex Graves
Date: 2016-03-29 00:00:00
Tasks: Language Modelling

This paper introduces Adaptive Computation Time (ACT), an algorithm that
allows recurrent neural networks to learn how many computational steps to take
between receiving an input and emitting an output. ACT requires minimal changes
to the network architecture, is deterministic and differentiable, and does not
add any noise to the parameter gradients. Experimental results are provided for
four synthetic problems: determining the parity of binary vectors, applying
binary logic operations, adding integers, and sorting real numbers. Overall,
performance is dramatically improved by the use of ACT, which successfully
adapts the number of computational steps to the requirements of the problem. We
also present character-level language modelling results on the Hutter prize
Wikipedia dataset. In this case ACT does not yield large gains in performance;
however it does provide intriguing insight into the structure of the data, with
more computation allocated to harder-to-predict transitions, such as spaces
between words and ends of sentences. This suggests that ACT or other adaptive
computation methods could provide a generic method for inferring segment
boundaries in sequence data.
Crowd Sourcing Image Segmentation with iaSTAPLE
Authors: Dmitrij Schlesinger, Florian Jug, Gene Myers, Carsten Rother, Dagmar Kainmüller
Date: 2017-02-21 00:00:00
Tasks: Image Segmentation, Segmentation, Semantic Segmentation

We propose a novel label fusion technique as well as a crowdsourcing protocol
to efficiently obtain accurate epithelial cell segmentations from non-expert
crowd workers. Our label fusion technique simultaneously estimates the true
segmentation, the performance levels of individual crowd workers, and an image
segmentation model in the form of a pairwise Markov random field. We term our
approach image-aware STAPLE (iaSTAPLE) since our image segmentation model
seamlessly integrates into the well-known and widely used STAPLE approach. In
an evaluation on a light microscopy dataset containing more than 5000 membrane
labeled epithelial cells of a fly wing, we show that iaSTAPLE outperforms
STAPLE in terms of segmentation accuracy as well as in terms of the accuracy of
estimated crowd worker performance levels, and is able to correctly segment 99%
of all cells when compared to expert segmentations. These results show that
iaSTAPLE is a highly useful tool for crowd sourcing image segmentation.
Deep Multi-Species Embedding
Authors: Di Chen, Yexiang Xue, Shuo Chen, Daniel Fink, Carla Gomes
Date: 2016-09-28 00:00:00

Understanding how species are distributed across landscapes over time is a
fundamental question in biodiversity research. Unfortunately, most species
distribution models only target a single species at a time, despite strong
ecological evidence that species are not independently distributed. We propose
Deep Multi-Species Embedding (DMSE), which jointly embeds vectors corresponding
to multiple species as well as vectors representing environmental covariates
into a common high-dimensional feature space via a deep neural network. Applied
to bird observational data from the citizen science project \textit{eBird}, we
demonstrate how the DMSE model discovers inter-species relationships to
outperform single-species distribution models (random forests and SVMs) as well
as competing multi-label models. Additionally, we demonstrate the benefit of
using a deep neural network to extract features within the embedding and show
how they improve the predictive performance of species distribution modelling.
An important domain contribution of the DMSE model is the ability to discover
and describe species interactions while simultaneously learning the shared
habitat preferences among species. As an additional contribution, we provide a
graphical embedding of hundreds of bird species in the Northeast US.
Stochastic Composite Least-Squares Regression with convergence rate O(1/n)
Authors: Nicolas Flammarion, Francis Bach
Date: 2017-02-21 00:00:00
Tasks: regression

We consider the minimization of composite objective functions composed of the
expectation of quadratic functions and an arbitrary convex function. We study
the stochastic dual averaging algorithm with a constant step-size, showing that
it leads to a convergence rate of O(1/n) without strong convexity assumptions.
This thus extends earlier results on least-squares regression with the
Euclidean geometry to (a) all convex regularizers and constraints, and (b) all
geome-tries represented by a Bregman divergence. This is achieved by a new
proof technique that relates stochastic and deterministic recursions.
A Discriminative Event Based Model for Alzheimer's Disease Progression Modeling
Authors: Vikram Venkatraghavan, Esther Bron, Wiro Niessen, Stefan Klein
Date: 2017-02-21 00:00:00

The event-based model (EBM) for data-driven disease progression modeling
estimates the sequence in which biomarkers for a disease become abnormal. This
helps in understanding the dynamics of disease progression and facilitates
early diagnosis by staging patients on a disease progression timeline. Existing
EBM methods are all generative in nature. In this work we propose a novel
discriminative approach to EBM, which is shown to be more accurate as well as
computationally more efficient than existing state-of-the art EBM methods. The
method first estimates for each subject an approximate ordering of events, by
ranking the posterior probabilities of individual biomarkers being abnormal.
Subsequently, the central ordering over all subjects is estimated by fitting a
generalized Mallows model to these approximate subject-specific orderings based
on a novel probabilistic Kendall's Tau distance. To evaluate the accuracy, we
performed extensive experiments on synthetic data simulating the progression of
Alzheimer's disease. Subsequently, the method was applied to the Alzheimer's
Disease Neuroimaging Initiative (ADNI) data to estimate the central event
ordering in the dataset. The experiments benchmark the accuracy of the new
model under various conditions and compare it with existing state-of-the-art
EBM methods. The results indicate that discriminative EBM could be a simple and
elegant approach to disease progression modeling.
Delving Deeper into MOOC Student Dropout Prediction
Authors: Jacob Whitehill, Kiran Mohan, Daniel Seaton, Yigal Rosen, Dustin Tingley
Date: 2017-02-21 00:00:00
Tasks: Prediction, regression, Student dropout

In order to obtain reliable accuracy estimates for automatic MOOC dropout
predictors, it is important to train and test them in a manner consistent with
how they will be used in practice. Yet most prior research on MOOC dropout
prediction has measured test accuracy on the same course used for training the
classifier, which can lead to overly optimistic accuracy estimates. In order to
understand better how accuracy is affected by the training+testing regime, we
compared the accuracy of a standard dropout prediction architecture
(clickstream features + logistic regression) across 4 different training
paradigms. Results suggest that (1) training and testing on the same course
("post-hoc") can overestimate accuracy by several percentage points; (2)
dropout classifiers trained on proxy labels based on students' persistence are
surprisingly competitive with post-hoc training (87.33% versus 90.20% AUC
averaged over 8 weeks of 40 HarvardX MOOCs); and (3) classifier performance
does not vary significantly with the academic discipline. Finally, we also
research new dropout prediction architectures based on deep, fully-connected,
feed-forward neural networks and find that (4) networks with as many as 5
hidden layers can statistically significantly increase test accuracy over that
of logistic regression.
Mimicking Ensemble Learning with Deep Branched Networks
Authors: Byungju Kim, Youngsoo Kim, Yeakang Lee, Junmo Kim
Date: 2017-02-21 00:00:00
Tasks: Classification, Ensemble Learning, General Classification, image-classification, Image Classification

This paper proposes a branched residual network for image classification. It
is known that high-level features of deep neural network are more
representative than lower-level features. By sharing the low-level features,
the network can allocate more memory to high-level features. The upper layers
of our proposed network are branched, so that it mimics the ensemble learning.
By mimicking ensemble learning with single network, we have achieved better
performance on ImageNet classification task.
Amortised MAP Inference for Image Super-resolution
Authors: Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, Ferenc Huszár
Date: 2016-10-14 00:00:00
Tasks: Denoising, Image Super-Resolution, Super-Resolution, Variational Inference

Image super-resolution (SR) is an underdetermined inverse problem, where a
large number of plausible high-resolution images can explain the same
downsampled image. Most current single image SR methods use empirical risk
minimisation, often with a pixel-wise mean squared error (MSE) loss. However,
the outputs from such methods tend to be blurry, over-smoothed and generally
appear implausible. A more desirable approach would employ Maximum a Posteriori
(MAP) inference, preferring solutions that always have a high probability under
the image prior, and thus appear more plausible. Direct MAP estimation for SR
is non-trivial, as it requires us to build a model for the image prior from
samples. Furthermore, MAP inference is often performed via optimisation-based
iterative algorithms which don't compare well with the efficiency of
neural-network-based alternatives. Here we introduce new methods for amortised
MAP inference whereby we calculate the MAP estimate directly using a
convolutional neural network. We first introduce a novel neural network
architecture that performs a projection to the affine subspace of valid SR
solutions ensuring that the high resolution output of the network is always
consistent with the low resolution input. We show that, using this
architecture, the amortised MAP inference problem reduces to minimising the
cross-entropy between two distributions, similar to training generative models.
We propose three methods to solve this optimisation problem: (1) Generative
Adversarial Networks (GAN) (2) denoiser-guided SR which backpropagates
gradient-estimates from denoising to train the network, and (3) a baseline
method using a maximum-likelihood-trained image prior. Our experiments show
that the GAN based approach performs best on real image data. Lastly, we
establish a connection between GANs and amortised variational inference as in
e.g. variational autoencoders.
Interpreting Outliers: Localized Logistic Regression for Density Ratio Estimation
Authors: Makoto Yamada, Song Liu, Samuel Kaski
Date: 2017-02-21 00:00:00
Tasks: Density Ratio Estimation, Outlier Detection, regression, Variable Selection

We propose an inlier-based outlier detection method capable of both
identifying the outliers and explaining why they are outliers, by identifying
the outlier-specific features. Specifically, we employ an inlier-based outlier
detection criterion, which uses the ratio of inlier and test probability
densities as a measure of plausibility of being an outlier. For estimating the
density ratio function, we propose a localized logistic regression algorithm.
Thanks to the locality of the model, variable selection can be
outlier-specific, and will help interpret why points are outliers in a
high-dimensional space. Through synthetic experiments, we show that the
proposed algorithm can successfully detect the important features for outliers.
Moreover, we show that the proposed algorithm tends to outperform existing
algorithms in benchmark datasets.
Hybrid Dialog State Tracker with ASR Features
Authors: Miroslav Vodolán, Rudolf Kadlec, Jan Kleindienst
Date: 2017-02-21 00:00:00
Proceeding: EACL 2017 4
Tasks: dialog state tracking, slot-filling, Slot Filling, Spoken Language Understanding

This paper presents a hybrid dialog state tracker enhanced by trainable
Spoken Language Understanding (SLU) for slot-filling dialog systems. Our
architecture is inspired by previously proposed neural-network-based
belief-tracking systems. In addition, we extended some parts of our modular
architecture with differentiable rules to allow end-to-end training. We
hypothesize that these rules allow our tracker to generalize better than pure
machine-learning based systems. For evaluation, we used the Dialog State
Tracking Challenge (DSTC) 2 dataset - a popular belief tracking testbed with
dialogs from restaurant information system. To our knowledge, our hybrid
tracker sets a new state-of-the-art result in three out of four categories
within the DSTC2.
Towards a Common Implementation of Reinforcement Learning for Multiple Robotic Tasks
Authors: Angel Martínez-Tenor, Juan Antonio Fernández-Madrigal, Ana Cruz-Martín, Javier González-Jiménez
Date: 2017-02-21 00:00:00
Tasks: Decision Making, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Mobile robots are increasingly being employed for performing complex tasks in
dynamic environments. Reinforcement learning (RL) methods are recognized to be
promising for specifying such tasks in a relatively simple manner. However, the
strong dependency between the learning method and the task to learn is a
well-known problem that restricts practical implementations of RL in robotics,
often requiring major modifications of parameters and adding other techniques
for each particular task. In this paper we present a practical core
implementation of RL which enables the learning process for multiple robotic
tasks with minimal per-task tuning or none. Based on value iteration methods,
this implementation includes a novel approach for action selection, called
Q-biased softmax regression (QBIASSR), which avoids poor performance of the
learning process when the robot reaches new unexplored states. Our approach
takes advantage of the structure of the state space by attending the physical
variables involved (e.g., distances to obstacles, X,Y,{\theta} pose, etc.),
thus experienced sets of states may favor the decision-making process of
unexplored or rarely-explored states. This improvement has a relevant role in
reducing the tuning of the algorithm for particular tasks. Experiments with
real and simulated robots, performed with the software framework also
introduced here, show that our implementation is effectively able to learn
different robotic tasks without tuning the learning method. Results also
suggest that the combination of true online SARSA({\lambda}) with QBIASSR can
outperform the existing RL core algorithms in low-dimensional robotic tasks.
Is Saki #delicious? The Food Perception Gap on Instagram and Its Relation to Health
Authors: Ferda Ofli, Yusuf Aytar, Ingmar Weber, Raggi al Hammouri, Antonio Torralba
Date: 2017-02-21 00:00:00
Tasks: Relation

Food is an integral part of our life and what and how much we eat crucially
affects our health. Our food choices largely depend on how we perceive certain
characteristics of food, such as whether it is healthy, delicious or if it
qualifies as a salad. But these perceptions differ from person to person and
one person's "single lettuce leaf" might be another person's "side salad".
Studying how food is perceived in relation to what it actually is typically
involves a laboratory setup. Here we propose to use recent advances in image
recognition to tackle this problem. Concretely, we use data for 1.9 million
images from Instagram from the US to look at systematic differences in how a
machine would objectively label an image compared to how a human subjectively
does. We show that this difference, which we call the "perception gap", relates
to a number of health outcomes observed at the county level. To the best of our
knowledge, this is the first time that image recognition is being used to study
the "misalignment" of how people describe food images vs. what they actually
depict.
Generative Temporal Models with Memory
Authors: Mevlana Gemici, Chia-Chun Hung, Adam Santoro, Greg Wayne, Shakir Mohamed, Danilo J. Rezende, David Amos, Timothy Lillicrap
Date: 2017-02-15 00:00:00
Tasks: Variational Inference

We consider the general problem of modeling temporal data with long-range
dependencies, wherein new observations are fully or partially predictable based
on temporally-distant, past observations. A sufficiently powerful temporal
model should separate predictable elements of the sequence from unpredictable
elements, express uncertainty about those unpredictable elements, and rapidly
identify novel elements that may help to predict the future. To create such
models, we introduce Generative Temporal Models augmented with external memory
systems. They are developed within the variational inference framework, which
provides both a practical training methodology and methods to gain insight into
the models' operation. We show, on a range of problems with sparse, long-term
temporal dependencies, that these models store information from early in a
sequence, and reuse this stored information efficiently. This allows them to
perform substantially better than existing models based on well-known recurrent
neural networks, like LSTMs.
Joint Graph Decomposition and Node Labeling: Problem, Algorithms, Applications
Authors: Evgeny Levinkov, Jonas Uhrig, Siyu Tang, Mohamed Omran, Eldar Insafutdinov, Alexander Kirillov, Carsten Rother, Thomas Brox, Bernt Schiele, Bjoern Andres
Date: 2016-11-14 00:00:00
Tasks: Combinatorial Optimization, Multiple Object Tracking, Object Tracking, Pose Estimation, Semantic Segmentation

We state a combinatorial optimization problem whose feasible solutions define
both a decomposition and a node labeling of a given graph. This problem offers
a common mathematical abstraction of seemingly unrelated computer vision tasks,
including instance-separating semantic segmentation, articulated human body
pose estimation and multiple object tracking. Conceptually, the problem we
state generalizes the unconstrained integer quadratic program and the minimum
cost lifted multicut problem, both of which are NP-hard. In order to find
feasible solutions efficiently, we define two local search algorithms that
converge monotonously to a local optimum, offering a feasible solution at any
time. To demonstrate their effectiveness in tackling computer vision tasks, we
apply these algorithms to instances of the problem that we construct from
published data, using published algorithms. We report state-of-the-art
application-specific accuracy for the three above-mentioned applications.