pwc-archive/papers-with-abstracts huggingface dataset:
Spikes as regularizers
Authors: Anders Søgaard
Date: 2016-11-18 00:00:00

We present a confidence-based single-layer feed-forward learning algorithm
SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of
activation spikes. We adaptively update a weight vector relying on confidence
estimates and activation offsets relative to previous activity. We regularize
updates proportionally to item-level confidence and weight-specific support,
loosely inspired by the observation from neurophysiology that high spike rates
are sometimes accompanied by low temporal precision. Our experiments suggest
that the new learning algorithm SPIRAL is more robust and less prone to
overfitting than both the averaged perceptron and AROW.
ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning
Authors: Hui Miao, Ang Li, Larry S. Davis, Amol Deshpande
Date: 2016-11-18 00:00:00
Tasks: Deep Learning, Management

Deep learning has improved state-of-the-art results in many important fields,
and has been the subject of much research in recent years, leading to the
development of several systems for facilitating deep learning. Current systems,
however, mainly focus on model building and training phases, while the issues
of data management, model sharing, and lifecycle management are largely
ignored. Deep learning modeling lifecycle generates a rich set of data
artifacts, such as learned parameters and training logs, and comprises of
several frequently conducted tasks, e.g., to understand the model behaviors and
to try out new models. Dealing with such artifacts and tasks is cumbersome and
largely left to the users. This paper describes our vision and implementation
of a data and lifecycle management system for deep learning. First, we
generalize model exploration and model enumeration queries from commonly
conducted tasks by deep learning modelers, and propose a high-level domain
specific language (DSL), inspired by SQL, to raise the abstraction level and
accelerate the modeling process. To manage the data artifacts, especially the
large amount of checkpointed float parameters, we design a novel model
versioning system (dlv), and a read-optimized parameter archival storage system
(PAS) that minimizes storage footprint and accelerates query workloads without
losing accuracy. PAS archives versioned models using deltas in a
multi-resolution fashion by separately storing the less significant bits, and
features a novel progressive query (inference) evaluation algorithm. Third, we
show that archiving versioned models using deltas poses a new dataset
versioning problem and we develop efficient algorithms for solving it. We
conduct extensive experiments over several real datasets from computer vision
domain to show the efficiency of the proposed techniques.
Foundations of Structural Causal Models with Cycles and Latent Variables
Authors: Stephan Bongers, Patrick Forré, Jonas Peters, Joris M. Mooij
Date: 2016-11-18 00:00:00
Tasks: counterfactual

Structural causal models (SCMs), also known as (nonparametric) structural equation models (SEMs), are widely used for causal modeling purposes. In particular, acyclic SCMs, also known as recursive SEMs, form a well-studied subclass of SCMs that generalize causal Bayesian networks to allow for latent confounders. In this paper, we investigate SCMs in a more general setting, allowing for the presence of both latent confounders and cycles. We show that in the presence of cycles, many of the convenient properties of acyclic SCMs do not hold in general: they do not always have a solution; they do not always induce unique observational, interventional and counterfactual distributions; a marginalization does not always exist, and if it exists the marginal model does not always respect the latent projection; they do not always satisfy a Markov property; and their graphs are not always consistent with their causal semantics. We prove that for SCMs in general each of these properties does hold under certain solvability conditions. Our work generalizes results for SCMs with cycles that were only known for certain special cases so far. We introduce the class of simple SCMs that extends the class of acyclic SCMs to the cyclic setting, while preserving many of the convenient properties of acyclic SCMs. With this paper we aim to provide the foundations for a general theory of statistical causal modeling with SCMs.
A Reinforcement Learning Approach to the View Planning Problem
Authors: Mustafa Devrim Kaba, Mustafa Gokhan Uzunbas, Ser Nam Lim
Date: 2016-10-19 00:00:00
Proceeding: CVPR 2017 7
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We present a Reinforcement Learning (RL) solution to the view planning
problem (VPP), which generates a sequence of view points that are capable of
sensing all accessible area of a given object represented as a 3D model. In
doing so, the goal is to minimize the number of view points, making the VPP a
class of set covering optimization problem (SCOP). The SCOP is NP-hard, and the
inapproximability results tell us that the greedy algorithm provides the best
approximation that runs in polynomial time. In order to find a solution that is
better than the greedy algorithm, (i) we introduce a novel score function by
exploiting the geometry of the 3D model, (ii) we model an intuitive human
approach to VPP using this score function, and (iii) we cast VPP as a Markovian
Decision Process (MDP), and solve the MDP in RL framework using well-known RL
algorithms. In particular, we use SARSA, Watkins-Q and TD with function
approximation to solve the MDP. We compare the results of our method with the
baseline greedy algorithm in an extensive set of test objects, and show that we
can out-perform the baseline in almost all cases.
Generative Deep Neural Networks for Dialogue: A Short Review
Authors: Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, Joelle Pineau
Date: 2016-11-18 00:00:00
Tasks: Decoder, Response Generation

Researchers have recently started investigating deep neural networks for
dialogue applications. In particular, generative sequence-to-sequence (Seq2Seq)
models have shown promising results for unstructured tasks, such as word-level
dialogue response generation. The hope is that such models will be able to
leverage massive amounts of data to learn meaningful natural language
representations and response generation strategies, while requiring a minimum
amount of domain knowledge and hand-crafting. An important challenge is to
develop models that can effectively incorporate dialogue context and generate
meaningful and diverse responses. In support of this goal, we review recently
proposed models based on generative encoder-decoder neural network
architectures, and show that these models have better ability to incorporate
long-term dialogue history, to model uncertainty and ambiguity in dialogue, and
to generate responses with high-level compositional structure.
NoiseOut: A Simple Way to Prune Neural Networks
Authors: Mohammad Babaeizadeh, Paris Smaragdis, Roy H. Campbell
Date: 2016-11-18 00:00:00

Neural networks are usually over-parameterized with significant redundancy in
the number of required neurons which results in unnecessary computation and
memory usage at inference time. One common approach to address this issue is to
prune these big networks by removing extra neurons and parameters while
maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated
pruning algorithm based on the correlation between activations of neurons in
the hidden layers. We prove that adding additional output neurons with entirely
random targets results into a higher correlation between neurons which makes
pruning by NoiseOut even more efficient. Finally, we test our method on various
networks and datasets. These experiments exhibit high pruning rates while
maintaining the accuracy of the original network.
Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
Authors: Volkan Cirik, Eduard Hovy, Louis-Philippe Morency
Date: 2016-11-18 00:00:00
Tasks: Sentiment Analysis

Curriculum Learning emphasizes the order of training instances in a
computational learning setup. The core hypothesis is that simpler instances
should be learned early as building blocks to learn more complex ones. Despite
its usefulness, it is still unknown how exactly the internal representation of
models are affected by curriculum learning. In this paper, we study the effect
of curriculum learning on Long Short-Term Memory (LSTM) networks, which have
shown strong competency in many Natural Language Processing (NLP) problems. Our
experiments on sentiment analysis task and a synthetic task similar to sequence
prediction tasks in NLP show that curriculum learning has a positive effect on
the LSTM's internal states by biasing the model towards building constructive
representations i.e. the internal representation at the previous timesteps are
used as building blocks for the final prediction. We also find that smaller
models significantly improves when they are trained with curriculum learning.
Lastly, we show that curriculum learning helps more when the amount of training
data is limited.
Ear Recognition: More Than a Survey
Authors: Žiga Emeršič, Vitomir Štruc, Peter Peer
Date: 2016-11-18 00:00:00
Tasks: Survey

Automatic identity recognition from ear images represents an active field of
research within the biometric community. The ability to capture ear images from
a distance and in a covert manner makes the technology an appealing choice for
surveillance and security applications as well as other application domains.
Significant contributions have been made in the field over recent years, but
open research problems still remain and hinder a wider (commercial) deployment
of the technology. This paper presents an overview of the field of automatic
ear recognition (from 2D images) and focuses specifically on the most recent,
descriptor-based methods proposed in this area. Open challenges are discussed
and potential research directions are outlined with the goal of providing the
reader with a point of reference for issues worth examining in the future. In
addition to a comprehensive review on ear recognition technology, the paper
also introduces a new, fully unconstrained dataset of ear images gathered from
the web and a toolbox implementing several state-of-the-art techniques for ear
recognition. The dataset and toolbox are meant to address some of the open
issues in the field and are made publicly available to the research community.
An Empirical Study of Continuous Connectivity Degree Sequence Equivalents
Authors: Daniel Moyer, Boris A. Gutman, Joshua Faskowitz, Neda Jahanshad, Paul M. Thompson
Date: 2016-11-18 00:00:00
Tasks: Point Processes

In the present work we demonstrate the use of a parcellation free
connectivity model based on Poisson point processes. This model produces for
each subject a continuous bivariate intensity function that represents for
every possible pair of points the relative rate at which we observe tracts
terminating at those points. We fit this model to explore degree sequence
equivalents for spatial continuum graphs, and to investigate the local
differences between estimated intensity functions for two different
tractography methods. This is a companion paper to Moyer et al. (2016), where
the model was originally defined.
Learning Interpretability for Visualizations using Adapted Cox Models through a User Experiment
Authors: Adrien Bibal, Benoit Frénay
Date: 2016-11-18 00:00:00

In order to be useful, visualizations need to be interpretable. This paper
uses a user-based approach to combine and assess quality measures in order to
better model user preferences. Results show that cluster separability measures
are outperformed by a neighborhood conservation measure, even though the former
are usually considered as intuitively representative of user motives. Moreover,
combining measures, as opposed to using a single measure, further improves
prediction performances.
Stratified Knowledge Bases as Interpretable Probabilistic Models (Extended Abstract)
Authors: Ondrej Kuzelka, Jesse Davis, Steven Schockaert
Date: 2016-11-18 00:00:00

In this paper, we advocate the use of stratified logical theories for
representing probabilistic models. We argue that such encodings can be more
interpretable than those obtained in existing frameworks such as Markov logic
networks. Among others, this allows for the use of domain experts to improve
learned models by directly removing, adding, or modifying logical formulas.
End-to-End Subtitle Detection and Recognition for Videos in East Asian Languages via CNN Ensemble with Near-Human-Level Performance
Authors: Yan Xu, Siyuan Shan, Ziming Qiu, Zhipeng Jia, Zhengyang Shen, Yipei Wang, Mengfei Shi, Eric I-Chao Chang
Date: 2016-11-18 00:00:00

In this paper, we propose an innovative end-to-end subtitle detection and
recognition system for videos in East Asian languages. Our end-to-end system
consists of multiple stages. Subtitles are firstly detected by a novel image
operator based on the sequence information of consecutive video frames. Then,
an ensemble of Convolutional Neural Networks (CNNs) trained on synthetic data
is adopted for detecting and recognizing East Asian characters. Finally, a
dynamic programming approach leveraging language models is applied to
constitute results of the entire body of text lines. The proposed system
achieves average end-to-end accuracies of 98.2% and 98.3% on 40 videos in
Simplified Chinese and 40 videos in Traditional Chinese respectively, which is
a significant outperformance of other existing methods. The near-perfect
accuracy of our system dramatically narrows the gap between human cognitive
ability and state-of-the-art algorithms used for such a task.
Communication Efficient Distributed Agnostic Boosting
Authors: Shang-Tse Chen, Maria-Florina Balcan, Duen Horng Chau
Date: 2015-06-21 00:00:00

We consider the problem of learning from distributed data in the agnostic
setting, i.e., in the presence of arbitrary forms of noise. Our main
contribution is a general distributed boosting-based procedure for learning an
arbitrary concept space, that is simultaneously noise tolerant, communication
efficient, and computationally efficient. This improves significantly over
prior works that were either communication efficient only in noise-free
scenarios or computationally prohibitive. Empirical results on large synthetic
and real-world datasets demonstrate the effectiveness and scalability of the
proposed approach.
Team-maxmin equilibrium: efficiency bounds and algorithms
Authors: Nicola Basilico, Andrea Celli, Giuseppe De Nittis, Nicola Gatti
Date: 2016-11-18 00:00:00

The Team-maxmin equilibrium prescribes the optimal strategies for a team of
rational players sharing the same goal and without the capability of
correlating their strategies in strategic games against an adversary. This
solution concept can capture situations in which an agent controls multiple
resources-corresponding to the team members-that cannot communicate. It is
known that such equilibrium always exists and it is unique (unless degeneracy)
and these properties make it a credible solution concept to be used in
real-world applications, especially in security scenarios. Nevertheless, to the
best of our knowledge, the Team-maxmin equilibrium is almost completely
unexplored in the literature. In this paper, we investigate bounds of
(in)efficiency of the Team-maxmin equilibrium w.r.t. the Nash equilibria and
w.r.t. the Maxmin equilibrium when the team members can play correlated
strategies. Furthermore, we study a number of algorithms to find and/or
approximate an equilibrium, discussing their theoretical guarantees and
evaluating their performance by using a standard testbed of game instances.
Faster variational inducing input Gaussian process classification
Authors: Pavel Izmailov, Dmitry Kropotov
Date: 2016-11-18 00:00:00
Tasks: Classification, Dimensionality Reduction, Gaussian Processes, General Classification, Stochastic Optimization, Variational Inference

Gaussian processes (GP) provide a prior over functions and allow finding
complex regularities in data. Gaussian processes are successfully used for
classification/regression problems and dimensionality reduction. In this work
we consider the classification problem only. The complexity of standard methods
for GP-classification scales cubically with the size of the training dataset.
This complexity makes them inapplicable to big data problems. Therefore, a
variety of methods were introduced to overcome this limitation. In the paper we
focus on methods based on so called inducing inputs. This approach is based on
variational inference and proposes a particular lower bound for marginal
likelihood (evidence). This bound is then maximized w.r.t. parameters of kernel
function of the Gaussian process, thus fitting the model to data. The
computational complexity of this method is $O(nm^2)$, where $m$ is the number
of inducing inputs used by the model and is assumed to be substantially smaller
than the size of the dataset $n$. Recently, a new evidence lower bound for
GP-classification problem was introduced. It allows using stochastic
optimization, which makes it suitable for big data problems. However, the new
lower bound depends on $O(m^2)$ variational parameter, which makes optimization
challenging in case of big m. In this work we develop a new approach for
training inducing input GP models for classification problems. Here we use
quadratic approximation of several terms in the aforementioned evidence lower
bound, obtaining analytical expressions for optimal values of most of the
parameters in the optimization, thus sufficiently reducing the dimension of
optimization space. In our experiments we achieve as well or better results,
compared to the existing method. Moreover, our method doesn't require the user
to manually set the learning rate, making it more practical, than the existing
method.
Neural Simpletrons - Minimalistic Directed Generative Networks for Learning with Few Labels
Authors: Dennis Forster, Abdul-Saboor Sheikh, Jörg Lücke
Date: 2015-06-28 00:00:00
Tasks: GPU

Classifiers for the semi-supervised setting often combine strong supervised
models with additional learning objectives to make use of unlabeled data. This
results in powerful though very complex models that are hard to train and that
demand additional labels for optimal parameter tuning, which are often not
given when labeled data is very sparse. We here study a minimalistic
multi-layer generative neural network for semi-supervised learning in a form
and setting as similar to standard discriminative networks as possible. Based
on normalized Poisson mixtures, we derive compact and local learning and neural
activation rules. Learning and inference in the network can be scaled using
standard deep learning tools for parallelized GPU implementation. With the
single objective of likelihood optimization, both labeled and unlabeled data
are naturally incorporated into learning. Empirical evaluations on standard
benchmarks show, that for datasets with few labels the derived minimalistic
network improves on all classical deep learning approaches and is competitive
with their recent variants without the need of additional labels for parameter
tuning. Furthermore, we find that the studied network is the best performing
monolithic (`non-hybrid') system for few labels, and that it can be applied in
the limit of very few labels, where no other system has been reported to
operate so far.
Fast low-level pattern matching algorithm
Authors: Janja Paliska Soldo, Ana Sovic Krzic, and Damir Sersic
Date: 2016-11-18 00:00:00

This paper focuses on pattern matching in the DNA sequence. It was inspired
by a previously reported method that proposes encoding both pattern and
sequence using prime numbers. Although fast, the method is limited to rather
small pattern lengths, due to computing precision problem. Our approach
successfully deals with large patterns, due to our implementation that uses
modular arithmetic. In order to get the results very fast, the code was adapted
for multithreading and parallel implementations. The method is reduced to
assembly language level instructions, thus the final result shows significant
time and memory savings compared to the reference algorithm.
Generalizing diffuse interface methods on graphs: non-smooth potentials and hypergraphs
Authors: Jessica Bosch, Steffen Klamt, Martin Stoll
Date: 2016-11-18 00:00:00

Diffuse interface methods have recently been introduced for the task of
semi-supervised learning. The underlying model is well-known in materials
science but was extended to graphs using a Ginzburg--Landau functional and the
graph Laplacian. We here generalize the previously proposed model by a
non-smooth potential function. Additionally, we show that the diffuse interface
method can be used for the segmentation of data coming from hypergraphs. For
this we show that the graph Laplacian in almost all cases is derived from
hypergraph information. Additionally, we show that the formerly introduced
hypergraph Laplacian coming from a relaxed optimization problem is well suited
to be used within the diffuse interface method. We present computational
experiments for graph and hypergraph Laplacians.
A Generalized Stochastic Variational Bayesian Hyperparameter Learning Framework for Sparse Spectrum Gaussian Process Regression
Authors: Quang Minh Hoang, Trong Nghia Hoang, Kian Hsiang Low
Date: 2016-11-18 00:00:00
Tasks: regression, Stochastic Optimization

While much research effort has been dedicated to scaling up sparse Gaussian
process (GP) models based on inducing variables for big data, little attention
is afforded to the other less explored class of low-rank GP approximations that
exploit the sparse spectral representation of a GP kernel. This paper presents
such an effort to advance the state of the art of sparse spectrum GP models to
achieve competitive predictive performance for massive datasets. Our
generalized framework of stochastic variational Bayesian sparse spectrum GP
(sVBSSGP) models addresses their shortcomings by adopting a Bayesian treatment
of the spectral frequencies to avoid overfitting, modeling these frequencies
jointly in its variational distribution to enable their interaction a
posteriori, and exploiting local data for boosting the predictive performance.
However, such structural improvements result in a variational lower bound that
is intractable to be optimized. To resolve this, we exploit a variational
parameterization trick to make it amenable to stochastic optimization.
Interestingly, the resulting stochastic gradient has a linearly decomposable
structure that can be exploited to refine our stochastic optimization method to
incur constant time per iteration while preserving its property of being an
unbiased estimator of the exact gradient of the variational lower bound.
Empirical evaluation on real-world datasets shows that sVBSSGP outperforms
state-of-the-art stochastic implementations of sparse GP models.