pwc-archive/papers-with-abstracts huggingface dataset:
The Power of Depth for Feedforward Neural Networks
Authors: Ronen Eldan, Ohad Shamir
Date: 2015-12-12 00:00:00

We show that there is a simple (approximately radial) function on $\reals^d$,
expressible by a small 3-layer feedforward neural networks, which cannot be
approximated by any 2-layer network, to more than a certain constant accuracy,
unless its width is exponential in the dimension. The result holds for
virtually all known activation functions, including rectified linear units,
sigmoids and thresholds, and formally demonstrates that depth -- even if
increased by 1 -- can be exponentially more valuable than width for standard
feedforward neural networks. Moreover, compared to related results in the
context of Boolean functions, our result requires fewer assumptions, and the
proof techniques and construction are very different.
Active Learning for Community Detection in Stochastic Block Models
Authors: Akshay Gadde, Eyal En Gad, Salman Avestimehr, Antonio Ortega
Date: 2016-05-08 00:00:00
Tasks: Active Learning, Benchmarking, Clustering, Community Detection, Stochastic Block Model

The stochastic block model (SBM) is an important generative model for random
graphs in network science and machine learning, useful for benchmarking
community detection (or clustering) algorithms. The symmetric SBM generates a
graph with $2n$ nodes which cluster into two equally sized communities. Nodes
connect with probability $p$ within a community and $q$ across different
communities. We consider the case of $p=a\ln (n)/n$ and $q=b\ln (n)/n$. In this
case, it was recently shown that recovering the community membership (or label)
of every node with high probability (w.h.p.) using only the graph is possible
if and only if the Chernoff-Hellinger (CH) divergence
$D(a,b)=(\sqrt{a}-\sqrt{b})^2 \geq 1$. In this work, we study if, and by how
much, community detection below the clustering threshold (i.e. $D(a,b)<1$) is
possible by querying the labels of a limited number of chosen nodes (i.e.,
active learning). Our main result is to show that, under certain conditions,
sampling the labels of a vanishingly small fraction of nodes (a number
sub-linear in $n$) is sufficient for exact community detection even when
$D(a,b)<1$. Furthermore, we provide an efficient learning algorithm which
recovers the community memberships of all nodes w.h.p. as long as the number of
sampled points meets the sufficient condition. We also show that recovery is
not possible if the number of observed labels is less than $n^{1-D(a,b)}$. The
validity of our results is demonstrated through numerical experiments.
Asymmetric Move Selection Strategies in Monte-Carlo Tree Search: Minimizing the Simple Regret at Max Nodes
Authors: Yun-Ching Liu, Yoshimasa Tsuruoka
Date: 2016-05-08 00:00:00

The combination of multi-armed bandit (MAB) algorithms with Monte-Carlo tree
search (MCTS) has made a significant impact in various research fields. The UCT
algorithm, which combines the UCB bandit algorithm with MCTS, is a good example
of the success of this combination. The recent breakthrough made by AlphaGo,
which incorporates convolutional neural networks with bandit algorithms in
MCTS, also highlights the necessity of bandit algorithms in MCTS. However,
despite the various investigations carried out on MCTS, nearly all of them
still follow the paradigm of treating every node as an independent instance of
the MAB problem, and applying the same bandit algorithm and heuristics on every
node. As a result, this paradigm may leave some properties of the game tree
unexploited. In this work, we propose that max nodes and min nodes have
different concerns regarding their value estimation, and different bandit
algorithms should be applied accordingly. We develop the Asymmetric-MCTS
algorithm, which is an MCTS variant that applies a simple regret algorithm on
max nodes, and the UCB algorithm on min nodes. We will demonstrate the
performance of the Asymmetric-MCTS algorithm on the game of $9\times 9$ Go,
$9\times 9$ NoGo, and Othello.
Recurrent Neural Network Training with Dark Knowledge Transfer
Authors: Zhiyuan Tang, Dong Wang, Zhiyong Zhang
Date: 2015-05-18 00:00:00
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Transfer Learning

Recurrent neural networks (RNNs), particularly long short-term memory (LSTM),
have gained much attention in automatic speech recognition (ASR). Although some
successful stories have been reported, training RNNs remains highly
challenging, especially with limited training data. Recent research found that
a well-trained model can be used as a teacher to train other child models, by
using the predictions generated by the teacher model as supervision. This
knowledge transfer learning has been employed to train simple neural nets with
a complex one, so that the final performance can reach a level that is
infeasible to obtain by regular training. In this paper, we employ the
knowledge transfer learning approach to train RNNs (precisely LSTM) using a
deep neural network (DNN) model as the teacher. This is different from most of
the existing research on knowledge transfer learning, since the teacher (DNN)
is assumed to be weaker than the child (RNN); however, our experiments on an
ASR task showed that it works fairly well: without applying any tricks on the
learning scheme, this approach can train RNNs successfully even with limited
training data.
Deep Learning with Eigenvalue Decay Regularizer
Authors: Oswaldo Ludwig
Date: 2016-04-24 00:00:00
Tasks: Classification, Deep Learning, General Classification, image-classification, Image Classification, Sentiment Analysis, Sentiment Classification

This paper extends our previous work on regularization of neural networks
using Eigenvalue Decay by employing a soft approximation of the dominant
eigenvalue in order to enable the calculation of its derivatives in relation to
the synaptic weights, and therefore the application of back-propagation, which
is a primary demand for deep learning. Moreover, we extend our previous
theoretical analysis to deep neural networks and multiclass classification
problems. Our method is implemented as an additional regularizer in Keras, a
modular neural networks library written in Python, and evaluated in the
benchmark data sets Reuters Newswire Topics Classification, IMDB database for
binary sentiment classification, MNIST database of handwritten digits and
CIFAR-10 data set for image classification.
Ultradense Word Embeddings by Orthogonal Transformation
Authors: Sascha Rothe, Sebastian Ebert, Hinrich Schütze
Date: 2016-02-24 00:00:00
Proceeding: NAACL 2016 6
Tasks: Sentiment Analysis, Word Embeddings

Embeddings are generic representations that are useful for many NLP tasks. In
this paper, we introduce DENSIFIER, a method that learns an orthogonal
transformation of the embedding space that focuses the information relevant for
a task in an ultradense subspace of a dimensionality that is smaller by a
factor of 100 than the original space. We show that ultradense embeddings
generated by DENSIFIER reach state of the art on a lexicon creation task in
which words are annotated with three types of lexical information - sentiment,
concreteness and frequency. On the SemEval2015 10B sentiment analysis task we
show that no information is lost when the ultradense subspace is used, but
training is an order of magnitude more efficient due to the compactness of the
ultradense space.
Detecting Ground Control Points via Convolutional Neural Network for Stereo Matching
Authors: Zhun Zhong, Songzhi Su, Donglin Cao, Shaozi Li
Date: 2016-05-08 00:00:00
Tasks: Stereo Matching, Stereo Matching Hand

In this paper, we present a novel approach to detect ground control points
(GCPs) for stereo matching problem. First of all, we train a convolutional
neural network (CNN) on a large stereo set, and compute the matching confidence
of each pixel by using the trained CNN model. Secondly, we present a ground
control points selection scheme according to the maximum matching confidence of
each pixel. Finally, the selected GCPs are used to refine the matching costs,
and we apply the new matching costs to perform optimization with semi-global
matching algorithm for improving the final disparity maps. We evaluate our
approach on the KITTI 2012 stereo benchmark dataset. Our experiments show that
the proposed approach significantly improves the accuracy of disparity maps.
On-Average KL-Privacy and its equivalence to Generalization for Max-Entropy Mechanisms
Authors: Yu-Xiang Wang, Jing Lei, Stephen E. Fienberg
Date: 2016-05-08 00:00:00

We define On-Average KL-Privacy and present its properties and connections to
differential privacy, generalization and information-theoretic quantities
including max-information and mutual information. The new definition
significantly weakens differential privacy, while preserving its minimalistic
design features such as composition over small group and multiple queries as
well as closeness to post-processing. Moreover, we show that On-Average
KL-Privacy is **equivalent** to generalization for a large class of
commonly-used tools in statistics and machine learning that samples from Gibbs
distributions---a class of distributions that arises naturally from the maximum
entropy principle. In addition, a byproduct of our analysis yields a lower
bound for generalization error in terms of mutual information which reveals an
interesting interplay with known upper bounds that use the same quantity.
Predicting Performance on MOOC Assessments using Multi-Regression Models
Authors: Zhiyun Ren, Huzefa Rangwala, Aditya Johri
Date: 2016-05-08 00:00:00
Tasks: regression

The past few years has seen the rapid growth of data min- ing approaches for
the analysis of data obtained from Mas- sive Open Online Courses (MOOCs). The
objectives of this study are to develop approaches to predict the scores a stu-
dent may achieve on a given grade-related assessment based on information,
considered as prior performance or prior ac- tivity in the course. We develop a
personalized linear mul- tiple regression (PLMR) model to predict the grade for
a student, prior to attempting the assessment activity. The developed model is
real-time and tracks the participation of a student within a MOOC (via
click-stream server logs) and predicts the performance of a student on the next
as- sessment within the course offering. We perform a com- prehensive set of
experiments on data obtained from three openEdX MOOCs via a Stanford University
initiative. Our experimental results show the promise of the proposed ap-
proach in comparison to baseline approaches and also helps in identification of
key features that are associated with the study habits and learning behaviors
of students.
Deeply Exploit Depth Information for Object Detection
Authors: Saihui Hou, Zilei Wang, Feng Wu
Date: 2016-05-08 00:00:00
Tasks: Object, object-detection, Object Detection

This paper addresses the issue on how to more effectively coordinate the
depth with RGB aiming at boosting the performance of RGB-D object detection.
Particularly, we investigate two primary ideas under the CNN model: property
derivation and property fusion. Firstly, we propose that the depth can be
utilized not only as a type of extra information besides RGB but also to derive
more visual properties for comprehensively describing the objects of interest.
So a two-stage learning framework consisting of property derivation and fusion
is constructed. Here the properties can be derived either from the provided
color/depth or their pairs (e.g. the geometry contour adopted in this paper).
Secondly, we explore the fusion method of different properties in feature
learning, which is boiled down to, under the CNN model, from which layer the
properties should be fused together. The analysis shows that different semantic
properties should be learned separately and combined before passing into the
final classifier. Actually, such a detection way is in accordance with the
mechanism of the primary neural cortex (V1) in brain. We experimentally
evaluate the proposed method on the challenging dataset, and have achieved
state-of-the-art performance.
A corpus of preposition supersenses in English web reviews
Authors: Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Meredith Green, Kathryn Conger, Tim O'Gorman, Martha Palmer
Date: 2016-05-08 00:00:00

We present the first corpus annotated with preposition supersenses,
unlexicalized categories for semantic functions that can be marked by English
prepositions (Schneider et al., 2015). That scheme improves upon its
predecessors to better facilitate comprehensive manual annotation. Moreover,
unlike the previous schemes, the preposition supersenses are organized
hierarchically. Our data will be publicly released on the web upon publication.
On Image segmentation using Fractional Gradients-Learning Model Parameters using Approximate Marginal Inference
Authors: Anish Acharya, Uddipan Mukherjee, Charless Fowlkes
Date: 2016-05-07 00:00:00
Tasks: Edge Detection, General Classification, Image Segmentation, Segmentation, Semantic Segmentation

Estimates of image gradients play a ubiquitous role in image segmentation and
classification problems since gradients directly relate to the boundaries or
the edges of a scene. This paper proposes an unified approach to gradient
estimation based on fractional calculus that is computationally cheap and
readily applicable to any existing algorithm that relies on image gradients. We
show experiments on edge detection and image segmentation on the Stanford
Backgrounds Dataset where these improved local gradients outperforms state of
the art, achieving a performance of 79.2% average accuracy.
Distributed stochastic optimization for deep learning (thesis)
Authors: Sixin Zhang
Date: 2016-05-07 00:00:00
Tasks: Deep Learning, image-classification, Image Classification, Stochastic Optimization

We study the problem of how to distribute the training of large-scale deep
learning models in the parallel computing environment. We propose a new
distributed stochastic optimization method called Elastic Averaging SGD
(EASGD). We analyze the convergence rate of the EASGD method in the synchronous
scenario and compare its stability condition with the existing ADMM method in
the round-robin scheme. An asynchronous and momentum variant of the EASGD
method is applied to train deep convolutional neural networks for image
classification on the CIFAR and ImageNet datasets. Our approach accelerates the
training and furthermore achieves better test accuracy. It also requires a much
smaller amount of communication than other common baseline approaches such as
the DOWNPOUR method.
  We then investigate the limit in speedup of the initial and the asymptotic
phase of the mini-batch SGD, the momentum SGD, and the EASGD methods. We find
that the spread of the input data distribution has a big impact on their
initial convergence rate and stability region. We also find a surprising
connection between the momentum SGD and the EASGD method with a negative moving
average rate. A non-convex case is also studied to understand when EASGD can
get trapped by a saddle point.
  Finally, we scale up the EASGD method by using a tree structured network
topology. We show empirically its advantage and challenge. We also establish a
connection between the EASGD and the DOWNPOUR method with the classical Jacobi
and the Gauss-Seidel method, thus unifying a class of distributed stochastic
optimization methods.
All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehicles
Authors: Peter Radecki, Mark Campbell, Kevin Matzen
Date: 2016-05-07 00:00:00
Tasks: All, Classification, General Classification, Multi-Object Tracking, Object, Object Tracking

A novel probabilistic perception algorithm is presented as a real-time joint
solution to data association, object tracking, and object classification for an
autonomous ground vehicle in all-weather conditions. The presented algorithm
extends a Rao-Blackwellized Particle Filter originally built with a particle
filter for data association and a Kalman filter for multi-object tracking
(Miller et al. 2011a) to now also include multiple model tracking for
classification. Additionally a state-of-the-art vision detection algorithm that
includes heading information for autonomous ground vehicle (AGV) applications
was implemented. Cornell's AGV from the DARPA Urban Challenge was upgraded and
used to experimentally examine if and how state-of-the-art vision algorithms
can complement or replace lidar and radar sensors. Sensor and algorithm
performance in adverse weather and lighting conditions is tested. Experimental
evaluation demonstrates robust all-weather data association, tracking, and
classification where camera, lidar, and radar sensors complement each other
inside the joint probabilistic perception algorithm.
Matching models across abstraction levels with Gaussian Processes
Authors: Giulio Caravagna, Luca Bortolussi, Guido Sanguinetti
Date: 2016-05-07 00:00:00
Tasks: Gaussian Processes

Biological systems are often modelled at different levels of abstraction
depending on the particular aims/resources of a study. Such different models
often provide qualitatively concordant predictions over specific
parametrisations, but it is generally unclear whether model predictions are
quantitatively in agreement, and whether such agreement holds for different
parametrisations. Here we present a generally applicable statistical machine
learning methodology to automatically reconcile the predictions of different
models across abstraction levels. Our approach is based on defining a
correction map, a random function which modifies the output of a model in order
to match the statistics of the output of a different model of the same system.
We use two biological examples to give a proof-of-principle demonstration of
the methodology, and discuss its advantages and potential further applications.
Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences
Authors: Akira Taniguchi, Tadahiro Taniguchi, Tetsunari Inamura
Date: 2016-02-03 00:00:00

In this paper, we propose a novel unsupervised learning method for the
lexical acquisition of words related to places visited by robots, from human
continuous speech signals. We address the problem of learning novel words by a
robot that has no prior knowledge of these words except for a primitive
acoustic model. Further, we propose a method that allows a robot to effectively
use the learned words and their meanings for self-localization tasks. The
proposed method is nonparametric Bayesian spatial concept acquisition method
(SpCoA) that integrates the generative model for self-localization and the
unsupervised word segmentation in uttered sentences via latent variables
related to the spatial concept. We implemented the proposed method SpCoA on
SIGVerse, which is a simulation environment, and TurtleBot2, which is a mobile
robot in a real environment. Further, we conducted experiments for evaluating
the performance of SpCoA. The experimental results showed that SpCoA enabled
the robot to acquire the names of places from speech sentences. They also
revealed that the robot could effectively utilize the acquired spatial concepts
and reduce the uncertainty in self-localization.
Fast Bilateral Filtering of Vector-Valued Images
Authors: Sanjay Ghosh, Kunal. N. Chaudhury
Date: 2016-05-07 00:00:00

In this paper, we consider a natural extension of the edge-preserving
bilateral filter for vector-valued images. The direct computation of this
non-linear filter is slow in practice. We demonstrate how a fast algorithm can
be obtained by first approximating the Gaussian kernel of the bilateral filter
using raised-cosines, and then using Monte Carlo sampling. We present
simulation results on color images to demonstrate the accuracy of the algorithm
and the speedup over the direct implementation.
Belief Merging by Source Reliability Assessment
Authors: Paolo Liberatore
Date: 2016-05-07 00:00:00

Merging beliefs requires the plausibility of the sources of the information
to be merged. They are typically assumed equally reliable in lack of hints
indicating otherwise; yet, a recent line of research spun from the idea of
deriving this information from the revision process itself. In particular, the
history of previous revisions and previous merging examples provide information
for performing subsequent mergings.
  Yet, no examples or previous revisions may be available. In spite of the
apparent lack of information, something can still be inferred by a
try-and-check approach: a relative reliability ordering is assumed, the merging
process is performed based on it, and the result is compared with the original
information. The outcome of this check may be incoherent with the initial
assumption, like when a completely reliable source is rejected some of the
information it provided. In such cases, the reliability ordering assumed in the
first place can be excluded from consideration. The first theorem of this
article proves that such a scenario is indeed possible. Other results are
obtained under various definition of reliability and merging.
Learning with Memory Embeddings
Authors: Volker Tresp, Cristóbal Esteban, Yinchong Yang, Stephan Baier, Denis Krompaß
Date: 2015-11-25 00:00:00
Tasks: Knowledge Graphs, Representation Learning

Embedding learning, a.k.a. representation learning, has been shown to be able
to model large-scale semantic knowledge graphs. A key concept is a mapping of
the knowledge graph to a tensor representation whose entries are predicted by
models using latent representations of generalized entities. Latent variable
models are well suited to deal with the high dimensionality and sparsity of
typical knowledge graphs. In recent publications the embedding models were
extended to also consider time evolutions, time patterns and subsymbolic
representations. In this paper we map embedding models, which were developed
purely as solutions to technical problems for modelling temporal knowledge
graphs, to various cognitive memory functions, in particular to semantic and
concept memory, episodic memory, sensory memory, short-term memory, and working
memory. We discuss learning, query answering, the path from sensory input to
semantic decoding, and the relationship between episodic memory and semantic
memory. We introduce a number of hypotheses on human memory that can be derived
from the developed mathematical models.
On Improving Informativity and Grammaticality for Multi-Sentence Compression
Authors: Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond Wong, Fang Chen
Date: 2016-05-07 00:00:00
Tasks: Language Modeling, Language Modelling, Opinion Summarization, POS, Re-Ranking, Sentence, Sentence Compression

Multi Sentence Compression (MSC) is of great value to many real world
applications, such as guided microblog summarization, opinion summarization and
newswire summarization. Recently, word graph-based approaches have been
proposed and become popular in MSC. Their key assumption is that redundancy
among a set of related sentences provides a reliable way to generate
informative and grammatical sentences. In this paper, we propose an effective
approach to enhance the word graph-based MSC and tackle the issue that most of
the state-of-the-art MSC approaches are confronted with: i.e., improving both
informativity and grammaticality at the same time. Our approach consists of
three main components: (1) a merging method based on Multiword Expressions
(MWE); (2) a mapping strategy based on synonymy between words; (3) a re-ranking
step to identify the best compression candidates generated using a POS-based
language model (POS-LM). We demonstrate the effectiveness of this novel
approach using a dataset made of clusters of English newswire sentences. The
observed improvements on informativity and grammaticality of the generated
compressions show that our approach is superior to state-of-the-art MSC
methods.