pwc-archive/papers-with-abstracts huggingface dataset:
Neural Models for Sequence Chunking
Authors: Feifei Zhai, Saloni Potdar, Bing Xiang, Bo-Wen Zhou
Date: 2017-01-15 00:00:00
Tasks: Chunking, Natural Language Understanding, Sentence, slot-filling, Slot Filling

Many natural language understanding (NLU) tasks, such as shallow parsing
(i.e., text chunking) and semantic slot filling, require the assignment of
representative labels to the meaningful chunks in a sentence. Most of the
current deep neural network (DNN) based methods consider these tasks as a
sequence labeling problem, in which a word, rather than a chunk, is treated as
the basic unit for labeling. These chunks are then inferred by the standard IOB
(Inside-Outside-Beginning) labels. In this paper, we propose an alternative
approach by investigating the use of DNN for sequence chunking, and propose
three neural models so that each chunk can be treated as a complete unit for
labeling. Experimental results show that the proposed neural sequence chunking
models can achieve start-of-the-art performance on both the text chunking and
slot filling tasks.
Boosting Dictionary Learning with Error Codes
Authors: Yigit Oktar, Mehmet Turkan
Date: 2017-01-15 00:00:00
Tasks: Dictionary Learning

In conventional sparse representations based dictionary learning algorithms,
initial dictionaries are generally assumed to be proper representatives of the
system at hand. However, this may not be the case, especially in some systems
restricted to random initializations. Therefore, a supposedly optimal
state-update based on such an improper model might lead to undesired effects
that will be conveyed to successive iterations. In this paper, we propose a
dictionary learning method which includes a general feedback process that codes
the intermediate error left over from a less intensive initial learning
attempt, and then adjusts sparse codes accordingly. Experimental observations
show that such an additional step vastly improves rates of convergence in
high-dimensional cases, also results in better converged states in the case of
random initializations. Improvements also scale up with more lenient sparsity
constraints.
Towards Music Captioning: Generating Music Playlist Descriptions
Authors: Keunwoo Choi, George Fazekas, Brian McFee, Kyunghyun Cho, Mark Sandler
Date: 2016-08-17 00:00:00
Tasks: Music Captioning

Descriptions are often provided along with recommendations to help users'
discovery. Recommending automatically generated music playlists (e.g.
personalised playlists) introduces the problem of generating descriptions. In
this paper, we propose a method for generating music playlist descriptions,
which is called as music captioning. In the proposed method, audio content
analysis and natural language processing are adopted to utilise the information
of each track.
Quantitative Entropy Study of Language Complexity
Authors: R. R. Xie, W. B. Deng, D. J. Wang, L. P. Csernai
Date: 2016-11-14 00:00:00

We study the entropy of Chinese and English texts, based on characters in
case of Chinese texts and based on words for both languages. Significant
differences are found between the languages and between different personal
styles of debating partners. The entropy analysis points in the direction of
lower entropy, that is of higher complexity. Such a text analysis would be
applied for individuals of different styles, a single individual at different
age, as well as different groups of the population.
DyNet: The Dynamic Neural Network Toolkit
Authors: Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, Pengcheng Yin
Date: 2017-01-15 00:00:00
Tasks: graph construction

We describe DyNet, a toolkit for implementing neural network models based on
dynamic declaration of network structure. In the static declaration strategy
that is used in toolkits like Theano, CNTK, and TensorFlow, the user first
defines a computation graph (a symbolic representation of the computation), and
then examples are fed into an engine that executes this computation and
computes its derivatives. In DyNet's dynamic declaration strategy, computation
graph construction is mostly transparent, being implicitly constructed by
executing procedural code that computes the network outputs, and the user is
free to use different network structures for each input. Dynamic declaration
thus facilitates the implementation of more complicated network architectures,
and DyNet is specifically designed to allow users to implement their models in
a way that is idiomatic in their preferred programming language (C++ or
Python). One challenge with dynamic declaration is that because the symbolic
computation graph is defined anew for every training example, its construction
must have low overhead. To achieve this, DyNet has an optimized C++ backend and
lightweight graph representation. Experiments show that DyNet's speeds are
faster than or comparable with static declaration toolkits, and significantly
faster than Chainer, another dynamic declaration toolkit. DyNet is released
open-source under the Apache 2.0 license and available at
http://github.com/clab/dynet.
Infinite-dimensional Log-Determinant divergences II: Alpha-Beta divergences
Authors: Minh Ha Quang
Date: 2016-10-13 00:00:00

This work presents a parametrized family of divergences, namely Alpha-Beta
Log- Determinant (Log-Det) divergences, between positive definite unitized
trace class operators on a Hilbert space. This is a generalization of the
Alpha-Beta Log-Determinant divergences between symmetric, positive definite
matrices to the infinite-dimensional setting. The family of Alpha-Beta Log-Det
divergences is highly general and contains many divergences as special cases,
including the recently formulated infinite dimensional affine-invariant
Riemannian distance and the infinite-dimensional Alpha Log-Det divergences
between positive definite unitized trace class operators. In particular, it
includes a parametrized family of metrics between positive definite trace class
operators, with the affine-invariant Riemannian distance and the square root of
the symmetric Stein divergence being special cases. For the Alpha-Beta Log-Det
divergences between covariance operators on a Reproducing Kernel Hilbert Space
(RKHS), we obtain closed form formulas via the corresponding Gram matrices.
Balancing Novelty and Salience: Adaptive Learning to Rank Entities for Timeline Summarization of High-impact Events
Authors: Tuan Tran, Claudia Niederée, Nattiya Kanhabua, Ujwal Gadiraju, Avishek Anand
Date: 2017-01-14 00:00:00
Tasks: Informativeness, Learning-To-Rank, Timeline Summarization

Long-running, high-impact events such as the Boston Marathon bombing often
develop through many stages and involve a large number of entities in their
unfolding. Timeline summarization of an event by key sentences eases story
digestion, but does not distinguish between what a user remembers and what she
might want to re-check. In this work, we present a novel approach for timeline
summarization of high-impact events, which uses entities instead of sentences
for summarizing the event at each individual point in time. Such entity
summaries can serve as both (1) important memory cues in a retrospective event
consideration and (2) pointers for personalized event exploration. In order to
automatically create such summaries, it is crucial to identify the "right"
entities for inclusion. We propose to learn a ranking function for entities,
with a dynamically adapted trade-off between the in-document salience of
entities and the informativeness of entities across documents, i.e., the level
of new information associated with an entity for a time point under
consideration. Furthermore, for capturing collective attention for an entity we
use an innovative soft labeling approach based on Wikipedia. Our experiments on
a real large news datasets confirm the effectiveness of the proposed methods.
Scalable and Incremental Learning of Gaussian Mixture Models
Authors: Rafael Pinto, Paulo Engel
Date: 2017-01-14 00:00:00
Tasks: General Classification, Incremental Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

This work presents a fast and scalable algorithm for incremental learning of
Gaussian mixture models. By performing rank-one updates on its precision
matrices and determinants, its asymptotic time complexity is of \BigO{NKD^2}
for $N$ data points, $K$ Gaussian components and $D$ dimensions. The resulting
algorithm can be applied to high dimensional tasks, and this is confirmed by
applying it to the classification datasets MNIST and CIFAR-10. Additionally, in
order to show the algorithm's applicability to function approximation and
control tasks, it is applied to three reinforcement learning tasks and its
data-efficiency is evaluated.
Hedera: Scalable Indexing and Exploring Entities in Wikipedia Revision History
Authors: Tuan Tran, Tu Ngoc Nguyen
Date: 2017-01-14 00:00:00
Tasks: Articles

Much of work in semantic web relying on Wikipedia as the main source of
knowledge often work on static snapshots of the dataset. The full history of
Wikipedia revisions, while contains much more useful information, is still
difficult to access due to its exceptional volume. To enable further research
on this collection, we developed a tool, named Hedera, that efficiently
extracts semantic information from Wikipedia revision history datasets. Hedera
exploits Map-Reduce paradigm to achieve rapid extraction, it is able to handle
one entire Wikipedia articles revision history within a day in a medium-scale
cluster, and supports flexible data structures for various kinds of semantic
web study.
QCRI Machine Translation Systems for IWSLT 16
Authors: Nadir Durrani, Fahim Dalvi, Hassan Sajjad, Stephan Vogel
Date: 2017-01-14 00:00:00
Tasks: Domain Adaptation, Language Modeling, Language Modelling, Machine Translation, NMT, Translation

This paper describes QCRI's machine translation systems for the IWSLT 2016
evaluation campaign. We participated in the Arabic->English and English->Arabic
tracks. We built both Phrase-based and Neural machine translation models, in an
effort to probe whether the newly emerged NMT framework surpasses the
traditional phrase-based systems in Arabic-English language pairs. We trained a
very strong phrase-based system including, a big language model, the Operation
Sequence Model, Neural Network Joint Model and Class-based models along with
different domain adaptation techniques such as MML filtering, mixture modeling
and using fine tuning over NNJM model. However, a Neural MT system, trained by
stacking data from different genres through fine-tuning, and applying ensemble
over 8 models, beat our very strong phrase-based system by a significant 2 BLEU
points margin in Arabic->English direction. We did not obtain similar gains in
the other direction but were still able to outperform the phrase-based system.
We also applied system combination on phrase-based and NMT outputs.
Marked Temporal Dynamics Modeling based on Recurrent Neural Network
Authors: Yongqing Wang, Shenghua Liu, Hua-Wei Shen, Xue-Qi Cheng
Date: 2017-01-14 00:00:00

We are now witnessing the increasing availability of event stream data, i.e.,
a sequence of events with each event typically being denoted by the time it
occurs and its mark information (e.g., event type). A fundamental problem is to
model and predict such kind of marked temporal dynamics, i.e., when the next
event will take place and what its mark will be. Existing methods either
predict only the mark or the time of the next event, or predict both of them,
yet separately. Indeed, in marked temporal dynamics, the time and the mark of
the next event are highly dependent on each other, requiring a method that
could simultaneously predict both of them. To tackle this problem, in this
paper, we propose to model marked temporal dynamics by using a mark-specific
intensity function to explicitly capture the dependency between the mark and
the time of the next event. Extensive experiments on two datasets demonstrate
that the proposed method outperforms state-of-the-art methods at predicting
marked temporal dynamics.
On Hölder projective divergences
Authors: Frank Nielsen, Ke Sun, Stéphane Marchand-Maillet
Date: 2017-01-14 00:00:00
Tasks: Clustering

We describe a framework to build distances by measuring the tightness of
inequalities, and introduce the notion of proper statistical divergences and
improper pseudo-divergences. We then consider the H\"older ordinary and reverse
inequalities, and present two novel classes of H\"older divergences and
pseudo-divergences that both encapsulate the special case of the Cauchy-Schwarz
divergence. We report closed-form formulas for those statistical
dissimilarities when considering distributions belonging to the same
exponential family provided that the natural parameter space is a cone (e.g.,
multivariate Gaussians), or affine (e.g., categorical distributions). Those new
classes of H\"older distances are invariant to rescaling, and thus do not
require distributions to be normalized. Finally, we show how to compute
statistical H\"older centroids with respect to those divergences, and carry out
center-based clustering toy experiments on a set of Gaussian distributions that
demonstrate empirically that symmetrized H\"older divergences outperform the
symmetric Cauchy-Schwarz divergence.
The Landscape of Empirical Risk for Non-convex Losses
Authors: Song Mei, Yu Bai, Andrea Montanari
Date: 2016-07-22 00:00:00
Tasks: Binary Classification, General Classification, regression

Most high-dimensional estimation and prediction methods propose to minimize a
cost function (empirical risk) that is written as a sum of losses associated to
each data point. In this paper we focus on the case of non-convex losses, which
is practically important but still poorly understood. Classical empirical
process theory implies uniform convergence of the empirical risk to the
population risk. While uniform convergence implies consistency of the resulting
M-estimator, it does not ensure that the latter can be computed efficiently.
  In order to capture the complexity of computing M-estimators, we propose to
study the landscape of the empirical risk, namely its stationary points and
their properties. We establish uniform convergence of the gradient and Hessian
of the empirical risk to their population counterparts, as soon as the number
of samples becomes larger than the number of unknown parameters (modulo
logarithmic factors). Consequently, good properties of the population risk can
be carried to the empirical risk, and we can establish one-to-one
correspondence of their stationary points. We demonstrate that in several
problems such as non-convex binary classification, robust regression, and
Gaussian mixture model, this result implies a complete characterization of the
landscape of the empirical risk, and of the convergence properties of descent
algorithms.
  We extend our analysis to the very high-dimensional setting in which the
number of parameters exceeds the number of samples, and provide a
characterization of the empirical risk landscape under a nearly
information-theoretically minimal condition. Namely, if the number of samples
exceeds the sparsity of the unknown parameters vector (modulo logarithmic
factors), then a suitable uniform convergence result takes place. We apply this
result to non-convex binary classification and robust regression in very
high-dimension.
Learning to Invert: Signal Recovery via Deep Convolutional Networks
Authors: Ali Mousavi, Richard G. Baraniuk
Date: 2017-01-14 00:00:00
Tasks: Compressive Sensing

The promise of compressive sensing (CS) has been offset by two significant
challenges. First, real-world data is not exactly sparse in a fixed basis.
Second, current high-performance recovery algorithms are slow to converge,
which limits CS to either non-real-time applications or scenarios where massive
back-end computing is available. In this paper, we attack both of these
challenges head-on by developing a new signal recovery framework we call {\em
DeepInverse} that learns the inverse transformation from measurement vectors to
signals using a {\em deep convolutional network}. When trained on a set of
representative images, the network learns both a representation for the signals
(addressing challenge one) and an inverse map approximating a greedy or convex
recovery algorithm (addressing challenge two). Our experiments indicate that
the DeepInverse network closely approximates the solution produced by
state-of-the-art CS recovery algorithms yet is hundreds of times faster in run
time. The tradeoff for the ultrafast run time is a computationally intensive,
off-line training procedure typical to deep networks. However, the training
needs to be completed only once, which makes the approach attractive for a host
of sparse recovery problems.
A Threshold-based Scheme for Reinforcement Learning in Neural Networks
Authors: Thomas H. Ward
Date: 2016-09-12 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

A generic and scalable Reinforcement Learning scheme for Artificial Neural
Networks is presented, providing a general purpose learning machine. By
reference to a node threshold three features are described 1) A mechanism for
Primary Reinforcement, capable of solving linearly inseparable problems 2) The
learning scheme is extended to include a mechanism for Conditioned
Reinforcement, capable of forming long term strategy 3) The learning scheme is
modified to use a threshold-based deep learning algorithm, providing a robust
and biologically inspired alternative to backpropagation. The model may be used
for supervised as well as unsupervised training regimes.
Learning Linear Dynamical Systems with High-Order Tensor Data for Skeleton based Action Recognition
Authors: Wenwen Ding, Kai Liu
Date: 2017-01-14 00:00:00
Tasks: Action Recognition, Dictionary Learning, Skeleton Based Action Recognition, Temporal Action Localization, Time Series, Time Series Analysis

In recent years, there has been renewed interest in developing methods for
skeleton-based human action recognition. A skeleton sequence can be naturally
represented as a high-order tensor time series. In this paper, we model and
analyze tensor time series with Linear Dynamical System (LDS) which is the most
common for encoding spatio-temporal time-series data in various disciplines dut
to its relative simplicity and efficiency. However, the traditional LDS treats
the latent and observation state at each frame of video as a column vector.
Such a vector representation fails to take into account the curse of
dimensionality as well as valuable structural information with human action.
Considering this fact, we propose generalized Linear Dynamical System (gLDS)
for modeling tensor observation in the time series and employ Tucker
decomposition to estimate the LDS parameters as action descriptors. Therefore,
an action can be represented as a subspace corresponding to a point on a
Grassmann manifold. Then we perform classification using dictionary learning
and sparse coding over Grassmann manifold. Experiments on MSR Action3D Dataset,
UCF Kinect Dataset and Northwestern-UCLA Multiview Action3D Dataset demonstrate
that our proposed method achieves superior performance to the state-of-the-art
algorithms.
Minimally Naturalistic Artificial Intelligence
Authors: Steven Stenberg Hansen
Date: 2017-01-14 00:00:00
Tasks: Common Sense Reasoning, Inductive Bias, Meta-Learning

The rapid advancement of machine learning techniques has re-energized
research into general artificial intelligence. While the idea of
domain-agnostic meta-learning is appealing, this emerging field must come to
terms with its relationship to human cognition and the statistics and structure
of the tasks humans perform. The position of this article is that only by
aligning our agents' abilities and environments with those of humans do we
stand a chance at developing general artificial intelligence (GAI). A broad
reading of the famous 'No Free Lunch' theorem is that there is no universally
optimal inductive bias or, equivalently, bias-free learning is impossible. This
follows from the fact that there are an infinite number of ways to extrapolate
data, any of which might be the one used by the data generating environment; an
inductive bias prefers some of these extrapolations to others, which lowers
performance in environments using these adversarial extrapolations. We may
posit that the optimal GAI is the one that maximally exploits the statistics of
its environment to create its inductive bias; accepting the fact that this
agent is guaranteed to be extremely sub-optimal for some alternative
environments. This trade-off appears benign when thinking about the environment
as being the physical universe, as performance on any fictive universe is
obviously irrelevant. But, we should expect a sharper inductive bias if we
further constrain our environment. Indeed, we implicitly do so by defining GAI
in terms of accomplishing that humans consider useful. One common version of
this is need the for 'common-sense reasoning', which implicitly appeals to the
statistics of physical universe as perceived by humans.
Long Timescale Credit Assignment in NeuralNetworks with External Memory
Authors: Steven Stenberg Hansen
Date: 2017-01-14 00:00:00

Credit assignment in traditional recurrent neural networks usually involves
back-propagating through a long chain of tied weight matrices. The length of
this chain scales linearly with the number of time-steps as the same network is
run at each time-step. This creates many problems, such as vanishing gradients,
that have been well studied. In contrast, a NNEM's architecture recurrent
activity doesn't involve a long chain of activity (though some architectures
such as the NTM do utilize a traditional recurrent architecture as a
controller). Rather, the externally stored embedding vectors are used at each
time-step, but no messages are passed from previous time-steps. This means that
vanishing gradients aren't a problem, as all of the necessary gradient paths
are short. However, these paths are extremely numerous (one per embedding
vector in memory) and reused for a very long time (until it leaves the memory).
Thus, the forward-pass information of each memory must be stored for the entire
duration of the memory. This is problematic as this additional storage far
surpasses that of the actual memories, to the extent that large memories on
infeasible to back-propagate through in high dimensional settings. One way to
get around the need to hold onto forward-pass information is to recalculate the
forward-pass whenever gradient information is available. However, if the
observations are too large to store in the domain of interest, direct
reinstatement of a forward pass cannot occur. Instead, we rely on a learned
autoencoder to reinstate the observation, and then use the embedding network to
recalculate the forward-pass. Since the recalculated embedding vector is
unlikely to perfectly match the one stored in memory, we try out 2
approximations to utilize error gradient w.r.t. the vector in memory.