pwc-archive/papers-with-abstracts huggingface dataset:
Nonparametric Spherical Topic Modeling with Word Embeddings
Authors: Kayhan Batmanghelich, Ardavan Saeedi, Karthik Narasimhan, Sam Gershman
Date: 2016-04-01 00:00:00
Proceeding: ACL 2016 8
Tasks: Topic Models, Variational Inference, Word Embeddings

Traditional topic models do not account for semantic regularities in
language. Recent distributional representations of words exhibit semantic
consistency over directional metrics such as cosine similarity. However,
neither categorical nor Gaussian observational distributions used in existing
topic models are appropriate to leverage such correlations. In this paper, we
propose to use the von Mises-Fisher distribution to model the density of words
over a unit sphere. Such a representation is well-suited for directional data.
We use a Hierarchical Dirichlet Process for our base topic model and propose an
efficient inference algorithm based on Stochastic Variational Inference. This
model enables us to naturally exploit the semantic structures of word
embeddings while flexibly discovering the number of topics. Experiments
demonstrate that our method outperforms competitive approaches in terms of
topic coherence on two different text corpora while offering efficient
inference.
Zipf's law emerges asymptotically during phase transitions in communicative systems
Authors: Bohdan B. Khomtchouk, Claes Wahlestedt
Date: 2016-03-10 00:00:00

Zipf's law predicts a power-law relationship between word rank and frequency
in language communication systems, and is widely reported in texts yet remains
enigmatic as to its origins. Computer simulations have shown that language
communication systems emerge at an abrupt phase transition in the fidelity of
mappings between symbols and objects. Since the phase transition approximates
the Heaviside or step function, we show that Zipfian scaling emerges
asymptotically at high rank based on the Laplace transform. We thereby
demonstrate that Zipf's law gradually emerges from the moment of phase
transition in communicative systems. We show that this power-law scaling
behavior explains the emergence of natural languages at phase transitions. We
find that the emergence of Zipf's law during language communication suggests
that the use of rare words in a lexicon is critical for the construction of an
effective communicative system at the phase transition.
Studying Very Low Resolution Recognition Using Deep Networks
Authors: Zhangyang Wang, Shiyu Chang, Yingzhen Yang, Ding Liu, Thomas S. Huang
Date: 2016-01-16 00:00:00
Proceeding: CVPR 2016 6
Tasks: Domain Adaptation, Face Identification, Font Recognition, Super-Resolution

Visual recognition research often assumes a sufficient resolution of the
region of interest (ROI). That is usually violated in practice, inspiring us to
explore the Very Low Resolution Recognition (VLRR) problem. Typically, the ROI
in a VLRR problem can be smaller than $16 \times 16$ pixels, and is challenging
to be recognized even by human experts. We attempt to solve the VLRR problem
using deep learning methods. Taking advantage of techniques primarily in super
resolution, domain adaptation and robust regression, we formulate a dedicated
deep learning method and demonstrate how these techniques are incorporated step
by step. Any extra complexity, when introduced, is fully justified by both
analysis and simulation results. The resulting \textit{Robust Partially Coupled
Networks} achieves feature enhancement and recognition simultaneously. It
allows for both the flexibility to combat the LR-HR domain mismatch, and the
robustness to outliers. Finally, the effectiveness of the proposed models is
evaluated on three different VLRR tasks, including face identification, digit
recognition and font recognition, all of which obtain very impressive
performances.
The Open World of Micro-Videos
Authors: Phuc Xuan Nguyen, Gregory Rogez, Charless Fowlkes, Deva Ramanan
Date: 2016-03-31 00:00:00
Tasks: Diversity, TAG, Video Understanding

Micro-videos are six-second videos popular on social media networks with
several unique properties. Firstly, because of the authoring process, they
contain significantly more diversity and narrative structure than existing
collections of video "snippets". Secondly, because they are often captured by
hand-held mobile cameras, they contain specialized viewpoints including
third-person, egocentric, and self-facing views seldom seen in traditional
produced video. Thirdly, due to to their continuous production and publication
on social networks, aggregate micro-video content contains interesting
open-world dynamics that reflects the temporal evolution of tag topics. These
aspects make micro-videos an appealing well of visual data for developing
large-scale models for video understanding. We analyze a novel dataset of
micro-videos labeled with 58 thousand tags. To analyze this data, we introduce
viewpoint-specific and temporally-evolving models for video understanding,
defined over state-of-the-art motion and deep visual features. We conclude that
our dataset opens up new research opportunities for large-scale video analysis,
novel viewpoints, and open-world dynamics.
A Compositional Approach to Language Modeling
Authors: Kushal Arora, Anand Rangarajan
Date: 2016-04-01 00:00:00
Tasks: Language Modeling, Language Modelling, Sentence

Traditional language models treat language as a finite state automaton on a
probability space over words. This is a very strong assumption when modeling
something inherently complex such as language. In this paper, we challenge this
by showing how the linear chain assumption inherent in previous work can be
translated into a sequential composition tree. We then propose a new model that
marginalizes over all possible composition trees thereby removing any
underlying structural assumptions. As the partition function of this new model
is intractable, we use a recently proposed sentence level evaluation metric
Contrastive Entropy to evaluate our model. Given this new evaluation metric, we
report more than 100% improvement across distortion levels over current state
of the art recurrent neural network based language models.
Variational reaction-diffusion systems for semantic segmentation
Authors: Paul Vernaza
Date: 2016-04-01 00:00:00
Tasks: Image Segmentation, Semantic Segmentation

A novel global energy model for multi-class semantic image segmentation is
proposed that admits very efficient exact inference and derivative calculations
for learning. Inference in this model is equivalent to MAP inference in a
particular kind of vector-valued Gaussian Markov random field, and ultimately
reduces to solving a linear system of linear PDEs known as a reaction-diffusion
system. Solving this system can be achieved in time scaling near-linearly in
the number of image pixels by reducing it to sequential FFTs, after a linear
change of basis. The efficiency and differentiability of the model make it
especially well-suited for integration with convolutional neural networks, even
allowing it to be used in interior, feature-generating layers and stacked
multiple times. Experimental results are shown demonstrating that the model can
be employed profitably in conjunction with different convolutional net
architectures, and that doing so compares favorably to joint training of a
fully-connected CRF with a convolutional net.
Neural Attention Models for Sequence Classification: Analysis and Application to Key Term Extraction and Dialogue Act Detection
Authors: Sheng-syun Shen, Hung-Yi Lee
Date: 2016-03-31 00:00:00
Tasks: Caption Generation, Classification, General Classification, Machine Translation, Question Answering, speech-recognition, Speech Recognition, Term Extraction, Translation, Visual Question Answering, Visual Question Answering (VQA)

Recurrent neural network architectures combining with attention mechanism, or
neural attention model, have shown promising performance recently for the tasks
including speech recognition, image caption generation, visual question
answering and machine translation. In this paper, neural attention model is
applied on two sequence classification tasks, dialogue act detection and key
term extraction. In the sequence labeling tasks, the model input is a sequence,
and the output is the label of the input sequence. The major difficulty of
sequence labeling is that when the input sequence is long, it can include many
noisy or irrelevant part. If the information in the whole sequence is treated
equally, the noisy or irrelevant part may degrade the classification
performance. The attention mechanism is helpful for sequence classification
task because it is capable of highlighting important part among the entire
sequence for the classification task. The experimental results show that with
the attention mechanism, discernible improvements were achieved in the sequence
labeling task considered here. The roles of the attention mechanism in the
tasks are further analyzed and visualized in this paper.
To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction
Authors: Wenbin Li, Seyedmajid Azimi, Aleš Leonardis, Mario Fritz
Date: 2016-03-31 00:00:00

Understanding physical phenomena is a key competence that enables humans and
animals to act and interact under uncertain perception in previously unseen
environments containing novel object and their configurations. Developmental
psychology has shown that such skills are acquired by infants from observations
at a very early stage.
  In this paper, we contrast a more traditional approach of taking a
model-based route with explicit 3D representations and physical simulation by
an end-to-end approach that directly predicts stability and related quantities
from appearance. We ask the question if and to what extent and quality such a
skill can directly be acquired in a data-driven way bypassing the need for an
explicit simulation.
  We present a learning-based approach based on simulated data that predicts
stability of towers comprised of wooden blocks under different conditions and
quantities related to the potential fall of the towers. The evaluation is
carried out on synthetic data and compared to human judgments on the same
stimuli.
Kernel Methods for the Approximation of Some Key Quantities of Nonlinear Systems
Authors: Jake Bouvrie, Boumediene Hamzi
Date: 2012-04-03 00:00:00

We introduce a data-based approach to estimating key quantities which arise
in the study of nonlinear control systems and random nonlinear dynamical
systems. Our approach hinges on the observation that much of the existing
linear theory may be readily extended to nonlinear systems - with a reasonable
expectation of success - once the nonlinear system has been mapped into a high
or infinite dimensional feature space. In particular, we develop computable,
non-parametric estimators approximating controllability and observability
energy functions for nonlinear systems, and study the ellipsoids they induce.
In all cases the relevant quantities are estimated from simulated or observed
data. It is then shown that the controllability energy estimator provides a key
means for approximating the invariant measure of an ergodic, stochastically
forced nonlinear system.
Kernel Methods for the Approximation of Nonlinear Systems
Authors: Jake Bouvrie, Boumediene Hamzi
Date: 2011-08-14 00:00:00
Tasks: BIG-bench Machine Learning, Dimensionality Reduction

We introduce a data-driven order reduction method for nonlinear control
systems, drawing on recent progress in machine learning and statistical
dimensionality reduction. The method rests on the assumption that the nonlinear
system behaves linearly when lifted into a high (or infinite) dimensional
feature space where balanced truncation may be carried out implicitly. This
leads to a nonlinear reduction map which can be combined with a representation
of the system belonging to a reproducing kernel Hilbert space to give a closed,
reduced order dynamical system which captures the essential input-output
characteristics of the original model. Empirical simulations illustrating the
approach are also provided.
Modeling Visual Compatibility through Hierarchical Mid-level Elements
Authors: Jose Oramas, Tinne Tuytelaars
Date: 2016-03-31 00:00:00
Tasks: Object

In this paper we present a hierarchical method to discover mid-level elements
with the objective of modeling visual compatibility between objects. At the
base-level, our method identifies patterns of CNN activations with the aim of
modeling different variations/styles in which objects of the classes of
interest may occur. At the top-level, the proposed method discovers patterns of
co-occurring activations of base-level elements that define visual
compatibility between pairs of object classes. Experiments on the massive
Amazon dataset show the strength of our method at describing object classes and
the characteristics that drive the compatibility between them.
Hierarchical Quickest Change Detection via Surrogates
Authors: Prithwish Chakraborty, Sathappan Muthiah, Ravi Tandon, Naren Ramakrishnan
Date: 2016-03-31 00:00:00
Tasks: Change Detection, Time Series, Time Series Analysis

Change detection (CD) in time series data is a critical problem as it reveal
changes in the underlying generative processes driving the time series. Despite
having received significant attention, one important unexplored aspect is how
to efficiently utilize additional correlated information to improve the
detection and the understanding of changepoints. We propose hierarchical
quickest change detection (HQCD), a framework that formalizes the process of
incorporating additional correlated sources for early changepoint detection.
The core ideas behind HQCD are rooted in the theory of quickest detection and
HQCD can be regarded as its novel generalization to a hierarchical setting. The
sources are classified into targets and surrogates, and HQCD leverages this
structure to systematically assimilate observed data to update changepoint
statistics across layers. The decision on actual changepoints are provided by
minimizing the delay while still maintaining reliability bounds. In addition,
HQCD also uncovers interesting relations between changes at targets from
changes across surrogates. We validate HQCD for reliability and performance
against several state-of-the-art methods for both synthetic dataset (known
changepoints) and several real-life examples (unknown changepoints). Our
experiments indicate that we gain significant robustness without loss of
detection delay through HQCD. Our real-life experiments also showcase the
usefulness of the hierarchical setting by connecting the surrogate sources
(such as Twitter chatter) to target sources (such as Employment related
protests that ultimately lead to major uprisings).
Design of false color palettes for grayscale reproduction
Authors: Filip A. Sala
Date: 2016-02-06 00:00:00

Design of false color palette is quite easy but some effort has to be done to
achieve good dynamic range, contrast and overall appearance of the palette.
Such palettes, for instance, are commonly used in scientific papers for
presenting the data. However, to lower the cost of the paper most scientists
decide to let the data to be printed in grayscale. The same applies to e-book
readers based on e-ink where most of them are still grayscale. For majority of
false color palettes reproducing them in grayscale results in ambiguous mapping
of the colors and may be misleading for the reader. In this article design of
false color palettes suitable for grayscale reproduction is described. Due to
the monotonic change of luminance of these palettes grayscale representation is
very similar to the data directly presented with a grayscale palette. Some
suggestions and examples how to design such palettes are provided.
Neural Language Correction with Character-Based Attention
Authors: Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, Andrew Y. Ng
Date: 2016-03-31 00:00:00
Tasks: Decoder, Language Modeling, Language Modelling, Machine Translation, Translation

Natural language correction has the potential to help language learners
improve their writing skills. While approaches with separate classifiers for
different error types have high precision, they do not flexibly handle errors
such as redundancy or non-idiomatic phrasing. On the other hand, word and
phrase-based machine translation methods are not designed to cope with
orthographic errors, and have recently been outpaced by neural models.
Motivated by these issues, we present a neural network-based approach to
language correction. The core component of our method is an encoder-decoder
recurrent neural network with an attention mechanism. By operating at the
character level, the network avoids the problem of out-of-vocabulary words. We
illustrate the flexibility of our approach on dataset of noisy, user-generated
text collected from an English learner forum. When combined with a language
model, our method achieves a state-of-the-art $F_{0.5}$-score on the CoNLL 2014
Shared Task. We further demonstrate that training the network on additional
data with synthesized errors can improve performance.
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
Authors: Shihao Ji, S. V. N. Vishwanathan, Nadathur Satish, Michael J. Anderson, Pradeep Dubey
Date: 2015-11-21 00:00:00
Tasks: CPU, Language Modeling, Language Modelling

We propose BlackOut, an approximation algorithm to efficiently train massive
recurrent neural network language models (RNNLMs) with million word
vocabularies. BlackOut is motivated by using a discriminative loss, and we
describe a new sampling strategy which significantly reduces computation while
improving stability, sample efficiency, and rate of convergence. One way to
understand BlackOut is to view it as an extension of the DropOut strategy to
the output layer, wherein we use a discriminative training loss and a weighted
sampling scheme. We also establish close connections between BlackOut,
importance sampling, and noise contrastive estimation (NCE). Our experiments,
on the recently released one billion word language modeling benchmark,
demonstrate scalability and accuracy of BlackOut; we outperform the
state-of-the art, and achieve the lowest perplexity scores on this dataset.
Moreover, unlike other established methods which typically require GPUs or CPU
clusters, we show that a carefully implemented version of BlackOut requires
only 1-10 days on a single machine to train a RNNLM with a million word
vocabulary and billions of parameters on one billion words. Although we
describe BlackOut in the context of RNNLM training, it can be used to any
networks with large softmax output layers.
Stochastic subGradient Methods with Linear Convergence for Polyhedral Convex Optimization
Authors: Tianbao Yang, Qihang Lin
Date: 2015-10-06 00:00:00
Tasks: BIG-bench Machine Learning

In this paper, we show that simple {Stochastic} subGradient Decent methods
with multiple Restarting, named {\bf RSGD}, can achieve a \textit{linear
convergence rate} for a class of non-smooth and non-strongly convex
optimization problems where the epigraph of the objective function is a
polyhedron, to which we refer as {\bf polyhedral convex optimization}. Its
applications in machine learning include $\ell_1$ constrained or regularized
piecewise linear loss minimization and submodular function minimization. To the
best of our knowledge, this is the first result on the linear convergence rate
of stochastic subgradient methods for non-smooth and non-strongly convex
optimization problems.
Multi-agent Reinforcement Learning with Sparse Interactions by Negotiation and Knowledge Transfer
Authors: Luowei Zhou, Pei Yang, Chunlin Chen, Yang Gao
Date: 2015-08-21 00:00:00
Tasks: Multi-agent Reinforcement Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Transfer Learning

Reinforcement learning has significant applications for multi-agent systems,
especially in unknown dynamic environments. However, most multi-agent
reinforcement learning (MARL) algorithms suffer from such problems as
exponential computation complexity in the joint state-action space, which makes
it difficult to scale up to realistic multi-agent problems. In this paper, a
novel algorithm named negotiation-based MARL with sparse interactions (NegoSI)
is presented. In contrast to traditional sparse-interaction based MARL
algorithms, NegoSI adopts the equilibrium concept and makes it possible for
agents to select the non-strict Equilibrium Dominating Strategy Profile
(non-strict EDSP) or Meta equilibrium for their joint actions. The presented
NegoSI algorithm consists of four parts: the equilibrium-based framework for
sparse interactions, the negotiation for the equilibrium set, the minimum
variance method for selecting one joint action and the knowledge transfer of
local Q-values. In this integrated algorithm, three techniques, i.e., unshared
value functions, equilibrium solutions and sparse interactions are adopted to
achieve privacy protection, better coordination and lower computational
complexity, respectively. To evaluate the performance of the presented NegoSI
algorithm, two groups of experiments are carried out regarding three criteria:
steps of each episode (SEE), rewards of each episode (REE) and average runtime
(AR). The first group of experiments is conducted using six grid world games
and shows fast convergence and high scalability of the presented algorithm.
Then in the second group of experiments NegoSI is applied to an intelligent
warehouse problem and simulated results demonstrate the effectiveness of the
presented NegoSI algorithm compared with other state-of-the-art MARL
algorithms.
Total Variation Applications in Computer Vision
Authors: Vania V. Estrela, Hermes Aguiar Magalhaes, Osamu Saotome
Date: 2016-03-31 00:00:00

The objectives of this chapter are: (i) to introduce a concise overview of
regularization; (ii) to define and to explain the role of a particular type of
regularization called total variation norm (TV-norm) in computer vision tasks;
(iii) to set up a brief discussion on the mathematical background of TV
methods; and (iv) to establish a relationship between models and a few existing
methods to solve problems cast as TV-norm. For the most part, image-processing
algorithms blur the edges of the estimated images, however TV regularization
preserves the edges with no prior information on the observed and the original
images. The regularization scalar parameter {\lambda} controls the amount of
regularization allowed and it is an essential to obtain a high-quality
regularized output. A wide-ranging review of several ways to put into practice
TV regularization as well as its advantages and limitations are discussed.
Contrastive Entropy: A new evaluation metric for unnormalized language models
Authors: Kushal Arora, Anand Rangarajan
Date: 2016-01-03 00:00:00
Tasks: Language Model Evaluation, Language Modeling, Language Modelling, Sentence

Perplexity (per word) is the most widely used metric for evaluating language
models. Despite this, there has been no dearth of criticism for this metric.
Most of these criticisms center around lack of correlation with extrinsic
metrics like word error rate (WER), dependence upon shared vocabulary for model
comparison and unsuitability for unnormalized language model evaluation. In
this paper, we address the last problem and propose a new discriminative
entropy based intrinsic metric that works for both traditional word level
models and unnormalized language models like sentence level models. We also
propose a discriminatively trained sentence level interpretation of recurrent
neural network based language model (RNN) as an example of unnormalized
sentence level model. We demonstrate that for word level models, contrastive
entropy shows a strong correlation with perplexity. We also observe that when
trained at lower distortion levels, sentence level RNN considerably outperforms
traditional RNNs on this new metric.