pwc-archive/papers-with-abstracts huggingface dataset:
Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
Authors: Anh Nguyen, Jason Yosinski, Jeff Clune
Date: 2016-02-11 00:00:00

We can better understand deep neural networks by identifying which features
each of their neurons have learned to detect. To do so, researchers have
created Deep Visualization techniques including activation maximization, which
synthetically generates inputs (e.g. images) that maximally activate each
neuron. A limitation of current techniques is that they assume each neuron
detects only one type of feature, but we know that neurons can be multifaceted,
in that they fire in response to many different types of features: for example,
a grocery store class neuron must activate either for rows of produce or for a
storefront. Previous activation maximization techniques constructed images
without regard for the multiple different facets of a neuron, creating
inappropriate mixes of colors, parts of objects, scales, orientations, etc.
Here, we introduce an algorithm that explicitly uncovers the multiple facets of
each neuron by producing a synthetic visualization of each of the types of
images that activate a neuron. We also introduce regularization methods that
produce state-of-the-art results in terms of the interpretability of images
obtained by activation maximization. By separately synthesizing each type of
image a neuron fires in response to, the visualizations have more appropriate
colors and coherent global structure. Multifaceted feature visualization thus
provides a clearer and more comprehensive description of the role of each
neuron.
Matrix Factorization-Based Clustering Of Image Features For Bandwidth-Constrained Information Retrieval
Authors: Jacob Chakareski, Immanuel Manohar, Shantanu Rane
Date: 2016-05-07 00:00:00
Tasks: Clustering, Information Retrieval, Retrieval, Translation

We consider the problem of accurately and efficiently querying a remote
server to retrieve information about images captured by a mobile device. In
addition to reduced transmission overhead and computational complexity, the
retrieval protocol should be robust to variations in the image acquisition
process, such as translation, rotation, scaling, and sensor-related
differences. We propose to extract scale-invariant image features and then
perform clustering to reduce the number of features needed for image matching.
Principal Component Analysis (PCA) and Non-negative Matrix Factorization (NMF)
are investigated as candidate clustering approaches. The image matching
complexity at the database server is quadratic in the (small) number of
clusters, not in the (very large) number of image features. We employ an
image-dependent information content metric to approximate the model order,
i.e., the number of clusters, needed for accurate matching, which is preferable
to setting the model order using trial and error. We show how to combine the
hypotheses provided by PCA and NMF factor loadings, thereby obtaining more
accurate retrieval than using either approach alone. In experiments on a
database of urban images, we obtain a top-1 retrieval accuracy of 89% and a
top-3 accuracy of 92.5%.
Neural Recovery Machine for Chinese Dropped Pronoun
Authors: Wei-Nan Zhang, Ting Liu, Qingyu Yin, Yu Zhang
Date: 2016-05-07 00:00:00
Tasks: Feature Engineering

Dropped pronouns (DPs) are ubiquitous in pro-drop languages like Chinese, Japanese etc. Previous work mainly focused on painstakingly exploring the empirical features for DPs recovery. In this paper, we propose a neural recovery machine (NRM) to model and recover DPs in Chinese, so that to avoid the non-trivial feature engineering process. The experimental results show that the proposed NRM significantly outperforms the state-of-the-art approaches on both two heterogeneous datasets. Further experiment results of Chinese zero pronoun (ZP) resolution show that the performance of ZP resolution can also be improved by recovering the ZPs to DPs.
Robust Dialog State Tracking for Large Ontologies
Authors: Franck Dernoncourt, Ji Young Lee, Trung H. Bui, Hung H. Bui
Date: 2016-05-07 00:00:00
Tasks: coreference-resolution, Coreference Resolution, dialog state tracking, Spoken Language Understanding

The Dialog State Tracking Challenge 4 (DSTC 4) differentiates itself from the
previous three editions as follows: the number of slot-value pairs present in
the ontology is much larger, no spoken language understanding output is given,
and utterances are labeled at the subdialog level. This paper describes a novel
dialog state tracking method designed to work robustly under these conditions,
using elaborate string matching, coreference resolution tailored for dialogs
and a few other improvements. The method can correctly identify many values
that are not explicitly present in the utterance. On the final evaluation, our
method came in first among 7 competing teams and 24 entries. The F1-score
achieved by our method was 9 and 7 percentage points higher than that of the
runner-up for the utterance-level evaluation and for the subdialog-level
evaluation, respectively.
Adobe-MIT submission to the DSTC 4 Spoken Language Understanding pilot task
Authors: Franck Dernoncourt, Ji Young Lee, Trung H. Bui, Hung H. Bui
Date: 2016-05-07 00:00:00
Tasks: dialog state tracking, Spoken Language Understanding

The Dialog State Tracking Challenge 4 (DSTC 4) proposes several pilot tasks.
In this paper, we focus on the spoken language understanding pilot task, which
consists of tagging a given utterance with speech acts and semantic slots. We
compare different classifiers: the best system obtains 0.52 and 0.67 F1-scores
on the test set for speech act recognition for the tourist and the guide
respectively, and 0.52 F1-score for semantic tagging for both the guide and the
tourist.
Clustering on Multiple Incomplete Datasets via Collective Kernel Learning
Authors: Weixiang Shao, Xiaoxiao Shi, Philip S. Yu
Date: 2013-10-04 00:00:00
Tasks: Clustering, Recommendation Systems

Multiple datasets containing different types of features may be available for
a given task. For instance, users' profiles can be used to group users for
recommendation systems. In addition, a model can also use users' historical
behaviors and credit history to group users. Each dataset contains different
information and suffices for learning. A number of clustering algorithms on
multiple datasets were proposed during the past few years. These algorithms
assume that at least one dataset is complete. So far as we know, all the
previous methods will not be applicable if there is no complete dataset
available. However, in reality, there are many situations where no dataset is
complete. As in building a recommendation system, some new users may not have a
profile or historical behaviors, while some may not have a credit history.
Hence, no available dataset is complete. In order to solve this problem, we
propose an approach called Collective Kernel Learning to infer hidden sample
similarity from multiple incomplete datasets. The idea is to collectively
completes the kernel matrices of incomplete datasets by optimizing the
alignment of the shared instances of the datasets. Furthermore, a clustering
algorithm is proposed based on the kernel matrix. The experiments on both
synthetic and real datasets demonstrate the effectiveness of the proposed
approach. The proposed clustering algorithm outperforms the comparison
algorithms by as much as two times in normalized mutual information.
Distributed Learning with Infinitely Many Hypotheses
Authors: Angelia Nedić, Alex Olshevsky, César Uribe
Date: 2016-05-06 00:00:00

We consider a distributed learning setup where a network of agents
sequentially access realizations of a set of random variables with unknown
distributions. The network objective is to find a parametrized distribution
that best describes their joint observations in the sense of the
Kullback-Leibler divergence. Apart from recent efforts in the literature, we
analyze the case of countably many hypotheses and the case of a continuum of
hypotheses. We provide non-asymptotic bounds for the concentration rate of the
agents' beliefs around the correct hypothesis in terms of the number of agents,
the network parameters, and the learning abilities of the agents. Additionally,
we provide a novel motivation for a general set of distributed Non-Bayesian
update rules as instances of the distributed stochastic mirror descent
algorithm.
Sequence Level Training with Recurrent Neural Networks
Authors: Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba
Date: 2015-11-20 00:00:00
Tasks: Machine Translation

Many natural language processing applications use language models to generate
text. These models are typically trained to predict the next word in a
sequence, given the previous words and some context such as an image. However,
at test time the model is expected to generate the entire sequence from
scratch. This discrepancy makes generation brittle, as errors may accumulate
along the way. We address this issue by proposing a novel sequence level
training algorithm that directly optimizes the metric used at test time, such
as BLEU or ROUGE. On three different tasks, our approach outperforms several
strong baselines for greedy generation. The method is also competitive when
these baselines employ beam search, while being several times faster.
Towards Automated Melanoma Screening: Proper Computer Vision & Reliable Results
Authors: Michel Fornaciali, Micael Carvalho, Flávia Vasques Bittencourt, Sandra Avila, Eduardo Valle
Date: 2016-04-14 00:00:00
Tasks: Melanoma Diagnosis

In this paper we survey, analyze and criticize current art on automated
melanoma screening, reimplementing a baseline technique, and proposing two
novel ones. Melanoma, although highly curable when detected early, ends as one
of the most dangerous types of cancer, due to delayed diagnosis and treatment.
Its incidence is soaring, much faster than the number of trained professionals
able to diagnose it. Automated screening appears as an alternative to make the
most of those professionals, focusing their time on the patients at risk while
safely discharging the other patients. However, the potential of automated
melanoma diagnosis is currently unfulfilled, due to the emphasis of current
literature on outdated computer vision models. Even more problematic is the
irreproducibility of current art. We show how streamlined pipelines based upon
current Computer Vision outperform conventional models - a model based on an
advanced bags of words reaches an AUC of 84.6%, and a model based on deep
neural networks reaches 89.3%, while the baseline (a classical bag of words)
stays at 81.2%. We also initiate a dialog to improve reproducibility in our
community
Some Simulation Results for Emphatic Temporal-Difference Learning Algorithms
Authors: Huizhen Yu
Date: 2016-05-06 00:00:00

This is a companion note to our recent study of the weak convergence
properties of constrained emphatic temporal-difference learning (ETD)
algorithms from a theoretic perspective. It supplements the latter analysis
with simulation results and illustrates the behavior of some of the ETD
algorithms using three example problems.
Stochastic Contextual Bandits with Known Reward Functions
Authors: Pranav Sakulkar, Bhaskar Krishnamachari
Date: 2016-04-30 00:00:00
Tasks: Decision Making, Multi-Armed Bandits, Sequential Decision Making

Many sequential decision-making problems in communication networks can be
modeled as contextual bandit problems, which are natural extensions of the
well-known multi-armed bandit problem. In contextual bandit problems, at each
time, an agent observes some side information or context, pulls one arm and
receives the reward for that arm. We consider a stochastic formulation where
the context-reward tuples are independently drawn from an unknown distribution
in each trial. Motivated by networking applications, we analyze a setting where
the reward is a known non-linear function of the context and the chosen arm's
current state. We first consider the case of discrete and finite context-spaces
and propose DCB($\epsilon$), an algorithm that we prove, through a careful
analysis, yields regret (cumulative reward gap compared to a distribution-aware
genie) scaling logarithmically in time and linearly in the number of arms that
are not optimal for any context, improving over existing algorithms where the
regret scales linearly in the total number of arms. We then study continuous
context-spaces with Lipschitz reward functions and propose CCB($\epsilon,
\delta$), an algorithm that uses DCB($\epsilon$) as a subroutine.
CCB($\epsilon, \delta$) reveals a novel regret-storage trade-off that is
parametrized by $\delta$. Tuning $\delta$ to the time horizon allows us to
obtain sub-linear regret bounds, while requiring sub-linear storage. By
exploiting joint learning for all contexts we get regret bounds for
CCB($\epsilon, \delta$) that are unachievable by any existing contextual bandit
algorithm for continuous context-spaces. We also show similar performance
bounds for the unknown horizon case.
The Actias system: supervised multi-strategy learning paradigm using categorical logic
Authors: Carlos Leandro, Helder Pita, Luís Monteiro
Date: 2016-05-06 00:00:00

One of the most difficult problems in the development of intelligent systems
is the construction of the underlying knowledge base. As a consequence, the
rate of progress in the development of this type of system is directly related
to the speed with which knowledge bases can be assembled, and on its quality.
We attempt to solve the knowledge acquisition problem, for a Business
Information System, developing a supervised multistrategy learning paradigm.
This paradigm is centred on a collaborative data mining strategy, where groups
of experts collaborate using data-mining process on the supervised acquisition
of new knowledge extracted from heterogeneous machine learning data models.
  The Actias system is our approach to this paradigm. It is the result of
applying the graphic logic based language of sketches to knowledge integration.
The system is a data mining collaborative workplace, where the Information
System knowledge base is an algebraic structure. It results from the
integration of background knowledge with new insights extracted from data
models, generated for specific data modelling tasks, and represented as rules
using the sketches language.
Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds
Authors: Mark Bun, Thomas Steinke
Date: 2016-05-06 00:00:00
Tasks: Privacy Preserving

"Concentrated differential privacy" was recently introduced by Dwork and
Rothblum as a relaxation of differential privacy, which permits sharper
analyses of many privacy-preserving computations. We present an alternative
formulation of the concept of concentrated differential privacy in terms of the
Renyi divergence between the distributions obtained by running an algorithm on
neighboring inputs. With this reformulation in hand, we prove sharper
quantitative results, establish lower bounds, and raise a few new questions. We
also unify this approach with approximate differential privacy by giving an
appropriate definition of "approximate concentrated differential privacy."
Deformably Registering and Annotating Whole CLARITY Brains to an Atlas via Masked LDDMM
Authors: Kwame S. Kutten, Joshua T. Vogelstein, Nicolas Charon, Li Ye, Karl Deisseroth, Michael I. Miller
Date: 2016-05-06 00:00:00

The CLARITY method renders brains optically transparent to enable
high-resolution imaging in the structurally intact brain. Anatomically
annotating CLARITY brains is necessary for discovering which regions contain
signals of interest. Manually annotating whole-brain, terabyte CLARITY images
is difficult, time-consuming, subjective, and error-prone. Automatically
registering CLARITY images to a pre-annotated brain atlas offers a solution,
but is difficult for several reasons. Removal of the brain from the skull and
subsequent storage and processing cause variable non-rigid deformations, thus
compounding inter-subject anatomical variability. Additionally, the signal in
CLARITY images arises from various biochemical contrast agents which only
sparsely label brain structures. This sparse labeling challenges the most
commonly used registration algorithms that need to match image histogram
statistics to the more densely labeled histological brain atlases. The standard
method is a multiscale Mutual Information B-spline algorithm that dynamically
generates an average template as an intermediate registration target. We
determined that this method performs poorly when registering CLARITY brains to
the Allen Institute's Mouse Reference Atlas (ARA), because the image histogram
statistics are poorly matched. Therefore, we developed a method (Mask-LDDMM)
for registering CLARITY images, that automatically find the brain boundary and
learns the optimal deformation between the brain and atlas masks. Using
Mask-LDDMM without an average template provided better results than the
standard approach when registering CLARITY brains to the ARA. The LDDMM
pipelines developed here provide a fast automated way to anatomically annotate
CLARITY images. Our code is available as open source software at
http://NeuroData.io.
Low-Complexity Stochastic Generalized Belief Propagation
Authors: Farzin Haddadpour, Mahdi Jafari Siavoshani, Morteza Noshad
Date: 2016-05-06 00:00:00

The generalized belief propagation (GBP), introduced by Yedidia et al., is an
extension of the belief propagation (BP) algorithm, which is widely used in
different problems involved in calculating exact or approximate marginals of
probability distributions. In many problems, it has been observed that the
accuracy of GBP considerably outperforms that of BP. However, because in
general the computational complexity of GBP is higher than BP, its application
is limited in practice.
  In this paper, we introduce a stochastic version of GBP called stochastic
generalized belief propagation (SGBP) that can be considered as an extension to
the stochastic BP (SBP) algorithm introduced by Noorshams et al. They have
shown that SBP reduces the complexity per iteration of BP by an order of
magnitude in alphabet size. In contrast to SBP, SGBP can reduce the computation
complexity if certain topological conditions are met by the region graph
associated to a graphical model. However, this reduction can be larger than
only one order of magnitude in alphabet size. In this paper, we characterize
these conditions and the amount of computation gain that we can obtain by using
SGBP. Finally, using similar proof techniques employed by Noorshams et al., for
general graphical models satisfy contraction conditions, we prove the
asymptotic convergence of SGBP to the unique GBP fixed point, as well as
providing non-asymptotic upper bounds on the mean square error and on the high
probability error.
Regret Analysis of the Anytime Optimally Confident UCB Algorithm
Authors: Tor Lattimore
Date: 2016-03-29 00:00:00

I introduce and analyse an anytime version of the Optimally Confident UCB
(OCUCB) algorithm designed for minimising the cumulative regret in finite-armed
stochastic bandits with subgaussian noise. The new algorithm is simple,
intuitive (in hindsight) and comes with the strongest finite-time regret
guarantees for a horizon-free algorithm so far. I also show a finite-time lower
bound that nearly matches the upper bound.
Training Neural Networks Without Gradients: A Scalable ADMM Approach
Authors: Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, Tom Goldstein
Date: 2016-05-06 00:00:00

With the growing importance of large network models and enormous training
datasets, GPUs have become increasingly necessary to train neural networks.
This is largely because conventional optimization algorithms rely on stochastic
gradient methods that don't scale well to large numbers of cores in a cluster
setting. Furthermore, the convergence of all gradient methods, including batch
methods, suffers from common problems like saturation effects, poor
conditioning, and saddle points. This paper explores an unconventional training
method that uses alternating direction methods and Bregman iteration to train
networks without gradient descent steps. The proposed method reduces the
network training problem to a sequence of minimization sub-steps that can each
be solved globally in closed form. The proposed method is advantageous because
it avoids many of the caveats that make gradient methods slow on highly
non-convex problems. The method exhibits strong scaling in the distributed
setting, yielding linear speedups even when split over thousands of cores.
Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec
Authors: Christopher E Moody
Date: 2016-05-06 00:00:00
Tasks: Topic Models, Word Embeddings

Distributed dense word vectors have been shown to be effective at capturing
token-level semantic and syntactic regularities in language, while topic models
can form interpretable representations over documents. In this work, we
describe lda2vec, a model that learns dense word vectors jointly with
Dirichlet-distributed latent document-level mixtures of topic vectors. In
contrast to continuous dense document representations, this formulation
produces sparse, interpretable document mixtures through a non-negative simplex
constraint. Our method is simple to incorporate into existing automatic
differentiation frameworks and allows for unsupervised document representations
geared for use by scientists while simultaneously learning word vectors and the
linear relationships between them.
Device and System Level Design Considerations for Analog-Non-Volatile-Memory Based Neuromorphic Architectures
Authors: Sukru Burc Eryilmaz, Duygu Kuzum, Shimeng Yu, H. -S. Philip Wong
Date: 2015-12-25 00:00:00

This paper gives an overview of recent progress in the brain inspired
computing field with a focus on implementation using emerging memories as
electronic synapses. Design considerations and challenges such as requirements
and design targets on multilevel states, device variability, programming
energy, array-level connectivity, fan-in/fanout, wire energy, and IR drop are
presented. Wires are increasingly important in design decisions, especially for
large systems, and cycle-to-cycle variations have large impact on learning
performance.
Visual Saliency Based on Scale-Space Analysis in the Frequency Domain
Authors: Jian Li, Martin Levine, Xiangjing An, Xin Xu, Hangen He
Date: 2016-05-06 00:00:00
Tasks: Saliency Detection, Video Saliency Detection

We address the issue of visual saliency from three perspectives. First, we
consider saliency detection as a frequency domain analysis problem. Second, we
achieve this by employing the concept of {\it non-saliency}. Third, we
simultaneously consider the detection of salient regions of different size. The
paper proposes a new bottom-up paradigm for detecting visual saliency,
characterized by a scale-space analysis of the amplitude spectrum of natural
images. We show that the convolution of the {\it image amplitude spectrum} with
a low-pass Gaussian kernel of an appropriate scale is equivalent to such an
image saliency detector. The saliency map is obtained by reconstructing the 2-D
signal using the original phase and the amplitude spectrum, filtered at a scale
selected by minimizing saliency map entropy. A Hypercomplex Fourier Transform
performs the analysis in the frequency domain. Using available databases, we
demonstrate experimentally that the proposed model can predict human fixation
data. We also introduce a new image database and use it to show that the
saliency detector can highlight both small and large salient regions, as well
as inhibit repeated distractors in cluttered images. In addition, we show that
it is able to predict salient regions on which people focus their attention.