pwc-archive/papers-with-abstracts huggingface dataset:
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Authors: Sergey Ioffe, Christian Szegedy
Date: 2015-02-11 00:00:00
Tasks: General Classification, image-classification, Image Classification

Training Deep Neural Networks is complicated by the fact that the
distribution of each layer's inputs changes during training, as the parameters
of the previous layers change. This slows down the training by requiring lower
learning rates and careful parameter initialization, and makes it notoriously
hard to train models with saturating nonlinearities. We refer to this
phenomenon as internal covariate shift, and address the problem by normalizing
layer inputs. Our method draws its strength from making normalization a part of
the model architecture and performing the normalization for each training
mini-batch. Batch Normalization allows us to use much higher learning rates and
be less careful about initialization. It also acts as a regularizer, in some
cases eliminating the need for Dropout. Applied to a state-of-the-art image
classification model, Batch Normalization achieves the same accuracy with 14
times fewer training steps, and beats the original model by a significant
margin. Using an ensemble of batch-normalized networks, we improve upon the
best published result on ImageNet classification: reaching 4.9% top-5
validation error (and 4.8% test error), exceeding the accuracy of human raters.
Bayesian Optimization of Text Representations
Authors: Dani Yogatama, Noah A. Smith
Date: 2015-03-02 00:00:00
Proceeding: EMNLP 2015 9
Tasks: Bayesian Optimization, General Classification, global-optimization, Sentiment Analysis, Topic Classification

When applying machine learning to problems in NLP, there are many choices to
make about how to represent input texts. These choices can have a big effect on
performance, but they are often uninteresting to researchers or practitioners
who simply need a module that performs well. We propose an approach to
optimizing over this space of choices, formulating the problem as global
optimization. We apply a sequential model-based optimization technique and show
that our method makes standard linear models competitive with more
sophisticated, expensive state-of-the-art methods based on latent variable
models or neural networks on various topic classification and sentiment
analysis problems. Our approach is a first step towards black-box NLP systems
that work with raw text and do not require manual tuning.
A review of mean-shift algorithms for clustering
Authors: Miguel Á. Carreira-Perpiñán
Date: 2015-03-02 00:00:00
Tasks: Clustering, Denoising, Image Segmentation, Semantic Segmentation

A natural way to characterize the cluster structure of a dataset is by
finding regions containing a high density of data. This can be done in a
nonparametric way with a kernel density estimate, whose modes and hence
clusters can be found using mean-shift algorithms. We describe the theory and
practice behind clustering based on kernel density estimates and mean-shift
algorithms. We discuss the blurring and non-blurring versions of mean-shift;
theoretical results about mean-shift algorithms and Gaussian mixtures;
relations with scale-space theory, spectral clustering and other algorithms;
extensions to tracking, to manifold and graph data, and to manifold denoising;
K-modes and Laplacian K-modes algorithms; acceleration strategies for large
datasets; and applications to image segmentation, manifold denoising and
multivalued regression.
A Hebbian/Anti-Hebbian Network Derived from Online Non-Negative Matrix Factorization Can Cluster and Discover Sparse Features
Authors: Cengiz Pehlevan, Dmitri B. Chklovskii
Date: 2015-03-02 00:00:00
Tasks: Anatomy, Clustering

Despite our extensive knowledge of biophysical properties of neurons, there
is no commonly accepted algorithmic theory of neuronal function. Here we
explore the hypothesis that single-layer neuronal networks perform online
symmetric nonnegative matrix factorization (SNMF) of the similarity matrix of
the streamed data. By starting with the SNMF cost function we derive an online
algorithm, which can be implemented by a biologically plausible network with
local learning rules. We demonstrate that such network performs soft clustering
of the data as well as sparse feature discovery. The derived algorithm
replicates many known aspects of sensory anatomy and biophysical properties of
neurons including unipolar nature of neuronal activity and synaptic weights,
local synaptic plasticity rules and the dependence of learning rate on
cumulative neuronal activity. Thus, we make a step towards an algorithmic
theory of neuronal function, which should facilitate large-scale neural circuit
simulations and biologically inspired artificial intelligence.
A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data
Authors: Cengiz Pehlevan, Tao Hu, Dmitri B. Chklovskii
Date: 2015-03-02 00:00:00

Neural network models of early sensory processing typically reduce the
dimensionality of streaming input data. Such networks learn the principal
subspace, in the sense of principal component analysis (PCA), by adjusting
synaptic weights according to activity-dependent learning rules. When derived
from a principled cost function these rules are nonlocal and hence biologically
implausible. At the same time, biologically plausible local rules have been
postulated rather than derived from a principled cost function. Here, to bridge
this gap, we derive a biologically plausible network for subspace learning on
streaming data by minimizing a principled cost function. In a departure from
previous work, where cost was quantified by the representation, or
reconstruction, error, we adopt a multidimensional scaling (MDS) cost function
for streaming data. The resulting algorithm relies only on biologically
plausible Hebbian and anti-Hebbian local learning rules. In a stochastic
setting, synaptic weights converge to a stationary state which projects the
input data onto the principal subspace. If the data are generated by a
nonstationary distribution, the network can track the principal subspace. Thus,
our result makes a step towards an algorithmic theory of neural computation.
An $\mathcal{O}(n\log n)$ projection operator for weighted $\ell_1$-norm regularization with sum constraint
Authors: Weiran Wang
Date: 2015-03-02 00:00:00

We provide a simple and efficient algorithm for the projection operator for
weighted $\ell_1$-norm regularization subject to a sum constraint, together
with an elementary proof. The implementation of the proposed algorithm can be
downloaded from the author's homepage.
Deep Transfer Network: Unsupervised Domain Adaptation
Authors: Xu Zhang, Felix Xinnan Yu, Shih-Fu Chang, Shengjin Wang
Date: 2015-03-02 00:00:00
Tasks: Domain Adaptation, Unsupervised Domain Adaptation

Domain adaptation aims at training a classifier in one dataset and applying
it to a related but not identical dataset. One successfully used framework of
domain adaptation is to learn a transformation to match both the distribution
of the features (marginal distribution), and the distribution of the labels
given features (conditional distribution). In this paper, we propose a new
domain adaptation framework named Deep Transfer Network (DTN), where the highly
flexible deep neural networks are used to implement such a distribution
matching process.
  This is achieved by two types of layers in DTN: the shared feature extraction
layers which learn a shared feature subspace in which the marginal
distributions of the source and the target samples are drawn close, and the
discrimination layers which match conditional distributions by classifier
transduction. We also show that DTN has a computation complexity linear to the
number of training samples, making it suitable to large-scale problems. By
combining the best paradigms in both worlds (deep neural networks in
recognition, and matching marginal and conditional distributions in domain
adaptation), we demonstrate by extensive experiments that DTN improves
significantly over former methods in both execution time and classification
accuracy.
Recovering PCA from Hybrid-$(\ell_1,\ell_2)$ Sparse Sampling of Data Elements
Authors: Abhisek Kundu, Petros Drineas, Malik Magdon-Ismail
Date: 2015-03-02 00:00:00

This paper addresses how well we can recover a data matrix when only given a
few of its elements. We present a randomized algorithm that element-wise
sparsifies the data, retaining only a few its elements. Our new algorithm
independently samples the data using sampling probabilities that depend on both
the squares ($\ell_2$ sampling) and absolute values ($\ell_1$ sampling) of the
entries. We prove that the hybrid algorithm recovers a near-PCA reconstruction
of the data from a sublinear sample-size: hybrid-($\ell_1,\ell_2$) inherits the
$\ell_2$-ability to sample the important elements as well as the regularization
properties of $\ell_1$ sampling, and gives strictly better performance than
either $\ell_1$ or $\ell_2$ on their own. We also give a one-pass version of
our algorithm and show experiments to corroborate the theory.
Data granulation by the principles of uncertainty
Authors: Lorenzo Livi, Alireza Sadeghian
Date: 2014-07-26 00:00:00

Researches in granular modeling produced a variety of mathematical models,
such as intervals, (higher-order) fuzzy sets, rough sets, and shadowed sets,
which are all suitable to characterize the so-called information granules.
Modeling of the input data uncertainty is recognized as a crucial aspect in
information granulation. Moreover, the uncertainty is a well-studied concept in
many mathematical settings, such as those of probability theory, fuzzy set
theory, and possibility theory. This fact suggests that an appropriate
quantification of the uncertainty expressed by the information granule model
could be used to define an invariant property, to be exploited in practical
situations of information granulation. In this perspective, a procedure of
information granulation is effective if the uncertainty conveyed by the
synthesized information granule is in a monotonically increasing relation with
the uncertainty of the input data. In this paper, we present a data granulation
framework that elaborates over the principles of uncertainty introduced by
Klir. Being the uncertainty a mesoscopic descriptor of systems and data, it is
possible to apply such principles regardless of the input data type and the
specific mathematical setting adopted for the information granules. The
proposed framework is conceived (i) to offer a guideline for the synthesis of
information granules and (ii) to build a groundwork to compare and
quantitatively judge over different data granulation procedures. To provide a
suitable case study, we introduce a new data granulation technique based on the
minimum sum of distances, which is designed to generate type-2 fuzzy sets. We
analyze the procedure by performing different experiments on two distinct data
types: feature vectors and labeled graphs. Results show that the uncertainty of
the input data is suitably conveyed by the generated type-2 fuzzy set models.
A neuromorphic hardware framework based on population coding
Authors: Chetan Singh Thakur, Tara Julia Hamilton, Runchun Wang, Jonathan Tapson, André van Schaik
Date: 2015-03-02 00:00:00

In the biological nervous system, large neuronal populations work
collaboratively to encode sensory stimuli. These neuronal populations are
characterised by a diverse distribution of tuning curves, ensuring that the
entire range of input stimuli is encoded. Based on these principles, we have
designed a neuromorphic system called a Trainable Analogue Block (TAB), which
encodes given input stimuli using a large population of neurons with a
heterogeneous tuning curve profile. Heterogeneity of tuning curves is achieved
using random device mismatches in VLSI (Very Large Scale Integration) process
and by adding a systematic offset to each hidden neuron. Here, we present
measurement results of a single test cell fabricated in a 65nm technology to
verify the TAB framework. We have mimicked a large population of neurons by
re-using measurement results from the test cell by varying offset. We thus
demonstrate the learning capability of the system for various regression tasks.
The TAB system may pave the way to improve the design of analogue circuits for
commercial applications, by rendering circuits insensitive to random mismatch
that arises due to the manufacturing process.
FPGA Implementation of the CAR Model of the Cochlea
Authors: Chetan Singh Thakur, Tara Julia Hamilton, Jonathan Tapson, Richard F. Lyon, André van Schaik
Date: 2015-03-02 00:00:00

The front end of the human auditory system, the cochlea, converts sound
signals from the outside world into neural impulses transmitted along the
auditory pathway for further processing. The cochlea senses and separates sound
in a nonlinear active fashion, exhibiting remarkable sensitivity and frequency
discrimination. Although several electronic models of the cochlea have been
proposed and implemented, none of these are able to reproduce all the
characteristics of the cochlea, including large dynamic range, large gain and
sharp tuning at low sound levels, and low gain and broad tuning at intense
sound levels. Here, we implement the Cascade of Asymmetric Resonators (CAR)
model of the cochlea on an FPGA. CAR represents the basilar membrane filter in
the Cascade of Asymmetric Resonators with Fast-Acting Compression (CAR-FAC)
cochlear model. CAR-FAC is a neuromorphic model of hearing based on a pole-zero
filter cascade model of auditory filtering. It uses simple nonlinear extensions
of conventional digital filter stages that are well suited to FPGA
implementations, so that we are able to implement up to 1224 cochlear sections
on Virtex-6 FPGA to process sound data in real time. The FPGA implementation of
the electronic cochlea described here may be used as a front-end sound analyser
for various machine-hearing applications.
Utility-Theoretic Ranking for Semi-Automated Text Classification
Authors: Giacomo Berardi, Andrea Esuli, Fabrizio Sebastiani
Date: 2015-03-02 00:00:00
Tasks: Classification, General Classification, text-classification, Text Classification

\emph{Semi-Automated Text Classification} (SATC) may be defined as the task
of ranking a set $\mathcal{D}$ of automatically labelled textual documents in
such a way that, if a human annotator validates (i.e., inspects and corrects
where appropriate) the documents in a top-ranked portion of $\mathcal{D}$ with
the goal of increasing the overall labelling accuracy of $\mathcal{D}$, the
expected increase is maximized. An obvious SATC strategy is to rank
$\mathcal{D}$ so that the documents that the classifier has labelled with the
lowest confidence are top-ranked. In this work we show that this strategy is
suboptimal. We develop new utility-theoretic ranking methods based on the
notion of \emph{validation gain}, defined as the improvement in classification
effectiveness that would derive by validating a given automatically labelled
document. We also propose a new effectiveness measure for SATC-oriented ranking
methods, based on the expected reduction in classification error brought about
by partially validating a list generated by a given ranking method. We report
the results of experiments showing that, with respect to the baseline method
above, and according to the proposed measure, our utility-theoretic ranking
methods can achieve substantially higher expected reductions in classification
error.
Optimality of Poisson processes intensity learning with Gaussian processes
Authors: Alisa Kirichenko, Harry van Zanten
Date: 2014-09-17 00:00:00
Tasks: Gaussian Processes

In this paper we provide theoretical support for the so-called "Sigmoidal
Gaussian Cox Process" approach to learning the intensity of an inhomogeneous
Poisson process on a $d$-dimensional domain. This method was proposed by Adams,
Murray and MacKay (ICML, 2009), who developed a tractable computational
approach and showed in simulation and real data experiments that it can work
quite satisfactorily. The results presented in the present paper provide
theoretical underpinning of the method. In particular, we show how to tune the
priors on the hyper parameters of the model in order for the procedure to
automatically adapt to the degree of smoothness of the unknown intensity and to
achieve optimal convergence rates.
Signal inference with unknown response: Calibration-uncertainty renormalized estimator
Authors: Sebastian Dorn, Torsten A. Enßlin, Maksim Greiner, Marco Selig, Vanessa Boehm
Date: 2014-10-23 00:00:00

The calibration of a measurement device is crucial for every scientific
experiment, where a signal has to be inferred from data. We present CURE, the
calibration uncertainty renormalized estimator, to reconstruct a signal and
simultaneously the instrument's calibration from the same data without knowing
the exact calibration, but its covariance structure. The idea of CURE,
developed in the framework of information field theory, is starting with an
assumed calibration to successively include more and more portions of
calibration uncertainty into the signal inference equations and to absorb the
resulting corrections into renormalized signal (and calibration) solutions.
Thereby, the signal inference and calibration problem turns into solving a
single system of ordinary differential equations and can be identified with
common resummation techniques used in field theories. We verify CURE by
applying it to a simplistic toy example and compare it against existent
self-calibration schemes, Wiener filter solutions, and Markov Chain Monte Carlo
sampling. We conclude that the method is able to keep up in accuracy with the
best self-calibration methods and serves as a non-iterative alternative to it.
Block stochastic gradient iteration for convex and nonconvex optimization
Authors: Yangyang Xu, Wotao Yin
Date: 2014-08-12 00:00:00
Tasks: regression, Stochastic Optimization

The stochastic gradient (SG) method can minimize an objective function
composed of a large number of differentiable functions, or solve a stochastic
optimization problem, to a moderate accuracy. The block coordinate
descent/update (BCD) method, on the other hand, handles problems with multiple
blocks of variables by updating them one at a time; when the blocks of
variables are easier to update individually than together, BCD has a lower
per-iteration cost. This paper introduces a method that combines the features
of SG and BCD for problems with many components in the objective and with
multiple (blocks of) variables.
  Specifically, a block stochastic gradient (BSG) method is proposed for
solving both convex and nonconvex programs. At each iteration, BSG approximates
the gradient of the differentiable part of the objective by randomly sampling a
small set of data or sampling a few functions from the sum term in the
objective, and then, using those samples, it updates all the blocks of
variables in either a deterministic or a randomly shuffled order. Its
convergence for both convex and nonconvex cases are established in different
senses. In the convex case, the proposed method has the same order of
convergence rate as the SG method. In the nonconvex case, its convergence is
established in terms of the expected violation of a first-order optimality
condition. The proposed method was numerically tested on problems including
stochastic least squares and logistic regression, which are convex, as well as
low-rank tensor recovery and bilinear logistic regression, which are nonconvex.
Phase Transitions in Sparse PCA
Authors: Thibault Lesieur, Florent Krzakala, Lenka Zdeborova
Date: 2015-03-01 00:00:00

We study optimal estimation for sparse principal component analysis when the
number of non-zero elements is small but on the same order as the dimension of
the data. We employ approximate message passing (AMP) algorithm and its state
evolution to analyze what is the information theoretically minimal mean-squared
error and the one achieved by AMP in the limit of large sizes. For a special
case of rank one and large enough density of non-zeros Deshpande and Montanari
[1] proved that AMP is asymptotically optimal. We show that both for low
density and for large rank the problem undergoes a series of phase transitions
suggesting existence of a region of parameters where estimation is information
theoretically possible, but AMP (and presumably every other polynomial
algorithm) fails. The analysis of the large rank limit is particularly
instructive.
Sparse Approximation of a Kernel Mean
Authors: E. Cruz Cortés, C. Scott
Date: 2015-03-01 00:00:00
Tasks: Clustering

Kernel means are frequently used to represent probability distributions in
machine learning problems. In particular, the well known kernel density
estimator and the kernel mean embedding both have the form of a kernel mean.
Unfortunately, kernel means are faced with scalability issues. A single point
evaluation of the kernel density estimator, for example, requires a computation
time linear in the training sample size. To address this challenge, we present
a method to efficiently construct a sparse approximation of a kernel mean. We
do so by first establishing an incoherence-based bound on the approximation
error, and then noticing that, for the case of radial kernels, the bound can be
minimized by solving the $k$-center problem. The outcome is a linear time
construction of a sparse kernel mean, which also lends itself naturally to an
automatic sparsity selection scheme. We show the computational gains of our
method by looking at three problems involving kernel means: Euclidean embedding
of distributions, class proportion estimation, and clustering using the
mean-shift algorithm.
Constructive sparse trigonometric approximation for functions with small mixed smoothness
Authors: V. N. Temlyakov
Date: 2015-03-01 00:00:00

The paper gives a constructive method, based on greedy algorithms, that
provides for the classes of functions with small mixed smoothness the best
possible in the sense of order approximation error for the $m$-term
approximation with respect to the trigonometric system.
An Online Convex Optimization Approach to Blackwell's Approachability
Authors: Nahum Shimkin
Date: 2015-03-01 00:00:00

The notion of approachability in repeated games with vector payoffs was
introduced by Blackwell in the 1950s, along with geometric conditions for
approachability and corresponding strategies that rely on computing {\em
steering directions} as projections from the current average payoff vector to
the (convex) target set. Recently, Abernethy, Batlett and Hazan (2011) proposed
a class of approachability algorithms that rely on the no-regret properties of
Online Linear Programming for computing a suitable sequence of steering
directions. This is first carried out for target sets that are convex cones,
and then generalized to any convex set by embedding it in a higher-dimensional
convex cone. In this paper we present a more direct formulation that relies on
the support function of the set, along with suitable Online Convex Optimization
algorithms, which leads to a general class of approachability algorithms. We
further show that Blackwell's original algorithm and its convergence follow as
a special case.
Novel Metaknowledge-based Processing Technique for Multimedia Big Data clustering challenges
Authors: Nima Bari, Roman Vichr, Kamran Kowsari, Simon Y. Berkovich
Date: 2015-03-01 00:00:00
Tasks: Clustering

Past research has challenged us with the task of showing relational patterns
between text-based data and then clustering for predictive analysis using Golay
Code technique. We focus on a novel approach to extract metaknowledge in
multimedia datasets. Our collaboration has been an on-going task of studying
the relational patterns between datapoints based on metafeatures extracted from
metaknowledge in multimedia datasets. Those selected are significant to suit
the mining technique we applied, Golay Code algorithm. In this research paper
we summarize findings in optimization of metaknowledge representation for
23-bit representation of structured and unstructured multimedia data in order
to