pwc-archive/papers-with-abstracts huggingface dataset:
Scheduled denoising autoencoders
Authors: Krzysztof J. Geras, Charles Sutton
Date: 2014-06-12 00:00:00
Tasks: Denoising, Representation Learning

We present a representation learning method that learns features at multiple
different levels of scale. Working within the unsupervised framework of
denoising autoencoders, we observe that when the input is heavily corrupted
during training, the network tends to learn coarse-grained features, whereas
when the input is only slightly corrupted, the network tends to learn
fine-grained features. This motivates the scheduled denoising autoencoder,
which starts with a high level of noise that lowers as training progresses. We
find that the resulting representation yields a significant boost on a later
supervised task compared to the original input, or to a standard denoising
autoencoder trained at a single noise level. After supervised fine-tuning our
best model achieves the lowest ever reported error on the CIFAR-10 data set
among permutation-invariant methods.
Phase Transitions in Spectral Community Detection of Large Noisy Networks
Authors: Pin-Yu Chen, Alfred O. Hero III
Date: 2015-04-09 00:00:00
Tasks: Clustering, Community Detection

In this paper, we study the sensitivity of the spectral clustering based
community detection algorithm subject to a Erdos-Renyi type random noise model.
We prove phase transitions in community detectability as a function of the
external edge connection probability and the noisy edge presence probability
under a general network model where two arbitrarily connected communities are
interconnected by random external edges. Specifically, the community detection
performance transitions from almost perfect detectability to low detectability
as the inter-community edge connection probability exceeds some critical value.
We derive upper and lower bounds on the critical value and show that the bounds
are identical when the two communities have the same size. The phase transition
results are validated using network simulations. Using the derived expressions
for the phase transition threshold we propose a method for estimating this
threshold from observed data.
Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
Authors: Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, Yann Lecun
Date: 2014-12-24 00:00:00
Tasks: GPU

We examine the performance profile of Convolutional Neural Network training
on the current generation of NVIDIA Graphics Processing Units. We introduce two
new Fast Fourier Transform convolution implementations: one based on NVIDIA's
cuFFT library, and another based on a Facebook authored FFT implementation,
fbfft, that provides significant speedups over cuFFT (over 1.5x) for whole
CNNs. Both of these convolution implementations are available in open source,
and are faster than NVIDIA's cuDNN implementation for many common convolutional
layers (up to 23.5x for some synthetic kernel configurations). We discuss
different performance regimes of convolutions, comparing areas where
straightforward time domain convolutions outperform Fourier frequency domain
convolutions. Details on algorithmic applications of NVIDIA GPU hardware
specifics in the implementation of fbfft are also provided.
Deep Networks With Large Output Spaces
Authors: Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, Jay Yagnik
Date: 2014-12-23 00:00:00
Tasks: Video Recognition

Deep neural networks have been extremely successful at various image, speech,
video recognition tasks because of their ability to model deep structures
within the data. However, they are still prohibitively expensive to train and
apply for problems containing millions of classes in the output layer. Based on
the observation that the key computation common to most neural network layers
is a vector/matrix product, we propose a fast locality-sensitive hashing
technique to approximate the actual dot product enabling us to scale up the
training and inference to millions of output classes. We evaluate our technique
on three diverse large-scale recognition tasks and show that our approach can
train large-scale models at a faster rate (in terms of steps/total time)
compared to baseline methods.
A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition
Authors: Roozbeh Mottaghi, Yu Xiang, Silvio Savarese
Date: 2015-04-10 00:00:00
Proceeding: CVPR 2015 6
Tasks: 3D Pose Estimation, Object, object-detection, Object Detection, Pose Estimation

Despite the fact that object detection, 3D pose estimation, and sub-category
recognition are highly correlated tasks, they are usually addressed
independently from each other because of the huge space of parameters. To
jointly model all of these tasks, we propose a coarse-to-fine hierarchical
representation, where each level of the hierarchy represents objects at a
different level of granularity. The hierarchical representation prevents
performance loss, which is often caused by the increase in the number of
parameters (as we consider more tasks to model), and the joint modelling
enables resolving ambiguities that exist in independent modelling of these
tasks. We augment PASCAL3D+ dataset with annotations for these tasks and show
that our hierarchical model is effective in joint modelling of object
detection, 3D pose estimation, and sub-category recognition.
Move Evaluation in Go Using Deep Convolutional Neural Networks
Authors: Chris J. Maddison, Aja Huang, Ilya Sutskever, David Silver
Date: 2014-12-20 00:00:00
Tasks: Board Games, Game of Go, Position

The game of Go is more challenging than other board games, due to the
difficulty of constructing a position or move evaluation function. In this
paper we investigate whether deep convolutional networks can be used to
directly represent and learn this knowledge. We train a large 12-layer
convolutional neural network by supervised learning from a database of human
professional games. The network correctly predicts the expert move in 55% of
positions, equalling the accuracy of a 6 dan human player. When the trained
convolutional network was used directly to play games of Go, without any
search, it beat the traditional search program GnuGo in 97% of games, and
matched the performance of a state-of-the-art Monte-Carlo tree search that
simulates a million positions per move.
Tailoring Word Embeddings for Bilexical Predictions: An Experimental Comparison
Authors: Pranava Swaroop Madhyastha, Xavier Carreras, Ariadna Quattoni
Date: 2014-12-22 00:00:00
Tasks: Relation, Word Embeddings

We investigate the problem of inducing word embeddings that are tailored for
a particular bilexical relation. Our learning algorithm takes an existing
lexical vector space and compresses it such that the resulting word embeddings
are good predictors for a target bilexical relation. In experiments we show
that task-specific embeddings can benefit both the quality and efficiency in
lexical prediction tasks.
Training Convolutional Networks with Noisy Labels
Authors: Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, Rob Fergus
Date: 2014-06-09 00:00:00
Tasks: General Classification

The availability of large labeled datasets has allowed Convolutional Network
models to achieve impressive recognition results. However, in many settings
manual annotation of the data is impractical; instead our data has noisy
labels, i.e. there is some freely available label for each image which may or
may not be accurate. In this paper, we explore the performance of
discriminatively-trained Convnets when trained on such noisy data. We introduce
an extra noise layer into the network which adapts the network outputs to match
the noisy label distribution. The parameters of this noise layer can be
estimated as part of the training process and involve simple modifications to
current training infrastructures for deep networks. We demonstrate the
approaches on several datasets, including large scale experiments on the
ImageNet classification benchmark.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Authors: Karen Simonyan, Andrew Zisserman
Date: 2014-09-04 00:00:00
Tasks: Activity Recognition In Videos, Classification, Domain Generalization, Face Anti-Spoofing, General Classification, Image Classification

In this work we investigate the effect of the convolutional network depth on
its accuracy in the large-scale image recognition setting. Our main
contribution is a thorough evaluation of networks of increasing depth using an
architecture with very small (3x3) convolution filters, which shows that a
significant improvement on the prior-art configurations can be achieved by
pushing the depth to 16-19 weight layers. These findings were the basis of our
ImageNet Challenge 2014 submission, where our team secured the first and the
second places in the localisation and classification tracks respectively. We
also show that our representations generalise well to other datasets, where
they achieve state-of-the-art results. We have made our two best-performing
ConvNet models publicly available to facilitate further research on the use of
deep visual representations in computer vision.
Diffusion Component Analysis: Unraveling Functional Topology in Biological Networks
Authors: Hyunghoon Cho, Bonnie Berger, Jian Peng
Date: 2015-04-10 00:00:00
Tasks: Dimensionality Reduction

Complex biological systems have been successfully modeled by biochemical and
genetic interaction networks, typically gathered from high-throughput (HTP)
data. These networks can be used to infer functional relationships between
genes or proteins. Using the intuition that the topological role of a gene in a
network relates to its biological function, local or diffusion based
"guilt-by-association" and graph-theoretic methods have had success in
inferring gene functions. Here we seek to improve function prediction by
integrating diffusion-based methods with a novel dimensionality reduction
technique to overcome the incomplete and noisy nature of network data. In this
paper, we introduce diffusion component analysis (DCA), a framework that plugs
in a diffusion model and learns a low-dimensional vector representation of each
node to encode the topological properties of a network. As a proof of concept,
we demonstrate DCA's substantial improvement over state-of-the-art
diffusion-based approaches in predicting protein function from molecular
interaction networks. Moreover, our DCA framework can integrate multiple
networks from heterogeneous sources, consisting of genomic information,
biochemical experiments and other resources, to even further improve function
prediction. Yet another layer of performance gain is achieved by integrating
the DCA framework with support vector machines that take our node vector
representations as features. Overall, our DCA framework provides a novel
representation of nodes in a network that can be used as a plug-in architecture
to other machine learning algorithms to decipher topological properties of and
obtain novel insights into interactomes.
Deep Structured Output Learning for Unconstrained Text Recognition
Authors: Max Jaderberg, Karen Simonyan, Andrea Vedaldi, Andrew Zisserman
Date: 2014-12-18 00:00:00
Tasks: Language Modeling, Language Modelling, Multi-Task Learning

We develop a representation suitable for the unconstrained recognition of
words in natural images: the general case of no fixed lexicon and unknown
length.
  To this end we propose a convolutional neural network (CNN) based
architecture which incorporates a Conditional Random Field (CRF) graphical
model, taking the whole word image as a single input. The unaries of the CRF
are provided by a CNN that predicts characters at each position of the output,
while higher order terms are provided by another CNN that detects the presence
of N-grams. We show that this entire model (CRF, character predictor, N-gram
predictor) can be jointly optimised by back-propagating the structured output
loss, essentially requiring the system to perform multi-task learning, and
training uses purely synthetically generated data. The resulting model is a
more accurate system on standard real-world text recognition benchmarks than
character prediction alone, setting a benchmark for systems that have not been
trained on a particular lexicon. In addition, our model achieves
state-of-the-art accuracy in lexicon-constrained scenarios, without being
specifically modelled for constrained recognition. To test the generalisation
of our model, we also perform experiments with random alpha-numeric strings to
evaluate the method when no visual language model is applicable.
Gradient of Probability Density Functions based Contrasts for Blind Source Separation (BSS)
Authors: Dharmani Bhaveshkumar C
Date: 2015-04-10 00:00:00
Tasks: blind source separation

The article derives some novel independence measures and contrast functions
for Blind Source Separation (BSS) application. For the $k^{th}$ order
differentiable multivariate functions with equal hyper-volumes (region bounded
by hyper-surfaces) and with a constraint of bounded support for $k>1$, it
proves that equality of any $k^{th}$ order derivatives implies equality of the
functions. The difference between product of marginal Probability Density
Functions (PDFs) and joint PDF of a random vector is defined as Function
Difference (FD) of a random vector. Assuming the PDFs are $k^{th}$ order
differentiable, the results on generalized functions are applied to the
independence condition. This brings new sets of independence measures and BSS
contrasts based on the $L^p$-Norm, $ p \geq 1$ of - FD, gradient of FD (GFD)
and Hessian of FD (HFD). Instead of a conventional two stage indirect
estimation method for joint PDF based BSS contrast estimation, a single stage
direct estimation of the contrasts is desired. The article targets both the
efficient estimation of the proposed contrasts and extension of the potential
theory for an information field. The potential theory has a concept of
reference potential and it is used to derive closed form expression for the
relative analysis of potential field. Analogous to it, there are introduced
concepts of Reference Information Potential (RIP) and Cross Reference
Information Potential (CRIP) based on the potential due to kernel functions
placed at selected sample points as basis in kernel methods. The quantities are
used to derive closed form expressions for information field analysis using
least squares. The expressions are used to estimate $L^2$-Norm of FD and
$L^2$-Norm of GFD based contrasts.
N-gram-Based Low-Dimensional Representation for Document Classification
Authors: Rémi Lebret, Ronan Collobert
Date: 2014-12-19 00:00:00
Tasks: Classification, Clustering, Document Classification, General Classification, Sentiment Analysis, Sentiment Classification

The bag-of-words (BOW) model is the common approach for classifying
documents, where words are used as feature for training a classifier. This
generally involves a huge number of features. Some techniques, such as Latent
Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA), have been
designed to summarize documents in a lower dimension with the least semantic
information loss. Some semantic information is nevertheless always lost, since
only words are considered. Instead, we aim at using information coming from
n-grams to overcome this limitation, while remaining in a low-dimension space.
Many approaches, such as the Skip-gram model, provide good word vector
representations very quickly. We propose to average these representations to
obtain representations of n-grams. All n-grams are thus embedded in a same
semantic space. A K-means clustering can then group them into semantic
concepts. The number of features is therefore dramatically reduced and
documents can be represented as bag of semantic concepts. We show that this
model outperforms LSA and LDA on a sentiment classification task, and yields
similar results than a traditional BOW-model with far less features.
The Ordered Weighted $\ell_1$ Norm: Atomic Formulation, Projections, and Algorithms
Authors: Xiangrong Zeng, Mário A. T. Figueiredo
Date: 2014-09-15 00:00:00
Tasks: Clustering, regression

The ordered weighted $\ell_1$ norm (OWL) was recently proposed, with two
different motivations: its good statistical properties as a sparsity promoting
regularizer; the fact that it generalizes the so-called {\it octagonal
shrinkage and clustering algorithm for regression} (OSCAR), which has the
ability to cluster/group regression variables that are highly correlated. This
paper contains several contributions to the study and application of OWL
regularization: the derivation of the atomic formulation of the OWL norm; the
derivation of the dual of the OWL norm, based on its atomic formulation; a new
and simpler derivation of the proximity operator of the OWL norm; an efficient
scheme to compute the Euclidean projection onto an OWL ball; the instantiation
of the conditional gradient (CG, also known as Frank-Wolfe) algorithm for
linear regression problems under OWL regularization; the instantiation of
accelerated projected gradient algorithms for the same class of problems.
Finally, a set of experiments give evidence that accelerated projected gradient
algorithms are considerably faster than CG, for the class of problems
considered.
Unsupervised Neural Architecture for Saliency Detection: Extended Version
Authors: Natalia Efremova, Sergey Tarasenko
Date: 2014-11-18 00:00:00
Tasks: Saliency Detection

We propose a novel neural network architecture for visual saliency
detections, which utilizes neurophysiologically plausible mechanisms for
extraction of salient regions. The model has been significantly inspired by
recent findings from neurophysiology and aimed to simulate the bottom-up
processes of human selective attention. Two types of features were analyzed:
color and direction of maximum variance. The mechanism we employ for processing
those features is PCA, implemented by means of normalized Hebbian learning and
the waves of spikes. To evaluate performance of our model we have conducted
psychological experiment. Comparison of simulation results with those of
experiment indicates good performance of our model.
NICE: Non-linear Independent Components Estimation
Authors: Laurent Dinh, David Krueger, Yoshua Bengio
Date: 2014-10-30 00:00:00
Tasks: Image Generation

We propose a deep learning framework for modeling complex high-dimensional
densities called Non-linear Independent Component Estimation (NICE). It is
based on the idea that a good representation is one in which the data has a
distribution that is easy to model. For this purpose, a non-linear
deterministic transformation of the data is learned that maps it to a latent
space so as to make the transformed data conform to a factorized distribution,
i.e., resulting in independent latent variables. We parametrize this
transformation so that computing the Jacobian determinant and inverse transform
is trivial, yet we maintain the ability to learn complex non-linear
transformations, via a composition of simple building blocks, each based on a
deep neural network. The training criterion is simply the exact log-likelihood,
which is tractable. Unbiased ancestral sampling is also easy. We show that this
approach yields good generative models on four image datasets and can be used
for inpainting.
Deep Narrow Boltzmann Machines are Universal Approximators
Authors: Guido Montufar
Date: 2014-11-14 00:00:00

We show that deep narrow Boltzmann machines are universal approximators of
probability distributions on the activities of their visible units, provided
they have sufficiently many hidden layers, each containing the same number of
units as the visible layer. We show that, within certain parameter domains,
deep Boltzmann machines can be studied as feedforward networks. We provide
upper and lower bounds on the sufficient depth and width of universal
approximators. These results settle various intuitions regarding undirected
networks and, in particular, they show that deep narrow Boltzmann machines are
at least as compact universal approximators as narrow sigmoid belief networks
and restricted Boltzmann machines, with respect to the currently available
bounds for those models.
Maximum Entropy Linear Manifold for Learning Discriminative Low-dimensional Representation
Authors: Wojciech Marian Czarnecki, Rafał Józefowicz, Jacek Tabor
Date: 2015-04-10 00:00:00
Tasks: Data Visualization, Dimensionality Reduction, General Classification, Representation Learning

Representation learning is currently a very hot topic in modern machine
learning, mostly due to the great success of the deep learning methods. In
particular low-dimensional representation which discriminates classes can not
only enhance the classification procedure, but also make it faster, while
contrary to the high-dimensional embeddings can be efficiently used for visual
based exploratory data analysis.
  In this paper we propose Maximum Entropy Linear Manifold (MELM), a
multidimensional generalization of Multithreshold Entropy Linear Classifier
model which is able to find a low-dimensional linear data projection maximizing
discriminativeness of projected classes. As a result we obtain a linear
embedding which can be used for classification, class aware dimensionality
reduction and data visualization. MELM provides highly discriminative 2D
projections of the data which can be used as a method for constructing robust
classifiers.
  We provide both empirical evaluation as well as some interesting theoretical
properties of our objective function such us scale and affine transformation
invariance, connections with PCA and bounding of the expected balanced accuracy
error.
Study of Some Recent Crossovers Effects on Speed and Accuracy of Genetic Algorithm, Using Symmetric Travelling Salesman Problem
Authors: Hassan Ismkhan, Kamran Zamanifar
Date: 2015-04-10 00:00:00

The Travelling Salesman Problem (TSP) is one of the most famous optimization
problems. The Genetic Algorithm (GA) is one of metaheuristics that have been
applied to TSP. The Crossover and mutation operators are two important elements
of GA. There are many TSP solver crossover operators. In this paper, we state
implementation of some recent TSP solver crossovers at first and then we use
each of them in GA to solve some Symmetric TSP (STSP) instances and finally
compare their effects on speed and accuracy of presented GA.
Visual Saliency Based on Multiscale Deep Features
Authors: Guanbin Li, Yizhou Yu
Date: 2015-03-30 00:00:00
Proceeding: CVPR 2015 6
Tasks: Image Segmentation, Semantic Segmentation

Visual saliency is a fundamental problem in both cognitive and computational
sciences, including computer vision. In this CVPR 2015 paper, we discover that
a high-quality visual saliency model can be trained with multiscale features
extracted using a popular deep learning architecture, convolutional neural
networks (CNNs), which have had many successes in visual recognition tasks. For
learning such saliency models, we introduce a neural network architecture,
which has fully connected layers on top of CNNs responsible for extracting
features at three different scales. We then propose a refinement method to
enhance the spatial coherence of our saliency results. Finally, aggregating
multiple saliency maps computed for different levels of image segmentation can
further boost the performance, yielding saliency maps better than those
generated from a single segmentation. To promote further research and
evaluation of visual saliency models, we also construct a new large database of
4447 challenging images and their pixelwise saliency annotation. Experimental
results demonstrate that our proposed method is capable of achieving
state-of-the-art performance on all public benchmarks, improving the F-Measure
by 5.0% and 13.2% respectively on the MSRA-B dataset and our new dataset
(HKU-IS), and lowering the mean absolute error by 5.7% and 35.1% respectively
on these two datasets.