pwc-archive/papers-with-abstracts huggingface dataset:
Variational Gaussian Dropout is not Bayesian
Authors: Jiri Hron, Alexander G. de G. Matthews, Zoubin Ghahramani
Date: 2017-11-08 00:00:00
Tasks: Bayesian Inference

Gaussian multiplicative noise is commonly used as a stochastic regularisation
technique in training of deterministic neural networks. A recent paper
reinterpreted the technique as a specific algorithm for approximate inference
in Bayesian neural networks; several extensions ensued. We show that the
log-uniform prior used in all the above publications does not generally induce
a proper posterior, and thus Bayesian inference in such models is ill-posed.
Independent of the log-uniform prior, the correlated weight noise approximation
has further issues leading to either infinite objective or high risk of
overfitting. The above implies that the reported sparsity of obtained solutions
cannot be explained by Bayesian or the related minimum description length
arguments. We thus study the objective from a non-Bayesian perspective, provide
its previously unknown analytical form which allows exact gradient evaluation,
and show that the later proposed additive reparametrisation introduces minima
not present in the original multiplicative parametrisation. Implications and
future research directions are discussed.
Simultaneously Color-Depth Super-Resolution with Conditional Generative Adversarial Network
Authors: Lijun Zhao, Huihui Bai, Jie Liang, Bing Zeng, Anhong Wang, Yao Zhao
Date: 2017-08-30 00:00:00
Tasks: Edge Detection, Generative Adversarial Network, image smoothing, Image Super-Resolution, Image-to-Image Translation, Style Transfer, Super-Resolution, Translation

Recently, Generative Adversarial Network (GAN) has been found wide
applications in style transfer, image-to-image translation and image
super-resolution. In this paper, a color-depth conditional GAN is proposed to
concurrently resolve the problems of depth super-resolution and color
super-resolution in 3D videos. Firstly, given the low-resolution depth image
and low-resolution color image, a generative network is proposed to leverage
mutual information of color image and depth image to enhance each other in
consideration of the geometry structural dependency of color-depth image in the
same scene. Secondly, three loss functions, including data loss, total
variation loss, and 8-connected gradient difference loss are introduced to
train this generative network in order to keep generated images close to the
real ones, in addition to the adversarial loss. Experimental results
demonstrate that the proposed approach produces high-quality color image and
depth image from low-quality image pair, and it is superior to several other
leading methods. Besides, we use the same neural network framework to resolve
the problem of image smoothing and edge detection at the same time.
Learning Non-overlapping Convolutional Neural Networks with Multiple Kernels
Authors: Kai Zhong, Zhao Song, Inderjit S. Dhillon
Date: 2017-11-08 00:00:00

In this paper, we consider parameter recovery for non-overlapping
convolutional neural networks (CNNs) with multiple kernels. We show that when
the inputs follow Gaussian distribution and the sample size is sufficiently
large, the squared loss of such CNNs is $\mathit{~locally~strongly~convex}$ in
a basin of attraction near the global optima for most popular activation
functions, like ReLU, Leaky ReLU, Squared ReLU, Sigmoid and Tanh. The required
sample complexity is proportional to the dimension of the input and polynomial
in the number of kernels and a condition number of the parameters. We also show
that tensor methods are able to initialize the parameters to the local strong
convex region. Hence, for most smooth activations, gradient descent following
tensor initialization is guaranteed to converge to the global optimal with time
that is linear in input dimension, logarithmic in precision and polynomial in
other factors. To the best of our knowledge, this is the first work that
provides recovery guarantees for CNNs with multiple kernels under polynomial
sample and computational complexities.
Scene-adapted plug-and-play algorithm with convergence guarantees
Authors: Afonso M. Teodoro, José M. Bioucas-Dias, Mário A. T. Figueiredo
Date: 2017-02-08 00:00:00
Tasks: Denoising, Image Denoising

Recent frameworks, such as the so-called plug-and-play, allow us to leverage
the developments in image denoising to tackle other, and more involved,
problems in image processing. As the name suggests, state-of-the-art denoisers
are plugged into an iterative algorithm that alternates between a denoising
step and the inversion of the observation operator. While these tools offer
flexibility, the convergence of the resulting algorithm may be difficult to
analyse. In this paper, we plug a state-of-the-art denoiser, based on a
Gaussian mixture model, in the iterations of an alternating direction method of
multipliers and prove the algorithm is guaranteed to converge. Moreover, we
build upon the concept of scene-adapted priors where we learn a model targeted
to a specific scene being imaged, and apply the proposed method to address the
hyperspectral sharpening problem.
Is Simple Better? Revisiting Non-linear Matrix Factorization for Learning Incomplete Ratings
Authors: Vaibhav Krishna, Tian Guo, Nino Antulov-Fantulin
Date: 2017-10-16 00:00:00
Tasks: Clustering, Collaborative Filtering, Recommendation Systems

Matrix factorization techniques have been widely used as a method for
collaborative filtering for recommender systems. In recent times, different
variants of deep learning algorithms have been explored in this setting to
improve the task of making a personalized recommendation with user-item
interaction data. The idea that the mapping between the latent user or item
factors and the original features is highly nonlinear suggest that classical
matrix factorization techniques are no longer sufficient. In this paper, we
propose a multilayer nonlinear semi-nonnegative matrix factorization method,
with the motivation that user-item interactions can be modeled more accurately
using a linear combination of non-linear item features. Firstly, we learn
latent factors for representations of users and items from the designed
multilayer nonlinear Semi-NMF approach using explicit ratings. Secondly, the
architecture built is compared with deep-learning algorithms like Restricted
Boltzmann Machine and state-of-the-art Deep Matrix factorization techniques. By
using both supervised rate prediction task and unsupervised clustering in
latent item space, we demonstrate that our proposed approach achieves better
generalization ability in prediction as well as comparable representation
ability as deep matrix factorization in the clustering task.
ASAGA: Asynchronous Parallel SAGA
Authors: Rémi Leblond, Fabian Pedregosa, Simon Lacoste-Julien
Date: 2016-06-15 00:00:00

We describe ASAGA, an asynchronous parallel version of the incremental
gradient algorithm SAGA that enjoys fast linear convergence rates. Through a
novel perspective, we revisit and clarify a subtle but important technical
issue present in a large fraction of the recent convergence rate proofs for
asynchronous parallel optimization algorithms, and propose a simplification of
the recently introduced "perturbed iterate" framework that resolves it. We
thereby prove that ASAGA can obtain a theoretical linear speedup on multi-core
systems even without sparsity assumptions. We present results of an
implementation on a 40-core architecture illustrating the practical speedup as
well as the hardware overhead.
Investigation of event-based memory surfaces for high-speed tracking, unsupervised feature extraction and object recognition
Authors: Saeed Afshar, Gregory Cohen, Tara Julia Hamilton, Jonathan Tapson, Andre van Schaik
Date: 2016-03-14 00:00:00
Tasks: Object Recognition

In this paper we compare event-based decaying and time based-decaying memory
surfaces for high-speed eventbased tracking, feature extraction, and object
classification using an event-based camera. The high-speed recognition task
involves detecting and classifying model airplanes that are dropped free-hand
close to the camera lens so as to generate a challenging dataset exhibiting
significant variance in target velocity. This variance motivated the
investigation of event-based decaying memory surfaces in comparison to
time-based decaying memory surfaces to capture the temporal aspect of the
event-based data. These surfaces are then used to perform unsupervised feature
extraction, tracking and recognition. In order to generate the memory surfaces,
event binning, linearly decaying kernels, and exponentially decaying kernels
were investigated with exponentially decaying kernels found to perform best.
Event-based decaying memory surfaces were found to outperform time-based
decaying memory surfaces in recognition especially when invariance to target
velocity was made a requirement. A range of network and receptive field sizes
were investigated. The system achieves 98.75% recognition accuracy within 156
milliseconds of an airplane entering the field of view, using only twenty-five
event-based feature extracting neurons in series with a linear classifier. By
comparing the linear classifier results to an ELM classifier, we find that a
small number of event-based feature extractors can effectively project the
complex spatio-temporal event patterns of the dataset to an almost linearly
separable representation in feature space.
Dictionary-based Tensor Canonical Polyadic Decomposition
Authors: Jérémy E. Cohen, Nicolas Gillis
Date: 2017-04-03 00:00:00
Tasks: Tensor Decomposition

To ensure interpretability of extracted sources in tensor decomposition, we
introduce in this paper a dictionary-based tensor canonical polyadic
decomposition which enforces one factor to belong exactly to a known
dictionary. A new formulation of sparse coding is proposed which enables high
dimensional tensors dictionary-based canonical polyadic decomposition. The
benefits of using a dictionary in tensor decomposition models are explored both
in terms of parameter identifiability and estimation accuracy. Performances of
the proposed algorithms are evaluated on the decomposition of simulated data
and the unmixing of hyperspectral images.
Variational Fourier features for Gaussian processes
Authors: James Hensman, Nicolas Durrande, Arno Solin
Date: 2016-11-21 00:00:00
Tasks: Gaussian Processes

This work brings together two powerful concepts in Gaussian processes: the
variational approach to sparse approximation and the spectral representation of
Gaussian processes. This gives rise to an approximation that inherits the
benefits of the variational approach but with the representational power and
computational scalability of spectral representations. The work hinges on a key
result that there exist spectral features related to a finite domain of the
Gaussian process which exhibit almost-independent covariances. We derive these
expressions for Matern kernels in one dimension, and generalize to more
dimensions using kernels with specific structures. Under the assumption of
additive Gaussian noise, our method requires only a single pass through the
dataset, making for very fast and accurate computation. We fit a model to 4
million training points in just a few minutes on a standard laptop. With
non-conjugate likelihoods, our MCMC scheme reduces the cost of computation from
O(NM2) (for a sparse Gaussian process) to O(NM) per iteration, where N is the
number of data and M is the number of features.
Universal consistency and minimax rates for online Mondrian Forests
Authors: Jaouad Mourtada, Stéphane Gaïffas, Erwan Scornet
Date: 2017-11-08 00:00:00
Proceeding: NeurIPS 2017 12
Tasks: General Classification, regression

We establish the consistency of an algorithm of Mondrian Forests, a
randomized classification algorithm that can be implemented online. First, we
amend the original Mondrian Forest algorithm, that considers a fixed lifetime
parameter. Indeed, the fact that this parameter is fixed hinders the
statistical consistency of the original procedure. Our modified Mondrian Forest
algorithm grows trees with increasing lifetime parameters $\lambda_n$, and uses
an alternative updating rule, allowing to work also in an online fashion.
Second, we provide a theoretical analysis establishing simple conditions for
consistency. Our theoretical analysis also exhibits a surprising fact: our
algorithm achieves the minimax rate (optimal rate) for the estimation of a
Lipschitz regression function, which is a strong extension of previous results
to an arbitrary dimension.
LatentPoison - Adversarial Attacks On The Latent Space
Authors: Antonia Creswell, Anil A. Bharath, Biswa Sengupta
Date: 2017-11-08 00:00:00
Tasks: Decoder, General Classification, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

Robustness and security of machine learning (ML) systems are intertwined,
wherein a non-robust ML system (classifiers, regressors, etc.) can be subject
to attacks using a wide variety of exploits. With the advent of scalable deep
learning methodologies, a lot of emphasis has been put on the robustness of
supervised, unsupervised and reinforcement learning algorithms. Here, we study
the robustness of the latent space of a deep variational autoencoder (dVAE), an
unsupervised generative framework, to show that it is indeed possible to
perturb the latent space, flip the class predictions and keep the
classification probability approximately equal before and after an attack. This
means that an agent that looks at the outputs of a decoder would remain
oblivious to an attack.
Un résultat intrigant en commande sans modèle
Authors: Cédric Join, Emmanuel Delaleau, Michel Fliess, Claude H. Moog
Date: 2017-11-08 00:00:00

An elementary mathematical example proves, thanks to the Routh-Hurwitz
criterion, a result that is intriguing with respect to today's practical
understanding of model-free control, i.e., an "intelligent" proportional
controller (iP) may turn to be more difficult to tune than an intelligent
proportional-derivative one (iPD). The vast superiority of iPDs when compared
to classic PIDs is shown via computer simulations. The introduction as well as
the conclusion analyse model-free control in the light of recent advances.
Dimension Estimation Using Random Connection Models
Authors: Paulo Serra, Michel Mandjes
Date: 2017-11-08 00:00:00
Tasks: Dimensionality Reduction

Information about intrinsic dimension is crucial to perform dimensionality
reduction, compress information, design efficient algorithms, and do
statistical adaptation. In this paper we propose an estimator for the intrinsic
dimension of a data set. The estimator is based on binary neighbourhood
information about the observations in the form of two adjacency matrices, and
does not require any explicit distance information. The underlying graph is
modelled according to a subset of a specific random connection model, sometimes
referred to as the Poisson blob model. Computationally the estimator scales
like n log n, and we specify its asymptotic distribution and rate of
convergence. A simulation study on both real and simulated data shows that our
approach compares favourably with some competing methods from the literature,
including approaches that rely on distance information.
The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
Authors: Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, Davide Scaramuzza
Date: 2016-10-26 00:00:00
Tasks: Motion Estimation, Pose Estimation, Visual Odometry

New vision sensors, such as the Dynamic and Active-pixel Vision sensor
(DAVIS), incorporate a conventional global-shutter camera and an event-based
sensor in the same pixel array. These sensors have great potential for
high-speed robotics and computer vision because they allow us to combine the
benefits of conventional cameras with those of event-based sensors: low
latency, high temporal resolution, and very high dynamic range. However, new
algorithms are required to exploit the sensor characteristics and cope with its
unconventional output, which consists of a stream of asynchronous brightness
changes (called "events") and synchronous grayscale frames. For this purpose,
we present and release a collection of datasets captured with a DAVIS in a
variety of synthetic and real environments, which we hope will motivate
research on new algorithms for high-speed and high-dynamic-range robotics and
computer-vision applications. In addition to global-shutter intensity images
and asynchronous events, we provide inertial measurements and ground-truth
camera poses from a motion-capture system. The latter allows comparing the pose
accuracy of ego-motion estimation algorithms quantitatively. All the data are
released both as standard text files and binary files (i.e., rosbag). This
paper provides an overview of the available data and describes a simulator that
we release open-source to create synthetic event-camera data.
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Authors: Maithra Raghu, Justin Gilmer, Jason Yosinski, Jascha Sohl-Dickstein
Date: 2017-06-19 00:00:00
Proceeding: NeurIPS 2017 12

We propose a new technique, Singular Vector Canonical Correlation Analysis
(SVCCA), a tool for quickly comparing two representations in a way that is both
invariant to affine transform (allowing comparison between different layers and
networks) and fast to compute (allowing more comparisons to be calculated than
with previous methods). We deploy this tool to measure the intrinsic
dimensionality of layers, showing in some cases needless over-parameterization;
to probe learning dynamics throughout training, finding that networks converge
to final representations from the bottom up; to show where class-specific
information in networks is formed; and to suggest new training regimes that
simultaneously save computation and overfit less. Code:
https://github.com/google/svcca/
Learning Sparse Visual Representations with Leaky Capped Norm Regularizers
Authors: Jianqiao Wangni, Dahua Lin
Date: 2017-11-08 00:00:00

Sparsity inducing regularization is an important part for learning
over-complete visual representations. Despite the popularity of $\ell_1$
regularization, in this paper, we investigate the usage of non-convex
regularizations in this problem. Our contribution consists of three parts.
First, we propose the leaky capped norm regularization (LCNR), which allows
model weights below a certain threshold to be regularized more strongly as
opposed to those above, therefore imposes strong sparsity and only introduces
controllable estimation bias. We propose a majorization-minimization algorithm
to optimize the joint objective function. Second, our study over monocular 3D
shape recovery and neural networks with LCNR outperforms $\ell_1$ and other
non-convex regularizations, achieving state-of-the-art performance and faster
convergence. Third, we prove a theoretical global convergence speed on the 3D
recovery problem. To the best of our knowledge, this is the first convergence
analysis of the 3D recovery problem.
Transductive Zero-Shot Hashing via Coarse-to-Fine Similarity Mining
Authors: Hanjiang Lai, Yan Pan
Date: 2017-11-08 00:00:00
Tasks: Transfer Learning

Zero-shot Hashing (ZSH) is to learn hashing models for novel/target classes
without training data, which is an important and challenging problem. Most
existing ZSH approaches exploit transfer learning via an intermediate shared
semantic representations between the seen/source classes and novel/target
classes. However, due to having disjoint, the hash functions learned from the
source dataset are biased when applied directly to the target classes. In this
paper, we study the transductive ZSH, i.e., we have unlabeled data for novel
classes. We put forward a simple yet efficient joint learning approach via
coarse-to-fine similarity mining which transfers knowledges from source data to
target data. It mainly consists of two building blocks in the proposed deep
architecture: 1) a shared two-streams network, which the first stream operates
on the source data and the second stream operates on the unlabeled data, to
learn the effective common image representations, and 2) a coarse-to-fine
module, which begins with finding the most representative images from target
classes and then further detect similarities among these images, to transfer
the similarities of the source data to the target data in a greedy fashion.
Extensive evaluation results on several benchmark datasets demonstrate that the
proposed hashing method achieves significant improvement over the
state-of-the-art methods.
Intriguing Properties of Adversarial Examples
Authors: Ekin D. Cubuk, Barret Zoph, Samuel S. Schoenholz, Quoc V. Le
Date: 2017-11-08 00:00:00
Proceeding: ICLR 2018 1
Tasks: Adversarial Robustness, Neural Architecture Search, Reinforcement Learning

It is becoming increasingly clear that many machine learning classifiers are
vulnerable to adversarial examples. In attempting to explain the origin of
adversarial examples, previous studies have typically focused on the fact that
neural networks operate on high dimensional data, they overfit, or they are too
linear. Here we argue that the origin of adversarial examples is primarily due
to an inherent uncertainty that neural networks have about their predictions.
We show that the functional form of this uncertainty is independent of
architecture, dataset, and training protocol; and depends only on the
statistics of the logit differences of the network, which do not change
significantly during training. This leads to adversarial error having a
universal scaling, as a power-law, with respect to the size of the adversarial
perturbation. We show that this universality holds for a broad range of
datasets (MNIST, CIFAR10, ImageNet, and random data), models (including
state-of-the-art deep networks, linear models, adversarially trained networks,
and networks trained on randomly shuffled labels), and attacks (FGSM, step
l.l., PGD). Motivated by these results, we study the effects of reducing
prediction entropy on adversarial robustness. Finally, we study the effect of
network architectures on adversarial sensitivity. To do this, we use neural
architecture search with reinforcement learning to find adversarially robust
architectures on CIFAR10. Our resulting architecture is more robust to white
\emph{and} black box attacks compared to previous attempts.
A Benchmark for Sparse Coding: When Group Sparsity Meets Rank Minimization
Authors: Zhiyuan Zha, Xin Yuan, Bihan Wen, Jiantao Zhou, Jiachao Zhang, Ce Zhu
Date: 2017-09-12 00:00:00
Tasks: Dictionary Learning, Image Inpainting, Image Restoration

Sparse coding has achieved a great success in various image processing tasks. However, a benchmark to measure the sparsity of image patch/group is missing since sparse coding is essentially an NP-hard problem. This work attempts to fill the gap from the perspective of rank minimization. More details please see the manuscript....
Revealing structure components of the retina by deep learning networks
Authors: Qi Yan, Zhaofei Yu, Feng Chen, Jian. K. Liu
Date: 2017-11-08 00:00:00
Tasks: Deep Learning

Deep convolutional neural networks (CNNs) have demonstrated impressive
performance on visual object classification tasks. In addition, it is a useful
model for predication of neuronal responses recorded in visual system. However,
there is still no clear understanding of what CNNs learn in terms of visual
neuronal circuits. Visualizing CNN's features to obtain possible connections to
neuronscience underpinnings is not easy due to highly complex circuits from the
retina to higher visual cortex. Here we address this issue by focusing on
single retinal ganglion cells with a simple model and electrophysiological
recordings from salamanders. By training CNNs with white noise images to
predicate neural responses, we found that convolutional filters learned in the
end are resembling to biological components of the retinal circuit. Features
represented by these filters tile the space of conventional receptive field of
retinal ganglion cells. These results suggest that CNN could be used to reveal
structure components of neuronal circuits.