pwc-archive/papers-with-abstracts huggingface dataset:
Ergodic Inference: Accelerate Convergence by Optimisation
Authors: Yichuan Zhang, José Miguel Hernández-Lobato
Date: 2018-05-25 00:00:00
Tasks: Computational Efficiency, Variational Inference

Statistical inference methods are fundamentally important in machine learning. Most state-of-the-art inference algorithms are variants of Markov chain Monte Carlo (MCMC) or variational inference (VI). However, both methods struggle with limitations in practice: MCMC methods can be computationally demanding; VI methods may have large bias. In this work, we aim to improve upon MCMC and VI by a novel hybrid method based on the idea of reducing simulation bias of finite-length MCMC chains using gradient-based optimisation. The proposed method can generate low-biased samples by increasing the length of MCMC simulation and optimising the MCMC hyper-parameters, which offers attractive balance between approximation bias and computational efficiency. We show that our method produces promising results on popular benchmarks when compared to recent hybrid methods of MCMC and VI.
Robust Subspace Learning: Robust PCA, Robust Subspace Tracking, and Robust Subspace Recovery
Authors: Namrata Vaswani, Thierry Bouwmans, Sajid Javed, Praneeth Narayanamurthy
Date: 2017-11-26 00:00:00
Tasks: Dimensionality Reduction

PCA is one of the most widely used dimension reduction techniques. A related
easier problem is "subspace learning" or "subspace estimation". Given
relatively clean data, both are easily solved via singular value decomposition
(SVD). The problem of subspace learning or PCA in the presence of outliers is
called robust subspace learning or robust PCA (RPCA). For long data sequences,
if one tries to use a single lower dimensional subspace to represent the data,
the required subspace dimension may end up being quite large. For such data, a
better model is to assume that it lies in a low-dimensional subspace that can
change over time, albeit gradually. The problem of tracking such data (and the
subspaces) while being robust to outliers is called robust subspace tracking
(RST). This article provides a magazine-style overview of the entire field of
robust subspace learning and tracking. In particular solutions for three
problems are discussed in detail: RPCA via sparse+low-rank matrix decomposition
(S+LR), RST via S+LR, and "robust subspace recovery (RSR)". RSR assumes that an
entire data vector is either an outlier or an inlier. The S+LR formulation
instead assumes that outliers occur on only a few data vector indices and hence
are well modeled as sparse corruptions.
Inference in Probabilistic Graphical Models by Graph Neural Networks
Authors: KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq Pitkow
Date: 2018-03-21 00:00:00
Tasks: Decision Making

A fundamental computation for statistical inference and accurate decision-making is to compute the marginal probabilities or most probable states of task-relevant variables. Probabilistic graphical models can efficiently represent the structure of such complex data, but performing these inferences is generally difficult. Message-passing algorithms, such as belief propagation, are a natural way to disseminate evidence amongst correlated variables while exploiting the graph structure, but these algorithms can struggle when the conditional dependency graphs contain loops. Here we use Graph Neural Networks (GNNs) to learn a message-passing algorithm that solves these inference tasks. We first show that the architecture of GNNs is well-matched to inference tasks. We then demonstrate the efficacy of this inference approach by training GNNs on a collection of graphical models and showing that they substantially outperform belief propagation on loopy graphs. Our message-passing algorithms generalize out of the training set to larger graphs and graphs with different structure.
Static and Dynamic Robust PCA and Matrix Completion: A Review
Authors: Namrata Vaswani, Praneeth Narayanamurthy
Date: 2018-03-01 00:00:00
Tasks: Dimensionality Reduction, Low-Rank Matrix Completion, Matrix Completion

Principal Components Analysis (PCA) is one of the most widely used dimension
reduction techniques. Robust PCA (RPCA) refers to the problem of PCA when the
data may be corrupted by outliers. Recent work by Cand{\`e}s, Wright, Li, and
Ma defined RPCA as a problem of decomposing a given data matrix into the sum of
a low-rank matrix (true data) and a sparse matrix (outliers). The column space
of the low-rank matrix then gives the PCA solution. This simple definition has
lead to a large amount of interesting new work on provably correct, fast, and
practical solutions to RPCA. More recently, the dynamic (time-varying) version
of the RPCA problem has been studied and a series of provably correct, fast,
and memory efficient tracking solutions have been proposed. Dynamic RPCA (or
robust subspace tracking) is the problem of tracking data lying in a (slowly)
changing subspace while being robust to sparse outliers. This article provides
an exhaustive review of the last decade of literature on RPCA and its dynamic
counterpart (robust subspace tracking), along with describing their theoretical
guarantees, discussing the pros and cons of various approaches, and providing
empirical comparisons of performance and speed.
  A brief overview of the (low-rank) matrix completion literature is also
provided (the focus is on works not discussed in other recent reviews). This
refers to the problem of completing a low-rank matrix when only a subset of its
entries are observed. It can be interpreted as a simpler special case of RPCA
in which the indices of the outlier corrupted entries are known.
Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
Authors: Josh Fromm, Shwetak Patel, Matthai Philipose
Date: 2018-05-25 00:00:00
Proceeding: ICLR 2018 1
Tasks: Binarization

Recent work has shown that fast, compact low-bitwidth neural networks can be
surprisingly accurate. These networks use homogeneous binarization: all
parameters in each layer or (more commonly) the whole model have the same low
bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs
where each arithmetic instruction can have a custom bitwidth, motivating
heterogeneous binarization, where every parameter in the network may have a
different bitwidth. In this paper, we show that it is feasible and useful to
select bitwidths at the parameter granularity during training. For instance a
heterogeneously quantized version of modern networks such as AlexNet and
MobileNet, with the right mix of 1-, 2- and 3-bit parameters that average to
just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these
networks. Further, we provide analyses to show that the heterogeneously
binarized systems yield FPGA- and ASIC-based implementations that are
correspondingly more efficient in both circuit area and energy efficiency than
their homogeneous counterparts.
Analysing Symbolic Regression Benchmarks under a Meta-Learning Approach
Authors: Luiz Otavio Vilas Boas Oliveira, Joao Francisco Barreto da Silva Martins, Luis Fernando Miranda, Gisele Lobo Pappa
Date: 2018-05-25 00:00:00
Tasks: Meta-Learning, regression, Symbolic Regression

The definition of a concise and effective testbed for Genetic Programming
(GP) is a recurrent matter in the research community. This paper takes a new
step in this direction, proposing a different approach to measure the quality
of the symbolic regression benchmarks quantitatively. The proposed approach is
based on meta-learning and uses a set of dataset meta-features---such as the
number of examples or output skewness---to describe the datasets. Our idea is
to correlate these meta-features with the errors obtained by a GP method. These
meta-features define a space of benchmarks that should, ideally, have datasets
(points) covering different regions of the space. An initial analysis of 63
datasets showed that current benchmarks are concentrated in a small region of
this benchmark space. We also found out that number of instances and output
skewness are the most relevant meta-features to GP output error. Both
conclusions can help define which datasets should compose an effective testbed
for symbolic regression methods.
Detecting Deceptive Reviews using Generative Adversarial Networks
Authors: Hojjat Aghakhani, Aravind Machiry, Shirin Nilizadeh, Christopher Kruegel, Giovanni Vigna
Date: 2018-05-25 00:00:00
Tasks: General Classification, Reinforcement Learning, Reinforcement Learning (RL), text-classification, Text Classification

In the past few years, consumer review sites have become the main target of
deceptive opinion spam, where fictitious opinions or reviews are deliberately
written to sound authentic. Most of the existing work to detect the deceptive
reviews focus on building supervised classifiers based on syntactic and lexical
patterns of an opinion. With the successful use of Neural Networks on various
classification applications, in this paper, we propose FakeGAN a system that
for the first time augments and adopts Generative Adversarial Networks (GANs)
for a text classification task, in particular, detecting deceptive reviews.
Unlike standard GAN models which have a single Generator and Discriminator
model, FakeGAN uses two discriminator models and one generative model. The
generator is modeled as a stochastic policy agent in reinforcement learning
(RL), and the discriminators use Monte Carlo search algorithm to estimate and
pass the intermediate action-value as the RL reward to the generator. Providing
the generator model with two discriminator models avoids the mod collapse issue
by learning from both distributions of truthful and deceptive reviews. Indeed,
our experiments show that using two discriminators provides FakeGAN high
stability, which is a known issue for GAN architectures. While FakeGAN is built
upon a semi-supervised classifier, known for less accuracy, our evaluation
results on a dataset of TripAdvisor hotel reviews show the same performance in
terms of accuracy as of the state-of-the-art approaches that apply supervised
machine learning. These results indicate that GANs can be effective for text
classification tasks. Specifically, FakeGAN is effective at detecting deceptive
reviews.
Provable Dynamic Robust PCA or Robust Subspace Tracking
Authors: Praneeth Narayanamurthy, Namrata Vaswani
Date: 2017-05-24 00:00:00
Tasks: Compressive Sensing

Dynamic robust PCA refers to the dynamic (time-varying) extension of robust
PCA (RPCA). It assumes that the true (uncorrupted) data lies in a
low-dimensional subspace that can change with time, albeit slowly. The goal is
to track this changing subspace over time in the presence of sparse outliers.
We develop and study a novel algorithm, that we call simple-ReProCS, based on
the recently introduced Recursive Projected Compressive Sensing (ReProCS)
framework. Our work provides the first guarantee for dynamic RPCA that holds
under weakened versions of standard RPCA assumptions, slow subspace change and
a lower bound assumption on most outlier magnitudes. Our result is significant
because (i) it removes the strong assumptions needed by the two previous
complete guarantees for ReProCS-based algorithms; (ii) it shows that it is
possible to achieve significantly improved outlier tolerance, compared with all
existing RPCA or dynamic RPCA solutions by exploiting the above two simple
extra assumptions; and (iii) it proves that simple-ReProCS is online (after
initialization), fast, and, has near-optimal memory complexity.
What Face and Body Shapes Can Tell About Height
Authors: Semih Günel, Helge Rhodin, Pascal Fua
Date: 2018-05-25 00:00:00
Tasks: Anatomy, Autonomous Driving, Face Recognition

Recovering a person's height from a single image is important for virtual
garment fitting, autonomous driving and surveillance, however, it is also very
challenging due to the absence of absolute scale information. We tackle the
rarely addressed case, where camera parameters and scene geometry is unknown.
To nevertheless resolve the inherent scale ambiguity, we infer height from
statistics that are intrinsic to human anatomy and can be estimated from images
directly, such as articulated pose, bone length proportions, and facial
features. Our contribution is twofold. First, we experiment with different
machine learning models to capture the relation between image content and human
height. Second, we show that performance is predominantly limited by dataset
size and create a new dataset that is three magnitudes larger, by mining
explicit height labels and propagating them to additional images through face
recognition and assignment consistency. Our evaluation shows that monocular
height estimation is possible with a MAE of 5.56cm.
Self-Net: Lifelong Learning via Continual Self-Modeling
Authors: Blake Camp, Jaya Krishna Mandivarapu, Rolando Estrada
Date: 2018-05-25 00:00:00
Tasks: Continual Learning, Lifelong learning

Learning a set of tasks over time, also known as continual learning (CL), is one of the most challenging problems in artificial intelligence. While recent approaches achieve some degree of CL in deep neural networks, they either (1) grow the network parameters linearly with the number of tasks, (2) require storing training data from previous tasks, or (3) restrict the network's ability to learn new tasks. To address these issues, we propose a novel framework, Self-Net, that uses an autoencoder to learn a set of low-dimensional representations of the weights learned for different tasks. We demonstrate that these low-dimensional vectors can then be used to generate high-fidelity recollections of the original weights. Self-Net can incorporate new tasks over time with little retraining and with minimal loss in performance for older tasks. Our system does not require storing prior training data and its parameters grow only logarithmically with the number of tasks. We show that our technique outperforms current state-of-the-art approaches on numerous datasets---including continual versions of MNIST, CIFAR10, CIFAR100, and Atari---and we demonstrate that our method can achieve over 10X storage compression in a continual fashion. To the best of our knowledge, we are the first to use autoencoders to sequentially encode sets of network weights to enable continual learning.
Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
Authors: Jiahao Su, Jingling Li, Bobby Bhattacharjee, Furong Huang
Date: 2018-05-25 00:00:00
Tasks: Model Compression, tensor algebra, Tensor Decomposition

We propose tensorial neural networks (TNNs), a generalization of existing
neural networks by extending tensor operations on low order operands to those
on high order ones. The problem of parameter learning is challenging, as it
corresponds to hierarchical nonlinear tensor decomposition. We propose to solve
the learning problem using stochastic gradient descent by deriving nontrivial
backpropagation rules in generalized tensor algebra we introduce. Our proposed
TNNs has three advantages over existing neural networks: (1) TNNs naturally
apply to high order input object and thus preserve the multi-dimensional
structure in the input, as there is no need to flatten the data. (2) TNNs
interpret designs of existing neural network architectures. (3) Mapping a
neural network to TNNs with the same expressive power results in a TNN of fewer
parameters. TNN based compression of neural network improves existing low-rank
approximation based compression methods as TNNs exploit two other types of
invariant structures, periodicity and modulation, in addition to the low
rankness. Experiments on LeNet-5 (MNIST), ResNet-32 (CIFAR10) and ResNet-50
(ImageNet) demonstrate that our TNN based compression outperforms (5% test
accuracy improvement universally on CIFAR10) the state-of-the-art low-rank
approximation based compression methods under the same compression rate,
besides achieving orders of magnitude faster convergence rates due to the
efficiency of TNNs.
Guaranteed Simultaneous Asymmetric Tensor Decomposition via Orthogonalized Alternating Least Squares
Authors: Furong Huang, Jialin Li, Xuchen You
Date: 2018-05-25 00:00:00
Tasks: Tensor Decomposition

Tensor CANDECOMP/PARAFAC (CP) decomposition is an important tool that solves a wide class of machine learning problems. Existing popular approaches recover components one by one, not necessarily in the order of larger components first. Recently developed simultaneous power method obtains only a high probability recovery of top $r$ components even when the observed tensor is noiseless. We propose a Slicing Initialized Alternating Subspace Iteration (s-ASI) method that is guaranteed to recover top $r$ components ($\epsilon$-close) simultaneously for (a)symmetric tensors almost surely under the noiseless case (with high probability for a bounded noise) using $O(\log(\log \frac{1}{\epsilon}))$ steps of tensor subspace iterations. Our s-ASI method introduces a Slice-Based Initialization that runs $O(1/\log(\frac{\lambda_r}{\lambda_{r+1}}))$ steps of matrix subspace iterations, where $\lambda_r$ denotes the r-th top singular value of the tensor. We are the first to provide a theoretical guarantee on simultaneous orthogonal asymmetric tensor decomposition. Under the noiseless case, we are the first to provide an \emph{almost sure} theoretical guarantee on simultaneous orthogonal tensor decomposition. When tensor is noisy, our algorithm for asymmetric tensor is robust to noise smaller than $\min\{O(\frac{(\lambda_r - \lambda_{r+1})\epsilon}{\sqrt{r}}), O(\delta_0\frac{\lambda_r -\lambda_{r+1}}{\sqrt{d}})\}$, where $\delta_0$ is a small constant proportional to the probability of bad initializations in the noisy setting.
A Reduction for Optimizing Lattice Submodular Functions with Diminishing Returns
Authors: Alina Ene, Huy L. Nguyen
Date: 2016-06-27 00:00:00

A function $f: \mathbb{Z}_+^E \rightarrow \mathbb{R}_+$ is DR-submodular if
it satisfies $f({\bf x} + \chi_i) -f ({\bf x}) \ge f({\bf y} + \chi_i) - f({\bf
y})$ for all ${\bf x}\le {\bf y}, i\in E$. Recently, the problem of maximizing
a DR-submodular function $f: \mathbb{Z}_+^E \rightarrow \mathbb{R}_+$ subject
to a budget constraint $\|{\bf x}\|_1 \leq B$ as well as additional constraints
has received significant attention \cite{SKIK14,SY15,MYK15,SY16}.
  In this note, we give a generic reduction from the DR-submodular setting to
the submodular setting. The running time of the reduction and the size of the
resulting submodular instance depends only \emph{logarithmically} on $B$. Using
this reduction, one can translate the results for unconstrained and constrained
submodular maximization to the DR-submodular setting for many types of
constraints in a unified manner.
Pathology Segmentation using Distributional Differences to Images of Healthy Origin
Authors: Simon Andermatt, Antal Horváth, Simon Pezold, Philippe Cattin
Date: 2018-05-25 00:00:00
Tasks: Image-to-Image Translation, Segmentation, Translation, Weakly supervised segmentation

Fully supervised segmentation methods require a large training cohort of already segmented images, providing information at the pixel level of each image. We present a method to automatically segment and model pathologies in medical images, trained solely on data labelled on the image level as either healthy or containing a visual defect. We base our method on CycleGAN, an image-to-image translation technique, to translate images between the domains of healthy and pathological images. We extend the core idea with two key contributions. Implementing the generators as residual generators allows us to explicitly model the segmentation of the pathology. Realizing the translation from the healthy to the pathological domain using a variational autoencoder allows us to specify one representation of the pathology, as this transformation is otherwise not unique. Our model hence not only allows us to create pixelwise semantic segmentations, it is also able to create inpaintings for the segmentations to render the pathological image healthy. Furthermore, we can draw new unseen pathology samples from this model based on the distribution in the data. We show quantitatively, that our method is able to segment pathologies with a surprising accuracy being only slightly inferior to a state-of-the-art fully supervised method, although the latter has per-pixel rather than per-image training information. Moreover, we show qualitative results of both the segmentations and inpaintings. Our findings motivate further research into weakly-supervised segmentation using image level annotations, allowing for faster and cheaper acquisition of training data without a large sacrifice in segmentation accuracy.
An end-to-end Differentially Private Latent Dirichlet Allocation Using a Spectral Algorithm
Authors: Christopher DeCarolis, Mukul Ram, Seyed A. Esmaeili, Yu-Xiang Wang, Furong Huang
Date: 2018-05-25 00:00:00
Proceeding: ICML 2020 1
Tasks: Sensitivity, Variational Inference

We provide an end-to-end differentially private spectral algorithm for learning LDA, based on matrix/tensor decompositions, and establish theoretical guarantees on utility/consistency of the estimated model parameters. The spectral algorithm consists of multiple algorithmic steps, named as "{edges}", to which noise could be injected to obtain differential privacy. We identify \emph{subsets of edges}, named as "{configurations}", such that adding noise to all edges in such a subset guarantees differential privacy of the end-to-end spectral algorithm. We characterize the sensitivity of the edges with respect to the input and thus estimate the amount of noise to be added to each edge for any required privacy level. We then characterize the utility loss for each configuration as a function of injected noise. Overall, by combining the sensitivity and utility characterization, we obtain an end-to-end differentially private spectral algorithm for LDA and identify the corresponding configuration that outperforms others in any specific regime. We are the first to achieve utility guarantees under the required level of differential privacy for learning in LDA. Overall our method systematically outperforms differentially private variational inference.
Zero-Shot Dual Machine Translation
Authors: Lierni Sestorain, Massimiliano Ciaramita, Christian Buck, Thomas Hofmann
Date: 2018-05-25 00:00:00
Tasks: Machine Translation, NMT, Reinforcement Learning, Translation

Neural Machine Translation (NMT) systems rely on large amounts of parallel
data. This is a major challenge for low-resource languages. Building on recent
work on unsupervised and semi-supervised methods, we present an approach that
combines zero-shot and dual learning. The latter relies on reinforcement
learning, to exploit the duality of the machine translation task, and requires
only monolingual data for the target language pair. Experiments show that a
zero-shot dual system, trained on English-French and English-Spanish,
outperforms by large margins a standard NMT system in zero-shot translation
performance on Spanish-French (both directions). The zero-shot dual method
approaches the performance, within 2.2 BLEU points, of a comparable supervised
setting. Our method can obtain improvements also on the setting where a small
amount of parallel data for the zero-shot language pair is available. Adding
Russian, to extend our experiments to jointly modeling 6 zero-shot translation
directions, all directions improve between 4 and 15 BLEU points, again,
reaching performance near that of the supervised setting.