pwc-archive/papers-with-abstracts huggingface dataset:
Lipschitz regularity of deep neural networks: analysis and efficient estimation
Authors: Kevin Scaman, Aladin Virmaux
Date: 2018-05-28 00:00:00
Proceeding: NeurIPS 2018 12

Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-of-art methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.
Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training
Authors: Xu Han, Zhiyuan Liu, Maosong Sun
Date: 2018-05-28 00:00:00
Tasks: Denoising, Relation, Relation Extraction

Existing neural relation extraction (NRE) models rely on distant supervision
and suffer from wrong labeling problems. In this paper, we propose a novel
adversarial training mechanism over instances for relation extraction to
alleviate the noise issue. As compared with previous denoising methods, our
proposed method can better discriminate those informative instances from noisy
ones. Our method is also efficient and flexible to be applied to various NRE
architectures. As shown in the experiments on a large-scale benchmark dataset
in relation extraction, our denoising method can effectively filter out noisy
instances and achieve significant improvements as compared with the
state-of-the-art models.
Discrete flow posteriors for variational inference in discrete dynamical systems
Authors: Laurence Aitchison, Vincent Adam, Srinivas C. Turaga
Date: 2018-05-28 00:00:00
Proceeding: ICLR 2019 5
Tasks: GPU, Variational Inference

Each training step for a variational autoencoder (VAE) requires us to sample
from the approximate posterior, so we usually choose simple (e.g. factorised)
approximate posteriors in which sampling is an efficient computation that fully
exploits GPU parallelism. However, such simple approximate posteriors are often
insufficient, as they eliminate statistical dependencies in the posterior.
While it is possible to use normalizing flow approximate posteriors for
continuous latents, some problems have discrete latents and strong statistical
dependencies. The most natural approach to model these dependencies is an
autoregressive distribution, but sampling from such distributions is inherently
sequential and thus slow. We develop a fast, parallel sampling procedure for
autoregressive distributions based on fixed-point iterations which enables
efficient and accurate variational inference in discrete state-space latent
variable dynamical systems. To optimize the variational bound, we considered
two ways to evaluate probabilities: inserting the relaxed samples directly into
the pmf for the discrete distribution, or converting to continuous logistic
latent variables and interpreting the K-step fixed-point iterations as a
normalizing flow. We found that converting to continuous latent variables gave
considerable additional scope for mismatch between the true and approximate
posteriors, which resulted in biased inferences, we thus used the former
approach. Using our fast sampling procedure, we were able to realize the
benefits of correlated posteriors, including accurate uncertainty estimates for
one cell, and accurate connectivity estimates for multiple cells, in an order
of magnitude less time.
Temporal Event Knowledge Acquisition via Identifying Narratives
Authors: Wenlin Yao, Ruihong Huang
Date: 2018-05-28 00:00:00
Proceeding: ACL 2018 7
Tasks: General Classification, Relation Classification, Temporal Relation Classification

Inspired by the double temporality characteristic of narrative texts, we
propose a novel approach for acquiring rich temporal "before/after" event
knowledge across sentences in narrative stories. The double temporality states
that a narrative story often describes a sequence of events following the
chronological order and therefore, the temporal order of events matches with
their textual order. We explored narratology principles and built a weakly
supervised approach that identifies 287k narrative paragraphs from three large
text corpora. We then extracted rich temporal event knowledge from these
narrative paragraphs. Such event knowledge is shown useful to improve temporal
relation classification and outperform several recent neural network models on
the narrative cloze task.
Fusion of Methods Based on Minutiae, Ridges and Pores for Robust Fingerprint Recognition
Authors: Lucas Alexandre Ramos, Aparecido Nilceu Marana
Date: 2018-05-28 00:00:00

The use of physical and behavioral characteristics for human identification
is known as biometrics. Among the many biometrics traits available, the
fingerprint is the most widely used. The fingerprint identification is based on
the impression patterns, as the pattern of ridges and minutiae, characteristics
of first and second levels respectively. The current identification systems use
these two levels of fingerprint features due to the low cost of the sensors.
However, due the recent advances in sensor technology, it is possible to use
third level features present within the ridges, such as the perspiration pores.
Recent studies have shown that the use of third-level features can increase
security and fraud protection in biometric systems, since they are difficult to
reproduce. In addition, recent researches have also focused on multibiometrics
recognition due to its many advantages. The goal of this work was to apply
fusion techniques for fingerprint recognition in order to combine minutiae,
ridges and pore-based methods and, thus, provide more robust biometrics
recognition systems. We evaluated isotropic-based and adaptive-based automatic
pore extraction methods and the fusion of pore-based method with the
identification methods based on minutiae and ridges. The experiments were
performed on the public database PolyU HRF and showed a reduction of
approximately 16% in the Equal Error Rate compared to the best results obtained
by the methods individually.
Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization
Authors: Dmitry Kobak, Jonathan Lomond, Benoit Sanchez
Date: 2018-05-28 00:00:00

A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined $n\ll p$ situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we demonstrate that an explicit positive ridge penalty can fail to provide any improvement over the minimum-norm least squares estimator. Moreover, the optimal value of ridge penalty in this situation can be negative. This happens when the high-variance directions in the predictor space can predict the response variable, which is often the case in the real-world high-dimensional data. In this regime, low-variance directions provide an implicit ridge regularization and can make any further positive ridge penalty detrimental. We prove that augmenting any linear model with random covariates and using minimum-norm estimator is asymptotically equivalent to adding the ridge penalty. We use a spiked covariance model as an analytically tractable example and prove that the optimal ridge penalty in this case is negative when $n\ll p$.
Face hallucination using cascaded super-resolution and identity priors
Authors: Klemen Grm, Simon Dobrišek, Walter J. Scheirer, Vitomir Štruc
Date: 2018-05-28 00:00:00
Tasks: Face Hallucination, Face Recognition, Hallucination, Super-Resolution

In this paper we address the problem of hallucinating high-resolution facial
images from unaligned low-resolution inputs at high magnification factors. We
approach the problem with convolutional neural networks (CNNs) and propose a
novel (deep) face hallucination model that incorporates identity priors into
the learning procedure. The model consists of two main parts: i) a cascaded
super-resolution network that upscales the low-resolution images, and ii) an
ensemble of face recognition models that act as identity priors for the
super-resolution network during training. Different from competing
super-resolution approaches that typically rely on a single model for upscaling
(even with large magnification factors), our network uses a cascade of multiple
SR models that progressively upscale the low-resolution images using steps of
$2\times$. This characteristic allows us to apply supervision signals (target
appearances) at different resolutions and incorporate identity constraints at
multiple-scales. Our model is able to upscale (very) low-resolution images
captured in unconstrained conditions and produce visually convincing results.
We rigorously evaluate the proposed model on a large datasets of facial images
and report superior performance compared to the state-of-the-art.
Inductive Framework for Multi-Aspect Streaming Tensor Completion with Side Information
Authors: Madhav Nimishakavi, Bamdev Mishra, Manish Gupta, Partha Talukdar
Date: 2018-02-18 00:00:00
Tasks: Missing Values

Low rank tensor completion is a well studied problem and has applications in
various fields. However, in many real world applications the data is dynamic,
i.e., new data arrives at different time intervals. As a result, the tensors
used to represent the data grow in size. Besides the tensors, in many real
world scenarios, side information is also available in the form of matrices
which also grow in size with time. The problem of predicting missing values in
the dynamically growing tensor is called dynamic tensor completion. Most of the
previous work in dynamic tensor completion make an assumption that the tensor
grows only in one mode. To the best of our Knowledge, there is no previous work
which incorporates side information with dynamic tensor completion. We bridge
this gap in this paper by proposing a dynamic tensor completion framework
called Side Information infused Incremental Tensor Analysis (SIITA), which
incorporates side information and works for general incremental tensors. We
also show how non-negative constraints can be incorporated with SIITA, which is
essential for mining interpretable latent clusters. We carry out extensive
experiments on multiple real world datasets to demonstrate the effectiveness of
SIITA in various different settings.
Deep Anomaly Detection Using Geometric Transformations
Authors: Izhak Golan, Ran El-Yaniv
Date: 2018-05-28 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Anomaly Detection

We consider the problem of anomaly detection in images, and present a new
detection technique. Given a sample of images, all known to belong to a
"normal" class (e.g., dogs), we show how to train a deep neural model that can
detect out-of-distribution images (i.e., non-dog objects). The main idea behind
our scheme is to train a multi-class model to discriminate between dozens of
geometric transformations applied on all the given images. The auxiliary
expertise learned by the model generates feature detectors that effectively
identify, at test time, anomalous images based on the softmax activation
statistics of the model when applied on transformed images. We present
extensive experiments using the proposed detector, which indicate that our
algorithm improves state-of-the-art methods by a wide margin.
Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification
Authors: Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, Maurizio Filippone
Date: 2018-05-28 00:00:00
Proceeding: NeurIPS 2018 12
Tasks: Classification, Gaussian Processes, General Classification, Uncertainty Quantification

In this paper, we study the problem of deriving fast and accurate
classification algorithms with uncertainty quantification. Gaussian process
classification provides a principled approach, but the corresponding
computational burden is hardly sustainable in large-scale problems and devising
efficient alternatives is a challenge. In this work, we investigate if and how
Gaussian process regression directly applied to the classification labels can
be used to tackle this question. While in this case training time is remarkably
faster, predictions need be calibrated for classification and uncertainty
estimation. To this aim, we propose a novel approach based on interpreting the
labels as the output of a Dirichlet distribution. Extensive experimental
results show that the proposed approach provides essentially the same accuracy
and uncertainty quantification of Gaussian process classification while
requiring only a fraction of computational resources.
Hierarchical clustering with deep Q-learning
Authors: Richard Forster, Agnes Fulop
Date: 2018-05-28 00:00:00
Tasks: Clustering, Q-Learning, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

The reconstruction and analyzation of high energy particle physics data is
just as important as the analyzation of the structure in real world networks.
In a previous study it was explored how hierarchical clustering algorithms can
be combined with kt cluster algorithms to provide a more generic clusterization
method. Building on that, this paper explores the possibilities to involve deep
learning in the process of cluster computation, by applying reinforcement
learning techniques. The result is a model, that by learning on a modest
dataset of 10; 000 nodes during 70 epochs can reach 83; 77% precision in
predicting the appropriate clusters.
Machine learning for prediction of extreme statistics in modulation instability
Authors: Mikko Närhi, Lauri Salmela, Juha Toivonen, Cyril Billet, John M. Dudley, Goëry Genty
Date: 2018-05-28 00:00:00
Tasks: BIG-bench Machine Learning

A central area of research in nonlinear science is the study of instabilities
that drive the emergence of extreme events. Unfortunately, experimental
techniques for measuring such phenomena often provide only partial
characterization. For example, real-time studies of instabilities in nonlinear
fibre optics frequently use only spectral data, precluding detailed predictions
about the associated temporal properties. Here, we show how Machine Learning
can overcome this limitation by predicting statistics for the maximum intensity
of temporal peaks in modulation instability based only on spectral
measurements. Specifically, we train a neural network based Machine Learning
model to correlate spectral and temporal properties of optical fibre modulation
instability using data from numerical simulations, and we then use this model
to predict the temporal probability distribution based on high-dynamic range
spectral data from experiments. These results open novel perspectives in all
systems exhibiting chaos and instability where direct time-domain observations
are difficult.
Hyper-Hue and EMAP on Hyperspectral Images for Supervised Layer Decomposition of Old Master Drawings
Authors: AmirAbbas Davari, Nikolaos Sakaltras, Armin Haeberle, Sulaiman Vesal, Vincent Christlein, Andreas Maier, Christian Riess
Date: 2018-01-29 00:00:00
Tasks: Attribute

Old master drawings were mostly created step by step in several layers using
different materials. To art historians and restorers, examination of these
layers brings various insights into the artistic work process and helps to
answer questions about the object, its attribution and its authenticity.
However, these layers typically overlap and are oftentimes difficult to
differentiate with the unaided eye. For example, a common layer combination is
red chalk under ink.
  In this work, we propose an image processing pipeline that operates on
hyperspectral images to separate such layers. Using this pipeline, we show that
hyperspectral images enable better layer separation than RGB images, and that
spectral focus stacking aids the layer separation. In particular, we propose to
use two descriptors in hyperspectral historical document analysis, namely
hyper-hue and extended multi-attribute profile (EMAP). Our comparative results
with other features underline the efficacy of the three proposed improvements.
Mapping the Americanization of English in Space and Time
Authors: Bruno Gonçalves, Lucía Loureiro-Porto, José J. Ramasco, David Sánchez
Date: 2017-07-03 00:00:00

As global political preeminence gradually shifted from the United Kingdom to
the United States, so did the capacity to culturally influence the rest of the
world. In this work, we analyze how the world-wide varieties of written English
are evolving. We study both the spatial and temporal variations of vocabulary
and spelling of English using a large corpus of geolocated tweets and the
Google Books datasets corresponding to books published in the US and the UK.
The advantage of our approach is that we can address both standard written
language (Google Books) and the more colloquial forms of microblogging messages
(Twitter). We find that American English is the dominant form of English
outside the UK and that its influence is felt even within the UK borders.
Finally, we analyze how this trend has evolved over time and the impact that
some cultural events have had in shaping it.
Glassy nature of the hard phase in inference problems
Authors: Fabrizio Antenucci, Silvio Franz, Pierfrancesco Urbani, Lenka Zdeborová
Date: 2018-05-15 00:00:00

An algorithmically hard phase was described in a range of inference problems:
even if the signal can be reconstructed with a small error from an information
theoretic point of view, known algorithms fail unless the noise-to-signal ratio
is sufficiently small. This hard phase is typically understood as a metastable
branch of the dynamical evolution of message passing algorithms. In this work
we study the metastable branch for a prototypical inference problem, the
low-rank matrix factorization, that presents a hard phase. We show that for
noise-to-signal ratios that are below the information theoretic threshold, the
posterior measure is composed of an exponential number of metastable glassy
states and we compute their entropy, called the complexity. We show that this
glassiness extends even slightly below the algorithmic threshold below which
the well-known approximate message passing (AMP) algorithm is able to closely
reconstruct the signal. Counter-intuitively, we find that the performance of
the AMP algorithm is not improved by taking into account the glassy nature of
the hard phase. This result provides further evidence that the hard phase in
inference problems is algorithmically impenetrable for some deep computational
reasons that remain to be uncovered.
Block-optimized Variable Bit Rate Neural Image Compression
Authors: Caglar Aytekin, Xingyang Ni, Francesco Cricri, Jani Lainema, Emre Aksu, Miska Hannuksela
Date: 2018-05-28 00:00:00
Tasks: Binarization, Image Compression

In this work, we propose an end-to-end block-based auto-encoder system for
image compression. We introduce novel contributions to neural-network based
image compression, mainly in achieving binarization simulation, variable bit
rates with multiple networks, entropy-friendly representations, inference-stage
code optimization and performance-improving normalization layers in the
auto-encoder. We evaluate and show the incremental performance increase of each
of our contributions.
Importance Weighted Transfer of Samples in Reinforcement Learning
Authors: Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, Marcello Restelli
Date: 2018-05-28 00:00:00
Proceeding: ICML 2018 7
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL)

We consider the transfer of experience samples (i.e., tuples < s, a, s', r >)
in reinforcement learning (RL), collected from a set of source tasks to improve
the learning process in a given target task. Most of the related approaches
focus on selecting the most relevant source samples for solving the target
task, but then all the transferred samples are used without considering anymore
the discrepancies between the task models. In this paper, we propose a
model-based technique that automatically estimates the relevance (importance
weight) of each source sample for solving the target task. In the proposed
approach, all the samples are transferred and used by a batch RL algorithm to
solve the target task, but their contribution to the learning process is
proportional to their importance weight. By extending the results for
importance weighting provided in supervised learning literature, we develop a
finite-sample analysis of the proposed batch RL algorithm. Furthermore, we
empirically compare the proposed algorithm to state-of-the-art approaches,
showing that it achieves better learning performance and is very robust to
negative transfer, even when some source tasks are significantly different from
the target task.
Image Distortion Detection using Convolutional Neural Network
Authors: Namhyuk Ahn, Byungkon Kang, Kyung-Ah Sohn
Date: 2018-05-28 00:00:00
Tasks: General Classification, Image Compression

Image distortion classification and detection is an important task in many
applications. For example when compressing images, if we know the exact
location of the distortion, then it is possible to re-compress images by
adjusting the local compression level dynamically. In this paper, we address
the problem of detecting the distortion region and classifying the distortion
type of a given image. We show that our model significantly outperforms the
state-of-the-art distortion classifier, and report accurate detection results
for the first time. We expect that such results prove the usefulness of our
approach in many potential applications such as image compression or distortion
restoration.
Investigating Label Noise Sensitivity of Convolutional Neural Networks for Fine Grained Audio Signal Labelling
Authors: Rainer Kelz, Gerhard Widmer
Date: 2018-05-28 00:00:00
Tasks: Sensitivity

We measure the effect of small amounts of systematic and random label noise
caused by slightly misaligned ground truth labels in a fine grained audio
signal labeling task. The task we choose to demonstrate these effects on is
also known as framewise polyphonic transcription or note quantized multi-f0
estimation, and transforms a monaural audio signal into a sequence of note
indicator labels. It will be shown that even slight misalignments have clearly
apparent effects, demonstrating a great sensitivity of convolutional neural
networks to label noise. The implications are clear: when using convolutional
neural networks for fine grained audio signal labeling tasks, great care has to
be taken to ensure that the annotations have precise timing, and are free from
systematic or random error as much as possible - even small misalignments will
have a noticeable impact.