pwc-archive/papers-with-abstracts huggingface dataset:
Activation Adaptation in Neural Networks
Authors: Farnoush Farhadi, Vahid Partovi Nia, Andrea Lodi
Date: 2019-01-28 00:00:00

Many neural network architectures rely on the choice of the activation function for each hidden layer. Given the activation function, the neural network is trained over the bias and the weight parameters. The bias catches the center of the activation, and the weights capture the scale. Here we propose to train the network over a shape parameter as well. This view allows each neuron to tune its own activation function and adapt the neuron curvature towards a better prediction. This modification only adds one further equation to the back-propagation for each neuron. Re-formalizing activation functions as CDF generalizes the class of activation function extensively. We aimed at generalizing an extensive class of activation functions to study: i) skewness and ii) smoothness of activation functions. Here we introduce adaptive Gumbel activation function as a bridge between Gumbel and sigmoid. A similar approach is used to invent a smooth version of ReLU. Our comparison with common activation functions suggests different data representation especially in early neural network layers. This adaptation also provides prediction improvement.
A new evaluation framework for topic modeling algorithms based on synthetic corpora
Authors: Hanyu Shi, Martin Gerlach, Isabel Diersen, Doug Downey, Luis A. N. Amaral
Date: 2019-01-28 00:00:00
Tasks: General Classification, Topic Models

Topic models are in widespread use in natural language processing and beyond.
Here, we propose a new framework for the evaluation of probabilistic topic
modeling algorithms based on synthetic corpora containing an unambiguously
defined ground truth topic structure. The major innovation of our approach is
the ability to quantify the agreement between the planted and inferred topic
structures by comparing the assigned topic labels at the level of the tokens.
In experiments, our approach yields novel insights about the relative strengths
of topic models as corpus characteristics vary, and the first evidence of an
"undetectable phase" for topic models when the planted structure is weak. We
also establish the practical relevance of the insights gained for synthetic
corpora by predicting the performance of topic modeling algorithms in
classification tasks in real-world corpora.
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
Authors: Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian U. Stich, Martin Jaggi
Date: 2019-01-28 00:00:00

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does converge, signSGD may generalize poorly when compared with SGD. These issues arise because of the biased nature of the sign compression operator. We then show that using error-feedback, i.e. incorporating the error made by the compression operator into the next step, overcomes these issues. We prove that our algorithm EF-SGD with arbitrary compression operator achieves the same rate of convergence as SGD without any additional assumptions. Thus EF-SGD achieves gradient compression for free. Our experiments thoroughly substantiate the theory and show that error-feedback improves both convergence and generalization. Code can be found at \url{https://github.com/epfml/error-feedback-SGD}.
Interpreting Deep Neural Networks Through Variable Importance
Authors: Jonathan Ish-Horowicz, Dana Udwin, Seth Flaxman, Sarah Filippi, Lorin Crawford
Date: 2019-01-28 00:00:00

While the success of deep neural networks (DNNs) is well-established across a variety of domains, our ability to explain and interpret these methods is limited. Unlike previously proposed local methods which try to explain particular classification decisions, we focus on global interpretability and ask a universally applicable question: given a trained model, which features are the most important? In the context of neural networks, a feature is rarely important on its own, so our strategy is specifically designed to leverage partial covariance structures and incorporate variable dependence into feature ranking. Our methodological contributions in this paper are two-fold. First, we propose an effect size analogue for DNNs that is appropriate for applications with highly collinear predictors (ubiquitous in computer vision). Second, we extend the recently proposed "RelATive cEntrality" (RATE) measure (Crawford et al., 2019) to the Bayesian deep learning setting. RATE applies an information theoretic criterion to the posterior distribution of effect sizes to assess feature significance. We apply our framework to three broad application areas: computer vision, natural language processing, and social science.
Semi-supervised Learning in Network-Structured Data via Total Variation Minimization
Authors: Alexander Jung, Alfred O. Hero III, Alexandru Mara, Saeed Jahromi, Ayelet Heimowitz, Yonina C. Eldar
Date: 2019-01-28 00:00:00
Tasks: Clustering, compressed sensing

We propose and analyze a method for semi-supervised learning from partially-labeled network-structured data. Our approach is based on a graph signal recovery interpretation under a clustering hypothesis that labels of data points belonging to the same well-connected subset (cluster) are similar valued. This lends naturally to learning the labels by total variation (TV) minimization, which we solve by applying a recently proposed primal-dual method for non-smooth convex optimization. The resulting algorithm allows for a highly scalable implementation using message passing over the underlying empirical graph, which renders the algorithm suitable for big data applications. By applying tools of compressed sensing, we derive a sufficient condition on the underlying network structure such that TV minimization recovers clusters in the empirical graph of the data. In particular, we show that the proposed primal-dual method amounts to maximizing network flows over the empirical graph of the dataset. Moreover, the learning accuracy of the proposed algorithm is linked to the set of network flows between data points having known labels. The effectiveness and scalability of our approach is verified by numerical experiments.
Depth creates no more spurious local minima
Authors: Li Zhang
Date: 2019-01-28 00:00:00

We show that for any convex differentiable loss, a deep linear network has no spurious local minima as long as it is true for the two layer case. This reduction greatly simplifies the study on the existence of spurious local minima in deep linear networks. When applied to the quadratic loss, our result immediately implies the powerful result in [Kawaguchi 2016]. Further, with the work in [Zhou and Liang 2018], we can remove all the assumptions in [Kawaguchi 2016]. This property holds for more general "multi-tower" linear networks too. Our proof builds on [Laurent and von Brecht 2018] and develops a new perturbation argument to show that any spurious local minimum must have full rank, a structural property which can be useful more generally.
Squeezed Very Deep Convolutional Neural Networks for Text Classification
Authors: Andréa B. Duque, Luã Lázaro J. Santos, David Macêdo, Cleber Zanchettin
Date: 2019-01-28 00:00:00
Tasks: Classification, General Classification, Sentiment Analysis, text-classification, Text Classification

Most of the research in convolutional neural networks has focused on
increasing network depth to improve accuracy, resulting in a massive number of
parameters which restricts the trained network to platforms with memory and
processing constraints. We propose to modify the structure of the Very Deep
Convolutional Neural Networks (VDCNN) model to fit mobile platforms constraints
and keep performance. In this paper, we evaluate the impact of Temporal
Depthwise Separable Convolutions and Global Average Pooling in the network
parameters, storage size, and latency. The squeezed model (SVDCNN) is between
10x and 20x smaller, depending on the network depth, maintaining a maximum size
of 6MB. Regarding accuracy, the network experiences a loss between 0.4% and
1.3% and obtains lower latencies compared to the baseline model.
Generalization of feature embeddings transferred from different video anomaly detection domains
Authors: Fernando Pereira dos Santos, Leonardo Sampaio Ferraz Ribeiro, Moacir Antonelli Ponti
Date: 2019-01-28 00:00:00
Tasks: Anomaly Detection, Domain Generalization, Transfer Learning, Video Anomaly Detection

Detecting anomalous activity in video surveillance often involves using only
normal activity data in order to learn an accurate detector. Due to lack of
annotated data for some specific target domain, one could employ existing data
from a source domain to produce better predictions. Hence, transfer learning
presents itself as an important tool. But how to analyze the resulting data
space? This paper investigates video anomaly detection, in particular feature
embeddings of pre-trained CNN that can be used with non-fully supervised data.
By proposing novel cross-domain generalization measures, we study how source
features can generalize for different target video domains, as well as analyze
unsupervised transfer learning. The proposed generalization measures are not
only a theorical approach, but show to be useful in practice as a way to
understand which datasets can be used or transferred to describe video frames,
which it is possible to better discriminate between normal and anomalous
activity.
Analogies Explained: Towards Understanding Word Embeddings
Authors: Carl Allen, Timothy Hospedales
Date: 2019-01-28 00:00:00
Tasks: Word Embeddings

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "$w_x$ is to $w_y$". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.
Evaluating Word Embedding Models: Methods and Experimental Results
Authors: Bin Wang, Angela Wang, Fenxiao Chen, Yuncheng Wang, C. -C. Jay Kuo
Date: 2019-01-28 00:00:00
Tasks: Word Embeddings

Extensive evaluation on a large number of word embedding models for language
processing applications is conducted in this work. First, we introduce popular
word embedding models and discuss desired properties of word models and
evaluation methods (or evaluators). Then, we categorize evaluators into
intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a
representation independent of specific natural language processing tasks while
extrinsic evaluators use word embeddings as input features to a downstream task
and measure changes in performance metrics specific to that task. We report
experimental results of intrinsic and extrinsic evaluators on six word
embedding models. It is shown that different evaluators focus on different
aspects of word models, and some are more correlated with natural language
processing tasks. Finally, we adopt correlation analysis to study performance
consistency of extrinsic and intrinsic evalutors.
Leveraging Outdoor Webcams for Local Descriptor Learning
Authors: Milan Pultar, Dmytro Mishkin, Jiří Matas
Date: 2019-01-28 00:00:00
Tasks: Clustering

We present AMOS Patches, a large set of image cut-outs, intended primarily for the robustification of trainable local feature descriptors to illumination and appearance changes. Images contributing to AMOS Patches originate from the AMOS dataset of recordings from a large set of outdoor webcams. The semiautomatic method used to generate AMOS Patches is described. It includes camera selection, viewpoint clustering and patch selection. For training, we provide both the registered full source images as well as the patches. A new descriptor, trained on the AMOS Patches and 6Brown datasets, is introduced. It achieves state-of-the-art in matching under illumination changes on standard benchmarks.
Attribute-Guided Sketch Generation
Authors: Hao Tang, Xinya Chen, Wei Wang, Dan Xu, Jason J. Corso, Nicu Sebe, Yan Yan
Date: 2019-01-28 00:00:00
Tasks: Attribute, Generative Adversarial Network, Translation

Facial attributes are important since they provide a detailed description and
determine the visual appearance of human faces. In this paper, we aim at
converting a face image to a sketch while simultaneously generating facial
attributes. To this end, we propose a novel Attribute-Guided Sketch Generative
Adversarial Network (ASGAN) which is an end-to-end framework and contains two
pairs of generators and discriminators, one of which is used to generate faces
with attributes while the other one is employed for image-to-sketch
translation. The two generators form a W-shaped network (W-net) and they are
trained jointly with a weight-sharing constraint. Additionally, we also propose
two novel discriminators, the residual one focusing on attribute generation and
the triplex one helping to generate realistic looking sketches. To validate our
model, we have created a new large dataset with 8,804 images, named the
Attribute Face Photo & Sketch (AFPS) dataset which is the first dataset
containing attributes associated to face sketch images. The experimental
results demonstrate that the proposed network (i) generates more
photo-realistic faces with sharper facial attributes than baselines and (ii)
has good generalization capability on different generative tasks.
CollaGAN : Collaborative GAN for Missing Image Data Imputation
Authors: Dongwook Lee, Junyoung Kim, Won-Jin Moon, Jong Chul Ye
Date: 2019-01-28 00:00:00
Tasks: Generative Adversarial Network, Image Imputation, Imputation, Translation

In many applications requiring multiple inputs to obtain a desired output, if
any of the input data is missing, it often introduces large amounts of bias.
Although many techniques have been developed for imputing missing data, the
image imputation is still difficult due to complicated nature of natural
images. To address this problem, here we proposed a novel framework for missing
image data imputation, called Collaborative Generative Adversarial Network
(CollaGAN). CollaGAN converts an image imputation problem to a multi-domain
images-to-image translation task so that a single generator and discriminator
network can successfully estimate the missing data using the remaining clean
data set. We demonstrate that CollaGAN produces the images with a higher visual
quality compared to the existing competing approaches in various image
imputation tasks.
Language Independent Sequence Labelling for Opinion Target Extraction
Authors: Rodrigo Agerri, German Rigau
Date: 2019-01-28 00:00:00
Tasks: Aspect-Based Sentiment Analysis, Aspect-Based Sentiment Analysis (ABSA), Clustering, Sentiment Analysis

In this research note we present a language independent system to model
Opinion Target Extraction (OTE) as a sequence labelling task. The system
consists of a combination of clustering features implemented on top of a simple
set of shallow local features. Experiments on the well known Aspect Based
Sentiment Analysis (ABSA) benchmarks show that our approach is very competitive
across languages, obtaining best results for six languages in seven different
datasets. Furthermore, the results provide further insights into the behaviour
of clustering features for sequence labelling tasks. The system and models
generated in this work are available for public use and to facilitate
reproducibility of results.
Fairwashing: the risk of rationalization
Authors: Ulrich Aïvodji, Hiromi Arai, Olivier Fortineau, Sébastien Gambs, Satoshi Hara, Alain Tapp
Date: 2019-01-28 00:00:00
Tasks: BIG-bench Machine Learning, Fairness

Black-box explanation is the problem of explaining how a machine learning model -- whose internal logic is hidden to the auditor and generally complex -- produces its outcomes. Current approaches for solving this problem include model explanation, outcome explanation as well as model inspection. While these techniques can be beneficial by providing interpretability, they can be used in a negative manner to perform fairwashing, which we define as promoting the false perception that a machine learning model respects some ethical values. In particular, we demonstrate that it is possible to systematically rationalize decisions taken by an unfair black-box model using the model explanation as well as the outcome explanation approaches with a given fairness metric. Our solution, LaundryML, is based on a regularized rule list enumeration algorithm whose objective is to search for fair rule lists approximating an unfair black-box model. We empirically evaluate our rationalization technique on black-box models trained on real-world datasets and show that one can obtain rule lists with high fidelity to the black-box model while being considerably less unfair at the same time.
Making Deep Q-learning methods robust to time discretization
Authors: Corentin Tallec, Léonard Blier, Yann Ollivier
Date: 2019-01-28 00:00:00
Tasks: Deep Reinforcement Learning, Q-Learning, Reinforcement Learning, Sensitivity

Despite remarkable successes, Deep Reinforcement Learning (DRL) is not robust
to hyperparameterization, implementation details, or small environment changes
(Henderson et al. 2017, Zhang et al. 2018). Overcoming such sensitivity is key
to making DRL applicable to real world problems. In this paper, we identify
sensitivity to time discretization in near continuous-time environments as a
critical factor; this covers, e.g., changing the number of frames per second,
or the action frequency of the controller. Empirically, we find that
Q-learning-based approaches such as Deep Q- learning (Mnih et al., 2015) and
Deep Deterministic Policy Gradient (Lillicrap et al., 2015) collapse with small
time steps. Formally, we prove that Q-learning does not exist in continuous
time. We detail a principled way to build an off-policy RL algorithm that
yields similar performances over a wide range of time discretizations, and
confirm this robustness empirically.
Edge, Ridge, and Blob Detection with Symmetric Molecules
Authors: Rafael Reisenhofer, Emily J. King
Date: 2019-01-28 00:00:00

We present a novel approach to the detection and characterization of edges, ridges, and blobs in two-dimensional images which exploits the symmetry properties of directionally sensitive analyzing functions in multiscale systems that are constructed in the framework of alpha-molecules. The proposed feature detectors are inspired by the notion of phase congruency, stable in the presence of noise, and by definition invariant to changes in contrast. We also show how the behavior of coefficients corresponding to differently scaled and oriented analyzing functions can be used to obtain a comprehensive characterization of the geometry of features in terms of local tangent directions, widths, and heights. The accuracy and robustness of the proposed measures are validated and compared to various state-of-the-art algorithms in extensive numerical experiments in which we consider sets of clean and distorted synthetic images that are associated with reliable ground truths. To further demonstrate the applicability, we show how the proposed ridge measure can be used to detect and characterize blood vessels in digital retinal images and how the proposed blob measure can be applied to automatically count the number of cell colonies in a Petri dish.
CLIC: Curriculum Learning and Imitation for object Control in non-rewarding environments
Authors: Pierre Fournier, Olivier Sigaud, Cédric Colas, Mohamed Chetouani
Date: 2019-01-28 00:00:00
Tasks: reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Unsupervised Reinforcement Learning

In this paper we study a new reinforcement learning setting where the
environment is non-rewarding, contains several possibly related objects of
various controllability, and where an apt agent Bob acts independently, with
non-observable intentions. We argue that this setting defines a realistic
scenario and we present a generic discrete-state discrete-action model of such
environments. To learn in this environment, we propose an unsupervised
reinforcement learning agent called CLIC for Curriculum Learning and Imitation
for Control. CLIC learns to control individual objects in its environment, and
imitates Bob's interactions with these objects. It selects objects to focus on
when training and imitating by maximizing its learning progress. We show that
CLIC is an effective baseline in our new setting. It can effectively observe
Bob to gain control of objects faster, even if Bob is not explicitly teaching.
It can also follow Bob when he acts as a mentor and provides ordered
demonstrations. Finally, when Bob controls objects that the agent cannot, or in
presence of a hierarchy between objects in the environment, we show that CLIC
ignores non-reproducible and already mastered interactions with objects,
resulting in a greater benefit from imitation.
Ising Models with Latent Conditional Gaussian Variables
Authors: Frank Nussbaum, Joachim Giesen
Date: 2019-01-28 00:00:00

Ising models describe the joint probability distribution of a vector of binary feature variables. Typically, not all the variables interact with each other and one is interested in learning the presumably sparse network structure of the interacting variables. However, in the presence of latent variables, the conventional method of learning a sparse model might fail. This is because the latent variables induce indirect interactions of the observed variables. In the case of only a few latent conditional Gaussian variables these spurious interactions contribute an additional low-rank component to the interaction parameters of the observed Ising model. Therefore, we propose to learn a sparse + low-rank decomposition of the parameters of an Ising model using a convex regularized likelihood problem. We show that the same problem can be obtained as the dual of a maximum-entropy problem with a new type of relaxation, where the sample means collectively need to match the expected values only up to a given tolerance. The solution to the convex optimization problem has consistency properties in the high-dimensional setting, where the number of observed binary variables and the number of latent conditional Gaussian variables are allowed to grow with the number of training samples.
Bayesian Differential Privacy for Machine Learning
Authors: Aleksei Triastcyn, Boi Faltings
Date: 2019-01-28 00:00:00
Proceeding: ICML 2020 1
Tasks: BIG-bench Machine Learning

Traditional differential privacy is independent of the data distribution. However, this is not well-matched with the modern machine learning context, where models are trained on specific data. As a result, achieving meaningful privacy guarantees in ML often excessively reduces accuracy. We propose Bayesian differential privacy (BDP), which takes into account the data distribution to provide more practical privacy guarantees. We also derive a general privacy accounting method under BDP, building upon the well-known moments accountant. Our experiments demonstrate that in-distribution samples in classic machine learning datasets, such as MNIST and CIFAR-10, enjoy significantly stronger privacy guarantees than postulated by DP, while models maintain high classification accuracy.