pwc-archive/papers-with-abstracts huggingface dataset:
Numerically Recovering the Critical Points of a Deep Linear Autoencoder
Authors: Charles G. Frye, Neha S. Wadia, Michael R. DeWeese, Kristofer E. Bouchard
Date: 2019-01-29 00:00:00

Numerically locating the critical points of non-convex surfaces is a
long-standing problem central to many fields. Recently, the loss surfaces of
deep neural networks have been explored to gain insight into outstanding
questions in optimization, generalization, and network architecture design.
However, the degree to which recently-proposed methods for numerically
recovering critical points actually do so has not been thoroughly evaluated. In
this paper, we examine this issue in a case for which the ground truth is
known: the deep linear autoencoder. We investigate two sub-problems associated
with numerical critical point identification: first, because of large parameter
counts, it is infeasible to find all of the critical points for contemporary
neural networks, necessitating sampling approaches whose characteristics are
poorly understood; second, the numerical tolerance for accurately identifying a
critical point is unknown, and conservative tolerances are difficult to
satisfy. We first identify connections between recently-proposed methods and
well-understood methods in other fields, including chemical physics, economics,
and algebraic geometry. We find that several methods work well at recovering
certain information about loss surfaces, but fail to take an unbiased sample of
critical points. Furthermore, numerical tolerance must be very strict to ensure
that numerically-identified critical points have similar properties to true
analytical critical points. We also identify a recently-published Newton method
for optimization that outperforms previous methods as a critical point-finding
algorithm. We expect our results will guide future attempts to numerically
study critical points in large nonlinear neural networks.
Decentralized Online Learning: Take Benefits from Others' Data without Sharing Your Own to Track Global Trend
Authors: Yawei Zhao, Chen Yu, Peilin Zhao, Hanlin Tang, Shuang Qiu, Ji Liu
Date: 2019-01-29 00:00:00

Decentralized Online Learning (online learning in decentralized networks) attracts more and more attention, since it is believed that Decentralized Online Learning can help the data providers cooperatively better solve their online problems without sharing their private data to a third party or other providers. Typically, the cooperation is achieved by letting the data providers exchange their models between neighbors, e.g., recommendation model. However, the best regret bound for a decentralized online learning algorithm is $\Ocal{n\sqrt{T}}$, where $n$ is the number of nodes (or users) and $T$ is the number of iterations. This is clearly insignificant since this bound can be achieved \emph{without} any communication in the networks. This reminds us to ask a fundamental question: \emph{Can people really get benefit from the decentralized online learning by exchanging information?} In this paper, we studied when and why the communication can help the decentralized online learning to reduce the regret. Specifically, each loss function is characterized by two components: the adversarial component and the stochastic component. Under this characterization, we show that decentralized online gradient (DOG) enjoys a regret bound $\Ocal{n\sqrt{T}G + \sqrt{nT}\sigma}$, where $G$ measures the magnitude of the adversarial component in the private data (or equivalently the local loss function) and $\sigma$ measures the randomness within the private data. This regret suggests that people can get benefits from the randomness in the private data by exchanging private information. Another important contribution of this paper is to consider the dynamic regret -- a more practical regret to track users' interest dynamics. Empirical studies are also conducted to validate our analysis.
Throttling Malware Families in 2D
Authors: Mohamed Nassar, Haidar Safa
Date: 2019-01-29 00:00:00
Tasks: Clustering, Dimensionality Reduction, valid

Malicious software are categorized into families based on their static and
dynamic characteristics, infection methods, and nature of threat. Visual
exploration of malware instances and families in a low dimensional space helps
in giving a first overview about dependencies and relationships among these
instances, detecting their groups and isolating outliers. Furthermore, visual
exploration of different sets of features is useful in assessing the quality of
these sets to carry a valid abstract representation, which can be later used in
classification and clustering algorithms to achieve a high accuracy. In this
paper, we investigate one of the best dimensionality reduction techniques known
as t-SNE to reduce the malware representation from a high dimensional space
consisting of thousands of features to a low dimensional space. We experiment
with different feature sets and depict malware clusters in 2-D. Surprisingly,
t-SNE does not only provide nice 2-D drawings, but also dramatically increases
the generalization power of SVM classifiers. Moreover, obtained results showed
that cross-validation accuracy is much better using the 2-D embedded
representation of samples than using the original high-dimensional
representation.
Rare geometries: revealing rare categories via dimension-driven statistics
Authors: Henry Kvinge, Elin Farnell, Jingya Li, Yujia Chen
Date: 2019-01-29 00:00:00
Tasks: Translation

In many situations, classes of data points of primary interest also happen to be those that are least numerous. A well-known example is detection of fraudulent transactions among the collection of all financial transactions, the vast majority of which are legitimate. These types of problems fall under the label of `rare-category detection.' There are two challenging aspects of these problems. The first is a general lack of labeled examples of the rare class and the second is the potential non-separability of the rare class from the majority (in terms of available features). Statistics related to the geometry of the rare class (such as its intrinsic dimension) can be significantly different from those for the majority class, reflecting the different dynamics driving variation in the different classes. In this paper we present a new supervised learning algorithm that uses a dimension-driven statistic, called the kappa-profile, to classify whether unlabeled points belong to a rare class. Our algorithm requires very few labeled examples and is invariant with respect to translation so that it performs equivalently on both separable and non-separable classes.
Trading-off Accuracy and Energy of Deep Inference on Embedded Systems: A Co-Design Approach
Authors: Nitthilan Kannappan Jayakodi, Anwesha Chatterjee, Wonje Choi, Janardhan Rao Doppa, Partha Pratim Pande
Date: 2019-01-29 00:00:00
Tasks: image-classification, Image Classification

Deep neural networks have seen tremendous success for different modalities of
data including images, videos, and speech. This success has led to their
deployment in mobile and embedded systems for real-time applications. However,
making repeated inferences using deep networks on embedded systems poses
significant challenges due to constrained resources (e.g., energy and computing
power). To address these challenges, we develop a principled co-design
approach. Building on prior work, we develop a formalism referred to as
Coarse-to-Fine Networks (C2F Nets) that allow us to employ classifiers of
varying complexity to make predictions. We propose a principled optimization
algorithm to automatically configure C2F Nets for a specified trade-off between
accuracy and energy consumption for inference. The key idea is to select a
classifier on-the-fly whose complexity is proportional to the hardness of the
input example: simple classifiers for easy inputs and complex classifiers for
hard inputs. We perform comprehensive experimental evaluation using four
different C2F Net architectures on multiple real-world image classification
tasks. Our results show that optimized C2F Net can reduce the Energy Delay
Product (EDP) by 27 to 60 percent with no loss in accuracy when compared to the
baseline solution, where all predictions are made using the most complex
classifier in C2F Net.
Stochastic Gradient MCMC for Nonlinear State Space Models
Authors: Christopher Aicher, Srshti Putcha, Christopher Nemeth, Paul Fearnhead, Emily B. Fox
Date: 2019-01-29 00:00:00
Tasks: Bayesian Inference, State Space Models, Time Series, Time Series Analysis

State space models (SSMs) provide a flexible framework for modeling complex time series via a latent stochastic process. Inference for nonlinear, non-Gaussian SSMs is often tackled with particle methods that do not scale well to long time series. The challenge is two-fold: not only do computations scale linearly with time, as in the linear case, but particle filters additionally suffer from increasing particle degeneracy with longer series. Stochastic gradient MCMC methods have been developed to scale Bayesian inference for finite-state hidden Markov models and linear SSMs using buffered stochastic gradient estimates to account for temporal dependencies. We extend these stochastic gradient estimators to nonlinear SSMs using particle methods. We present error bounds that account for both buffering error and particle error in the case of nonlinear SSMs that are log-concave in the latent process. We evaluate our proposed particle buffered stochastic gradient using stochastic gradient MCMC for inference on both long sequential synthetic and minute-resolution financial returns data, demonstrating the importance of this class of methods.
Personalized Treatment Selection using Causal Heterogeneity
Authors: Ye Tu, Kinjal Basu, Cyrus DiCiccio, Romil Bansal, Preetam Nandy, Padmini Jaikumar, Shaunak Chatterjee
Date: 2019-01-29 00:00:00
Tasks: Stochastic Optimization

Randomized experimentation (also known as A/B testing or bucket testing) is widely used in the internet industry to measure the metric impact obtained by different treatment variants. A/B tests identify the treatment variant showing the best performance, which then becomes the chosen or selected treatment for the entire population. However, the effect of a given treatment can differ across experimental units and a personalized approach for treatment selection can greatly improve upon the usual global selection strategy. In this work, we develop a framework for personalization through (i) estimation of heterogeneous treatment effect at either a cohort or member-level, followed by (ii) selection of optimal treatment variants for cohorts (or members) obtained through (deterministic or stochastic) constrained optimization. We perform a two-fold evaluation of our proposed methods. First, a simulation analysis is conducted to study the effect of personalized treatment selection under carefully controlled settings. This simulation illustrates the differences between the proposed methods and the suitability of each with increasing uncertainty. We also demonstrate the effectiveness of the method through a real-life example related to serving notifications at Linkedin. The solution significantly outperformed both heuristic solutions and the global treatment selection baseline leading to a sizable win on top-line metrics like member visits.
Latent Normalizing Flows for Discrete Sequences
Authors: Zachary M. Ziegler, Alexander M. Rush
Date: 2019-01-29 00:00:00
Tasks: Language Modeling, Language Modelling, Music Generation

Normalizing flows are a powerful class of generative models for continuous random variables, showing both strong model flexibility and the potential for non-autoregressive generation. These benefits are also desired when modeling discrete random variables such as text, but directly applying normalizing flows to discrete sequences poses significant additional challenges. We propose a VAE-based generative model which jointly learns a normalizing flow-based distribution in the latent space and a stochastic mapping to an observed discrete space. In this setting, we find that it is crucial for the flow-based distribution to be highly multimodal. To capture this property, we propose several normalizing flow architectures to maximize model flexibility. Experiments consider common discrete sequence tasks of character-level language modeling and polyphonic music generation. Our results indicate that an autoregressive flow-based model can match the performance of a comparable autoregressive baseline, and a non-autoregressive flow-based model can improve generation speed with a penalty to performance.
Comprehensive Evaluation of Deep Learning Architectures for Prediction of DNA/RNA Sequence Binding Specificities
Authors: Ameni Trabelsi, Mohamed Chaabane, Asa Ben Hur
Date: 2019-01-29 00:00:00
Tasks: Automatic Machine Learning Model Selection, Deep Learning, Model Selection, Multi-Label Text Classification, Text Classification

Motivation: Deep learning architectures have recently demonstrated their
power in predicting DNA- and RNA-binding specificities. Existing methods fall
into three classes: Some are based on Convolutional Neural Networks (CNNs),
others use Recurrent Neural Networks (RNNs), and others rely on hybrid
architectures combining CNNs and RNNs. However, based on existing studies it is
still unclear which deep learning architecture is achieving the best
performance. Thus an in-depth analysis and evaluation of the different methods
is needed to fully evaluate their relative. Results: In this study, We present
a systematic exploration of various deep learning architectures for predicting
DNA- and RNA-binding specificities. For this purpose, we present deepRAM, an
end-to-end deep learning tool that provides an implementation of novel and
previously proposed architectures; its fully automatic model selection
procedure allows us to perform a fair and unbiased comparison of deep learning
architectures. We find that an architecture that uses k-mer embedding to
represent the sequence, a convolutional layer and a recurrent layer,
outperforms all other methods in terms of model accuracy. Our work provides
guidelines that will assist the practitioner in choosing the best architecture
for the task at hand, and provides some insights on the differences between the
models learned by convolutional and recurrent networks. In particular, we find
that although recurrent networks improve model accuracy, this comes at the
expense of a loss in the interpretability of the features learned by the model.
Availability and implementation: The source code for deepRAM is available at
https://github.com/MedChaabane/deepRAM
On the Transferability of Spectral Graph Filters
Authors: Ron Levie, Elvin Isufi, Gitta Kutyniok
Date: 2019-01-29 00:00:00

This paper focuses on spectral filters on graphs, namely filters defined as
elementwise multiplication in the frequency domain of a graph. In many graph
signal processing settings, it is important to transfer a filter from one graph
to another. One example is in graph convolutional neural networks (ConvNets),
where the dataset consists of signals defined on many different graphs, and the
learned filters should generalize to signals on new graphs, not present in the
training set. A necessary condition for transferability (the ability to
transfer filters) is stability. Namely, given a graph filter, if we add a small
perturbation to the graph, then the filter on the perturbed graph is a small
perturbation of the original filter. It is a common misconception that spectral
filters are not stable, and this paper aims at debunking this mistake. We
introduce a space of filters, called the Cayley smoothness space, that contains
the filters of state-of-the-art spectral filtering methods, and whose filters
can approximate any generic spectral filter. For filters in this space, the
perturbation in the filter is bounded by a constant times the perturbation in
the graph, and filters in the Cayley smoothness space are thus termed linearly
stable. By combining stability with the known property of equivariance, we
prove that graph spectral filters are transferable.
Spectral Multi-scale Community Detection in Temporal Networks with an Application
Authors: Zhana Kuncheva, Giovanni Montana
Date: 2019-01-29 00:00:00
Tasks: Community Detection

The analysis of temporal networks has a wide area of applications in a world
of technological advances. An important aspect of temporal network analysis is
the discovery of community structures. Real data networks are often very large
and the communities are observed to have a hierarchical structure referred to
as multi-scale communities. Changes in the community structure over time might
take place either at one scale or across all scales of the community structure.
The multilayer formulation of the modularity maximization (MM) method
introduced captures the changing multi-scale community structure of temporal
networks. This method introduces a coupling between communities in neighboring
time layers by allowing inter-layer connections, while different values of the
resolution parameter enable the detection of multi-scale communities. However,
the range of this parameter's values must be manually selected. When dealing
with real life data, communities at one or more scales can go undiscovered if
appropriate parameter ranges are not selected. A novel Temporal Multi-scale
Community Detection (TMSCD) method overcomes the obstacles mentioned above.
This is achieved by using the spectral properties of the temporal network
represented as a multilayer network. In this framework we select automatically
the range of relevant scales within which multi-scale community partitions are
sought.
Reparameterizable Subset Sampling via Continuous Relaxations
Authors: Sang Michael Xie, Stefano Ermon
Date: 2019-01-29 00:00:00
Tasks: feature selection, Stochastic Optimization

Many machine learning tasks require sampling a subset of items from a collection based on a parameterized distribution. The Gumbel-softmax trick can be used to sample a single item, and allows for low-variance reparameterized gradients with respect to the parameters of the underlying distribution. However, stochastic optimization involving subset sampling is typically not reparameterizable. To overcome this limitation, we define a continuous relaxation of subset sampling that provides reparameterization gradients by generalizing the Gumbel-max trick. We use this approach to sample subsets of features in an instance-wise feature selection task for model interpretability, subsets of neighbors to implement a deep stochastic k-nearest neighbors model, and sub-sequences of neighbors to implement parametric t-SNE by directly comparing the identities of local neighbors. We improve performance in all these tasks by incorporating subset sampling in end-to-end training.
Hyperspherical Prototype Networks
Authors: Pascal Mettes, Elise van der Pol, Cees G. M. Snoek
Date: 2019-01-29 00:00:00
Proceeding: NeurIPS 2019 12
Tasks: Classification, General Classification, regression

This paper introduces hyperspherical prototype networks, which unify classification and regression with prototypes on hyperspherical output spaces. For classification, a common approach is to define prototypes as the mean output vector over training examples per class. Here, we propose to use hyperspheres as output spaces, with class prototypes defined a priori with large margin separation. We position prototypes through data-independent optimization, with an extension to incorporate priors from class semantics. By doing so, we do not require any prototype updating, we can handle any training size, and the output dimensionality is no longer constrained to the number of classes. Furthermore, we generalize to regression, by optimizing outputs as an interpolation between two prototypes on the hypersphere. Since both tasks are now defined by the same loss function, they can be jointly trained for multi-task problems. Experimentally, we show the benefit of hyperspherical prototype networks for classification, regression, and their combination over other prototype methods, softmax cross-entropy, and mean squared error approaches.
Adversarial Examples Are a Natural Consequence of Test Error in Noise
Authors: Nic Ford, Justin Gilmer, Nicolas Carlini, Dogus Cubuk
Date: 2019-01-29 00:00:00
Proceeding: ICLR 2019 5
Tasks: Adversarial Robustness

Over the last few years, the phenomenon of adversarial examples ---
maliciously constructed inputs that fool trained machine learning models ---
has captured the attention of the research community, especially when the
adversary is restricted to small modifications of a correctly handled input.
Less surprisingly, image classifiers also lack human-level performance on
randomly corrupted images, such as images with additive Gaussian noise. In this
paper we provide both empirical and theoretical evidence that these are two
manifestations of the same underlying phenomenon, establishing close
connections between the adversarial robustness and corruption robustness
research programs. This suggests that improving adversarial robustness should
go hand in hand with improving performance in the presence of more general and
realistic image corruptions. Based on our results we recommend that future
adversarial defenses consider evaluating the robustness of their methods to
distributional shift with benchmarks such as Imagenet-C.
Time-Space tradeoff in deep learning models for crop classification on satellite multi-spectral image time series
Authors: Vivien Sainte Fare Garnot, Loic Landrieu, Sebastien Giordano, Nesrine Chehata
Date: 2019-01-29 00:00:00
Tasks: Classification, Crop Classification, Deep Learning, General Classification, Time Series, Time Series Analysis

In this article, we investigate several structured deep learning models for
crop type classification on multi-spectral time series. In particular, our aim
is to assess the respective importance of spatial and temporal structures in
such data. With this objective, we consider several designs of convolutional,
recurrent, and hybrid neural networks, and assess their performance on a large
dataset of freely available Sentinel-2 imagery. We find that the
best-performing approaches are hybrid configurations for which most of the
parameters (up to 90%) are allocated to modeling the temporal structure of the
data. Our results thus constitute a set of guidelines for the design of bespoke
deep learning models for crop type classification.
Repairing without Retraining: Avoiding Disparate Impact with Counterfactual Distributions
Authors: Hao Wang, Berk Ustun, Flavio P. Calmon
Date: 2019-01-29 00:00:00
Tasks: counterfactual, Fairness

When the performance of a machine learning model varies over groups defined by sensitive attributes (e.g., gender or ethnicity), the performance disparity can be expressed in terms of the probability distributions of the input and output variables over each group. In this paper, we exploit this fact to reduce the disparate impact of a fixed classification model over a population of interest. Given a black-box classifier, we aim to eliminate the performance gap by perturbing the distribution of input variables for the disadvantaged group. We refer to the perturbed distribution as a counterfactual distribution, and characterize its properties for common fairness criteria. We introduce a descent algorithm to learn a counterfactual distribution from data. We then discuss how the estimated distribution can be used to build a data preprocessor that can reduce disparate impact without training a new model. We validate our approach through experiments on real-world datasets, showing that it can repair different forms of disparity without a significant drop in accuracy.