pwc-archive/papers-with-abstracts huggingface dataset:
Online Continual Learning from Imbalanced Data
Authors: Aristotelis Chrysakis, Marie-Francine Moens
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Computational Efficiency, Continual Learning

A well-documented weakness of neural networks is the fact that they suffer from catastrophic forgetting when trained on data provided by a non-stationary distribution. Recent work in the field of continual learning attempts to understand and overcome this issue. Unfortunately, the majority of relevant work embraces the implicit assumption that the distribution of observed data is perfectly balanced. In contrast, humans and animals learn from observations that are temporally correlated and severely imbalanced. Motivated by this remark, we aim to evaluate memory population methods that are used in online continual learning, when dealing with highly imbalanced and temporally correlated streams of data. More importantly, we introduce a new memory population approach, which we call class-balancing reservoir sampling (CBRS). We demonstrate that CBRS outperforms the state-of-the-art memory population algorithms in a considerably challenging learning setting, over a range of different datasets, and for multiple architectures. Finally, we probe the computational efficiency of CBRS compared to the state of the art, both in terms of time and memory overhead.
Online Multi-Kernel Learning with Graph-Structured Feedback
Authors: Pouya M Ghari, Yanning Shen
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Multi-kernel learning (MKL) exhibits reliable performance in nonlinear function approximation tasks. Instead of using one kernel, it learns the optimal kernel from a pre-selected dictionary of kernels.  The selection of the dictionary has crucial impact on both the performance and complexity of MKL. Specifically, inclusion of a large number of irrelevant kernels may impair the accuracy, and increase the complexity of MKL algorithms. To enhance the accuracy, and alleviate the computational burden, the present paper develops a novel scheme which actively chooses relevant kernels. The proposed framework models the pruned kernel combination as feedback collected from a graph, that is refined 'on the fly.' Leveraging the random feature approximation, we propose an online scalable multi-kernel learning approach with graph feedback, and prove that the proposed algorithm enjoys sublinear regret. Numerical tests on real datasets demonstrate the effectiveness of the novel approach.
Optimistic bounds for multi-output learning
Authors: Henry Reeve, Ata Kaban
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: General Classification, Multi-class Classification, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, Multi-target regression

We investigate the challenge of multi-output learning, where the goal is to learn a vector-valued function based on a supervised data set. This includes a range of important problems in Machine Learning including multi-target regression, multi-class classification and multi-label classification. We begin our analysis by introducing the self-bounding Lipschitz condition for multi-output loss functions, which interpolates continuously between a classical Lipschitz condition and a multi-dimensional analogue of a smoothness condition. We then show that the self-bounding Lipschitz condition gives rise to optimistic bounds for multi-output learning, which are minimax optimal up to logarithmic factors. The proof exploits local Rademacher complexity combined with a powerful minoration inequality due to Srebro, Sridharan and Tewari.  As an application we derive a state-of-the-art generalization bound for multi-class gradient boosting.
Supervised Quantile Normalization for Low Rank Matrix Factorization
Authors: Marco Cuturi, Olivier Teboul, Jonathan Niles-Weed, Jean-Philippe Vert
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Low rank matrix factorization is a fundamental building block in machine learning, used for instance to summarize gene expression profile data or word-document counts. To be robust to outliers and differences in scale across features, a matrix factorization step is usually preceded by ad-hoc feature normalization steps, such as tf-idf scaling or data whitening. We propose in this work to learn these normalization operators jointly with the factorization itself. More precisely, given a $d\times n$ matrix $X$ of $d$ features measured on $n$ individuals, we propose to learn the parameters of quantile normalization operators that can operate row-wise on the values of $X$ and/or of its factorization $UV$  to improve the quality of the low-rank representation of $X$ itself. This optimization is facilitated by the introduction of differentiable quantile normalization operators derived using regularized optimal transport algorithms.
Efficient proximal mapping of the path-norm regularizer of shallow networks
Authors: Fabian Latorre, Paul Rolland, Shaul Nadav Hallak, Volkan Cevher
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We demonstrate two new important properties of the path-norm regularizer for shallow neural networks. First, despite its non-smoothness and non-convexity it allows a closed form proximal operator which can be efficiently computed, allowing the use of stochastic proximal-gradient-type methods for regularized empirical risk minimization. Second, it provides an upper bound on the Lipschitz constant of the network, which is tighter than the trivial layer-wise product of Lipschitz constants, motivating its use for training networks robust to adversarial perturbations. Finally, in practical experiments we show that it provides a better robustness-accuracy trade-off when compared to $\ell_1$-norm regularization or training with a layer-wise constrain of the Lipschitz constant.
Influenza Forecasting Framework based on Gaussian Processes
Authors: Christoph Zimmer, Reza Yaesoubi
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Decision Making, Gaussian Processes

The seasonal epidemic of influenza costs thousands
of lives each year in the US. While influenza
epidemics occur every year, timing and size of the
epidemic vary strongly from season to season.
This complicates the public health efforts to adequately
respond to such epidemics. Forecasting
techniques to predict the development of seasonal
epidemics such as influenza, are of great help to
public health decision making. Therefore, the
US Center for Disease Control and Prevention
(CDC) has initiated a yearly challenge to forecast
influenza-like illness. Here, we propose a
new framework based on Gaussian process (GP)
for seasonal epidemics forecasting and demonstrate
its capability on the CDC reference data
on influenza like illness: our framework leads to
accurate forecasts with small but reliable uncertainty
estimation. We compare our framework
to several state of the art benchmarks and show
competitive performance. We, therefore, believe
that our GP based framework for seasonal epidemics
forecasting will play a key role for future
influenza forecasting and, lead to further research
in the area.
Latent Bernoulli Autoencoder
Authors: Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, Paolo Remagnino
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

In this work, we pose a question whether it is possible to design and train an
autoencoder model in an end-to-end fashion to learn latent representations in
multivariate Bernoulli space, and achieve performance comparable with the
current state-of-the-art variational methods. Moreover, we investigate how to
generate novel samples and perform smooth interpolation in the binary latent
space.  To meet our objective, we propose a simplified deterministic model
with a straight-through estimator to learn the binary latents and show its
competitiveness with the latest VAE methods.  Furthermore, we propose a novel
method based on a random hyperplane rounding for sampling and smooth
interpolation in the multivariate Bernoulli latent space.  Although not a main
objective, we demonstrate that our methods perform on par or better than the
current state-of-the-art methods on common CelebA, CIFAR-10 and MNIST 
datasets. PyTorch code and trained models to reproduce published results 
will be released with the camera ready version.
Teaching with Limited Information on the Learner's Behaviour
Authors: Ferdinando Cicalese, Francisco Sergio de Freitas Filho, Eduardo Laber, Marco Molinaro
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Machine Teaching studies how efficiently a Teacher  can guide a Learner to a target hypothesis. We focus on the model of Machine Teaching with a black box learner introduced in [Dasgupta et al., ICML 2019], where the teaching is done  interactively   without having any knowledge of the Learner's   algorithm and  class of hypotheses, apart from the fact that it contains the target hypothesis $h^*$. 

We first refine some existing results for this   model and,  then, we study new variants of it. Motivated  by the realistic possibility that  $h^*$ is not available to the learner, we consider the case where the teacher can only aim at having the learner converge to a best available approximation of $h^*$. We also consider weaker black box learners, where, in each round, the choice of the consistent hypothesis returned to the Teacher is not adversarial, and in particular, we show that better provable bounds can be obtained for a type of Learner  that moves to the next  hypothesis smoothly, preferring hypotheses that are  close  to the current one; and for another type of Learner that can provide to the Teacher hypotheses chosen at random among those consistent with the examples received so far.  
Finally, we present an empirical evaluation of  our basic interactive teacher on real datasets.
Towards Adaptive Residual Network Training: A Neural-ODE Perspective
Authors: chengyu dong, Liyuan Liu, Zichao Li, Jingbo Shang
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Serving as a crucial factor, the depth of residual networks balances model capacity, performance, and training efficiency. However, depth has been long fixed as a hyper-parameter and needs laborious tuning, due to the lack of theories describing its dynamics. Here, we conduct theoretical analysis on network depth and introduce adaptive residual network training, which gradually increases model depth during training. Specifically, from an ordinary differential equation perspective, we describe the effect of depth growth with embedded errors, characterize the impact of model depth with truncation errors, and derive bounds for them. Illuminated by these derivations, we propose an adaptive training algorithm for residual networks, LipGrow, which automatically increases network depth and accelerates model training. In our experiments, it achieves better or comparable performance while reducing ~50% of training time.
Towards non-parametric drift detection via Dynamic Adapting Window Independence Drift Detection (DAWIDD)
Authors: Fabian Hinder, André Artelt, CITEC Barbara Hammer
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Drift Detection

The notion of concept drift refers to the phenomenon that the distribution, which is underlying the observed data, changes over time; as a consequence machine learning models may become inaccurate and need adjustment. Many online learning schemes include drift detection to actively detect and react to observed changes. Yet, reliable drift detection constitutes a challenging problem in particular in the context of high dimensional data, varying drift characteristics, and the absence of a parametric model such as a classification scheme which reflects the drift. In this paper we present a novel concept drift detection method, Dynamic Adapting Window Independence Drift Detection (DAWIDD), which aims for non-parametric drift detection of diverse drift characteristics. For this purpose, we establish a mathematical equivalence of the presence of drift to the dependency of specific random variables in an according drift process. This allows us to rely on independence tests rather than parametric models or the classification loss, resulting in a fairly robust scheme to universally detect different types of drift, as is also confirmed in experiments.
Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks
Authors: Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, Dan Alistarh
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: image-classification, Image Classification

Optimizing convolutional neural networks for fast inference has recently become an extremely active area of research. One of the go-to solutions in this context is weight pruning, which aims to reduce computational and memory footprint by removing large subsets of the connections in a neural network.
Surprisingly, much less attention has been given to exploiting sparsity in the activation maps, which tend to be naturally sparse in many settings thanks to the structure of rectified linear (ReLU) activation functions.

In this paper, we present an in-depth analysis of methods for maximizing the sparsity of the activations in a trained neural network, and show that, when coupled with an efficient sparse-input convolution algorithm, we can leverage this sparsity for significant performance gains.
To induce highly sparse activation maps without accuracy loss, we introduce a new regularization technique, coupled with a new threshold-based sparsification method based on a parameterized activation function called Forced-Activation-Threshold Rectified Linear Unit (FATReLU). We examine the impact of our methods on popular image classification models, showing that most architectures can adapt to significantly sparser activation maps without any accuracy loss.
Our second contribution is showing that these these compression gains can be translated into inference speedups: we provide a new algorithm to enable fast convolution operations over networks with sparse activations, and show that it can enable significant speedups for end-to-end inference on a range of popular models on the large-scale ImageNet image classification task on modern Intel CPUs, with little or no retraining cost.
Deep Streaming Label Learning
Authors: Zhen Wang, Liu Liu, DaCheng Tao
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Multi-Label Learning

In multi-label learning, each instance can be associated with multiple and non-exclusive labels. Previous studies assume that all the labels in the learning process are fixed and static; however, they ignore the fact that the labels will emerge continuously in changing environments. In order to fill in these research gaps, we propose a novel deep neural network (DNN) based framework, Deep Streaming Label Learning (DSLL), to classify instances with newly emerged labels effectively. DSLL can explore and incorporate the knowledge from past labels and historical models to understand and develop emerging new labels. DSLL consists of three components: 1) a streaming label mapping to extract deep relationships between new labels and past labels with a novel label-correlation aware loss; 2) a streaming feature distillation propagating feature-level knowledge from the historical model to a new model; 3) a senior student network to model new labels with the help of knowledge learned from the past. Theoretically, we prove that DSLL admits tight generalization error bounds for new labels in the DNN framework. Experimentally, extensive empirical results show that the proposed method performs significantly better than the existing state-of-the-art multi-label learning methods to handle the continually emerging new labels.
Learning Factorized Weight Matrix for Joint Image Filtering
Authors: Xiangyu Xu, Yongrui Ma, Wenxiu Sun
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Feature Correlation

Joint image filtering is a fundamental problem in computer vision with applications in many different areas. Most existing algorithms solve this problem with a weighted averaging process to aggregate input pixels. However, the weight matrix of this process is often empirically designed and not robust to complex input. In this work, we propose to learn the weight matrix for joint image filtering. This is a challenging problem, as directly learning a large weight matrix is computationally intractable. To address this issue, we introduce the correlation of deep features to approximate the aggregation weights. However, this strategy only uses inner product for the weight matrix estimation, which limits the performance of the proposed algorithm. Therefore, we further propose to learn a nonlinear function to predict sparse residuals of the feature correlation matrix. Note that the proposed method essentially factorizes the weight matrix into a low-rank and a sparse matrix and then learn both of them simultaneously with deep neural networks. Extensive experiments show that the proposed algorithm compares favorably against the state-of-the-art approaches on a wide variety of joint image filtering tasks.
Neural Network Control Policy Verification With Persistent Adversarial Perturbation
Authors: Yuh-Shyang Wang, Tsui-Wei Weng, Luca Daniel
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Deep neural networks are known to be fragile to small adversarial perturbations. This issue becomes more critical when a neural network is interconnected with a physical system in a closed loop. In this paper, we show how to combine recent works on static neural network certification tools with robust control theory to certify a neural network policy in a control loop. We give a sufficient condition and an algorithm to ensure that the closed loop state and control constraints are satisfied when the persistent adversarial perturbation is linf norm bounded. Our method is based on finding a positively invariant set of the closed loop dynamical system, and thus we do not require the continuity of the neural network policy. Along with the verification result, we also develop an effective attack strategy for neural network control systems that outperforms exhaustive Monte-Carlo search significantly. We show that our certification algorithm works well on learned models and achieves 5 times better result than the traditional Lipschitz-based method to certify the robustness of a neural network policy on multiple control problems.
Randomization matters How to defend against strong adversarial attacks
Authors: Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, Jamal Atif
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

\emph{Is there a classifier that ensures optimal robustness against all adversarial attacks?}
This paper answers this question by adopting a game-theoretic point of view. We show that adversarial attacks and defenses form an \emph{infinite} zero-sum game where classical results (e.g. Nash or Sion theorems) do not apply. We demonstrate the non-existence of a Nash equilibrium in our game when the classifier and the adversary are both deterministic, hence giving a negative answer to the above question in the deterministic regime. Nonetheless, the question remains open in the randomized regime. We tackle this problem by showing that, under mild conditions on the dataset distribution, any deterministic classifier can be outperformed by a randomized one. This gives arguments for using randomization, and leads us to a new algorithm for building randomized classifiers that are robust to \emph{strong} adversarial attacks. Empirical results validate our theoretical analysis, and show that our defense method considerably outperforms Adversarial Training against state-of-the-art attacks.
A Tree-Structured Decoder for Image-to-Markup Generation
Authors: Jianshu Zhang, Jun Du, Yongxin Yang, Yi-Zhe Song, Si Wei, Li-Rong Dai
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Decoder, Handwritten Mathmatical Expression Recognition, Math

Recent encoder-decoder approaches typically employ string decoders to convert images into serialized strings for image-to-markup. However, for tree-structured representational markup, string representations can hardly cope with the structural complexity. In this work, we first show via a set of toy problems that string decoders struggle to decode tree structures, especially as structural complexity increases. We then propose a tree-structured decoder that specifically aims at generating a tree-structured markup. Our decoders works sequentially, where at each step a child node and its parent node are simultaneously generated to form a sub-tree. This sub-tree is consequently used to construct the final tree structure in a recurrent manner. Key to the success of our tree decoder is twofold, (i) it strictly respects the parent-child relationship of trees, and (ii) it explicitly outputs trees as oppose to a linear string. Evaluated on both math formula recognition and chemical formula recognition, the proposed tree decoder is shown to greatly outperform strong string decoder baselines.
Cost-effectively Identifying Causal Effect When Only Response Variable Observable
Authors: Tian-Zuo Wang, Xi-Zhu Wu, Sheng-Jun Huang, Zhi-Hua Zhou
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Decision Making

In many real tasks, we care about how to make decisions other than mere predictions on an event, e.g. how to increase the revenue next month instead of knowing it will drop. The key is to identify the causal effects on the desired event. Pearl proposed do-calculus to make it given the knowledge of causal structure (Pearl, 2009). But sometimes, we have to discover it at first. In this paper, we propose a novel solution for this challenging task where only the response variable is observable under intervention. By an active strategy introducing limited interventions and exploiting the exact distribution of the response variable, the proposed approach can cost-effectively identify the causal effect of each intervention, and thus guide the decision-making. Theoretical analysis along with empirical studies is presented to show that our approach can achieve causal effect identification with fewer interventions.
Dual-Path Distillation: A Unified Framework to Improve Black-Box Attacks
Authors: Yonggang Zhang, Ya Li, Tongliang Liu, Xinmei Tian
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We study the problem of constructing black-box adversarial attacks, where no model information is revealed except for the feedback knowledge of the given inputs. To obtain sufficient knowledge for crafting adversarial examples, previous methods query the target model with inputs that are perturbed with different searching directions. However, these methods suffer from poor query efficiency since the employed searching directions are sampled randomly. To mitigate this issue, we formulate the goal of mounting efficient attacks as an optimization problem in which the adversary tries to fool the target model with a limited number of queries. Under such settings, the adversary has to select appropriate searching directions to reduce the number of model queries. By solving the efficient-attack problem, we find that we need to distill the knowledge in both the path of the adversarial examples and the path of the searching directions. Therefore, we propose a novel framework, dual-path distillation, that utilizes the feedback knowledge not only to craft adversarial examples but also to alter the  searching directions to achieve efficient attacks.  Experimental results suggest that our framework can significantly increase the query efficiency.