pwc-archive/papers-with-abstracts huggingface dataset:
Self-Attentive Hawkes Process
Authors: Qiang Zhang, Aldo Lipani, Omer Kirnap, Emine Yilmaz
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Capturing the occurrence dynamics is crucial to predicting which type of events will happen next and when. A common method to do this is Hawkes processes. To enhance their capacity, recurrent neural networks (RNNs) have been incorporated due to RNNs’ successes in processing sequential data such as languages. Recent evidence suggests self-attention is more competent than RNNs in dealing with languages. However, we are unaware of the effectiveness of selfattention in the context of Hawkes processes. This study attempts to fill the gap by designing a self-attentive Hawkes process (SAHP). The SAHP employed self-attention to summarize influence from history events and compute the probability of the next event. One deficit of the conventional self-attention is that position embeddings only
considered order numbers in a sequence, which ignored time intervals between temporal events. To overcome this deficit, we modified the conventional method by translating time intervals into phase shifts of sinusoidal functions. Experiments
on goodness-of-fit and prediction tasks showed the improved capability of SAHP. Furthermore, the SAHP is more interpretable than RNN-based counterparts because the learnt attention weights revealed contributions of one event type to the happening of another type. To the best of our knowledge, this is the first work that studies the effectiveness of self-attention in Hawkes processes
The Natural Lottery Ticket Winner: Reinforcement Learning with Ordinary Neural Circuits
Authors: Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus, Radu Grosu
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: reinforcement-learning, Reinforcement Learning (RL)

We propose a neural information processing system which is obtained by re-purposing the function of a biological neural circuit model to govern simulated and real-world control tasks. Inspired by the structure of the nervous system of the soil-worm, C. elegans, we introduce ordinary neural circuits (ONCs), defined as the model of biological neural circuits reparameterized for the control of alternative tasks. We first demonstrate that ONCs realize networks with higher maximum flow compared to arbitrary wired networks. We then learn instances of ONCs to control a series of robotic tasks, including the autonomous parking of a real-world rover robot. For reconfiguration of the purpose of the neural circuit, we adopt a search-based optimization algorithm. Ordinary neural circuits perform on par and, in some cases, significantly surpass the performance of contemporary deep learning models. ONC networks are compact, 77% sparser than their counterpart neural controllers, and their neural dynamics are fully interpretable at the cell-level.
DeepCoDA: personalized interpretability for compositional health
Authors: Thomas Quinn, Dang Nguyen, Santu Rana, Sunil Gupta, Svetha Venkatesh
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Interpretability allows the domain-expert to directly evaluate the model's relevance and reliability, a practice that offers assurance and builds trust. In the healthcare setting, interpretable models should implicate relevant biological mechanisms independent of technical factors like data pre-processing. We define personalized interpretability as a measure of sample-specific feature attribution, and view it as a minimum requirement for a precision health model to justify its conclusions. Some health data, especially those generated by high-throughput sequencing experiments, have nuances that compromise precision health models and their interpretation. These data are compositional, meaning that each feature is conditionally dependent on all other features. We propose the DeepCoDA framework to extend precision health modelling to high-dimensional compositional data, and to provide personalized interpretability through patient-specific weights. Our architecture maintains state-of-the-art performance across 25 real-world data sets, all while producing interpretations that are both personalized and fully coherent for compositional data.
Explicit Gradient Learning for Black-Box Optimization
Authors: Elad Sarafian, Mor Sinay, yoram louzoun, Noa Agmon, Sarit Kraus
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Image Generation

Black-Box Optimization (BBO) methods can find optimal policies for systems that interact with complex environments with no analytical representation. As such, they are of interest in many Artificial Intelligence (AI) domains. Yet classical BBO methods fall short in high-dimensional non-convex problems. They are thus often overlooked in real-world AI tasks. Here we present a BBO method, termed Explicit Gradient Learning (EGL), that is designed to optimize high-dimensional ill-behaved functions. We derive EGL by finding weak spots in methods that fit the objective function with a parametric Neural Network (NN) model and obtain the gradient signal by calculating the parametric gradient. Instead of fitting the function, EGL trains a NN to estimate the objective gradient directly. We prove the convergence of EGL to a stationary point and its robustness in the optimization of integrable functions. We evaluate EGL and achieve state-of-the-art results in two challenging problems: (1) the COCO test suite against an assortment of standard BBO methods; and (2) in a high-dimensional non-convex image generation task.
Healing Gaussian Process Experts
Authors: samuel cohen, Rendani Mbuvha, Tshilidzi Marwala, Marc Deisenroth
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Gaussian Processes, General Classification, regression, Uncertainty Quantification

Gaussian processes (GPs) are nonparametric Bayesian models that have been applied to regression and classification problems. One of the approaches to alleviate their cubic training cost is the use of local GP experts trained on subsets of the data. In particular, product-of-expert models combine the predictive distributions of local experts through a tractable product operation.
While these expert models allow for massively distributed computation, their predictions typically suffer from erratic behaviour of the mean or uncalibrated uncertainty quantification. By calibrating predictions via a tempered softmax weighting, we provide a solution to these problems for multiple product-of-expert models, including the generalised product of experts and the robust Bayesian committee machine. Furthermore, we leverage the optimal transport literature and propose a new product-of-expert model that combines predictions of local experts by computing their Wasserstein barycenter, which can be applied to both regression and classification.
Multi-Precision Policy Enforced Training (MuPPET) : A Precision-Switching Strategy for Quantised Fixed-Point Training of CNNs
Authors: Aditya Rajagopal, Diederik Vink, Stylianos Venieris, Christos-Savvas Bouganis
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: GPU

Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners. As networks grow in size and complexity, training time can be reduced through low-precision data representations and
computations. However, in doing so the final accuracy suffers due to the problem of vanishing gradients. Existing state-of-the-art methods combat this issue by means of a mixed-precision approach utilising two different precision levels, FP32 (32-bit floating-point) and FP16/FP8 (16-/8-bit floating-point), leveraging the hardware support of recent GPU architectures for FP16 operations to obtain performance gains. This work pushes the boundary of quantised training by employing a multilevel optimisation approach
that utilises multiple precisions including low-precision fixed-point representations. The novel training strategy, MuPPET, combines the use of multiple number representation regimes together with a precision-switching mechanism that decides at run time the transition point between precision regimes. Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the target hardware architecture and yields improvements in training time and energy efficiency compared to state-of-the-art
approaches. Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, MuPPET achieves the same accuracy as standard full-precision training with training-time speedup of up to 1.84× and an average speedup of 1.58× across the networks.
VideoOneNet: Bidirectional Convolutional Recurrent OneNet with Trainable Data Steps for Video Processing
Authors: Zoltán Milacski, Barnabás Póczos, Andras Lorincz
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Colorization, Compressive Sensing, Deblurring, Denoising, Rotated MNIST, Super-Resolution

Deep Neural Networks (DNNs) achieve the state-of-the-art results on a wide range of image processing tasks, however, the majority of such solutions are problem-specific, like most AI algorithms. The One Network to Solve Them All (OneNet) procedure has been suggested to resolve this issue by exploiting a DNN as the proximal operator in Alternating Direction Method of Multipliers (ADMM) solvers for various imaging problems. In this work, we make two contributions, both facilitating end-to-end learning using backpropagation. First, we generalize OneNet to videos by augmenting its convolutional prior network with bidirectional recurrent connections; second, we extend the fixed fully connected linear ADMM data step with another trainable bidirectional convolutional recurrent network. In our computational experiments on the Rotated MNIST, Scanned CIFAR-10 and UCF-101 data sets, the proposed modifications improve performance by a large margin compared to end-to-end convolutional OneNet and 3D Wavelet sparsity on several video processing problems: pixelwise inpainting-denoising, blockwise inpainting, scattered inpainting, super resolution, compressive sensing, deblurring, frame interpolation, frame prediction and colorization. Our two contributions are complementary, and using them together yields the best results.
Adversarial Robustness via Runtime Masking and Cleansing
Authors: Yi-Hsuan Wu, Chia-Hung Yuan, Shan-Hung (Brandon) Wu
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Adversarial Robustness

Deep neural networks are shown to be vulnerable to adversarial attacks. This motivates robust learning techniques, such as the adversarial training, whose goal is to learn a network that is robust against adversarial attacks.  However, the sample complexity of robust learning can be significantly larger than that of “standard” learning. In this paper, we propose improving the adversarial robustness of a network by leveraging the potentially large test data seen at runtime. We devise a new defense method, called runtime masking and cleansing (RMC), that adapts the network at runtime before making a prediction to dynamically mask network gradients and cleanse the model of the non-robust features inevitably learned during the training process due to the size limit of the training set. We conduct experiments on real-world datasets and the results demonstrate the effectiveness of RMC empirically.
Boosted Histogram Transform for Regression
Authors: Yuchao Cai, Hanyuan Hang, Hanfang Yang, Zhouchen Lin
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: HTR, regression

In this paper, we propose a boosting algorithm for regression problems called \textit{boosted histogram transform for regression} (BHTR) based on histogram transforms composed of random rotations, stretchings, and translations. From the theoretical perspective, we first prove fast convergence rates for BHTR under the assumption that the target function lies in the spaces $C^{0,\alpha}$. Moreover, if the target function resides in the subspace $C^{1,\alpha}$, by establishing the upper bound of the convergence rate for the boosted regressor, i.e. BHTR, and the lower bound for base regressors, i.e. histogram transform regressors (HTR), we manage to explain the benefits of the boosting procedure. In the experiments, compared with other state-of-the-art algorithms such as gradient boosted regression tree (GBRT), Breiman's forest, and kernel-based methods, our BHTR algorithm shows promising performance on both synthetic and real datasets.
From Chaos to Order: Symmetry and Conservation Laws in Game Dynamics
Authors: Sai Ganesh Nagarajan, David Balduzzi, Georgios Piliouras
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Games are an increasingly useful tool for training and testing learning algorithms. Recent examples include GANs, AlphaZero and the AlphaStar league. However, multi-agent learning can be extremely difficult to predict and control. Even simple games learning dynamics can yield chaotic behavior.  In this paper, we present basic \emph{mechanism design} tools for constructing games with predictable and controllable dynamics. We show that arbitrarily large and complex network games, encoding both cooperation (team play) and competition (zero-sum interaction), exhibit conservation laws when agents use the standard regret-minimizing dynamics known as Follow-the-Regularized-Leader. These laws persist even if different agents use different dynamics and encode long-range correlations between agents' behavior even though the agents may not interact directly. Moreover, we provide sufficient conditions under which the dynamics have multiple, linearly independent, conservation laws. Increasing the number of conservation laws results in more predictable dynamics, eventually making chaotic behavior in some cases even formally impossible.
Non-separable Non-stationary random fields
Authors: Kangrui Wang, Oliver Hamelijnck, Theodoros Damoulas, Mark Steel
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We describe a framework for constructing non-separable non-stationary random fields that is based on an infinite mixture of convolved stochastic processes. When the mixing process is stationary and the convolution function is non-stationary we arrive at expressive kernels that are available in closed form. When the mixing is non-stationary and the convolution function is stationary the resulting random fields exhibit varying degrees of non-separability that better preserve local structure. These kernels have natural interpretations through corresponding stochastic differential equations (SDEs) and are demonstrated on a range of synthetic benchmarks and spatio-temporal applications in geostatistics and machine learning. We show how a single Gaussian process (GP) with these kernels can computationally and statistically outperform both separable and existing non-stationary non-separable approaches such as treed GPs and deep GP constructions.
On Lp-norm Robustness of Ensemble Decision Stumps and Trees
Authors: Yihan Wang, huan zhang, Hongge Chen, Duane Boning, Cho-Jui Hsieh
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Recent papers have demonstrated that ensemble stumps and trees could be vulnerable to small input perturbations, so robustness verification and defense for those models have become an important research problem. However, due to the structure of decision trees, where each node makes decision purely based on one feature value, all the previous works only consider the $\ell_\infty$ norm perturbation. To study robustness with respect to a general $\ell_p$ norm perturbation, one has to consider correlation between perturbations on different features, which has not been handled by previous algorithms. In this paper, we study the robustness verification and defense with respect to general $\ell_p$ norm perturbation for ensemble trees and stumps. For robustness verification, we prove that exact verification is NP-complete for $p\in(0, \infty)$ while polynomial time algorithms exist for $p=0$ or $\infty$. Approximation algorithms based on dynamic programming is then developed for verifying ensemble trees and stumps. For robustness training, we propose the first certified defense method for training ensemble stumps and trees with respect to $\ell_p$ norm perturbations. The effectiveness of proposed algorithms is verified empirically on real datasets.
On Relativistic f-Divergences
Authors: Alexia Jolicoeur-Martineau
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

We take a more rigorous look at Relativistic Generative Adversarial Networks (RGANs) and prove that the objective function of the discriminator is a statistical divergence for any concave function $f$ with minimal properties ($f(0)=0$, $f'(0) \neq 0$, $\sup_x f(x)>0$). We devise additional variants of relativistic $f$-divergences. We show that the Wasserstein distance is weaker than $f$-divergences which are weaker than relativistic $f$-divergences. Given the good performance of RGANs, this suggests that Wasserstein GAN does not performs well primarily because of the weak metric, but rather because of regularization and the use of a relativistic discriminator. We introduce the minimum-variance unbiased estimator (MVUE) for Relativistic paired GANs (RpGANs; originally called RGANs which could bring confusion) and show that it does not perform better. We show that the estimator of Relativistic average GANs (RaGANs) is asymptotically unbiased and that the finite-sample bias is small; removing this bias does not improve performance.
Acceleration through spectral density estimation
Authors: Fabian Pedregosa, Damien Scieur
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Density Estimation, regression

We develop a framework for designing optimal optimization methods in terms of their average-case runtime. This yields a new class of methods that achieve acceleration through a model of the Hessian's expected spectral density. We develop explicit algorithms for the uniform, Marchenko-Pastur and exponential distribution. These methods are momentum-based gradient algorithms whose hyper-parameters can be estimated cheaply using only the norm and the trace of the Hessian, in stark contrast with classical accelerated methods like Nesterov acceleration and Polyak momentum that require knowledge of the Hessian's largest and smallest singular value. Empirical results on quadratic, logistic regression and neural network show the proposed methods always match and in many cases significantly improve upon classical accelerated methods.
Adaptive Adversarial Multi-task Representation Learning
Authors: YUREN MAO, Weiwei Liu, Xuemin Lin
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Representation Learning

Adversarial Multi-task Representation Learning (AMTRL) methods are able to boost the performance of Multi-task Representation Learning (MTRL) models. However, the theoretical mechanism behind AMTRL is less investigated. To fill this gap, we study the generalization error bound of AMTRL through the lens of Lagrangian duality . Based on the duality, we proposed an novel adaptive AMTRL algorithm which improves the performance of original AMTRL methods. The extensive experiments back up our theoretical analysis and validate the superiority of our proposed algorithm.
A Flexible Framework for Nonparametric Graphical Modeling that Accommodates Machine Learning
Authors: Yunhua Xiang, Noah Simon
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: BIG-bench Machine Learning, valid

Graphical modeling has been broadly useful for exploring the dependence structure among features in a dataset. However, the strength of graphical modeling hinges on our ability to encode and estimate conditional dependencies. In particular, commonly used measures such as partial correlation are only meaningful under strongly parametric (in this case, multivariate Gaussian) assumptions. These assumptions are unverifiable, and there is often little reason to believe they hold in practice. In this paper, we instead consider 3 non-parametric measures of conditional dependence. These measures are meaningful without structural assumptions on the multivariate distribution of the data. In addition, we show that for 2 of these measures there are simple, strong plug-in estimators that require only the estimation of a conditional mean. These plug-in estimators (1) are asymptotically linear and non-parametrically efficient, (2) allow incorporation of flexible machine learning techniques for conditional mean estimation, and (3) enable the construction of valid Wald-type confidence intervals. In addition, by leveraging the influence function of these estimators, one can obtain intervals with simultaneous coverage guarantees for all pairs of features.
Amortized Finite Element Analysis for Fast PDE-Constrained Optimization
Authors: Tianju Xue, Alex Beatson, Sigrid Adriaenssens , Ryan Adams
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1

Optimizing the parameters of partial differential equations (PDEs), i.e., PDE-constrained optimization (PDE-CO), allows us to model natural systems from observations or perform rational design of structures with complicated mechanical, thermal, or electromagnetic properties.  However, PDE-CO is often computationally prohibitive due to the need to solve the PDE---typically via finite element analysis (FEA)---at each step of the optimization procedure. In this paper we propose amortized finite element analysis (AmorFEA), in which a neural network learns to produce accurate PDE solutions, while preserving many of the advantages of traditional finite element methods. This network is trained to directly minimize the potential energy from which the PDE and finite element method are derived, avoiding the need to generate costly supervised training data by solving PDEs with traditional FEA. As FEA is a variational procedure, AmorFEA is a direct analogue to popular amortized inference approaches in latent variable models, with the finite element basis acting as the variational family. AmorFEA can perform PDE-CO without the need to repeatedly solve the associated PDE, accelerating optimization when compared to a traditional workflow using FEA and the adjoint method.
Black-Box Variational Inference as a Parametric Approximation to Langevin Dynamics
Authors: Matthew Hoffman, Yi-An Ma
Date: 2020-01-01 00:00:00
Proceeding: ICML 2020 1
Tasks: Variational Inference

Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased. In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think. In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to an VI procedure based on optimizing a nonparametric normalizing flow. The evolution under gradient descent of real-world VI approximations that use tractable, parametric flows can thus be seen as an approximation to the evolution of a population of LD-MCMC chains. This result suggests that the transient bias of LD (due to the Markov chain not having burned in) may track that of VI (due to the optimizer not having converged), up to differences due to VI’s asymptotic bias and parameter geometry. Empirically, we find that the transient biases of these algorithms (and their momentum-accelerated counterparts) do evolve similarly. This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if it is stopped before fully burning in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).