pwc-archive/papers-with-abstracts huggingface dataset:
Neural Networks as Inter-Domain Inducing Points
Authors: Shengyang Sun, Jiaxin Shi, Roger Baker Grosse
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Gaussian Processes, regression

Equivalences between infinite neural networks and Gaussian processes have been established for explaining the functional prior and training dynamics of deep learning models. In this paper we cast the hidden units of finite-width neural networks as the inter-domain inducing points of a kernel, then a one-hidden-layer network becomes a kernel regression model. For dot-product kernels on both $R^d$ and $S^{d−1}$, we derive the kernel functions for inducing points. Empirically we conduct toy experiments to validate the proposed approaches.
Gradient Regularisation as Approximate Variational Inference
Authors: Ali Unlu, Laurence Aitchison
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Variational Inference

Variational inference in Bayesian neural networks is usually performed using stochastic sampling which gives very high-variance gradients, and hence slow learning. Here, we show that it is possible to obtain a deterministic approximation of the ELBO for a Bayesian neural network by doing a Taylor-series expansion around the mean of the current variational distribution. The resulting approximate ELBO is the training-log-likelihood plus a squared gradient regulariser. In addition to learning the approximate posterior variance, we also consider a uniform-variance approximate posterior, inspired by the stationary distribution of SGD. The corresponding approximate ELBO has a simple form, as the log-likelihood plus a simple squared-gradient regulariser. We argue that this squared-gradient regularisation may at the root of the excellent empirical performance of SGD.
Adaptive Strategy for Resetting a Non-stationary Markov Chain during Learning via Joint Stochastic Approximation
Authors: Hyunsu Kim, Juho Lee, Hongseok Yang
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

In this paper, we tackle the non-stationary kernel problem of the JSA algorithm by Ou and Song 2020, a recent proposal that learns a deep generative model $p_\theta(\mathbf{x},\mathbf{h})$ and a corresponding approximate posterior $q_\phi(\mathbf{h}|\mathbf{x})$ by drawing samples from a non-stationary Markov chain and estimating gradients with these samples. The non-stationary kernel problem refers to the degraded performance of the algorithm due to the constant change of the transition kernel of the chain throughout the run of the algorithm. We present an automatic adaptive strategy for checking whether this change is significant at each gradient-update step or not, and resetting the chain with a sample drawn from the current approximate posterior $q_\phi(\mathbf{h}|\mathbf{x})$ if the answer to the check is yes. In our experiments with the binarized MNIST, our strategy gives results comparable with or slightly better than those reported in the original paper on JSA, while avoiding the nontrivial manual intervention required for handling the non-stationary kernel problem in the original JSA algorithm.
Roundoff Error in Metropolis-Hastings Accept-Reject Steps
Authors: Matthew Douglas Hoffman
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

In this note, we consider what happens to the Metropolis-Hastings (M-H) algorithm when it is fed log-density calculations that are subject to roundoff error and catastrophic cancellations. We will briefly review the nature of these errors and how they can arise in M-H algorithms. Next, we will develop a theoretical model of roundoff error in M-H corrections, and find that it can lead to exponentially low acceptance rates in the magnitude of the errors (if the errors have Gaussian tails) and bias (if different types of states produce different types of errors). Finally, we will discuss some consequences of this phenomenon, and touch on some practical ways to avoid these consequences of catastrophic cancellations in software.
Variational Refinement for Importance SamplingUsing the Forward Kullback-Leibler Divergence
Authors: Ghassen Jerfel, Serena Lutong Wang, Clara Fannjiang, Katherine A Heller, Yian Ma, Michael Jordan
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Bayesian Inference, Variational Inference

Variational Inference (VI) is a popular alternative to asymptotically exact sampling in Bayesian inference. 
Its main workhorse is optimization over a reverse Kullback-Leibler divergence (RKL), which typically underestimates the tail of the posterior and causes miscalibration and potential degeneracy (over-pruning). 
Importance sampling (IS), on the other hand, is often used to fine-tune and debias the estimates of approximate Bayesian inference procedures. 
The quality of IS crucially depends on the choice of the proposal distribution. 
Ideally, the proposal distribution has heavier tails than the target, which is unachievable by minimizing the RKL.
We thus propose a novel combination of optimization and sampling techniques for approximate Bayesian inference by constructing an IS proposal distribution through the minimization of a forward KL (FKL) divergence. 
This approach guarantees asymptotic consistency and a fast convergence towards both the optimal IS estimator and the optimal variational approximation.
Transforming Worlds: Automated Involutive MCMC for Open-Universe Probabilistic Models
Authors: George Matheos, Alexander K. Lew, Matin Ghavamizadeh, Stuart Russell, Marco Cusumano-Towner, Vikash Mansinghka
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Bayesian Inference, Probabilistic Programming, valid

Open-universe probabilistic models enable Bayesian inference about how many objects underlie data, and how they are related. Effective inference in OUPMs remains a challenge, however, often requiring the use of custom, trans-dimensional MCMC kernels, based on heuristics, deep learning, or domain knowledge, that can be difficult to derive and to implement correctly. This paper adapts the recently introduced involutive MCMC framework to the open-universe setting, and shows how error-prone aspects of kernel design and implementation (e.g., the computation of valid accept/reject probabilities) can be automated, using techniques from probabilistic and differentiable programming. The result is an intuitive design space for MCMC kernels for OUPMs: users write programs that propose incremental changes to possible worlds, creating, deleting, or modifying objects according to arbitrary application-specific logic, and their proposals are automatically converted into stationary MCMC kernels. We demonstrate in preliminary experiments that data-driven involutive MCMC kernels outperform generic probabilistic programming language inference, as well as generic birth/death reversible-jump kernels without application-specific logic.
Marginal Likelihood Gradient for Bayesian Neural Networks
Authors: Marcin B. Tomczak, Richard E Turner
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Variational Inference

Bayesian learning of neural networks is attractive as it can protecting against over-fitting and provide automatic methods for inferring important hyperparameters by maximizing the marginal probability of the data. However, existing approaches in this vein, such as those based on variational inference, do not perform well. In this paper, we take a different approach and directly derive a practical estimator of the gradient of the marginal log-likelihood for BNNs by combining local reparametrization of the network w.r.t.~the prior distribution with the self-normalized importance sampling estimator. We show promising preliminary results on a toy example and on vectorized MNIST classification where the new method results in significantly improved performance of variational inference compared to existing approaches to tune hyperparameters.
Generative Video Compression as Hierarchical Variational Inference
Authors: Ruihan Yang, Yibo Yang, Joseph Marino, Stephan Mandt
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Density Estimation, Variational Inference, Video Compression

Recent work by Marino et al. (2020) showed improved performance in sequential density estimation by combining masked autoregressive flows with hierarchical latent variable models. We draw a connection between such autoregressive generative models and the task of lossy video compression. Specifically, we view recent neural video compression methods (Lu et al., 2019; Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight. Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.
Evidence Estimation by Kullback-Leibler Integration for Flow-Based Methods
Authors: Nikolai Zaki, Théo Galy-Fajou, Manfred Opper
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

Flow-based methods such as Stein Variational Gradient Descent caught a lot of interest due to their flexibility and the strong theory going with them.
An usual issue however is to be able to compute the KL divergence or even possibly the log evidence.
In this preliminary work we show that  it's possible to obtain an approximation during inference by reusing the training computations.
We show preliminary results on Gaussian targets but face issues with more complex problems.
Gaussian Process Latent Variable Flows for Massively Missing Data
Authors: Vidhi Lalchand, Aditya Ravuri, Neil D Lawrence
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Dimensionality Reduction, Gaussian Processes, Normalising Flows, Variational Inference

Gaussian process latent variable models (GPLVM) are used to perform nonlinear and probabilistic dimensionality reduction. They extend Gaussian processes (GP) to the domain of unsupervised learning. The Bayesian incarnation of the GPLVM uses a variational framework, where the posterior over all unknown quantities is approximated by a well-behaved variational family, a factorised Gaussian. This gives not only implicit regularisation but also mathematical convenience. In this work we narrow our focus on examining the quality of the latent representation learnt under this Gaussian assumption. We introduce non-Gaussianity in the distribution of the latent space through normalising flows. The flexibility afforded by flows is critical in modelling massively missing data. Inference is performed using Stochastic Variational Inference (SVI) with a structured variational lower bound that factorizes across data points permitting efficient and scalable mini-batching of gradients. We call this flexible model class $\textit{Gaussian process latent variable flows}$(GPLVF). We compare this framework with traditional models like the Bayesian GPLVM. Our experiments focus on massively missing data settings.
Neural Linear Models with Functional Gaussian Process Priors
Authors: Joe Watson, Jihao Andreas Lin, Pascal Klink, Jan Peters
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Gaussian Processes, regression, Variational Inference

Neural linear models (NLM) and Gaussian processes (GP) are both examples of Bayesian linear regression on rich feature spaces. 
In contrast to the widespread use of nonparametric GPs for probabilistic nonlinear regression,
NLMs remain an underused parametric alternative because standard type II maximum likelihood (ML) training
leads to overconfidence outside of the data distribution.
Therefore, we propose to augment this training procedure through functional variational inference (fVI) proposed by Sun et. al. (2019), which is particularly well suited for NLMs due to their closed-form predictive distribution.
Additionally, we investigate whether an appropriate functional prior can guide parametric NLMs to attain nonparametric GP performance, despite using fewer parameters.
Results show that functional priors do improve performance of NLM over ML training, and that the NLM performs on par with weight space BNNs in this setting.
Why Cold Posteriors? On the Suboptimal Generalization of Optimal Bayes Estimates
Authors: Chen Zeno, Itay Golan, Ari Pakman, Daniel Soudry
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Deep Learning

Recent works have shown that the predictive accuracy of Bayesian deep learning models exhibit substantial improvements when the posterior is raised to a 1/T power with T<1. In this work, we explore several possible reasons for this surprising behavior.
The Gaussian Process Latent Autoregressive Model
Authors: Rui Xia, Wessel Bruinsma, William Tebbutt, Richard E Turner
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Denoising, Gaussian Processes, model, Variational Inference

Many real-world prediction problems involve modelling the dependencies between multiple different outputs across the input space. Multi-output Gaussian Processes (MOGP) are a particularly important approach to such problems. In this paper, we build on the Gaussian Process Autoregressive Regression (GPAR) model which is one of the best performing MOGP models, but which fails when observation noise is large, when there are missing data, and when non-Gaussian observation models are required. We extend the original GPAR model to handle these settings and provide a variational inference procedure similar to that used in deep Gaussian Processes which replaces the ad hoc denoising approximation used in the original work. We show that the new approach naturally handles noisy outputs, missing data and that it also enables the model to handle heterogeneous non-Gaussian observation models.
Marginalised Spectral Mixture Kernels with Nested Sampling
Authors: Fergus Simpson, Vidhi Lalchand, Carl Edward Rasmussen
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Prediction Intervals, regression, Time Series, Time Series Analysis, Time Series Regression

Gaussian Process (GPs) models are a rich distribution over functions with inductive biases controlled by a kernel function. Learning occurs through optimisation of the kernel hyperparameters using the marginal likelihood as the objective (ML-II). This work analyses the benefits of marginalising kernel hyperparameters using nested sampling (NS), a technique well-suited to sample from complex, multi-modal distributions. We benchmark against Hamiltonian Monte Carlo (HMC) on time-series regression tasks and find that a principled approach to quantifying hyperparameter uncertainty substantially improves the quality of prediction intervals.
Scalable Hybrid Hidden Markov Model with Gaussian Process Emission for Sequential Time-series Observations
Authors: Yohan Jung, Jinkyoo Park
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Time Series, Time Series Analysis, Variational Inference

A hidden Markov model (HMM) using Gaussian Process as an emission model has been widely used to model sequential data in complex form. This study particularly introduces the hybrid Bayesian HMM with GP emission using SM kernel, which we call HMM-GPSM, for estimating a hidden state of each time-series observation sequentially observed from a single channel. We then propose a scalable learning method to train the HMM-GPSM model using large-scale data having (1) long sequences of state transitions and (2) a large number of time-series observations for each hidden state. For a long sequence of state transitions, we employ stochastic variational inference (SVI) to efficiently update the parameters of HMM-GPSM. For a large number of data points in each time series observation, we propose the approximate GP emission using the spectral points sampled from the spectral density of SM kernel by Random Fourier feature (RFF) and the efficient inference for the kernel hyperparameters of approximate GP emission and corresponding HMM-GPSM. Specifically, we derive the training loss, i.e., the evidence lower bound of the HMM-GPSM that can be scalably computed for a large number of time-series observations by employing the regularized lower bound of GP emission likelihood with KL divergence. The proposed methods can be used together for the sequential time-series dataset having both (1) and (2). We validate the proposed method on the synthetic using the clustering accuracy and training time as the performance metrics.
On the Inconsistency of Bayesian Inference for Misspecified Neural Networks
Authors: Yijie Zhang, Eric Nalisnick
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Bayesian Inference, regression, Variational Inference

Grunwald and Van Ommen (2017) show that Bayesian inference for linear regression can be inconsistent under model misspecification. In this paper, we extend their analysis to Bayesian neural networks (BNNs), investigating if they too can be inconsistent under misspecification. We find that BNNs exhibit the same inconsistency when Hamiltonian Monte Carlo is used for posterior inference. However, variational inference changes this behavior. Surprisingly, we find that variational Bayes leads to BNNs that are consistent in the setting studied by Grunwald and Van Ommen (2017). We conjecture that the success of variational Bayes is due to its optimization objective: the evidence lower bound (ELBO) implicitly encourages the posterior approximation to concentrate, mitigating the ill-effects of the misspecification.
Decoupled Sparse Gaussian Processes Components]{Decoupled Sparse Gaussian Processes Components : Separating Decision Making from Data Manifold Fitting
Authors: Sebastian Popescu, David J. Sharp, James H. Cole, Ben Glocker
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Decision Making, Gaussian Processes, regression

 We propose a decoupling in Reproducing Kernel Hilbert Space of the parametric and non-parametric components of Sparse Gaussian Processes. We demonstrate that this decoupling results in a set of inducing points concentrated around the decision boundaries, respectively one that fits the data manifold in terms of variance. Moreover, we make connections between Bayesian Kernel Ridge Regression and Sparse Gaussian Processes in posterior function space. Inspired from this equivalence, we introduce a inducing points' locations linked parametrization of the variational variance term of inducing point values, hence resulting in models that only have variational mean parameters to optimize. We compare our model to counterparts and show improvements at testing time on large scale regression and classification datasets, alongside faster convergence rates to the full Gaussian Process.
Variational Beam Search for Novelty Detection
Authors: Aodong Li, Alex James Boyd, Padhraic Smyth, Stephan Mandt
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Novelty Detection

We consider the problem of online learning in the presence of sudden distribution shifts, which may be hard to detect and can lead to a slow but steady degradation in model performance. To address this problem we propose a new Bayesian meta-algorithm that can both (i) make inferences about subtle distribution shifts based on minimal sequential observations and (ii) accordingly adapt a model in an online fashion. The approach uses beam search over multiple change point hypotheses to perform inference on a hierarchical sequential latent variable modeling framework. Our proposed approach is model-agnostic, applicable to both supervised and unsupervised learning, and yields significant improvements over state-of-the-art Bayesian online learning approaches.
Efficient Calculation of Adversarial Examples for Bayesian Neural Networks
Authors: Sina Däubener, Joel Frank, Thorsten Holz, Asja Fischer
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

Calculating adversarial examples for Bayesian neural networks is cumbersome. Due to the inherent stochasticity, the gradient of the network can only be reliable approximated by sampling multiple times from the posterior, leading to a greatly increased computational cost. In this paper we propose to efficiently attack Bayesian neural networks with adversarial examples calculated for a deterministic network with parameters given by the mean of the posterior distribution. We show in a series of experiments, that the proposed approach can be used to effectively attack Bayesian neural networks while using 4.2 times less of the resources of existing adversarial example estimation methods with comparable strength. We demonstrate that this is especially helpful during adversarial training, when multiple different model configuration need to be evaluated.
Approximating the clusters' prior distribution in Bayesian nonparametric models
Authors: Daria Bystrova, Julyan Arbel, Guillaume Kon Kam King, François Deslandes
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

In Bayesian nonparametrics, knowledge of the prior distribution induced on the number of clusters is key for prior specification and calibration. However, evaluating this prior is infamously difficult even for moderate sample size.
We evaluate several statistical approximations to the prior distribution on the number of clusters for Gibbs-type processes, a class including the Pitman--Yor process and the normalized generalized gamma process.
We introduce a new approximation based on the predictive distribution of Gibbs-type process, which compares favourably with the existing methods.
We thoroughly discuss the limitations of these various approximations by comparing them against an exact implementation of the prior distribution of the number of clusters.
Distilling Ensembles Improves Uncertainty Estimates
Authors: Zelda E Mariet, Rodolphe Jenatton, Florian Wenzel, Dustin Tran
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1

We seek to bridge the performance gap between batch ensembles (ensembles of deep networks with shared parameters)  and deep ensembles on tasks which require not only predictions, but also uncertainty estimates for these predictions. We obtain negative theoretical results on the possibility of approximating deep ensemble weights by  batch ensemble weights, and so turn to distillation. Training a batch ensemble on the outputs of deep ensembles improves accuracy and uncertainty estimates, without requiring hyper-parameter tuning. This result is specific to the choice of batch ensemble architectures: distilling deep ensembles to a single network is unsuccessful, despite single networks having only marginally fewer parameters than batch ensembles.
Gaussian Density Parametrization Flow: Particle and Stochastic Approaches
Authors: Théo Galy-Fajou, Valerio Perrone, Manfred Opper
Date: 2020-11-23 00:00:00
Proceeding: pproximateinference AABI Symposium 2021 1
Tasks: Bayesian Inference, Variational Inference

Bayesian inference is intractable for most practical problems and requires approximation schemes with several trade-offs.
Variational inference provides one of such approximations which, while powerful, has thus far seen limited use in high-dimensional applications due to its complexity and computational cost. This paper introduces a scalable, theoretically-grounded, and simple-to-implement algorithm for approximate inference with a variational Gaussian distribution. Specifically, we establish a practical particle-based algorithm to perform variational Gaussian inference that scales linearly in the problem dimensionality. We show that our approach performs on par with the state of the art on a set of challenging high-dimensional problems.