pwc-archive/papers-with-abstracts huggingface dataset:
Towards Characterizing the High-dimensional Bias of Kernel-based Particle Inference Algorithms
Authors: Jimmy Ba, Murat A. Erdogdu, Marzyeh Ghassemi, Taiji Suzuki, Shengyang Sun, Denny Wu, Tianzong Zhang
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: LEMMA

Particle-based inference algorithm is a promising method to efficiently generate samples for an intractable target distribution by iteratively updating a set of particles. As a noticeable example, Stein variational gradient descent (SVGD) provides a deterministic and computationally efficient update, but it is known to underestimate the variance in high dimensions, the mechanism of which is poorly understood. In this work we explore a connection between SVGD and MMD-based inference algorithm via Stein's lemma. By comparing the two update rules, we identify the source of bias in SVGD as a combination of high variance and deterministic bias, and empirically demonstrate that the removal of either factors leads to accurate estimation of the variance. In addition, for learning high-dimensional Gaussian target, we analytically derive the converged variance for both algorithms, and confirm that only SVGD suffers from the "curse of dimensionality".
Graph Tracking in Dynamic Probabilistic Programs via Source Transformations
Authors: Philipp Gabler, Martin Trapp, Hong Ge, Franz Pernkopf
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, BIG-bench Machine Learning, Probabilistic Programming

Many modern machine learning algorithms, such as automatic differentiation (AD) and versions of approximate Bayesian inference, can be understood as a particular case of message passing on some computation graph.
To permit learning complex models, recent approaches to extract a computation graph focus on dynamic graphs using so-called operator overloading techniques.
However, in contrast to source transformation, which is commonly used for static graphs, using operator overloading does not allow to analyse the graph and naturally leads to higher overhead costs.
In this paper, we present a combination of source transformation and operator overloading, which allows extracting the underlaying computation graph of complex machine learning models, e.g.~Bayesian nonparametric models or models with stochastic control flow.
To track the execution of operations in a machine learning model during run-time, we inject additional statements into the existing program at compile-time using the intermediate representation of the program.
We introduce an extension of the well-known Wengert lists, used in many AD implementations, to record all necessary control flow information and any additional metadata.
Finally, we discuss future applications of our approach, such as conjugacy exploitation in universal probabilistic programming languages, BUGS like Gibbs sampling for dynamic models and variational message-passing.
Neural Permutation Processes
Authors: Ari Pakman, Yueqi Wang, Liam Paninski
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference

We introduce a neural architecture to perform amortized approximate Bayesian inference over latent random permutations of two sets of objects. The method involves approximating permanents of matrices of pairwise probabilities using recent ideas on functions defined over sets. Each sampled permutation comes with a probability estimate, a quantity unavailable in MCMC approaches. We illustrate the method in sets of 2D points and MNIST images.
Langevin Dynamics as Nonparametric Variational Inference
Authors: Matthew D. Hoffman, Yian Ma
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

Variational inference (VI) and Markov chain Monte Carlo (MCMC) are approximate posterior inference algorithms that are often said to have complementary strengths, with VI being fast but biased and MCMC being slower but asymptotically unbiased. In this paper, we analyze gradient-based MCMC and VI procedures and find theoretical and empirical evidence that these procedures are not as different as one might think. In particular, a close examination of the Fokker-Planck equation that governs the Langevin dynamics (LD) MCMC procedure reveals that LD implicitly follows a gradient flow that corresponds to a variational inference procedure based on optimizing a nonparametric normalizing flow. This result suggests that the transient bias of LD (due to too few warmup steps) may track that of VI (due to too few optimization steps), up to differences due to VI’s parameterization and asymptotic bias. Empirically, we find that the transient biases of these algorithms (and momentum-accelerated versions) do evolve similarly. This suggests that practitioners with a limited time budget may get more accurate results by running an MCMC procedure (even if it’s far from burned in) than a VI procedure, as long as the variance of the MCMC estimator can be dealt with (e.g., by running many parallel chains).
BooVAE: A Scalable Framework for Continual VAE Learning under Boosting Approach
Authors: Anna Kuzina, Evgenii Egorov, Evgeny Burnaev
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Image Generation

Variational Auto Encoders (VAE) are capable of generating realistic images, sounds and video sequences. From practitioners point of view, we are usually interested in solving problems where tasks are learned sequentially, in a way that avoids revisiting all previous data at each stage. We address this problem by introducing a conceptually simple and scalable end-to-end approach of incorporating past knowledge by learning prior directly from the data.  We consider scalable boosting-like approximation for intractable theoretical optimal prior. We provide empirical studies on two commonly used benchmarks, namely  MNIST and Fashion MNIST on disjoint sequential image generation tasks. For each dataset proposed method delivers the best results among comparable approaches,  avoiding catastrophic forgetting in a fully automatic way with a fixed model architecture.
Coping With Simulators That Don’t Always Return
Authors: Andrew Warrington, Saeid Naderiparizi, Frank Wood
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Computational Efficiency

Deterministic models are approximations of reality that are often easier to build and interpret than stochastic alternatives.  
Unfortunately, as nature is capricious, observational data can never be fully explained by deterministic models in practice.  
Observation and process noise need to be added to adapt deterministic models to behave stochastically, such that they are capable of explaining and extrapolating from noisy data.
Adding process noise to deterministic simulators can induce a failure in the simulator resulting in no return value for certain inputs -- a property we describe as ``brittle.''
We investigate and address the wasted computation that arises from these failures, and the effect of such failures on downstream inference tasks.
We show that performing inference in this space can be viewed as rejection sampling, and train a conditional normalizing flow as a proposal over noise values such that there is a low probability that the simulator crashes, increasing computational efficiency and inference fidelity for a fixed sample budget when used as the proposal in an approximate inference algorithm.
Seeing the whole picture instead of a single point: Self-supervised likelihood learning for deep generative models
Authors: Petra Poklukar, Judith Bütepage, Danica Kragic
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

Recent findings show that deep generative models can judge out-of-distribution samples as more likely than those drawn from the same distribution as the training data. In this work, we focus on variational autoencoders (VAEs) and address the problem of misaligned likelihood estimates on image data. We develop a novel likelihood function that is based not only on the parameters returned by the VAE but also on the features of the data learned in a self-supervised fashion. In this way, the model additionally captures the semantic information that is disregarded by the usual VAE likelihood function. We demonstrate the improvements in reliability of the estimates with experiments on the FashionMNIST and MNIST datasets.
Deep Amortized Variational Inference for Multivariate Time Series Imputation with Latent Gaussian Process Models
Authors: Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, Stephan Mandt
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Deep Learning, Dimensionality Reduction, Imputation, Missing Values, Multivariate Time Series Imputation, Time Series, Time Series Analysis, Variational Inference

Multivariate time series with missing values are common in areas such as healthcare and finance, and have grown in number and complexity over the years. This raises the question whether deep learning methodologies can outperform classical data imputation methods in this domain. However, naive applications of deep learning fall short in giving reliable confidence estimates and lack interpretability. We propose a new deep sequential latent variable model for dimensionality reduction and data imputation. Our modeling assumption is simple and interpretable: the high dimensional time series has a lower-dimensional representation which evolves smoothly in time according to a Gaussian process. The non-linear dimensionality reduction in the presence of missing data is achieved using a VAE approach with a novel structured variational approximation. We demonstrate that our approach outperforms several classical and deep learning-based data imputation methods on high-dimensional data from the domains of computer vision and healthcare, while additionally improving the smoothness of the imputations and providing interpretable uncertainty estimates.
Batch simulations and uncertainty quantification in Gaussian process surrogate-based approximate Bayesian computation
Authors: Marko Järvenpää, Aki Vehtari, Pekka Marttinen
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Uncertainty Quantification

Surrogate models can be used to accelerate approximate Bayesian computation (ABC). In one such framework the discrepancy between simulated and observed data is modelled with a Gaussian process. So far principled strategies have been proposed only for sequential selection of the simulation locations. To address this limitation, we develop Bayesian optimal design strategies to parallellise the expensive simulations. We also touch the problem of quantifying the uncertainty of the ABC posterior due to the limited budget of simulations.
Automatically Trading off Time and Variance when Selecting Gradient Estimators
Authors: Tomas Geffner, Justin Domke
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

Stochastic gradient descent (SGD) is the workhorse of modern machine learning. Sometimes, there are many different potential gradient estimators that can be used. When so, choosing the one with the best tradeoff between cost and variance is important. This paper analyzes the convergence rates of SGD as a function of time, rather than iterations. This results in a simple rule to select the estimator that leads to the best optimization convergence guarantee. This choice is the same for different variants of SGD, and with different assumptions about the objective (e.g. convexity or smoothness). Inspired by this principle, we propose a technique to automatically select an estimator when a finite pool of estimators is given. Then, we extend to infinite pools of estimators, where each one is indexed by control variate weights. This is enabled by a reduction to a mixed-integer quadratic program. Empirically, automatically choosing an estimator performs comparably to the best estimator chosen with hindsight.
Variational Gaussian Process Models without Matrix Inverses
Authors: Mark van der Wilk, ST John, Artem Artemev, James Hensman
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

Large matrix inversions have often been cited as a major impediment to scaling Gaussian process (GP) models. With the use of GPs as building blocks for ever more sophisticated Bayesian deep learning models, removing these impediments is a necessary step for achieving large scale results. We present a variational approximation for a wide range of GP models that does not require a matrix inverse to be performed at each optimisation step. Our bound instead directly parameterises a free matrix, which is an additional variational parameter. At the local maxima of the bound, this matrix is equal to the matrix inverse. We prove that our bound gives the same guarantees as earlier variational approximations. We demonstrate some beneficial properties of the bound experimentally, although significant wall clock time speed improvements will require future improvements in optimisation and implementation.
Variational Selective Autoencoder
Authors: Yu Gong, Hossein Hajimirsadeghi, JiaWei He, Megha Nawhal, Thibaut Durand, Greg Mori
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Image Inpainting, Imputation

Despite promising progress on unimodal data imputation (e.g. image inpainting), models for multimodal data imputation are far from satisfactory. In this work, we propose variational selective autoencoder (VSAE) for this task. Learning only from partially-observed data, VSAE can model the joint distribution of observed/unobserved modalities and the imputation mask, resulting in a unified model for various down-stream tasks including data generation and imputation. Evaluation on synthetic high-dimensional and challenging low-dimensional multimodal datasets shows significant improvement over state-of-the-art imputation models.
Pseudo-Bayesian Learning via Direct Loss Minimization with Applications to Sparse Gaussian Process Models
Authors: Rishit Sheth, Roni Khardon
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

We propose that approximate Bayesian algorithms should optimize a new criterion, directly derived from the loss, to calculate their approximate posterior which we refer to as pseudo-posterior. Unlike standard variational inference which optimizes a lower bound on the log marginal likelihood, the new algorithms can be analyzed to provide loss guarantees on the predictions with the pseudo-posterior. Our criterion can be used to derive new sparse Gaussian process algorithms that have error guarantees applicable to various likelihoods.
Newtonian Monte Carlo: a second-order gradient method for speeding up MCMC
Authors: Nimar S. Arora, Nazanin Khosravani Tehrani, Kinjal Divesh Shah, Michael Tingley, Yucen Lily Li, Narjes Torabi, David Noursi, Sepehr Akhavan Masouleh, Eric Lippert, Erik Meijer
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

We present Newtonian Monte Carlo (NMC), a method to improve Markov Chain Monte Carlo (MCMC) convergence by analyzing the first and second order gradients of the target density to determine a suitable proposal density at each point. Existing first order gradient-based methods suffer from the problem of determining an appropriate step size. Too small a step size and it will take a large number of steps to converge, while a very large step size will cause it to overshoot the high density region. NMC is similar to the Newton-Raphson update in optimization where the second order gradient is used to automatically scale the step size in each dimension. However, our objective is not to find a maxima but instead to find a parameterized density that can best match the local curvature of the target density.  This parameterized density is then used as a single-site Metropolis-Hastings proposal.

As a further improvement on first order methods, we show that random variables with constrained supports don't need to be transformed before taking a gradient step. NMC directly matches constrained random variables to a proposal density with the same support thus keeping the curvature of the target density intact.

We demonstrate the efficiency of NMC on a number of different domains. For statistical models where the prior is conjugate to the likelihood, our method recovers the posterior quite trivially in one step. However, we also show results on fairly large non-conjugate models, where NMC performs better than adaptive first order methods such as NUTS or other inexact scalable inference methods such as Stochastic Variational Inference or bootstrapping.
Global Approximate Inference via Local Linearisation for Temporal Gaussian Processes
Authors: William J. Wilkinson, Paul E. Chang, Michael Riis Andersen, Arno Solin
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Gaussian Processes, Variational Inference

The extended Kalman filter (EKF) is a classical signal processing algorithm which performs efficient approximate Bayesian inference in non-conjugate models by linearising the local measurement function, avoiding the need to compute intractable integrals when calculating the posterior. In some cases the EKF outperforms methods which rely on cubature to solve such integrals, especially in time-critical real-world problems. The drawback of the EKF is its local nature, whereas state-of-the-art methods such as variational inference or expectation propagation (EP) are considered global approximations. We formulate power EP as a nonlinear Kalman filter, before showing that linearisation results in a globally iterated algorithm that exactly matches the EKF on the first pass through the data, and iteratively improves the linearisation on subsequent passes. An additional benefit is the ability to calculate the limit as the EP power tends to zero, which removes the instability of the EP-like algorithm. The resulting inference scheme solves non-conjugate temporal Gaussian process models in linear time, $\mathcal{O}(n)$, and in closed form.
The Gaussian Process Prior VAE for Interpretable Latent Dynamics from Pixels
Authors: Michael Arthur Leopold Pearce
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: State Space Models

We consider the problem of unsupervised learning of a low dimensional, interpretable, latent state of a video containing a moving object. The problem of distilling dynamics from pixels has been extensively considered through the lens of graphical/state space models that exploit Markov structure for cheap computation and structured graphical model priors for enforcing interpretability on latent representations. We take a step towards extending these approaches by discarding the Markov structure; instead, repurposing the recently proposed Gaussian Process Prior Variational Autoencoder for learning sophisticated latent trajectories. We describe the model and perform experiments on a synthetic dataset and see that the model reliably reconstructs smooth dynamics exhibiting U-turns and loops. We also observe that this model may be trained without any beta-annealing or freeze-thaw of training parameters. Training is performed purely end-to-end on the unmodified evidence lower bound objective. This is in contrast to previous works, albeit for slightly different use cases, where application specific training tricks are often required.
Genomic variety prediction via Bayesian nonparametrics
Authors: Lorenzo Masoero, Federico Camerlenghi, Stefano Favaro, Tamara Broderick
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Experimental Design, Prediction

Despite the advent of Big Data, data-gathering in many domains can still be an expensive process that necessitates careful planning. For instance, in genomics, researchers can spend money and time to sequence a greater number of individual genomes -- or alternatively they can spend these resources to sequence individual genomes with increased accuracy. In either case, spending resources has the potential to reveal new variations in the genome and thereby new genetic insights. We consider the case where scientists have already conducted a pilot study to reveal some variants in a genome and are contemplating a follow-up study. We provide a novel prediction method, using Bayesian nonparametric methods, for how many variants scientists can expect to find in the follow-up based on the information in the pilot. When sequencing accuracy is kept constant between the pilot and follow-up, we demonstrate on (real) data from the gnomAD project that our prediction is more accurate than two recent proposals -- and as accurate as a more classic proposal. Unlike other existing methods though, our method allows practitioners to change the sequencing accuracy between the pilot and the follow-up. We demonstrate how this distinction allows our method to be used both for more realistic predictions as well as for optimal experimental design of the follow-up study under a resource budget.
Non-reversibly updating a uniform [0,1] value for accept/reject decisions
Authors: Radford M. Neal
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform [0,1] value, and to then update this uniform value non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration. This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions.  It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories.  This is of significance when some variables are updated by other methods, since if HMC is used, these updates can be done only between trajectories, whereas they can be done more often with Langevin updates.  This is seen for a Bayesian neural network model, in which connection weights are updated by persistent Langevin or HMC, while hyperparameters are updated by Gibbs sampling.
Efficient Bayesian Inference for Nested Simulators
Authors: Bradley Gram-Hansen, Christian Schroeder de Witt, Robert Zinkov, Saeid Naderiparizi, Adam Scibior, Andreas Munk, Frank Wood, Mehrdad Ghadiri, Philip Torr, Yee Whye Teh, Atilim Gunes Baydin, Tom Rainforth
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference

We introduce two approaches for conducting efficient Bayesian inference in stochastic simulators containing nested stochastic sub-procedures, i.e., internal procedures for which the density cannot be calculated directly such as rejection sampling loops. The resulting class of simulators are used extensively throughout the sciences and can be interpreted as probabilistic generative models. However, drawing inferences from them poses a substantial challenge due to the inability to evaluate even their unnormalised density, preventing the use of many standard inference procedures like Markov Chain Monte Carlo (MCMC). To address this, we introduce inference algorithms based on a two-step approach that first approximates the conditional densities of the individual sub-procedures, before using these approximations to run MCMC methods on the full program. Because the sub-procedures can be dealt with separately and are lower-dimensional than that of the overall problem, this two-step process allows them to be isolated and thus be tractably dealt with, without placing restrictions on the overall dimensionality of the problem. We demonstrate the utility of our approach on a simple, artificially constructed simulator.