pwc-archive/papers-with-abstracts huggingface dataset:
Towards Hierarchical Discrete Variational Autoencoders
Authors: Valentin Liévin, Andrea Dittadi, Lars Maaløe, Ole Winther
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

Variational Autoencoders (VAEs) have proven to be powerful latent variable models. How- ever, the form of the approximate posterior can limit the expressiveness of the model. Categorical distributions are flexible and useful building blocks for example in neural memory layers. We introduce the Hierarchical Discrete Variational Autoencoder (HD-VAE): a hi- erarchy of variational memory layers. The Concrete/Gumbel-Softmax relaxation allows maximizing a surrogate of the Evidence Lower Bound by stochastic gradient ascent. We show that, when using a limited number of latent variables, HD-VAE outperforms the Gaussian baseline on modelling multiple binary image datasets. Training very deep HD-VAE remains a challenge due to the relaxation bias that is induced by the use of a surrogate objective. We introduce a formal definition and conduct a preliminary theoretical and empirical study of the bias.
Scalable Gradients and Variational Inference for Stochastic Differential Equations
Authors: Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, David K. Duvenaud
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Time Series, Time Series Analysis, Variational Inference

We derive reverse-mode (or adjoint) automatic differentiation for solutions of stochastic differential equations (SDEs), allowing time-efficient and constant-memory computation of pathwise gradients, a continuous-time analogue of the reparameterization trick.
Specifically, we construct a backward SDE whose solution is the gradient and provide conditions under which numerical solutions converge.
We also combine our stochastic adjoint approach with a stochastic variational inference scheme for continuous-time SDE models, allowing us to learn distributions over functions using stochastic gradient descent.
Our latent SDE model achieves competitive performance compared to existing approaches on time series modeling.
Sinkhorn Permutation Variational Marginal Inference
Authors: Gonzalo Mena, Erdem Varol, Amin Nejatbakhsh, Eviatar Yemini, Liam Paninski
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

We address the problem of marginal inference for an exponential family defined over the set of permutation matrices. This problem is known to quickly become intractable as the size of the permutation increases, since its involves the computation of the permanent of a matrix, a #P-hard problem. We introduce Sinkhorn variational marginal inference as a scalable alternative, a method whose validity is ultimately justified by the so-called Sinkhorn approximation of the permanent. We demonstrate the efectiveness of our method in the problem of probabilistic identification of neurons in the worm C.elegans
Structured Semi-Implicit Variational Inference
Authors: Iuliia Molchanova, Dmitry Molchanov, Novi Quadrianto, Dmitry Vetrov
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

In this work we construct flexible joint distributions from low-dimensional conditional semi-implicit distributions. Explicitly defining the structure of the approximation allows to make the variational lower bound tighter, resulting in more accurate inference.
Arbitrarily-conditioned Data Imputation
Authors: Micael Carvalho, Thibaut Durand, JiaWei He, Nazanin Mehrasa, Greg Mori
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Imputation

In this paper, we propose an arbitrarily-conditioned data imputation framework built upon variational autoencoders and normalizing flows. The proposed model is capable of mapping any partial data to a multi-modal latent variational distribution. Sampling from such a distribution leads to stochastic imputation. Preliminary evaluation on MNIST dataset shows promising stochastic imputation conditioned on partial images as input.
Bijectors.jl: Flexible transformations for probability distributions
Authors: Tor Erlend Fjelde, Kai Xu, Mohamed Tarek, Sharan Yalburgi, Hong Ge
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Variational Inference

Transforming one probability distribution to another is a powerful tool in Bayesian inference and machine learning. Some prominent examples are constrained-to-unconstrained transformations of distributions for use in Hamiltonian Monte-Carlo and constructing flexible and learnable densities such as normalizing flows. We present Bijectors.jl, a software package for transforming distributions implemented in Julia, available at github.com/TuringLang/Bijectors.jl. The package provides a flexible and composable way of implementing transformations of distributions without being tied to a computational framework. 

We demonstrate the use of Bijectors.jl on improving variational inference by encoding known statistical dependencies into the variational posterior using normalizing flows, providing a general approach to relaxing the mean-field assumption usually made in variational inference.
Exchangeable Variational Autoencoders with Applications to Genomic Data
Authors: Jeffrey Chan, Jeffrey Spence, and Yun Song
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference

Exchangeable-structured datapoints are ubiquitous in statistical problems ranging from point clouds to graphs to sets. Particularly in biological settings where multiple experiments derived from a noisy scientific process attempt to measure a latent variable of interest, experimental datapoints are often exchangeable demanding the development of methods which can exploit this structure. Modern machine learning approaches to scalable Bayesian inference typically use autoencoding variational Bayes -- marrying ideas from deep learning and probabilistic modeling to achieve practical inference for expressive models. Current VAE-based approaches do not naturally handle exchangeable (but non-iid) datapoints. Often exchangeable-structured datapoints may contain heterogeneity in datapoint dimensions precluding a staightforward application of the vanilla VAE framework. In this work, we develop the Exchangeable Variational Autoencoder which provides inferential and computational benefits while enabling varying set size data to be robustly handled in the VAE framework. We then demonstrate its efficacy in two settings: (1) on the well-studied Latent Dirichlet Allocation model and (2) on the bootstrapped, isoform-level uncertainty estimates of single-cell RNA-seq data.
AdvancedHMC.jl: A robust, modular and efficient implementation of advanced HMC algorithms
Authors: Kai Xu, Hong Ge, Will Tebbutt, Mohamed Tarek, Martin Trapp, Zoubin Ghahramani
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Benchmarking

Stan's Hamilton Monte Carlo (HMC) has demonstrated remarkable sampling robustness and efficiency in a wide range of Bayesian inference problems through carefully crafted adaption schemes to the celebrated No-U-Turn sampler (NUTS) algorithm.  It is challenging to implement these adaption schemes robustly in practice, hindering wider adoption amongst practitioners who are not directly working with the Stan modelling language. AdvancedHMC.jl (AHMC) contributes a modular, well-tested, standalone implementation of NUTS that recovers and extends Stan's NUTS algorithm. AHMC is written in Julia, a modern high-level language for scientific computing, benefiting from optional hardware acceleration and interoperability with a wealth of existing software written in both Julia and other languages, such as Python. Efficacy is demonstrated empirically by comparison with Stan through a third-party Markov chain Monte Carlo benchmarking suite.
Optimal Transport for Distribution Adaptation in Bayesian Hilbert Maps
Authors: Anthony Tompkins, Ransalu Senanayake, Fabio Ramos
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

Parameters are one of the most critical components of machine learning models. As datasets and learning domains change, it is often necessary and time-consuming to re-learn entire models. Rather than re-learning the parameters from scratch, replacing learning with optimization, we propose a framework building upon the theory of \emph{optimal transport} to adapt model parameters by discovering correspondences between models and data, significantly amortizing the training cost. We demonstrate our idea on the challenging problem of creating probabilistic spatial representations for autonomous robots. Although recent mapping techniques have facilitated robust occupancy mapping, learning all spatially-diverse parameters in such approximate Bayesian models demand considerable computational time, discouraging them to be used in real-world robotic mapping. Considering the fact that the geometric features a robot would observe with its sensors are similar across various environments, in this paper, we demonstrate how to re-use parameters and hyperparameters learned in different domains. This adaptation is computationally more efficient than variational inference and Monte Carlo techniques. A series of experiments conducted on realistic settings verified the possibility of transferring thousands of such parameters with a negligible time and memory cost, enabling large-scale mapping in urban environments.
Distributional Bayesian optimisation for variational inference on black-box simulators
Authors: Rafael Oliveira, Lionel Ott, Fabio Ramos
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Optimisation, Variational Inference

Inverse problems are ubiquitous in natural sciences and refer to the challenging task of inferring complex and potentially multi-modal posterior distributions over hidden parameters given a set of observations. Typically, a model of the physical process in the form of differential equations is available but leads to intractable inference over its parameters. While the forward propagation of parameters through the model simulates the evolution of the system, the inverse problem of finding the parameters given the sequence of states is not unique. In this work, we propose a generalisation of the Bayesian optimisation framework to approximate inference. The resulting method learns approximations to the posterior distribution by applying Stein variational gradient descent on top of estimates from a Gaussian process model. Preliminary results demonstrate the method's performance on likelihood-free inference for reinforcement learning environments.
Spectral Mixture Kernel Approximation Using Reparameterized Random Fourier Feature
Authors: Yohan Jung, Jinkyoo Park
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

We propose a method for Spectral Mixture kernel approximation using the Reparameterized Random Fourier Feature (R-RFF) in the sense of both general parameter and natural parameter view. Meanwhile, we provide the effective sampling methods of spectral points which samples the number of spectral points by considering the normalized weight parameters of SM kernel. Also, we develop the regularized sparse spectrum approximation by using Stochastic Gradient Variational Bayes for scalable learning of GP model with SM kernel.
Rapid Model Comparison by Amortizing Across Models
Authors: Lily H. Zhang, Michael C. Hughes
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: model, Topic Models, Variational Inference

Comparing the inferences of diverse candidate models is an essential part of model checking and escaping local optima. To enable efficient comparison, we introduce an amortized variational inference framework that can perform fast and reliable posterior estimation across models of the same architecture. Our Any Parameter Encoder (APE) extends the encoder neural network common in amortized inference to take both a data feature vector and a model parameter vector as input. APE thus reduces posterior inference across unseen data and models to a single forward pass. In experiments comparing candidate topic models for synthetic data and product reviews, our Any Parameter Encoder yields comparable posteriors to more expensive methods in far less time, especially when the encoder architecture is designed in model-aware fashion.
Bayesian Model Selection for Identifying Markov Equivalent Causal Graphs
Authors: Mehmet Burak Kurutmaz, Melih Barsbey, Ali Taylan Cemgil, Sinan Yildirim, Umut Şimşekli
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Causal Discovery, Model Selection

Many approaches to causal discovery are limited by their inability to discriminate between Markov equivalent graphs given only observational data. We formulate causal discovery as a marginal likelihood based Bayesian model selection problem. We adopt a parameterization based on the notion of the independence of causal mechanisms which renders Markov equivalent graphs distinguishable. We complement this with an empirical Bayesian approach to setting priors so that the actual underlying causal graph is assigned a higher marginal likelihood than its alternatives. Adopting a Bayesian approach also allows for straightforward modeling of unobserved confounding variables, for which we provide a variational algorithm to approximate the marginal likelihood, since this desirable feat renders the computation of the marginal likelihood intractable. We believe that the Bayesian approach to causal discovery both allows the rich methodology of Bayesian inference to be used in various difficult aspects of this problem and provides a unifying framework to causal discovery research. We demonstrate promising results in experiments conducted on real data, supporting our modeling approach and our inference methodology.
Variationally Inferred Sampling Through a Refined Bound
Authors: Victor Gallego, David Rios Insua
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Density Estimation, State Space Models, Time Series, Time Series Analysis

A framework for efficient Bayesian inference in probabilistic programs is introduced by embedding a sampler inside a variational posterior approximation. Its strength lies in both ease of implementation and automatically tuning sampler parameters to speed up mixing time. Several strategies to approximate the evidence lower bound (ELBO) computation are introduced, including a rewriting of the ELBO objective. Experimental evidence is shown by performing experiments on an unconditional VAE on density estimation tasks; solving an influence diagram in a high-dimensional space with a conditional variational autoencoder (cVAE) as a deep Bayes classifier; and state-space models for time-series data.
Gaussian Process Meta-Representations For Hierarchical Neural Network Weight Priors
Authors: Theofanis Karaletsos, Thang D. Bui
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Active Learning, Bayesian Inference

Bayesian inference offers a theoretically grounded and general way to train neural networks and can potentially give calibrated uncertainty. However, it is challenging to specify a meaningful and tractable prior over the network parameters, and deal with the weight correlations in the posterior. To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for the network parameters based on recently introduced unit embeddings that can flexibly encode weight structures, and (ii) input-dependent contextual variables for the weight prior that can provide convenient ways to regularize the function space being modeled by the network through the use of kernels. 
We show these models provide desirable test-time uncertainty estimates, demonstrate cases of modeling inductive biases for neural networks with kernels and demonstrate competitive predictive performance on an active learning benchmark.
Interpretable User Models via Decision-rule Gaussian Processes: Preliminary Results on Energy Storage
Authors: Danial Mohseni-Taheri, Selvaprabu Nadarajah, Theja Tulabandhula
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Gaussian Processes, Variational Inference

Models of user behavior are critical inputs in many prescriptive settings and can be viewed as decision rules that transform state information available to the user into actions. Gaussian processes (GPs), as well as nonlinear extensions thereof, provide a flexible framework to learn user models in conjunction with approximate Bayesian inference. However, the resulting models may not be interpretable in general. We propose decision-rule GPs (DRGPs) that apply GPs in a transformed space defined by decision rules that have immediate interpretability to practitioners. We illustrate this modeling tool on a real application and show that structural variational inference techniques can be used with DRGPs. We find that DRGPs outperform the direct use of GPs in terms of out-of-sample performance.
HM-VAEs: a Deep Generative Model for Real-valued Data with Heterogeneous Marginals
Authors: Chao Ma, Sebastian Tschiatschek, Yingzhen Li, Richard Turner, Jose Miguel Hernandez-Lobato, Cheng Zhang
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

In this paper, we focused on improving VAEs for real-valued data that has heterogeneous marginal distributions. We propose the heterogeneous-marginal VAE (HM-VAE), a method that explicitly decomposes intra-variable uncertainties  and inter-variable uncertainties. We experimentally observe that the HM-VAEs can generate realistic data with nearly indistinguishable marginals when compared with real data.
Sequential Learning for Dirichlet Process Mixtures
Authors: Chunlin Ji, Bin Liu, Yingkai Jiang, Ke Deng
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Variational Inference

Dirichlet process mixture model provides a flexible nonparametric framework for unsupervised learning. Monte Carlo based sampling methods always involve heavy computation efforts; conventional variational inference requires careful design of the variational distribution and the conditional expectation. In this work, we treat the DP mixture itself as the variational proposal,
and view the given data as drawn samples of the unknown target distribution. We propose an evidence upper bound (EUBO) to act as the surrogate loss, and fit a DP mixture to the given data by minimizing the EUBO, which is equivalent to minimizing the KL-divergence between the target distribution and the DP mixture. We provide three advantages of the EUBO based DP mixture fitting and show how to build the black-box style sequential learning algorithm. We use the stochastic gradient descent (SGD) algorithm for optimization that leverages on the automatic differentiation tools. Simulation studies are provided to demonstrate the efficiency of our proposed methods.
Stein Variational Gradient Descent for Approximate Bayesian Computation
Authors: Chunlin Ji, Jiangsheng Yi, Wanchuang Zhu
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Bayesian Inference, Variational Inference

Approximate Bayesian Computation (ABC) provides a generic framework of Bayesian inference for likelihood-free models, but sampling based posterior approximation is often time-consuming and has difficulty accessing the convergence. Stochastic variational inference forms the posterior inference to a optimization problem and enable the ABC scalable for large dataset. However, complex simulation models involved in ABC always lead to complex posteriors, which is not easy to approximate by simple parametric variational distributions. We draw upon recent advances in the implicit model of variational distribution and introduce the Stein variational gradient descent (SVGD) approach to approximate the posterior by nonparametric particles. We also find that the kernel in the SVGD algorithm helps in reducing the large variance of the gradient estimators of ABC likelihood. Moreover, energy distance is proposed as the statistics in the evaluation of ABC likelihood, which reduce the difficulty in selecting proper statistics. Simulation studies are provided to demonstrate the correctness and efficiency of our algorithm.
Challenges in Computing and Optimizing Upper Bounds of Marginal Likelihood based on Chi-Square Divergences
Authors: Melanie F. Pradier, Michael C. Hughes, Finale Doshi-Velez
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Diagnostic, Variational Inference

Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood. However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood. This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks. We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound. We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies.
Efficient Inference Amortization in Graphical Models using Structured Continuous Conditional Normalizing Flows
Authors: Christian Weilbach, Boyan Beronov, William Harvey, Frank Wood
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12
Tasks: Probabilistic Programming

We introduce a more efficient neural architecture for amortized inference, which combines continuous and conditional normalizing flows using a principled choice of structure. Our gradient flow derives its sparsity pattern from the minimally faithful inverse of its underlying graphical model. We find that this factorization reduces the necessary numbers both of parameters in the neural network and of adaptive integration steps in the ODE solver. Consequently, the throughput at training time and inference time is increased, without decreasing performance in comparison to unconstrained flows. By expressing the structural inversion and the flow construction as compilation passes of a probabilistic programming language, we demonstrate their applicability to the stochastic inversion of realistic models such as convolutional neural networks (CNN).
A Projection Operator to Balance Consistency and Complexity in Importance Sampling
Authors: Alec Koppel*, Amrit Singh Bedi*, Brian M. Sadler, and Victor Elvira
Date: 2019-10-16 00:00:00
Proceeding: pproximateinference AABI Symposium 2019 12

Importance sampling (IS) is a standard Monte Carlo (MC) tool to compute information about random variables such as moments or quantiles with unknown distributions.  IS is 
asymptotically consistent as the number of MC samples, and hence deltas (particles) that parameterize the density estimate, go to infinity. However, retaining infinitely many particles is intractable. We propose a scheme for only keeping a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we approximate importance sampling in two ways.  First, we replace the deltas by kernels, yielding kernel density estimates (KDEs).  Second, we sequentially project KDEs onto nearby lower-dimensional subspaces. We characterize the asymptotic bias of this scheme as determined by a compression parameter and kernel bandwidth, which yields a tunable tradeoff between consistency and memory. In experiments,  we observe a favorable tradeoff between memory and accuracy, providing for the first time near-consistent compressions of arbitrary posterior distributions.