The Gaussian Process Latent Autoregressive Model
Abstract
Many real-world prediction problems involve modelling the dependencies between multiple diﬀerent outputs across the input space.
Multi-output Gaussian Processes (MOGP) are a particularly important approach to such problems.
In this paper, we build on the Gaussian
Process Autoregressive Regression (GPAR) model which is one of the best performing
MOGP models, but which fails when observation noise is large, when there are missing data, and when non-Gaussian observation models are required.
We extend the original
GPAR model to handle these settings and provide a variational inference procedure similar to that used in deep Gaussian Processes which replaces the ad hoc denoising approximation used in the original work.
We show that the new approach naturally handles noisy outputs, missing data and that it also enables the model to handle heterogeneous non-Gaussian observation models.
1. Introduction
With the growing prevalence of complex decision making systems, there is increasing need for learning systems that predict multiple outputs simultaneously.
Multi-output problems appear in many diﬀerent forms: they can diﬀer in data types of the outputs or in the ways outputs depend on one another.
Examples that involve mixed output data types include real-valued multi-target regression (Borchani et al., 2015), multi-label classiﬁcation (Zhang and Zhou, 2013) and the heterogeneous case where a mix of continuous, categorical, or discrete variables are of interest (Moreno-Mu˜noz et al., 2018).
Complex dependencies between outputs also appear in various ways: one output might depend quite simply on inputs but can depend on certain other outputs in a complex way.
The sophisticated dependencies between these outputs require structured modelling.
Multi-output GPs (MOGPs) are a powerful and popular approach to multi-output modelling.
In this paper, we focus on one of the MOGPs that explicitly treats outputs as inputs, called the Gaussian Process Autoregressive Regression (GPAR), studied by Requeima et al.
We generalise GPAR to deal with noisy or missing values in the outputs in a principled way and enable it to model non-Gaussian likelihoods and even heterogeneous data.
We connect the proposed model to
Deep Gaussian Process (DGPs) and leverage the approximate inference methods developed for DGPs (Salimbeni and Deisenroth, 2017).
Main Contribution.
First, we present the Gaussian Process Latent Autoregressive model (GPLAR), combining ideas from DGPs and MOGPs.
Hidden variables are introduced corresponding to noiseless, unobserved but true latent function evaluations that require Bayesian inference.
Second, we ﬁnd that GPLAR can still perform poorly when there are missing values in the ﬁrst few outputs.
We propose a new version of GPLAR inspired by bi-directional
Recurrent Neural Networks to address this limitation.
2. Gaussian Process Latent Autoregressive Model 2.1. The GPAR Model
In the multi-output scenario, assume x and y =
{ y l } Ll =1 are the training inputs and associated observations for L outputs.
We assume all L outputs share the same input space.
We utilize the product rule to decompose the joint distribution over all outputs into a set of univariate conditional distributions.
In particular, GPAR factorizes the distribution of L outputs y 1: L ( x ) =
( y 1 ( x ) , . . .
, y L ( x )) as p ( y 1: L ( x ))
=
p ( y 1 ( x )) p ( y 2 ( x ) | y 1 ( x )) . . .
p ( y L ( x ) | y 1: L − 1 ( x )), so y l ( x ) is generated from y 1: l − 1 ( x ) according to some latent function f m .
GPAR models these latent functions f 1: L with GPs, where their kernels k 1: L can be linear, nonlinear, or composite.
GPAR is a state-of-the-art MOGP model on small multioutput regression problems.
Deﬁciencies of GPAR.
There are limitations in the above formulation of GPAR.
A graphical model of three outputs is shown in Fig.
1( a ), where observation y 1 is directly used as inputs to the functions f 2 and f 3 .
Noisy outputs from an earlier stage result in noisy inputs to a subsequent level.
The original paper solved this by employing a denoising transformation: the posterior predictive mean of preceding outputs are used as inputs instead of observed values.
Furthermore, the inference and learning procedures provided in the original paper are only valid for closed-downwards observations, i.e., for every observation
y ln = y l ( x n ), there are also observations y (1: l − 1) n .
For observations that are not closed downwards, the authors propose to impute the necessary observations with posterior predictive means.
We will show in the experiments section that this imputation method and
GPAR’s layer-by-layer ﬁtting procedure performs poorly on closed-upwards observations.
[Equation]
fyfyf( a ) GPAR x hyfhyfhyf( b ) GPLAR x hyfhyfhyfh (cid:48)gh (cid:48)gh (cid:48)gx ( c ) Bi
[Figure]
GPLAR
2.2. The GPLAR Model
A more principled approach is to perform Bayesian inference.
Instead of directly working on observations, we introduce latent variables h 1:3 for each output, graphically shown in
Fig.
1( b ).
Unfortunately, approximate inference is now required to deal with these variables, which we develop next.
The same approximate inference procedure will also enable nonGaussian likelihoods, such as for classiﬁcation or non-negative data.
We call this model the
Gaussian Process Latent Autoregressive (GPLAR) model.
We describe the probabilistic model for a GPLAR model with L outputs.
The main diﬀerence with DGPs is that all previous hidden variables are propagated to the next layer.
Like GPAR, the latent functions in GPLAR are modelled with GPs: p ( f l | θ l ) =
GP ( f l ; m l , K l ), for l = 1 , . . .
, L .
These functions are then connected in the following way:
[Equation]
Approximate Inference for GPLAR.
The posterior distributions over the latent function mappings, f 1: L , as well as over the intermediate hidden variables h 1: L − 1 are of interest.
Existing work by Salimbeni and Deisenroth (2017); Bui et al.
(2016) for approximating these distributions considers variational approximations (Titsias, 2009) over the latent functions but retains the prior conditionals p ( h ln | f l , h
( l − 1) n ).
Using the same idea, we introduce inducing points to every layer (w.r.t.
output dimensions) of GPLAR.
The approximate posterior and joint distribution (written in terms of the function values at inducing points u ) of a three output GPLAR are as follows:
[Equation]
Evidence Lower Bound.
By applying the Jensen’s inequality, we get a lower bound for the log-marginal likelihood (ELBO):
[Equation]
The diﬀerence between the exact log-marginal likelihood and the ELBO is equal to the KL divergence between the approximate posterior (Eq. 1) and the true one (Eq. 2).
Maximizing the ELBO w.r.t.
the variational parameters and the hyperparameters, we simultaneously obtain approximations to the log-marginal likelihood and the posterior.
The second term in Eq.
3 decomposes over across the training instances and output dimensions l .
This term can be rewritten as (cid:80)
[Equation]
[Equation]
( h ln ) log p ( y ln | h ln ) dh
ln , where q ( h ln ) =
(cid:82) q ( h ln | h (1: l − 1) n , x n ) . . .
q ( h 1 n | x n ) dh 1 n . . .
dh ( l − 1) n .
Positing a Gaussian form for each variational distribution, q ( u l ) =
N ( u l ; m l , S l ), we notice that the latent function, f l , can be analytically marginalized out at each layer:
q ( h ln | h (1: l − 1) n , x n ) = N ( µ h l | h 1: l − 1 (ˆ x ln ) , σ 2 h l | h 1: l − 1 (ˆ x ln ))
where,
[Equation]
with ˆ x ln = ( x n , h (1: l − 1) n ) is concatenation of the input and previous hidden variables and
Z l is the location of inducing points at each layer.
Notice that for q ( h 1 n | x n ), the distribution does not marginalise out previous hidden layers, and is therefore simply a Gaussian predictive distribution.
However, for q ( h 2 n | x n ) = (cid:82) h 1 n q ( h 2 n | h 1 n , x n ) q ( h 1 n | x n ), the resulting q ( h 2 n ) is a complicated inﬁnite mixture of Gaussian densities (Bui, 2018), which can be multi-modal or heavy-tailed.
To sample from this, we use a nested simple Monte
Carlo method (Salimbeni and Deisenroth, 2017).
When further propagating h 2 n and h 1 n to the posterior of h 3 n in GLPAR, samples are drawn from a uniformly weighted mixture of Gaussian: q ( h 3 n | x n ) ≈ 1 R (cid:80) r
q ( h 3 n | h 2 nr , h 1 nr , x n ), where h 1 nr ∼ q ( h 1 n | x n ) , h 2 nr ∼ q ( h 2 n | h 1 nr , x n ).
To obtain low variance gradients, we apply the reparametrisation trick (Kingma and Welling, 2013) to recursively draw samples h lnr ∼ q ( h ln | h (1: l − 1) n , x n ) as h lnr = µ h l | h 1: l − 1 (ˆ x ln ) + (cid:15) lnr · σ 2 h l | h 1: l − 1 (ˆ x ln ) , (cid:15) lnr ∼ N (0 , 1).
When using a non-Gaussian likelihood, log p ( y ln | h ln ), additional approximations such as another step of simple Monte
Carlo sampling are required.
Treatment of Inducing Inputs.
Treatment of the inducing inputs for the ﬁrst layer is standard.
However, for higher layers, choosing the locations for inducing points is less straightforward.
These locations also require values for the corresponding previous function values, which should be consistent with the inducing points at those previous layers.
Consequently, free optimization of inducing inputs at each layer is no longer appropriate.
To remedy this, we take the mean of q ( u l ) as inducing inputs for the next layer l + 1.
The resulting locations for the inducing inputs at each layer l are then (cid:2) Z m. . .
m l − 1 (cid:3) , where m l denotes the mean of each variational distribution.
In this setup, the inducing inputs are “automatically” optimized since they are variational parameters themselves.
Please refer to supplementary material (Xia et al.) for more details.
Bi-directional GPLAR.
In its current form, GPLAR has diﬃculty dealing with many outputs.
Although the end-to-end training of all layers ﬁts all layers simultaneously, unlike
GPAR, which ﬁts the layers in a greedy fashion, when the outputs dimension becomes large, back-propagation through the autoregressive structure becomes diﬃcult.
Moreover for realworld data, dependencies between two outputs is often asymmetric.
Hence, an incorrect ordering of the outputs in GPLAR would model dependencies in the wrong directions.
To alleviate this, we take inspiration from the bi-directional RNN model.
The basic idea is to split each hidden state into two in order to model the forward and the backward direction separately, both of which are connected to the same output.
Inspired by this idea, we run another GPLAR model in reverse; the graphical model is shown in Fig.
1( c ).
The hidden variables from both directions at each layer are aggregated and in the case of a regression problem, supplemented with noise to produce the observations.
We will demonstrate that this structure can produce a better predictive mean and better calibrated uncertainty estimates.
3. Experiments
Synthetic Data Experiments.
We ﬁrst compare the ability of GPAR and GPLAR to model synthetic data generated from a multi-output GP where dependencies between outputs are nonlinear (see supplementary for full details).
We calculate the held-out log-
[Figure]
likelihood (HLL) for a range of values for the variance of noise.
It is observed from Fig.
2( a ) that the higher outputs of GPAR are unstable, suggesting that noisy outputs from previous layers harms predictions for the next output.
In contrast, GPLAR’s HLL is observed to always overlap or locate higher than the 95% conﬁdence upper bounds of GPAR.
In the third output of GPAR, the HLL is extremely negative when the noise variance is close to zero.
It turns out that, in this case, the predictive variances for the third output are all near zero.
In comparison, the predictive variances of GPLAR are at an appropriate level.
These results indicate that GPLAR can better handle noisy observations, and is more robust to under-ﬁtting and over-ﬁtting.
Next, we extend GPLAR to non-Gaussian likelihoods and heterogeneous outputs.
Data are drawn from 4 synthetic GPs.
The last two outputs are converted to binary outputs using the sigmoid function; labels are then generated by drawing from the corresponding Bernoulli distributions.
It is observed from
Fig.
2( b ) that information learned from previous tasks helps GPLAR predict the second labeling task.
The predictive mean nearly recovers the true underlying process, and the uncertainty is greatly reduced.
As expected, independent GPs (IGPs) fail to capture the dependencies and revert to the prior distribution in areas without observations.
Real-World Data Experiments.
In this section, we evaluate GPLAR and bi-GPLAR and compare to other models on two standard datasets where GPAR has been shown to be the state-of-the-art.
One is the electroencephalogram (EEG) dataset 1 , consisting of one second of measurements from 7 electrodes mounted on a patient’s scalp.
The second dataset is the exchange rates dataset 2 , consisting of one year of exchange rates w.r.t.
USD of ten international currencies and three metals in one year.
The task is to predict missing values in some of the outputs given all other observations.
The results are presented in Table.
3( a ), and refer to supplementary materials for corresponding ﬁgures.
For EEG, it is observed that predictions of GPAR over F 1 are over-conﬁdent which leads to large HLL, while the uncertainty over F1 from GPLAR is well-calibrated.
As for the exchange rates, although only the SMSE of GPLAR over “USD/AUD” is signiﬁcantly lower than that of GPAR, observe from ﬁgures that GPLAR gives predictions with reasonable uncertainty even outside the missing area, while GPAR produces overconﬁdent predictions with a lot more wigglings.
We further compare the performance of bi-GPLAR to the that of IGPs, GPAR in single direction, and DGPs on the EEG/exchange-rate dataset averaged over 5 patients/years respectively.
The results are shown in Fig.
3( b ).
Since the EEG datasets contains almost noiseless measurements, GPAR in the forward direction performs better than GPLAR for closed-downwards observations.
However, GPAR performs signiﬁcantly worse on closedupwards observations.
Bi-directional GPLAR gives a performance between GPARs in the two directions but GPAR does take more time.
Hence, in situations when noise is not dominant, one should use GPLAR with a natural ordering of outputs.
For the exchange rate, the observations are much noisier and sometimes contain severe outliers.
Bi-GPLAR gives the best results for all four outputs except for “USD/MXN”, when correlations are hard to ﬁnd for all models.
Bi-GPLAR also gives the best average performance over all datasets with closed-upwards or closed-downwards observations, and generally performs better than DGPs and IGPs.
Although DGPs model correlation between outputs, the experiments show that DGPs struggle to leverage knowledge of observed outputs for predicting missing outputs.
GPLAR can also be applied to non-time-series data.
We refer the reader to supplementary materials for experiments on more real-world data including radar image features and latitude-longitude spatial points as input.
[Table]
[Figure]
4. Conclusion
We have introduced GPLAR, an extension of GPAR that deals with noisy outputs using a fully Bayesian approach, enabling the resulting model to work with non-Gaussian or even heterogeneous likelihoods.
We further extended the GPLAR model to a bi-directional version.