A UTOREGRESSIVE Q UANTILE F LOWS FOR P REDICTIVE U NCERTAINTY E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Numerous applications of machine learning involve predicting ﬂexible probability distributions over model outputs.
We propose Autoregressive Quantile Flows, a ﬂexible class of probabilistic models over high-dimensional variables that can be used to accurately capture predictive aleatoric uncertainties.
These models are instances of autoregressive ﬂows trained using a novel objective based on proper scoring rules, which simpliﬁes the calculation of computationally expensive determinants of Jacobians during training and supports new types of neural architectures.
We demonstrate that these models can be used to parameterize predictive conditional distributions and improve the quality of probabilistic predictions on time series forecasting and object detection
I NTRODUCTION
Reasoning about uncertainty via the language of probability is important in many application domains of machine learning, including medicine (Saria, 2018), robotics (Chua et al., 2018; Buckman et al., 2018), and operations research (Van Roy et al., 1997).
Especially important is the estimation of predictive uncertainties (e.g., conﬁdence intervals around forecasts); in applications such as clinical diagnosis (Jiang et al., 2012) or decision support systems (Werling et al., 2015), estimating uncertainty can be as important as obtaining high accuracy (Kuleshov and Liang, 2015).
Normalizing ﬂows (Rezende and Mohamed, 2016; Papamakarios et al., 2019; Kingma et al., 2016) are a popular framework for deﬁning probabilistic models, and can be used for density estimation
(Papamakarios et al., 2017), out-of-distribution detection (Nalisnick et al., 2019), content generation (Kingma and Dhariwal, 2018), and more.
Flows feature tractable posterior inference and maximum likelihood estimation; however, maximum likelihood estimation of ﬂows requires carefully designing a family of bijective functions that are simultaneously expressive and whose Jacobian has a tractable determinant.
In practice, this makes ﬂows time-consuming to design and computationally expensive to train.
This paper takes a step towards addressing this limitation of normalizing ﬂows by proposing new objectives that contribute towards alleviating the computational cost of calculating determinants of
Jacobians.
Speciﬁcally, we argue for training ﬂows using an objective that is different from classical maximum likelihood and is instead based on proper scoring rules (Gneiting and Raftery, 2007), a standard tool in the statistics literature for evaluating the quality of probabilistic forecasts.
We show that this objective can be used to train normalizing ﬂows and that it simpliﬁes the computation of
Jacobians in certain types of ﬂows.
We introduce autoregressive quantile ﬂows (AQFs), a framework that combines the above learning objective with a set of architectural choices inspired by classical autoregressive ﬂows.
Quantile ﬂows possess characteristics that represent an improvement over existing ﬂow models—including supporting neural architectures that simultaneously provide fast training and sampling— in addition to the usual beneﬁts of ﬂows (exact posterior inference and density estimation).
Interestingly, quantile ﬂows can be interpreted as extensions of quantile functions to multiple dimensions.
We use AQFs as the basis for quantile ﬂow regression (QFR), an approach to predictive uncertainty estimation in which a probabilistic model directly outputs a normalizing ﬂow as the predictive distribution.
The QFR approach enables neural networks to output highly expressive probabilistic predictions that make very little assumptions on the form of the predicted variable and that improve uncertainty estimates in probabilistic and Bayesian models.
In the one-dimensional case, our approach yields quantile function regression and cumulative distribution function regression , two simple, general, and principled approaches for ﬂexible probabilistic forecasting in regression.
In addition, we demonstrate the beneﬁts of AQFs on probabilistic modeling tasks that include density estimation and autoregressive generation.
Across our sets of experiments, we observe improved performance, and we demonstrate properties of quantile ﬂows that traditional ﬂow models do not possess (e.g., sampling with ﬂexible neural parameterizations).
Contributions.
In summary, this work (1) introduces new objectives for ﬂow models that simplify the computation of determinants of Jacobians, which in turn greatly simpliﬁes the implementation of ﬂow models and extends the class of models that can be used to parameterize ﬂows.
We also (2) deﬁne autoregressive quantile ﬂows based on this objective, and highlight new architectures supported by this framework.
Finally, (3) we deploy AQFs as part of quantile ﬂow regression, and show that this approach improves upon existing methods for predictive uncertainty estimation.
B ACKGROUND
Notation.
Our goal is to learn a probabilistic model
p ( y ) ∈ ∆( R d ) in the space ∆( R d ) of distributions over a high-dimensional y ∈ R d ; we use y j ∈ R to denote components of y .
In some cases, we have access to features x ∈ X associated with y and we want to train a forecaster
H : X → ∆( R d ) that outputs a predictive probability over y conditioned on x .
2. N ORMALIZING F LOWS AND A UTOREGRESSIVE G ENERATIVE M ODELS
A normalizing ﬂow deﬁnes a distribution p ( y ) via an invertible mapping f θ : R d → R d with parameters θ ∈ Θ that describes a transformation between y and a random variable z ∈ R d sampled from a simple prior z ∼ p ( z ) (Rezende and Mohamed, 2016; Papamakarios et al., 2019).
We may compute p ( y ) via the change of variables formula p ( y ) =
(cid:12)(cid:12)(cid:12) ∂f θ ( z ) − 1 ∂z (cid:12)(cid:12)(cid:12) p ( z ) , where (cid:12)(cid:12)(cid:12)
∂f θ ( z ) − 1 ∂z (cid:12)(cid:12)(cid:12) denotes the determinant of the inverse Jacobian of f θ .
In order to ﬁt ﬂow-based models using maximum likelihood, we typically choose f θ to be in a family for which the Jacobian is tractable.
A common way to deﬁne ﬂows with a tractable Jacobian is via autoregressive models of the form
[Equation]
where τ ( z j ; h j ) is an invertible transformer, a strictly monotonic function of z j , and c j is the j -th conditioner, which outputs parameters h j for the transformer.
As long as τ is invertible, such autoregressive models can be used to deﬁne ﬂows (Papamakarios et al., 2019).
2. E VALUATING F ORECASTS WITH P ROPER S CORING R ULES
A common way to represent a probabilistic forecast in the statistics and forecasting literature is via a cumulative distribution function (CDF)
F : R d → [0 , 1] ; any probability distribution can be represented this way, including discrete distributions.
Since F is monotonically increasing in each coordinate, when y is one dimensional, we may deﬁne its inverse Q : [0 , 1] → R called the quantile function (QF), deﬁned as Q ( α )
= inf { y (cid:48) ∈ R | F ( y (cid:48) ) ≥ α } .
In the statistics literature, the quality of forecasts is often evaluated using proper scoring rules (or proper scores; Gneiting and Raftery (2007)).
For example, when predictions take the form of
CDFs, a popular scoring rule is the continuous ranked probability score (CRPS), deﬁned for two
CDFs F and G as CRPS ( F, G ) = (cid:82) y ( F ( y )
− G ( y ))
2 dy.
When we only have samples y 1 , ..., y m from G , we can generalize this score as 1 m (cid:80) mi =1 (cid:82) y ( F ( y )
− I
( y − y i )) 2 dy.
Alternatively, we can evaluate the α -th quantile Q ( α ) of a QF Q via the check score L : R × R → R + deﬁned as
L α ( y, f ) =
α ( y − f ) if y ≥ f
and (1 − α )
( f − y ) otherwise.
The check score also provides a consistent estimator for the conditional quantile of any distribution.
T AKING S TEPS B EYOND M AXIMUM L IKELIHOOD L EARNING OF F LOWS
Maximum likelihood estimation of ﬂows requires carefully designing a family of bijective functions that are simultaneously expressive and whose Jacobian has a tractable determinant.
In practice, this makes ﬂows time-consuming to design and computationally expensive to train.
In this paper, we argue for training ﬂows using objectives based on proper scoring rules (Gneiting and Raftery, 2007).
3. L EARNING S IMPLE F LOWS WITH P ROPER S CORING R ULES
We begin with the one dimensional setting, where a ﬂow f θ : R → R is a bijective mapping that can be interpreted as a QF.
Alternatively, the reverse ﬂow f − 1 θ can be interpreted as a CDF.
We will use
Q θ , F θ to denote f θ and f − 1 θ , respectively; our goal is to ﬁt these models from data.
In order to ﬁt models of the cumulative distribution and the quantile function, we propose objectives based on proper scoring rules.
We propose ﬁtting models F θ of the CDF using the CRPS:
[Equation] [Equation]
When dealing with a model Q θ of the QF, we propose an objective based on the expected check score
[Equation] [Equation]
) where L α is a check score targeting quantile α .
We refer to this objective as the quantile loss .
This objective has been used previously to train value functions in reinforcement learning as well as conditional distributions in autoregressive models (Dabney et al., 2018a;b).
In this paper, we describe its application to modeling aleatoric predictive uncertainties.
The parametric form of Q θ or F θ can be any class of strictly monotonic (hence invertible) functions.
Previous works have relied on afﬁne or piecewise linear functions (Wehenkel and Louppe, 2021), sum-of-squares (Jaini et al., 2019), monotonic neural networks (Huang et al., 2018; Cao et al., 2019), and other models.
Any of these choices suits our framework; we provide more details below.
Equivalence Between the CRPS and Quantile Losses So far, we have described two methods for ﬁtting a one-dimensional ﬂow model.
Their objectives are actually equivalent.
Proposition 1.
For a CDF F : R → [0 , 1] and y (cid:48) ∈ R , the CRPS and quantile losses are equivalent:
[Equation]
This fact appears to be part of statistics folk knowledge, and we have only ever seen it stated brieﬂy in some works.
We provide a complete proof in the appendix.
See (Laio and Tamea, 2007) for another argument.
If the models F θ , Q θ are analytically invertible (e.g., they are piecewise linear), we are free to choose ﬁtting the CDF or its inverse.
Other representations for F will not lead to analytically invertible models, which require choosing a training direction, as we discuss below.
Practical Implementation.
The quantile and the CRPS losses both involve a potentially intractable integral.
We approximate the integrals using Monte-Carlo; this allows us to obtain gradients using backpropagation.
For the quantile loss, we sample α uniformly at random in [0 , 1] ; for the CRPS loss, we choose a reasonable range of y (usually, centered around y i ) and sample uniformly in that range.
This approach works well in practice and avoids the complexity of alternative methods such as quadrature (Durkan et al., 2019).
3. L EARNING H IGH -D IMENSIONAL A UTOREGRESSIVE F LOWS
Next, we extend our approach to high-dimensional autoregressive ﬂows by ﬁtting each conditional distribution with the quantile or the CRPS loss.
We start with a general autoregressive ﬂow deﬁned as
[Equation]
where τ ( z j ; h j ) is an invertible transformer and
c j is the j -th conditioner As in the previous section, we use Q h j , F h j to denote τ ( z j ; h j ) and τ − 1 ( y j ; h j ) , respectively.
We train autoregressive ﬂows using losses that decompose over each dimension as follows:
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
where L (1) and L (2) are, respectively, the CRPS and quantile objectives deﬁned in the previous section and applied component-wise.
We average the losses on a dataset of n points { y i } ni =1 .
Sampling-Based Training of Autoregressive Flows.
Crucially, Equation 5 deﬁnes not only new objectives but also new training procedures for ﬂows.
Speciﬁcally, it deﬁnes a sampling-based learning process that contrasts with traditional maximum likelihood training.
In order to train with objectives L (1) , L (2) , we perform Monte Carlo sampling to approximate an intractable integral.
Thus, we sample a source variable ( y or z ) from a uniform distribution and transform it into the other variable, which we call the target , after which we compute the loss and its gradient.
This is in contrast to maximum-likelihood learning, where we take the observed variable y , transform it into z , and evaluate p ( z ) as well as the determinant of the Jacobian.
The choice of source and target variable depends on the choice of objective function.
Each choice yields a different type of training procedure; we refer to these as forward and reverse training.
Forward and Reverse Training.
In forward training, we sample a z ∼ U ([0 , 1]) , pass it through
Q h j , obtaining y j .
Thus, at training time, we perform generation and our loss based on the similarity between samples and the real data y .
In contrast, we train standard ﬂows by passing an input y into the inverse ﬂow to obtain a latent z (and no sampling is involved).
Our process share similarities with the training of GANs, but we don’t rely on a discriminator.
The forward training process enables us to support neural architectures that maximum likelihood training does not support, as we discuss below.
In reverse training, we sample y ∼ U ([ y min , y max ]) (where the bounds in U are chosen via a heuristic), and we compute the resulting z via F θ .
The CRPS loss deﬁned on z seeks to make the distribution uniform on average, which is consistent with the properties of inverse transform sampling.
In this case τ ( · ; h j ) can be interpreted as a CDF function conditioned on y <
j via h j .
On Computing Determinants of Jacobians.
Interestingly, since quantile ﬂows are trained using variants of the CRPS rather than maximum likelihood, they do not rely on the change of variables formula, and do not require computing the derivative ∂τ/∂z for each transformer τ ( · ; h j ) .
This fact has a number of important beneﬁts: (1) it greatly simpliﬁes the implementation of ﬂow-based models, and (2) it may also support families of transformers for which computing derivatives ∂τ/∂z
would have otherwise been inconvenient.
Note however, that our approach does not fully obviate the need to think about the tractability of
Jacobians: we still rely on autoregressive models for whom the Jacobian is triangular.
We will explore how to further generalize our approach beyond triangular Jacobians in future work.
A UTOREGRESSIVE Q UANTILE F LOWS
Next, we introduce autoregressive quantile ﬂows (AQFs), a class of models trained with our set of novel learning objectives.
Quantile ﬂows improve over classical ﬂow models by, among other things, supporting neural architectures that simultaneously provide fast training and sampling.
An AQF deﬁnes a mapping between z and y that has the form y j
= τ ( z j ; h j ) h
j = c
j
( y <j ) , as in a classical ﬂow, with τ ( z j ; h j ) being an invertible transformer and c j being the j -th conditioner.
Architectures for Quantile Flows.
Crucially, we are free to select a parameterization for either τ ( z j ; h j ) or its inverse τ − 1 ( y j ; h j ) and the resulting model will be trainable using one of our objectives L (1) , L (2) .
This is in sharp contrast to maximum likelihood, in which we need to learn a model of τ − 1 ( y j ; h j ) in order to recover the latent z from the observed data y and compute p ( z ) .
This latter limitation has the consequence that in classical ﬂows in which the transformer τ − 1 is not invertible (which is in most non-afﬁne models), it is difﬁcult or not possible to perform sampling (since sampling requires τ ).
As a consequence, most non-afﬁne ﬂow models can only be used for density estimation; ours is the ﬁrst that be used for generation.
We give examples below.
Afﬁne, Piecewise Linear, and Neural Quantile Flows.
Afﬁne transformers have the form τ ( z ; a, b )
=
a · z + b , hence they are analytically invertible.
A special case is the Gaussian transformer with parameters µ, σ 2 , which yields the masked autoregressive ﬂow (MAF; Papamakarios et al.
(2017)) architecture.
The MAF can be trained in our framework using maximum likelihood or
L (1) , L (2) .
Afﬁne ﬂows further extend to piecewise linear transformers, increasing expressivity while maintaining analytical invertiblity.
Training afﬁne and piecewise linear transformers provides both a forwards and a reverse mapping between y and z in closed form.
For higher expressivity, we may parameterize transformers with monotonic neural networks, which can be constructed from a composition of perceptrons with strictly positive weights.
Huang et al.
(2018) proposed an approach where a conditioner c ( y <j ) directly outputs the weights of a monotonic neural network which models τ − 1 ( y j , c
( y <j )) .
In our experiments, τ ( z j , c ( y <j )) is simply parameterized by a network with positive weights that outputs y j as based on z j , y <j , similarly to (Cao et al., 2019), although AQFs also support the approach of Huang et al.
Comparison to Other Flow Families.
In general, any transformer family that can be used within autoregressive models can be used within the quantile ﬂows framework, including integration-based transformers (Wehenkel and Louppe, 2021) spline approximations (Müller et al., 2019; Durkan et al., 2019; Dolatabadi et al., 2020), piece-wise separable models, and others.
A key feature of our framework is that it supports both forwards and reverse training while alternatives do not.
For example, when training a neural autoregressive ﬂow (Huang et al., 2018), we need to compute y from z to compute the log-likelihood.
Hence, we can only parameterize τ in the reverse direction.
However, since τ is analytically invertible, this precludes us from sampling from it.
Our method on the other hand, can be parameterized in the forwards direction using a ﬂexible neural approximator for τ , making it possible to both sample and use a ﬂexible neural approximator for τ .
Additionally, modeling ﬂexible CDFs enables us to study broader classes of distributions, including discrete ones.
We perform early experiments and leave the full extension to future work.
P REDICTIVE U NCERTAINTY E STIMATION U SING Q UANTILE F LOWS
Probabilistic models typically assume a ﬁxed parametric form for their outputs (e.g., a Gaussian), which limits their expressivity and accuracy (Kuleshov et al., 2018).
Here, we use AQFs as the basis for quantile ﬂow regression (QFR), which models highly expressive aleatoric uncertainties over high-dimensional variables by directly outputting a normalizing ﬂow.
5. Q UANTILE F LOW R EGRESSION
The idea of Quantile Flow Regression (QFR) is to train models H :
X → ( R d → [0 , 1]) that directly output a normalizing ﬂow.
This approach does not impose any further parametric assumptions on the form of the output distribution (besides that it is parameterized by an expressive function approximator), enabling the use of ﬂexible probability distributions in regression, and avoiding restrictive assumptions of the kind that would be made by Gaussian regression.
From an implementation perspective, H outputs quantile ﬂows of the form
[Equation]
where each conditioner c i ( y <
i , g ( x )) in a function of an input x processed by a model g ( x ) .
We provide speciﬁc examples of this parameterization in our experiments section.
The entire structure can be represented in a single computational graph and is trained end-to-end using objectives L (1) , L (2) .
Estimating Probabilities from Quantile Flows.
In practice, we may also be interested to estimate the probability of various events under the joint distribution of y .
This may be done by sampling from the predictive distribution and computing the empirical occurrence of the events of interest.
See our time series experiments for details, as well as the appendix for pseudocode of the sampling algorithm.
Note that while we cannot directly estimate events such as P ( y 1 ≤ y ≤ y 2 ) from a multi-variate quantile ﬂow, we can always compute densities of the form P ( y ) .
When ﬁtting ﬂows in the reverse direction, we can simply differentiate each transformer dτ/dy to obtain conditional densities.
When ﬁtting ﬂows in the forward direction, we can use the formula dτ/dz = 1 /P ( τ ( z )) .
5. Q UANTILE AND C UMULATIVE D ISTRIBUTION F UNCTION R EGRESSION
In the one-dimensional case, our approach yields quantile function regression and cumulative distribution function regression (CDFR), two simple, general, and principled approaches for ﬂexible probabilistic forecasting in regression.
In quantile function regression, a baseline model H outputs a quantile function Q ( α ) ; in practice, we implement this model via a neural network f ( x, α ) that takes as input an extra α ∈ [0 , 1] and outputs the α -th quantile of the conditional distribution of y given x .
We train the model f with the quantile loss L (2) while sampling α ∼ U ([0 , 1]) during the optimization process.
We provide several example parameterizations of f in the experiments section.
Cumulative distribution function regression is deﬁned similarly to the above and is trained with the
CRPS loss L (1) .
Quantile vs. Cumulative Distribution Functions.
The quantile function Q makes it easy to derive α -% conﬁdence intervals around the median by querying Q (1 − α/ 2) and Q ( α/ 2) .
Conversely, the
CDF function f makes it easy to query the probability that y falls in a region [ y 1 , y 2 ] via f ( y 2 ) − f ( y 1 ) .
Each operation can also be performed in the alternative model, but would require bisection or grid search.
We suspect users will ﬁt one model or both, depending on their needs.
Extensions to Classiﬁcation.
A CDF can also be used to represent discrete random variables; thus our method is applicable to classiﬁcation, even though we focus on regression.
We may ﬁt a CDF F over K numerical class labels y 1 < ... <
y K ; the optimal form of F is a step function.
When F is not a step function, we can still extract from it approximate class membership probabilities.
For example, in binary classiﬁcation where classes are encoded as 0 and 1 , we may use ( F (1 − (cid:15) )
− F (0)) / 2 as an estimate of the probability of y = 0 .
We provide several experiments in this regime.
E XPERIMENTS
We evaluate quantile ﬂow regression and its extensions against several baselines, including Mixture
Density Networks (MDN; Bishop (1994)), Gaussian regression, Quantile regression (QRO; in which we ﬁt a separate estimator for 99 quantiles indexed by α = 0 .
We report calibration errors (as in Kuleshov et al.
(2018)), check scores (CHK), the CRPS (as in Gneiting and Raftery (2007)) and L1 loss (MAE); both calibration error (deﬁned in appendix) and check score are indexed by α = 0 .
Additional experiments on synthetic datasets can be found in the appendix.
6. E XPERIMENTS ON UCI D ATASETS
Datasets We used four benchmark UCI regression datasets (Dua and Graff, 2017) varying in size from 308-1030 instances and 6-10 continuous features.
We used two benchmark UCI binary classiﬁcation datasets with 512 and 722 instances with 22 and 9 features respectively.
We randomly hold out 25% of data for testing.
Models.
QFR and CDFR were trained with learning rates and dropout rates of 3 e − 3 , 3 e − 4 and (0 . 2 , 0 . 1) respectively as described in Section 5.2.
All of the models were two-hidden-layer neural networks with hidden layer size 64, and ReLU activation functions.
Results.
QFR, on average, outperforms the baselines in each of the one-dimensional UCI tasks (Table 1).
Interestingly, our method was also able to learn a CDF for binary classiﬁcation.
We demonstrated that the CDFR method is able to compete with the neural network baseline and that our
[Table]
method can be potentially extended to classiﬁcation, testing on the Diabetes and KC2 dataset where the Neural Network obtained metrics of 0.717 and 0.832 respectively, and our CDFR obtained 0. and 0.796 respectively.
6. O BJECT D ETECTION
Data.
We also conducted bounding box retrieval experiments against Gaussian baselines on the data corpus VOC 2007, obtained from Everingham et al.
with 20 classes of objects in 9963 images.
[Table]
Models.
We modiﬁed the architecture of He et al.
(2019) which uses a
KL-loss to ﬁt a Gaussian distribution for each bounding box.
The existing architecture has a MobileNet-v2 backbone (Sandler et al., 2019).
Keeping the backbone unchanged, the quantile regressor method adds an alternate head with separate layers, which adds uncertainty bounds to the original output.
The original Gaussian method was trained for a total of 60 epochs, with a learning rate of 2e-4 which is decremented by a factor of 10 at epochs 30 and 45.
The quantile ﬂow layer is trained for an additionalepochs, with the same learning rate.
[Table]
Results.
We implement QFR in conjunction in a standard model, as well as one that uses variational dropout (VD; Gal and Ghahramani (2016)).
In each case, the QFR layer improves uncertainty estimation (as measured by CRPS; Table 3), with the best results coming from a model trained with
QFR and VD.
MAP@50 is deﬁned in the appendix.
This suggests our methods improve both aleatoric and epistemic uncertainties (the latter being estimated by VD).
We illustrate examples of predicted uncertainties on real images in Figure 1.
We also swapped QFR for CDFR, while keeping approximately the same architecture, and compared to the Gaussian baseline using the CRPS.
CDFR also outperforms the Gaussian baseline (Table 2).
6. T IME S ERIES F ORECASTING
Data.
Our time series experiment uses quantile ﬂows to forecast hourly electricity usage.
We follow the experimental setup of Mashlakov et al.
(2021) using their publicly available code.
We use the 2011-2014 Electricity Load dataset, a popular dataset for benchmarking time series models.
It contains 70,299 time series derived from hourly usage by 321 clients across three years.
There exist different ways to pre-process this dataset; we follow the setup of Mashlakov et al.
Our models predict hourly electricity usage 24 hours into the future for each time series, for a total of 1,687,176 predictions.
There are four covariates.
Models.
We benchmark two state-of-the-art time series forecasting models: DeepAR (Salinas et al., 2019) and MQ-RNN (Wen et al., 2018).
Both use an autoregressive LSTM architecture as in
Mashlakov et al.
We train a recurrent model with same LSTM architecture and replace the predictive component with a quantile ﬂow.
Each transformer τ
( z i ; h i ) is a monotone neural network with one hidden layer ofneurons, conditioned on the output
h
i of the LSTM module, acting as the conditioner c i ( y <i , x ) .
[Table]
Results.
Figures 2a and 2b compare predictions from
DeepAR and AQF (in blue) on illustrative time series (red).
We take samples from both models and visualize the (0 . 1 , 0 . 9) quantiles in shaded blue.
While both time series underpredict, the Gaussian is overconﬁdent, while the samples from AQF contain the true (red) curve.
Moreover,
AQF quantiles expand in uncertain regions and contract with more certainty.
We also compare the three methods quantitatively in Table 4.
Both achieve similar levels of accuracy (ND and RMSE are as deﬁned in Salinas et al. , but we have included them in the appendix as well), and AQF produces better uncertainties.
We also found that for 37.2% of time series, DeepAR and AQF performed comparably for CRPS (CRPS within 5% of each other); on 30.3% of time series DeepAR fared better, and on 32.1% AQF did best.
6. G ENERATIVE M ODELING
A key feature of AQFs is that they can be used for generative modeling with expressive neural parameterizations since they can be trained in the forward direction, unlike Huang et al.
We benchmark three different methods for generative ﬂows, MAF trained via maximum likelihood (MAF-LL) and with our quantile loss (MAF-QL), and our proposed method, Neural Quantile Flow (NAQF) with the same number of parameters on various UCI datasets.
The models consist of an
LSTM layer followed by a FC layer, with 40 hidden states in each.
The MAFs generate the parameters for a Gaussian distribution from which the results are sampled.
Results.
Figure 3 shows the error curves of the MAF models trained with log-likelihood and with our loss.
Interestingly, there is little correlation between the two, which is
[Figure]
reminiscent of other model classes trained with different objectives (e.g., LL-Flows and
GANs); in some cases, likelihood can even diverge as we train with our loss.
evaluates CRPS trained with each model.
The best results are achieved with the NAQF, even though it uses the same number of parameters, indicating the value of neural transformers.
[Table]
6. I MAGE G ENERATION
We demonstrate the ability of our model to generate samples using a neural parameterization by generating images of digits from sklearn (Figure 4).
We use 5 LSTM layers with 80 neurons; all models are trained to convergence with learning rate 1e-3, with 500 epochs for the NAQF and MAF-QL, while the
MAF-LL was given an extra 500 epochs to reach convergence.
Additionally, we created a SVM discriminator to differentiate the real data versus the sampled data from the model (D-Loss in Table 6, deﬁned formally in appendix).
The better the sampled data is, the lower the accuracy achieved by the discriminator.
[Table]
NAQF
[Table]
Results.
The NAQF generates signiﬁncantly better samples that are the most challenging to discriminate from real samples, and also echieves the best CRPS values.
The MAF method trained with LL achieves worst results and visibly worse samples, despite having the same number of parameters and being trained for longer.
Note that despite the MAF-QL having a considerably worse NLL, the samples generated are still more representative of the data, according to the discriminative model, and visually the sample quality is much higher.
D ISCUSSION AND C ONCLUSION
Our work extends the framework of ﬂows Papamakarios et al.
(2017; 2019) to use a novel learning objectives.
Similarly, it generalizes implicit quantile networks Dabney et al.
(2018b)
into a normalizing ﬂow framework.
Interestingly, quantile ﬂows can be seen as a generalization of quantile functions to higher dimensions: they deﬁne a mapping that is increasing in each dimension and implements a form of inverse transform sampling.
While several authors previously proposed multi-variate generalizations of quantiles (Serﬂing, 2002; Liu and Wu, 2009), neither of them possess all the properties of a univariate quantile function, and ours is an additional proposal.
A UCI D ATASETS E XPANDED
Yacht .
The yacht dataset consists of 308 instances, 6 features and 1 continuous target.
Boston .
The boston house prices dataset consists of 506 instances, 14 features and 1 continuous target.
Concrete .
The concrete dataset consists of 1030 instances, 8 features and 1 continuous target.
Energy .
The energy efﬁciency dataset consists of 768 instances, 10 features and 1 continuous target.
Classiﬁcation Datasets KC2 .
The kc2 dataset consists of 522 instances, 22 features and 1 binary target.
Diabetes .
The diabetes dataset consists of 769 instances, 9 features and 1 binary target.
Multivariate Datasets ENB .
The enb dataset consists of 768 instances, 10 features and 2 continous targets.
B P ROOF OF P ROPOSITION
Proposition 2.
For a CDF F : R → [0 , 1] and y ∈ R , the CRPS and quantile losses are equivalent:
[Equation]
Proof.
Recall that the pinball loss is deﬁned as
[Equation]
y
[Equation]
f ) if y ≥ f and (1 − α )( f − y ) otherwise.
Observe that we can equivalently write
[Equation]
We take the expected pinball loss and perform the change of variables y
=
F − 1 ( α ) :
[Equation]
where f ( y ) is the probability density function associated with F .
Using integration by parts and the fact that F ( y (cid:48) ) − I ( y (cid:48) ≤ y )
→ 0 as y → ∞ and as y → −∞ , we obtain
[Equation]
C E XTRA F IGURES
[Figure]
D S YNTHETIC E XPERIMENTS
In addition to the real-world dataset experiments, we also conducted experiments with synthetic datasets, which provides additional insights into the properties of autoregressive quantile ﬂows.
[Table]
[Table]
D. O NE -D IMENSIONAL E XPERIMENTS
We start with a series of one-dimensional experiments involving the a simple 1D function to which we add additive Gaussian, Beta, and Poisson noise.
The Gaussian and Beta distribution experiments were generated from a base function of f ( x ) = sin( x/ 2) + x/ 10 such that
y
=
f
( x ) + N (0 , 1) and y =
f ( x ) + B (2 , 5) respectively.
The
Poisson distribution was generated from P ( f ( x ) + C ) , where C is a constant term to make all terms non-negative.
We ﬁt these functions using autoregressive quantile ﬂows (AQFs) as well as the following Baselines:
Gaussian regression, Mixture Density Networks (MDNs)
[Table]
and Poisson regression.
All models implement the same neural networks and differ only in how the ﬁnal layer represents aleatoric uncertainty.
We calculated check score and CRPS metrics for each model.
In addition, we calculated the metrics for the true distribution, as a upper bound on the performance of the models.
As shown in Table 9,
AQF is much closer to the true distribution compared to the other models with Gaussian assumptions, especially when performed on non-Gaussian distributions.
This shows that ﬁtting an expressive AQF model better captures predictive uncertainty than alternative models that make parametric assumptions on the output distribution.
In other words, using a parametric distribution (e.g., Poisson or Gaussian) works well if the underlying data actually follows that distribution.
When using an AQF, we don’t need to select a predictive distribution type , it just works out of the box as if we knew it.
[Table]
We also report calibration error, deﬁned as the absolute difference between the percentage of points below the predicted quantile curves and the actual quantile value (Table 7).
For a perfectly Gaussian noise distribution, the QRF (analogous to AQF in 1-dimension) and QRO methods (QRO consists of ﬁtting a separate estimator for 99 quantiles indexed by α = 0 .
01 , ..., 0 . 99 ) are comparable to a Gaussian regressor in terms of calibration.
When the distributions become non-Gaussian, QRF is adaptable whereas the Gaussian fails to capture the distribution.
Thus, the Gaussian maximum likelihood loss tends to be overconﬁdent, a fact also demonstrated in Kuleshov et al.
; AQF addresses the overconﬁdence.
Supervised Accuracy and Calibration at Key Quantiles We also report calibration values at key quantiles (25%, 50%, 75%) in Table 8).
We also measure predictive accuracy in terms of RMSE and demonstrate that our method obtains good estimates of uncertainty without sacriﬁcing supervised performance in terms of regression quality.
[Figure]
In Figure 6a and 6b, we report 80% conﬁdence intervals for AQF and Gaussian regression methods on the synthetic beta noise data for a visualization of the quantile predictions.
D. M ULTI -D IMENSIONAL E XPERIMENTS
We also conducted multi-dimensional experiments with a d -dimensional target y ; we model the ﬁrst dimension by y 0 = 5 sin( x/ 3) + x + N (0 , 1) ; then, further dimensions are modeled by y d = 5 sin( y d − 1 / 3) + y d − 1 + N (0 , 1) .
We use both the marginal CRPS and the joint CRPS metrics, as described in Jordan et al.
(2018), for both a simple 2d-experiment as well as a more complex 50-dimensional experiment.
The methods we evaluate are our proposed AQF, a Gaussian Markov Random Field (a type of structured prediction algorithm that assumes a multi-variate Gaussian distribution on the output), and a model in which we train a multi-variate Gaussian distribution using quantile loss instead of log likelihood (denoted as “Quantile" in the table).
As in the 1D synthetic experiments, we also compare metrics against samples drawn from the true distribution to obtain an upper bound.
Table 10 shows that AQF does a better job at capturing the high-dimensional distribution than models that assume a more restricted parametric form, such as the Gaussian MRF.
Simply replacing the objective function with quantile loss instead of standard Gaussian noise does not provide as signiﬁcant of a gain as AQF does.
In other words, existing models (e.g., a Gaussian MRF) are not sufﬁciently expressive to capture the multi-variate distribution that he we have deﬁned.
In contrast the AQF layer is able to achieve close to approach the theoretical upper bound without making any additional modeling assumptions.
[Table]
E A LGORITHM
Algorithm
Sample α = U (0 , 1)
Sample a training instance ( x, y ) from the dataset (or a batch of instances)
Compute prediction y pred =
f ( x, α )
Compute quantile loss L α ( y, y pred ) .
Take a gradient step quantile loss L α ( y, y pred ) .
F A DDITIONAL D EFINITIONS FOR U SED M ETRICS F. C ALIBRATION MAE
If we have a 1-dimensional model which predicts a curve F ( q ) for quantile q, then we have that our Calibration MAE on test set Y is given by  99 (cid:88) q = | P ( Y ≤ F ( q )) − q |
F. MAP@
MAP@50 denotes the mean average precision for an IoU value of 0.5.
IoU is computed as, given ground truth boxes G and predicted box B, G ∩ B G ∪ B ≥ 0 .
MAP is then calculated as the AUC of the precision-recall curve given the IoU.
F. RMSE AND ND (T IME S ERIES )
We will deﬁne data point
z
i,t to be the t-th time in the i-th context window, and ˆ z to be the prediction for point z .
Then RMSE is deﬁned as
RMSE = (cid:115)
[Equation]
(cid:88)
[Equation]
ND (Normalized Deviation) is calculated as
[Equation]
F.
LL, Q-L OSS , AND D-L OSS (I
MAGE
G ENERATION )
Log Likelihood and Q-Loss (quantile loss as deﬁned in our paper) with respect to the digits generation task are optimization objectives and not metrics.
We measure these with to depict how one is affected in the same model trained on the other loss function.
D-Loss is measured as the accuracy with which a SVM can discern the difference between generated samples S and true samples Y, which are labeled 0 and 1 respectively.
So to say, given validation set
X, we have that
[Equation]