Original: V ARIATIONAL D ETERMINISTIC U NCERTAINTY
Revised: V ARIATIONAL D ETERMINISTIC U NCERTAINTY

Original: Q UANTIFICATION
Revised: Q UANTIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Building on recent advances in uncertainty quantiﬁcation using a single deep deterministic model (DUQ), we introduce variational Deterministic Uncertainty
Revised: Building on recent advances in uncertainty quantiﬁcation using a single deep deterministic model (DUQ), we introduce variational Deterministic Uncertainty

Original: Quantiﬁcation (vDUQ).
Revised: Quantiﬁcation (vDUQ).

Original: We overcome several shortcomings of DUQ by recasting it as a Gaussian process (GP) approximation.
Revised: We overcome several shortcomings of DUQ by recasting it as a Gaussian process (GP) approximation.

Original: Our principled approximation is based on an inducing point GP in combination with Deep Kernel Learning.
Revised: Our principled approximation is based on an inducing point GP in combination with Deep Kernel Learning.

Original: This enables vDUQ to use rigorous probabilistic foundations, and work not only on classiﬁcation but also on regression problems.
Revised: This enables vDUQ to use rigorous probabilistic foundations, and work not only on classiﬁcation but also on regression problems.

Original: We avoid uncertainty collapse away from the training data by regularizing the spectral norm of the deep feature extractor.
Revised: We avoid uncertainty collapse away from the training data by regularizing the spectral norm of the deep feature extractor.

Original: Our method matches SotA accuracy, 96.2% on CIFAR-10, while maintaining the speed of softmax models, and provides uncertainty estimates competitive with Deep Ensembles.
Revised: Our method matches SotA accuracy, 96.2% on CIFAR-10, while maintaining the speed of softmax models, and provides uncertainty estimates competitive with Deep Ensembles.

Original: We demonstrate our method in regression problems and by estimating uncertainty in causal inference for personalized medicine.
Revised: We demonstrate our method in regression problems and by estimating uncertainty in causal inference for personalized medicine.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deploying machine learning algorithms as part of automated decision making systems, such as self driving cars and medical diagnostics, requires implementing fail-safes.
Revised: Deploying machine learning algorithms as part of automated decision making systems, such as self driving cars and medical diagnostics, requires implementing fail-safes.

Original: Whenever the model is presented with a novel or ambiguous situation, it would not be wise to simply trust its prediction.
Revised: Whenever the model is presented with a novel or ambiguous situation, it would not be wise to simply trust its prediction.

Original: Instead, the system should try to get more information or simply withhold or defer judgment.
Revised: Instead, the system should try to get more information or simply withhold or defer judgment.

Original: While signiﬁcant progress has been made towards estimating predictive uncertainty reliably in deep learning (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017), there is no single method that works on large datasets in classiﬁcation and regression without signiﬁcant computation overheads, such as multiple forward passes.
Revised: While signiﬁcant progress has been made towards estimating predictive uncertainty reliably in deep learning (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017), there is no single method that is shown to work on large datasets in classiﬁcation and regression without signiﬁcant computation overheads, such as multiple forward passes.

Original: We propose Variational Deterministic Uncertainty Quantiﬁcation (vDUQ), a method for obtaining predictive uncertainty in deep learning for both classiﬁcation and regression problems in only a single forward pass.
Revised: We propose Variational Deterministic Uncertainty Quantiﬁcation (vDUQ), a method for obtaining predictive uncertainty in deep learning for both classiﬁcation and regression problems in only a single forward pass.

Original: In previous work, van Amersfoort et al.
Revised: In previous work, van Amersfoort et al.

Original: (2020) show that combining a distance aware decision function with a regularized feature extractor, leads to a model (DUQ) that matches a softmax model in accuracy, but is competitive with Deep Ensembles for uncertainty on large datasets.
Revised: (2020) show that combining a distance aware decision function with a regularized feature extractor in the form of a deep RBF network, leads to a model (DUQ) that matches a softmax model in accuracy, but is competitive with Deep Ensembles for uncertainty on large datasets.

Original: The feature extractor is regularized using a two-sided gradient penalty, which encourages the model to be sensitive to changes in the input, avoiding feature collapse, and encouraging generalization by controlling the Lipschitz constant.
Revised: The feature extractor is regularized using a two-sided gradient penalty, which encourages the model to be sensitive to changes in the input, avoiding feature collapse, and encouraging generalization by controlling the Lipschitz constant.

Original: This model, however, has several limitations; for example the uncertainty (a distance in feature space) cannot be interpreted probabilistically and it is difﬁcult to disentangle aleatoric and epistemic uncertainty.
Revised: This model, however, has several limitations; for example the uncertainty (a distance in feature space) cannot be interpreted probabilistically and it is difﬁcult to disentangle aleatoric and epistemic uncertainty.

Original: Additionally, the loss function and centroid update scheme are not principled and do not extend to regression tasks.
Revised: Additionally, the loss function and centroid update scheme are not principled and do not extend to regression tasks.

Original: A probabilistic and principled alternative to Deep RBF networks are Gaussian Processes (GPs) in combination with Deep Kernel Learning (DKL) (Wilson et al., 2016b;a; Hinton & Salakhutdinov, 2008).
Revised: A probabilistic and principled alternative to deep RBF networks are Gaussian Processes (GPs) in combination with Deep Kernel Learning (DKL) (Hinton & Salakhutdinov, 2008; Wilson et al., 2016b).

Original: DKL was introduced as a “best of both worlds” solution: apply a deep model on the training data and learn the GP in feature space, ideally getting the advantages of both models.
Revised: DKL was introduced as a “best of both worlds” solution: apply a deep model on the training data and learn the GP in feature space, ideally getting the advantages of both models.

Original: In practice, however, DKL suffers from the same failure as Deep RBF networks: the deep model is free to map out of distribution data close to the feature representation of the training data, removing the attractive properties of GPs with distance sensitive kernels.
Revised: In practice, however, DKL suffers from the same failure as Deep RBF networks: the deep model is free to map out of distribution data close to the feature representation of the training data, removing the attractive properties of GPs with distance sensitive kernels.

Original: Using insights from DUQ, we are able to mitigate the problems of uncertainty collapse in DKL.
Revised: Using insights from DUQ, we are able to mitigate the problems of uncertainty collapse in DKL.

Original: In particular, we use direct spectral normalization (Gouk et al., 2018;
Revised: In particular, we use direct spectral normalization (Gouk et al., 2018;

Original: Miyato et al., 2018)
Revised: Miyato et al., 2018)

Original: in combination with a ResNet (He et al., 2016), a variation that was suggested in Liu et al.
Revised: in combination with a ResNet (He et al., 2016), a variation that was suggested in Liu et al.

Original: This
Revised: The spectral

Original: [Figure]
Revised: [Figure]

Original:  enforces an upper bound on the Lipschitz constant, while the residual connections enforce a lower bound, obtaining a similar result as the gradient penalty of DUQ by construction.
Revised: normalization enforces smoothness, while the residual connections enforce sensitivity of the feature represenation to changes in the input, obtaining a similar effect as the gradient penalty of DUQ.

Original: We use an  inducing point variational approximation of the GP predictive distribution (Hensman et al., 2015) to  speed up inference in the GP model and decouple it from the dataset size.
Revised: We use an inter-domain inducing point variational approximation of the GP predictive distribution (L´azaro-Gredilla & Figueiras-Vidal, 2009; Hensman et al., 2015), which places inducing points in feature space leading to needing fewer inducing points than previous work (Wilson et al., 2016a). These two techniques combined speed up inference in the GP model and decouple it from the dataset size.

Original: We release our code 1 and hope that it will become a drop in alternative for softmax models with improved uncertainty.
Revised: We release our code 1 and hope that it will become a drop in alternative for softmax models with improved uncertainty.

Original: In Figure 1, we show how vDUQ and Deep Ensembles (Lakshminarayanan et al., 2017) perform on simple 1D regression.
Revised: In Figure 1, we show how vDUQ and Deep Ensembles (Lakshminarayanan et al., 2017), the current state of the art for uncertainty quantiﬁcation (Ovadia et al., 2019), perform on simple 1D regression.

Revised: This task is particularly hard for deep networks as shown in Foong et al.

Original: vDUQ shows the desired behavior of reverting back to the prior away from the data, while the Deep Ensemble extrapolates arbitrarily and conﬁdently.
Revised: vDUQ shows the desired behavior of reverting back to the prior away from the data, while the Deep Ensemble extrapolates arbitrarily and conﬁdently.

Original: In between the two sinusoids, the Deep Ensemble is certain while vDUQ increases its uncertainty.
Revised: In between the two sinusoids, the Deep Ensemble is certain while vDUQ increases its uncertainty.

Original: This task is particularly hard for deep networks as shown in Foong et al.

Original: In summary, our contributions are as follows:
Revised: In summary, our contributions are as follows:

Original: • We stabilize training deep networks using DKL  and for the ﬁrst time match the accuracy  of training a deep network using regular softmax output.
Revised: • We improve training a DKL model and for the ﬁrst time match the accuracy and speed of training a deep network using regular softmax output on standard vision benchmarks.

Original: • Wediscuss the theoretical motivations of spectral normalization for uncertainty, and its relationship to prior work. • We demonstrate  uncertainty estimation whichobtains excellent results on toy data, matches or exceeds the state of the art on CIFAR-10, including ensembling approaches, and state of the art performance ona causal inference benchmark requiring uncertainty.
Revised: • We demonstrate excellent uncertainty quantiﬁcation in classsiﬁcation which matches or exceeds the state of the art on CIFAR-10, including ensembling approaches. • We show state of the art performance on causal inference for personalized medicine, an exciting real world application. This task requires calibrated uncertainty in regression to be able to defer treatment to an expert when uncertainty is high.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Gaussian Processes (GPs) provide an interpretable, explainable and principled way to make predictions, and can work well even with little training data due to their use of Bayesian inference.
Revised: Gaussian Processes (GPs) provide an interpretable, explainable and principled way to make predictions, and can work well even with little training data due to their use of Bayesian inference.

Original: In contrast with neural networks, GPs have high uncertainty away from the training data and on noisy inputs.
Revised: In contrast to deep neural networks, GPs have high uncertainty away from the training data and on noisy inputs.

Original: There are  two main issues with the standard GP setup:  performance on high dimensional inputs and  computational scaling with larger datasets.
Revised: There are however two main issues with the standard GP setup: poor performance on high dimensional inputs and inefﬁcient computational scaling with large datasets.

Original: The poor performance on high dimensional inputs is due to the fact that most standard shift-invariant kernels are based on  the Euclidean distance, which is a poor metric for highly structured data like images (Theis et al., 2016).
Revised: The poor performance on high dimensional inputs is due to the fact that most standard shift-invariant kernels are based on [Figure] the Euclidean distance, which is a poor metric for highly structured data like images (Theis et al., 2016).

Original: While kernels exist that better address this (Van der Wilk et al., 2017; Jacot et al., 2018), these are more computationally expensive and typically still under perform standard convolutional neural networks.
Revised: While kernels exist that better address this (Van der Wilk et al., 2017; Jacot et al., 2018), these are more computationally expensive and typically still underperform standard convolutional neural networks.

Original: Deep Kernel Learning (Calandra et al., 2016; Hinton & Salakhutdinov, 2008; Wilson et al., 2016b) is a way to combine the expressiveness of deep neural networks with the attractive properties of
Revised: Deep Kernel Learning (Calandra et al., 2016; Hinton & Salakhutdinov, 2008; Wilson et al., 2016b) is a way to combine the expressiveness of deep neural networks with the attractive properties of

Original: Gaussian Processes.
Revised: Gaussian Processes.

Original: The core idea is to use a deep neural network inside the kernel of a GP, 
Revised: The core idea is to use a deep neural network inside the kernel of a GP, k

Original: [Figure] k ( x i , x j ) → k ( f θ ( x i ) , f θ ( x j )) , where f θ ( · ) is a deep neural network, such as a Wide ResNet (Zagoruyko & Komodakis, 2016)
Revised: ( x i , x j ) → k ( f θ ( x i ) , f θ ( x j )) , where f θ ( · ) is a deep neural network, such as a Wide ResNet (Zagoruyko & Komodakis, 2016)

Original: up to the last linear layer, parametrized by θ .
Revised: up to the last linear layer, parametrized by θ .

Original: The kernel k ( · , · ) can be any of the standard kernels, such as the RBF  kernel.
Revised: The kernel k ( · , · ) can be any of the standard kernels, such as the RBF or Mat ´ ern kernel.

Original: With the deep network it becomes possible to train the GP on datasets that contain high dimensional points, such as images.
Revised: With the deep network it becomes possible to train the GP on datasets that contain high dimensional points, such as images.

Original: The main shortcoming of DKL is that much of the work is done by the parametric kernel, rather than the component of the model which is Bayesian.

Original: As this parametric model is unconstrained, uncertainty away from the data collapses (Bradshaw et al., 2017), and the model can be arbitrarily conﬁdent while no data has been observed in the region during training.
Revised: As the deep network is unconstrained, uncertainty away from the data collapses (Bradshaw et al., 2017), and the model can be arbitrarily conﬁdent while no data has been observed in that region during training.

Revised: This has been a major drawback to using DKL in practice.

Original: See Section 3 for a detailed discussion of this effect and ways to mitigate it.
Revised: In Section 3 we provide a detailed discussion of this effect and ways to mitigate it.

Original: While DKL is a potentially powerful solution to overcome the problem of inexpressive kernels, it does not address the other scaling problem of Gaussian Processes: large datasets.
Revised: While DKL is a potentially powerful solution to overcome the problem of inexpressive kernels, it does not address the other scaling problem of Gaussian Processes: large datasets.

Original: The poor computational scaling comes from the fact that making predictions with an exact GP requires solving a linear system the size of the training data, namely computing K ( X, X ) − 1 K ( X, x ∗ ) where K ( X, X ) is the kernel matrix evaluated on the training dataset  and K ( X, x ∗ ) the kernel between the training dataset and a test point.
Revised: The poor computational scaling comes from the fact that making predictions with an exact GP requires solving a linear system the size of the training data, namely computing K ( X, X ) − 1 K ( X, x ∗ ) where K ( X, X ) is the kernel matrix evaluated on the training dataset X and K ( X, x ∗ ) the kernel between the training dataset and a test point x ∗ .

Original: This is a signiﬁcant computational problem in its own right (Wang et al., 2019; Wilson et al., 2016a).
Revised: This is a signiﬁcant computational problem in its own right.

Original: A powerful and principled alternative to exact GP inference is inducing point GP approximate inference, where only a small set of inducing points in the input space U are used to represent the entire training set, reducing the linear system to be solved to K ( u, u ) − 1 K ( u, x ∗ ) .
Revised: A powerful and principled alternative to exact GP inference is inducing point GP approximate inference, where only a small set of inducing points in the input space U are used to represent the entire training set, reducing the linear system to be solved to K ( u, u ) − 1 K ( u, x ∗ ) .

Original: The new linear system is m by m , where m is the number of inducing points, rather than N by N .
Revised: The new linear system is m by m , where m is the number of inducing points, rather than N by N .

Original: Finding the optimal set of points, which are not necessarily part of our training set, can be done by treating them as variational parameters and maximizing a lower bound on the marginal log-likelihood (Titsias, 2009).
Revised: Finding the optimal set of points, which are not necessarily part of our training set, can be done by treating them as variational parameters and maximizing a lower bound on the marginal log-likelihood (Titsias, 2009).

Original: We follow the variational inducing point formulation of SVGPC (Hensman et al., 2015) as implemented in GPyTorch (Gardner et al., 2018).
Revised: We follow the variational inducing point formulation of SVGPC (Hensman et al., 2015) as implemented in GPyTorch (Gardner et al., 2018).

Original:  M ETHOD
Revised:  M ETHOD

Original: vDUQ combines Deep Kernel Learning with an inducing point GP, learned using variational inference.
Revised: vDUQ is an instance of Deep Kernel Learning, combining a constrained deep neural network with an inducing point GP.

Revised: The model is learned end-to-end using variational inference by means of gradient

Revised: - Residual NN f θ : x → R d with feature space dimensionality d and parameters θ initialized using

Revised: - Approximate GP with variational parameters φ and number of inducing points m .

Original: In this section we will analyse why the uncertainty of DKL can collapse and how to avoid this.
Revised: Using a random subset of k points of our training data, X init ⊂ X , compute: Initial inducing points K means

Original: We will further explain the steps necessary such that we can train a deep model from end-to-end and avoid the pre-training of previous work (Bradshaw et al., 2017).
Revised: on f ( X init ) with K = m .

Original: The crucial step we make to obtain reliable uncertainty with deep kernels is the introduction of spectral regularization on the deep model.
Revised: Use found centroids as initial inducing point locations in GP. [Table] [Table] [Table]

Revised: [Table]

Revised: descent.

Revised: In this section we discuss how vDUQ overcomes previous problems with collapsing uncertainty in DKL (Bradshaw et al., 2017) and explain how to learn the model with no pre-training, few inducing points, and a standard minibatch size, avoiding some shortcomings of Wilson et al.

Original:  Without it, the deep network, f θ ( · ) , is free to map data points that are far away from the training distribution to a feature representation that’s similar to in distribution data.
Revised: (2016a). Without restrictions, the deep network inside the kernel is free to map input points that are far away from the training distribution to a feature representation that resembles those of data points in the training distribution.

Original: This behavior is also exhibited in standard neural networks (Smith & Gal, 2018) and sometimes referred to as feature collapse .
Revised: This behavior is also exhibited in standard neural networks (Smith & Gal, 2018) and sometimes referred to as feature collapse .

Original: We visualize this in Figure 2, where we map a 2D input space into feature space.
Revised: We visualize this in Figure 2, where we map a 2D input space into feature space.

Original: With the feature space of the unconstrained model it is impossible to recover uncertainty: many points are collapsed on top of each other.
Revised: With the feature space of the unconstrained model it is impossible to recover uncertainty: many points are collapsed on top of each other.

Original: A GP with a stationary kernel reverts back to the prior away from the data (or in our case, the inducing points); in DKL we do not have this guarantee, since the “neural” component of the kernel is free to collapse the distance between any two inputs to an arbitrarily small value.

Original: Recent work has shown that it is possible to reduce feature collapse by regularizing the deep model using a biLipschitz constraint (van Amersfoort et al., 2020), i.e to try to make the model satisfy
Revised: In DUQ (van Amersfoort et al., 2020), the authors show that it is possible to reduce feature collapse by enforcing two constraints on the model: sensitivity and smoothness.

Original: [Equation] The lower Lipschitz constraint is responsible for making the model sensitive to changesin the input: this means the model cannot simply collapse feature representations arbitrarily, but is constrained by the mechanisms in the model, such as strided convolutions.
Revised: Sensitivity implies that when the input changes the feature representation also changes: this means the model cannot simply collapse feature representations arbitrarily.

Original: The upper Lipschitz constraint ensures the model is smooth and small changes in the input cannot cause massive shifts in the output.
Revised: Smoothness means small changes in the input cannot cause massive shifts in the output.

Original: This appears to help optimization, and ensures the feature space accords with the implicit assumptions that for example RBF kernels make about the data.
Revised: This appears to help optimization, and ensures the feature space accords with the implicit assumptions that for example RBF kernels make about the data.

Original: We discuss this theory behind this constraint in more depth in Section 3.1.
Revised: We discuss a connection between these two requirements, sensitivity and smoothness, and bi-Lipschitz functions in Section 6.

Original: There is a number of methods proposed in the literature that attempt to satisfy the bi-Lipschitz constraint and each comes with different trade-offs:
Revised: There is a number of methods proposed in the literature that attempt to satisfy these constraints and each comes with different trade-offs:

Original: • Two-sided Gradient Penalty: van Amersfoort et al. (2020) achieve the regularization using a two-sided gradient penalty,using a soft regularizer that penalizes the squared distance of the gradient from a ﬁxed value at every input point.
Revised: • Two-sided gradient penalty: In DUQ (van Amersfoort et al., 2020), the properties are achieved by regularizing using a two-sided gradient penalty, that penalizes the squared distance of the gradient from a ﬁxed value at every input point.

Original: This approach is easy to implement, but is not guaranteed to work, and in practice both the stability of training and its effectiveness as a regularizer can be fairly sensitive to the weighting of this penalty.
Revised: This approach is easy to implement, but is not guaranteed to work, and in practice both the stability of training and its effectiveness as a regularizer can be fairly sensitive to the weighting of this penalty.

Original: • Direct Spectral Normalization in combination with ResNets: Spectral Normalization (Miyato et al., 2018; Gouk et al., 2018) directly regularizes the upper Lipschitz constraint and it is possible to combine this with an architecture that contains residual connections for the lower constraint, as suggested in Liu et al. (2020), as long as batch normalization (Ioffe & Szegedy, 2015) is also appropriately constrained.
Revised: • Direct spectral normalization and residual connections: spectral normalization (Miyato et al., 2018; Gouk et al., 2018) on the weights leads to smoothness and it is possible to combine this with an architecture that contains residual connections for the sensitivity constraint (Liu et al., 2020).

Original: This method is faster than the gradient penalty, and in practice a more effective way of mitigating feature collapse.
Revised: This method is faster than the gradient penalty, and in practice a more effective way of mitigating feature collapse.

Original: • Reversible Model: A reversible model is constructed by using reversible layers and avoiding any down scaling operations (Jacobsen et al., 2018; Behrmann et al., 2019).
Revised: • Reversible model: A reversible model is constructed by using reversible layers and avoiding any down scaling operations (Jacobsen et al., 2018; Behrmann et al., 2019).

Original: This approach can guarantee that the overall function is bi-Lipschitz, but the resulting model consumes considerably more memory and can be difﬁcult to train.
Revised: This approach can guarantee that the overall function is bi-Lipschitz, but the resulting model consumes considerably more memory and can be difﬁcult to train.

Original: In this work, we use direct spectral normalization, as we ﬁnd it to be more stable than a direct gradient penalty, and  we ﬁnd it difﬁcult to get good performance using fully reversible models, as these preclude the use of layers that reduce dimensionality, like strided convolutions, which we ﬁnd crucial for good performance.
Revised: In this work, we use direct spectral normalization and residual connections, as we ﬁnd it to be more stable than a direct gradient penalty and signiﬁcantly more computationally efﬁcient than reversible models. In Figure 2, we show that a constrained model is unable to collapse points on top of each other in feature space, enabling the GP to correctly quantify uncertainty. Using a stationary kernel, the model then reverts back to the prior away from the training data just like a standard GP.

Original: The regularized feature extractor allows us to ofﬂoad computational complexity from the GP onto the deep model without sacriﬁcing uncertainty.
Revised: The regularized feature extractor allows us to ofﬂoad computational complexity from the GP onto the deep model without sacriﬁcing uncertainty.

Original: An expressive deep model is able to ﬁnd a feature representation which generalizes across all intra class variation, and thus cluster inputs of the same class in feature space.
Revised: An expressive deep model is able to ﬁnd a feature representation which generalizes across all intra class variation, and thus cluster inputs of the same class in feature space.

Original: Wechoose to deﬁne the inducing points in feature space  (L´azaro-Gredilla & Figueiras-Vidal, 2009; Hensman et al., 2017), which takes advantage of this clustering to reduce the number of inducing points required.
Revised: We deﬁne the inducing points in feature space instead of input space (also known as an interdomain approximation) (L´azaro-Gredilla & Figueiras-Vidal, 2009; Hensman et al., 2017), which takes advantage of this clustering to reduce the number of inducing points required.

Original: In practice, we ﬁnd that only very few inducing points, the number of classes in the case of classiﬁcation, are necessary to obtain a well performing GP and we found similarly low numbers (in the 10 to 100 range) necessary for regression, which means that solving the linear system is fast and the GP has minimal overhead compared to a Softmax output.
Revised: In practice, we ﬁnd that only very few inducing points, the number of classes in the case of classiﬁcation, are necessary to obtain a well performing GP and we found similarly low numbers (in the 10 to 100 range) work well for regression, which means that solving the linear system is fast and the GP has minimal overhead compared to a softmax output.

Original: The fact that we can use few inducing points is not true in general for DKL, but requires the extra restrictions we have placed on our feature extractor to avoid the pathological feature collapse.
Revised: The fact that we can use few inducing points is not true in general for DKL (e.g. Wilson et al. (2016a) use several hundred points in input space), but requires the extra restrictions we have placed on our feature extractor to avoid the pathological feature collapse.

Revised: We also ﬁnd we can train with standard minibatch sizes of 128, while previous work used 5,000 on CIFAR-10 to combat gradient variance (Wilson et al., 2016a).

Original: It is important to note that in DKL and also in this paper we are not being Bayesian about our neural network parameters θ , so we only need a single pass through the feature extractor both when training and during inference.
Revised: It is important to note that in DKL and also in this paper we are not being Bayesian about our neural network parameters θ , so we only need a single pass through the feature extractor both when training and during inference.

Original: In addition, we note that batch normalization, a very important component of training deep ResNets, has a non-trivial Lipschitz constant.This is not an issue with gradient penalties, but for methods which use spectral normalization, this is important to take into account.
Revised: When applying spectral normalization to the parameters of a deep model, it is important to note that batch normalization, a crucial component of training deep ResNets, has a non-trivial Lipschitz constant.

Original: In particular, since batch normalization, with scale and shift parameters γ and β ,transforms the input according to:
Revised: In particular, since batch normalization transforms the input, using scale and shift parameters γ and β , according to

Original: [Equation]
Revised: [Equation]

Original: It has a Lipschitz constant of max i | [Equation] √
Revised: it has a Lipschitz constant of max i | γ i √

Original: [Equation] | (Gouk et al., 2018).
Revised: Var( x ) i | (Gouk et al., 2018).

Original: Using the above equation, it is straightforward todeﬁne a version of batch normalizationwhich has a maximum Lipschitz of one by dividing the weight γ of the batch normalization by the (scaled) Lipschitz constant.
Revised: Using the above equation, we can extend spectral normalization to batch normalization by dividing the weight γ of the batch normalization by the (scaled) Lipschitz constant.

Original: In practice, we ﬁnd empirically that typical batch normalization layers in trained ResNets have a relatively high Lipschitz constant, up to around 12, with 95% having a Lipschitz greater than one, see also Figurein the Appendix.
Revised: In practice, we ﬁnd empirically that typical batch normalization layers in trained ResNets have a relatively high Lipschitz constant, up to around 12, with 95% having a Lipschitz greater than one (see also Figure 5 in the Appendix).

Original: This is counter to the claim of Liu et al.
Revised: This is counter to the claim of Liu et al.

Original: (2020) that batch normalization reduces the Lipschitz value of the network.
Revised: (2020) that batch normalization reduces the Lipschitz value of the network.

Original: Unless otherwise noted, in this paper we use the spectral normalized version of batch normalization.

Original: 3. S PECTRAL N ORMALIZATION AND B I -L IPSCHITZ

Original: It is worth thinking about what the condition in Equation 1 means.

Original: As pointed out in Liu et al.
Revised: Following these claims, Liu et al.

Original: (2020), there is an established connection between bi-Lipschitz mappings and metric spaces - if we have two metric spaces X and Y , with a continuous function f : X (cid:55)→ Y between them which is bijective, and f is bi-Lipschitz, then metric properties of X like boundedness and completeness have to be preserved by Y , and distances in X can be changed only by a bounded factor in Y .

Original: In this sense, a bi-Lipschitz map between X and Y can be considered to be “approximately” distance preserving, which is useful for measuring uncertainty in feature space.

Original: A function f whichis parameterized as a residual network and regularized appropriately can be constructed to be bi-Lipschitz if it is a residual block between two spaces of the same dimensionality (Liu et al., 2020; Behrmann et al., 2019). However, it is straightforward to demonstrate the following observation (which we do not claim to be novel) Proposition 1. Let f be a function from R n to R m , with m < n .
Revised: (2020) only constrain the convolutional layers, which as we demonstrate empirically results in models which are less sensitive to changes in the input, violating the constraint introduced in DUQ.

Original: Then f is not bi-Lipschitz.

Original: We provide a proof in Appendix B; it sufﬁces to show that a bi-Lipschitz function is necessarily a homeomorphism, and so cannot exist between spaces of different dimensionality. Since layers which reduce the dimensionality, like pooling layers andstrided convolutions, are common and often necessary for good performance on image data, this calls into question how informative the metric space perspective is on explaining the usefulness of bi-Lipschitz regularization for the model as a whole. Explanations of the effectiveness of these regularizers based on the assumption that the bi-Lipschitz condition holds are insufﬁcient if the network uses dimensionality reduction, which the feature extraction architectures used in both Liu et al. (2020) and van Amersfoort et al.
Revised: Unless otherwise noted, we apply spectral normalization to both the convolutional and the batch normalization layers.

Original: (2020) do. These arguments would hold for invertible models. Some progress towards a well performing bijective deep model was made in Behrmann et al. (2019), however the iResNet is signiﬁcantly slower and computationally expensive than the WRN, and we were unable to achieve the performance we report here without using dimension-reducing layers.
Revised: For convolutional layers with 1 × 1 ﬁlters we use exact normalization, while for large ﬁlters we use an approximation, implemented originally by

Original: It is also important to note that the proof of bi-Lipschitz-ness in Liu et al. (2020) (and corresponding proof of invertibility in Behrmann et al.(2019) only holds if the spectral norm of the residual connection is constrained to be less than one. In practice, both in our work and that of Liu et al.
Revised: Behrmann et al.

Original: , we ﬁnd that Lipschitz constraints looser than this work better in practice, raising further questions of how much this theory is explaining the observed usefulness of this regularization scheme.

Original: A complete, rigorous theory of why the spectral normalization as used in this and previous work is a useful regularization scheme is not remains an open question.

Original: However, in practice we ﬁnd that controlling the feature space distortion “enough” has a signiﬁcant effect in enabling the GP to maintain its prior reversion effect.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Currently, the simplest and most effective way to obtain uncertainty in deep learning classiﬁcation is using ensembling (Lakshminarayanan et al., 2017).
Revised: Currently, the simplest and most effective way to obtain uncertainty in deep learning classiﬁcation is using ensembling (Lakshminarayanan et al., 2017).

Original: Despite its simplicity it has shown to be

Original: [Figure]

Original: remarkably effective (Ovadia et al., 2019)

Original: , outperforming alternatives such as MC Dropout (Gal & Ghahramani, 2016) and mean-ﬁeld variational inference Bayesian Neural Networks(Farquhar et al., 2019) at the expense of having to train and evaluate multiple models.
Revised: Despite its simplicity it has shown to be remarkably effective (Ovadia et al., 2019), outperforming alternatives such as MC Dropout (Gal & Ghahramani, 2016) and mean-ﬁeld variational inference Bayesian Neural Networks (Farquhar et al., 2019) at the expense of having to train and evaluate multiple models.

Original: An important issue with
Revised: An important issue with

Original: Deep Ensembles is enforcing diversity; while on large and complicated datasets it is sufﬁcient to use a different initialization and data order, this is not sufﬁcient on easier or smaller datasets, as we highlight in Figure 1. 
Revised: Deep Ensembles is enforcing diversity; while on large and complicated datasets it is sufﬁcient to use a different initialization and data order, this is not sufﬁcient on easier or smaller datasets, as we highlight in Figure 1. [Figure]

Original: DUQ (van Amersfoort et al., 2020) demonstrated empirically  that it is possible to quantify uncertainty in deep learning using only a single deterministic model.
Revised: van Amersfoort et al. (2020) demonstrate empirically with DUQ that it is possible to quantify uncertainty in deep learning using only a single deterministic model.

Original: SNGP (Liu et al., 2020) takes the concept ofDUQ, a distance aware prediction function and a bi-Lipschitz constraint, and achieves them by using a deep neural network with a random fourier features (RFF) approximate GP and  spectral normalization on the convolutional weights. They do not account for batch normalization in their normalization scheme.
Revised: SNGP (Liu et al., 2020) extends on DUQ’s ideas of a distance aware prediction function and spectral regularization, and implements them by using a deep neural network with a random fourier features (RFF) approximate GP and direct spectral normalization on the convolutional weights (but not on the weights of batch normalization).

Original: While they obtain impressive results, it is important to note that the RFF approximation makes the GP parametric, meaning there are now a ﬁxed number of basis functions used in the prediction which can lead to unreliable uncertainty.
Revised: While they obtain impressive results, it is important to note that the RFF approximation makes the GP parametric, meaning there are now a ﬁxed number of basis functions used in the prediction which can lead to unreliable uncertainty.

Original: Using variational Bayes in the form of SVGPC conserves the non-parametric property and uses inﬁnitely many basis functions; it is an approximation to the true GP posterior, whereas a random Fourier expansion is a different model which converges in expectation to the true model as the number of random fourier features tends to inﬁnity.
Revised: In contrast, our use of variational Bayes in the form of SVGPC conserves the non-parametric property and uses inﬁnitely many basis functions; it is an approximation to the true GP posterior, whereas Liu et al. (2020)’s random Fourier expansion leads to a different model which converges in expectation to the true model as the number of random Fourier features tends to inﬁnity.

Original: vDUQ only needs a handful of inducing points and increasing this number is guaranteed to improve the model monotonically as the pseudo inputs are part of the  approximation  and not parameters of the model.
Revised: This is discussed in Section 6 of L ´ azaro-Gredilla et al. (2010) which comment that the RFF approximation is at risk of overﬁtting, while the variational inducing point approximation is safeguarded from it as they are parameters of the variational approximation (Titsias, 2009). Furthermore, the RFF approximation is only tractable for the RBF kernel and signiﬁcantly more expensive for other kernels, restricting the ﬂexibility of the model.

Original: Finally, although in principle the method of Liu et al.
Revised: Finally, although in principle the method of Liu et al.

Original: (2020) could be adapted to regression in a straightforward way, they do not investigate this, while our experiments demonstrate the applicability of this kind of model both on standard classiﬁcation benchmarks and uncertainty sensitive regression tasks, such as Causal Inference.
Revised: (2020) could be adapted to regression in a straightforward way, they do not investigate this. Our experiments demonstrate the applicability of this kind of model both on standard classiﬁcation benchmarks and uncertainty sensitive regression tasks, such as causal inference.

Original:  E XPERIMENTS 5. R EGRESSION AND IN - BETWEEN U NCERTAINTY
Revised:  E XPERIMENTS 5. R EGRESSION AND IN - BETWEEN U NCERTAINTY

Original: In Figure 1, we show results on a 1D dataset created from sinusoids.
Revised: In Figure 1, we show results on a 1D dataset created from sinusoids.

Original: For vDUQ, we use spectral normalization, a Mat´ern kernel with ν = 12 and 20 inducing points.
Revised: For vDUQ, we use spectral normalization, a Mat´ern kernel with ν = 12 and 20 inducing points.

Original: This kernel is known to lead to less smooth learned functions, which leads to quickly increasing the uncertainty when there is no data.
Revised: This kernel is known to lead to less smooth learned functions, which leads to quickly increasing the uncertainty when there is no data.

Original: Full experimental details are provided in Appendix A
Revised: Full experimental details are provided in Appendix A

Original: Deep Ensemble extrapolates incorrectly as x becomes more negative, at x = − 15 the prediction is far away from what is seen in the training data, however the model still has low uncertainty.
Revised: Deep Ensemble extrapolates incorrectly as x becomes more negative, at x = − 15 the prediction is far away from what is seen in the training data, however the model still has low uncertainty.

Original: Meanwhile vDUQ reverts to the prior directly at the edge of the training data domain, which is the ideal behavior in this regression setting.
Revised: Meanwhile vDUQ reverts to the prior directly at the edge of the training data domain, which is the ideal behavior in this regression setting.

Original: In Deep Ensembles (Lakshminarayanan et al., 2017), it was suggested to let the model predict both the mean and the standard deviation.
Revised: In Deep Ensembles (Lakshminarayanan et al., 2017), it was suggested to let the model predict both the mean and the standard deviation.

Original: This standard deviation only measures the aleatoric uncertainty (see also Chua et al.
Revised: This standard deviation only measures the aleatoric uncertainty (see also Chua et al.

Original: (2018) for a discussion), which is the uncertainty arising from noise inherent in the data.
Revised: (2018) for a discussion), which is the uncertainty arising from noise inherent in the data.

Original: In this dataset there is only minimal noise, and therefore the only way to assess the uncertainty is to compute the variance of the mean predictions.
Revised: In this dataset there is only minimal noise, and therefore the only way to assess the uncertainty is to compute the variance of the mean predictions.

Original: [Table]

Original: In contrast, vDUQ’s posterior variance is directly interpretable as the predictive uncertainty (also known as the total uncertainty).
Revised: In contrast, vDUQ’s posterior variance is directly interpretable as the predictive uncertainty (also known as the total uncertainty).

Original: In the Appendix, Figure 4, we visualize sampling from the posterior, which highlights the non-smoothness of the kernel choice. 
Revised: In the Appendix, Figure 4, we visualize sampling from the posterior, which highlights the non-smoothness of the kernel choice. [Table]

Original: 5. C LASSIFICATION ON T WO M OONS
Revised: 5. C LASSIFICATION ON T WO M OONS

Original: We show results on Two Moons dataset (Pedregosa et al., 2011) for three different models: a standard softmax model, vDUQ  and a variation where the spectral normalized ResNet is replaced by a fully connected model, similar to Bradshaw et al.
Revised: We show results on Two Moons dataset (Pedregosa et al., 2011) for three different models: a standard softmax model, vDUQ [Table] and a variation where the spectral normalized ResNet is replaced by a fully connected model (similar to Bradshaw et al.

Original: For experimental details see Appendix A.
Revised: For experimental details see Appendix A.

Original: On this simple dataset, a multi layer neural network easily achieves 100% accuracy with a single, non-linear, decision boundary going between the two moons.
Revised: On this simple dataset [Table] a multi layer neural network easily achieves 100% accuracy with a single, non-linear, decision boundary going between the two moons.

Original: However these models exhibit complete certainty everywhere in the input domain as shown in Figure 3a, with yellow the certain regions, and blue uncertain.
Revised: However these models exhibit complete certainty everywhere in the input domain as shown in Figure 3a, with yellow the certain regions [Table] and blue uncertain.

Original: The uncertainty is computed using the entropy of the class prediction: we model the problem as a two class classiﬁcation problem.
Revised: The uncertainty is computed using the entropy of the class prediction: we model the problem as a two class classiﬁcation problem.

Original: vDUQ (in Figure 3c) on the other hand quantiﬁes uncertainty exactly as one would expect for the two moons dataset: certain on the training data, uncertain away from it and in between the two half moons.
Revised: vDUQ (in Figure 3c) on the other hand quantiﬁes uncertainty exactly as one would expect for the two moons dataset: certain on the training data, uncertain away from it and in between the two half moons.

Original: Figure 3b highlights the importance of our contribution.
Revised: Figure 3b highlights the importance of our contribution.

Original: When using DKL with a deep network without the right constraints, the model generalizes beyond the training distribution and is certain even away from the training data.
Revised: When using DKL with a deep network without the right constraints, the model generalizes beyond the training distribution and is certain even away from the training data.

Original: 5. CIFAR-10 AND SVHN
Revised: 5. CIFAR-10 AND SVHN

Original: In this section we look at training a large model, the Wide Residual Network, on CIFAR-
Revised: In this section we look at training a large model, the Wide Residual Network, on CIFAR-

Original: (Krizhevsky et al., 2009).
Revised: (Krizhevsky et al., 2009).

Original: In vDUQ, we train the large ResNet end-to-end with the GP, this is in contrast with prior work that used a pre-trained model (Bradshaw et al., 2017), which is a signiﬁcant limitation.
Revised: For vDUQ, we train the large ResNet end-to-end with the GP, this is in contrast with prior work that used a pre-trained model (Bradshaw et al., 2017), which limits the ability to apply the method to new domains.

Original: We follow the experimental setup of Zagoruyko & Komodakis (2016), and use a 28 layer model with BasicBlocks and dropout.
Revised: We follow the experimental setup of Zagoruyko & Komodakis (2016), and use a 28 layer model with BasicBlocks and dropout.

Original: Interestingly, we can follow hyper parameter suggestions (such as dropout rate, learning rate, and optimizer) directly when training vDUQ, and no special care is necessary for the variational parameters.
Revised: Interestingly, we can follow hyper parameter suggestions (such as dropout rate, learning rate, and optimizer) directly when training vDUQ, and no special care is necessary for the variational parameters.

Original: We remove the ﬁnal linear layer of the model and the resulting 640 dimensional feature vector is directly used in the GP.
Revised: We remove the ﬁnal linear layer of the model and the resulting 640 dimensional feature vector is directly used in the GP.

Original: We train using only 10 inducing points , which are shared among the independent output GPs (although q ( u ) is different) and use Lipschitz factor 3.
Revised: We train using only 10 inducing points , which are shared among the independent output GPs (although q ( u ) is different) and use Lipschitz factor 3.

Original: This means the model is fast, and going through one epoch is just 3% slower than using a softmax output.
Revised: This means the model is fast, and going through one epoch is just 3% slower than using a softmax output.

Original: Further experimental details are discussed in Appendix A.
Revised: Further experimental details are discussed in Appendix A.

Revised: We provide an ablation study of the number of inducing points in

Revised: Appendix C.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: Interestingly, using a softmax model it is possible to get good results on this benchmark.

Original: vDUQ without any spectral normalization matches the accuracy of the WRN, but the uncertainty is only slightly better than a softmax model.
Revised: vDUQ without any spectral normalization matches the accuracy of the WRN, but the uncertainty is only slightly better than a softmax model.

Original: With spectral normalization the accuracy drops slightly, as expected, but the uncertainty improves signiﬁcantly.
Revised: With spectral normalization the accuracy drops slightly, as expected, but the uncertainty improves signiﬁcantly.

Revised: Note that SVDKL (Wilson et al., 2016a), our most direct baseline, obtains just 91% accuracy on CIFARusing a ResNet-20 (for which the softmax accuracy is 94%, see also van Amersfoort et al.

Revised: Meanwhile, convolutional GPs (Van der Wilk et al., 2017) obtain 64.6% accuracy while using 1, inducing points.

Revised: We perform an ablation where we train vDUQ with spectral normalization only on the convolutions, similar to (Liu et al., 2020), and we obtain an AUROC of 0 .

Original: Using spectral normalization only on the convolutions does not reduce the accuracy, but also does not improve the uncertainty.
Revised: 93 ± 0 . 003 and accuracy matching the model without spectral normalization. This provides further evidence that the batch normalization layers undo the effect, and need to be properly normalized. [Table]

Original: 5. C AUSAL I NFERENCE
Revised: C AUSAL I NFERENCE

Original: In this section, we consider the safety critical task of inferring the individual-level causal effects of abinary treatment t for an individual described by covariates x via the conditional average treatment
Revised: Personalized healthcare is an exciting application of machine learning, where the efﬁcacy of a treatment is predicted based on characteristics of the individual using a model trained on previously treated patients.

Original: [Table] effect: CATE :=
Revised: An individual’s response to treatment can only be known if they are a member of a group represented in the data and if there is prescription diversity in the group: treatment and no treatment.

Original: E [ Y t =1 ( x ) − Y t =0 ( x )] (Abrevaya et al., 2015).
Revised: Jesson et al.

Original: As it is only possible to observe one of the two potential outcomes, Y t =0 ( x ) or Y t =1 ( x ) , identiﬁcation of CATE hinges upon the assumption that we have conditioned on all features x of each patient which might have a confounding effect on treatment outcome. This assumption posits that the observed data is unconfounded ( { Y t =0 , Y t =1 } ⊥⊥ t | x ) and that there is sufﬁcient overlap (0 < p ( t = 1 | x ) < 1) .
Revised: (2020) show that measures of uncertainty can identify when personalized causal-effect inference fails due to such factors. Uncertainty can then deﬁne policies for deferring treatment recommendations to an expert when there is insufﬁcient knowledge about a person.

Original: To get some intuition, consider a case where a helpful treatment t is more likely to be given to the severely ill. As a result, the survival rate of treated patients will be lower, creating an erroneous conclusion that the treatment has a negative effect if  we don’t condition on the current symptoms x of each patient. Conditioning on  x removes this confounding effect.
Revised: The uncertainty estimates must be correct, or else individuals would receive treatment recommendations even if their response to treatment is not known, which can result in undue stress, ﬁnancial burdens, or worse. In this section we evaluate vDUQ on the task of inferring the personalized causal-effects of a binary treatment t on an individual x via the conditional average treatment effect: CATE( x ) :

Original: The unconfoundedness assumption says that we observe all possible factors that may affect the distribution of treatment.
Revised: =

Original: The overlap assumption means we need to have sufﬁcient samples from each subgroup - e.g our data needs to contain at least some people who were assigned treatment for every set of attributes x for us to determine the conditional effect. Jesson et al.
Revised: E [ Y t =1 ( x ) − Y t =0 ( x )] (Abrevaya et al., 2015).

Original: (2020) show that violation of the overlap assumption can be framed as an out-ofdistribution problem and that uncertainty aware models yield appropriately low conﬁdence predictions for such out of distribution test examples. Further, they show that  uncertainty estimatescan be used to deﬁne treatment recommendation policies that defer to experts when it is suspected that either overlap is violated, or a new example represents a covariate shift. Therefore, causal inference can be seen as an interesting application of regression uncertainty because uncertainty can identify when a model has insufﬁcient understanding to make a recommendation for a novel individual, as well as express conﬁdence in the causal estimand itself, which would be invalid if overlap is violated.
Revised: Intuitively, an individual would be recommended treatment if CATE( x ) > 0 . We show that vDUQ yields uncertainty estimates that lead to safer and more data-efﬁcient policies for withholding treatment recommendations than alternative methods signiﬁcantly improving on previous state-of-the-art results .

Original: We use the semi-synthetic IHDP (Hill, 2011), IHDP Cov.
Revised: We use the semi-synthetic IHDP (Hill, 2011), IHDP Cov.

Original: and CEMNIST(Jesson et al., 2020) datasets to assess the uncertainty estimates of vDUQ on this causal inference task following the experimental setup in (Jesson et al., 2020).
Revised: and CEMNIST datasets to assess the uncertainty estimates of vDUQ on this causal inference task following the experimental setup in (Jesson et al., 2020).

Original: IHDP and IHDP Cov.
Revised: IHDP and IHDP Cov.

Original: are regression datasets with only ∼ 750 data points, while CEMNIST is a binary classiﬁcation dataset with 10k points.
Revised: are regression datasets with only ∼ 750 data points, while CEMNIST is a binary classiﬁcation dataset with 10k points.

Original: For each dataset, predictions are withheld for a speciﬁed proportion ( 50% or 10% ) of the test set based on either a random sample ( random ), or  on thehighest uncertainty in  CATE estimate ( uncertainty ).
Revised: Treatment-effect recommendations are deferred to an expert either at random , or based on the uncertainty in the CATE estimate. Recommendations are deferred for 10% of the cases in the IHDP dataset, and 50% of the IHDP Cov. and CEMNIST datasets.

Original: We report the root expected Precision in Estimation of Heterogeneous Effect (Hill, 2011) ( √ (cid:15) PEHE ) to assess the error on the remaining examples (lower is better).
Revised: We report the root expected Precision in Estimation of Heterogeneous Effect (Hill, 2011) ( √ (cid:15) PEHE ) to assess the error on the remaining recommendations (lower is better).

Original: Table 2 summarizes our results and shows that vDUQ has improved performance and uncertainty estimates better suited to rejection policies than other uncertainty aware methods.
Revised: Table 2 summarizes our results and shows that vDUQ has improved performance and uncertainty estimates better suited to rejection policies than other uncertainty aware methods.

Original: In particular, we improve on DKLITE (Zhang et al., 2020) a methodspeciﬁcally designed  for CATE inference.
Revised: In particular, we improve on DKLITE (Zhang et al., 2020), the baseline related to our method: an alternative deep kernel learning method designed speciﬁcally for CATE inference.

Original:  C ONCLUSION
Revised:  L IMITATIONS

Revised: Even though demonstrated to work well empirically above, a theoretical justiﬁcation of the spectral normalization in combination with residual connection is still lacking.

Revised: Liu et al.

Revised: (2020) propose an explanation based on the fact that spectral normalization, in combination with residual architectures, induces a bi-Lipschitz constraint, which is deﬁned as:

Revised: [Equation]

Revised: From that constraint it follows that if we have two metric spaces X and Y , with a continuous function f : X (cid:55)→ Y between them which is bijective, and f is bi-Lipschitz, then metric properties of X like boundedness and completeness have to be preserved by Y , and distances in X can be changed only by a bounded factor in Y .

Revised: In this sense, a bi-Lipschitz map between X and Y can be considered to be “approximately” distance preserving.

Revised: Liu et al.

Revised: (2020) argue that this means that applying this penalty to a Wide ResNet causes it to be a distance preserving mapping.

Revised: However, the proof they provide is only valid for spectral regularisation with coefﬁcients less than 1, whereas empirically we ﬁnd that exceeding 1 has similar effectiveness in terms of uncertainty and is easier to optimize.

Revised: This suggests that the proof does not capture the underlying mechanism of the spectral normalization.

Revised: In addition, this argument is not applicable to models that do any form of downsampling as it is straightforward to demonstrate the following observation (which we do not claim to be novel):

Original:  We have presented vDUQ, a practical probabilistic method which allows for using deep architectures and we established its efﬁcacy in 2D, on a large image dataset and in causal inference for personalized medicine, obtaining or matching state of the art results in each.
Revised: Proposition 1. Let f be a function from R n to R m , with m < n . Then f is not bi-Lipschitz. We provide a proof in Appendix B; it sufﬁces to show that a bi-Lipschitz function is necessarily a homeomorphism, and so cannot exist between spaces of different dimensionality.

Original: There are interesting future research opportunities exploring the theoretical justiﬁcation for the regularization scheme, going beyond what is discussed in Section 3.1 and Liu et al. 
Revised: We thus consider explanations of the effectiveness of spectral normalization and residual connections based on the assumption that the bi-Lipschitz condition holds insufﬁcient if the network uses dimensionality reduction, which most feature extraction architectures do. Some progress towards a well performing bijective deep model was made in Behrmann et al. (2019), however the iResNet is signiﬁcantly slower and more computationally expensive than the WRN, and we were unable to achieve the performance we report here without using dimension-reducing layers. Despite these open theoretical questions, we have so far not found a practical example where the uncertainty quality was poor, and our empirical results call for further theoretical study.

Original:  and incorporating research from the existing kernel literature.
Revised: We present vDUQ, a practical probabilistic method which allows for using deep architectures and we established its efﬁcacy in 2D, on a large image dataset and in causal inference for personalized medicine, obtaining or matching state of the art results in each.

Original: Good uncertainty and fast inference in both classiﬁcation and regression make vDUQ a compelling plug in option for many applications, such as exploration in reinforcement learning, active learning and many real world tasks.
Revised: Good uncertainty and fast inference in both classiﬁcation and regression make vDUQ a compelling plug-in option for many applications, such as exploration in reinforcement learning, active learning, and many real world tasks.

Revised: Exciting directions for future research include a rigorous theory of why spectral normalization works despite downsampling, or alternatively a computationally efﬁcient way to enforce a bi-Lipschitz constraint.

Original: A E XPERIMENTAL D ETAILS A. 2D EXPERIMENTS
Revised: A E XPERIMENTAL D ETAILS A. 2D EXPERIMENTS

Original: We perform our 2D experiments using a simple feed forward ResNet.
Revised: We perform our 2D experiments using a simple feed forward ResNet.

Original: The ﬁrst linear layer maps from the input to the feature representation and does not have an activation function.
Revised: The ﬁrst linear layer maps from the input to the feature representation and does not have an activation function.

Original: From there on, the model is a ResNet, x (cid:48)
Revised: From there on, the model is a ResNet, x (cid:48)

Original: =
Revised: =

Original: x
Revised: x

Original: + f ( x ) , with f ( · ) a combination of a linear mapping and a relu activation function.
Revised: + f ( x ) , with f ( · ) a combination of a linear mapping and a relu activation function.

Original: The linear mapping has optional Spectral Normalization for which we use the implementation of Behrmann et al.
Revised: The linear mapping has optional Spectral Normalization for which we use the implementation of Behrmann et al.

Original: We use the SGD optimizer for regression and Adam for the two moons with learning rate 0.01, and we use 4 layers with 128 features.
Revised: We use the SGD optimizer for regression and Adam for the two moons with learning rate 0.01, and we use 4 layers with 128 features.

Original: For the toy regression, we use a max Lipschitz constant 0.95, one power iteration and 20 inducing points.
Revised: For the toy regression, we use a max Lipschitz constant 0.95, one power iteration and 20 inducing points.

Original: For the two moons, we set the noise level to 0.1 and use ﬁve power iterations, a Lipschitz constant of 0.65 and four inducing points.
Revised: For the two moons, we set the noise level to 0.1 and use ﬁve power iterations, a Lipschitz constant of 0.65 and four inducing points.

Original: For the Deep Ensemble in Figure 1 we train 10 separate models, using a different initialization and data order for each and train to minimize the squared error.
Revised: For the Deep Ensemble in Figure 1 we train 10 separate models, using a different initialization and data order for each and train to minimize the squared error.

Original: A. W IDE R ES N ET E XPERIMENTS
Revised: A. W IDE R ES N ET E XPERIMENTS

Original: For the WRN, we follow the experimental setup and implementation of Zagoruyko & Komodakis (2016).
Revised: For the WRN, we follow the experimental setup and implementation of Zagoruyko & Komodakis (2016).

Original: This means that for CIFAR-10, we use depth 28 with widen factor 10 and BasicBlocks with dropout.
Revised: This means that for CIFAR-10, we use depth 28 with widen factor 10 and BasicBlocks with dropout.

Original: We train for the prescribed 200 epochs with batch size 128, starting with learning rate 0.1 and dropping with a factor of 0.2 at epoch 60, 120 and 160.
Revised: We train for the prescribed 200 epochs with batch size 128, starting with learning rate 0.1 and dropping with a factor of 0.2 at epoch 60, 120 and 160.

Original: We use the full training set and take the model at epoch 200 (no early stopping).
Revised: We use the full training set and take the model at epoch 200 (no early stopping).

Original: For Spectral Normalization, we again use the implementation of Behrmann et al.
Revised: For Spectral Normalization, we again use the implementation of Behrmann et al.

Original: (2019), in particular the convolutional version and also constrain batch normalization as described in (Gouk et al., 2018).
Revised: (2019), in particular the convolutional version and also constrain batch normalization as described in (Gouk et al., 2018).

Original: We use 1 power iteration and use the lowest Lipschitz constant that still allows for good accuracy, which we found to be around 3 in practice.
Revised: We use 1 power iteration and use the lowest Lipschitz constant that still allows for good accuracy, which we found to be around 3 in practice.

Original: We increase the momentum of batch normalization to 0.99 (from 0.9 default) to reduce the variance of the running average estimator of the empirical feature variance, which can be a source of instability (Gouk et al., 2018).
Revised: We increase the momentum of batch normalization to 0.99 (from 0.9 default) to reduce the variance of the running average estimator of the empirical feature variance, which can be a source of instability (Gouk et al., 2018).

Original: A. C AUSAL E XPERIMENTS
Revised: A. C AUSAL E XPERIMENTS

Original: Following Shalit et al.
Revised: Following Shalit et al.

Original: , we use 63/27/10 train/validation/test splits and report the PEHE evaluated on the test set over 1000, and 20 trials for the IHDP and CEMNIST datasets, respectively.
Revised: , we use 63/27/10 train/validation/test splits and report the PEHE evaluated on the test set over 1000, and 20 trials for the IHDP and CEMNIST datasets, respectively.

Original: For each trial, we train for a maximum of 750 epochs and evaluate the model with the lowest ELBO evaluated over the validation set.
Revised: For each trial, we train for a maximum of 750 epochs and evaluate the model with the lowest ELBO evaluated over the validation set.

Original: We employ Adam optimization with a learning rate of 0.001 and batch size of 100.
Revised: We employ Adam optimization with a learning rate of 0.001 and batch size of 100.

Original: The feature extractor uses a feed forward ResNet architecture with 3, 200 unit hidden layers, and
Revised: The feature extractor uses a feed forward ResNet architecture with 3, 200 unit hidden layers, and

Original: ELU activations.
Revised: ELU activations.

Original: Dropout is applied after each activation at a rate of 0.1.
Revised: Dropout is applied after each activation at a rate of 0.1.

Original: The feature extractor takes the covariates x and treatment t as input.
Revised: The feature extractor takes the individual x and treatment t as input.

Original: For the CEMNIST experiment, a depth 3 CNN
Revised: For the CEMNIST experiment, a depth 3 CNN

Original: Resnet is used to extract features from the image, which is then passed as an input to the above architecture.
Revised: Resnet is used to extract features from the image, which is then passed as an input to the above architecture.

Original: Spectral normalization is used on all layers of the feature extractor.
Revised: Spectral normalization is used on all layers of the feature extractor.

Original: We use a Mat´ern kernel with ν = 12 , 200 inducing points, and a smoothed box prior on the lengthscales with range (exp( − 1) , exp(1)) .
Revised: We use a Mat´ern kernel with ν = 12 , 200 inducing points, and a smoothed box prior on the lengthscales with range (exp( − 1) , exp(1)) .

Original: For the DKLITE experiments we use the open source code from the authors available at https://bitbucket.org/mvdschaar/mlforhealthlabpub/src/master/ alg/dklite/ .
Revised: For the DKLITE experiments we use the open source code from the authors available at https://bitbucket.org/mvdschaar/mlforhealthlabpub/src/master/ alg/dklite/ .

Original: We write a custom loop over the IHDP dataset to follow the above protocol.
Revised: We write a custom loop over the IHDP dataset to follow the above protocol.

Original: For DKLITE on CEMNIST, in initial experimentation we found that the method did not adapt well to images, so we omit the comparison from the table.
Revised: For DKLITE on CEMNIST, in initial experimentation we found that the method did not adapt well to images, so we omit the comparison from the table.

Original: We make this adaptation available at anonymized-for-review .
Revised: We make this adaptation available at anonymized-for-review .

Original: A. I NDUCING P OINT GP
Revised: A. I NDUCING P OINT GP

Original: We initialize the inducing points by doing k-means on the feature representation of 1,000 points.
Revised: We initialize the inducing points by doing k-means on the feature representation of 1,000 points.

Original: We compute the initial length scale by taking the pairwise euclidean distance between feature representation of the 1,000 points.
Revised: We compute the initial length scale by taking the pairwise euclidean distance between feature representation of the 1,000 points.

Original: We whiten the inducing points before training, as suggested in Matthews
Revised: We whiten the inducing points before training, as suggested in Matthews

Original: The kernel parameters, such as the length scale and output scale, are different per GP.
Revised: The kernel parameters, such as the length scale and output scale, are different per GP.

Original: We implement it using GPyTorch (Gardner et al., 2018) and use their default values if not otherwise speciﬁed.
Revised: We implement it using GPyTorch (Gardner et al., 2018) and use their default values if not otherwise speciﬁed.

Original: B P ROOF OF PROPOSITION
Revised: B P ROOF OF PROPOSITION

Original: This proof makes no claim to novelty, but is provided for completeness.
Revised: This proof makes no claim to novelty, but is provided for completeness.

Original: Proof of proposition 1.
Revised: Proof of proposition 1.

Original: We can show that this follows from the fact that the metric spaces R m and R n are not homeomorphic (topologically equivalent) to one another (Brouwer, 1911).
Revised: We can show that this follows from the fact that the metric spaces R m and R n are not homeomorphic (topologically equivalent) to one another (Brouwer, 1911).

Original: While this statement is extremely intuitive, proving it is surprisingly technical, so we will take it as given.
Revised: While this statement is extremely intuitive, proving it is surprisingly technical, so we will take it as given.

Original: First, we prove the following lemma.
Revised: First, we prove the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let f : X
Revised: Let f : X

Original: (cid:55)→ Y be a bi-Lipschitz and onto function, so 1 L || x 1 − x 2 || X ≤
Revised: (cid:55)→ Y be a bi-Lipschitz and onto function, so 1 L || x 1 − x 2 || X ≤

Original: || f ( x 1 ) − f ( x 2 )
Revised: || f ( x 1 ) − f ( x 2 )

Original: || Y ≤ L || x 1 − x 2 || X , and Y is the image of X under f .
Revised: || Y ≤ L || x 1 − x 2 || X , and Y is the image of X under f .

Original: Then f is a homeomorphism between
Revised: Then f is a homeomorphism between

Original: X and Y , and so X and Y are homeomorphic.
Revised: X and Y , and so X and Y are homeomorphic.

Original: Proof.
Revised: Proof.

Original: Recall that a function f is a homeomorphism if f is bijective, f is continuous, and f − 1 is also continuous.
Revised: Recall that a function f is a homeomorphism if f is bijective, f is continuous, and f − 1 is also continuous.

Original: We will address these in turn.
Revised: We will address these in turn.

Original: To see that f is bijective, note that f is injective iff.
Revised: To see that f is bijective, note that f is injective iff.

Original: ∀ x 1 , x 2 ∈ X , we have x 1 (cid:54) =
Revised: ∀ x 1 , x 2 ∈ X , we have x 1 (cid:54) =

Original: x 2 = ⇒ f ( x 1 ) (cid:54) =
Revised: x 2 = ⇒ f ( x 1 ) (cid:54) =

Original: f
Revised: f

Original: ( x 2 ) .
Revised: ( x 2 ) .

Original: But this follows directly from the lower Lipschitz property of f , since if x 1 (cid:54) =
Revised: But this follows directly from the lower Lipschitz property of f , since if x 1 (cid:54) =

Original: x 2 , then || x 1 − x 2 || X > 0 , so || f ( x 1 ) − f ( x 2 ) || Y > 0 , from which it follows that f ( x 1 ) (cid:54) =
Revised: x 2 , then || x 1 − x 2 || X > 0 , so || f ( x 1 ) − f ( x 2 ) || Y > 0 , from which it follows that f ( x 1 ) (cid:54) =

Original: f
Revised: f

Original: ( x 2 ) .
Revised: ( x 2 ) .

Original: Since f is injective (one-to-one) and onto, it is a bijection.
Revised: Since f is injective (one-to-one) and onto, it is a bijection.

Original: Since any function which is Lipschitz continuous is also continuous, the fact that f is continuous is given.
Revised: Since any function which is Lipschitz continuous is also continuous, the fact that f is continuous is given.

Original: Since f is bijective, the inverse function f − 1 exists, and we need to show that it is continuous.
Revised: Since f is bijective, the inverse function f − 1 exists, and we need to show that it is continuous.

Original: We have, from the bi-Lipschitzness of f , that 1 L || f − 1 ( f ( x 1 ))
Revised: We have, from the bi-Lipschitzness of f , that 1 L || f − 1 ( f ( x 1 ))

Original: − f − 1 ( f ( x 1 ))
Revised: − f − 1 ( f ( x 1 ))

Original: ||
Revised: ||

Original: X ≤ || f
Revised: X ≤ || f

Original: ( x 1 ) − f ( x 2 )
Revised: ( x 1 ) − f ( x 2 )

Original: || Y , which implies that the inverse function is also Lipschitz, and hence also continuous.
Revised: || Y , which implies that the inverse function is also Lipschitz, and hence also continuous.

Original: So f is a homeomorphism, and we are done.
Revised: So f is a homeomorphism, and we are done.

Original: We have therefore established that if a bi-Lipschitz (and onto) function exists between two spaces
Revised: We have therefore established that if a bi-Lipschitz (and onto) function exists between two spaces

Original: X and Y , then X and Y are homeomorphic.
Revised: X and Y , then X and Y are homeomorphic.

Original: This provides a proof of our claim by contradiction; if a function f : R n (cid:55)→ R m existed and was bi-Lipschitz, then we would have shown that R n and R m were homeomorphic, since any bi-Lipschitz function is a homeomorphism.
Revised: This provides a proof of our claim by contradiction; if a function f : R n (cid:55)→ R m existed and was bi-Lipschitz, then we would have shown that R n and R m were homeomorphic, since any bi-Lipschitz function is a homeomorphism.

Original: But R n and R m are not homeomorphic unless m = n , so no such function can exist.
Revised: But R n and R m are not homeomorphic unless m = n , so no such function can exist.

Revised: C A BLATION OF N UMBER OF I NDUCING P OINTS

Revised: In Table 3, we show the results of increasing the number of inducing points.

Revised: While more inducing points generally helps, the differences are minimal.

Revised: [Table]

Original: C S AMPLING FROM THE JOINT POSTERIOR
Revised: D S AMPLING FROM THE JOINT POSTERIOR

Original: In Figure 4, we show samples from the joint predictive posterior.
Revised: In Figure 4, we show samples from the joint predictive posterior.

Original: This type of sampling is not possible using standard deep learning models, because there is no covariance modelled between the batch elements.
Revised: This type of sampling is not possible using standard deep learning models, because there is no covariance modelled between the batch elements.

Original: Possible applications of this include batch active learning, where we want to estimate a joint posterior (Kirsch et al., 2019), but also for deep exploration in RL where we can effectively sample a full policy (Osband et al., 2016). 
Revised: Possible applications of this include batch active learning, where we want to estimate a joint posterior (Kirsch et al., 2019), but also for deep exploration in RL where we can effectively sample a full policy (Osband et al., 2016). [Figure]