The Implicit Delta Method
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Uncertainty quantiﬁcation is a crucial part of drawing credible conclusions frompredictive models, whether concerned about the prediction at a given point or anydownstream evaluation that uses the model as input.
When the predictive modelis simple and its evaluation differentiable, this task is solved by the delta method,where we propagate the asymptotically-normal uncertainty in the predictive modelthrough the evaluation to compute standard errors and Wald conﬁdence intervals.
However, this becomes difﬁcult when the model and/or evaluation becomes morecomplex.
Remedies include the bootstrap, but it can be computationally infeasiblewhen training the model even once is costly.
In this paper, we propose an alternative,the implicit delta method, which works by inﬁnitesimally regularizing the trainingloss of the predictive model to automatically assess downstream uncertainty.
Weshow that the change in the evaluation due to regularization is consistent for theasymptotic variance of the evaluation estimator, even when the inﬁnitesimal changeis approximated by a ﬁnite difference.
This provides both a reliable quantiﬁcationof uncertainty in terms of standard errors as well as permits the construction ofcalibrated conﬁdence intervals.
We discuss connections to other approaches touncertainty quantiﬁcation, both Bayesian and frequentist, and demonstrate ourapproach empirically.
Introduction
In this paper, we consider quantifying uncertainty in evaluations of predictive models trained on data.
Consider the following examples.
We ﬁt a complex model (such as a neural net) to predict meanservice time for an incoming call to a call center given some features, and we use it to prioritizecalls in a queuing system.
We may be interested in conﬁdence intervals on the average wait timeof incoming calls in the queue.
Such conﬁdence intervals would be crucial for drawing credibleconclusions about such evaluations, since we know we cannot take the point prediction at face valuegiven the sampling uncertainty in the data.
We may, alternatively, be ﬁtting a ranking algorithm bypredicting user interaction from user-item features and then applying some ﬁxed business rules ontop, and we want to assess how often certain item categories would end up at the top.
Of course,we would want to understand how certain we are in this assessment.
Or, we ﬁt a complex modelto predict mean demand given price and user features from a price experiment we ran, and we useit to target discounts by optimizing demand at a price times unit proﬁt.
We may be interested inconﬁdence intervals on the average proﬁt over a given distribution of features.
All of these examples have two important features: they involve (1) a computationally burdensomestep of ﬁtting a large-scale model and (2) evaluating the result using a complicated function that neednot even be known explicitly.
Were both of these simple (simple model and simple function thereof),we could just use the classic delta method [7] (see next section for detail).
However, when theseaspects are complex and the model involves many parameters, it may be too prohibitive to eitheranalytically derive the whole inverse Fisher information matrix in the many model parameters orcompute and invert the Hessian of the training loss empirically as well as compute the gradient ofthe ﬁnal evaluation as a function of all parameters [24].
Even one aspect being complex may pose a
[Table]
+xncBN8D34Fv2+jraDpvGD/IfjzF7dusfw=</latexit> serious challenge ( e.g. , uncertainty quantiﬁcation for the prediction of a complex model at a point).
Aremedy may be to bootstrap the whole process from data to ﬁnal evaluation, but that can prove verycomputationally burdensome [9].
Usually just ﬁtting the model once is already an expensive task;ﬁtting it hundreds of times can be operationally infeasible.
Other remedies, in the case of neural nets,may be the use of Langevin dynamics [33] or random dropout [12].
But these assess uncertainty innetwork weights and/or network predictions, which must then be translated to uncertainty in the ﬁnalevaluation.
In this paper, we propose a direct yet inexpensive way to generically assess uncertainty in suchsettings.
Speciﬁcally, we consider conducting inference when the estimator is some speciﬁed functionof a (conditional) maximum likelihood estimator (MLE), such as a regression or classiﬁcation model.
Our proposal, the implicit delta method , works by simply adding an inﬁnitesimal regularization to the
MLE objective ( e.g. , the sum of squared errors).
We prove that the inﬁnitesimal change in the ﬁnalestimator due to this regularization is consistent for its asymptotic variance, the same variance thatwould have been predicted by the delta method in theory.
Hence, the name of our method: we areconducting a delta-method quantiﬁcation of uncertainty implicitly without explicitly propagating theuncertainty through the derivative of the evaluation function, analytically deriving the possibly-huge
Fisher information matrix, or approximating it empirically.
We prove that even when we approximatethe inﬁnitesimal change with a ﬁnite difference with constant width, the change we measure is stillconsistent for the asymptotic variance.
This not only gives an assessment of uncertainty in terms ofstandard errors, it also permits us to construct calibrated conﬁdence intervals.
We demonstrate this inexperiments, comparing to other popular approaches for uncertainty quantiﬁcation, both Bayesianand frequentist.
Problem Set Up and the Delta Method
We consider an estimate constructed in two steps: ﬁrst we ﬁt a model using maximum likelihoodestimation (MLE) and then apply some function to it.
Namely, we consider data given by nindependent and identically distributed (iid) observations Z i ∈
Z , i = 1 , . . . , n , drawn from apopulation with density f
( z ; θ 0 ) with respect to some base measure µ on Z .
For example, the datamay consist of observations of features X and responses Y , with Z =
( X, Y ) .
In the ﬁrst step, we ﬁt a model to the data by MLE.
Given a model { f ( z ; θ ) : θ ∈ Θ } of densities(with respect to µ ) parametrized by θ ∈ Θ ⊆ R d , we set
Model ﬁtting
[Equation]
A prominent example is generalized regression, where we observe features and responses Z =
( X, Y ) ,have a parametrized predictor h θ ( x ) ∈ R p , and a parametric model g ( y ; ϑ ) with ϑ ∈ R p .
We then setf (( x, y ); θ ) =
g ( y ; h θ ( x )) .
1 Examples include least-squares regression, binary classiﬁcation with
Setting f
(( x, y ); θ ) =
g ( y ; h θ ( x ))
f ( x ) using the true unknown density f ( x ) of X does not change the MLE cross-entropy loss, and Poisson regression, all with possibly complex and nonlinear predictors ( e.g. ,neural nets).
In the second step, we process the trained model in some way to come up with our estimate.
Given some ψ : Θ (cid:55)→ R , we compute
Evaluation using ﬁtted model: ˆ ψ n = ψ (ˆ θ n ) .
One example in the case of generalized regression is evaluation of the predictor at a speciﬁedpoint, ψ ( θ )
=
g θ ( x 0 ) .
Another example is, when g θ ( x, p ) corresponds to predicted mean demand at price p given features x , we may be interested in the average optimal proﬁt, ψ ( θ ) =1 m (cid:80)
[Equation]
, for an evaluation dataset { x j : j = 1 , . . . , m } .
More generally,ψ could be more opaque: it could involve, for example, simulating a queuing system with a controllerparametrized by θ , such as a priority policy with priority score g θ ( x ) .
We are interested in conducting uncertainty quantiﬁcation for ˆ ψ n , and in particular in inference on itspopulation limit, ψ 0 = ψ ( θ 0 ) .
One way to do this inference is to propagate through ψ the uncertaintywithin ˆ θ n about θ 0 , provided we understand the latter uncertainty.
This is the so-called delta method .
To apply it, we must ﬁrst understand the uncertainty in ˆ θ n .
Provided some regularity holds, thisuncertainty can be characterized by the curvature of the objective function P (cid:96) ( θ, · ) at θ = θ 0 : if thecurvature is sharp (resp., ﬂat) then perturbing the objective and minimizing P n (cid:96) ( θ, · ) instead does not(resp.
, does) move the minimizer far away.
This curvature is exactly the Fisher information matrix :
[Equation]
Speciﬁcally, under appropriate regularity conditions,
[Equation]
In the above, N ( µ, Σ) refers to the multivariate normal distribution and (cid:32) refers to convergence indistribution.
There are a variety of speciﬁc technical conditions that can establish this result.
For anabstract presentation see theorems 9.27 and 9.28 in Wasserman [32].
For more rigorous treatmentssee theorem 13.2 of Wooldridge [35], theorem 3.3 of Newey and McFadden [23], theorem 5.1 of
Lehmann and Casella [20], or theorem 8.3 of Davidson and MacKinnon [5], each of which usesslightly different technical regularity conditions.
Given Eq.
(2) holds with I ( θ 0 )
(cid:31) 0 , the delta method would then guarantee that (see theorem 5.15 in32)
[Equation]
provided ∇ ψ ( θ 0 ) exists and V 0 > 0 .
An immediate and very important consequence of this is that we can construct calibrated conﬁdenceintervals for ψ 0 : under Eq.
(3),if
[Equation]
then
[Equation]
where Φ refers to the cumulative distribution function of the standard normal distribution and → prefers to convergence in probability.
For example, as long as I ( θ ) and ∇ ψ ( θ ) are continuous at θ 0 ,we can use
[Equation]
[Equation]
As discussed in the introduction, however, this approach may prove intractable in practice, especially when θ has many dimensions.
Since we are only truly concerned with the uncertainty in ˆ ψ n and not in ˆ θ n , it may seem unnecessary and overly cumbersome to ﬁrst compute the uncertainty in the latterand then propagate it.
We next present our method, which does this all implicitly, never workingdirectly with the vector θ except as an optimization variable in maximizing the MLE objective and aperturbation thereof.
The Implicit Delta Method
We would like to construct calibrated conﬁdence intervals as in Eq.
(4), but computing the estimatedstandard error as in Eq.
(5) can be prohibitive.
The IDM is a way to compute the estimated standarderror while neither explicitly computing the uncertainty in ˆ θ n nor propagating this uncertaintythrough ψ .
Instead, we will simply slightly perturb the original MLE in Eq.
(1) using a little bit ofregularization, which will implicitly do both of these difﬁcult tasks for us.
To deﬁne the IDM, we ﬁrst deﬁne a regularized version of the MLE.
Given any λ ≥ 0 , we consideradding the regularizer λψ ( θ ) to Eq.
(1) as well as the corresponding ﬁnal estimator after passingthrough ψ :
[Equation]
We refer to this as ψ -regularized MLE.
We then deﬁne the inﬁnitesimal IDM (IIDM) as the inﬁnitesimal change ( i.e. , derivative) in our ﬁnalestimate using ψ -regularized MLE as we inﬁnitesimally increase λ from 0 :
[Equation]
IIDM
[Equation]
Our ﬁrst result shows that the IIDM estimate is consistent for the true asymptotic variance in Eq.
Theorem 1.
Suppose that ˆ θ n → p θ 0 ∈ Interior(Θ) , I ( θ 0 ) (cid:31) 0 , and that, in a neighborhood of θ 0 , ψ ( θ ) is continuously differentiable and f ( Z ; θ ) is almost surely twice continuously differentiable in θ with a Hessian that is bounded in operator norm by an integrable function of Z .
Then
[Equation]
The signiﬁcance of Theorem 1 is that, per Eq.
(4), it implies that
[Equation]
is a calibrated β -conﬁdence interval for ψ 0 .
Note that, aside from conditions on ψ (which are the same as needed for Eqs.
(3) and (5) to work),the regularity conditions required in Theorem 1 are implied by the regularity conditions required forestablishing Eq.
(2) by, for example, any of Davidson and MacKinnon [5], Lehmann and Casella[20] , Newey and McFadden [23], Wooldridge [35].
In that sense, these conditions are not strong asthey are already needed for ˆ V DeltaMethod n to be a good estimate of uncertainty to begin with, and theyﬁt into the existing framework for the asymptotic analysis of MLE.
The implication of Theorem 1 is that we may be able to implicitly complete the steps of the deltamethod (compute the uncertainty in ˆ θ n , then propagate it through ψ ) by simply assessing theimpact of regularizing the MLE.
However, this requires we actually differentiate with respect to theregularization coefﬁcient.
While this requires computing just one ﬁrst derivative (rather than manyﬁrst and second derivatives as in Eq.
, it is still not clear how to do this in practice.
In practice, we might approximate this derivative using ﬁnite differences, i.e. , replace the limit in
Eq.
(7) with a very small λ .
This gives rise to what we call the ﬁnite-difference IDM (FDIDM),deﬁned as follows for a given λ n > 0 :
[Equation]
Our next result shows that it in fact sufﬁces to choose λ n constant.
In fact any choice of λ n growingstrictly slower than n , yields that n ˆ V FDIDM n is also consistent for V 0 , just like ˆ V IIDM n , provided justslightly more regularity holds.
Theorem 2.
Fix any λ n = o ( n ) .
Suppose that in addition to the assumptions of Theorem 1, in a neighborhood of θ 0 , ψ ( θ ) is thrice continuously differentiable and f ( Z ; θ ) is almost surely thrice continuously differentiable in θ with a third-order derivative that is bounded in operator norm by an integrable function of Z .
Then
[Equation]
It may seem surprising that a constant λ n sufﬁces or that λ n is even allowed to grow, but that can beseen as an artifact of the fact we did not normalize the sum over the data in Eq.
(6) by 1 /n .
If wedid normalize, it would be equivalent to rescaling λ by n , so that o ( n ) becomes o (1) , i.e. , requiringa vanishing increment for the ﬁnite differencing.
Nonetheless, writing Eq.
(6) as we did is veryconvenient, as it matches how one usually applies optimization algorithms such as stochastic gradientdescent to training objectives, and it makes the choice of λ n for Eq.
(8) very easy: just ﬁx someconstant and do not worry about the scaling with n .
For example, setting λ n = 1 suggests a verysimple-looking 95%-conﬁdence interval:
[Equation]
Remark 1 (Regression Using Squared Error Loss) .
When training regression models we usuallyminimize over model parameters ( e.g. , neural net weights) the sum over the data of squared errorloss, (cid:96) (( x, y ); θ ) =
( y − g θ ( x )) 2 .
This differs from the corresponding Gaussian log likelihood bya factor of − 12 σ 2 (and some constants that do not matter), where σ 2 is the residual variance of Ygiven X .
Therefore, to apply IDM, all we should do is simply regularize the sum-of-squared-errorsminimization problem by − 2 σ 2 λψ ( θ ) , as that would be equivalent to dividing the log likelihood partby − 2 σ 2 .
Of course, we do not know σ 2 , but we can estimate it by ˆ σ 2 n = 1 n
[Equation]
ni =
( y − g ˆ θ n ( x )) 2 , thatis, the minimum average sum of squared errors.
Since ˆ σ 2 n → p σ 2 , as it is in fact the MLE estimatefor σ 2 , the asymptotic guarantees of Theorems 1 and 2 will continue to hold after this rescaling.
Remark 2 (Using IDM to Compute the Fisher Information) .
A by-product of the proof of
Theorem 1 is that, if we looked at the (vector-valued) derivative ˆ W n =
∂∂λ ˆ θ n ( λ ; ψ ) (cid:12)(cid:12) λ
=
=lim λ
→
[Equation]
(cid:17) , then n ˆ W n → p I ( θ 0 ) − 1 ∇ ψ ( θ 0 ) .
Therefore, if we set ψ ( θ ) = θ
i , i.e. ,the i th component of θ , then ˆ W n converges to the i th column of I ( θ 0 ) − 1 .
Thus, by regularizing eachcomponent of θ in turn, we obtain the whole matrix.
Nonetheless, the whole raison d’être of IDM is to avoid working directly with the parameter vector θaltogether, and simply propagate its uncertainty automatically via the MLE optimization problem.
Forexample, if we consider neural net regression, IDM would never make explicit reference to the vectorof weights itself, only to the trained prediction model and its prediction performance on data.
Theabove, wherein we compute the uncertainty in θ directly, stands in contradiction to this.
Nonetheless,it can be a useful observation when inference on θ itself is for some reason of interest.
3.1 Extension to Multivariate Evaluations
We have so far focused on scalar evaluations for ease of presentation and as it covers the mostimportant cases.
We now show how our method easily extends to the multivariate case, whereψ ( θ ) =
( ψ (1) ( θ ) , . . .
, ψ ( K ) ( θ ))
∈ R K .
The reason it may not sufﬁce to run IDM separately for eachcomponent is that we may be interested in the covariance of the evaluations.
Under the appropriateconditions, the extension of the delta method for MLE (Eq.
(3)) to multivariate evaluations is
[Equation]
where J ij ( θ ) =
∂∂θ i ψ ( j ) ( θ ) is the K × d Jacobian of ψ ( θ ) .
Our extensions of IIDM and FDIDM to multivariate evaluations are as follows:
[Equation]
Theorem 3.
n ˆ V IIDM n → p V 0 under the conditions of Theorem 1, and n ˆ V FDIDM n → p V 0 under the 177 conditions of Theorem 2, both as K × K matrices.
Surprisingly, this shows one need only solve K + 1 (possibly) regularized MLEs to get the full
K × K covariance.
(See Alg. 2 in supplement.)
3.2 Handling Non-differentiable Evaluations and Evaluation Uncertainty
So far we have assumed that the evaluation function is a known and differentiable function.
Bothstatements may be false when we are interested in evaluating average performance on a populationbut we only have a ﬁnite evaluation data set and unit performance is not differentiable.
Speciﬁcally, let W 1 , . .
, W m ∼ W denote the evaluation data set (which may be the same as the training set or otherwise dependent or it may be an independent data set) and let h ( w ; θ ) the unit evaluation function.
Consider the empirical evaluation map
[Equation]
If h ( W ; θ ) is almost surely not differentiable in θ , then ψ is also almost surely not differentiable,which poses a challenge.
We will show, however, that even though ψ is not differentiable (whichwould break the usual delta method), FDIDM actually remains valid, without any changes to themethod , provided certain on-average-differentiability holds.
To motivate the challenge of nondifferentiability and the plausibility of on-average-differentiability,consider an example where g θ ( x ) represents an order quantity to stock in context x and w = ( x, d )represents features and demand.
If h (( x, d ); θ ) = max { d − g θ ( x ) , 0 } then ψ ( θ ) quantiﬁes average 191 unmet demand, but h is not differentiable.
Other non-differentiable examples include evaluatingregression and classiﬁcation models’ performance using non-differentiable utility functions.
Whileh may not be differentiable and hence neither ψ , it may still be plausible that its expectation 194 E [ h ( W ; θ )]
=
E [ ψ ( θ )] is differentiable.
For example, if the distribution of demand conditioned onfeatures is continuous, then the derivative of E [ h ( W ; θ )] in the example of average unmet demandwill be the average of the conditional cumulative distribution function at g θ ( x ) times ∇ θ g θ ( x ) , andthe second derivative will involve the conditional density.
We next show FDIDM still works with non-differentiable ψ , given some on-average-differentiability.
Theorem 4.
Consider either m (cid:16) n or m (cid:29) n .
Fix λ n = λ > 0 constant.
Suppose the assumptions 200 of Theorem 2 hold, that Eq.
(2) holds, and that E [ h ( W ; θ )] is twice continuously differentiable at θ 0 .
Suppose further that, for some functions L ( w ) and ˙ h 0 ( w ) with E [ L 2 ( W )] < ∞ , we have that in aneighborhood of θ 0 ,
[Equation]
almost surely ,
[Equation]
Then
[Equation]
In the above example, L ( w ) would be the Lipschitz constant of g θ ( x ) in θ and ˙ h 0 ( w ) would be thecumulative distribution function of demand at g θ
( X ) .
Although the asymptotic variance of ˆ ψ n − ψ 0 is now different (in particular both V 0 in Eq.
(3) andˆ V DeltaMethod n in Eq.
(5) are ill-deﬁned), Theorem 4 shows that ˆ V FDIDM n actually remains consistentfor this new asymptotic variance.
Thus, it provides a consistent estimate of standard errors and itstill gives calibrated conﬁdence intervals (note ψ 0 is now random but we can still have a conﬁdenceinterval for it).
In some cases we may want to directly conduct inference on the population version of the evaluation,ψ ∗
= E [ h ( W ; θ 0 )] .
To do this, all we have to do is simply also add the uncertainty due to ﬁniteevaluation data set.
Under standard regularity conditions, we have
[Equation]
where
[Equation]
Therefore, provided the training and evaluation data sets are independent,
[Equation]
If not independent, then
[Equation]
V ψm provides a consistent upper bound on the standard error.
3.3 Implementation
FDIDM is given in pseudocode in Alg.
Given an objective function L := (cid:80) ni log f
( Z i ; θ ) ,evaluation function ψ , and scalar width λ , FDIDM returns the estimated variance of ψ (ˆ θ n ) .
The ﬁrststep is to maximize the original objective w.r.t.
θ .
Usually, this task has already been solved as this isthe trained predictive model.
Then, maximize the ψ -regularized objective w.r.t.
θ .
Finally, return theestimated variance using the ﬁnite-difference method evaluated at λ = 0 .
See Appendix D for thecorresponding algorithm when ψ is multivariate.
In practice, one can further reduce the computational cost of FDIDM due to the fact that the ψregularized objective can be made arbitrarily close to original objective by choosing λ small enough.
Input: Learning objective L , evaluation ψ , scalar λ 1 Function FDIDM( , ψ , λ ) :
[Equation]
// optimize learning objective
[Equation]
// optimize ψ -regularized objective
// estimated variance of ψ (ˆ θ n )
Speciﬁcally, when using stochastic gradient ascent in FDIDM, once the optimum ˆ θ n has been found,only a small number of gradient updates may be required to also ﬁnd ˆ θ n ( λ ) .
FDIDM also admits non-gradient-based approaches.
Consider the case that ψ is a simulator that takesa ﬁtted model and returns a set of evaluations and no gradient.
Then the ψ -regularized objective mayoptimized by gradient-free methods such as Nelder-Mead [22] and Bayesian optimization [11].
Alternatives for Uncertainty Quantiﬁcation and Related Work
Uncertainty quantiﬁcation in machine learning is a topic of major interest due to the need to makedownstream inferences and decisions based on the predictions of large-scale networks trained onmassive datasets in either a frequentist [13, 24, 27] or Bayesian fashion [2, 6, 12].
An exhaustiveaccount of the literature is outside the scope of this paper.
We highlight the principal ideas and pointsof contact with our work.
The Bootstrap The bootstrap simulates sampling from the true data generating distribution byresampling from the observed dataset (see, e.g. , [9] for an introduction and [18] for theory on whenit works).
The key advantage is that it enables general-purpose and easy-to-implement uncertaintyquantiﬁcation for estimators.
It comes at a high computational burden because the estimator, whichmay comprise a model-ﬁtting algorithm and prediction, needs to be executed many times.
In thecontext of deep learning, many useful adaptations of the bootstrap and the related jackknife have beenproposed to increase its computational efﬁciency [13, 26, 27].
Maintaining an ensemble of models asa representation of the variability of the evaluation is an appealing intuition that does not restrict oneto local approximations, and may be combined with local approximations where necessary.
The Functional Delta Method The delta method [7] is a classic approach that is widely used withsmall models with an analytic Fisher information matrix ( e.g. , linear regression) and, more recently,auto-differentiation unlocks the delta method for a larger class of models [24].
The bottleneckis the need to calculate then invert the Fisher information matrix, for which there are variousapproximations [21, 28].
The functional delta method extends the delta method to evaluations ofinﬁnite-dimensional parameters (see Ch.
but is usually restricted to analytically derivinginﬂuence functions in theory by differentiating the population estimand with respect to distributionsand then approximating the inﬂuence function by plugging in estimates of unknown nuisances [3, 14].
Bayesian Uncertainty Quantiﬁcation Tractable methods for approximate Bayesian inferencein neural networks, such as variational inference in feed-forward nets [2], autoencoders [17, 30],normalizing ﬂows [29], dropout uncertainty [12], stochastic gradient Langevin dynamics [33] andrelated approaches, present an impressive range of options for uncertainty quantiﬁcation.
In caseswhere it is sufﬁcient to only consider a single mode in the posterior, local methods can prove useful.
In particular, an alternative interpretation of the delta method is as a special case of the Laplaceapproximation to Bayesian inference, where Eq.
(5) arises in the posterior predictive distributionfor a local multivariate Gaussian approximation around the maximum a posteriori estimate.
Severalrecent works have investigated the potential of the Laplace approximation as a way to avoid havingto characterizing the full posterior in deep networks [6, 15, 16].
IDM can provide another way toperform a Laplace approximation and may be orthogonally combined with the above methods.
Experiments
In this section, we evaluate ﬁnite-difference implicit delta method (FDIDM) on a range of tasks thatrequire conﬁdence intervals.
2 Our goal is to quantify the extent to which FDIDM applies in practice
[Figure]
y = − sin(3 x − 310 ) + 110 (cid:15) , where (cid:15) ∼ N (0 , 1) and how it compares to alternative methods.
We start with 1D synthetic data in Sec.
5.1 where weapply a neural net to recover known functions from small datasets.
Then, in Sec. 5.2, we considerthe task of inferring average utility under a neural net trained on a set of real-world benchmarkdatasets.
Finally, in Sec. 5.3, we apply FDIDM to variational autoencoders and use the implicit deltaperspective to understand the effect of KL down-weighting.
5.1 1D Synthetic Examples
We consider known quadratic and sinusoidal functions from which we draw a random dataset.
Fig 2gives
the data generating stochastic function for a sin wave and the resulting ﬁts for FDIDM, theclassic delta method, a Gaussian process (GP) with Matern-52 kernel, as well as simulation fromthe true data generating function.
(Appendices B and C provide the results on the quadratic functionand further experimental details, respectively.)
The quadratic example has evenly dispersed inputdata and there is close alignment between the methods.
The sin wave is more challenging because itrequires extrapolation – also known as “in-between” uncertainty in [10] – from outside the rangesof given inputs.
As expected, IDM agrees most with the delta method while the GP overestimatesuncertainty, particularly for extrapolation at the outer edges.
However, it should be noted that the
GP kernel 3 gives a different mean to IDM, DM, and simulation which are all using a neural net with1 hidden layer of 50 tanh units.
4 The corresponding full covariance matrix of the predictions is alsogiven in Fig.
All the methods recover the high level structure of covariance for both examples,though the scale factors differ considerably, especially for the sin wave.
5.2 Conﬁdence in Predicted Cost Downstream of Classiﬁcation
A set of classiﬁcation tasks are solved with a neural net with one hidden layer and 50 tanh hiddenunits.
In this setting, we wish to calculate conﬁdence intervals over total cost in a downstream taskunder predictions from the network.
An arbitrary cost function is set up, in this case, the average crossentropy of the observations on a held-out validation dataset, though in practice we could have a widevariety of cost functions relating to the task downstream of the classiﬁer.
It is challenging to form aconﬁdence interval for even this simple cost function because it is a function of predictions from thenetwork.
Under this scenario, it is typical to make a bootstrapping estimate, requiring B times thecost of training the network (here, we use B = 50 ).
FDIDM is also applicable in this setting.
Weshow both methods on MNIST image classiﬁcation [19] and a set of UCI benchmark datasets [8] in
Fig.
We ﬁnd that FDIDM has good coverage for a fraction of the cost of the bootstrap estimate.
[Figure] [Figure]
5.3 Down-Weighting KL in Variational Autoencoders
The variational autoencoder (VAE) is a prominent example of approximate inference in deep generative models [17, 30].
In practice, it has been observed that down-weighting the KL term in thevariational objective by a factor 1 T , where T > 1 , results in signiﬁcantly better accuracy on held-outtest data [1, 36].
This is closely related to other ways of reweighting the prior and likelihood termsin approximate inference such as data augmentation [25] and the cold posterior effect [34].
Variousexplanations for the beneﬁt of setting T > 1 have been posited, e.g. , model mispeciﬁcation.
Here, webrieﬂy evaluate a precise IDM-based interpretation of this phenomenon.
Observing that any objective does not change its critical points under (non-zero) rescaling, it holds thatdown-weighting the KL term yields the same optimization problem as up-weighting the reconstructionerror by T .
We consider this setting and deﬁne ψ to be the negative reconstruction error under a VAE.
Rearranging the terms in Eq.
(8), letting λ n := T − 1 , we have
[Equation]
[Equation]
The two immediate implications of Eq.
(10) for ﬁxed T > 1 of o ( n ) are
that,1.
the reconstruction quality for the objective implied by variational Bayes is upper bounded by thatof the objective with up-weighted likelihood; and,2.
the higher the variance in the reconstruction quality, as determined by the dataset and model, thegreater the beneﬁt of up-weighting the likelihood term in variational Bayes.
Since (1) is already supported by existing empirical work, we focus on evaluating whether (2) alsoholds in practice.
To do this, we artiﬁcially increase the variance of the reconstruction quality byperturbing a proportion of the dataset and compare ˆ ψ n for various T .
Fig.
4 shows the results forapplying a VAE to the MNIST dataset after adding i.i.d.
noise (cid:15) ∼ Uniform(0 , 120 ) to each pixel in arandomly chosen fraction of the images.
The ﬁgure indicates that, as predicted by Eq.
(10), the gapin the reconstruction quality for T > 1 relative to T = 1 increases as more variance is introduced.
Higher values of T do indeed result in better reconstruction quality and this advantage grows withthe amount of variance.
In sum, these ﬁndings are consistent with the hypothesis that the advantagefrom KL down-weighting may be explained as the residual between the ψ -regularized variationalobjective and the objective implied by the evidence lower bound, though further studies are required.
Discussion, Limitation, and Conclusions
In this paper we develop the implicit delta method for forming calibrated conﬁdence intervals viaa careful regularization of the model objective.
Like the delta method, the method requires the
MLE to be at a stable optimum, where perturbations around the optimum reliably capture samplinguncertainty.
If this is not the case, e.g. , the parameter has failed to converge or the objective itselfis changing, it yields unreliable results.
For this reason, IDM – like the delta method and thebootstrap – may be misleading for small data, and indeed uncertainty quantiﬁcation with small data isfundamentally difﬁcult.
The most appealing feature of IDM is that it does not require the variance ofthe parameters to be made explicit, which also suggests future research in exploring the compatibilityof nonparametric models with IDM.
There is also the potential to explore IDM in constrained MLE.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] Proof of consistency in Sec. 3; connections discussedin Sec. 4; evaluation in Sec.
(b) Did you describe the limitations of your work?
[Yes] See Sec. 6(c)
Did you discuss any potential negative societal impacts of your work?
[N/A]
To the
best of the authors’ knowledge, our work does not have negative social impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sec.
3 andAppendix A
(b) Did you include complete proofs of all theoretical results?
[Yes] See Sec. 3 and
Appendix A
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] In supplement;will also be posted on github outside of the anonymous review process
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec. 5, Appendix B and C, and source code
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
d) Did you discuss whether and how consent was obtained from people whose data you’re
[-= 
-]
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]