Roundoﬀ Error in Metropolis-Hastings Accept-Reject Steps
1. Introduction
The Metropolis-Hastings (M-H) accept-reject step (Metropolis et al., 1953; Hastings, 1970) is a central element of many Markov chain Monte Carlo (MCMC) algorithms that are essential to modern Bayesian inference.
If performed using exact arithmetic, it can be used to ensure that a Markov chain’s stationary distribution is any distribution of interest p ( θ ), even if that distribution is only known up to a normalizing constant.
But computers cannot do exact arithmetic on real numbers, and in fact the trend in machine learning is towards lower-precision computation.
Lower-precision formats oﬀer both memory savings and more ﬂoating-point operations per second (FLOPS) per transistor.
Rather than use the traditional 64-bit ﬂoating-point format (F64), modern ML software frameworks such as TensorFlow (Abadi et al., 2015), PyTorch (Paszke et al., 2019), and
JAX (Bradbury et al., 2018) tend to default to 32-bit ﬂoating-point (F32) numerics, or even lower-precision formats such as TensorFloat-32 (TF32; Kharya, 2020) or bﬂoat16 (Wang and Kanwar, 2019) on supported hardware.
These lower-precision formats often work ﬁne for training large neural networks using stochastic gradient descent (e.g., Kalamkar et al., 2019).
But are there consequences to using low-precision arithmetic in MCMC?
In this note, we consider what happens to the M-H algorithm when it is fed log-density calculations that are subject to roundoﬀ error and catastrophic cancellations.
Below, we will brieﬂy review the nature of these errors and how they can arise in M-H algorithms.
Next, we will develop a theoretical model of roundoﬀ error in M-H corrections, and ﬁnd that it can lead to exponentially low acceptance rates in the magnitude of the errors (if the errors have Gaussian tails) and bias (if diﬀerent types of states produce diﬀerent types of errors).
Finally, we will discuss some consequences of this phenomenon, and touch on some practical ways to avoid these consequences of catastrophic cancellations in software.
2. A Source of Roundoﬀ Error in Metropolis-Hastings
Metropolis-Hastings updates (M-H; Hastings, 1970) proceed by proposing a move from the current state θ to a new state θ (cid:48) drawn from some proposal distribution q ( θ (cid:48) | θ ).
If we are trying to sample from some distribution p ( θ ), then we replace the current state θ with the proposed state θ (cid:48) with probability α = min { 1 , p ( θ (cid:48) )
q ( θ | θ (cid:48) )
p ( θ ) q ( θ (cid:48) | θ ) } , and otherwise keep the current state.
(For both convenience and accuracy, M-H implementations typically compute the densities and their ratio in log space, e.g. exp { log p ( θ (cid:48) )
− log p ( θ ) } instead of p ( θ (cid:48) )
p ( θ ) .)
This procedure leaves the distribution p ( θ ) invariant; that is, if θ ∼ p then θ (cid:48) ∼ p as well.
So repeating it with a suitable q for long enough that it forgets its initial state will yield samples from p .
The derivation of the M-H algorithm assumes that we use exact arithmetic to compute the acceptance probability α .
Often, ﬂoating-point arithmetic is a good enough approximation to arithmetic on real numbers that we can ignore this issue.
We have found F arithmetic to be problematic in at least two scenarios.
First, if the magnitude of log p ( θ (cid:48) )
+ log q ( θ | θ (cid:48) ) is very large, then there may not be enough mantissa bits in an F32 scalar to accurately represent it.
For example, the number 10000000 .
51 cannot be represented as an F32, and gets rounded to 10000001 .
This is a small relative error, but in the M-H accept-reject step absolute error is what matters.
For example, if log p ( θ (cid:48) )
+ log q ( θ | θ (cid:48) )
= 10000000 .
49 and log p ( θ )
+ log q ( θ (cid:48) | θ ) =
51, then the true acceptance probability should be e − 0 .
02 ≈ 0 .
98, but if these values are rounded before subtraction the computed acceptance probability is e − 1 ≈ 0 .
Second, when log p ( θ ) is a sum of many terms, small errors in summing these terms can accumulate.
Figure 1 illustrates this phenomenon when computing two million independent Poisson log-likelihoods with diﬀerent rate parameters.
Summing the terms for each observation in F32 yields roughly Gaussian errors about 10 times larger than the nearly uniform errors obtained when summing in F64 and rounding the ﬁnal result to F32.
The errors due to summing in F32 are clearly large enough to aﬀect the dynamics of the M-H algorithm, causing some states that might be rejected to be accepted and vice versa.
In the next section, we will explore how these errors may introduce bias or degrade the eﬃciency of the M-H algorithm.
3. Analysis of Metropolis-Hastings with Roundoﬀ Error
In this section, we analyze the eﬀects of roundoﬀ error in the log-densities fed to the
Metropolis-Hastings algorithm.
We denote the state of the Markov chain as θ , the exact target log-density as log p ( θ ), and the computed target log-density with roundoﬀ error (cid:15) as log ˆ p ( θ )
= log p ( θ )+ (cid:15) ( θ ).
Since (cid:15) ( θ ) is obtained by rounding bounded ﬂoating-point numbers, it is also bounded, so we can deﬁne the normalizing constant Z = (cid:82) θ ˆ p ( θ ) =
E p [ e (cid:15) ( θ ) ].
Note that, although (cid:15) is a deterministic function of θ , when θ is a draw from a Markov chain (cid:15) is also a random variable whose distribution depends on that of θ .
At each step, the algorithm proceeds from a state θ and cached log ˆ p ( θ ) to sample θ (cid:48) ∼ q ( · | θ ), computes the perturbed Metropolis-Hastings acceptance probability
[Equation] [Equation]
and accepts the new state θ (cid:48) with probability ˆ α .
This algorithm leaves the perturbed distribution ˆ pZ invariant.
If expectations of interest with respect to ˆ pZ are close to their values under the true p , then the asymptotic bias of this algorithm will be small.
Loosely speaking, this bias will be small if we are trying to estimate the expected value of some function h ( θ ) such that the average of e (cid:15) ( θ ) over any region in which h ( θ ) is nearly constant is approximately Z .
But if the variance of (cid:15) ( θ ) is large, then the average acceptance rate ˆ α may get very small, dramatically slowing down the algorithm.
Intuitively, the issue is that e (cid:15) ( θ ) may have quite heavy tails, and so most of the mass in ˆ p will be concentrated on values of θ for which e (cid:15) ( θ ) are abnormally large.
Unfortunately, there is no way to preferentially select states whose roundoﬀ errors are large and positive, so the chain must wait until it gets a favorable error by chance.
(Pseudo-marginal MCMC suﬀers from a very similar issue (Andrieu and
Roberts, 2009).)
We can make this intuition a bit more quantitative.
At stationarity, the expected acceptance probability is
[Equation]
Consider what happens if we have an ideal proposal distribution q ( θ (cid:48) | θ )
= p ( θ (cid:48) ).
Without roundoﬀ, this would yield perfect samples and an acceptance rate of 1.
But with roundoﬀ, we can bound the acceptance rate as
[Equation]
where the bound follows from applying Markov’s inequality to e (cid:15) ( θ ) / 2 .
By the strict convexity of the square, this upper bound is strictly less than 1 unless the variance of (cid:15) is 0.
How much less depends on both the scale and the tails of the distribution of (cid:15) .
Two cases are of particular interest: ﬁrst, where (cid:15) follows a normal distribution (e.g., due to summing many independent roundoﬀ errors), and second, where (cid:15) follows a uniform distribution (e.g., due to rounding an accurately computed log-density to ﬁt into a single-precision ﬂoat).
In both of these cases we can compute the exact acceptance rate ˆ α (cid:63) .
The Gaussian case:
If (cid:15) ∼ N ( µ, σ ), then equation 5 can be written as
[Equation]
where φ ( (cid:15) ) and Φ( (cid:15) ) are the pdf and cdf of a standard normal distribution.
Since 2 φ ( x )Φ( − x ) is the pdf of a skew-normal distribution with shape parameter α = − 1, we can use the skewnormal’s moment-generating function M to compute the integral:
[Equation]
If σ is close to zero, then a ﬁrst-order Taylor approximation implies that the acceptance rate will be
[Equation]
(10) so small amounts of roundoﬀ error will not hurt the acceptance rate too badly.
However, as σ gets large, the acceptance rate approaches zero very rapidly, as Figure 2 shows.
For example, if σ = 2 then ˆ α (cid:63) ≈ 0 . 16; if σ = 4 then ˆ α (cid:63) ≈ 0 .
The uniform case:
If (cid:15) ∼ Uniform([ µ − σ, µ + σ ]), then equation 5 can be written as
[Equation]
For small σ , this is 1 − σ 3 + O ( σ 3 ).
For large σ , the tanh saturates and it becomes 1 σ .
This is a much more graceful degradation in acceptance rate than the Gaussian case, but for large roundoﬀ errors it may still be quite bad.
In summary, we have argued that roundoﬀ errors in computing the log-density that is used in the Metropolis-Hastings accept-reject step may or may not induce bias, but they can aggressively degrade the acceptance rate to the point that the algorithm becomes unusable.
4. Empirical Example: Bayesian Linear Regression
In this section, we will empirically demonstrate the issues described above by trying to use
Hamiltonian Monte Carlo (HMC; Neal, 2011) to sample from the posterior of a simple
Bayesian linear regression model applied to synthetic data.
The model is
[Equation]
(12) where β ∈ R 2 , x ∈ R N × 2 , and y ∈ R N .
To generate the dataset, we drew each x nd i.i.d.
from a standard normal and sampled β and y from the generative process above.
We ran HMC in both F32 and F64 on a CPU using TensorFlow Probability (Lao et al., 2020) in JAX (Bradbury et al., 2018) to sample from p ( β | x, y ) for N ∈ { 20000 , 100000 , 200000 , 500000 , 1000000 } .
All chains were run for 500 iterations with 20 leapfrog steps and a step size of 0 .
005 / √ N .
Each chain was initialized with the value of β that was used to generate y , which is a valid draw from the posterior p ( β | x, y ).
When run in F64, the average acceptance probabilities for these chains were very high: between 0 . 99975 and 0 .
for all values of N .
Figure 3 summarizes the results of the F32 experiments.
as N gets larger, the number of terms being added together to compute the posterior log-density gets larger, and the number of roundoﬀ errors being added together likewise grows.
As the magnitude of the roundoﬀ errors grows, the average acceptance probability drops roughly as equationpredicts it should if the roundoﬀ errors are Gaussian, although it is a bit higher than predicted for N = 1000000.
This may be because the chain has not actually converged to its stationary distribution; it might take many tries before it randomly selects a β with a large enough positive roundoﬀ error (cid:15) ( β ) to qualify as “typical” under the perturbed stationary distribution ˆ p ( β ).
5. Discussion and Practical Considerations
We have illustrated a way in which roundoﬀ error can severely degrade the performance of Metropolis-Hastings procedures, leading in some cases to bias and in others to such low acceptance rates that the method is useless.
Unfortunately these issues seem most pronounced in precisely the sort of large-scale problems that motivate the use of cheap, low-precision arithmetic.
We have found a few of ways of diagnosing these roundoﬀ issues.
The simplest is of course to run the algorithm in F64 and see if anything changes, but not all implementations make that easy.
One can also inspect the values of log p
( θ ) that the algorithm is generating; if they have more than six digits to the left of the decimal point, then there may be cause for concern.
Finally, most Metropolis-Hastings algorithms have some kind of step size parameter; in an exact-arithmetic implementation, as the step size approaches zero, the acceptance rate would go to one.
If it is diﬃcult or impossible to drive the acceptance rate to one by reducing the step size, that is a warning sign that numerical issues may be present.
Likewise, if one is employing a step-size adaptation scheme that targets a particular acceptance rate, and instead of converging it drives the step size to zero, then one should suspect roundoﬀ error as the culprit.
There are multiple ways to address roundoﬀ issues.
A simple approach is to compute the terms being summed in log p ( θ ) in F32, but convert them to F64 before the ﬁnal (relatively cheap) summation.
If F64 summation is not available or convenient, one can also use methods like Kahan summation to eliminate catastrophic cancellations.
In many cases it is more accurate to compute a sum of diﬀerences than a diﬀerence of sums, e.g.,
[Equation]
of
[Equation]
p
[Equation]
| θ )); however, this approach makes caching of previous results more memory-intensive and is somewhat cumbersome, since it breaks the abstraction of a scalar log-density.
Also, hardware that supports stochastic rounding may dramatically reduce roundoﬀ error (Gupta et al., 2015).
Acknowledgments
I wish to thank my many colleagues at Google (and on the TensorFlow Probability team in particular) who helped me identify and understand these issues, especially Brian Patton,
Alexey Radul, Jamie Smith and Pavel Sountsov
Appendix A. Code to generate ﬁgure
def p
o i s s o n l
o g p d f (x , rate ) : return ( − scipy .
s p e c
i a l . gammaln( x+1) + x ∗ np .
log ( rate ) − rate ) .
sum ( ) x = np .
random .
poisson (1 , s i z e =2000000) sum errors =
[ ] f i n a l r o u n d o f f e r r
o r s =
[ ] f
o r i in range ( 1 0 0 0 ) : rate = np .
exp ( 0 .
2 ∗ np .
random .
randn ( ) ) logpdf64 =
p o
i s s o n l o g p
d
f
(x , rate ) .
sum ( ) logpdf32 = p
o
i s s o n l o g p
d
f
( x . astype (np .
f l o a t 3 2 ) , rate .
astype (np .
f l o a t 3 2 ) ) .
sum ( ) sum errors .
append ( logpdf32 − logpdf64 )
f i n a l r o u n d
o
f f e r r o r s .
append ( logpdf64 .
astype (np .
f l o a t 3 2 ) − logpdf64 ) p l t .
f i g u r e
( f i g s i
z
e =[12 , 4 ] ) p l t .
subplot (1 , 2 , 1) p l t .
t i t l e ( ’Sum of F32s ’ ) sns .
d i s t p l o t ( sum errors ) p l t .
x l a b e l ( ’ Error in log − density c a l c u l a t i o n ’ )
p l t .
subplot (1 , 2 , 2) p l t .
t i t l e ( ’Sum of F64s rounded to F32 ’ ) sns .
d i s t p l o t ( f i n a l r o u n d o f f e r r
o
r s ) p l t .
x l a b e l ( ’ Error in log − density c a l c u l a t i o n ’ ) sns .
d i s t p l o t ( e r r o r s )