D IFFERENTIABLE B AYESIAN N EURAL N ETWORK I NFERENCE FOR D ATA S TREAMS
Anonymous authors
Paper under double-blind review
A BSTRACT
While deep neural networks (NNs) do not provide the conﬁdence of its prediction,
Bayesian neural network (BNN) can estimate the uncertainty of the prediction.
However, BNNs have not been widely used in practice due to the computational cost of predictive inference.
This prohibitive computational cost is a hindrance especially when processing stream data with low-latency.
To address this problem, we propose a novel model which approximate BNNs for data streams.
Instead of generating separate prediction for each data sample independently, this model estimates the increments of prediction for a new data sample from the previous predictions.
The computational cost of this model is almost the same as that of nonBayesian deep NNs.
Experiments including semantic segmentation on real-world data show that this model performs signiﬁcantly faster than BNNs, estimating uncertainty comparable to the results of BNNs.
I NTRODUCTION
While deterministic neural networks (DNNs) surpass human capability in some area in terms of prediction accuracy (He et al., 2015; Silver et al., 2016; Ardila et al., 2019), it could not estimate a trustworthy uncertainty of prediction.
Since the prediction can not be perfect and the misprediction might result in fatal consequences in areas such as medical analysis and autonomous vehicles control, estimating uncertainty as well as predictions will be crucial for the safer application of machine learning based systems.
Bayesian neural network (BNN), a neural network (NN) that uses probability distributions as weights, estimates not only predictive results but also uncertainties.
This allows computer systems to make better decisions by combining uncertainty with prediction.
Moreover, BNN can achieve high performance in a variety of ﬁelds, e.g. image recognition (Kendall et al., 2015; Kendall & Gal, 2017), language modeling (Fortunato et al., 2017), reinforcement learning (Kahn et al., 2017; Osband et al., 2018), meta-learning (Yoon et al., 2018;
Finn et al., 2018), and multi-task learning (Kendall et al., 2018), by exploiting uncertainty.
Although BNNs have these theoretical advantages, they have not been used as a practical tool.
The predictive inference speed has detained BNNs from wide applications.
BNN executes NN inference for dozens of samples from weight distributions.
Since sampling and multiple NN executions are difﬁcult to be parallelized, the inference execution takes an order of magnitude more time.
Particularly, this is a signiﬁcant barrier for processing data streams with low-latency.
Most time-varying data streams change continuously, and so do the predictions of BNNs.
Thus, we estimate prediction by calculating the increments between two consecutive results, instead of calculating the separate prediction for each input data.
This is equivalent to calculating the differentiation of the BNN’s prediction for arbitrary data, because the difference of prediction for the new data is the line integration of the gradient of prediction over the new data.
In this work, we propose a differentiable BNN (DBNN) inference with respect to an input data.
The prediction of the DBNN is given by a Monte Carlo (MC) estimator for distributions of data streams and weights.
We speed up the inference by approximating the distribution using histogram and calculating the gradient for this MC estimator.
We show that the time complexity of this model is nearly the same as that of deep DNNs.
We evaluate DBNN with semantic segmentation using road scene video sequences and the results show that DBNN has almost no degradation in computational performance compared to deep DNNs.
The uncertainty predicted by DBNN is comparable to that of
BNN in various situations
The main contributions of this work are as follows.
• We propose online codevector histogram that estimates the probability of a high-dimensional data stream.
Then, we show that this histogram can be used to obtain the MC gradient estimation.
• We propose differentiable Bayesian neural network inference with respect to input data as an approximation of Bayesian neural network inference for data streams.
DBNN uses the online codevector histogram to approximate probabilities.
This model is nonparametric and applied to trained BNN without signiﬁcant modiﬁcations.
• We theoretically and empirically show that the computational performance of DBNN is almost the same as that of deep DNNs.
B ACKGROUND AND R ELATED W ORK
BNNs are a state-of-the-art method to estimate predictive uncertainty while DNNs based approaches have been developed recently (Lakshminarayanan et al., 2017; Guo et al., 2017).
BNNs with probability distributions as weights produce probabilistic results.
To make prediction, BNNs sample from the weight probabilities and performs DNN for each sample.
In this section, we describe the details and challenges of BNNs inference process.
2. B AYESIAN N EURAL N ETWORK I NFERENCE
Suppose that p ( w ) is posterior probability of NN weights w and p ( y | x , w ) is the BNN model for input data x .
Then, the inference result of BNN is a predictive distribution:
[Equation]
(1) where x (cid:63) is observed input data vector and y is output vector.
For simplicity, BNNs are usually modeled to have a probability distribution with the mean of the prediction of DNN, e.g. in HernándezLobato & Adams (2015); Gal & Ghahramani (2016):
[Equation]
where NN ( · ) is prediction of DNN and τ is a given parameter.
(1) can be approximated using MC estimator:
[Equation]
where w (cid:63) ∼ p ( w ) and N w (cid:63) is the number of samples.
The expected value of the obtained predictive distribution is the predictive result of BNN and the variance is the predictive uncertainty.
The process of calculating the equation consists of two steps: sampling weights from p ( w ) , e.g. using Markov chain Monte Carlo (MCMC) (Neal et al., 2011; Hoffman & Gelman, 2014), and executing the DNN for each weight sample.
Since real-world data is large and practical NNs are deep, the repetitive computation of NNs is difﬁcult to be fully parallelized due to various problems, such as memory limitations of the GPU (Kendall & Gal, 2017), and the iterative computation of DNNs results in the decrease of computation speed.
MCMCs are slow and, despite recent achievements (Tran et al., 2018), it is challenging to achieve linearly scaling performance for multi-GPU MCMC.
In addition, it is difﬁcult to check for the convergence of MCMCs.
To mitigate these problems, we approximate the difference in the prediction for one new data as one NN calculation, and sample the weights from the posterior before testing.
Several sampling-free BNNs, e.g. Hernández-Lobato & Adams (2015); Wang et al. (2016); Hwang et al.
; Wu et al.
(2018), have been proposed recently.
The sampling-free BNNs predict results with only one forward pass.
These methods, however, have limitations.
First, sampling-free BNNs necessitate speciﬁc types of weight-distribution with parameters such as Gaussian distribution or exponential family instead of using MC estimator to approximate (1).
When a posterior has multimodality or skew, they can give inaccurate approximations.
Second, in many cases, sampling-free BNN uses a method other than variational inference to obtain a posterior because evidence lower bound (ELBO) is not amenable.
These constraints make it difﬁcult to apply sampling-free BNNs to deep NNs.
Therefore, we mainly consider sampling-based BNN for comparison in this paper.
2. B AYESIAN N EURAL N ETWORK G RADIENT E STIMATION
It is standard to use a gradient of MC estimator to optimize the loss function of BNN, e.g. ELBO.
Several direct and indirect methods are used to obtain the MC gradient estimation: reparametrization trick (Kingma & Welling, 2013; Blundell et al., 2015;
Fortunato et al., 2017), score function (Kleijnen & Rubinstein, 1996; Williams, 1992; Glynn, 1990), dropout (Gal & Ghahramani, 2016; McClure &
Kriegeskorte, 2016), batch normalization (Teye et al., 2018), expectation propagation (HernándezLobato & Adams, 2015), and smooth transformation for samples (Liu & Wang, 2016; Burda et al., 2015; Maddison et al., 2017; Naesseth et al., 2017; Le et al., 2017; van den Oord et al., 2017).
However, it is not appropriate to use these techniques to calculate a gradient of MC estimator
L = E p ( x ) (cid:2) f
( x ) (cid:3) for arbitrary functions f ( · ) when a data stream is non-stationary and p ( x ) , the distribution of the data stream, is time-variant.
For example, the reparameterization trick, one of the most widely used methods for MC gradient estimation, separates the probability p ( x ) into a deterministic function h ( · ) and an invariant probability p ( (cid:15) ) , i.e., x = h ( (cid:15) ) and ∂ x p ( (cid:15) )
= 0 .
Then, the L is expectation over the invariant distribution, i.e., L = E p ( (cid:15) ) (cid:2)
f
( x ) (cid:3) , and the approximated L becomes differentiable, i.e., ∂ x L = E p ( (cid:15) ) (cid:2) ∂
x f ( x ) (cid:3) .
To use this technique, we need to adapt h ( · ) to the non-stationary data stream.
If h ( · ) is modeled in an NN, we may need to continuously train this NN on the data stream; this leads to a decrease in computational performance.
To alleviate this problem, we use histogram to estimate probability with nearest neighbor search, instead of using an NN-based generative model.
In addition, the histogram is a linear sum of disjoint bins that is analytically useful while simplifying calculations.
O NLINE C ODEVECTOR H ISTOGRAM
Vector quantization was introduced in Gersho (1982); Gray (1984) in order to compress a probability distribution to a dozens of samples called codevectors.
Kotani et al.
(2002) showed that the histogram of codevectors—codevectors augmented with counts—can effectively represent the features of face image dataset.
Xu et al.
(2012); Frezza-Buet (2014); Ghesmoune et al. (2016) have proposed algorithms that changes codevectors depending on the data stream.
We introduce online codevector histogram (OCH) to estimate the probability distribution of non-stationary data stream as well as dataset.
OCH is a probabilistic algorithm that changes not only codevectors but their counts and calculates the difference in distribution as the data stream changes, at low computational cost.
OCH can add a new codevector and delete old ones, while counting the matches of each codevector for the past data stream.
OCH maps the input vector to the codevector using nearest neighbor search.
It is a high-dimensional histogram where the Voronoi diagram is the boundary and a codevector represents the corresponding bin.
DBNN uses OCH to approximate the probability distributions of input and output vector data streams.
Algorithm 1 shows how OCH operates in three steps.
(a) Given a new input vector, OCH ﬁnds the nearest codevector and increases its count.
Then, it divides the corresponding bin by inserting the input vector as a new codevector with probability proportional to the count.
(b) OCH decrease all counts by the same rate to reduce the contribution of old data.
(c) To keep the number of codevectors small, OCH deletes codevectors with probability in inverse proportion to its counts.
OCH takes three hyperparameters: K , λ , and φ .
K is the number of codevectors kept in OCH on average.
λ determines how fast OCH depreciates old counts with the rate of γ .
φ with counts of each bin regulates the probability of adding and deleting codevectors.
Nearest neighbor search is the most computationally intensive step in alg.
To lower the computational complexity, we use locality-sensitive hashing with stable distribution, h = (cid:98) ( a · x (cid:63) + b ) / w (cid:99) for a vector a and scalars b and w .
It requires up to log K hashes for precise search.
The upper bound of the computational complexity is O (dim( x ) log K ) .
This is faster than continuous training of NN-based generative models for the data stream changing over time.
input : input vector x (cid:63) , count of input vector n (cid:63) , OCH =
{ ( c i , n i ) } where c i is codevector and n i is its count, hyperparameters K , λ , and φ output : updated OCH 1 c
i ← Search the nearest codevector to x (cid:63) in OCH 2 n i ← n i + n (cid:63) where n i is count of c i 3 p ∼ Bernoulli (cid:0) σ ( π i − ¯ π + φ ) (cid:1) where σ ( · ) is sigmoid, N = (cid:80)
i n i , π i = n i /N , and ¯ π = 1 /K 4 if p = 1 then
OCH ← OCH ∪ { ( x (cid:63) , (1 − γ ) · n i ) } where γ = e − λ / N n i ← γ · n i 7 forall
OCH do n j ←
· n j 9 forall ( c k , n k ) ∈ OCH do q ∼ Bernoulli (cid:0)
(¯ π − π k + φ ) · ¯ π (cid:1) if q = 1 then
OCH ← OCH \ { ( c k , n k ) }
Meanwhile, OCH approximates probability distribution p ( x ) as: p
[Equation]
(4) where N = (cid:80) i n i , π i = n i /N , and V ( x | c ) is the bin or neighborhood of c with (cid:82) V ( x
| c ) d x = 1 .
The right-hand-side of (4) is clearly differentiable with respect to π j , while a set of random samples from p ( x ) is not differentiable.
Given a new input data, OCH changes only one count of the nearest codevector.
Therefore, if n (cid:63) (cid:28) N , the difference of OCH for a new input vector is approximately
[Equation]
where c (cid:63) is the nearest codevector to the input vector, α = δn / N , and δn is the difference of the count of c (cid:63) .
When OCH creates a new codevector, δn is n (cid:63) .
Here, we used the fact that codevectors are invariant for input data, so V ( x | c i ) is also invariant.
This property is useful when the linear operator g ( · ) is applied to p ( x ) .
By deﬁnition of linear operator, g ( p ( x )) = (cid:80) i
π
i g
( V ( x | c i )) , then its difference is δg ( p ( x ))
= (cid:80) i δπ i g
( V ( x | c i ))
=
αg ( V ( x | c (cid:63) )) .
D IFFERENTIABLE B AYESIAN N EURAL N ETWORK I NFERENCE
When a data stream S
= { . . .
, x (cid:63) } is given, where each subsequent data sample changes continuously, the prediction of a NN for S also changes continuously, i.e., p ( y ) → p ( y )
+ δp ( y ) , because a NN (with continuous activation) is a homeomorphism.
We show that δp ( y ) is approximately proportional to the prediction of a NN augmented with OCH to the input and output for the recent data x (cid:63) .
Figureshows the structure of DBNN, where DNN is augmented with OCHs as distribution estimator units of input and output stream.
DBNN calculates predictive uncertainty as well as the predictive result using the weighted ensemble of output codevectors.
4. D IFFERENTIATION OF B AYESIAN N EURAL N ETWORK I NFERENCE AS D IFFERENCE OF P REDICTION
It is challenging to calculate the difference of (1) for a new data x (cid:63) from S .
The differentiation of p ( y | x (cid:63) , w ) with respect to x (cid:63) is analytically intractable, because the samples from the probability of the data stream p ( x |S ) are discrete, although p ( x |S ) varies continuously.
In other words, ∂ x (cid:63) p ( y | x (cid:63) , w ) = (cid:82) p ( y | x , w ) ∂ x (cid:63) δ ( x − x (cid:63) )
d x where δ ( · ) is the delta function, and the delta function is not differentiable.
To address this issue, DBNN smoothens the delta function of the data
[Figure]
mean variance
DNN OCH Z ⋮ w w w
[Figure]
OCH Y predictions ⋮
[Figure]
sample δ ( x − x (cid:63) ) to the probability of the data stream p ( x |S ) as follows:
[Equation]
where z = ( x , w ) and p ( z |S ) =
p ( x |S ) p ( w ) .
p ( x |S ) is separately learned with respect to the data stream S by another model and is called data uncertainty .
p ( w ) is the posterior distribution obtained from BNN training and is called model uncertainty .
DBNN inference decouples data (stream) from the model.
Instead, it joins the probability of data and the probability of the NN weights.
When the most recent data sample is considered instead of the data stream, i.e., p ( x |S ) =
δ ( x − x (cid:63) ) , DBNN inference is reduced to BNN inference as mentioned before.
To calculate (8), we use OCH and (4) to represent p ( z |S ) , and call it OCH Z :
[Equation]
where π i is weight of codevector c i and proportional to the count of the codevector.
V ( z | c i ) is neighborhood of c i .
Then, δp ( y |S ) can be approximated by changing the weight of the codevector nearest to x (cid:63) as:
[Equation]
according to the (6).
This means that (cid:82) p ( y | z ) V ( z | c i )
d z is invariant with respect to the change of S , i.e., δ (cid:2) (cid:82) p ( y | z ) V ( z | c i )
d z (cid:3)
= 0 , since histogram only changes its counts, not the codevectors.
We approximate V ( z | c (cid:63) ) as delta-function distribution at c (cid:63) , i.e., V ( z | c (cid:63) )
(cid:39) δ ( z − c (cid:63) ) :
[Equation]
It is safe to assume that p ( y | c (cid:63) ) is dominantly distributed near NN ( c
[Equation]
) , i.e., p ( y | c (cid:63) ) (cid:39)
V (cid:0) y | NN ( c (cid:63) ) (cid:1) , because the expected value of y should be equal to NN ( c (cid:63) ) :
[Equation]
(13) where NN ( c (cid:63) ) is prediction of DNN for c (cid:63) .
This result is equal to the changes of OCH when we update
OCH for data, as (6) expresses.
In conclusion, the difference of the DBNN prediction contributes only to the neighborhood of the DNN prediction for the codevector in OCH Z approximately.
Thus, given the OCH Y representing p ( y |S ) , p ( y |S )
+ δp ( y |S ) can be approximated by updating the OCH Y for the prediction of DNN.
=
NN ( c
I MPLEMENTATION OF D IFFERENTIABLE B AYESIAN N EURAL N ETWORK I NFERENCE
DBNN is composed of three stages and Algorithm 2 describes the DBNN inference process in detail as follows: (a) First, to estimate the probability of the input data stream, the algorithm updates OCH Z for z (cid:63) =
( x (cid:63) , w (cid:63) ) where x (cid:63) is an input data and w (cid:63) is a random sample from given approximated posterior distribution OCH Z which represents the given p ( w ) .
OCH W consists of weights sampled from trained p ( w ) using MCMC.
OCH Z and OCH Y are initialized to empty sets.
(b) Second, if
OCH Z generated a new codevector in OCH Z , DBNN generates the prediction for the new codevector using DNN and keeps the prediction in a cache table.
As a result, the cache table contains the results of DNN corresponding to all codevectors in OCH Z .
(c) Third, DBNN ﬁnds the nearest neighbor codevector to x (cid:63) in OCH Z , looks up the corresponding prediction in the cache table and update
OCH Y for the prediction for the codevector.
In conclusion, OCH Z approximating p ( z |S ) consists of recent data and weights from posterior.
OCH Y approximating p ( y |S ) consists of recent predictions.
DBNN calculates the difference of prediction for a new data sample from the previous predictions.
In the process, it executes DNN once if necessary, which is computationally expensive, in contrast to
BNNs’ repetitive execution DNNs.
Furthermore, DBNN sometimes does not execute DNN, but only updates OCH Z and OCH Y for the input and output vectors, by updating the counts and estimates prediction and uncertainty based on cached results.
The dominant part of the computational cost of updating OCH Z is the inner product of a given input vector, i.e., a · z , for nearest neighbor search.
Given z =
( x , w ) and a = ( a 0 , a 1 ) , a · z = a 0 · x + a 1 · w holds.
The codevector samples w from
OCH W is ﬁxed and ﬁnite, and DBNN caches all a 1 · w .
Therefore, the average computational cost on a 1 · w is minuscule, when updating OCH Z .
In conclusion, the upper bound of the computational complexity of DBNN inference is O (dim( x ) log K Z + NN ( · ) + dim( y ) log K Y ) where O ( NN ( · )) is computational complexity of NN and K Z and K Y are hyperparameters K of OCH Z and OCH Y , respectively.
Modern deep NN performs vector operations dozens of times, if not hundreds, so the computation time of OCH is very small compared to DNN execution.
DBNN uses the ﬂexible parametric probability estimator OCH to represent the distributions of input and output vector streams.
Unlike the most BNNs that depend on a parameterized model described in (2), DBNN does not depend on the speciﬁc model.
OCH represents both continuous and discrete vector spaces, and so does DBNN.
If DNN and posterior are given, we can easily convert them to
DBNN without signiﬁcant modiﬁcations: just add OCH to the input and output of a DNN to estimate the probability of the input and output vector streams.
E XPERIMENTS
Although DBNN uses assumptions that seem reasonable, it is necessary to measure empirical performances to show that this assumption works for real-world problems.
This section evaluates the performance of DBNN in three set of experiments.
The ﬁrst experiment visualizes the characteristics of DBNN by performing simple linear regression on synthetic data.
The second experiment classiﬁes various real-world multivariate datasets using shallow and narrow NNs.
This experiment shows the
2 x 0.
w  2 x    y (a) DNN  2 x 0.
w  2 x    y (b) MU  2 x 0.
w  2 x    y (c) DU  2 x 0.
w  2 x    y (d) DBNN 0 2
6 t 0.
[ y ]
DNNMUDUDBNN
2 4 6 t 0.2 0.
s t d
[ y ]
DNNMUDUDBNN (e) v = 0 .0 20 40 60 t 0.
[ y ]
DNNMUDUDBNN 0 20 40 60 t 0.2 0.
s t d
[ y ]
DNNMUDUDBNN (f) v = 0 .
difference between computational and predictive performance of BNN and DBNN under various conditions when converting BNN to DBNN using small NN.
The third experiment performs semantic segmentation on real-world video sequences.
This experiment compares the performances of DBNN with other baselines when using a deep and wide modern NNs in practical situation.
We use the following four methods as baselines:
• DNN.
Let Softmax ( y (cid:48) ) be probability of NN where y (cid:48) is DNN logits.
It is easy to implement, but it differs from the actual classiﬁcation probability when the NN is deepened, broadened, and regularized well (Guo et al., 2017).
• Model Uncertainty (MU).
BNN is referred to as MU in this section since it introduces model uncertainty to NN.
MU have to calculate (3) to predict the result.
It is difﬁcult to analytically determine the number of samples that the prediction converges.
Instead, in section 5.1 and section 5.2, we experimentally set the number of samples from posterior to 30 so that the accuracy and the negative log-likelihood (NLL) converge.
In section 5.3, MU containing MC dropout (Gal & Ghahramani, 2016) layers predicts results with 30 forward passes.
Appendix E shows the performance of the MU for different numbers of forward passes.
• Data Uncertainty (DU).
DU is a method that assigns only data uncertainty to DNN by adding OCHs to input and output of trained DNN.
In other words, it uses one weight instead of weight distribution in the DBNN, i.e., (8) with p ( w ) =
δ ( w − w (cid:63) ) .
This method shows the effect of OCH on prediction when it is applied to the NN.
Also, it shows that DNNs modiﬁed using OCH can achieve higher uncertainty than vanilla DNNs in semantic segmentation experiment.
• DBNN.
DBNN takes both data uncertainty and model uncertainty.
DBNN adds OCH to the input and output of trained BNN.
In section 5.1 and section 5.2, the approximated posterior distribution of this method OCH W consists of 30 samples from the BNN’s posterior.
5. S IMPLE L INEAR R EGRESSION
A neural network with a linear activation function is a linear regression model y =
w 0 x + w 1 .
The posterior is given by p ( w 0 ) =
N (1 . 0 , 0 .
02 2 ) and p ( w 1 ) =
N (0 . 0 , 0 .
The distribution of time-varying input data streams is given by p ( x | t )
=
δ ( x − vt ) where t is integer timestamp from t = − 10 .
v is 0 .
3 and 0 .
The top of ﬁgs.
2a to 2d are approximated distributions of input and weight samples p ( x, w 0 ) and bottom are approximated distributions of output samples with data p ( x, y ) at t = 0 .
w 1 is omitted from w in these ﬁgures, but it behaves like w 0 .
[Table]
To predict result, DNN uses point-estimated weight and data x = 0 , which is the data at t = 0 .
MU uses weight distribution, instead of point-estimated weight, and data x = 0 .
DU uses point-estimated weight, which is the same as DNN.
However, since DU contains OCHs, it uses not only the most recent data ( x = 0 at t = 0 ) but also past data ( x < 0 when t < 0 ).
As a result, DU predicts the distribution of y .
DBNN predicts the distribution of y using both weight distribution and data from the past to now.
DBNN without distribution of x is equivalent to MU, and DBNN without distribution of w is equivalent to DU.
The ﬁgs. 2e to 2f show expected value and standard deviation of DBNN prediction on time.
First, as shown in top of ﬁg. 2e, the result of DBNN lags behind since DBNN smoothens its prediction with respect to time.
1 Second, unlike MU uses dozens of samples, DBNN uses a small number of samples, so the error is relatively large.
Third, as shown in bottom of ﬁg. 2e, the DBNN prediction is under-conﬁdent because DBNN considers not only MU but also DU.
Fourth, as shown in the comparison of ﬁg. 2e and ﬁg. 2f, DBNN converges to BNN and DU converges to DNN as the input data stream changes more slowly.
5. C LASSIFICATION ON M ULTIVARIATE D ATASETS
A classiﬁcation experiment is designed to compare computational and predictive performance changes when converting BNNs to DBNNs using shallow and narrow NNs in various situations.
In this experiment, we use three time-series real-world datasets with input dimensions between 4 and 8.
Localization and EMG dataset have relatively large number of classes, eleven and eight respectively while Occupancy dataset has only two classes.
The NNs consist of 2 fully-connected hidden layers with 50 units.
The distributions of weights are optimized using variational inference.
See appendix A for more information about experimental settings and datasets.
Computational Performance.
Execution times for one batch of MU and DBNN are 26.7 ± 10.6ms and 19.5 ±
10.0ms, respectively.
MU parallelizes execution by using a batch size of 30.
This result shows that DBNN is 37% faster than MU even though MU is parallelized and DBNN uses additional
OCHs.
DBNN improves the computational performance of DBNN by sampling the weights from the posterior before testing.
As NN gets deeper and larger, the execution time of the DBNN will be reduced compared to the MU, because the larger the NN, the greater the burden of sampling weights.
Predictive Performance.
As shown in Table 1, we compared MU and DBNN in terms of the rootmean-square error (RMSE), RMSE for predictive results with conﬁdence above 90% (RMSE-90), negative log-likelihood (NLL), and the percentage of predictive results with conﬁdence above 90% (Cov-90) of the MU and DBNN for three datasets.
DBNN is more accurate than MU and predicts uncertainty better than MU on Localization and EMG datasets.
DBNN predicts less accurately than MU on Occupancy dataset with comparable uncertainties.
The conﬁdence of DBNN is always less than or equal to that of MU. Overall, DBNN provides predictive performance comparable to BNN on various datasets.
[Table]
Semantic segmentation experiment shows the computational and predictive performance of DBNN with a modern deep NN in practical situation.
We use CamVid dataset (Brostow et al., 2009) consisting of 30 frame-per-second (fps) video sequences of real-world day and dusk road scenes.
We use the
U-Net (Ronneberger et al., 2015) as the backbone architecture.
Bayesian U-Net, similar to Kendall et al.
(2015), contains MC dropout (Gal & Ghahramani, 2016) layers.
For more information about experimental settings, see appendix A.
Computational Performance.
The throughput (Thr) column of table 2 shows the number of video frames processed by each model per second.
This table shows that DNN takes 162 ± 12ms on average to process one frame.
In comparison, DBNN takes 191 ± 70ms on average to process one frame, which is only 17% higher than DNN.
According to the difference between the execution time of DNN and DU, the average execution time of one OCH is 14ms, which is only 7% of the total.
Besides, due to GPU memory limitations, the batch size of MU is limited to 1.
Thus, the MU predicts results for 30 batches of size 1.
In conclusion, the execution time of MU is 5291 ± 69ms, which is 32 × higher than that of DNN, and 27 × higher than that of DBNN.
Moreover, DBNN can increase throughput at the expense of accuracy by adjusting a hyperparameter.
See appendix C for more information.
Predictive Performance.
The Acc to Cov-90 columns of the table 2 show the quantitative comparison of the predictive performance for each method.
We measure global pixel accuracy (Acc), mean intersection over unit (IoU), and negative log-likelihood (NLL).
At the same time, we selected only those pixels with conﬁdence greater than 90% and measure the accuracy (Acc-90), i.e., p ( accurate | conﬁdent ) , and IoU (IoU-90).
We also measure the probability that the pixel is not 90% conﬁdent pixel when prediction is incorrect (Unc-90), i.e., p ( unconﬁdent | inaccurate ) , and the percentage of pixels with 90% conﬁdence (Cov-90).
According to this table, the accuracies of DU and DBNN are 0.46% and 0.69% lower than that of
DNN and MU, respectively.
For pixels with ninety-percent or higher conﬁdence, DNN, MU, DU, and
DBNN predicts with 3.8%, 7.6%, 7.1%, and 7.5% higher accuracy, respectively, compared to the accuracy for all pixels.
MU, DU and DBNN improves the accuracy more than DNN for conﬁdent pixels, which means that MU and DBNN estimates high uncertainty for misclassiﬁed region.
IoUs for certain pixels increase by 6.8%, 14%, 10%, 16%, respectively, compared to the IoUs for all pixels, which shows the same trend as in accuracy.
As shown in this results, DNN is the most improper way to distinguish uncertain pixels because it has the least performance improvement compared to other methods for pixels with high conﬁdence.
On the other hand, MU has the highest performance improvement compared to DNN when considering all pixels, and it also has the highest performance for certain pixels.
DBNN has improved performance for certain pixels, similar to MU.
The predictive performance of DU for all pixels is similar to the predictive performance of DNN, but for certain pixels, the performance is much better than that of
DNN.
See appendix D for more information on semantic segmentation experiment.
With a small number of forward passes, BNN can reduce computational overhead at the expense of predictive performance.
To achieve equivalent performance to DBNN in uncertainty measures, MU requiresforward passes.
See appendix E for more details.
C ONCLUSION
We present a differentiable BNN (DBNN) inference with respect to input data, which is a novel approximation of BNN inference, to improve the computational performance of BNN inference for data streams.
The derivative of DBNN predictive inference with respect to input data derives the increment of prediction when one data is newly given from the data stream.
However, the inference of vanilla BNN cannot be differentiated with respect to data.
To address this issue, DBNN introduce a new term that is the probability of data streams in the BNN inference.
Then, it approximate its prediction with a histogram for high-dimensional vector streams.
Consequently, the DBNN inference executes DNN only once to calculate the prediction changed by a new data from data streams.
This results in an order of magnitude times improvement in computational performance compared to deep
BNN.
Experiments with semantic segmentation using real-world datasets show that the computational performance of DBNN is almost the same as that of DNN, and uncertainty is comparable to that of
BNN
A E XPERIMENTAL S ETUP AND D ATASETS
We conduct all the experiments with the Intel Xeon W-2123 Processor, 32GB memory, and a single
GeForce RTX
Ti.
NN models are implemented in TensorFlow (Abadi et al., 2016).
Classiﬁcation on Multivariate Datasets.
In section 5.2, we train BNNs with Adam with a constant learning rate of 0.001 using ELBO loss.
We set the batch size to 30.
The BNN consist of 2 fullyconnected layers with 50 units using Flipout estimator (Wen et al., 2018).
We set hyperparameters K , λ , and σ ( φ ) to 10, 1.3, and 1.0, respectively, in all OCHs of DBNN; this is the same conﬁguration in the semantic segmentation experiment.
We use the following time-sequential real-world dataset for classiﬁcation in section 5.2: Localization
Data for Person Activity Dataset (Kaluža et al., 2010), Occupancy Detection Dataset (Candanedo &
Feldheim, 2016), and EMG Data for Gestures Dataset (Lobov et al., 2018).
If there are no distinction between the test set and the training set, NNs use 90% of the dataset for training and the rest for testing.
NNs test the datasets in time-sequential order.
See table 3 for more information about datasets.
Semantic Segmentation.
In section 5.3 and appendix D, NNs are trained using categorical crossentropy loss, with RMSprop with a constant learning rate of 0.0001 and discounting factor of 0.995.
Batch size is limited to 1 because of memory limitations.
In these experiments, we use U-Net (Ronneberger et al., 2015) based on VGG-16 (Simonyan & Zisserman, 2014).
Bayesian U-Net, similar to Kendall et al.
(2015), contains six MC dropout (Gal & Ghahramani, 2016) layers behind the layers that receives the smallest input vector sizes.
The overhead of dropout layers are negligible and DBNN always uses a new dropout every time it predicts a result.
To optimize the predictive performances, we set hyperparameters K , λ , and σ ( φ ) to 10, 1.3, and 1.0, respectively, in all OCHs of DU and DBNN.
See appendix C for the change in NLL with hyperparameters.
To improve computational performance, OCH Y executes nearest neighbor search for arg max of y with respect to the classes.
We use CamVid dataset (Brostow et al., 2009) and CityScape dataset (Cordts et al., 2016) in the semantic segmentation experiments.
We feed sequential images to DBNN and DU before evaluation since DU and DBNN use multiple previous images.
For the CamVid dataset, we resized images to 720 × 960 pixel bilinearly from 360 × 480 pixels.
Since some test sets do not have previous sequences, we use only 70 of 168 test sets in the CamVid dataset.
For CityScape dataset, we use validation set, which was not used during training, as test set because only the video sequence corresponding to the validation set has been disclosed.
See table 3 for more information about datasets.
B C OMPUTATIONAL P ERFORMANCE FOR N EURAL N ETWORK D EPTH
Figure 3 shows a trend that the execution time of MU and DBNN increases as hidden layers get deeper without changing the input and output dimensions.
In this case, as in section 5.2, NN consists ofunits of fully-connected hidden layers, with input and output dimensions of 4 and 11, respectively.
MU and DBNN use batches of size 30 and 1, respectively, but execution time is the same even if MU uses batch size of 1.
According to this ﬁgure, when the number of hidden layers is 0, the execution time of DBNN is 15.1 ± 6.7ms, which is 4.6ms slower than MU.
This is because DBNN uses two additional OCHs compared to MU.
However, unlike MU, DBNN does not sample the weight in the
[Figure] [Figure]
posterior when it predicts the result. Therefore, the increase of execution time of DBNN per layer is lower than that of MU.
As a result, the execution time of the MU increases by 8.6ms while that of
DBNN increases by 3.5ms when one layer is added.
When the number of hidden layers is 10, the execution time of MU and DBNN is 98.9 ± 20.1ms and 49.0 ± 23.6ms, respectively—DBNN is 2.02 × faster than MU.
C P REDICTIVE P ERFORMANCE FOR H YPERPARAMETERS
DBNN has three hyperparameters; K , λ , and σ ( φ ) , in the input and output OCHs.
The ﬁrst to third ﬁgures in ﬁg. 4 shows how NLL changes over the variuos hyperparameters of the output OCH in semantic segmentation.
As shown in this ﬁgure, as K increases, NLL decreases, and the NLL stagnates when K goes over 15. The lower the λ , the lower the NLL, until 0.7.
The higher the σ ( φ ) , the lower the NLL.
The effect of hyperparameters of the input OCH is the similar trend as that of the output OCH.
With high K and and low λ , DBNN maintains more codevectors—recent frames in video sequence and obtains more accurate model uncertainty using more codevectors.
As a result, at high K and low λ , DBNN exhibits lower NLL. When K and λ get lower than a certain point ( λ = 0 . 7 in this example), however, the data uncertainty is too high to estimate results.
NLL is minimized when model uncertainty and data uncertainty are balanced.
On the other hand, if σ ( φ ) is low, DBNN mostly adjusts the weights of codevectors and occasionally adds new codevectors.
This results in the DBNN becoming inaccurate, but the calculation is faster.
Accuracy Throughput Trade-off.
If σ ( φ ) is less than 1.0, DBNN occasionally adds a new codevector to the input OCH and executes NN, at the rate of σ ( φ ) on average.
Since NN occupies the most of the execution time of DBNN, throughput decreases when σ ( φ ) decreases.
The fourth ﬁgure in ﬁg. 4 shows the changes in throughput and NLL as σ ( φ ) changes.
According to this ﬁgure, DBNN can increase throughput up to 12fps when it achieves same NLL as DNN.
In this case, DBNN is 2 ×
[Table] faster than DNN.
Conversely, DBNN can update a batch of two or more sizes instead of one.
In this case, throughput of DBNN decreases but NLL increases (not shown in this ﬁgure).
D E XTENDED I NFORMATIONS OF S EMANTIC S EGMENTATION E XPERIMENT
Section 5.3 shows the performance with semantic segmentation on the CamVid dataset.
This section shows the performance on the CityScape dataset, another real-world road scene video sequence.
Furthermore, we analyze the predictive performances of DNN, MU, DU, and DBNN in more detail. Predictive Performance of Models for Datasets. Table 4 shows the computational and predictive performance of various methods on the CityScape dataset. We measure the followings as in section 5.3: throughput (Thr), global pixel accuracy (Acc), accuracy for 90% certain pixels (Acc-90), the probability that the pixel is not 90% certain pixel when prediction is incorrect (Unc-90), mean intersection over unit (IoU), IoU for 90% certain pixels (IoU-90), negative log-likelihood (NLL), and the ratio of 90% certain pixels (Cov-90).
Table 4 implies the followings: First, The computational performance of DNN and DBNN are relatively comparable, and that of DBNN is signiﬁcantly higher than that of MU.
Second, the predictive performances of DU and DBNN for all pixels is lower than that of DNN and BNN, respectively.
Third, predictive performances for certain pixels and other measures for uncertainties of
MU and DBNN are comparable.
These results are consistent with the result of section 5.3.
Extended Analysis of Conﬁdence and Accuracy.
Figure 5 shows conﬁdence histograms (Guo et al., 2017) and reliability diagrams (DeGroot & Fienberg, 1983; Niculescu-Mizil & Caruana, 2005;
Naeini et al., 2015; Guo et al., 2017) of U-Net on the CamVid dataset.
A conﬁdence histogram shows proportion of samples for conﬁdence as a histogram.
The reliability diagram is a histogram of accuracy for conﬁdence.
We also measure the Expected Calibration Error (ECE) (Naeini et al., 2015;
Guo et al., 2017), which is deﬁned as follows for the reliability diagram:
[Equation]
(14) where | B i | is the number of samples in bin B i , M = (cid:80) i | B i | , acc ( B i ) is mean accuracy of samples in B i , and conf ( B i ) is mean conﬁdence of samples in B i .
ECE stands for the difference between conﬁdence and accuracy.
ECEs of DNN, MU, DU, and DBNN are 7.43%, 4.27%, 5.86%, and 5.13%, respectively.
As shown in ﬁg. 5a, DNN is overconﬁdent and miscalibrated; its conﬁdence is generally high and there is signiﬁcant discrepancy between conﬁdence and accuracy.
On the other hand, MU is underconﬁdent and well-calibrated.
DBNN is more under-conﬁdent than MU, and the conﬁdence of DBNN is calibrated relatively comparable to that of MU.
Also, DU calibrates the conﬁdence of DNN.
In conclusion, the conﬁdence of DBNN is well-calibrated, whild the conﬁdence of the DNN is relatively unreliabl.
These are consistent with the results of the predictive performance.
Qualitative Results.
Figure 6 shows the qualitative comparison of the predictions for each method.
According to this ﬁgure, DNN is overconﬁdent, i.e., conﬁdence is generally high, and uncertainty is mostly distributed at the boundaries of the classiﬁed chunks.
Even when DNN generates wrong prediction, the conﬁdence level is very high.
The uncertainty of MU is distributed on the boundaries
[Figure]
[Figure]
[Table]
as in the case of the DNN [Table] but is also distributed in the misclassiﬁed areas.
The uncertainty of DU differs from that of DNN.
First, DU is under-conﬁdent compared to DNN.
Second, although DU does not identify all of the misclassiﬁcations compared to MU, it sometimes estimates high uncertainty in the misclassiﬁed areas.
The uncertainty of DBNN is similar to the MU as we expected
[Table]
E P ERFORMANCE OF BNN FOR N UMBER OF F ORWARD P ASSES
BNNs, i.e., MUs, in section 5.3 and appendix D contain MC dropout (Gal & Ghahramani, 2016) layers and require multiple forward passes to predict result.
Table 5 shows the computational and predictive performance of the Bayesian U-Net for the number of forward passes on the CamVid dataset.
According to table 5, the predictive performance of MU improves up to 30 forward passes.
However, there is little difference between the predictive performance for 30 forward passes and for 50 forward passes.
MU requires at least 30 forward passes to get the best predictive performance, so we execute 30 predictive inference in section 5.3 and appendix D.
Meanwhile, to achieve the same uncertainty measures as DBNN, MU requires 10 forward passes.
In particular, Unc-90 of DBNN has always outperformed that of MU.
For the improvement rate from
Acc and IoU to Acc-90 and IoU-90, the uncertainty of DBNN is the same as that of MU withforward passes.