E XPLAINING I MAGE C LASSIFIERS BY C OUNTERFACTUAL G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
When a black-box classiﬁer processes an input to render a prediction, which input features are relevant and why?
We propose to answer this question by efﬁciently marginalizing over the universe of plausible alternative values for a subset of features by conditioning a generative model of the input distribution on the remaining features.
In contrast with recent approaches that compute alternative feature values ad-hoc—generating counterfactual inputs far from the natural data distribution— our model-agnostic method produces realistic explanations, generating plausible inputs that either preserve or alter the classiﬁcation conﬁdence.
When applied to image classiﬁcation, our method produces more compact and relevant per-feature saliency assignment, with fewer artifacts compared to previous methods.
I NTRODUCTION
Neural networks achieve state-of-the-art performance in many domains, but their decisions are difﬁcult to interpret.
Saliency maps are a tool for interpreting neural network classiﬁcation that, given a particular input example and output class, score the relevance of each input dimension to the resulting classiﬁcation.
Fong & Vedaldi (2017) and Dabkowski & Gal (2017) cast saliency computation an optimization problem informally described by the following question: which inputs, when replaced by an uninformative reference value, maximally change the classiﬁer output?
Because these methods use heuristic reference values, e.g. blurred input (Fong & Vedaldi, 2017) or random colors (Dabkowski & Gal, 2017), they ignore the context of the surrounding pixels, often producing unnatural in-ﬁlled images (Figure 2).
If we think of a saliency map as interrogating the neural network classiﬁer, these approaches have to deal with a somewhat unusual question of how the classiﬁer responds to images outside of its training distribution.
To encourage explanations that are consistent with the data distribution, we modify the question at hand: which region, when replaced by plausible alternative values, would maximally change classiﬁer output?
We marginalize out the masked region, conditioning the generative model on the non-masked parts of the image to sample counterfactual inputs that either change or preserve classiﬁer behavior.
In this paper we provide a new model-agnostic framework for visualizing feature importance from any differentiable classiﬁer, based on variational Bernoulli dropout (Gal & Ghahramani, 2016).
By leveraging a powerful in-ﬁlling conditional generative model we produce saliency maps on ImageNet that identify relevant and concentrated pixels better than existing methods.
R ELATED WORK
Gradient-based approaches (Simonyan et al., 2013; Springenberg et al., 2014; Zhang et al., 2016;
Selvaraju et al., 2016) derive a saliency map for a given input example and class target by computing the gradient of the classiﬁer output with respect to each component (e.g., pixel) of the input.
The reliance on the local gradient information induces a bias due to gradient saturation or discontinuity in the DNN activations (Shrikumar et al., 2017).
Reference-based approaches analyze the sensitivity of classiﬁer outputs to the substitution of certain inputs/pixels with an uninformative reference value.
Shrikumar et al.
(2017) linearly approximates this change in classiﬁer output using an algorithm resembling backpropagation.
This method is efﬁcient and addresses gradient discontinuity, but ignores nonlinear interactions between inputs.
Chen et al.
(2018) optimizes a variational bound on the mutual information between a subset of inputs and the target, using a variational family that sets
[Figure]
Heuristics
Generative Methods
[Figure]
Mean
Blur
Random
Local
VAE
CA
[Figure]
I n - ﬁ ll
., Contextual Attention GAN (CA) (Yu et al., 2018)—ameliorate in-ﬁll artifacts, making explanations more plausible under the data distribution.
input features outside the chosen subset to zero.
In both cases, the choice of background value as reference limits applicability to simple image domains with static background like MNIST.
Zintgraf et al.
(2017) computes the saliency of a pixel (or image patch) by treating it as unobserved and marginalizing it out, then measuring the change in classiﬁcation outcome.
This approach is similar in spirit to ours.
The key difference is that where Zintgraf et al.
(2017) iteratively execute this computation for each region, we leverage a variational Bernoulli distribution to efﬁciently search for optimal solution while encouraging sparsity.
This reduces computational complexity and allows us to model the interaction between disjoint regions of input space.
Fong & Vedaldi (2017) computes saliency by optimizing the change in classiﬁer outputs with respect to a perturbed input, expressed as the pixel-wise convex combination of the original input with a reference image.
They offer three heuristics for choosing the reference: mean input pixel value (typically gray), Gaussian noise, and blurred input.
Dabkowski & Gal (2017) amortize the cost of estimating these perturbations by training an auxiliary neural network.
P ROPOSED METHOD
Dabkowski & Gal (2017) propose two objectives for computing the saliency map:
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure]
• Smallest Deletion Region (SDR) considers a saliency map as an answer to the question: What is the smallest input region that could be removed and swapped with alternative reference values in order to minimize the classiﬁcation score?
Smallest Supporting Region (SSR) instead poses the question: What is the smallest input region that could substituted into a ﬁxed reference input in order to maximize the classiﬁcation score?
Solving these optimization problems (which we formalize below) involves a search over input masks, and necessitates reference values to be substituted inside (SDR) or outside (SSR) the masked region.
These values were previously chosen heuristically, e g., mean pixel value per channel.
We instead consider inputs inside (SDR) or outside (SSR) the masked region as unobserved variables to be marginalized efﬁciently by sampling from a strong conditional generative model 1 .
We describe our approach for an image application where the input comprises pixels, but our method is more broadly applicable to any domain where the classiﬁer is differentiable.
Consider an input image x , class c , and classiﬁer with output distribution p M ( c | x ) .
Denote by r a subset of the input pixels that implies a partition of the input x = x r ∪ x \ r .
We refer to r as a region, although it may be disjoint.
We are interested in the classiﬁer output when x r are unobserved, which can be expressed by marginalization as
[Equation]
We then approximate p ( x r | x \ r ) by some generative model with distribution p G ( x r | x \ r ) (speciﬁc implementations are discussed in section 4.1).
Then given a binary mask 2 z and the original image x ,
Input image x , classiﬁer score s M , sparsity hyperparameter λ , objective
L ∈ { L SSR , L SDR }
Initialize single mask z ∈ [0 , 1] d while loss L is not converged do
Clip z to [0 , 1]
Compute reference image ˆ x heuristically
Compute in-ﬁll φ ( x , z ) = x · z + ˆ x · (1 − z )
With φ , compute L by Equation 4 or
Update z with ∇ z L end while
Return z as per-feature saliency map
Algorithm 2: FIDO (Ours)
Input image x , classiﬁer score s M , sparsity hyperparameter λ , objective
L ∈ { L SSR , L SDR } , generative model p G
Initialize dropout rate θ ∈ [0 , 1] d while loss L is not converged do
Sample minibatch z ∈ { 0 , 1 } d ∼ Bern ( θ )
Sample reference image ˆ x ∼ p G (ˆ x | z , x )
Compute in-ﬁll φ ( x , z ) = x · z + ˆ x · (1 − z )
With φ , compute L by Equation 4 or
Update θ with ∇ θ L (Maddison et al., 2016) end while
Return θ as per-feature saliency map we deﬁne an inﬁlling function 3 φ as a convex mixture of the input and reference with binary weights, φ ( x , z ) =
z (cid:12) x
+ ( 1 − z ) (cid:12) ˆ x where ˆ x ∼ p G ( ˆ x | x z = 0 ) .
3. O BJECTIVE FUNCTIONS
The classiﬁcation score function s M ( c ) represents a score of classiﬁer conﬁdence on class c ; in our experiments we use log-odds:
[Equation]
SDR seeks a mask z yielding low classiﬁcation score when a small number of reference pixels are mixed into the mask regions.
Without loss of generality 4 , we can specify a parameterized distribution over masks q θ ( z ) and optimize its parameters.
The SDR problem is a minimization w.r.t θ of
[Equation]
On the other hand, SSR aims to ﬁnd a masked region that maximizes classiﬁcation score while penalizing the size of the mask.
For sign consistency with the previous problem, we express this as a minimization w.r.t θ of
[Equation]
Naively searching over all possible z is exponentially costly in the number of pixels U .
Therefore we specify q θ ( z ) as a factorized Bernoulli:
[Equation]
This corresponds to applying Bernoulli dropout (Srivastava et al., 2014) to the input pixels and optimizing the per-pixel dropout rate.
θ is our saliency map since it has the same dimensionality as the input and provides a probability of each pixel being marginalized (SDR) or retained (SSR) prior to classiﬁcation.
We call our method FIDO because it uses a strong generative model (see section 4.1) to Fill-In the DropOut region.
To optimize the θ through the discrete random mask z , we follow Gal et al.
(2017) in computing biased gradients via the Concrete distribution (Maddison et al., 2016; Jang et al., 2016); we use temperature 0 .
We initialize all our dropout rates θ to 0 .
5 since we ﬁnd it increases the convergence speed and avoids trivial solutions.
We optimize using Adam (Kingma & Ba, 2014) with learning rate 0 .
05 and linearly decay the learning rate for 300 batches in all our experiments.
Our PyTorch implementation takes about one minute on a single GPU to ﬁnish one image
3. C OMPARISON TO F ONG & V EDALDI (2017)
Fong & Vedaldi (2017) compute saliency by directly optimizing the continuous mask z ∈ [0 , 1] under the SDR objective, with ˆ x chosen heuristically; we call this approach Black Box Meaningful
Perturbations (BBMP).
We instead optimize the parameters of a Bernoulli dropout distribution q θ ( z ) , which enables us to sample reference values ˆ x from a learned generative model.
Our method uses mini-batches of samples z ∼ q θ ( z ) to efﬁciently explore the huge space of binary masks and obtain uncertainty estimates, whereas BBMP is limited to a local search around the current point estimate of the mask z .
See Figure 5 for a pseudo code comparison.
In Appendix A.1 we investigate how the choice of algorithm affects the resulting saliency maps.
To avoid unnatural artifacts in φ ( x , z ) , (Fong & Vedaldi, 2017) and (Dabkowski & Gal, 2017) additionally included two forms of regularization: upsampling and total variation penalization.
Upsampling is used to optimize a coarser θ (e.g. 56 × 56 pixels), which is upsampled to the full dimensionality (e.g. 224 × 224 ) using bilinear interpolation.
Total variation penalty smoothes θ by a (cid:96) 2 regularization penalty between spatially adjacent θ u .
To avoid losing too much signal from regularization, we use upsampling size 56 and total variation {+as 0 . 
+} unless otherwise mentioned.
We examine the individual effects of these regularization terms in Appendices A.2 and A.4, respectively.
E XPERIMENTS
We ﬁrst evaluate the various inﬁlling strategies and objective functions for FIDO.
We then compare explanations under several classiﬁer architectures.
In section 4.5 we show that FIDO saliency maps outperform BBMP (Fong & Vedaldi, 2017) in a successive pixel removal task where pixels are in-ﬁlled by a generative model (instead of set to the heuristic value).
FIDO also outperforms the method from (Dabkowski & Gal, 2017) on the so-called Saliency Metric on ImageNet.
Appendices
A.1–A.6 provide further analysis, including consistency and the effects of additional regularization.
4. I NFILLING M ETHODS
We describe several methods for producing the reference value ˆ x .
The heuristics do not depend on z and are from the literature.
The generative approaches, which produce ˆ x by conditioning on the non-masked inputs x z =0 , are novel to saliency computation.
Heuristics: Mean sets each pixel of ˆ x according to its per-channel mean across the training data.
Blur generates ˆ x by blurring x with Gaussian kernel ( σ = 10 ) (Fong & Vedaldi, 2017).
Random samples ˆ x from independent per-pixel per-channel uniform color with Gaussians ( σ = 0 . 2 ).
Generative Models: Local computes ˆ x as the average value of the surrounding non-dropped-out pixels x z =
(we use a 15 × 15 window).
VAE is an image completion Variational Autoencoder (Iizuka et al., 2017).
Using the predictive mean of the decoder network worked better than sampling.
CA is the Contextual Attention GAN (Yu et al., 2018); we use the authors’ pre-trained model.
Figure 3 compares these methods with a centered mask.
The heuristic in-ﬁlls appear far from the distribution of natural images.
This is ameliorated by using a strong generative model like CA, which in-ﬁlls texture consistent with the surroundings.
See Appendix A.5 for a quantitative comparison.
4. C OMPARING THE SSR AND SDR OBJECTIVE FUNCTIONS
Here we examine the choice of objective function between L SSR and L SDR ; see Figure 6.
We observed more artifacts in the L SDR saliency maps, especially when a weak in-ﬁlling method (Mean) is used.
We suspect this unsatisfactory behavior is due to the relative ease of optimizing L SDR .
There are many degrees of freedom in input space that can increase the probability of any of the 999 classes besides c ; this property is exploited when creating adversarial examples (Szegedy et al., 2013).
Since it is more difﬁcult to inﬁll unobserved pixels that increase the probability of a particular class c , we believe L SSR encourages FIDO to ﬁnd explanations more consistent with the classiﬁer’s training distribution.
It is also possible that background texture is easier for a conditional generative model to ﬁt.
To mitigate the effect of artifacts, we use L SSR for the remaining experiments.
Heuristics baselines (Mean, Blur and Random) tend to produce more artifacts, while generative approaches (Local, VAE, CA) produce more focused explanations on the targets.
4. C OMPARING INFILLING METHODS
Here we demonstrate the merits of using strong generative model that produces substantially fewer artifacts and a more concentrated saliency map.
In Figure 7 we generate saliency maps of different inﬁlling techniques by interpreting ResNet using L SSR with sparsity penalty λ
= 10 − 3 .
We observed a susceptibility of the heuristic in-ﬁlling methods (Mean, Blur, Random) to artifacts in the resulting saliency maps, which may fool edge ﬁlters in the low level of the network.
The use of generative in-ﬁlling (Local, VAE, CA) tends to mitigate this effect; we believe they encourage in-ﬁlled images to lie closer to the natural image manifold.
To quantify the artifacts in the saliency maps by a proxy: the proportion of the MAP conﬁguration ( θ > 0 . 5 ) that lies outside of the ground truth bounding box.
FIDO-CA produces the fewest artifacts by this metric (Figure 8).
4. I NTERPRETING V ARIOUS C LASSIFIER A RCHITECTURES
We use FIDO-CA to compute saliency of the same image under three classiﬁer architectures: AlexNet,
VGG and ResNet; see Figure 9.
Each architecture correctly classiﬁed all the examples.
We observed a qualitative difference in the how the classiﬁers prioritize different input regions (according to the saliency maps).
For example in the last image, we can see AlexNet focuses more on the body region of the bird, while Vgg and ResNet focus more on the head features.
4. Q UANTITATIVE E VALUATION
We follow Fong & Vedaldi (2017) and Shrikumar et al.
(2017) in measuring the classiﬁer’s sensitivity to successively altering pixels in order of their saliency scores.
Intuitively, the “best” saliency map should compactly identify relevant pixels, so that the predictions are changed with a minimum number of altered pixels.
Whereas previous works ﬂipped salient pixel values or set them to zero, we note that this moves the classiﬁer inputs out of distribution.
We instead dropout pixels in saliency order and inﬁll their values with our strongest generative model, CA-GAN.
To make the log-odds score suppression comparable between images, we normalize per-image by the ﬁnal log-odds suppression score (all pixels inﬁlled).
In Figure 10 we evaluate on ResNet and carry out our scoring procedure on 1 , 533 randomly-selected correctly-predicted ImageNet validation images, and report the number of pixels required to reduce the normalized log-odds score by a given percent.
We evaluate FIDO under various in-ﬁlling strategies as well as BBMP with Blur and Random
in-ﬁlling strategies.
We attempt to put both algorithms on equal footing by using λ = 1e − 3 for FIDO and λ = 5e − 3 for BBMP (see
Section A.1 for further discussion).
We ﬁnd that strong generative inﬁlling (VAE and CA) yields more parsimonious saliency maps, which is consistent with our qualitative comparisons.
FIDO-CA can achieve a given normalized log-odds score suppression using fewer pixels than competing methods.
Although FIDO-CA already trains on the CA-GAN and might be better adapted to it, we still ﬁnd other generative approaches, Local and VAE, still perform better than competing heuristics.
We compare our algorithm to several strong baselines on two established metrics.
We ﬁrst evaluate whether the FIDO saliency map can solve weakly supervised localization (WSL) (Dabkowski & Gal, 2017).
After thresholding the saliency map θ above 0.5, we compute the smallest bounding box containing all salient pixels.
This prediction is “correct” if it has intersection-over-union (IoU) ratio over 0 . 5 with any of the ground truth bounding boxes.
Using FIDO with various inﬁlling methods, we report the average error rate across all 50 , 000 validation images in Table 1.
We evaluate the
[Figure] [Figure]
[Table]
authors’ pre-trained model of Dabkowski & Gal (2017) 5 and call it Realtime. We also include ﬁve baselines: Max (entire input as the bounding box), Center (centered bounding box occupying half the image), Grad (Simonyan et al., 2013), Deconvnet (Springenberg et al., 2014), and GradCAM (Selvaraju et al., 2016).
We follow the procedure of mean thresholding in Fong & Vedaldi (2017): we normalize the heatmap between 0 and 1 and binarize by threshold θ = αµ i where µ i is the average heatmap for image i .
Then we take the smallest bounding box that encompasses all the binarized heatmap. We search α between 0 to 5 with 0 . 2 step size on a holdout set to get minimun WSL error. The best α are 1 . 2 , 2 and 1 respectively.
FIDO-CA frugally assigns saliency to contextually important pixels, preserving classiﬁer conﬁdence (Figure 4), so we do not necessarily expect our saliency maps to correlate with the typically large human labeled bounding boxes.
The reliance on human-labeled bounding boxes makes WSL suboptimal for evaluating saliency maps, so we evaluate the so-called Saliency Metric proposed by Dabkowski & Gal (2017), which eschews the human labeled bounding boxes.
The smallest bounding box A is computed as before.
The image is then cropped using this bounding box and upscaling to its original size.
The Saliency Metric is log max ( Area ( A ) , 0 . 05)
− log p ( c | CropAndUpscale ( x , A )) , the log ratio between the bounding box area and the in-class classiﬁer probability after upscaling.
This metric represents the information concentration about the label within the bounded region. From the superior performance of FIDO-CA we conclude that a strong generative model regularizes explanations towards the natural image manifold and ﬁnds concentrated region of features relevant to the classiﬁer’s prediction. 4. A BLATION S TUDY
Can existing algorithms be improved by adding an in-ﬁlling generative model without modeling a discrete distribution over per-feature masks?
And does ﬁlling in the dropped-out region sufﬁce without an expressive generative model?
We carried out a ablation study that suggests no on both counts.
We compare FIDO-CA to a BBMP variant that uses CA-GAN inﬁlling (called BBMP-CA); we also evaluate FIDO with heuristic inﬁlling (FIDO-Blur, FIDO-Random).
Because the continuous mask of BBMP does not naturally partition the features into observed/unobserved, BBMP-CA ﬁrst thresholds the masked region r = I ( z > τ ) before generating the reference φ ( x r , x \ r ) with a sample from CA-GAN. We sweep the value of τ as 1 , 0 . 7 , 0 . 5 , 0 . 3 , 0 . 1 and 0 .
We ﬁnd BBMP-CA is brittle with respect to its threshold value, producing either too spread-out or stringent saliency maps (Figure 11).
We observed that FIDO-Blur and FIDO-Random produce more concentrated saliency map than their BBMP counterpart with less artifacts, while FIDO-CA produces the most concentrated region on the target with fewest artifacts.
Each of these baselines were evaluated on the two quantitative metrics (Appendix A.7); BBMP-CA considerably underperformed relative to FIDO-CA.
S COPE AND L IMITATIONS
Because the classiﬁer behavior is ill-deﬁned for out-of-distribution inputs, any explanation that relies on out-of-distribution feature values is unsatisfactory.
By modeling the input distribution via an expressive generative model, we can encourage explanations that rely on counterfactual inputs close to the natural manifold.
However, our performance is then upper-bounded by the ability of the generative model to capture the conditional input density.
Fortunately, this bound will improve alongside future improvements in generative modeling.
We proposed FIDO, a new framework for explaining differentiable classiﬁers that uses adaptive
Bernoulli dropout with strong generative in-ﬁlling to combine the best properties of recently proposed methods (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017).
We compute saliency by marginalizing over plausible alternative inputs, revealing concentrated pixel areas that preserve label information.
By quantitative comparisons we ﬁnd the FIDO saliency map provides more parsimonious explanations than existing methods.
FIDO provides novel but relevant explanations for the classiﬁer in question by highlighting contextual information relevant to the prediction and consistent with the training distribution.
A F URTHER ANALYSIS A. C OMPARISONS OF BBMP AND FIDO
Here we compare FIDO with two previously proposed methods, BBMP with Blur in-ﬁlling strategy (Fong & Vedaldi, 2017) and BBMP with Random in-ﬁlling strategy (Dabkowski & Gal, 2017).
One potential concern in qualitatively comparing these methods is that each method might have a different sensitivity to the sparsity parameter λ .
Subjectively, we observe that BBMP requires roughly 5 times higher sparsity penalty λ to get visually comparable saliency maps.
In our comparisons we sweep λ over a reasonable range for each method and show the resulting sequence of increasingly sparse saliency maps (Figure 12).
We use λ = 5e − 4 , 1e − 3 , 2e − 3 , 5e − 3 .
We observe that all methods are prone to artifacts in the low λ regime, so the appropriate selection of this value is clearly important.
Interestingly, BBMP Blur and Random respectively ﬁnd artifacts with different quality: small patches and pixels for Blur and structure off-object lines for Random.
FIDO with CA is arguably the best saliency map, producing fewer artifacts and concentrating saliency on small regions for the images.
Note that here BBMP and FIDO methods use different λ .
We show the heuristics produce more artifacts (BBMP Blur) or produces weird lines (BBMP Random) compared to our method (FIDO CAGAN).
A. U PSAMPLING E FFECT
Here we examine the effect of learning a reduced dimensionality θ that upsampled to the full image size during optimization.
We consider a variety of upsampling rates, and in a slight abuse of terminology we refer to the upsampling “size” as the square root of the dimensionality of θ before upsampling, so smaller size implies more upsampling.
In Figure 13, we demonstrate two examples with different upsampling size under Mean and CA inﬁlling methods with SSR objectives.
The weaker inﬁlling strategy Mean apparently requires stronger regularization to avoid artifacts compared to CA.
Note that although CA produces much less artifacts compared to Mean, it still produces some small artifacts outside of the objects which is unfavored.
We then choose 56 for the rest of our experiments to balance between details and the removal of the artifacts.
A. S TABILITY
To show the stability of our method, we test our method with different random seeds and observe if they are similar.
In Figure 14, our method produces similar saliency map for 4 different random seeds.
A. T OTAL V ARIATION E FFECT
Here we test the effect of total variation prior regularization in Figure 15.
We ﬁnd the total variation can reduce the adversarial artifacts further, while risking losing signals when the total variation penalty is too strong.
A. A NALYSIS OF GENERATIVE MODEL INFILLING
Here we quantitatively compare the in-ﬁlling strategies.
The generative approaches (VAE and CA) perform visually sharper images than four other baselines.
Since we expect this random removal should not remove the target information, we use the classiﬁcation probability of the ResNet as our metric to measure how good the inﬁlling method recover the target prediction.
We quantitatively evaluate the probability for 1 , 000 validation images in Figure 16.
We ﬁnd that VAE and CA consistently outperform other methods, having higher target probability.
We also note that all the heuristic baselines (Mean, Blur, Random) perform much worse since the heuristic nature of these approaches, the images they generate are not likely under the distribution of natural images leading to the poor performance by the classiﬁer [Figure] P r [Figure]
b a
[Figure]
ili t y A. B ATCH S IZE E FFECTS
Figure 17 shows the effect of batch size on the saliency map. We found unsatisfactory results for batch size less than 4 , which we attribute this to the high variance in the resulting gradient estimates. A. A BLATION S TUDY We show the performance of BBMP-CA with various thresholds τ on both WSL and SM on subset of 1 , 000 images in Table 2.
We also show more qaulitative examples in Figure 22.
We ﬁnd BBMP-CA is relatively brittle across different thresholds of τ . Though with τ = 0 . 3 , the BBMP-CA perform slightly better than BBMP and FIDO with heuristics inﬁlling, it still performs substantially inferior to FIDO-CA. We also perform the ﬂipping experiment in Figure 18 and show our FIDO-CA substantially outperforms BBMP-CA with varying different thresholds.
[Table]
[Figure]
replaced with CA-GAN in-ﬁlled values.
lower better.
B M ORE E XAMPLES
Figure 19 shows several more inﬁlled counterfactual images, along with the counterfactuals produced by the method from Dabkowski & Gal (2017).
More examples comparing the various FIDO inﬁlling approaches can be found in Figure 20 and 21.
Input
Realtime
FIDO-CA
Input
Realtime
FIDO-CA
Sa li e n cy
I n - ﬁ ll 0 .0 .1 .0 .0 .0 .
Sa li e n cy
I n - ﬁ ll 0 .0 .1 .1 .0 .1 .