Structuring Uncertainty for Fine-Grained Sampling in Stochastic Segmentation Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In image segmentation, the classic approach of learning a deterministic segmentation neither accounts for noise and ambiguity in the data nor for expert disagreements about the correct segmentation.
This has been addressed by architectures thatpredict heteroscedastic (input-dependent) segmentation uncertainty, which indicates regions of segmentations that should be treated with care.
What is missing arestructural insights into the uncertainty, which would be desirable for interpretabilityand systematic adjustments.
In the context of state-of-the-art stochastic segmentation networks (SSNs), we solve this issue by dismantling the overall predicteduncertainty into smaller uncertainty components.
We obtain them directly fromthe low-rank Gaussian distribution for the logits in the network head of SSNs,based on a previously unconsidered view of this distribution as a factor model.
The rank subsequently encodes a number of latent variables, each of which controls an individual uncertainty component.
Hence, we can use the latent variables(called factors) for ﬁne-grained sample control, thereby solving an open problemfrom previous work.
There is one caveat though–factors are only unique up toorthogonal rotations.
Factor rotations allow us to structure the uncertainty in away that endorses simplicity, non-redundancy, and separation among the individualuncertainty components.
To make the overall and factor-speciﬁc uncertainties atplay comprehensible, we introduce ﬂow probabilities that quantify deviations fromthe mean prediction and can also be used for uncertainty visualization.
We showon medical-imaging, earth-observation, and trafﬁc-scene data that rotation criteriabased on factor-speciﬁc ﬂow probabilities consistently yield the best factors forﬁne-grained sampling.
Introduction
Semantic Segmentation is the computer vision task of assigning a class to each pixel of an image.
Examples for popular applications are the segmentation of medical images [1, 2, 35], land-coverclassiﬁcation in earth observation [27, 33, 38], and segmentation of imagery taken by autonomousvehicles [14, 24, 39].
Semantic segmentation tasks are affected by heteroscedastic (input-dependent)aleatoric uncertainty [25, 26] that is also called data uncertainty.
Aleatoric uncertainty can emergein form of label noise in the training data, differing expert opinions about the true segmentation, orambiguity already contained in the data, for example, caused by technical restrictions like the imageresolution [16, 19, 26].
To account for the prevailing aleatoric uncertainty, various probabilistic architectures have beenproposed [3, 20, 23, 26, 27, 30].
The work [11] demonstrates that in general, uncertainties predictedby probabilistic segmentation architectures correlate positively with estimation errors, includingthose obtained from ensemble methods [22, 28] and MC-dropout [15].
Nevertheless, deterministic
Ground truth
Mean prediction
FP full
Entropy
ForestShrublandSavannaGrasslandWetlandsCroplandsUrban /
[Figure]
/Ice
BarrenWater
Image
U n r o t a t e d
Factor Factor
Factor Factor
Factor Factor
Factor Factor
Factor Factor
F P - Q u a r t i m a x segmentation architectures [7, 31] are still pre-dominantly used [2, 40].
This may be because forpractitioners it is often not clear how to take advantage of the predicted uncertainties.
For instance,a limited number of sampled segmentations usually does not represent the predicted uncertaintywell.
As shown in Figure 1 (top), overview plots for the predicted uncertainty like entropy [11, 26]can be generated.
They indicate areas of high uncertainty, where practitioners should take care.
However, they do not convey pixel-wise correlations of uncertainty, that is, how changes in thesegmentation of one region of an image affect changes in another.
Moreover, overview plots cannotexplain the overall uncertainty in terms of smaller uncertainty components, which would be desirabletowards an interpretation of the uncertainty following the principles of problem decomposition anddivide-and-conquer [29].
In the context of the recently introduced stochastic segmentation networks (SSNs) [30], the lack ofunderstanding the overall uncertainty in terms of independent components connects to a problemthat was also admitted by the authors of [30] who pointed out in the accompanying demo that’ﬁne-grained sample control’ is still missing.
Indeed, the identiﬁcation of independent or at leastreasonably distinguished components of uncertainty represents a natural solution to this problem.
Weseek such components for SSNs with the goal of manipulating them individually for generating andﬁne-adjusting segmentations.
SSNs model uncertainty via a low-rank multivariate Gaussian distribution on the logits, that is, on thenetwork output before the softmax is applied, see Section 2.
Surprisingly, it has not been made useof the fact that this uncertainty model itself offers a straightforward way for distinguishing betweenuncertainty components.
This may be because the low-rank model was originally proposed forreducing parameters.
Here, we center our approach around the semantics of the low-rank modelas a factor model.
Factor models structure the overall uncertainty into individual components ofuncertainty, each of which is governed by a single latent variable (called factor).
Therefore, we solvethe open problem from [30] by using the latent factor variables as control variables for a systematicexploration of the predicted heteroscedastic segmentation uncertainty.
For the best result, however,some additional work is necessary: As is known from exploratory factor analysis [13, 36], the latentvariables in factor models are only unique up to orthogonal rotations.
Hence, they should be rotatedfor increased interpretability [6, 9, 36], which in our case amounts to generating more useful controls.
Good controls encode uncertainty components that are simple, non-redundant, and separable in thesense that they affect distinguished image regions or classes.
To evaluate these aspects, in Section 3we introduce ﬂow probabilities that quantify deviations from the mean prediction, which also enables uncertainty visualization, see Figure 1.
Speciﬁcally, we compute factor-speciﬁc ﬂow probabilitiesthat quantify the impact of the uncertainty components encoded by individual factors.
In Section 4,we fuse factor-speciﬁc ﬂow probabilities with classic rotation criteria [6, 10] from exploratory factoranalysis.
In Section 5, we show that these fused criteria generally result in the best possible controls.
Note that as a by-product, computing full ﬂow probabilities for the overall uncertainty also yields anew type of overview plot that does not aggregate class-speciﬁc information about the uncertainty,see Figure 1 (top).
Before we summarize our main contributions, we would like to emphasize that we do not benchmark
SSNs as they have already been proven to produce state-of-the-art results w.r.t.
various metrics, forinstance, generalized energy distance to the ground truth distribution [23, 30].
Hence, it is safe for usto assume that after successful training, SSNs are capable of predicting the aleatoric uncertainty for agiven input image reasonably well.
Based on that our main contributions are:
control variables for the contributions of individual, factor-speciﬁc uncertainty components
for ﬁne-grained sampling (given by the latent factor variables),2.
ﬂow probabilities for quantifying and visualizing overall and factor-speciﬁc uncertainties,3.
rotations based on factor-speciﬁc ﬂow probabilities, which structure the uncertainty compo-
nents and thereby provide simpler, less redundant, and well-separated control variables.
Factor modeling in stochastic segmentation networks
Stochastic segmentation networks (SSNs) [30] are characterized by modeling the pixel-wise logits inthe network head as a low-rank multivariate Gaussian distribution, that is,
[Equation]
Here, η ∈ R n are the n = hwc logits for an input image x of size h × w and a classiﬁcationproblem with c classes.
The parameters of the Gaussian distribution are the mean µ ( x ) ∈ R n andthe covariance matrix, which decomposes into a matrix of rank bounded by r (cid:28) n with square rootΓ ( x ) ∈ R n × r and a diagonal matrix Ψ ( x ) ∈ R n × n with positive diagonal elements.
The parametersof the Gaussian distribution are the output of a backbone segmentation network with input x [7, 31].
Originally, the low-rank parameterization was solely introduced as a means for reducing the numberof parameters [30].
However, the low-rank covariance model has a deeper structural meaning asa factor model [13, 34, 36].
Factor models are characterized by a typically small number of latentvariables, called factors, that explain all correlations among a larger number of observed variables.
Inour case, the joint distribution of the observed logits and the latent factor variables z ∈ R r is given by
[Equation]
where I r is the ( r × r ) identity matrix, and for brevity, we omit the dependence on the input x inthe notation from now on.
Here, the interactions of the latent variables with the observed logits aredescribed in the matrix Γ of factor loadings: Each column contains the loadings of one latent factorvariable on the observed logits, yielding structured uncertainty.
The loading characteristic becomesclear in the following sampling procedure for the logits from the factor model:
[Equation]
This procedure results from sampling from the joint distribution p ( η , z ) =
p ( z ) p ( η | z ) as follows:
First, the latent variables are sampled according to z ∼ N ( 0 , I r ) .
Second
, the logits are sampledfrom the conditional distribution η | z ∼ N ( µ + Γ z , Ψ ) .
Subsequently, only the logits are observed.
Sampling the logits as in Equation (1) provides control over the contributions of the different latentfactor variables.
This invites for an individual manipulation of the factors, enabling ﬁne-grainedsampling.
However, as pointed out in the introduction, factors should be rotated beforehand becausethey are only unique up to orthogonal rotations, see Lemma 1 in the supplement.
In particular,orthogonal rotations of the latent factor variables do not change the marginal distribution p ( η ) ofthe logits.
Indeed, replacing Γ z by Γ Oz for an orthogonal matrix O ∈ R r × r in Equation (1) yieldsan equivalent sampling procedure.
Therefore, one way to understand orthogonal rotations is thatthey change the basis of the r -dimensional afﬁne space { µ + Γ z : z ∈ R r } of the (noiseless) logits,where the basis elements are the columns of the factor loading matrix.
Flow probabilities
In this section, we develop the notion of ﬂow probabilities as our main tool for the analysis of factormodels in SSNs.
Flow probabilities can roughly be understood as probabilities of deviations from themean prediction.
We use ﬂow probabilities for uncertainty quantiﬁcation and visualization.
3.1 Factor-speciﬁc ﬂow probabilities
Because we need to understand and assess individual factors, it is important to analyze and quantifythe uncertainty that is encoded in them.
A useful tool for that is to compute factor-wise distributionsof class predictions, for which we vary an individual latent factor variable z with associated factorloadings γ (a column of Γ , later we use the notation Γ : ,j for a speciﬁc column of Γ ).
We keep theinﬂuence of all other latent factor and noise variables ﬁxed to zero.
Consequently, for z ∼ N (0 , 1)we compute the following expected value:
[Equation]
Here, E ( µ + γ z ) ∈ { 0 , 1 } ( wh ) × c is the matrix whose rows correspond to the pixel-wise one-hotencoded class predictions, which are obtained by reshaping the logits µ + γ z into shape ( wh, c )and then applying an arg max along the class dimension.
Note that the class probabilities P canbe understood as a function of the factor loadings γ since the mean logits µ are ﬁxed.
We solve
Equation (2) analytically.
For that, we consider pixels separately since for (ﬂat) spatial indexi ∈ [ wh ] = { 1 , . . .
, wh } , the probabilities p ik from the i -th row of P only depend on its associatedmean logits and factor loadings.
Speciﬁcally, with the deﬁnition g ik ( z ) = µ ik + γ ik z for k ∈ [ c ] , forﬁxed z
the predicted class is arg max k g ik ( z ) .
Hence, from Equation (2) we get that
[Equation]
where 1 is the indicator function.
We solve Equation (3) for binary classiﬁcation ﬁrst ( k ∈ { 1 , 2 } ).
Assuming that only the logits for the class k = 2 are learned, we set g i 1 ( z ) = 0 for consistency in
Equation (3).
Then, with µ = µ i 2 and assuming that γ = γ
i 2 (cid:54) = 0 , the probability p i 2 evaluates as
[Equation]
Here, ψ is the cumulative distribution function of a standard normal random variable.
For the lastequality, observe that − µ/γ is the intersection point of the straight line g i 2 ( z ) = µ
+ γz with thez -axis
g
i 1 .
If γ = 0 , the arg max is not unique and probabilities can be split.
For clarity of thetechnical exposition, we assume that the arg max is unique in the following.
For general multi-class problems, the probabilities p ik in Equation (3) can be derived fromthe class-prediction function z (cid:55)→ arg max k (cid:48)
g ik (cid:48)
( z ) .
In this function, the class predictioncan only change at intersection points z of two non-parallel straight lines g ik and g ik (cid:48) , that is,z = ( µ ik − µ ik (cid:48) ) /
( γ ik (cid:48) − γ ik ) .
Generally, if a class k is predicted for some z , then all z values forwhich the k -th class is predicted form a non-empty interval ( z ik , z ik ) ⊂ R .
The end points of thisinterval can either be − ∞ , an intersection point of g ik , or ∞ .
In practice, the intervals ( z ik , z ik ) canbe computed by sorting all intersection points and checking the values of the class-prediction functionon the resulting partition of the z -axis.
If a class k is never predicted, we set z ik = z
ik = − ∞ .
Finally, the class probability is given by p ik = ψ ( z ik ) − ψ
( z ik ) , where we use the conventions thatψ ( − ∞ )
=
and ψ ( ∞ ) = 1 .
Observe that the formula for binary problems given above is a specialcase of the one given for p ik here.
Overall, we obtain the following result:
Proposition 1.
Deﬁne Z =
( z ik ) and Z = ( z ik ) with entries i ∈
[ wh ] and k ∈ [ c ] .
Then, thedistribution of predicted classes under variation of the factor with associated loadings γ is given by
[Equation]
where ψ applies the cumulative distribution function of a standard normal variable element-wise.
Now, to highlight the difference to the prediction from the mean µ , we compute factor-speciﬁc ﬂowprobabilities as
[Equation]
Positive entries in the k -th column F ( γ ) : ,k indicate that the prediction for the corresponding pixelschanges with positive probability from the mean prediction to class k .
Based on this fact, factorspeciﬁc ﬂow probabilities enable visualizations of the impact of individual factors, see Figure 1(bottom rows).
The visualizations are obtained by calculating a mixture of class-speciﬁc colors withweights given by the (factor-speciﬁc) ﬂow probabilities, see the supplement for details.
As factor-speciﬁc ﬂow probabilities represent the real impact of a factor on output segmentations,they will also be a key to quality assessment of the factors, see Section 4.
For future reference, wedenote by F ( Γ ) ∈ [ − 1 , 1] ( whc ) × r the matrix of all factor-speciﬁc ﬂow probabilities that is obtainedby concatenating the factor-speciﬁc ﬂow probabilities F ( Γ : ,j ) as columns after ﬂattening, whereΓ : ,j is the j -th column of Γ .
Finally, since we use the latent factor variables as control variablesfor ﬁne-grained sampling, it is helpful to also compute one-sided ﬂow probabilities that encode theuncertainty for respectively positive and negative values of the latent factor variable.
Corollary 1.
Using the notation from Proposition 1, the one-sided factor-speciﬁc ﬂow probabilitiesfor a factor with loadings γ compute as
[Equation]
3.2 Uncertainty quantiﬁcation for the full factor model
The idea of computing factor-speciﬁc ﬂow probabilities for uncertainty quantiﬁcation and visualization extends to the full factor model.
For that, analogous to Equation (2), we compute the distributionof class predictions.
However, this time we take the expected value over the full distribution of thelogits given in Equation (1):
[Equation]
The change from the mean prediction E ( µ ) is then given by the full ﬂow probabilities, which wecompute as F full =
P full − E ( µ ) ∈ [ − 1 , 1] ( wh ) × c .
Visualizing full ﬂow probabilities as above byweighted mixtures of class-speciﬁc colors yields a new type of overview plot for the uncertainty, see
Figure 1 (top row) for an example.
Though for our work only a by-product, it has the advantage thatit does not aggregate information about class-speciﬁc uncertainties, in contrast to overview plots likeentropy (see also Figure 1, top row).
In practice, the integral from Equation (4) is difﬁcult to evaluate.
This is because the arg max in
E ( η ) technically amounts to determining a maximum of multivariate linear functions.
Hence, weapproximate the integral using Monte-Carlo integration with m i.i.d.
samples z (1) , . . .
, z ( m ) ∈ R rdrawn from N ( 0 , I r ) and i.i.d.
samples ε (1) , . . .
, ε ( m ) ∈
R whc drawn from N ( 0 , I whc ) .
The matrix
P full of class probabilities is thus approximated by
[Equation]
The matrix F full of ﬂow probabilities can be approximated similarly.
In the supplement, we showempirically that the diagonal noise term has little impact on the ﬂow probabilities.
Hence, we canfocus on the structural uncertainty that is induced by the latent factor variables.
Factor rotations
As pointed out in Section 2, the latent variables/factors in factor models are only unique up toorthogonal rotations.
Therefore, it is common practice in exploratory factor analysis to rotate themin order to maximize their interpretability [6, 21, 34].
The factor model in a SSN represents thepredicted uncertainty for a given input image, where the factors themselves encode components ofthe overall uncertainty.
We intend to use them as control variables for ﬁne-grained sampling.
Fromthat we derive the following quality criteria: (1) The number of ’relevant’ factors should be small, where relevant factors are characterized by having a ’signiﬁcant’ effect on output segmentations.
Relevant factors should be separable from each other in the sense that they encode distinguisheduncertainty components.
(3) Each area in the input image should be affected by only few factors.
Here, the ﬁrst criterion ensures that the number of impactful control variables is reduced to anecessary minimum, and the second criterion requires that the corresponding uncertainty componentsare distinct.
Together, the ﬁrst two criteria discourage factor redundancy.
The last criterion reﬂectsthe general requirement of sparsity and simplicity that is also found among Thurstone’s rules [37] forsimple structure of a factor loading matrix, which is the primary goal in exploratory factor analysis[13].
However, in our case we rather require a simple structure on the matrix F ( Γ ) of factor-speciﬁcﬂow probabilities (see Section 3.1) since they measure the actual impact of the factors on outputsegmentations.
In Section 5, we evaluate different rotation criteria that we present in the following.
First, we consider classic rotation criteria.
Here, for a factor loading matrix Γ = ( γ ij ) ∈ R n × r ,
Crawson and Ferguson [10] deﬁned the CF family of rotation criteria:
[Equation]
The CF family is a generalization of the widely used orthomax family [17], where the parameter κcontrols a trade-off between row complexity (ﬁrst sum) and column complexity (second sum).
Wefocus on popular choices: κ = 1 /n yields an equivalent version of the Varimax criterion [21], whichis the most used method.
Intuitively, it tries to maximize the variance of the squared factor loadings.
Next, κ = 0 yields the Quartimax criterion that minimizes the number of factors needed to explain avariable (in our case segmentation uncertainty of a pixel).
Finally, κ = r/ (2 n ) yields the Equamaxcriterion that represents a combination of Varimax and Quartimax.
Classic rotation criteria do not consider the actual impact of factors on predicted segmentationsbecause they only take the factor loadings Γ but not the mean µ into account.
Therefore, we incorporate factor-speciﬁc ﬂow probabilities into rotation criteria by applying a base rotation criterion q onthe ﬂow probabilities instead of the factor loadings.
Hence, the objective function to be minimizedbecomes
O
(cid:55)→ q ( F ( Γ O )) instead of O
(cid:55)→ q ( Γ O ) .
We call the new family of rotation criteria the FPfamily.
For instance, FP-Varimax applies the Varimax criterion on the ﬂow probabilities.
Experiments
The purpose of our experiments is to (1) evaluate rotation criteria based on the quality of rotatedfactors, (2) demonstrate the merits of ﬁne-grained sample control based on reasonably-rotated factors.
Data sets and training.
First, we use the LIDC data set [1] in its pre-processed version from [27]that contains 2D slices of 3D thorax scans of size 128 × 128 pixels.
Each slice respectively has fourground truth segmentations from different experts.
Second, we use the multi-spectral Sentinel-2 datafrom the SEN12MS data set [32] with images of size 244 × 244 pixels and coarse labels for semanticsegmentation of 10 types of land cover.
Third, we use the CamVid data set [5], which contains imagesof road scenes in resolution 480 × 360 and is pixel-wise labeled into 11 different classes.
Additionaldetails and statistics about the data sets (including splits) can be found in the supplement, where wealso detail all training procedures.
We respectively use r = 10 in our experiments, which accountsfor the varying uncertainty in different images and has also been used in [30].
We would like toemphasize again that we do not benchmark SSNs since they have already shown to be state of the art[23, 30].
For examples of uncertainty predictions, see Figure 1, Figure 2, and the supplement.
[Figure]
Computational aspects.
On a single core of an Intel Xeon Platinum 8260 and using Python 3.7,factor-speciﬁc and full ﬂow probabilities can be computed in the sub-second range without signiﬁcantdifferences w.r.t.
the used rotation, see the supplement for details.
To obtain the optimal rotation matrices for the different rotation criteria, we adapted gradientprojection algorithms from [4] to our needs.
In our current implementation, optimization for criteriabased on ﬂow probabilities can take up to a few minutes, see the supplement for details.
In practice,we recommend pre-computing rotations whenever possible.
5.1 Evaluation of rotation criteria
We evaluate rotations according to the quality criteria from Section 4, that is, (1) the relevance ofindividual factors, (2) the separability of the relevant factors, and (3) the sparsity of the factors.
5.1.1 Factor relevance
Here, we measure the impact of individual factors on the segmentation.
A simple measure forthe impact of the j -th factor with loadings Γ : ,j ∈ R n is given by the (cid:96) 1 -norm (cid:107) F ( Γ : ,j ) (cid:107) 1 of thefactor-speciﬁc ﬂow probabilities.
In what follows, we consider relevance curves that show howmany factors exceed the overall uncertainty for varying thresholds τ ≥ 0 .
Speciﬁcally, we computen τ =
| R τ | , where R τ = { j : (cid:107) F ( Γ : ,j ) (cid:107) 1 ≥ τ (cid:107) F full ( Γ ) (cid:107) 1 } and we measure the overall uncertaintyby the (cid:96) 1 -norm of the full ﬂow probabilities, approximated by 100 Monte-Carlo samples.
Results and discussion.
The results of averaging n τ over the respective test images are shown in
Figure 3 (top row).
First, classic rotations barely reduce the number of relevant factors compared tothe unrotated representation.
This is no surprise since they do not take the mean logits into accountand only try to simplify the structure of the factor loadings Γ .
Nevertheless, even classic rotationsalready seem to decrease redundancy.
However, as intended by design, FP rotations reduce thenumber of relevant factors to a much greater extent.
Especially for LIDC and SEN12MS, alreadysmall thresholds τ are sufﬁcient to cut off most factors below the threshold: Figure 3 (top) showsthat all FP rotations behave similarly with curves declining sharply for small τ .
Consequently, FProtations tend to produce a huge gap between a small number of relevant factors and the remainingones, see Figure 1 for a visual example.
This is desirable since it allows to focus only on a fewrelevant and meaningful factors during the exploration of the predicted uncertainty.
However, it maybe harder to ﬁnd such factors if the predicted uncertainty has less inherent structure.
CamVid is anexample in this regard since for this data set, uncertainty predictions are often restricted to class [Figure] borders, which means that they are less spatially correlated.
Nevertheless, even for CamVid, FProtations signiﬁcantly reduce the number of relevant factors.
5.1.2 Separability of relevant factors
The second quality criterion from Section 4 concerns factor separation.
Here, for a separationthreshold ρ ∈ [0 , 1] , we compute the largest possible fraction of pairwise separated relevant factors:
[Equation]
If n τ = 0 , we set s τ ( ρ ) = 0 for all ρ .
The separation of two factors is measured by the cosinesimilarity of their factor-speciﬁc ﬂow probabilities, which is always non-negative since correspondingentries cannot have opposing signs.
For s τ ( ρ ) , a value of one is best since it means that all relevantfactors are also separated.
For ﬁxed relevance thresholds τ , we also compute the area under the curve
AUC ( s τ ) for the comparison of different rotation criteria.
Results and discussion.
In Figure 3 (bottom row), we show the separation scores AUC ( s τ ) fordifferent relevance thresholds τ , respectively averaged over all test images.
FP rotations consistentlybeat classic rotations by a factor of around two in terms of AUC.
Classic rotation criteria are stillbetter than the unrotated representations (which form the real baseline).
The AUC separation scoresdrop for thresholds τ that fail to determine the number of relevant factors sensefully because theyare too small or too large.
Notably, for classic rotation criteria, the separation scores AUC ( s τ )respectively peak at a threshold τ for which the number of relevant factors nearly coincides with theone from FP rotations, compare the intersection of the curves in Figure 3 (top row).
The peak of theseparation scores is less pronounced for FP rotations, particularly for LIDC and SEN12MS, wherethe set of relevant factors is more stable across different thresholds τ .
For SEN12MS, the results alsodistinguish among the FP-rotation criteria, where FP-Quartimax seems to be slightly favored over theother FP rotations.
This may be because Quartimax emphasizes row sparsity the most, which reducescosine similarities.
We investigate row sparsity further in the next section.
5.1.3 Factor sparsity
To evaluate to which degree different factors affect the same regions of the input image, we measurerow sparsity of the factor-speciﬁc ﬂow probabilities F ( Γ ) ∈ [ − 1 , 1] ( whc ) × r .
For that, for a (row)vector v ∈ R r let h ( v ) =
( √ r − (cid:107) v (cid:107) 1 / (cid:107) v (cid:107) 2 ) / ( √ r − 1) ∈ [0 , 1] be the Hoyer measure [18], wherevalues close to one indicate a high degree of sparsity.
For us, sparsity only matters in rows with actualuncertainty for the pixel/class, therefore we additionally weigh each row proportional to its (cid:96) 1 -norm.
Hence, as a ﬁnal measure, we compute the weighted Hoyer measure
[Equation]
Results and discussion.
FP rotations generally concentrate the uncertainty for single regions/classes in only few components, see Table 1.
This means that FP rotation yield the most disentangled uncertainty components, which also indicates strong separation.
For LIDC and SEN12MS,the amount of predicted uncertainty varies greatly across test images, causing high standard deviations.
However, in general, large correlating uncertainty components can be found, allowing high rowsparsity.
This is in contrast to CamVid, where uncertainty is typically predicted for class borders.
different rotations.
FP rotations yield the most sparsity.
For a more detailed ﬁgure, see the supplement
[Table]
F a c t o r F a c t o r
F a c t o r F a c t o r
5.2 Fine-grained sampling
Monteiro et al.
[30] already manipulated samples post-hoc by simple linear inter- or extrapolationw.r.t.
the mean.
However, they noted that additional more ﬁne-grained sample control is necessary fora systematic exploration of the sample space: The interpolation approach lacks a solid foundation inthe uncertainty model, and it relies on having a useful sample to start with.
The meaningful controlvariables that we obtain by rotating factors provide all that has been missing.
They enable users tosystematically explore the sample space by ﬁne-grained sampling: Starting from the mean prediction,they can inspect alternatives, correct possible mistakes, and ﬁne-adjust borders.
Particularly, they canmanipulate the contribution of individual uncertainty components by manually setting the values ofthe corresponding factors.
Pseudo-samples obtained in this way are shown in Figure 4.
Alongside this paper, we provide an interface for ﬁne-grained sampling.
It allows the selection ofa rotation criterion for a given input image (we recommend FP-Quartimax for a start), and controlvariables can be set conveniently using sliders.
In the supplement, we provide some visuals.
Discussion and conclusion
In this work, we interpreted the uncertainty model of stochastic segmentation networks (SSNs) as afactor model, which provides control variables for ﬁne-grained sampling as requested by the authorsof [30].
By (re-)structuring the uncertainty using rotations, we improved the controls and obtainedas few as possible, but as many as necessary relevant uncertainty components.
Here, it turned outthat rotation criteria based on ﬂow probabilities yield the most meaningful controls, where ﬂowprobabilities are a new quantiﬁcation and visualization technique for the uncertainty in SSNs.
Our controls allow to systematically explore the predicted uncertainty and to ﬁne-adjust samples.
However, the exploration of the sample space is only useful if the overall predicted uncertaintymakes sense.
This can be ensured by proper training.
Structuring and examining the uncertainty isespecially useful if there is a signiﬁcant amount of aleatoric uncertainty.
We note that one limitationcaused by our current implementation is that the computation of ﬂow-probability based rotationsmay take too long for performing it in an interactive scenario.
However, there is signiﬁcant potentialfor improving the optimization (scheme and parallelization).
In any case, rotations should be precomputed whenever possible.
Overall, we see a broader impact of our approach, which extends beyond the scope of SSNs.
Forinstance, we believe that it could be used for large-scale image classiﬁcation, where factor modelshave recently been employed for modeling class correlations [8].
Another promising applicationis to learn and inspect more structured latent spaces in (variational) autoencoders [12].
Structuringuncertainty as we do may be useful whenever a multivariate Gaussian forms part of a model.
Next,ﬂow probabilities can also be used for other probabilistic segmentation architectures that have a meanprediction as a reference point.
Notably, ﬂow-probability overview plots for the predicted uncertaintykeep class-speciﬁc information, in contrast to other overview plots like entropy.
To sum up, it is often easier to understand the whole in terms of smaller parts.
In this light, westructured the predicted uncertainty of SSNs into meaningful smaller uncertainty components.
Jointly,they enable ﬁne-grained sample control, so for us, the sum of the parts is also greater than the whole.