Structuring Uncertainty for Fine-Grained Sampling in Stochastic Segmentation Networks
Abstract
In image segmentation, the classic approach of learning a deterministic segmentation neither accounts for noise and ambiguity in the data nor for expert disagreements about the correct segmentation.
This has been addressed by architectures that predict heteroscedastic (input-dependent) segmentation uncertainty, which indicates regions of segmentations that should be treated with care.
What is missing are structural insights into the uncertainty, which would be desirable for interpretability and systematic adjustments.
In the context of state-of-the-art stochastic segmentation networks (SSNs), we solve this issue by dismantling the overall predicted uncertainty into smaller uncertainty components.
We obtain them directly from the low-rank Gaussian distribution for the logits in the network head of SSNs, based on a previously unconsidered view of this distribution as a factor model.
The rank subsequently encodes a number of latent variables, each of which controls an individual uncertainty component.
Hence, we can use the latent variables (called factors) for ﬁne-grained sample control, thereby solving an open problem from previous work.
There is one caveat though–factors are only unique up to orthogonal rotations.
Factor rotations allow us to structure the uncertainty in a way that endorses simplicity, non-redundancy, and separation among the individual uncertainty components.
To make the overall and factor-speciﬁc uncertainties at play comprehensible, we introduce ﬂow probabilities that quantify deviations from the mean prediction and can also be used for uncertainty visualization.
We show on medical-imaging, earth-observation, and trafﬁc-scene data that rotation criteria based on factor-speciﬁc ﬂow probabilities consistently yield the best factors for ﬁne-grained sampling.
Introduction
Semantic Segmentation is the computer vision task of assigning a class to each pixel of an image.
Examples for popular applications are the segmentation of medical images [1, 2, 39], land-cover classiﬁcation in earth observation [28, 37, 42], and segmentation of imagery taken by autonomous vehicles [14, 25, 43].
Semantic segmentation tasks are affected by heteroscedastic (input-dependent) aleatoric uncertainty [26, 27] that is also called data uncertainty.
Aleatoric uncertainty can emerge in form of label noise in the training data, differing expert opinions about the true segmentation, or ambiguity already contained in the data, for example, caused by technical restrictions like the image resolution [16, 20, 27].
To account for the prevailing aleatoric uncertainty, various probabilistic architectures have been proposed [3, 21, 24, 27, 28, 31].
The work [11] demonstrates that in general, uncertainties predicted by probabilistic segmentation architectures correlate positively with estimation errors, including those obtained from ensemble methods [23, 29] and MC-dropout [15].
Nevertheless, deterministic segmentation architectures [7, 35] are still pre-dominantly used [2, 44].
This may be because for practitioners it is often not clear how to take advantage of the predicted uncertainties.
For instance, a limited number of sampled segmentations usually does not represent the predicted uncertainty well.
As shown in Figure 1 (top), overview plots for the predicted uncertainty like entropy [11, 27] can be generated.
They indicate areas of high uncertainty, where practitioners should take care.
However, they do not convey pixel-wise correlations of uncertainty, that is, how changes in the segmentation of one region of an image affect changes in another.
Moreover, overview plots cannot explain the overall uncertainty in terms of smaller uncertainty components, which would be desirable towards an interpretation of the uncertainty following the principles of problem decomposition and divide-and-conquer [30].
[Figure]
[Table]
[Figure]
[Table]
[Figure]
a r t i m a x
In the context of the recently introduced stochastic segmentation networks (SSNs) [31] that we explain below the lack of understanding the overall uncertainty in terms of independent components connects to a problem that was also observed by the authors of [31].
They pointed out in their accompanying demo that ’ﬁne-grained sample control’ is still missing.
Indeed, the identiﬁcation of independent or at least reasonably distinguished components of uncertainty represents a natural solution to this problem.
We seek such components for SSNs with the goal of manipulating them individually for generating and ﬁne-adjusting segmentations.
SSNs model uncertainty via a low-rank multivariate Gaussian distribution on the logits, that is, on the network output before the softmax is applied, see Section 2.
Surprisingly, it has not been made use of the fact that this uncertainty model itself offers a straightforward way for distinguishing between uncertainty components.
This may be because the low-rank model was originally proposed for reducing the number of parameters.
Here, we center our approach around the semantics of the low-rank model as a factor model.
Factor models structure the overall uncertainty into individual components of uncertainty, each of which is governed by a single latent variable (called factor).
Therefore, we solve the open problem from [31] by using the latent factor variables as control variables for a systematic exploration of the predicted heteroscedastic segmentation uncertainty.
For the best result, however, some additional work is necessary: As is known from exploratory factor analysis [13, 40], the latent variables in factor models are only unique up to orthogonal rotations.
Hence, they should be rotated for increased interpretability [6, 9, 40], which in our case amounts to generating more useful controls.
Good controls encode uncertainty components that are simple, non-redundant, and separable in the sense that they affect distinguished image regions or classes.
To evaluate these aspects, in Sectionwe introduce ﬂow probabilities that quantify deviations from the mean prediction, which also enables uncertainty visualization, see Figure 1.
Speciﬁcally, we compute factor-speciﬁc ﬂow probabilities that quantify the impact of the uncertainty components encoded by individual factors.
In Section 4, we fuse factor-speciﬁc ﬂow probabilities with classic rotation criteria [6, 10] from exploratory factor analysis.
In Section 5, we show that these fused criteria generally result in the best possible controls.
Note that as a by-product, computing full ﬂow probabilities for the overall uncertainty also yields a new type of overview plot that does not aggregate class-speciﬁc information about the uncertainty, see Figure 1 (top).
Before we summarize our main contributions, we would like to emphasize that we do not benchmark
SSNs as they have already been proven to produce state-of-the-art results w.r.t.
various metrics, for instance, generalized energy distance to the ground truth distribution [24, 31].
Hence, it is safe for us to assume that after successful training, SSNs are capable of predicting the aleatoric uncertainty for a given input image reasonably well.
Based on that our main contributions are:
(1) control variables for the contributions of individual, factor-speciﬁc uncertainty components for ﬁne-grained sampling (given by the latent factor variables), (2) ﬂow probabilities for quantifying and visualizing overall and factor-speciﬁc uncertainties, (3) rotations based on factor-speciﬁc ﬂow probabilities, which structure the uncertainty components and thereby provide simpler, less redundant, and well-separated control variables.
Please ﬁnd an overview ﬁgure of our contributions in Section A of our supplement.
Additionally, we made the code for the proposed methods and experiments available under https://github.com/JakobCode/StructuringSSNs.
Factor modeling in stochastic segmentation networks
Stochastic segmentation networks (SSNs) [31] are characterized by modeling the pixel-wise logits in the network head as a low-rank multivariate Gaussian distribution, that is,
[Equation]
Here, η ∈ R n are the n = hwc logits for an input image x of size h × w and a classiﬁcation problem with c classes.
The parameters of the Gaussian distribution are the mean µ ( x ) ∈ R n and the covariance matrix, which decomposes into a matrix of rank bounded by r (cid:28) n with square root Γ ( x ) ∈ R n × r and a diagonal matrix Ψ ( x ) ∈ R n × n with positive diagonal elements.
The parameters of the Gaussian distribution are the output of a backbone segmentation network with input x [7, 35].
Originally, the low-rank parameterization was solely introduced as a means for reducing the number of parameters [31].
However, the low-rank covariance model has a deeper structural meaning as a factor model [13, 38, 40].
Factor models are characterized by a typically small number of latent variables, called factors, that explain all correlations among a larger number of observed variables.
In our case, the joint distribution of the observed logits and the latent factor variables z ∈ R r is given by
[Equation]
where I r is the ( r × r ) identity matrix, and for brevity, we omit the dependence on the input x in the notation from now on.
Here, the interactions of the latent variables with the observed logits are described in the matrix Γ of factor loadings: Each column contains the loadings of one latent factor variable on the observed logits, yielding structured uncertainty.
The loading characteristic becomes clear in the following sampling procedure for the logits from the factor model:
[Equation]
z ∼
[Equation]
This procedure results from sampling from the joint distribution p ( η , z ) =
p ( z ) p ( η | z ) as follows:
First, the latent variables are sampled according to z ∼ N ( 0 , I r ) .
Second, the logits are sampled from the conditional distribution η | z ∼ N ( µ + Γ z , Ψ ) .
Subsequently, only the logits are observed.
Sampling the logits as in Equation (1) provides control over the contributions of the different latent factor variables.
This invites for an individual manipulation of the factors, enabling ﬁne-grained sampling.
However, as pointed out in the introduction, factors should be rotated beforehand because they are only unique up to orthogonal rotations, see Lemma 1 in the supplement.
In particular, orthogonal rotations of the latent factor variables do not change the marginal distribution p ( η ) of the logits.
Indeed, replacing Γ z by Γ Oz for an orthogonal matrix O ∈ R r × r in Equation (1) yields an equivalent sampling procedure.
Therefore, one way to understand orthogonal rotations is that they change the basis of the r -dimensional afﬁne space { µ + Γ z : z ∈ R r } of the (noiseless) logits, where the basis elements are the columns of the factor loading matrix.
Flow probabilities
In this section, we develop the notion of ﬂow probabilities as our main tool for the analysis of factor models in SSNs.
Flow probabilities can roughly be understood as probabilities of deviations from the mean prediction.
We use ﬂow probabilities for uncertainty quantiﬁcation and visualization.
3.1 Factor-speciﬁc ﬂow probabilities
Because we need to understand and assess individual factors, it is important to analyze and quantify the uncertainty that is encoded in them.
A useful tool for that is to compute factor-wise distributions of class predictions, for which we vary an individual latent factor variable z with associated factor loadings γ (a column of Γ , later we use the notation Γ : ,j for a speciﬁc column of Γ ).
We keep the inﬂuence of all other latent factor and noise variables ﬁxed to zero.
Consequently, for z ∼ N (0 , 1) we compute the following expected value:
[Equation]
Here, E ( µ + γ z ) ∈ { 0 , 1 } ( wh ) × c is the matrix whose rows correspond to the pixel-wise one-hot encoded class predictions, which are obtained by reshaping the logits µ + γ z into shape ( wh, c ) and then applying an arg max along the class dimension.
Note that the class probabilities P can be understood as a function of the factor loadings γ since the mean logits µ are ﬁxed.
We solve
Equation (2) analytically.
For that, we consider pixels separately since for (ﬂat) spatial index i ∈ [ wh ] = { 1 , . . .
, wh } , the probabilities p ik from the i -th row of P only depend on its associated mean logits and factor loadings.
Speciﬁcally, with the deﬁnition g ik ( z ) = µ ik + γ ik z for k ∈ [ c ] , for ﬁxed z the predicted class is arg max k g ik ( z ) .
Hence, from Equation (2) we get that
[Equation]
where 1 is the indicator function.
We solve Equation (3) for binary classiﬁcation ﬁrst ( k ∈ { 1 , 2 } ).
Assuming that only the logits for the class k = 2 are learned, we set g i 1 ( z ) = 0 for consistency in
Equation (3).
Then, with µ = µ i 2 and assuming that γ = γ
i 2 (cid:54) = 0 , the probability p i 2 evaluates as
[Equation]
i
[Equation]
Here, ψ is the cumulative distribution function of a standard normal random variable.
For the last equality, observe that − µ/γ is the intersection point of the straight line g i 2 ( z ) = µ + γz with the z -axis g
i 1 .
If γ = 0 , the arg max is not unique and probabilities can be split.
For clarity of the technical exposition, we assume that the arg max is unique in the following.
For general multi-class problems, the probabilities p ik in Equation (3) can be derived from the class-prediction function z (cid:55)→ arg max k (cid:48)
g ik (cid:48)
( z ) .
In this function, the class prediction can only change at intersection points z of two non-parallel straight lines g ik and g ik (cid:48) , that is, z = ( µ ik − µ ik (cid:48) ) /
( γ ik (cid:48) − γ ik ) .
Generally, if a class k is predicted for some z , then all z values for which the k -th class is predicted form a non-empty interval ( z ik , z ik ) ⊂ R .
The end points of this interval can either be − ∞ , an intersection point of g ik , or ∞ .
In practice, the intervals ( z ik , z ik ) can be computed by sorting all intersection points and checking the values of the class-prediction function on the resulting partition of the z -axis.
If a class k is never predicted, we set z ik = z
ik = − ∞ .
Finally, the class probability is given by p ik = ψ ( z ik ) − ψ
( z ik ) , where we use the conventions that ψ ( − ∞ ) = 0 and ψ ( ∞ ) = 1 .
Observe that the formula for binary problems given above is a special case of the one given for p ik here.
Overall, we obtain the following result:
Proposition 1.
Deﬁne Z =
( z ik ) and Z = ( z ik ) with entries i ∈
[ wh ] and k ∈ [ c ] .
Then, the distribution of predicted classes under variation of the factor with associated loadings γ is given by
[Equation]
where ψ applies the cumulative distribution function of a standard normal variable element-wise.
Now, to highlight the difference to the prediction from the mean µ , we compute factor-speciﬁc ﬂow probabilities as
[Equation]
Positive entries in the k -th column F ( γ ) : ,k indicate that the prediction for the corresponding pixels changes with positive probability from the mean prediction to class k .
Based on this fact, factorspeciﬁc ﬂow probabilities enable visualizations of the impact of individual factors, see Figure(bottom rows).
The visualizations are obtained by calculating a mixture of class-speciﬁc colors with weights given by the (factor-speciﬁc) ﬂow probabilities, see the supplement for details.
As factor-speciﬁc ﬂow probabilities represent the real impact of a factor on output segmentations, they will also be a key to quality assessment of the factors, see Section 4.
For future reference, we denote by F ( Γ ) ∈ [ − 1 , 1] ( whc ) × r the matrix of all factor-speciﬁc ﬂow probabilities that is obtained by concatenating the factor-speciﬁc ﬂow probabilities F ( Γ : ,j ) as columns after ﬂattening, where Γ : ,j is the j -th column of Γ .
Finally, since we use the latent factor variables as control variables for ﬁne-grained sampling, it is helpful to also compute one-sided ﬂow probabilities that encode the uncertainty for respectively positive and negative values of the latent factor variable.
Corollary 1.
Using the notation from Proposition 1, the one-sided factor-speciﬁc ﬂow probabilities for a factor with loadings γ compute as
[Equation]
3.2 Uncertainty quantiﬁcation for the full factor model
The idea of computing factor-speciﬁc ﬂow probabilities for uncertainty quantiﬁcation and visualization extends to the full factor model.
For that, analogous to Equation (2), we compute the distribution of class predictions.
However, this time we take the expected value over the full distribution of the logits given in Equation (1):
[Equation]
The change from the mean prediction E ( µ ) is then given by the full ﬂow probabilities, which we compute as F full =
P full − E ( µ ) ∈ [ − 1 , 1] ( wh ) × c .
Visualizing full ﬂow probabilities as above by weighted mixtures of class-speciﬁc colors yields a new type of overview plot for the uncertainty, see
Figure 1 (top row) for an example.
Though for our work only a by-product, it has the advantage that it does not aggregate information about class-speciﬁc uncertainties, in contrast to overview plots like entropy (see also Figure 1, top row).
In practice, the integral from Equation (4) is difﬁcult to evaluate.
This is because the arg max in
E ( η ) technically amounts to determining a maximum of multivariate linear functions.
Hence, we approximate the integral using Monte-Carlo integration with m i.i.d.
samples z (1) , . . .
, z ( m ) ∈ R r drawn from N ( 0 , I r ) and i.i.d.
samples ε (1) , . . .
, ε ( m ) ∈
R whc drawn from N ( 0 , I whc ) .
The matrix
P full of class probabilities is thus approximated by
[Equation]
The matrix F full of ﬂow probabilities can be approximated similarly.
In the supplement, we show empirically that the diagonal noise term has little impact on the ﬂow probabilities.
Hence, we can focus on the structural uncertainty that is induced by the latent factor variables.
Factor rotations
As pointed out in Section 2, the latent variables/factors in factor models are only unique up to orthogonal rotations.
Therefore, it is common practice in exploratory factor analysis to rotate them in order to maximize their interpretability [6, 22, 38].
The factor model in a SSN represents the predicted uncertainty for a given input image, where the factors themselves encode components of the overall uncertainty.
We intend to use them as control variables for ﬁne-grained sampling.
From that we derive the following quality criteria:
(1) The number of ’relevant’ factors should be small, where relevant factors are characterized by having a ’signiﬁcant’ effect on output segmentations.
(2) Relevant factors should be separable from each other in the sense that they encode distinguished uncertainty components.
(3) Each area in the input image should be affected by only few factors.
Here, the ﬁrst criterion ensures that the number of impactful control variables is reduced to a necessary minimum, and the second criterion requires that the corresponding uncertainty components are distinct.
Together, the ﬁrst two criteria discourage factor redundancy.
The last criterion reﬂects the general requirement of sparsity and simplicity that is also found among Thurstone’s rules [41] for simple structure of a factor loading matrix, which is the primary goal in exploratory factor analysis [13].
However, in our case we rather require a simple structure on the matrix F ( Γ ) of factor-speciﬁc ﬂow probabilities (see Section 3.1) since they measure the actual impact of the factors on output segmentations.
In Section 5, we evaluate different rotation criteria that we present in the following.
First, we consider classic rotation criteria.
Here, for a factor loading matrix Γ = ( γ ij ) ∈ R n × r ,
Crawson and Ferguson [10] deﬁned the CF family of rotation criteria:
[Equation]
The CF family is a generalization of the widely used orthomax family [17], where the parameter κ controls a trade-off between row complexity (ﬁrst sum) and column complexity (second sum).
We focus on popular choices: κ = 1 /n yields an equivalent version of the Varimax criterion [22], which is the most used method.
Intuitively, it tries to maximize the variance of the squared factor loadings.
Next, κ = 0 yields the Quartimax criterion that minimizes the number of factors needed to explain a variable (in our case segmentation uncertainty of a pixel).
Finally, κ = r/ (2 n ) yields the Equamax criterion that represents a combination of Varimax and Quartimax.
Classic rotation criteria do not consider the actual impact of factors on predicted segmentations because they only take the factor loadings Γ but not the mean µ into account.
Therefore, we incorporate factor-speciﬁc ﬂow probabilities into rotation criteria by applying a base rotation criterion q on the ﬂow probabilities instead of the factor loadings.
Hence, the objective function to be minimized becomes O (cid:55)→ q ( F ( Γ O )) instead of O (cid:55)→ q ( Γ O ) .
We call the new family of rotation criteria the FP family.
For instance, FP-Varimax applies the Varimax criterion on the ﬂow probabilities.
Experiments
The purpose of our experiments is to (1) evaluate rotation criteria based on the quality of rotated factors, (2) demonstrate the merits of ﬁne-grained sample control based on reasonably-rotated factors.
Data sets and training.
First, we use the LIDC data set [1] in its pre-processed version from [28] that contains 2D slices of 3D thorax scans of size 128 × 128 pixels.
Each slice respectively has four ground truth segmentations from different experts.
Second, we use the multi-spectral Sentinel-2 data from the SEN12MS data set [36] with images of size 244 × 244 pixels and coarse labels for semantic segmentation of 10 types of land cover.
Third, we use the CamVid data set [5], which contains images of road scenes in resolution 480 × 360 and is pixel-wise labeled into 11 different classes.
Additional details and statistics about the data sets (including splits) can be found in the supplement, where we also detail all training procedures.
We respectively use r = 10 in our experiments, which accounts for the varying uncertainty in different images and has also been used in [31].
We would like to emphasize again that we do not benchmark SSNs since they have already shown to be state of the art [24, 31].
For examples of uncertainty predictions, see Figure 1, Figure 2, and the supplement.
[Figure]
Computational aspects.
We used Python 3.7, particularly with the libraries PyTorch 1.11 [32], scikit-learn [33], NumPy [18], and einops [34].
On a single core of an Intel Xeon Platinum 8260, factor-speciﬁc and full ﬂow probabilities can be computed in the sub-second range without signiﬁcant differences w.r.t.
the used rotation, see the supplement for details.
To obtain the optimal rotation matrices for the different rotation criteria, we adapted gradient projection algorithms from [4] to our needs.
In our current implementation, optimization for criteria based on ﬂow probabilities can take up to a few minutes, see the supplement for details.
In practice, we recommend pre-computing rotations whenever possible.
5.1 Evaluation of rotation criteria
We evaluate rotations according to the quality criteria from Section 4, that is, (1) the relevance of individual factors, (2) the separability of the relevant factors, and (3) the sparsity of the factors.
5.1.1 Factor relevance
Here, we measure the impact of individual factors on the segmentation.
In this section, we use the notation ˜Γ to denote a matrix of factor loadings that can be either rotated or unrotated.
A simple measure for the impact of the j -th factor with loadings ˜Γ : ,j ∈ R n is given by the (cid:96) 1 -norm (cid:107) F ( ˜Γ : ,j ) (cid:107)of the factor-speciﬁc ﬂow probabilities.
In what follows, we consider relevance curves that show how many factors exceed the overall uncertainty for varying thresholds τ ≥ 0 .
Speciﬁcally, we compute n τ = |
R τ | , where R τ = { j : (cid:107) F ( Γ : ,j ) (cid:107) 1 ≥ τ (cid:107) F full ( Γ ) (cid:107) 1 } , and we measure the overall uncertainty by the (cid:96) 1 -norm of the full ﬂow probabilities, approximated by 100 Monte-Carlo samples.
Results and discussion.
The results of averaging n τ over the respective test images are shown in
Figure 3 (top row).
First, classic rotations barely reduce the number of relevant factors compared to the unrotated representation.
This is no surprise since they do not take the mean logits into account and only try to simplify the structure of the factor loadings ˜Γ .
Nevertheless, even classic rotations already seem to decrease redundancy.
However, as intended by design, FP rotations reduce the number of relevant factors to a much greater extent.
Especially for LIDC and SEN12MS, already small thresholds τ are sufﬁcient to cut off most factors below the threshold: Figure 3 (top) shows that all FP rotations behave similarly with curves declining sharply for small τ .
Consequently, FP rotations tend to produce a huge gap between a small number of relevant factors and the remaining ones, see
Figure 1 for a visual example.
This is desirable since it allows to focus only on a few relevant and meaningful factors during the exploration of the predicted uncertainty.
It may be harder to ﬁnd such factors if the predicted uncertainty has less inherent structure.
CamVid is an example in this regard as uncertainty predictions are often restricted to class borders, which means that they are less spatially correlated.
However, even for CamVid, there is structured uncertainty, see Figure 13 (Section D.2.3) in the supplement.
Figure 3 (top) shows that also for CamVid, FP rotations signiﬁcantly reduce the number of relevant factors.
5.1.2 Separability of relevant factors
The second quality criterion from Section 4 concerns factor separation.
Here, for a separation threshold ρ ∈ [0 , 1] , we compute the largest possible fraction of pairwise separated relevant factors:
[Equation]
for all
[Equation]
(cid:54) =
[Equation]
∈ J } ∈ [0 , 1] .
If n τ = 0 , we set s τ ( ρ ) = 0 for all ρ .
The separation of two factors is measured by the cosine similarity of their factor-speciﬁc ﬂow probabilities, which is always non-negative since corresponding entries cannot have opposing signs.
For s τ ( ρ ) , a value of one is best since it means that all relevant factors are also separated.
For ﬁxed relevance thresholds τ , we also compute the area under the curve
AUC ( s τ ) for the comparison of different rotation criteria.
Results and discussion.
In Figure 3 (bottom row), we show the separation scores AUC ( s τ ) for different relevance thresholds τ , respectively averaged over all test images.
FP rotations consistently beat classic rotations by a factor of around two in terms of AUC.
Classic rotation criteria are still better than the unrotated representations (which form the real baseline).
The AUC separation scores drop for thresholds τ that fail to determine the number of relevant factors sensefully because they are too small or too large.
Notably, for classic rotation criteria, the separation scores AUC ( s τ ) respectively peak at a threshold τ for which the number of relevant factors nearly coincides with the one from FP rotations, compare the intersection of the curves in Figure 3 (top row).
The peak of the separation scores is less pronounced for FP rotations, particularly for LIDC and SEN12MS, where the set of relevant factors is more stable across different thresholds τ .
For SEN12MS, the results also distinguish among the FP-rotation criteria, where FP-Quartimax seems to be slightly favored over the
[Figure]
other FP rotations.
This may be because Quartimax emphasizes row sparsity the most, which reduces cosine similarities.
We investigate row sparsity further in the next section.
5.1.3 Factor sparsity
To evaluate to which degree different factors affect the same regions of the input image, we measure row sparsity of the factor-speciﬁc ﬂow probabilities F ( ˜Γ ) ∈ [ − 1 , 1] ( whc ) × r .
For that, for a (row) vector v ∈ R r let
[Equation]
be the Hoyer measure [19], where values close to one indicate a high degree of sparsity.
For us, sparsity only matters in rows with actual uncertainty for the pixel/class, therefore we additionally weigh each row proportional to its (cid:96) 1 -norm.
Hence, as a ﬁnal measure, we compute the weighted
Hoyer measure
[Equation]
Results and discussion.
FP rotations generally concentrate the uncertainty for single regions/classes in only few components, see Table 1.
This means that FP rotation yield the most disentangled uncertainty components, which also indicates strong separation.
For LIDC and SEN12MS, the amount of predicted uncertainty varies greatly across test images, causing high standard deviations.
However, in general, large correlating uncertainty components can be found, allowing high row sparsity.
This is in contrast to CamVid, where uncertainty is typically predicted for class borders.
[Table]
5.2 Fine-grained sampling
Monteiro et al.
[31] already manipulated samples post-hoc by simple linear inter- or extrapolation w.r.t.
the mean.
However, they noted that additional more ﬁne-grained sample control is necessary for a systematic exploration of the sample space: The interpolation approach lacks a solid foundation in the uncertainty model, and it relies on having a useful sample to start with.
The meaningful control variables that we obtain by rotating factors provide all that has been missing.
They enable users to systematically explore the sample space by ﬁne-grained sampling: Starting from the mean prediction, they can inspect alternatives, correct possible mistakes, and ﬁne-adjust borders.
Particularly, they can
F a c t o r F a c t o r
F a c t o r
F a c t o r manipulate the contribution of individual uncertainty components by manually setting the values of the corresponding factors.
Pseudo-samples obtained in this way are shown in Figure 4.
Alongside this paper, we provide an interface for ﬁne-grained sampling.
It allows the selection of a rotation criterion for a given input image (we recommend FP-Quartimax for a start), and control variables can be set conveniently using sliders.
In the supplement, we provide some visuals.
Discussion and conclusion
In this work, we interpreted the uncertainty model of stochastic segmentation networks (SSNs) as a factor model, which provides control variables for ﬁne-grained sampling as requested by the authors of [31].
By (re-)structuring the uncertainty using rotations, we improved the controls and obtained as few as possible, but as many as necessary relevant uncertainty components.
Here, it turned out that rotation criteria based on ﬂow probabilities yield the most meaningful controls, where ﬂow probabilities are a new quantiﬁcation and visualization technique for the uncertainty in SSNs.
Our controls allow to systematically explore the predicted uncertainty and to ﬁne-adjust samples.
However, the exploration of the sample space is only useful if the overall predicted uncertainty makes sense.
This can be ensured by proper training.
Structuring and examining the uncertainty is especially useful if there is a signiﬁcant amount of aleatoric uncertainty.
We note that one limitation caused by our current implementation is that the computation of ﬂow-probability based rotations may take too long for performing it in an interactive scenario.
However, there is signiﬁcant potential for improving the optimization (scheme and parallelization).
In any case, rotations should be precomputed whenever possible.
Overall, we see a broader impact of our approach, which extends beyond the scope of SSNs.
For instance, we believe that it could be used for large-scale image classiﬁcation, where factor models have recently been employed for modeling class correlations [8].
Another promising application is to learn and inspect more structured latent spaces in (variational) autoencoders [12].
Structuring uncertainty as we do may be useful whenever a multivariate Gaussian forms part of a model.
Next, ﬂow probabilities can also be used for other probabilistic segmentation architectures that have a mean prediction as a reference point.
Notably, ﬂow-probability overview plots for the predicted uncertainty keep class-speciﬁc information, in contrast to other overview plots like entropy.
To sum up, it is often easier to understand the whole in terms of smaller parts.
In this light, we structured the predicted uncertainty of SSNs into meaningful smaller uncertainty components.
Jointly, they enable ﬁne-grained sample control, so for us, the sum of the parts is also greater than the whole.
Acknowledgements
We thank Prof. Dr.-Ing.
Joachim Denzler for helpful comments and Ferdinand Rewicki for checking our work multiple times.
We also thank all anonymous reviewers for their insightful feedback.