On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice
Abstract
We study two of the most popular performance metrics in medical image segmentation, Accuracy and Dice, when the target labels are noisy.
For both metrics,several statements related to characterization and volume properties of the set ofoptimal segmentations are proved, and associated experiments are provided.
Ourmain insights are: (i) the volume of the solutions to both metrics may deviatesigniﬁcantly from the expected volume of the target, (ii) the volume of a solutionto Accuracy is always less than or equal to the volume of a solution to Dice and(iii) the optimal solutions to both of these metrics coincide when the set of feasiblesegmentations is constrained to the set of segmentations with the volume equal tothe expected volume of the target.
Introduction
One of the most central problems in medical image analysis is to identify the region of an imageassociated with a certain target structure.
This problem, referred to as image segmentation ordelineation, is often very time consuming to solve manually.
Consequently, there is great interestin the development of methods that can assist in automation of the procedure.
Since 2015, it hasbecome increasingly popular to address the segmentation problem using machine learning basedmethods, and in particular, fully convolutions neural networks with U-net architecture.
Such methodscommonly dominate the winning submissions to segmentation contests and are backed by a largebase of supporting literature [27, 2, 7, 9, 13, 19, 30]
Despite the success, performance of these methods will, like any machine learning method, dependon the quality of the available data [35, 5].
Since it is well known that the data commonly used inpractice is produced by medical practitioners that delineate structures in an inconsistent manner [24],it is important to understand the impact of label noise.
One way to study the inﬂuence of labelnoise is to consider how the noise impacts segmentations that are theoretically optimal with respectto the metric used for measuring performance.
Even if theoretically optimal solutions may not beattainable in practice, studying them gives important insights into what the effect label noise has onthe particular metric and any training method that is desgined to maximize it.
Arguably, the most simple and classical choice of metric is Accuracy; the fraction of the imagethat is correctly delineated.
This metric is most commonly targeted by taking a 1 / 2 -threshold ofpredictions from a model trained with the cross-entropy loss [23].
However, since Accuracy maynot reﬂect the desired behaviour when the data is unbalanced, that is, when the target structure ismuch smaller than the background, alternative metrics are often preferred.
The most popular suchalternative is the Sørensen-Dice coefﬁcient, or Dice for short and is related to the F 1 -metric used inbinary classiﬁcation.
This metric is most commonly targeted by taking a 1 / 2 -threshold of predictions from a model trained with the soft-Dice loss, a smoothed version of the Dice metric [23].
Otherexamples considered in the literature include the Jaccard index and variations of the Haussdorffmetric [31].
In this work, we conduct a theoretical investigation of the effect label noise has on optimal segmentations with respect to the performance metrics Accuracy and Dice.
Because the volume of a proposedsegmentation may be used for important properties such as estimating the size of a tumor [10], wepay special attention to the effect noise may have on the volume of the optimal segmentations.
Contributions: A characterization of all of optimal solutions to Accuracy and Dice when the targetis noisy is provided.
This characterization is used to analyze the volume of the optimal solutions andwe prove: (i) sharp upper and lower bounds on the volume of optimal segmentations with respect to
Accuracy and Dice, (ii) that the volume of the optimal solutions to Accuracy always is less or equalto the volume of the optimal solutions to Dice and (iii) that the optimal solutions to both metricscoincide when the volume is held ﬁx.
We also show the relevance of the problem in a practical settingby including experiments on data from the Gold Atlas project [24] and The Lung Image Database
Consortium (LIDC) and Image Database Resource Initiative (IDRI) [1].
Related work
Deep learning methods is playing an increasingly vital part in the development of medical imageanalysis.
However, deep learning models require large annotated data sets for successful trainingthat rarely are available in the clinics.
Even if some data sets are being curated for training deeplearning models, it is generally difﬁcult and expensive to accurately annotate large collections ofmedical images.
Moreover, the data sets are often small from the outset, due to privacy issues andinstitutional policies, which makes it difﬁcult to share data across borders.
Moreover, training datamay include corrupted or noisy labels.
This is particularly the case in image segmentation wheredifferent annotators may have different views on the correct delineation of a region of interest, leadingto uncertainty about the true label, see e.g. [4, 22].
Noisy labels may also appear due to automatedsystems or non-expert systems being used to annotate large volumes of data, see [26, 6].
There is a large body of literature on the impact of label noise in image segmentation, see [32] and[14] for recent reviews.
The proposed solutions to limit the loss of performance when the labels arenoisy include label cleaning and pre-processing, e.g. [11], modiﬁcation of network architechtures, e.g.[34], robustiﬁcation of loss functions, e.g. [20], reweighting of training data, see [37, 21], and manyothers.
These approaches are of practical nature and generally address methodology that improve theperformance on some chosen noisy data set.
On the contrary, the literature that address the effect oflabel noise from a theoretical point of view in the context of image segmentation is rather limited.
Itwas shown that the loss function soft-Dice, in contrast to the loss function cross-entropy, does notyield optimal predictions that coincide with the pixel-wise marginals and that the associated volumeis biased [2, 3].
This motivated methods for post-calibrating uncalibrated marginal estimates [28] anda more general investigation of the relationship between volume and marginal calibration [25].
Another domain of related work can be found in the binary classiﬁcation literature.
From a technical point of view, there is a connection between the study of solutions to Accuracy and Dice insegmentation and the study of Accuracy and F 1 in binary classiﬁcation [23].
Of importance to us isoptimal plug-in classiﬁers to Accuracy and F 1 .
That is, classiﬁers that are obtained by processing theposterior class probabilities or estimates thereof using a threshold.
For Accuracy, this relates to theclassical Bayes classiﬁer which has been studied since the origin of the ﬁeld [33].
Early work onthe existence of such a classiﬁer for F 1 , and the fact that this threshold is lower than the thresholdfor Accuracy can be traced to [36].
Further work showed this threshold to be equal to half of themaximal attainable F 1 -score [18].
Lots of extensions of these works have been proposed, but to thebest of our knowledge no such extension is in the direction of our work.
Preliminaries
In our work we ﬁnd that it is convenient to do the theoretical analysis over a continuous domain.
Formally, let Ω =
[0 , 1] n ⊂ R n be the unit cube of dimension n ≥ 1 and λ be the associated standardnormalized Lebesgue measure such that λ (Ω) = 1 .
The space of segmentations is denoted by S and is the space of measurable functions from Ω to the binary numbers { 0 , 1 } .
For any segmentations ∈ S , s ( w ) = 1 implies that the object of interest occupies the site ω ∈ Ω .
In a numerical setting, adiscretization of the domain Ω is commonly used.
For instance, 3D CT scans are often representedby a three dimensional voxel grid of an approximate order of 512 × 512 × 100 .
In such situations thespace of segmentations are given by all possible binary functions deﬁned on this voxel grid.
Notethat any segmentation on a discretized domain can be incorporated in our continuous framework byusing appropriately normalized step functions.
Details on this can be found in the end of Section 4.
Beyond the space of segmentations, several other technical constructions are introduced.
Thisincludes the space of measurable functions from Ω to [0 , 1] which we denote by M and refer to as themarginal functions .
We also let | f | .
= (cid:82) Ω | f ( ω ) | λ ( dω ) be the volume of f and ¯ f .
= 1 − f , where f isany measurable function deﬁned on Ω .
Throughout we adopt the convention that two λ -measurablefunctions f, g are equal if they are equal λ -a.e.
We will use I ( · ) to denote the identity function, andwhen F is a cumulative distribution function, we will denote the left limit F ( t − ) =
lim s ↑ t F ( s ) .
Finally, for a given volume v ∈ [0 , 1] , we let S v .
= { s ∈ S , | s | = v } be the set of segmentations withvolume v .
Classically, metrics in medical image segmentation are deﬁned per image as functionals over twodeterministic segmentations [31].
When noise is present, the label becomes a random variableand the metrics need to be extended to a functional over one deterministic segmentation and onerandom label segmentation.
In this work, the soft labeling convention for this extension is adopted[15, 12, 29, 16, 17].
Also, since we do our analysis with respect to measurable functions on acontinuous domain instead of functions on a ﬁnite index set, we replace the sums usually used with
Lebesgue integrals.
This, however, does not change the intuition of the metrics, and the deﬁnitionusing sums can be seen a special case.
Deﬁnition 1.
For any m ∈ M , Accuracy is given by
[Equation]
Deﬁnition 2.
For any m ∈ M , Dice is given by
[Equation]
For a noisy segmentation L , that is, a random variable taking values in S , m can be taken to bethe exact marginal success probability m ( ω ) =
E [ L ( ω )] , ω ∈ Ω .
Such marginal functions areimportant in theory but can seldom be obtained in practice.
Alternative choices of marginal functionsinclude ﬁnite sample approximations, that is, point-wise averages over ﬁnite observations of L , andestimates of m according to a single annotator [15].
These choices of m are important because theyare sometimes used for training machine learning models.
Finally, also the predictions of machinelearning models can be used as marginal functions.
Examples of observations of a particular L for acouple of different target structures are depicted in Figure 1.
Because of the fact that | m | =
E [ | L | ] when m is taken to be the exact marginal success probability,|
m | plays an important role in the medical segmentation context, either theoretically as the expectedvolume of the target or as an approximation thereof.
Understanding how | m | relates to | s | , wheres is an optimal segmentation to Accuracy or Dice will be central in our work.
To the best of ourknowledge, this has not been studied in prior work.
Main results
The objective of our analysis is to characterize the optimizers to A m and D m and give a detaileddescription of the volume of the optimal segmentations.
That is, for a given m ∈ M , identifyproperties (e.g. volume) of the optimal segmentation s ∈ S that maximimize Accuracy or Dice.
Tothis end, consider the probability measure on [0 , 1] given by the push-forward measure λ ◦ ¯ m − 1 ( · )and let F m denote its cumulative distribution function,
[Equation]
The function F m ( t ) may be interpreted as the volume of the set of sites with non-success probabilityless than or equal to t .
In other words, the volume of the sub-level set of ¯ m at level t .
Since F mis the cumulative distribution function of a probability distribution on [0 , 1] , it has several wellknown properties making it easy to work with, e.g., F m is non-decreasing and right-continuous with
F m (0) = 0 and F m (1) = 1 .
Of particular interest to us is that it has a generalized inverse given by
[Equation]
which can be interpreted as the minimum level at which the volume of the corresponding sub-levelset of ¯ m , is at least v .
This function is often referred to as the quantile function and also has severalwell known properties; it is non-decreasing and left-continuous.
Moreover, it allows us to deﬁne thefollowing important class of segmentations for a given m ∈ M .
[Equation]
The described class S m,v is informally the set of segmentations with volume v that assigns 1 to sites
ωwhere m ( ω ) is large.
If t = F − 1 m ( v ) is a continuity point of F m , i.e. λ ( ω : ¯ m ( ω ) =
t )
= 0 , then S m,vonly consist of the elements that are λ -a.e.
equal to the segmentation s ( ω )
=
I { m ( ω ) ≥ 1 − F − 1 m ( v ) } .
A lot of our analysis can be simpliﬁed if F m is assumed to be invertible almost everywhere.
However,this would require m to not have any non-neglible constant regions, which for instance excludes anym that is given by an empirical approximations using a ﬁnite number of samples.
Consequently, inthe sequel, we treat general F m .
Our ﬁrst result contains the essential ingredients for characterizingthe optimizers to Accuracy and Dice.
Lemma 1.
For any m ∈ M and v ∈ [0 , 1]
[Equation]
and the elements where the supremum is attained is given by S m,v .
A complete proof is given in the Supplementary Material and outlined as follows.
The ﬁrst partshows that the class S m,v is the class of optimal solutions by showing that for any s ∗ ∈ S m,v ands ∈ S v \ S m,v , (cid:82) Ω ( s ∗ ( ω ) − s ( ω )) m ( ω ) λ ( dω ) > 0 .
The second part proves the equality (6) using anapplication of the quantile transform.
That is, if U has uniform distribution on [0 , 1] then F − 1 m ( U )has cdf given by F m and (cid:82) 1 0 tF m ( dt ) =
E [ F − 1 m ( U )] =
(cid:82) 1 0 F − 1 m ( u ) du .
Lemma 1 allows us to reduce the constrained optimization problem over the rather complicated spaceof segmentations, to a one-dimensional integral with respect to the quantile function.
It is the startingpoint for our analysis.
In the remaining section our main theoretical results are presented.
In Theorem 1, we provide acharacterization of all of the optimal solutions to Accuracy based on volume.
In addition, sharp upperand lower bounds on the volume of the associated optimal segmentations are provided.
Theorem 1.
For any m ∈ M , the class of maximizers to A m is given by ∪ v ∈V A m S m,v where
[Equation]
Moreover, V A m satisﬁes the following bounds
[Equation]
where the bounds are sharp in the sense that there for any v ∈ [0 , 1] exist m 0 , m 1 ∈ M such that| m 0 | = | m 1 | = v and
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First, thefunction
[Equation]
is introduced and then Lemma 1 is used to show that sup s ∈S A m
[Equation] s ) = sup [Equation]
Consequently, the class of optimal solutions to A m is given by ∪ v ∈V A m S m,v , where V A m is the set ofoptimizers to a m .
The rest of the proof consists of detailed analysis of a m and is composed of threeparts.
The ﬁrst part is to show (7) by ﬁnding one optimal solution and then identifying all volumesthat yield the same optimal value.
The second part is to provide the lower and upper bounds on theelements of V A m in terms of | m | given by (8).
The third part is to provide examples of situationswhere the extreme cases occur 9.
In Figure 2, a case that is extreme both in the lower and in the uppersense is illustrated.
[Figure]
In Theorem 2, we provide a characterization of all of the optimal solutions to Dice based on volume.
In addition, sharp upper and lower bounds on the volume of the associated optimal segmentations areprovided.
Theorem 2.
For any m ∈ M , the class of maximizers to D m is given by ∪ v ∈V D m S m,v where
[Equation]
Moreover, V D m satisﬁes the following bounds
[Equation]
(12) where the bounds are sharp in the sense that there for any v ∈ (0 , 1] exist m 0 , m 1 ∈ M such {+that| m 
+}
| = | m 1 | = v and
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First, thefunction
[Equation]
(14) is introduced and then Lemma 1 is used to show that sup s ∈S D m ( s ) = sup v ∈ [0 , 1]
d m ( v ) .
Consequently, the class of optimal solutions to D m is given by ∪ v ∈V D m S m,v , where V D m is the set ofoptimizers to d m .
The remaining proof consists of detailed analysis of d m and composed of threeparts.
The ﬁrst part is to show (11) which is derived by careful investigation of the properties of thefunction δ ( v ) =
( | m | + v ) 2∂ v d m ( v ) , which has the same sign as ∂ v d m ( v ) and therefore can be usedto identify optimal values of d m .
The second part is to provide the lower and upper bounds on theelements of V D m in terms of | m | given by (12).
The third part is to provide examples of situationswhere the extreme values occur (13).
In Figure 3, a case that is extreme both in the lower and in theupper sense is illustrated.
[Figure]
and consequently is an extreme case to (13) in both the lower and the upper sense.
In Theorem 3, we relate the volume of the optimal segmentations of Accuracy and the optimalsegmentations of Dice for a given marginal probability m ∈ M .
Theorem 3.
For any m ∈ M , V A m given by (7) and V D m given by (11) satisfy
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First notethat D m ( s ) ≤ 1 for any s ∈ S and then consider separately the cases when D m ( s ) < 1 for alls ∈ S and when there exist some s ∈ S such that D m ( s ) = 1 .
For the ﬁrst case, it is obviousthat
sup s ∈S D m ( s ) / 2 < 1 / 2 which implies that F m (1 / 2) ≤ F m ((1 − sup s ∈S D m ( s ) / 2) − ) .
Forthe second case, we show that the volume of the optimizers are uniquely given by V A m =
V D m =
{ F m (1 / 2) } .
In either case, (15) holds.
In Theorem 4, the set of optimal solutions to Accuracy and Dice when constrained to a speciﬁcvolume is shown to coincide.
Theorem 4.
For any m ∈ M and v ∈ [0 , 1] the maximizers to the problems,
[Equation]
coincide and are given by S m,v .
The complete proof is given in the Supplementary Material and is a straightforward applicationof Lemma 1 Of particular interest is the case when v = | m | , since this correspond to the situationwhen the metrics are maximized under the constraint that there should be no volume bias.
It follows from Theorem 1 and Theorem 2 that the optimizers to both Accuracy and Dice are of theform ∪ v ∈V S m,v , where V =
[ F m ( t − ) , F m ( t )] for some t ∈ [0 , 1] .
This type of charecterization is practical for proving properties on volume, but inconvenient for other tasks.
In Theorem 5, we providean alternative charecterization using threshold segmentations of the form s ( ω ) =
I { m ( ω ) > α } ors ( ω ) =
I { m ( ω ) ≥ α } , for some α .
Even if there exist optimal segmentations that are not necessarilyof threshold type, they can always be bounded, above and below, by optimal segmentations ofthreshold type.
Theorem 5.
For any m ∈ M and t ∈ (0 , 1] , let s 1 ( ω )
=
I { m ( ω ) ≥ 1 − t } and s 0 ( ω ) = I { m ( ω ) >1 − t } .
Then, | s 0 | =
F ( t − ) , | s 1 |
=
F ( t ) and for each s ∈ S m,v , v ∈ [ F ( t − ) , F ( t )] ,
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First notethat | s 0 |
= (cid:82) Ω I { ¯ m ( ω ) < t } λ ( dω ) =
F m ( t − ) and | s 1 |
=
(cid:82) Ω I { ¯ m ( ω ) ≤ t }
λ ( dω ) =
F m ( t ) .
For theupper bound, with A = { ω : s ( ω ) > s 1 ( ω ) } , we ﬁrst observe that I { ω ∈ A } = s ( ω )¯ s 1 ( ω ) and then,using the deﬁnition of S m,v we prove that
[Equation]
The lower bound is similar, but slightly more involved.
In numerical applications, the continuum Ω is usually partitioned into a ﬁnite collection of voxels{ Ω i } i ∈I .
Marginal functions are then constrained to the subset of M that is compatible with thevoxelization in the sense that m is measurable with respect to the σ -ﬁeld generated by the partition.
Note that if s 1 ( ω ) =
I { m ( ω ) ≥ 1 − t } and s 0 ( ω ) = I { m ( ω ) > 1 − t } for some t ∈ (0 , 1] , thenalso s 0 and s 1 are compatible with the voxelization.
By Theorem 1 (Theorem 2) and Theorem 5,the segmentations with least and greatest volume that are optimal with respect to Accuracy (Dice)are compatible with the voxelization, and, consequently, it is not possible to obtain tighter boundsthan (8) by restricting the class of segmentations to the class of segmentations compatible with thevoxelization.
For Accuracy and Dice respectively, we denote the segmentations with the greatest volume by:
[Equation]
Note that s A m is analogous to the Bayes classiﬁer and s D m is analogous to the threshold classiﬁerdescribed in [18].
Both of these are trivial to compute from a given marginal function m compatibleto some voxelization.
Experiments
The sharp bounds on volume in Thereom 1 and Theorem 2 implies that there exist marginal functionsfor which the volume of the optimal segmentations to Accuracy and Dice deviate signiﬁcantly fromthe expected target volume.
In this section we conduct experiments on marginal functions formedfrom real world data to compare the volume of optimal segmentations to the expected volume inpractice.
For our experiments we investigate two data sets.
The ﬁrst data set (G) contains segmentations inthe pelvic area and is part of the Gold Atlas project [24].
The data is in 3D with a resolution of512 × 512 pixels per slice and consist of 19 patients with 9 different ROI’s (region of interest), eachof which have been delineated by 5 experts (see Figure 1 for an illustration of the segmentationsassociated with one patient and ROI). The second data set (L) contains segmentations in the thoraxarea and is part of The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) [1] and is hosted by TCIA [8].
The data is in 3D with a resolution of 512 × 512pixels per slice and contains 1018 cases with lung nodules delineated by 4 experts.
For each dataset, ROI and patient, a marginal function m is formed by taking the fraction of which each pixel hasbeen selected by the annotators, that is, a ﬁnite sample approximation is considered.
The resultingmarginal functions are then used to compute the segmentations s A m (17) and s D m (18).
Detailson the experiment, implementation and how to obtain the data is available in the Supplementary Material.
[Table]
|/|
[Table]
[Table]
Anal
[Table] |/ [Table] |/|
[Table]
|/
[Table]
[Figure]
[Figure] [Table] From our experiments we report the quantities | s A m | / | m | and | s D m | / | m | , which in a relative sensedescribe how much the volume of the computed optimal segmentations with respect to Accuracy and Dice deviate from the expected target volume.
In Table 1, aggregated statistics of these quantitieswith respect to all patients are shown. In Figure 4 and Figure 5, these quantities are illustrated foreach patient and ROI in scatter plots.
By simple inspection it is clear that the volume of optimalsegmentations to Accuracy and Dice often deviate signiﬁcantly from the expected target volume.
For (G) the extreme cases are given by some marginal function m for which | s A m | / | m | ≈ 1 / 2and some marginal function m for which | s D m | / | m | ≈ 1 . 5 .
For (L) the extreme cases are givenby some marginal function m for which | s A m | / | m | ≈ 0 and some marginal function m for which| s D m | / | m | ≈ 4 .
Conclusion
In this work, we have theoretically investigated the optimal segmentations with respect to theperformance metrics Accuracy and Dice.
We have given a detailed rigorous characterization of theoptimizers and upper and lower bounds on the volume of optimal segmentations.
Finally, we haveshown the relevance of our theoretical observations in practice by comparing the volume of optimalsegmentations with respect to the performance metrics to the expected volume, on two real worlddata set.
We conclude that noise may cause optimal segmentations to have a volume that deviatessigniﬁcantly from the expected target volume and that the reason for this may be what metric ischosen, or implicitly, what metric a chosen training method targets.
Broader impacts: Formalizing the evaluation process of automated segmentation methods can bedone in many ways, each with its pros and cons.
Even if this work can be interpreted as describing theproblems with using Dice for this formalization, it can still paradoxically contribute to an unhealthyﬁxation of Dice as the gold standard for segmentation evaluation in medical image analysis.
Thisin turn can lead to that medical practitioners put two much faith in models that have been shown toperform well with respect to the metric on some test data.
One solution to this is to make sure thatclinical practitioners using such models are educated in the problems associated with the metric.
Limitations:
In order for the volume bounds to be sharp in Theorem 1 and Theorem 2, weconstruct extreme cases of m .
These extreme cases might only be representable approximately withstep functions for a particular choice of voxelization.
Consequently, the most extreme cases wecan construct in a numerical setting may not be as extreme as those we have constructed in thecontinuous setting.
However, in medical image analysis it is common to deal voxelizations of theorder of 512 × 512 × 100 voxels which means that the approximation error would be negligable.
Ourwork is also limited by the amount of experiments included.
Additional numerical experiments on awider range of data sets would give a more comprehensive picture on the impact of different noisedistributions.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] Our main claims are informal descriptions of the
statements we prove in the Main results section.
(b) Did you describe the limitations of your work?
[Yes] See Limitations in the end of theConclusions section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeBroader impacts in the Discussion section.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Generalitiesare described in the preliminaries and the rest in the Main results section.(b)
Did you include complete proofs of all theoretical results?
[Yes] The, proofs are listedin the Supplementary and a short scetch is provided after each stated theorem.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All these detailsare found in the Supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A] .
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
No random experiments of this kind was conducted.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Our experiments does not requireany heavy computation.
It can be done on a regular laptop.
We mention this in theSupplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Several times with
(b) Did you mention the license of the assets?
[Yes] Yes, in the Supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes, code is included in the Supplementary Material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Yes, see the Supplementary Material.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] Yes, see the Supplementary Material.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]