On Image Segmentation With Noisy Labels: Characterization and Volume Properties of the Optimal Solutions to Accuracy and Dice
Abstract
We study two of the most popular performance metrics in medical image segmentation, Accuracy and Dice, when the target labels are noisy.
For both metrics,several statements related to characterization and volume properties of the set ofoptimal segmentations are proved, and associated experiments are provided.
Ourmain insights are: (i) the volume of the solutions to both metrics may deviatesigniﬁcantly from the expected volume of the target, (ii) the volume of a solutionto Accuracy is always less than or equal to the volume of a solution to Dice and(iii) the optimal solutions to both of these metrics coincide when the set of feasiblesegmentations is constrained to the set of segmentations with the volume equal tothe expected volume of the target.
Introduction
One of the most central problems in medical image analysis is to identify the region of an imageassociated with a certain target structure.
This problem, referred to as image segmentation ordelineation, is often very time consuming to solve manually.
Consequently, there is great interestin the development of methods that can assist in automation of the procedure.
Since 2015, it hasbecome increasingly popular to address the segmentation problem using machine learning basedmethods, and in particular, fully convolutions neural networks with U-net architecture.
Such methodscommonly dominate the winning submissions to segmentation contests and are backed by a largebase of supporting literature [25, 1, 6, 7, 11, 17, 28]
Despite the success, performance of these methods will, like any machine learning method, dependon the quality of the available data [33, 4].
Since it is well known that the data commonly used inpractice is produced by medical practitioners that delineate structures in an inconsistent manner [22],it is important to understand the impact of label noise.
One way to study the inﬂuence of labelnoise is to consider how the noise impacts segmentations that are theoretically optimal with respectto the metric used for measuring performance.
Even if theoretically optimal solutions may not beattainable in practice, studying them gives important insights into what the effect label noise has onthe particular metric and any training method that is desgined to maximize it.
Arguably, the most simple and classical choice of metric is Accuracy; the fraction of the imagethat is correctly delineated.
This metric is most commonly targeted by taking a 1 / 2 -threshold ofpredictions from a model trained with the cross-entropy loss [21].
However, since Accuracy maynot reﬂect the desired behaviour when the data is unbalanced, that is, when the target structure ismuch smaller than the background, alternative metrics are often preferred.
The most popular suchalternative is the Sørensen-Dice coefﬁcient, or Dice for short and is related to the F 1 -metric used inbinary classiﬁcation.
This metric is most commonly targeted by taking a 1 / 2 -threshold of predictions from a model trained with the soft-Dice loss, a smoothed version of the Dice metric [21].
Otherexamples considered in the literature include the Jaccard index and variations of the Haussdorffmetric [29].
In this work, we conduct a theoretical investigation of the effect label noise has on optimal segmentations with respect to the performance metrics Accuracy and Dice.
Because the volume of a proposedsegmentation may be used for important properties such as estimating the size of a tumor [8], we payspecial attention to the effect noise may have on the volume of the optimal segmentations.
Contributions: A characterization of all of optimal solutions to Accuracy and Dice when the targetis noisy is provided.
This characterization is used to analyze the volume of the optimal solutions andwe prove: (i) sharp upper and lower bounds on the volume of optimal segmentations with respect to
Accuracy and Dice, (ii) that the volume of the optimal solutions to Accuracy always is less or equalto the volume of the optimal solutions to Dice and (iii) that the optimal solutions to both metricscoincide when the volume is held ﬁx.
We also show the relevance of the problem in a practical settingby including experiments on data from the Gold Atlas project [22].
Related work
Deep learning methods is playing an increasingly vital part in the development of medical imageanalysis.
However, deep learning models require large annotated data sets for successful trainingthat rarely are available in the clinics.
Even if some data sets are being curated for training deeplearning models, it is generally difﬁcult and expensive to accurately annotate large collections ofmedical images.
Moreover, the data sets are often small from the outset, due to privacy issues andinstitutional policies, which makes it difﬁcult to share data across borders.
Moreover, training datamay include corrupted or noisy labels.
This is particularly the case in image segmentation wheredifferent annotators may have different views on the correct delineation of a region of interest, leadingto uncertainty about the true label, see e.g. [3, 20].
Noisy labels may also appear due to automatedsystems or non-expert systems being used to annotate large volumes of data, see [24, 5].
There is a large body of literature on the impact of label noise in image segmentation, see [30] and[12] for recent reviews.
The proposed solutions to limit the loss of performance when the labels arenoisy include label cleaning and pre-processing, e.g. [9], modiﬁcation of network architechtures, e.g.[32], robustiﬁcation of loss functions, e.g. [18], reweighting of training data, see [35, 19], and manyothers.
These approaches are of practical nature and generally address methodology that improve theperformance on some chosen noisy data set.
On the contrary, the literature that address the effect oflabel noise from a theoretical point of view in the context of image segmentation is rather limited.
Itwas shown that the loss function soft-Dice, in contrast to the loss function cross-entropy, does notyield optimal predictions that coincide with the pixel-wise marginals and that the associated volumeis biased [1, 2].
This motivated methods for post-calibrating uncalibrated marginal estimates [26] anda more general investigation of the relationship between volume and marginal calibration [23].
Another domain of related work can be found in the binary classiﬁcation literature.
From a technical point of view, there is a connection between the study of solutions to Accuracy and Dice insegmentation and the study of Accuracy and F 1 in binary classiﬁcation [21].
Of importance to us isoptimal plug-in classiﬁers to Accuracy and F 1 .
That is, classiﬁers that are obtained by processing theposterior class probabilities or estimates thereof using a threshold.
For Accuracy, this relates to theclassical Bayes classiﬁer which has been studied since the origin of the ﬁeld [31].
Early work onthe existence of such a classiﬁer for F 1 , and the fact that this threshold is lower than the thresholdfor Accuracy can be traced to [34].
Further work showed this threshold to be equal to half of themaximal attainable F 1 -score [16].
Lots of extensions of these works have been proposed, but to thebest of our knowledge no such extension is in the direction of our work.
Preliminaries
In our work we ﬁnd that it is convenient to do the theoretical analysis over a continuous domain.
Formally, let Ω =
[0 , 1] n ⊂ R n be the unit cube of dimension n ≥ 1 and λ be the associated standardnormalized Lebesgue measure such that λ (Ω) = 1 .
The space of segmentations is denoted by S andis the space of measurable functions from Ω to the binary numbers { 0 , 1 } .
For any segmentation s ∈ S , s ( w ) = 1 implies that the object of interest occupies the site ω ∈ Ω .
In a numerical setting, adiscretization of the domain Ω is commonly used.
For instance, 3D CT scans are often representedby a three dimensional voxel grid of an approximate order of 512 × 512 × 100 .
In such situations thespace of segmentations are given by all possible binary functions deﬁned on this voxel grid.
Notethat any segmentation on a discretized domain can be incorporated in our continuous framework byusing appropriately normalized step functions.
Details on this can be found in the end of Section 4.
Beyond the space of segmentations, several other technical constructions are introduced.
Thisincludes the space of measurable functions from Ω to [0 , 1] which we denote by M and refer to as themarginal functions .
We also let | f | .
= (cid:82) Ω | f ( ω ) | λ ( dω ) be the volume of f and ¯ f .
= 1 − f , where f isany measurable function deﬁned on Ω .
Throughout we adopt the convention that two λ -measurablefunctions f, g are equal if they are equal λ -a.e.
We will use I ( · ) to denote the identity function, andwhen F is a cumulative distribution function, we will denote the left limit F ( t − ) =
lim s ↑ t F ( s ) .
Finally, for a given volume v ∈ [0 , 1] , we let S v .
= { s ∈ S , | s | = v } be the set of segmentations withvolume v .
Classically, metrics in medical image segmentation are deﬁned per image as functionals over twodeterministic segmentations [29].
When noise is present, the label becomes a random variableand the metrics need to be extended to a functional over one deterministic segmentation and onerandom label segmentation.
In this work, the soft labeling convention for this extension is adopted[13, 10, 27, 14, 15].
Also, since we do our analysis with respect to measurable functions on acontinuous domain instead of functions on a ﬁnite index set, we replace the sums usually used with
Lebesgue integrals.
This, however, does not change the intuition of the metrics, and the deﬁnitionusing sums can be seen a special case.
Deﬁnition 1.
For any m ∈ M , Accuracy is given by
[Equation]
Deﬁnition 2.
For any m ∈ M , Dice is given by
[Equation]
For a noisy segmentation L , that is, a random variable taking values in S , m can be taken to bethe exact marginal success probability m ( ω ) =
E [ L ( ω )] , ω ∈ Ω .
Such marginal functions areimportant in theory but can seldom be obtained in practice.
Alternative choices of marginal functionsinclude ﬁnite sample approximations, that is, point-wise averages over ﬁnite observations of L , andestimates of m according to a single annotator [13].
These choices of m are important because theyare sometimes used for training machine learning models.
Finally, also the predictions of machinelearning models can be used as marginal functions.
Examples of observations of a particular L for acouple of different target structures are depicted in Figure 1.
Because of the fact that | m | =
E [ | L | ] when m is taken to be the exact marginal success probability,|
m | plays an important role in the medical segmentation context, either theoretically as the expectedvolume of the target or as an approximation thereof.
Understanding how | m | relates to | s | , wheres is an optimal segmentation to Accuracy or Dice will be central in our work.
To the best of ourknowledge, this has not been studied in prior work.
Main results
The objective of our analysis is to characterize the optimizers to A m and D m and give a detaileddescription of the volume of the optimal segmentations.
That is, for a given m ∈ M , identifyproperties (e.g. volume) of the optimal segmentation s ∈ S that maximimize Accuracy or Dice.
Tothis end, consider the probability measure on [0 , 1] given by the push-forward measure λ ◦ ¯ m − 1 ( · )and let F m denote its cumulative distribution function,
[Equation]
The function F m ( t ) may be interpreted as the volume of the set of sites with non-success probabilityless than or equal to t .
In other words, the volume of the sub-level set of ¯ m at level t .
Since F m is the cumulative distribution function of a probability distribution on [0 , 1] , it has several wellknown properties making it easy to work with, e.g., F m is non-decreasing and right-continuous with
F m (0) = 0 and F m (1) = 1 .
Of particular interest to us is that it has a generalized inverse given by
[Equation]
which can be interpreted as the minimum level at which the volume of the corresponding sub-levelset of ¯ m , is at least v .
This function is often referred to as the quantile function and also has severalwell known properties; it is non-decreasing and left-continuous.
Moreover, it allows us to deﬁne thefollowing important class of segmentations for a given m ∈ M .
[Equation]
The described class S m,v is informally the set of segmentations with volume v that assigns 1 to sites
ωwhere m ( ω ) is large.
If t = F − 1 m ( v ) is a continuity point of F m , i.e. λ ( ω : ¯ m ( ω ) =
t )
= 0 , then S m,vonly consist of the elements that are λ -a.e.
equal to the segmentation s ( ω )
=
I { m ( ω ) ≥ 1 − F − 1 m ( v ) } .
A lot of our analysis can be simpliﬁed if F m is assumed to be invertible almost everywhere.
However,this would require m to not have any non-neglible constant regions, which for instance excludes anym that is given by an empirical approximations using a ﬁnite number of samples.
Consequently, inthe sequel, we treat general F m .
Our ﬁrst result contains the essential ingredients for characterizingthe optimizers to Accuracy and Dice.
Lemma 1.
For any m ∈ M and v ∈ [0 , 1]
[Equation]
and the elements where the supremum is attained is given by S m,v .
A complete proof is given in the Supplementary Material and outlined as follows.
The ﬁrst partshows that the class S m,v is the class of optimal solutions by showing that for any s ∗ ∈ S m,v and s ∈ S v \ S m,v , (cid:82) Ω ( s ∗ ( ω ) − s ( ω )) m ( ω ) λ ( dω ) > 0 .
The second part proves the equality (6) using anapplication of the quantile transform.
That is, if U has uniform distribution on [0 , 1] then F − 1 m ( U )has cdf given by F m and (cid:82) 1 0 tF m ( dt ) =
E [ F − 1 m ( U )] =
(cid:82) 1 0 F − 1 m ( u ) du .
Lemma 1 allows us to reduce the constrained optimization problem over the rather complicated spaceof segmentations, to a one-dimensional integral with respect to the quantile function.
It is the startingpoint for our analysis.
In the remaining section our main theoretical results are presented.
In Theorem 1, we provide acharacterization of all of the optimal solutions to Accuracy based on volume.
In addition, sharp upperand lower bounds on the volume of the associated optimal segmentations are provided.
Theorem 1.
For any m ∈ M , the class of maximizers to A m is given by ∪ v ∈V A m S m,v where
[Equation]
Moreover, V A m satisﬁes the following bounds
[Equation]
where the bounds are sharp in the sense that there for any v ∈ [0 , 1] exist m 0 , m 1 ∈ M such that| m 0 | = | m 1 | = v and
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First, thefunction
[Equation]
is introduced and then Lemma 1 is used to show that sup s ∈S
A m ( s ) = sup v ∈ [0 , 1] a m ( v ) .
Consequently, the class of optimal solutions to A m is given by ∪ v ∈V A m S m,v , where V A m is the set ofoptimizers to a m .
The rest of the proof consists of detailed analysis of a m and is composed of threeparts.
The ﬁrst part is to show (7) by ﬁnding one optimal solution and then identifying all volumesthat yield the same optimal value.
The second part is to provide the lower and upper bounds on theelements of V A m in terms of | m | given by (8).
The third part is to provide examples of situationswhere the extreme cases occur 9.
In Figure 2, a case that is extreme both in the lower and in the uppersense is illustrated.
[Figure]
In Theorem 2, we provide a characterization of all of the optimal solutions to Dice based on volume.
In addition, sharp upper and lower bounds on the volume of the associated optimal segmentations areprovided.
Theorem 2.
For any m ∈ M , the class of maximizers to D m is given by ∪ v ∈V D m S m,v where
[Equation]
Moreover, V D m satisﬁes the following bounds
[Equation]
where the bounds are sharp in the sense that there for any v ∈ (0 , 1] exist m 0 , m 1 ∈ M such that
[Equation]
= | m 1 | = v and
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First, thefunction
[Equation]
is introduced and then Lemma 1 is used to show that sup s ∈S D m ( s ) = sup v ∈ [0 , 1]
d m ( v ) .
Consequently, the class of optimal solutions to D m is given by ∪ v ∈V D m S m,v , where V D m is the set ofoptimizers to d m .
The remaining proof consists of detailed analysis of d m and composed of threeparts.
The ﬁrst part is to show (11) which is derived by careful investigation of the properties of thefunction δ ( v ) =
( | m | + v ) 2∂ v d m ( v ) , which has the same sign as ∂ v d m ( v ) and therefore can be usedto identify optimal values of d m .
The second part is to provide the lower and upper bounds on theelements of V D m in terms of | m | given by (12).
The third part is to provide examples of situationswhere the extreme values occur (13).
In Figure 3, a case that is extreme both in the lower and in theupper sense is illustrated.
[Figure]
In Theorem 3, we relate the volume of the optimal segmentations of Accuracy and the optimalsegmentations of Dice for a given marginal probability m ∈ M .
Theorem 3.
For any m ∈ M , V A m given by (7) and V D m given by (11) satisfy
[Equation]
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First notethat D m ( s ) ≤ 1 for any s ∈ S and then consider separately the cases when D m ( s ) < 1 for alls ∈ S and when there exist some s ∈ S such that D m ( s ) = 1 .
For the ﬁrst case, it is obviousthat
sup s ∈S D m ( s ) / 2 < 1 / 2 which implies that F m (1 / 2) ≤ F m ((1 − sup s ∈S D m ( s ) / 2) − ) .
Forthe second case, we show that the volume of the optimizers are uniquely given by V A m =
V D m =
{ F m (1 / 2) } .
In either case, (15) holds.
In Theorem 4, the set of optimal solutions to Accuracy and Dice when constrained to a speciﬁcvolume is shown to coincide.
Theorem 4.
For any m ∈ M and v ∈ [0 , 1] the maximizers to the problems,
[Equation]
coincide and are given by S m,v .
The complete proof is given in the Supplementary Material and is a straightforward applicationof Lemma 1 Of particular interest is the case when v = | m | , since this correspond to the situationwhen the metrics are maximized under the constraint that there should be no volume bias.
It follows from Theorem 1 and Theorem 2 that the optimizers to both Accuracy and Dice are of theform ∪ v ∈V S m,v , where V =
[ F m ( t − ) , F m ( t )] for some t ∈ [0 , 1] .
This type of charecterization ispractical for proving properties on volume, but inconvenient for other tasks.
In Theorem 5, we providean alternative charecterization using threshold segmentations of the form s ( ω ) =
I { m ( ω ) > α } ors ( ω ) =
I { m ( ω ) ≥ α } , for some α .
Even if there exist optimal segmentations that are not necessarilyof threshold type, they can always be bounded, above and below, by optimal segmentations ofthreshold type.
Theorem 5.
For any m ∈ M and t ∈ (0 , 1] , let s 1 ( ω )
=
I { m ( ω ) ≥ 1 − t } and s 0 ( ω ) = I { m ( ω ) >1 − t } .
Then, | s 0 | =
F ( t − ) , | s 1 |
=
F ( t ) and for each s ∈ S m,v , v ∈ [ F ( t − ) , F ( t )] ,
[Equation]
The complete proof is given in the Supplementary Material and is outlined as follows.
First notethat | s 0 |
= (cid:82) Ω I { ¯ m ( ω ) < t } λ ( dω ) =
F m ( t − ) and | s 1 |
=
(cid:82) Ω I { ¯ m ( ω ) ≤ t }
λ ( dω ) =
F m ( t ) .
For theupper bound, with A = { ω : s ( ω ) > s 1 ( ω ) } , we ﬁrst observe that I { ω ∈ A } = s ( ω )¯ s 1 ( ω ) and then,using the deﬁnition of S m,v we prove that
[Equation]
The lower bound is similar, but slightly more involved.
In numerical applications, the continuum Ω is usually partitioned into a ﬁnite collection of voxels{ Ω i } i ∈I .
Marginal functions are then constrained to the subset of M that is compatible with thevoxelization in the sense that m is measurable with respect to the σ -ﬁeld generated by the partition.
Note that if s 1 ( ω ) =
I { m ( ω ) ≥ 1 − t } and s 0 ( ω ) = I { m ( ω ) > 1 − t } for some t ∈ (0 , 1] , thenalso s 0 and s 1 are compatible with the voxelization.
By Theorem 1 (Theorem 2) and Theorem 5,the segmentations with least and greatest volume that are optimal with respect to Accuracy (Dice)are compatible with the voxelization, and, consequently, it is not possible to obtain tighter boundsthan (8) by restricting the class of segmentations to the class of segmentations compatible with thevoxelization.
For Accuracy and Dice respectively, we denote the segmentations with the greatest volume by:
[Equation]
[Equation]
Note that s A m is analogous to the Bayes classiﬁer and s D m is analogous to the threshold classiﬁerdescribed in [16].
Both of these are trivial to compute from a given marginal function m compatibleto some voxelization.
Experiments
The sharp bounds on volume in Thereom 1 and Theorem 2 implies that there exist marginal functionsfor which the volume of the optimal segmentations to Accuracy and Dice deviate signiﬁcantly fromthe expected volume of the target.
In this section we conduct experiments on marginal functionsformed from real world data to compare the volume of optimal segmentations to the expected volumein practice.
The data set we investigate contains segmentations in the pelvic area and is part of the Gold Atlasproject [22].
It is freely availible for reserch and educational purposes.
More speciﬁcally, the data is in 3D with a resolution of 512 × 512 pixels per slice and consist of 18 patients with 9 different
ROI’s (region of interest), each of which have been delineated by several medical practitioners(see Figure 1 for an illustration of the segmentations associated with one patient and ROI).
Foreach ROI in every patient, a marginal function m is formed by taking the pixel-wise average of thedifferent segmentations, that is, a ﬁnite sample approximation is considered.
The resulting marginalfunctions are then used compute the segmentations s A m (17) and s D m (18).
Details on the experiment,implementation and how to obtain the data is available in the Supplementary Material.
[Table]
Results of the experiments are illustrated in Table 1 and Figure 4.
The quantities of interest are| s A m | / | m | and | s D m | / | m | , which in a relative sense describe how much the volume of the computedoptimal segmentations with respect to Accuracy and Dice deviate from the expected volume of thetarget.
In Table 1, aggregated statistics of these quantities with respect to all patients are shown andin Figure 4, these quantities are illustrated for each patient and ROI in scatter plots.
We make threeobservations from the results.
Firstly, there exist a marginal function m for which | s A m | / | m | < 1 / 2and a marginal function m for which | s D m | / | m | > 1 . 5 .
This conﬁrms that the volume of optimalsegmentations to Accuracy and Dice in practice may indeed deviate signiﬁcantly from the expectedvolume of the target.
Secondly, on average there seems to be a tendency that | s A m | / | m | ≤ 1 and| s D m | / | m | ≥ 1 .
An interesting question is if this holds exactly for some important class of marginals.
Thirdly, there seems to be a relationship between the relative volume bias and the shape of the ROI,where large structures that are more spherical like the Urinary bladder are less affected than smallstructures that are less spherical like the Neurovascular bundles.
This could be connected to the factthat larger more spherical structures have greater volume in relation to the boundary than smaller lessspherical structures, and that noise primarily exist on the boundary.
Conclusion
In this work, we have theoretically investigated the optimal segmentations with respect to theperformance metrics Accuracy and Dice.
We have given a detailed rigorous characterization of theoptimizers and upper and lower bounds on the volume of optimal segmentations.
Finally, we haveshown the relevance of our theoretical observations in practice by comparing the volume of optimalsegmentations with respect to the performance metrics to the expected volume, on a real worlddata set.
We conclude that noise may cause optimal segmentations to have a volume that deviatessigniﬁcantly from the expected volume of the target structure and that the reason for this is may becaused by the choice of training method rather than the choice of model.
Broader impacts: Formalizing the evaluation process of automated segmentation methods can bedone in many ways, each with its pros and cons.
Even if this work can be interpreted as describing theproblems with using Dice for this formalization, it can still paradoxically contribute to an unhealthyﬁxation of Dice as the gold standard for segmentation evaluation in medical image analysis.
This
[Figure]
Urinary
[Figure]
in turn can lead to that medical practitioners put two much faith in models that have been shown toperform well with respect to the metric on some test data.
One solution to this is to make sure thatclinical practitioners using such models are educated in the problems associated with the metric.
Limitations:
In order for the volume bounds to be sharp in Theorem 1 and Theorem 2, weconstruct extreme cases of m .
These extreme cases might only be representable approximately withstep functions for a particular choice of voxelization.
Consequently, the most extreme cases wecan construct in a numerical setting may not be as extreme as those we have constructed in thecontinuous setting.
However, in medical image analysis it is common to deal voxelizations of theorder of 512 × 512 × 100 voxels which means that the approximation error would be negligable.
Ourwork is also limited by the amount of experiments included.
Additional numerical experiments on awider range of data sets would give a more comprehensive picture on the impact of different noisedistributions. Checklist For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] Our main claims are informal descriptions of the
statements we prove in the Main results section.
(b) Did you describe the limitations of your work?
[Yes] See Limitations in the end of theConclusions section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
Broader impacts in the Discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Generalities
are described in the preliminaries and the rest in the Main results section.
(b) Did you include complete proofs of all theoretical results?
[Yes] The, proofs are listedin the Supplementary and a short scetch is provided after each stated theorem.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All these details
are found in the Supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[N/A] .
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
No random experiments of this kind was conducted.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Our experiments does not require
any heavy computation.
It can be done on a regular laptop.
We mention this in the
Supplementary.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Several times with
(b) Did you mention the license of the assets?
[Yes] Yes, in the Supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Yes, code is included in the Supplementary Material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Yes, see the Supplementary Material.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] Yes, see the Supplementary Material.
If you used crowdsourcing or conducted research with human subjects..
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]