Active Learning Through a Covering Lens
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep active learning aims to reduce the annotation cost for deep neural networks,which are notoriously data-hungry.
Until recently, deep active learning methodsstruggled in the low-budget regime, where only a small amount of samples areannotated.
The situation has been alleviated by recent advances in self-supervisedrepresentation learning methods, which impart the geometry of the data representation with rich information about the points.
Taking advantage of this progress,we study the problem of subset selection for annotation through a “covering” lens,proposing ProbCover – a new active learning algorithm for the low budget regime,which seeks to maximize Probability Coverage .
We describe a dual way to viewour formulation, from which one can derive strategies suitable for the high budgetregime of active learning, related to existing methods like Coreset .
We concludewith extensive experiments, evaluating ProbCover in the low budget regime.
Weshow that our principled active learning strategy improves the state-of-the-art inthe low-budget regime in several image recognition benchmarks.
This method isespecially beneficial in semi-supervised settings, allowing state-of-the-art semisupervised methods to achieve high accuracy with only a few labels.
Introduction
For the most part, deep learning technology critically depends on access to large amounts of annotateddata.
Yet annotations are costly and remain so even in our era of Big Data .
Deep active learning (AL)aims to alleviate this problem by improving the quality of the annotated data.
Specifically, given afixed budget b of examples that can be annotated and some deep learner, AL algorithms aim to querythose b examples that will most benefit this learner.
In order to optimally choose unlabeled examples to be annotated most deep AL strategies followsome combination of two main principles: 1) Uncertainty sampling (e.g, [3, 13, 14, 24, 25, 35, 46]),in which examples that the learner is most uncertain about are picked, to maximize the added value ofthe new annotations.
2) Diversity Sampling (e.g, [16, 36, 39, 41, 44]), in which examples are chosenfrom diverse regions of the data distribution, to represent it wholly and reduce redundancy in theannotation.
Most AL methods fail to improve over random selection when the annotation budget is very small[2, 7, 20, 31, 34, 38, 51], a phenomenon sometimes termed "cold start" [15, 22, 47].
When the budgetcontains only a few examples, they will struggle to improve the model’s performance, not evenreaching the accuracy of the random baseline.
Recently, it was shown that uncertainty sampling isinherently unsuited for the low budget regime, which may explain the cold start phenomenon [18].
The low-budget scenario is relevant in many applications, especially those requiring an expert taggerwhose time is expensive (e.g a radiologist tagger for tumor detection).
If we want to expand deeplearning to new domains, overcoming the cold start problem is an ever-important task.
[Figure]
In this work, we focus on understanding the very low budget regime of AL, where the budget of bexamples cannot dependably represent the data distribution.
In Section 2 we model the problem as
Max Probability Cover , defined as follows: given some data distribution, and a radius δ , select the bexamples which maximize the probability of the union of balls with radius δ around each example.
Under labeling separation assumptions on the data (which are realistic in semantic embedding spaces),
Max Probability Cover is the optimal strategy for nearest-neighbor models, in that it minimizes anupper bound on their generalization error.
In Section 2.3, we show a connection with existing deep
AL methods, like Coreset [36], and explain why those methods are more suitable for the high-budgetregime than the low-budget regime.
This phenomenon is visualized in Fig.
When the data distribution is not known, as is the case in most practical settings, we show that Max
Probability Cover can be reduced to Max Coverage when using the empirical data distribution.
Max
Coverage is known to be NP-hard [33].
We then adapt a greedy algorithm for the selection of bexamples from a fixed finite training set of unlabeled examples (the training set), which guarantees1 − 1 e approximation to the problem (see Section 3).
We call this new method ProbCover .
In Section 4 we empirically evaluate the performance of ProbCover on several computer visiondatasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and subsets of ImageNet.
ProbCover isthus shown to significantly outperform all alternative deep AL methods in the very low budget regime.
Additionally, ProbCover improves the performance of state-of-the-art
semi-supervised methods,which were thought until recently to make AL redundant [6], allowing for the learning of computervision tasks with very few annotated examples.
Relation to prior art In addition to the principle of uncertainty sampling, many AL strategiesemploy the principle of diversity sampling, querying examples from different parts of the distribution(e.g, [1, 15, 17, 21, 25, 29, 37, 45]).
While diverse sets of examples are expected to cover more ofthe training distribution, we maximize the coverage directly, which is especially beneficial in thelow-budget regime.
Recently, a line of works propose AL methods based on an approximation to the facility locationproblem [12, 29, 36, 42, 43, 50], which is a variation of a covering problem.
In the facility locationproblem, the entire distribution is covered with a fixed number of balls, which can vary in size,while in ProbCover the size of the balls is fixed, and we allow the coverage of smaller parts of thedistribution.
While this difference may seem minor, in the low-budget regime, when the budget isnot large enough to represent the data, examples chosen by the facility location problem are notrepresentative, and lead to poor performance (see Fig.
9 and discussion in Section 2.3)
Summary of contribution
• Develop a theoretical framework to analyze AL strategies in embedding spaces, with "dual" lowand high-budget interpretations.• Introduce ProbCover , a low-budget AL strategy motivated by our framework, which significantlyoutperforms other methods in the low-budget regime.• Demonstrate the outstanding competence of ProbCover in semi-supervised learning with veryfew labeled examples.
Theoretical Analysis
We develop a rigorous framework, within which different AL methods can be evaluated and designed.
Since AL is about the selection of examples, we chose to analyze the Nearest Neighbor (NN)classification model, as it is a model that depends almost exclusively on the set of training examplesand does not involve almost any external inductive bias.
Accordingly, in Section 2.1 we develop a bound on the generalization error in 1-NN models.
Thebound involves two terms, which ideally should be jointly minimized.
We show that one term is moresuitable for the low budget regime, where the number of labeled examples is small, while the otherterm is more suitable for the high budget regime, where the number of labeled examples is large.
In the low budget regime, we are likely to benefit more from minimizing the first term, which leadsus to the Max Probability Cover problem defined in Section 2.2.
We then show that this problem is
NP-hard.
Furthermore, when the data distribution is not known and is being approximated by theempirical distribution, we show that it is equivalent to the Max Coverage Problem.
Our method,described in Section 3, is designed to solve this problem.
In contrast, in the high budget regime, we are likely to benefit more from minimizing the secondterm, which is equivalent to optimal facility location , a problem that is also known to be NP-hard andwhich the AL Coreset algorithm
[36] is designed to solve.
Not surprisingly, Coreset is indeed onlyeffective in the high budget regime, as we empirically show in Section 4.
2.1 Bounding the Generalization Error
We shall now derive a bound on the generalization error of a 1 Nearest Neighbor (1-NN) classifier.
We start with some necessary notations and definitions.
Most important is the assumption of δ -purity,which states that most of the time, points that are less than δ apart have the same label.
We then provea lemma, showing that given a labeled set L and the coverage it achieves, and given the δ -purityassumption, the probability of a point being inside this cover and still being falsely labeled is small.
From this, we finally derive our bound on the generalization error, which is stated in Thm. 1.
Notations Let X denote the input domain whose underlying distribution is D , and let Y =
[ k ]denote the target domain.
Assume that a true labeling function f : X → Y exists.
The Generalization
Error of a hypothesis ˆ f : X → Y is defined as L D ( ˆ f ) =
E x ∼D (cid:104)
ˆ f ( x ) ̸ = f ( x ) (cid:105) .
Let X = { x i } mi =1 ∼ D m denote an unlabeled set of points, and b ≤ m the annotation budget.
Let
L ⊆
X denote the labeled set, where | L |
=
b .
Let B δ ( x ) =
{ x ′ :
∥ x ′ − x ∥ 2 ≤ δ } denote a ballcentered at x of radius δ .
Let C ( L, δ ) =
(cid:83) c ∈
L B δ ( c ) denote the region covered by δ -balls centeredat the labeled examples in L (sometimes abbreviated as C ).
We call C
( L, δ ) the covered region and
P ( C ) the coverage .
Definition 2.1.
We say that a ball B δ ( x ) is pure if ∀ x ′ ∈
B δ ( x ) : f ( x ′ )
=
f ( x ) .
Definition 2.2.
We define the purity of δ as: π ( δ ) =
P ( { x : B δ ( x ) is pure } ) .
Notice that π ( δ ) ismonotonically decreasing.
Let ˆ f denote the 1-NN classifier based on L .
We split the covered region C ( L, δ ) to two sets:
C right = { x ∈ C :
ˆ f ( x ) =
f ( x ) } , C wrong =
C \ C right
Lemma 1.
C wrong ⊆ { x :
Proof.
Let x ∈ C wrong .
Let c ∈
L denote the nearest neighbor to x .
Then they have the same predicted label, ˆ f ( x ) =
ˆ f ( c ) , and f ( c )
=
ˆ f ( c ) because c is labeled.
Since x is labeled wrongly, ˆ f ( x ) ̸ = f ( x ) , which means
[Equation]
Finally, since x ∈ C wrong ⊆
C is in the coverage, d ( x, c ) <
δ , which means that c ∈ B δ ( x ) with adifferent label and so B δ ( x ) is not pure.
Corollary 1.
[Equation]
Theorem 1.
The generalization error of the 1-NN classifier ˆ f is bounded by
[Equation]
Proof.
[Equation]
Notice that this gives us a different bound for different δ values, and different labeled sets L .
This bound introduces a trade-off: as δ increases, the coverage increases, but the purity decreases.
Therefore, we wish to find the pair δ, L that gives the tightest bound.
Discussion We can interpret (1) in the context of two boundary conditions of AL: high-budget andlow-budget.
In the high-budget regime, achieving full coverage P ( C ) = 1 is easy as we have manypoints, and the remaining challenge is to reduce 1 − π ( δ ) .
Since π ( δ ) is monotonically decreasing,we can seek to minimize δ subject to the constraint P ( C )
= 1 .
This is similar to Coreset [36].
In thelow-budget regime, full coverage entails very low purity, which (if sufficiently low) makes the boundtrivially 1.
Thus, instead of insisting on full coverage, we can fix a δ that yields "large enough" purityπ ( δ ) > 0 , and then seek a labeled set L that maximizes the coverage P ( C ) .
We call this problem
Max Probability Cover .
2.2 Max Probability Cover
Definition 2.3 ( Max Probability Cover ) .
Fix δ > 0 , and obtain a subset L ⊂ X, | L | = b , thatmaximizes coverage
[Equation]
An optimal solution to (2) would minimize the bound in (1), when δ is fixed (which fixes 1 − π ( δ ) ).
Unfortunately, when moving to practical settings there are two obstacles.
The first is about complexity:
Theorem 2.
Max Probability Cover is NP-hard.
Proof.
(Sketch, full proof in App.
B)
We construct a reduction from an established NP-hardproblem ( Max Coverage , see Def. A.1) to Max Probability Cover .
For the collection of subsets
S =
{ S 1 , . . .
, S m } , we consider the space R m and a collection of δ -balls { B δ ( x i ) } mi =1 with thespecial property that for any subset of the balls, their intersection includes at least one unique point(see Lemma 2).
We then assign each S i to B δ ( x i ) , and each element in S i is sent to a point in theintersection of all the balls assigned to subsets that contain it.
Each such point then defines a Diracmeasure, the normalized sum of which determines a probability distribution on R m .
The selectionof δ -balls that is the solution to the Max Probability Cover can be translated back to a selection ofsubsets, which is the solution to the original Max Coverage problem.
The second problem is that we are almost always not supplied with the data distribution, and even ifwe were, the probability computation would most likely be intractable and hard to approximate.
If we are not supplied with the data distribution, we can use the empirical distribution ˜ P
( A )
=1 n (cid:80) ni
=1 1 x i ∈
A to approximate it.
This gives us the following result:
Proposition.
When P is the empirical distribution ˜ P , the Max Probability Cover objective functionis equivalent to the Max Coverage objective, with { B δ ( x i ) ∩ X } ni =1 as the collection of subsets.
Proof.
Given a labeled set L = { x i } bi =1 , we show equality of objectives up to constant 1 | X |
P
[Equation]
b (cid:91)
[Equation]
=
[Equation]
151 2.3 The "Duality" of Max Probability Cover and Coreset
The Coreset AL method [36] minimizes the objective
[Equation]
We can rewrite the above in the language of distributions as
[Equation]
If we use the empirical distribution then δ ( L ) = δ ′ ( L ) .
In this framework we can say that Max
Probability Cover and Coreset are dual problems (dual in a loose sense) as follows:
Max Probability Cover minimizes the generalization error bound (1) when we fix δ and seek tomaximize the coverage, which is suitable for the low budget regime.2.
Coreset minimizes the generalization error bound (1) when we fix the coverage to 1 and minimizeδ , which is suitable for the high budget regime because only then can we fix the coverage to 1.This duality is visualized in Fig
Method: ProbCover
In order to deliver a practical method, we first note that our approach implicitly relies on the existenceof a good embedding space [4, 9, 49], where distance is correlated with semantic similarity, andwhere similar points are likely to bunch together in high-density regions.
In such spaces the labelingis also likely to be correlated with distance, making it friendly to 1-NN classification.
Additionally, itimplies the existence of large enough δ balls, with good purity and coverage properties.
Secondly, we note that since Max Coverage is NP-hard, it cannot be solved efficiently.
Instead,as its objective is a submodular monotone function, we use the greedy approximate algorithm,which achieves (cid:0) 1 − 1 e (cid:1) -approximation [26].
Using an algorithm that gives a better approximation isimpractical, see App.
D.3 for an in-depth discussion.
We now turn to describe the greedy algorithmand our method for estimating the ball size δ .
3.1 Greedy Algorithm
The algorithm goes as follows:
First, construct a directed graph G =
( V, E ) , with V
= X
and( x, x ′ ) ∈
E
⇐⇒ x ′ ∈
B δ ( x ) , where each vertex represents a specific example, and there is anedge between two vertices ( x, x ′ ) if x ′ is covered by the δ -ball centered at x .
The algorithm thenperforms b iterations of the following 2 steps: (i) pick the vertex x max with the highest out-degree forannotation; (ii) remove all incoming edges to x max and its neighbors. See Alg. 1 for pseudo-code.
Input: unlabeled pool U , labeled pool L , budget b , ball-size δ
Output: a set of points to query
G =
( V, E ) ← Construct Graph for all c ∈ L do
C = { x : ( c, x ) ∈ E }
E =
E \ { ( z, x ) : x ∈ C } end for
Queries ← ∅ for all i=1,...
,b do
Choose c ∈ U with highest out-degree in G
C = { x : ( c, x ) ∈ E }
E = E \ { ( z, x ) : x ∈ C } Queries = Queries ∪ { c } end for return Queries
[Figure] 3.2 Estimating δ
Our algorithm requires us to specify the hyper-parameter δ , the ball radius, whose value depends on details of the embedding space.
In choosing δ , we need to consider the trade-off between large coverage P ( C ) and high purity π ( δ ) .
We resolve this trade-off with a heuristic, where we pick the largest δ possible, while maintaining purity above a certain threshold α ∈ (0 , 1) .
Specifically,
[Equation]
Importantly, α is more intuitive to tune, and is kept constant across different datasets (unlike δ ).
Westill need to be able to estimate the purity π ( δ ) , which depends on the labels, from unlabeled data.
Toaddress this challenge, we estimate purity using unsupervised representation learning and clustering.
First, we cluster self-supervised features using k -Means with k equal to the number of classes.
For agiven δ , we compute the purity π ( δ ) using the clustering labels as pseudo-labels for each example.
Searching for the best δ , we repeat the process and pick the largest δ so that at least α = 0 .
95 of theballs are pure.
In Fig.
2, we plot the percentage of pure balls across different datasets as a function of δ , where thedashed line represents the δ ∗ chosen by our method.
Empirical Results
We report a set of empirical results, comparing ProbCover to other AL strategies in a variety ofsettings.
We focus on the very low budget regime, with a budget size b of a similar order of magnitudeas the number of classes.
Note that since the data is picked from an unlabeled pool, chances are thatthe initial labeled set is not going to be balanced across classes, and in the early stages of trainingsome classes will almost always be missing.
4.1 Methodology
Three deep AL frameworks are evaluated:
(i) Fully supervised : train the deep learner only on the annotated data, as a fully supervised task.
(ii) Semi-supervised by transfer learning : create a representation of the data by training with a self-
supervised task on the unlabeled data, then train a simple model on the ensuing representation
in a supervised manner.
This framework is intended to capture the basic benefits of semi- supervised learning, regardless of the added benefits provided by the sophistication of modern semi-supervised learning methods.
(iii) Fully semi-supervised : train a semi-supervised model on both the annotated and unlabeleddata using FlexMatch [48].
In frameworks (i) and (ii) we adopt the evaluation framework created by Munjal et al.
[32], in whichwe can compare multiple deep AL strategies in a principled way.
In framework (iii), we adopt thecode and hyper-parameters provided by FlexMatch.
When evaluating frameworks (i) and (ii), we compare ProbCover to 9 deep AL strategies as baselines.(1)
Random query uniformly.
(2)-(4) query examples with the lowest score, using the followingbasic scores: (2) Uncertainty – max softmax output, (3)
Margin – margin between the two highestsoftmax outputs, (4) Entropy – inverse entropy of softmax outputs.
(5) BADGE [1].
(6) DBAL [14].(7) TypiClust [18].
(8) BALD [25].
(9) Coreset [36].
We note that while most baseline methods aresuitable for the high budget regime, TypiClust is also suitable for the low budget regime.
These AL methods are evaluated on the following classification datasets: CIFAR-10/100 [27], TinyImageNet [28] and ImageNet-50/100 (the latter group includes subsets of ImageNet [11] containing50/100 classes respectively, following Van Gansbeke et al.
When considering CIFAR-10/100,and TinyImageNet, we use as input the embedding of SimCLR [8].
When considering ImageNet50/100 we use as input the embedding of DINO [5].
Results on ImageNet-50 are deferred to theappendix and can be seen in App.
E. Details on the specific networks and hyperparameters canbe found in App.
D.2.
When evaluating frameworks (i) and (ii), we perform 5 active learningrounds, querying a fixed budget of b examples in each round.
In framework (iii), as FlexMatch iscomputationally demanding, we only evaluate methods on their initial pool selection capabilities.
4.2 Main Results
(i) Fully supervised framework.
We evaluate different AL methods based on the performance of adeep neural network trained directly on the raw queried data.
On each round, we query b
samples(where b is equal to the number of and train a ResNet-18 on the accumulated queried set.
We repeatthis for 5 active learning rounds.
In Fig.
3, we plot the mean accuracy of 5 repetitions.
Additionalresults can be seen in App.
E.
[Figure]
(ii) Semi-supervised by transfer learning.
In this framework, we make use of pretrained selfsupervised features, and measure classification performance using the 1-NN classifier.
Accordingly,each point is classified by the label of its nearest neighbor (within the selected labeled set L ) in theself-supervised features space.
In low budgets, this framework outperforms the fully-supervisedframework (i), but is not as effective as fully semi-supervised learning (iii), demonstrating that thesefindings are not specific to any given semi-supervised method.
Similarly to Fig.
3, in Fig.
4, we plotthe mean accuracy of 5 repetitions for the different tasks.
[Figure]
(iii) Semi-supervised framework.
We compare the performance of different AL strategies used priorto running FlexMatch, a state-of-the-art semi-supervised method.
In Fig.
5 we show results with 3repetitions of FlexMatch, using the labeled sets provided by different AL strategies and budget bequal to the number of classes.
We see that our method outperforms random sampling and other ALbaselines by a large margin.
We note that in agreement with previous works [6, 18], AL strategiesthat are suited for high budgets do not improve the results of random sampling, while AL strategiesthat are suited for low budgets do.
[Figure]
4.3 Ablation Study
We report a set of ablation studies, evaluating the added value of each step of ProbCover .
Random initial selection When following the uncertainty sampling principle, as many AL methodsdo, a trained learner is needed.
Any such method requires therefore a non-empty initial pool oflabeled examples to train a rudimentary learner, from which uncertainty selection can be bootstrapped.
In the set of methods evaluated here (see Section 4.1), only two - ProbCover and TypiClust - are notaffected by this problem.
This can be seen in Fig.
3, noting that only these two methods do betterthan random in the initial step.
Is this the only reason they outperform other methods on low budgets?
To address this concern, we repeat the experiments reported in Fig.
3a-3b, using an initial random setof annotated examples across the board and by all methods.
Results are reported in Fig.
Whencomparing Fig.
3a-3b and Fig.
6, we see that the advantage of ProbCover and TypiClust goes beyondthe initial set selection, and can be seen even if this factor is eliminated.
Input space distances As discussed in Section 3, our approach relies on the existence of a goodembedding space, where distance is correlated with semantic similarity.
We now verify this claim byrepeating our basic fully-supervised experiments (Fig. 3) with one difference: ProbCover can onlyuse the original pixel space representation to compute coverage.
Results are show in Fig.
Whencomparing Fig.
3 with Fig.
7, a significant drop in performance is seen as expected, demonstratingthe importance of the semantic embedding space.
The interaction between δ and budget size To understand the interaction between the hyperparameter δ and budget b , we repeat our basic experiments (Fig. 3) with different choices of δ and bon CIFAR-10.
For each pair ( δ, b ) , we select an initial pool of b examples using ProbCover with δballs, and report the difference in accuracy from the selection of b random points.
Average results
[Figure] [Figure]
across 3 repetitions are shown in Fig.
8 as a function of b .
We see that as the budget b increases,smaller δ ’s are preferred.
[Figure]
Coreset vs. ProbCover .
In Section 2.3 we argue that ProbCover is suitable for low budgets, while
Coreset is suitable for high budgets.
To verify this claim, we compare their performance under thefollowing 3 setups while using the same embedding space, and report results on CIFAR-10:
• Low budget - Select an initial pool of 100 samples using the SimCLR representation.• High budget - Train a model on 5K randomly selected examples.
Then select an additional setof 5K examples using the learner’s latent representation.
This is the setup used in Sener and
Savarese [36]. • Mid budget - Same as high budget, except the initial pool size and added budget are 1K.
Results are reported in Fig.
In the low budget, ProbCover outperforms Coreset as would beexpected.
In the mid budget, where the feature space of the learner is informative, only ProbCoverachieves significant improvement over random selection.
In the high budget, Coreset improves overrandom selection, while ProbCover does not.
Summary and Discussion
We study the problem of AL in the low-budget regime.
We model the problem as Max Probability
Cover , showing that under certain assumptions on the data distribution, which are likely to holdin self-supervised embedding spaces, it optimizes an upper-bound on the generalization error ofa 1-NN classifier.
We devise an AL strategy ProbCover that approximates the optimal solution.
We empirically evaluate it in supervised and semi-supervised frameworks across different datasets,showing that ProbCover significantly outperforms other methods in the low-budget regime.
In future work we intend to investigate (i) possible avenues for improving the choice of δ by makinguse of already known labels, or inferring a score for δ through the topology of the resulting coveringgraph.
(ii) extending the current formulation of Max Probability Cover , by making δ - the radius ofthe balls - dependent on the samples instead of being uniform.
(iii) soft-coverage approaches, wherethe covering notion is not binary but some continuous measure, may allow us to do away with δ .
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Throughout the entire paper. (c) Did you discuss any potential negative societal impacts of your work?
[No] Irrelevantfor this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Section 2.
(b) Did you include complete proofs of all theoretical results?
[Yes] Section 2, and App. B
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We included
the necessary information for reproducibility.
Furthermore, the code will be published
upon acceptance.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] 4.
(b) Did you mention the license of the assets?
[Yes] All assets are publicly available.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The data is publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] Irrelevant.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]