Smoothed Embeddings for Certiﬁed Few-Shot Learning
Abstract
Randomized smoothing is considered to be the state-of-the-art provable defenseagainst adversarial perturbations.
However, it heavily exploits the fact that classiﬁers map input objects to class probabilities and do not focus on the ones thatlearn a metric space in which classiﬁcation is performed by computing distances toembeddings of classes prototypes.
In this work, we extend randomized smoothingto few-shot learning models that map inputs to normalized embeddings.
We provideanalysis of Lipschitz continuity of such models and derive robustness certiﬁcateagainst ` 2 -bounded perturbations that may be useful in few-shot learning scenarios.
Our theoretical results are conﬁrmed by experiments on different datasets.
Introduction
Few-shot learning is a setting in which a classiﬁcation model is evaluated on the classes not seenduring the training phase.
Nowadays quite a few few-shot learning approaches based on neuralnetworks are known.
Unfortunately, neural network based classiﬁers are intimately vulnerable toadversarial perturbations [41, 10] – accurately crafted small modiﬁcations of the input that maysigniﬁcantly alter the model’s prediction.
This vulnerability poses a restriction on the deployment ofsuch approaches in safety-critical scenarios, so the research interest in the ﬁeld of attacks on neuralnetworks has been great in recent years.
Several works studied this phenomena in different practical applications – image classiﬁcation[2, 34, 33, 40], object detection [17, 26, 48, 50], face recognition [18, 5, 54], semantic segmentation[7, 12, 50].
These studies show that it is easy to force a model to behave in the desired way byapplying imperceptible change to its input.
As a result, defenses, both empirical [4, 55, 14] andprovable [51, 21, 3, 47, 52, 15, 46, 35], were proposed recently.
Although empirical ones can be (andoften are) broken by more powerful attacks, the provable ones are of a big interest since they make itpossible to provide guarantees of the correctness of the work of a model under certain assumptions,and, thus, possibly broaden the scope of tasks which may be trusted to the neural networks.
Randomized smoothing [21, 3, 25] is the state-of-the-art approach used for constructing classiﬁcationmodels provably robust against small-norm additive adversarial perturbations.
This approach isscalable to large datasets and can be applied to any classiﬁer since it does not use any assumptionsabout model’s architecture.
Generally, the idea is following.
Suppose, we are given a base neuralnetwork classier f : R D !
[0 , 1] K that maps an input image x to a ﬁxed number of K classprobabilities.
Its smoothed version with the standard Gaussian distribution is:
[Equation]
As shown in [3], the new (smoothed) classiﬁer is provably robust at x to ` 2 -bounded perturbations ifthe base classiﬁer f is conﬁdent enough at x .
However, the proof of certiﬁcation heavily exploits n
[Figure] [Figure]
Smoothed
[Figure]
constant
[Figure]
Yes
[Figure]
Certify all the fact that classiﬁers are restricted to map an input to a ﬁxed number of class probabilities.
Thus,directly applying randomized smoothing to classiﬁers in metric space, such as in few-shot learning,is a challenging task.
There are several works that aim at improving the robustness for few-shot classiﬁcation [20, 9, 53, 28].
However, the focus in such works is either on the improvement of empirical robustness or probabilisticguarantees of certiﬁed robustness; none of them provide theoretical guarantees on the worst-casemodel behavior.
In this work we ﬁll this gap, by generalizing and theoretically justifying the idea of randomizedsmoothing to few-shot learning.
In this scenario, provable certiﬁcation needs to be obtained not in thespace of output class probabilities, but in the space of descriptive embeddings.
This work is the ﬁrst,to our knowledge, where the theoretical robustness guarantees for few-shot scenario is provided.
Our contributions are summarized as follows:
• We provide the ﬁrst theoretical robustness guarantee for few-shot learning classiﬁcation.
• Analysis of Lipschitz continuity of such models and providing the robustness certiﬁcatesagainst ` 2  bounded perturbations for few-shot learning scenarios.•
We propose to estimate conﬁdence intervals not for distances between the approximationof smoothed embedding and class prototype but for the dot product of vectors which hasexpectation equal to the distance between actual smoothed embedding and class prototype.
Problem statement
2.1 Notation
We consider a few-shot classiﬁcation problem where we are given a set of labeled objects( x 1 , y 1 ) , . . .
, ( x m , y m ) where x i 2 R D and y i 2 { 1 , . .
, K } are corresponding labels.
We follow the notation from [39] and denote S k as the set of objects of class k.
2.2 Few-shot learning classiﬁcation
Suppose we have a function f : R D !
R d that maps input objects to the space of normalizedembeddings.
Then, d  dimensional prototypes of classes are computed as follows (expression isgiven for the prototype of class k ):
[Equation]
In order to classify a sample, one should compute the distances between its embedding and classprototypes – a sample is assigned to the class with the closest prototype.
Namely, given a distancefunction ⇢ : R d ⇥ R d ! [0 , + 1 ) , the class c of the sample x is computed as below:
[Equation]
Given an embedding function f , our goal is to construct a classiﬁer g provably robust to additiveperturbations  of a small norm.
In other words, we want to ﬁnd a norm threshold t such that equality
[Equation]
will be satisﬁed for all  : k  k 2  t.
In this paper, the solution of a problem of constructing a classiﬁer that satisﬁes the condition in (4) isapproached by extending the analysis of the robustness of smoothed classiﬁers described in (1) tothe case of vector functions.
The choice of the distance metric in (4) is motivated by an analysis of
Lipschitz-continuity given in the next section.
Randomized smoothing3.1 Background
Randomized smoothing [21, 3] is described as a technique of convolving a base classiﬁer f with anisotropic Gaussian noise such that the new classiﬁer g ( x ) returns the most probable prediction of fof a random variable ⇠ ⇠ N ( x,  2 I ) , where the choice of Gaussian distribution is motivated by therestriction on g to be robust against additive perturbations of bounded norm.
In this case, given aclassiﬁer f : R D !
[0 , 1] and smoothing distribution N (0 ,  2 I ) , the classiﬁer g looks as follows:
[Equation]
One can show by Stein’s Lemma that given the fact that the function f in (5) is bounded (namely,8 x 2 D ( f ) , | f ( x ) |  1 ), then the function g is L  Lipschitz:
[Equation]
with L = q 2 ⇡ 2 , what immediately produce theoretical robustness guarantee on g.
Although this approach is simple and effective, it has a serious drawback: in practice, it is impossibleto compute the expectation in (5)
exactly and, thus, impossible to compute the prediction of thesmoothed function g at any point.
Instead the integral is computed with the use of Monte-Carloapproximation with n samples to obtain the prediction with an arbitrary level of conﬁdence.
Notably,to achieve an appropriate accuracy of the Monte-Carlo approximation, the number of samples nshould be large enough that may dramatically affect inference time.
In this work, we generalize the analysis of Lipschitz-continuity to the case of vector functionsg : R D !
R d and provide robustness guarantees for classiﬁcation performed in the space ofembeddings.
The certiﬁcation pipeline is illustrated in the Figure 1.
3.2 Randomized smoothing for vector functions
Lipschitz-continuity of vector function.
In the work of [38], the robustness guarantee from [3] isproved by estimating the Lipschitz constant of a smoothed classiﬁer.
Unfortunately, a straightforwardgeneralization of this approach to the case of vector functions leads to the estimation of the expectationof the norm of a multivariate Gaussian which is known to depend on the number of dimensions ofthe space.
Instead, we show that a simple adjustment to this technique may be done such that theestimate of the Lipschitz constant is the same as for the function in (5).
Our results are formulated inthe theorems below proofs of which are moved to the Appendix in order not to distract the reader.
Theorem 1.
(Lipschitz-continuity of smoothed vector function) Suppose that f : R D !
R d is adeterministic function and g ( x ) =
E " ⇠ N (0 ,  2 I )
f
( x + " ) is continuously differentiable for all x .
Iffor all x , k f ( x ) k 2 = 1 , then g ( x ) is L  Lipschitz in l 2  norm with L = q 2 ⇡ 2 .
Remark 1.
We perform the analysis of Lipschitz-continuity in the Theorem 1 in l 2  norm, so thedistance metric in (4) is l 2  distance.
We do not consider other norms in this paper.
Robust classiﬁcation in the space of embeddings.
To provide certiﬁcation for a classiﬁcation in the space of embeddings, one should estimate the maximum deviation of the classiﬁed embedding that does not change the closest class prototype.
In the Theorem 2, we showhow this deviation is connected with the mutual arrangement of embedding and class prototype.
[Figure]
[Equation]
[Figure] [Figure]
[Figure]
(Adversarial embedding risk)
[Figure]
Given an input image x 2 R D and the embed-
[Figure]
ding g : R D !
R d the closest point on to decision boundary in the embedding space (see
Figure 2) is located at a distance (deﬁned asadversarial embedding risk):
[Equation]
(7) where c 1 2 R d and c 2 2 R d are the two closestprototypes.
The value of  is the distance between classifying embedding and the decisionboundary between classes represented by c 1 andc 2 .
Note that this is the minimum l 2  distortionin the embedding space required to change theprediction of g.
Two theorems combined result in an l 2  robustness guarantee for few-shot classiﬁcation as:
Theorem 3.
(Robustness guarantee)
L 2 -robustness guarantee r for an input image x
in then  dimensional input metric space under classiﬁcation by a classiﬁer g from the Theorem 1 isr =
L , where L is the Lipschitz constant from the Theorem 1 and  is the adversarial risk from the
Theorem 2.
The value of r is the certiﬁed radius of g at x , or, in other words, minimum l 2  distortionin the input space required to change the prediction of g. The proof of this fact straightforwardlyfollows from the deﬁnition (6) and results from Theorems 1-2.
Certiﬁcation protocol
In this section, we describe the numerical implementation of our approach and estimate its failureprobability.
4.1 Estimation of prediction of smoothed classiﬁer
As mentioned previously, the procedure of few-shot classiﬁcation is performed by assigning an objectx to the closest class prototype.
Unfortunately, given the smoothed function g in the form from the
Theorem 1 and class prototype c k from (2), it is impossible to compute the value ⇢ ( g ( x ) , c k ) explicitlyas well as to determine the closest prototype, since it is in general unknown how does g ( x ) look like.
We propose both to estimate the closest prototype for classiﬁcation and to estimate the distance tothe closest decision boundary  from Theorem 2 as the largest class-preserving perturbation in theembedding space by computing two-sided conﬁdence intervals for random variables
[Equation]
(8) where ˆ g ( x ) = 1 n
P ni =1 f ( x + " i ) is the estimation of g ( x ) computed as empirical mean by n samplesof noise, and one-sided conﬁdence interval for  from the Theorem 2, respectively.
Pseudo-code forboth procedures is presented in Algorithms 1-2.
Algorithm 1 Closest prototype computation algorithm.
GIVEN: base classiﬁer f , noise  , object x , number of samples n for ˆ g ( x ) , class prototypes { c i } Ki =1 , maximum number of samples T and conﬁdence level ↵ ,
RETURNS: index A of the closest prototype.
Function CLOSEST( f,  , x, n, { c i } Ki =1 , T, ↵ ) fs ; i n 0 = n while n  T do " 1 , . .
, " n ⇠ N (0 ,  2 I )
ˆ g ( x ) 1 n
P ni =1 f ( x + " i ) fs fs
[ { ˆ g ( x ) } for all i 2 [1 , . . .
, K ] do
C = c
i dsToC ⇢ ( fs, C ) ( l C , u C )
TwoSidedConfInt ( dsToC, ↵ ) l i l C , u i u C end for
A arg min { l i } Ki = if u A < min( { l i } Ki =1 ,i 6 = A ) then Return A, fs else n n + n 0 {Increase number of samples used for computing an approximation ˆ g ( x ) until the number of observations is large enough to determine two leftmost intervals or until n = T } end if end while
EndFunction
The Algorithm 1 describes an inference procedure for the smoothed classiﬁer from the Theorem 1;the Algorithm 2 uses Algorithm 1 and, given input parameters, estimates an adversarial risk from the
Theorem 2 – it determines two closest to smoothed embedding g ( x ) prototypes A and B and producesthe lower conﬁdence bound for the distance between g ( x ) and the decision boundary between Aand B. Combined with analysis from the Theorem 1, it provides the certiﬁed radius for a sample –the smallest value of l 2  norm of perturbation in the input space required to change the predictionof the smoothed classiﬁer.
In the next subsection, we discuss in detail the procedure of computingconﬁdence intervals in Algorithms 1-2.
4.2 Applicability of algorithms
The computations of smoothed function and distances to class prototypes and decision boundary in
Algorithms 1-2 are based on the estimations of corresponding random variables, thus, it is necessaryto analyze applicability of the algorithms.
In this section, we propose a way to compute conﬁdenceintervals for squares of the distances between the estimates of embeddings and class prototypes.
Computation of conﬁdence intervals for the squares of distances.
Recall that one way to estimatethe value of a parameter of a random variable is to compute a conﬁdence interval for the correspondingstatistic.
In this work, we construct intervals by applying well-known Hoeffding inequality [13] inthe form
[Equation]
where X and E ( X ) are sample mean and population mean of random variable X , respectively, n isthe number of samples and numbers a i ,
b i are such that P ( X i 2 ( a i , b i ))
= 1 .
Algorithm
GIVEN: base classiﬁer f , noise  , object x , number of samples n for ˆ g ( x ) , class prototypes { c i } Ki =1 , maximum number of samples T for ˆ g ( x ) and conﬁdence level ↵
RETURNS: lower bound  for the adversarial risk  .
Function EMBEDDING-RISK( f,  , x, n, { c i } Ki =1 , T, ↵ )
A, fs A
CLOSEST( f,  , x, n, { c i } Ki =1 , T, ↵ )
B, fs B
CLOSEST( f,  , x, n, { c i } Ki
[Equation]
[Equation] [Equation]
, T, ↵ ) fs
[Equation]
[Equation]
; for all f 2 fs do
[Equation]
s [ {  } end for
LowerConfBound (  s, ↵ )
Return
EndFunction
However, a conﬁdence interval for the distance
k ˆ g
( x )  c k k 2 with a certain conﬁdence covers 161 an expectation of distance
E ( k ˆ g ( x )  c k k 2 ) , not the distance for expectation k E
(ˆ g ( x )  c k ) k 2 =k g ( x )  c k k 2 .
To solve this problem, we propose to compute conﬁdence intervals for the dot product of vectors.
Namely, given a quantity ⇠ x,k = h g ( x )  c k , g ( x )  c k
i , we sample its unbiased estimate with 165 at maximum 2 n samples of noise (here we have to mention that the number of samples n from
Algorithm 1 actually doubles since we need a pair of estimates ˆ g ( x ) of smoothed embeddings):
[Equation]
[Equation]
and compute conﬁdence interval ( l x,k , u x,k ) such that given
[Equation]
the population mean E (ˆ ⇠ x,k ) is most probably located within it, or, equivalently,
P ⇣ l x,k  E (ˆ ⇠ x,k )  u x,k ⌘  1  ↵ .
We have to mention that there are three conﬁdence intervals for three terms (one with quadratic number of samples and two with the linear numbers ofsamples) in the expression (10), that is why there is fraction 13 in the equation (11).
Also note that the population mean E (ˆ ⇠ x,k ) is exactly k g ( x )  c k k 22 , since
[Equation]
since f ( x + " i ) and f ( x + " j ) are independent random variables for i
= j .
Finally, note that theconﬁdence interval ( l x,k , u x,k ) for the quantity k g ( x )  c k k 22 implies conﬁdence interval
[Equation]
[Equation]
for the quantity k g ( x )  c k k 2 .
Thus, the procedures TwoSidedConfInt and LowerConfBound from 176 algorithms return an interval from (13) and its left bound for the random variable representingcorresponding distance, respectively.
[Figure] [Figure]
Experiments
5.1 Datasets
For the experimental evaluation of our approach we use several well-known datasets for few-shotlearning classiﬁcation.
Cub-200-2011 [45] is a dataset with 11 , 788 images of 200 bird species, where5864 images of 100 species are in the train subset and 5924 images of other 100 species are in the testsubset.
It is notable that a lot of species presented in dataset have degree of visual similarity, makingclassiﬁcation of ones a challenging task even for humans.
mini ImageNet [44] is a substet of imagesfrom ILSVRC 2015 [37] dataset with 64 images categories in train subset, 16 categories in validationsubset and 20 categories in test subset with 600 images of size 84 ⇥ 84 in each category.
CIFAR FS[1] is a subset of CIFAR 100 [19] dataset which was generated in the same way as mini ImageNet andcontains 37800 images of 64 categories in the train set and 11400 images of 20 categories in the testset.
Experimental setup for all the datasets is presented in the next section.
5.2 Experimental settings and computation cost
Following [3], we compute approximate certiﬁed test set accuracy to estimate the performance of thesmoothed model prediction with the Algorithm 1 and embedding risk computation with the Algorithm2.
The baseline model we used for experiments is a prototypical network introduced in [39] with
ConvNet-4 backbone.
Compared to the original architecture, an additional fully-connected layer wasadded in the tail of the network to map embeddings to 512-dimensional vector space.
The model wastrained to solve 1-shot and 5-shot classiﬁcation tasks on each dataset, with 5-way classiﬁcation oneach iteration.
Parameters of expeiments.
For data augmentation, we applied Gaussian noise with zero mean,unit variance and probability 0 .
3 of augmentation.
Each dataset was certiﬁed on a subsample of 500images with default parameters for the Algorithm 1: number of samples n = 1000 , conﬁdence level↵ = 0 .
001 and variance  = 1 . 0 , unless stated otherwise.
For our settings, it may be shown fromsimple geometry that values ( a i , b i ) from (9) are such that b i  a i 
so we use b i  a i = 4 .
Themaximum number of samples T in the Algorithm 1 is set to be T = 5 ⇥ 10 5 .
Computation cost.
In the table below, we report the computation time of the certiﬁcation procedureper image on Tesla V100 GPU for Cub-200-2011 dataset.
Standard deviation in seconds appearsto be signiﬁcant because the number of main loop iterations required to separate the two leftmostconﬁdence intervals varies from image to image in the test set.
[Table]
5.3 Results of experiments
In this section, we report results of our experiments.
In our evaluation protocol, we computeapproximate certiﬁed test set accuracy, CA .
Given a sample x , a smoothed classiﬁer g ( · ) from the
Algorithm 1 with an assigned classiﬁcation rule h ( x ) =
arg min i 2 { 1 ,...,K } k g ( x )  c
k k 2 , thresholdvalue " for l 2  norm of additive perturbation and the robustness guarantee r = r ( x ) from the
Algorithm 3, we compute CA on test set S as follows:
[Equation]
In other words, we treat the model g ( · ) as certiﬁed at point x under perturbation of norm " if x iscorrectly classiﬁed by g ( · ) (what means that the procedure of classiﬁcation described in the Algorithm1 does not abstain from classiﬁcation of x ) and g ( · ) has the value of certiﬁed radius r ( x ) > " .
Visualization of results.
The ﬁgures 3-4 represent dependencies of certiﬁed accuracy on the value ofnorm of additive perturbation for different learning settings (1-shot and 5-shot learning).
The valueof attack radius corresponds to the threshold " from (14).
For Cub-200-2011 dataset we provide adependency of certiﬁed accuracy for different sample size n for the Algorithm 1 (in the Figure 5).
Limitations
In this section, we provide failure probability of Algorithms 1-2, discuss abstains from classiﬁcationin the Algorithm 1 and speculate on the application of our method in other few-shot scenarios.
6.1 Estimation of errors of algorithms
Note that the value of ↵ from (11) is the probability of the value of ⇢ x,k = k g ( x )  c k k 2 not to belongto the corresponding interval of the form from (13).
Given a sample x , the procedure in the Algorithm1 returns two closest prototypes to the g ( x ) .
To determine two leftmost conﬁdence intervals, all thedistances ⇢ x,k have to be located within corresponding intervals, thus, according to the independenceof computing these two intervals, the error probability for the Algorithm 1 is q 1 = K ↵ , where Kis the number of classes.
Similarly, the procedure in the Algorithm 2 outputs the lower bound forthe adversarial risk with coverage at least 1  ↵ and depend on the output of the Algorithm 1 inside,and, thus, has error probability q 2 = 1  (1  ↵ )(1  K ↵ ) =
↵ + K ↵  K ↵ 2 that corresponds toreturning an overestimated lower bound for the adversarial risk from the Theorem 2.
6.2 Abstains from classiﬁcation and extension to other few-shot approaches
It is crucial to note that the procedure in the Algorithm 1 may require a lot of samples to distinguish two leftmost conﬁdence intervals and sometimes does not ﬁnish before reaching thresholdvalue T for sample size.
As a result, there may be input objects at which the smoothed classiﬁer can be neither evaluated nor certiﬁed.
In this subsection, we report the fraction of objects in which the Algorithm 1 abstains from determining the closest class prototype (see Tables 2-3).
n of the Algorithm 1, CIFAR-FS dataset, 5shot case.
Is is notable that relatively small number of samples may be used to achieve satisfactory level of certiﬁed accuracy.
[Table] [Table]
Since our method is based on randomized smoothing, among the approaches presented in [43] itis applicable for matching networks and to some extent to MAML networks.
In case of matchingnetworks, new sample is labeled as weighted cosine distance to support samples, so it is easy totransfer guarantees for l 2  distance to the ones for cosine distance in case of normalized embeddings.
For MAML, the smoothing may be applied to the embedding function as well, but theoreticalderivations of certiﬁcates are required.
Related work
Breaking neural networks with adversarial attacks and empirical defending from them have a longhistory of cat-and-mouse game.
Namely, for a particular proposed defense against existing adversarialperturbations, a new more aggressive attack is found.
This motivated researchers to ﬁnd defenses thatare mathematically provable and certiﬁably robust to different kinds of input manipulations.
Severalworks proposed exactly veriﬁed neural networks based on Satisﬁability Modulo Theories solvers[16, 6], or mixed-integer linear programming [30, 8].
These methods are found to be computationallyinefﬁcient, although they guarantee to ﬁnd adversarial examples, in the case if they exist.
Another lineof works use more relaxed certiﬁcation [47, 11, 36].
Although these methods aim to guarantee thatan adversary does not exist in a certain region around a given input, they suffer from scalability to bignetworks and large datasets.
The only scalable to large datasets provable defense against adversarialperturbations is randomized smoothing .
Initially, it was found as an empirical defense to mitigateadversarial effects in neural networks [29, 49].
Later several works showed its mathematical proofof certiﬁed robustness [21, 25, 3, 38].
Lecuyer et al [21] ﬁrst provided proof of certiﬁcates againstadversarial examples using differential privacy.
Later, Cohen et al [3] provided the tightest boundusing Neyman-Pearson lemma.
Interestingly, alternative proof using Lipschitz continuity was found[38].
The scalability and simplicity of randomized smoothing attracted signiﬁcant attention, and itwas extended beyond l 2  perturbations [22, 42, 27, 24, 23, 20, 32, 51].
Conclusion and future work
In this work, we extended randomized smoothing as a defense against additive norm-boundedadversarial attacks to the case of classiﬁcation in the embedding space that is used in few-shotlearning scenarios.
We performed an analysis of Lipschitz continuity of smoothed normalizedembeddings and derived a robustness certiﬁcate against l 2  attacks.
Our theoretical ﬁndings aresupported experimentally on several datasets.
There are several directions for future work: ourapproach can possibly be extended to other types of attacks, such semantic transformations; also, it isimportant to reduce the computational complexity of the certiﬁcation procedure.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs areincluded in the supplementary material.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] The code withinstructions is included in the supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] Applicable training details are described in Section 5.2.
(c) Did you report error bars (e
g., with respect to the random seed after running experi- ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 5.2.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Sections 5.1 and
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] The datasets and models are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]