Improving Out-of-distribution Robustness by Adversarial Training with Structured Priors
Anonymous Author(s)
AffiliationAddress email
Abstract
Deep models often fail to generalize well in test domains when the data distribution differs from that in the training domain.
Among numerous approaches toaddress this Out-of-Distribution (OOD) generalization problem, there has been agrowing surge of interest to exploit the input-robustness obtained by Adversarial
Training (AT) to improve OOD performances.
Recent works have revealed that therobust model obtained by conducting sample-wise AT also retains transferabilityto biased test domains.
In this paper, we empirically show that sample-wise AThas limited improvement on OOD performance.
Specifically, we find that AT canonly maintain performance at smaller scales of perturbation while Universal AT(UAT) are more robust to larger-scale perturbations.
This provides us with clues that the adversarial perturbations with universal (low dimensional) structures can enhance the robustness to large data distribution shifts which are common in OODscenarios.
Inspired by this, we propose two AT variants with low-rank structures totrain OOD-robust models.
Extensive experiments on DomainBed benchmark showthat our proposed approaches outperform Empirical Risk Minimization (ERM) andsample-wise AT.
Introduction
Existing deep learning methods have achieved good performance on visual classification tasks underthe same distribution of training sets and test sets.
However, when the data distribution of the test setis different from that of the training set, the classification performance of the deep neural networks(DNNs) may decrease sharply [1].
This is mainly because the network may capture spurious featuressuch as background and style information to assist the fast fitting during the training process [2].
However, in real-world scenarios, test data may differ from training data in background and styleinformation, thus DNNs that use unstable spurious features to make predictions will fail.
Solving theabove problem is known as the out-of-distribution (OOD) generalization.
Another scenario where DNNs may fail is that deep models are often vulnerable to adversarialexamples [3].
Adversarial training (AT) is originally proposed as an effective way to defend againstadversarial attacks [4].
Moreover, there is work showing that adversarial training helps to solvethe OOD generalization problem because OOD data can be seen as stronger perturbations to someextent [5].
The reason why AT can defend adversarial attacks meanwhile benefit OOD generalizationis that it can make the network robust to the interference of spurious features, such as randomlyinjected noise (in adversarial examples) or the spurious correlation between labels and backgroundinformation (in OOD generalization).
In other words, AT enables the network to make predictionsusing intrinsic features rather than spurious features.
A potential problem, however, is that existing AT methods for solving OOD generalization ignore thespecific design of perturbations.
They usually simply conduct sample-wise AT [6], which only brings limited performance improvement for OOD generalization.
The essential reason for the failure of thistype of approach is that the perturbations it uses cannot distinguish between invariant and spuriousfeatures.
As a result, it improves the robustness at the expense of the standard accuracy decreasing[7].
Moreover, we empirically find that universal AT [8] , i.e. with domain-wise perturbations, showsstronger input-robustness when facing larger-scale perturbations comparing to sample-wise AT.
Sincethe sample injected with large-scale perturbations can be regarded as OOD samples [5], we drawinspiration from this phenomenon that AT with universal (low-dimensional) structures can be the keyto solving OOD generalization.
To address the above problems, we propose to use structured low-rankperturbations related to domain information in AT, which can help the model to filter out backgroundand style information, thus benefits OOD generalization.
We make the following contributions in ourwork:
• We identify the limitations of sample-wise AT on OOD generalization through a series ofexperiments.
To alleviate this problem, we further propose two simple but effective AT variants with structured priors to improve OOD performances.
• We theoretically prove that our proposed structured AT approach can accelerate the conver- gence of reliance on spurious features to 0 when using finite-time-stopped gradient descent,thus enhancing the robustness of the model against spurious correlations.
• By conducting experiments on the DomainBed benchmark [9], we demonstrate that our algorithms outperform ERM and sample-wise AT on various OOD datasets.
Related Work
Solving OOD Generalization with AT.
According to [3], the performance of deep models aresusceptible to small-scale perturbations injected in the input images, even if these perturbationsare imperceptible to humans.
Adversarial training (AT) [4] is an effective approach to improve therobustness of the model to input perturbations.
However, many of the recent works have begun tofocus on the connection between AT and OOD due to the fact that OOD data can be regard as onekind of large-scale perturbations.
These works seek to exploit the robustness provided by AT toenhance the OOD generalization performance.
For instance, [6] applies sample-wise AT to OODgeneralization.
They theoretically find that if a model is robust to input perturbation on trainingsamples, it also generalizes well on OOD data.
[5] theoretically establishes a link between theobjective of AT and the OOD robustness.
They reveals that the AT procedure can be regarded as aheuristic solution to the worst-case problem around the training domain distribution.
Nevertheless,the discussion of [6] and [5] is restricted to the framework of using Wasserstein distance to measurethe distribution shift, which is less practical for the real-world OOD setting where domain shifts arediverse.
Additionally, they only study the case of sample-wise AT and did not further investigatethe effect of different forms of AT (not sample-wise) on the robustness performance of OOD.
Otherworks such as [10] focuses on the structure design of the perturbations.
In [10] they use multi-scaleperturbations within one picture.
But they do not exploit the universal information within one trainingdomain.
In our work, we focus on real-world OOD scenarios where there are additional clues lyingin the distribution shifts, i.e, the low-rank structures in the spurious features (such as background andstyle information) across one domain.
We further design a low-rank structure in the perturbations tospecifically eliminate such low-rank spurious correlations.
OOD Evaluation Benchmark.
The DomainBed benchmark [9] provides a fair way of evaluatingdifferent state-of-the-art OOD methods, which has been widely accepted by the community.
Byconducting rigorous experiments in a consistent setting, [9] reveals that many algorithms that claimto outperform previous methods cannot even outperform ERM.
Unlike previous works using AT toaddress OOD generalization, such as [6] and [5], in this work we adopt the Domainbed benchmarkfor a fair comparison of our approach with existing state-of-the-art methods.
Weakness of Sample-wise AT for OOD
Generalization 3.1 Preliminaries
Out-of-distribution (OOD) Generalization .
Let x ∈ X denote the random data in the input space
X , y ∈ Y denotes the target random data in label space Y , and ϕ :
X → Z denotes the feature [Table]
extractor, we have the predictor as f = w ◦ ϕ
( x ) where z = ϕ ( x ) is the feature variable in the featurespace Z and w : Z → Y denotes the classifier.
Now we give the formal definition of the out-of-distribution generalization problem.
We have a setof m training domains E = { E 1 , E 2 , ..., E m } , where each domain E e is characterized by a inputdataset E e := { ( x ei , y ei ) } n e i =
1 containing n e i.i.d input samples drawn from the distribution of P e , anda test domain E m +1 with data following the distribution of P te , where P te ̸
= P i , i = 1 , 2 , ..., m .
L : X → R + denotes the loss function.
The ultimate goal of OOD generalization is to find an optimalpredictor f that minimizes the risk on the unseen test domain:
[Equation]
Adversarial Training (AT) 1 .
According to [4], AT can be expressed as the following optimizationproblem:
[Equation]
) where δ ∈ S is the random injected perturbation with l p norm bounded by ϵ > 0 .
The innermaximization problem can be optimized by fast gradient sign method (FGSM [11]), a simple one-stepscheme:
[Equation]
where sgn ( · ) is the sign function, or by projected gradient descent (PGD [4]), a more powerfulmulti-step variant:
[Equation]
where (cid:81) is the projection operator onto the ϵ -ball and α is the step size.
3.2 Weakness of AT for OOD Generalization
We now highlight some weakness of sample-wise AT for OOD generalization based on series ofempirical evidence.
We first conduct a toy experiment on the DomainBed benchmark [9] to evaluatethe OOD performance of AT.
We run ERM and AT on four OOD datasets: PACS [12], OfficeHome[13], VLCS [14], and NICO [15] with a fixed set of hyperparamters (Detailed experimental settingcan be found in A.1).
The results are shown in Table 1.
We can see that the improvement of OODperformance by AT is limited with an average improvement of only 0.1%.
We further investigate the reason behind the limitations of performance improvements on OOD datasets of AT.
Although previous works have revealed that the robust features obtained by AT can improve the model generalization under OOD ([6] [5] [16]), we find that existing sample-wise AT methods only tolerate small scale perturbations.
Thus, we design an experiment on NICO dataset with multiple scales of perturbations.
The experiment details are shown in
A.1. [Figure]
[Equation] [Figure] we show that AT will suffer severe performance degradation when using large perturbations (shownin Figure 3.2).
This provides clues to understand the failure of AT in OOD scenarios.
The distribution shifts in OOD data usually have much larger scales than the invisible perturbations commonly used in
AT [5].
Hence, AT methods designed for small perturbations cannot handle these large scale domainshifts that often appear in OOD data.
However, our experiment shows that we can alleviate thisproblem by adapting universal AT (UAT, [8]) to the OOD setting, i.e. using domain-wise perturbations.
Figure 3.2 shows that UAT will keep its generalization performance when the perturbation scaleis large.
There are two empirical explanations for this: first, the background and style informationusually have a low-rank structure, such as the grassland and snowfield that have recurring parts.
Second, similar spurious features often appear within one specific domain, such as PACS [12] and VLCS [14] datasets.
As stated in [8], the universal perturbation lies in a low dimensional space.
Hence using universal (domain-wise) perturbations will help to eliminate such low-rank shifts andimprove model robustness.
Inspired by this, we proposed two new AT algorithms with more sophisticated low-rank structures ondifferent dimensions to improve OOD generalization in the next section.
The Proposed Structured AT Method
In order to construct low-rank structured perturbations, we start by analyzing the structure of samplewise perturbations.
Assume that each input data x has a shape of N × N × C .
N is the size of the inputimage and C is the number of the channels.
For simplicity, we assume C = 3 .
We reparameterizethe sample-wise perturbations as a series of 2-D matrices { D 11 , D 12 , ..., D 1 m } , { D 21 , D 22 , ..., D 2 m } ,{ D 31 , D 32 , ..., D 3 m } where D ce ∈ R n e × ( N ∗ N ) denotes the perturbations in the e -th domain for theinput channel c , n e is the number of the samples in domain E e , and m is the number of domains.
The i -th row of D ce represents the c -th channel of the i -th sample in domain E e (see the first columnin Figure 1 for illustration).
By such reparameterization, it is natural to find that there are twoorientations to reduce the rank of the perturbations:
Along the dimension of the number of samples (along the red arrow in the upper left corner of Figure 1).
This corresponds to reduce the number of the perturbations used withinone domain.
Along the dimension of the input scale (along the blue arrow in the upper left corner ofFigure 1).
This corresponds to reduce the rank of the perturbation used for a specific inputsample.
In the following sections, we will propose two AT variants with structured priors that reduce the rankin these two directions.
4.1 MAT: Adversarial Training with Combinations of Multiple Perturbations
In this section, we propose domain-wise Multiple-perturbation Adversarial Training (MAT).
It aimsto conduct rank minimization along the dimension of the number of samples.
Instead of usingsample-wise perturbations, MAT constructs a combination of multiple perturbations and shares thismixed perturbation within a domain.
Specifically, we choose to train the linear combination of kperturbations for each domain E e to conduct AT.
Here k is a hyperparameter and k is far less thanthe number of samples in domain E e .
The optimization problem can be reformulated as:
[Equation]
[Equation]
[Equation]
where
[Equation]
[Equation]
Here e denotes a training domain and α ei is the weight that can be learned for each perturbation δ ei .
MAT works as a low-rank version of sample-wise AT.
In sample-wise AT, we maintain n e perturbations for each domain E e , where n e is the number of training samples in domain E e . As for MAT, Sample-wise AT
MAT rank= rank= LRAT rank= [Figure] it reduces the number of perturbations available to samples from n e to k and to obtain low-rankstructures (see the second column in Figure 1 for illustration).
Therefore, it fulfills rank reductionalong the sample-number dimension.
4.2 LDAT: Adversarial Training with Low-rank Decomposed Perturbations
Based on MAT, we further propose Adversarial Training with Low-rank Decomposed perturbations(LDAT).
Analogous to MAT, LDAT still shares one perturbation in a specific domain.
Moreover,
LDAT imposes a low-rank constraint on the perturbation itself, which corresponds to the dimension ofthe input scale.
Technically, we obtain the domain-wise low-rank perturbation matrix δ ∈ R N × N × Cby multiplying two matrices: δ = AB .
Here A ∈ R N × l × C and B ∈ R l × N × C , l is a hyperparameterand l ≪ N .
Since rank ( AB ) ≤ rank ( A ) and rank ( AB ) ≤ rank ( B ) hold for arbitrary matrices A ,
B , we have rank ( δ ) ≤ l .
Therefore LDAT reduces the rank of the perturbation from a large value Nto a relatively small value l (see the third column in Figure 1 for illustration).
The formal definitionof the LDAT objective is:
[Equation]
where
[Equation]
Comparison to MAT, LDAT xxx
4.3 Theoretical Analysis
In this part, we theoretically explain why the domain-wise perturbation proposed in MAT and LDATcan help to improve the robustness of the model against spurious correlations following [2] and [17].
In general, we prove that MAT and LDAT can prevent the model from relying more on the spurious features to make predictions as the spurious correlations in the training data increase.
Consequently,the model trained with MAT or LDAT will generalize better on OOD data.
Notations.
Let x ∈ X denote the random data in the input space X and y ∈ Y denotes the targetrandom data in label space Y .
For simplicity, let Y ∈ { 1 , − 1 } in this section.
Let D denote thedistribution over X × Y .
Let x inv and x sp denote the invariant features and the spurious featuresrespectively.
Also for simplicity, assume that there exists an identity mapping Φ : X inv × X sp → Xsuch that each D ∈ D is induced by a distribution over X inv × X sp (so x can be denoted asx = ( x inv , x sp ) ).
Let x sp take values in { + β, − β } for some β > 0 .
A Simple OOD Task.
Consider a simple OOD task where we have two training domains representingthe grass and desert background respectively.
Both domains have two classes: the cow class and thecamel class.
In the grass/desert domain, the cow/camel class predominates.
During test-time, thecorrelation between labels and backgrounds flips.
We can abstract this cow-camel dataset into thefollowing model: a training dataset S with four groups of data points drawn from the four quadrantsof the feature space {− 1 , +1 } × {− β, + β } respectively.
We set the invariant features x inv =
y andthe spurious features x sp to be yβ with probability p ∈
[0 . 5 , 1) and − yβ with probability 1 − p .
Forexample, the data points of the cow class in the grass and the desert domain can be expressed asx
=
(1 , β ) and x = (1 , − β ) respectively.
Note that p measures the intensity of spurious correlationsin a certain environment.
Specifically, in the desert domain, the ratio of the camel samples is p and asfor the grass domain, the ratio of the cow samples is p .
It is easy to know that when p = 0 .
5 , thereare no spurious correlations between the labels and the background in this dataset.
Consider a linear classifier h ( x ) =
w inv x inv + w sp x sp .
Following [17], let us consider MAT/LDATtrained with gradient descent algorithm stopped in finite time t .
In order to characterize the dependence of the model on spurious features during the training process, we investigate the convergencerate of w sp β w inv ( t ) x inv to 0 on the above toy dataset.
The value w sp β w inv ( t ) x inv denotes the ratio betweenthe output of the spurious component to that of the invariant component.
We will prove that afteradding the domain-wise perturbations in finite-time-stopped gradient descent, the lower bound ofthe convergence rate of this ratio does not increase monotonically with p .
Hence, the model will notlearn a large prediction weight based on spurious features even if the spurious correlation is strong ( pis large).
In the following theorem, we denote the domain-wise perturbation in MAT/LDAT as δ .
Theorem 4.1applies to both MAT and LDAT since they both use domain-wise perturbations.
See appendix B for aformal statement and full proof of 4.1.
Theorem 4.1.
(informal)
Let H be the set of linear classifiers h ( x ) =
w inv x inv + w sp x sp .
Considerthe above 2-D OOD dataset S .
Assume that the empirical distribution of x inv given x sp · y > 0is identical to the empirical distribution of x inv given x sp · y < 0 .
δ is the optimal perturbationobtained by optimizing (7) or (9).
Let w inv ( t ) x inv + w sp ( t ) x sp be initialized to the origin, andtrained with MAT/LDAT to minimize the exponential loss on S .
Then, for any ( x, y ) ∈ S , we have:
[Equation]
| , (10) where ϵ :
= δy 2 β is a real number close to 0, c := 2(2 M − 1) β, c 1 :=
2(2 M − 1) β ( β − δy ) 2 e winvδy , c 2 :=2(2 M − 1) β ( β 2 − δ 2 ) e winvδy ( β − δy β + δy ) β + δy 2 β .
M = max x ∈ S ˆ w · x denotes the maximum value of the margin of themax-margin classifier ˆ w on S .
Therefore, the lower bound of the convergence rate does not increasemonotonically with p under the condition that when c 2 + ( c 1 1 − c 1 ) [-
-]12 + ϵ ( c 1 − 2 ϵ ) < 0 , the lower bounddoes not increase with p monotonically.
To sum up, since we can prevent this lower bound from growing monotonically with p , we acceleratethe convergence rate of w sp ( t ) β w inv ( t ) x inv to 0 when there is stronger spurious correlation (larger p ).
Recall thatthe ratio w sp ( t ) β w inv ( t ) x inv reflects the degree of reliance on spurious features.
Therefore, a faster convergenceof this ratio to 0 (smaller lower bound) means that the model will end up relying less on the spuriouscorrelations in finite time.
In other words, the OOD robustness can be enhanced by using domain-wiseperturbations.
Remark.
Here, we demonstrate that MAT and LDAT show stronger OOD robustness comparingto ERM.
We compare the result in Theorem 4.1 to that in the Theorem 2 in [2].
According to the Theorem 2 in [2], even if the max-margin classifier does not rely on the x sp for any level of spuriouscorrelation p ∈ [0 . 5 , 1) , ERM trained by gradient descent stopped in finite time still fails to avoidusing spurious features.
Moreover, when conducting ERM with finite-time-stopped gradient descent,the lower bound of the convergence rate of w sp β | w inv x inv | to 0 is
[Equation]
where M and c follow the definition in Theorem 4.1.
This lower bound grows monotonically with p ,thus ERM will have slower convergence for larger spurious correlations.
However, with domain-wiseperturbations, we can slightly modify the lower bound so that it does not increase monotonically withthe spurious correlation p .
Experiments
5.1 Experimental Setup
We conduct experiments on the DomainBed benchmark [9], a testbed for OOD generalization thatimplements consistent experimental protocols across various approaches to ensure fair comparisons.
We evaluate on PACS [12], OfficeHome [13], VLCS [14], NICO [15], and Colored MNIST [1].
There are several changes in our experimentation setting comparing to DomainBed:
Backbone Network.
We use ResNet-18 [18] as our backbone network for datasets excludingColored MNIST instead of ResNet-50 used in [9] for efficiency.2.
Hyperparameter Search Space.
We use a smaller hyperparameter search space than [9].We conduct a random search of 8 trials for PACS, OfficeHome, and VLCS while 6 trials forNICO and Colored MNIST in the hyperparameter search space, instead of 20 trials adoptedin [9] for feasibility.
See Appendix A.2 for more details.
Model Selection Strategy.
Since hyperparameter choice has a significant impact on the OODperformance, it is critical to use appropriate model selection method.
For PACS, OfficeHome, and
VLCS datasets, we use training-domain validation proposed in [9] since it is more in line with the
OOD scenario.
For NICO, we adopt OOD validation following [19].
For Colored MNIST, we usetest-domain validation [9] since it can enlarge the gaps in OOD performance among the algorithmswhile the gap induced by training-domain validation on Colored MNIST is marginal.
More detailscan be found in Appendix A.2.
Hyperparameters for MAT and LDAT.
To retain low-rank structures in perturbations, we setthe upper bound of the search space of the perturbation number k in MAT to be 20.
Similarly,the upper bound of the rank of the perturbation used in LDAT l is 20.
The complete setup of thehyperparameters for MAT and LDAT is provided in Appendix A.2.
5.2 OOD Performance on Benchmark
datasets
Table 2 summarizes the results on the five OOD datasets.
The results of other approaches for PACS,
OfficeHome, NICO, and Colored MNIST datasets are adopted from [19].
The results on VLCS ofother algorithms are missing (denoted as "-") because [19] does not experiment on this dataset.
Comparison with ERM and Sample-wise AT.
From Table 2, we observe that both MAT and
LDAT outperform ERM (on both our runs and the results in [19]) and AT on average.
In particular,
MAT achieves consistently better results than ERM on all five datasets.
Additionally, the averageperformance of AT is worse than ERM, which is consistent with our observations in section 3.2.
Comparison with Existing SOTA Approaches.
Although the results from [19] uses a differenttraining protocol from ours: they use a larger hyperparameter search space and 20 random search forthe hyperparameter combinations, the comparison between ERM ([19]) and ERM (our runs) indicatesthat their corresponding performances are close
Wefind that MAT outperforms all previous algorithms and LDAT ranked fourth among all methods,merely IRM [Table] 3 ± 1.
GroupDRO[26] 80.4 ± 0.
63.2 ± 0. - 71.8 ± [Table] 32.5 ± 0.
DANN[27] 81.1 ± 0. 62.9 ± 0. - 68.6 ± [Table]
24.5 ± 0.
MTL[28] 81.2 ± 0. 62.9 ± 0. [Table] [Table]
3 ± 0.
Mixup[29] 79.8 ± 0. [Table] 3 ± [Table]
- 66.6 ±
[Table]
[Table]
[Table]
ANDMask[30] 79.5 ± [Table] [Table] 0 ±
[Table] [Table] [Table] [Table]
[Table]
[Table]
MLDG[31] 73.0 ± 0.
52.4 ± 0.
[Table]
[Table] [Table] [Table] MAT (Our work) 82.3 ± 0.
[Table]
[Table]
[Table]
[Table] [Table] [Table] 4 ±
[Table]
LDAT (Our work) 82.6 ± 0.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
5.3 Empirical Understanding
Visualization.
To empirically show that MAT and LDAT can reduce the reliance on spurious features,we visualize the pixel attention heatmap of ERM, AT, MAT, and LDAT on NICO dataset using [32].
The heatmap is calculated with the second-order gradients of the last layer of the network w.r.t theoutput for the corresponding class.
It reflects the contribution of different components of the featuremap to the prediction results.
We pick the model with the best performance for each method.
Theresults in 5.3 indicate that the model trained by MAT and LDAT focuses more on the object itself,while ERM and AT adopt the background information that spuriously correlates to the class to makepredictions.
Parameter Analysis.
Conclusions
In this work, we empirically reveal the limitations of AT on OOD tasks.
Due to the lack of constraintson the perturbation, AT fails to generalize well when facing large-scale perturbations which is closeto the real-world OOD scenarios.
Based on the observation that UAT shows a stronger robustness tolarge perturbations, we further proposed MAT and LDAT to exploit the low-rank structure to alleviatethis issue.
We validate the effectiveness of the proposed method on OOD tasks both theoretically andempirically. [Figure]
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[TODO]
(b) Did you describe the limitations of your work?
[TODO](c)
Did you discuss any potential negative societal impacts of your work?
[TODO](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[TODO]2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[TODO](b) Did you include complete proofs of all theoretical results?
[TODO]
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[TODO]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[TODO]
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[TODO]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[TODO]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[TODO]
(b) Did you mention the license of the assets?
[TODO]
(c) Did you include any new assets either in the supplemental material or as a URL?
[TODO] (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[TODO]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[TODO]
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[TODO](b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[TODO] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[TODO]