O RIENT : Submodular Mutual Information Measures for Data Subset Selection under Distribution Shift
Abstract
Real-world machine-learning applications require robust models that generalizewell to distribution shift settings, which is typical in real-world situations.
Domainadaptation techniques aim to address this issue of distribution shift by minimizingthe disparities between domains to ensure that the model trained on the sourcedomain performs well on the target domain.
Nevertheless, the existing domain adaptation methods are computationally very expensive.
In this work, we aim to improvethe efficiency of existing supervised domain adaptation (SDA) methods by using asubset of source data that is similar to target data for faster model training.
Specifically, we propose O RIENT , a subset selection framework that uses the submodularmutual information (SMI) functions to select a source data subset similar to the target data for faster training.
Additionally, we demonstrate how existing robust subsetselection strategies, such as G LISTER , G RAD M ATCH , and C RAIG , when used witha held-out query set, fit within our proposed framework and demonstrate the connections with them.
Finally, we empirically demonstrate that SDA approaches liked -SNE, CCSA, and standard Cross-entropy training, when employed together with
O RIENT , achieve a) faster training and b) better performance on the target data.
Introduction
The recent success of deep learning frameworks in applications such as image classification [10],speech recognition [21], and object detection [14] stems primarily from the availability of largeamounts of labeled data.
However, due to enormous labeling costs and the need for specialists inspecific domains such as medical imaging, it is not always possible to obtain vast amounts of labeleddata in all situations.
On the contrary, training deep models on limited amounts of data can lead topoor performance due to overfitting [2].
Consequently, where obtaining large amounts of labeled datais difficult for the target domain, closely related domain (source domain) is used to train the model.
However, this may result in a deep model with suboptimal performance on target domain as a resultof the distribution shift [1, 6, 5, 50], i.e., change of data distribution from source domains to targetdomains.
A change in the data distribution often renders the features learned by the model on thesource domain irrelevant for the target domain.
To address the problem of distribution shift, many domain adaptation [54, 42] and domain generalization [41] techniques have been proposed in recent years.
Domain adaptation methods assume thatsome target domain information is available (usually target data), whereas domain generalization methods do not.
Domain adaptation (DA) methods can be categorized into unsupervised [16, 13, 33, 52](using unlabeled target data), semi-supervised [17, 59, 44, 48] (using labeled and unlabeled targetdata), and supervised [40, 39, 58, 38, 20] (using labeled target data) methods.
UDA methods that donot require any labeled target data assume access to large volumes of unlabeled target data.
Whenproviding the same amount of target data, SDA methods are more effective than UDA methods [40].
As it is not difficult to obtain a few samples (as less as 2 examples per class) of labeled target data in practice, SDA methods are an attractive approach in scenarios with limited target data.
Hence, in thiswork we are explicitly focused on the SDA setting.
Recently, various effective SDA methods are proposed [40, 58, 38].
However, they are usuallycompute intensive.
For example, using one of the state-of-the-art SDA method, d -SNE [58], to traina ResNet50 model [19] on the Office-Home dataset [53] with 3022 samples of the a source dataand 338 samples a target data for 300 epochs takes > 18 hours using a GTX 1080 Ti GPU.
To putthat in perspective, training a ResNet50 model on the larger CIFAR100 dataset with 45000 trainingsamples for 300 epochs on the same machine for standard cross-entropy loss takes only 14 hours.
Theincrease in training time not only increases energy consumption and CO2 emissions but also restrictsthe usefulness of SDA methods in resource-constrained environments.
We address this problem byseeking an answer to the following question: Can we improve the efficiency of SDA methods bytraining on subsets of source data to ensure faster adaptation and reduction in training time?
To this end, we propose O RIENT , a subset selection framework that utilizes Submodular Mutual
Information (SMI) measures [22, 18] to select subsets of source data that are similar to target data fortraining.
Our key insight is that by training the model on data points that are similar to target data, wecan speed up the training.
The O RIENT framework, illustrated in Figure 1, complements existing
SDA methods and can be effectively combined with any of them, enabling us to achieve a reductionin training times, energy costs, and CO2 emissions.
The key contributions of our paper include: (1) O RIENT , an SMI-based subset selection framework for efficient and effective supervised domain adaptation.
In Section 3, we illustrate that
O RIENT unifies three existing subset selection methods: CRAIG [36], GLISTER [26], and G RADM ATCH [25].
Specifically, these methods fit into O RIENT ’s framework when used with a held-outvalidation set from the target domain.
(3) We empirically show the effectiveness of O RIENT whenused with existing SDA methods like CCSA [40], d -SNE [58], and Standard Cross-entropy trainingon two real-world datasets: Office-31 [43] and Office-Home [53].
Our experiments also demonstratethat model learn better class discrimination features with O RIENT , resulting in better classificationperformance.
1.1 Related work
SDA When the labeled training data is scarce for a domain τ (or a task), a powerful way to boostperformance is by pretraining a model on a related domain and fine-tuning it in the target domain[15, 60, 9].
I SDA techniques have been investigated for various distribution shift settings where thetarget domain ( τ t = { X t , Y t } ) has different marginal or conditional distributions than the sourcedomain ( τ s = { X s , Y t } ).
In this work, we consider covariate shift [1], wherein the marginaldistibution is different (cid:0)
i.e. P ( X s ) ̸
= P ( X t ) (cid:1) , but the conditional distribution remains the same 71 (cid:0) i.e. P ( Y s | X s ) =
P ( Y t | X t ) =
P ( Y | X ) (cid:1) .
A notable body of work for covariate shift SDA hasfocused on identifying a latent feature space that is domain-invariant [40, 39, 58, 38, 20].
This lineof work train a network consisting of a feature extractor and a classifier end-to-end.
The featureextractor learns a non-linear transformation of the samples from different domains to a shared latentspace, and the classifier learns to assign class labels.
Different researchers have proposed differentoptimization objectives and loss functions to encourage domain confusion and class separability.
Tzeng et al.
[51] designed a domain-confusion loss to optimize for domain invariance and match thedistribution over classes in the source domain to the soft label in the target domain.
Motiian et al.[40] propose a classification and contrastive semantic alignment loss (CCSA).
CCSA uses contrastiveloss with Siamese Network to encourage the same class samples from different distributions to benearby in the latent space (semantic alignment loss), and the different class samples from differentdistributions to be far apart (separation loss).
Few-shot adversarial domain adaptation (FADA)
[39]proposes to learn similar latent space with adversarial training.
Domain-adaptation using stochasticneighborhood embedding ( d -SNE) [58] proposes to maximizes the smallest distance between thesamples of different classes and minimizes the largest distance between the samples of the same classwhile being domain invariant.
Another approach, Second- or Higher-order Transfer of Knowledge(So-HoT) [30], aims to align the with-in class scatters and maintain separation of between-classscatters.
Finally, Morsing et al.
[38] propose domain adaptation using graph embedding (DAGE) andshow that CCSA and d -SNE can also be expressed as graph embedding methods [20].
Subset Selection methods Submodular function [34, 12] is one of the effective approach of datasubset selection which is used in variety of applications [7].
For example, it has been employedfor efficient training in speech recognition [56, 55], machine translation [29], computer vision [23],supervised classification [36, 26, 25], semi-supervised classification [27], active learning methods [57,47, 3, 26], and hyper-parameter optimization [28].
Another widely used method for selecting subsetsis coreset construction.
The coresets [11] are weighted subsets of the data that approximate thesemantic characteristics of the entire dataset (e.g., loss, marginal probability, etc.).
A number ofrecent coreset selection-based methods [37, 26, 25, 27] have demonstrated the promise to efficientlyand robustly train deep models.
Coresets are used for other deep learning applications like continuallearning, active learning and data summarization [8, 49].
Subset selection plays a significant role in robust learning under realistic scenarios, such as imbalancesor rare classes, out-of-distribution(OOD) data, or redundancy.
Axelrod et al.
[4] propose a simplecross-entropy based data selection methods for effective domain adaptation in statistical machinelearning.
Kothawade et al.
[31] employed SMI measures for active learning in realistic scenariostackling imbalance, OOD, and redundancy.
PRISM [32] employed SMI measures for targeted datasummarization.
GLISTER [26] and G RAD M ATCH [25] shows that using a clean held-out validationset with SMI mitigates the label noise and class imbalance in the training set.
Mirzasoleiman et al.
[37]showed the data samples with clean labels cluster together, whereas the ones with noisy labels spreadout in the gradient space, thus making k-medoid clustering an effective method for reducing noisein labeled data.
Furthermore, Killamsetty et al.
[27] tackles OOD and imbalance in the unlabeleddata in semi-supervised learning setting through data subset selection.
We show how these previoussubset selection strategies including a modified version of C RAIG [36] were also using instantiationsof SMI measures in Section 3.
Preliminaries
In this section, we introduce submodular functions, SMIs and SDA loss functions.
Submodular Functions:
Let D be a set of n data points D = { 1 , · · · , n } and f : 2 D → R bea set function returning real-value for any subset of set D .
A function f is submodular [12] iff ( A ∪ x ) − f
( A ) ≥ f
( B ∪ x ) − f ( B ) , ∀ x ∈ D , ∀ A ⊆ B ⊆ D and x / ∈ B .
Monotone Submodularfunctions, when maximized for a cardinality constraint using a simple greedy algorithm, admit aconstant factor of 1 − 1 e .
This can be done in near-linear time using a stochastic-greedy algorithm.
Submodular Mutual Information (SMI): The SMI between two sets A, B ⊆ D , instantiated witha submodular function f , is defined as I f
( A ; B ) =
f ( A ) + f ( B ) − f
( A ∪ B ) [22].
In this work, weaim to select data points A ⊂ D that maximize the SMI between A and B .
Intuitively, this allowsselection of data points that are similar to B while being diverse.
SDA Loss Functions: For supervised domain adaptation, specialized loss functions are introducedas described in the previous section.
Here we highlight two state-of-the-art domain adaptation lossfunctions, CCSA and d -SNE, which are used in our experiments.
In CCSA, the classifier is modeled as a composition of two functions– h ◦ g .
Here, g : X → Z is afeature extractor that transforms the input from the feature space X to an embedding space Z , andh : Z → Y is a predictor function.
The CCSA loss for supervised domain adaptation is defined as,
[Equation]
where L CE ( h ◦ g ) is the cross-entropy loss for multi-class classification, L SA is a semantic alignmentloss encouraging the samples from different domains but same label to map nearby in the embeddingspace, and L S is a separation loss encouraging the samples from different domains and differentlabels to map far away in the embedding space.
The d -SNE loss function for SDA is defined as
[Equation]
where L s CE and L t CE are the cross-entropy loss on the source and the target domain examples,respectively, and ˜ L minimizes the largest distance between the samples from different domains withsame label and maximizes the smallest distance between the samples from different domain withdifferent label in the embedding space.
SDA loss functions are further elaborated in Section A.4.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
COM
Appendix Eq.
[Equation]
O RIENT
For our SDA setting, we assume a small amount of labeled data (as less as 2 examples per class) isavailable from the target domain τ t and sufficient labeled data is available from the source domain τ s .
We denote the source dataset as D s =
( x i , y i )
| D S | i =1 and the target dataset as D t = ( x i , y i ) | D t | i =1 .
Thesource dataset is used for training the model M using an SDA loss function L .
We will introduceother necessary notations in the remainder of the paper, as necessary.
A complete list of notations ispresented in Appendix (A.1).
First, we extend the SMI measure to incorporate data subsets from different domains.
For a sourcedomain subset A ⊆ D s and the target domain data set D t , we denote SMI measure as I f
( A ; D t ) .
Essentially, assuming that both the domains form a set D , and A ⊆ D s ⊂ D and D t ⊂ D .
With Sdenoting the similarity matrix defined over a subset A and D t , Table 1 presents the mathematicalexpression of the SMI functions for different instantiations of submodular functions ( f ) [32].
Wedefer the readers to Kothawade et al.
[32] for more details.
Note that the FLMI, GCMI and COMonly need the pairwise similarities of the data points in A and D t .
Consequently, the similarity kernelis of size
| A | × | D t | , making it very efficient to optimize.
For efficient SDA, we want to select a subset A of the source domain τ s such that training a modelon A results in a classifier that is proficient on the target domain τ t .
To this effect, we use thegradients χ of the current model to represent the data points and use it to compute the similaritymatrix.
Specifically, we define the pairwise similarity between two data points as the cosine similaritybetween the gradients,
[Equation]
where, χ i = ∇ θ L ( x i , y i ) and χ j = ∇ θ L ( x j , y j )
Given a set size b , we select a diverse subset A of the source data D s that is similar to the target data
D t by maximizing the following SMI measure,
[Equation]
I f
( A ; D t ) is an instance of cardinality constrained monotone submodular maximization for all
SMI functions except for LogDetMI.
Therefore, we can achieve a 1 − 1 e approximation using thelazy greedy algorithm [35].
Even though LogDetMI is not submodular, previous works [32, 31]have reported good empirical performance using the lazy greedy algorithm for maximization of the
LogDetMI function.
Following the footsteps of Kothawade et al.
[32, 31], we also use the lazy greedyalgorithm to maximize the LogDetMI function.
However, as the number of parameters in the deeplearning model can be extremely high, our gradients χ can be very high dimensional.
Following thesuccess of targeted subset selection approaches, GLISTER [26], SIMILAR [31], CRAIG [37] and
BADGE [3], we circumvent this problem by using last-layer gradient approximations to represent thedata point in the similarity matrix S .
Further, we select a new subset every L epochs as the model istrained, and the subsets chosen are adapted accordingly.
Algorithm: We present the complete training procedure of O RIENT in Algorithm 1.
We firstrandomly initialize the training subset A and the model parameters θ in line 1 and 2 , respectively.
For each epoch, we update the model parameters by optimizing a gradient-descent (GD) based loss Lon the current training subset A in line 4 .
After a fixed interval of L epochs, we update the subset A( line 5–11 ).
To do this, we compute the gradients χ s and χ t for all the datapoints in D s and D t , inline 6 and 7 ,respectively.
Next, we use the gradients to compute the similarity matrix S in line 8 , asdescribed in Equation 1.
In line 9 we instantiate the SMI function I f with S and update the subset Ain line 10 using Equation 2.
Our implementation is made available online 1 .
Input: source domain data D s , query data D t , batch size b , total epochs E , subset selection interval
L , SMI function I f .
Output: Final model parameters θ 1: A ← RandomSubset( D s )
▷ Randomly initialize a subset 2: θ ▷ initial model parameters 3: for epoch
e in E do ▷ for each epoch 4: θ ← mini-batch-gd( θ, L ( A ))
▷ mini-batch gradient descent 5: if e mod L == 0 then ▷ perform subset selection every L epochs 6: χ s ← ∇ θ L ( D s )
▷ compute gradients for source data 7: χ t ← ∇ θ L ( D t )
▷ compute gradients for query data 8:
S ← S IMILARITY ( χ s , χ t )
▷ compute the similarity matrix 9:
Instantiate I f with S 10:
A ← arg max A ⊆
D s , | A |≤ b I f
( A ; D t )
▷ update subset A
Figure 1 illustrates the O RIENT workflow.
The general framework of O RIENT can incorporateany gradient decent (GD) based SDA technique.
It can train any model with loss function L byusing a subset of the source data, selected every L epochs.
In our experiments, we demonstratethe effectiveness of our method in conjunction with standard cross-entropy loss as well as twostate-of-the-art domain adaptation losses, CCSA [40] and d-SNE [58].
Connections to Previous Work
O RIENT generalizes three approaches, GLISTER [26], G RAD M ATCH [25], and a slightly adaptedversion of CRAIG [36], which were initially proposed for efficient and robust subset selection.
Specifically, these subset selection approaches maximize a submodular function corresponding to adifferent instantiations of SMI functions for subset selection.
Following theorems summarize theconnection of SMI functions with these subset selection approaches.
Theorem 1 When the outer level loss of the discrete bi-level optimization problem of GLISTER ishinge loss, logistic loss, and perceptron loss, then the optimization problem becomes an instance of
Concave over Modular (COM) SMI function.
Theorem 2 When the optimization problem of G RAD M ATCH is used to match gradients of a held-outvalidation set, it becomes an instance of the summation of GCMI and a diversity function.
(a.1)
A → D (a.2)
A → W (b.1)
D → A (b.2)
D → W (c.1)
W → A (c.2)
W → D
Theorem 3 When the optimization problem of C RAIG is adapted to match gradients of a held-outvalidation set, it becomes an instance of the FLMI function with η = 0 .
We present the proofs of these theorems in Appendix (A.3).
In conclusion, SMI measures have beenused in previous subset selection strategies for robust learning to deal with the class imbalance andnoisy labels in training data sets and achieved comparable performance to current state-of-the-artrobust supervised learning approaches.
Experimental Evaluation
Our experiments explicitly aim at answering the following questions,
Q1:
Does the proposed O RIENT approach substantially reduce the training time while maintaining comparable performance to training on complete source dataset?
Q2:
Can the proposed O RIENT approach augment the existing domain adaptation approaches?
We evaluate our proposed approach on two domain adaptation datasets: Office-31 [43] and OfficeHome [53].
Office-31 dataset consists of 4110 images of 31 object categories from three differentdomains:
Amazon (A), DSLR (D), and Webcam (W).
For our experiments we used all the imagesfrom the source domain in the training set ( D s ) and two examples of each category in the queryset ( D t ).
Office-Home dataset consists of 10812 images of 65 object categories from four differentdomains:
Art (A), Clipart (C), Product (P), and Real-world (R).
Here, the target data ( D t ) consists ofabout 20% of the images from the target domain.
We use L = 20 , that is, we sample the subset afterevery 20 epochs and use b = 0 .
3% , that is, we sample 30% of the source domain for training.
To answer the first question, we use O RIENT to train a ResNet50 architecture using the stochasticgradient descent (SGD) algorithm with the momentum of 0 .
9 and the weight decay ratio of 5 e 4 .
Wecompare our approach against three baselines:
Full —training on the source data and the target data,
(a.1)
A → C (a.2)
A → P (a.3)
A → R (b.1)
C
→ A (b.2)
C → P (b.3)
C → R (c.1) P → A (c.2)
P → C (c.3)
P → R (d.1)
R → A (d.2)
R → C (d.3)
R → P i.e. D s ∪ D t ; Random —a random subset of the source dataset and the target set, i.e. R ∪ D t , R =
RandomSubset( D s ) ; and CRAIG —a coreset of the source dataset, i.e coreset( D t ) , without anyinformation of the target dataset.
We use standard categorical cross-entropy loss function (for multiclass classification) and five different instantiations of the submodular mutual information function:
Facility Location Mutual Information ( ORIENT (FLMI) ), Graph Cut Mutual Information ( ORIENT(GCMI) ), Log Determinant Mutual Information ( ORIENT (LDMI) ), GLISTER ( ORIENT (G) ),and GradMatch ( ORIENT (GM) ).
[Table]
[Table]
Figure 2 presents the scatter plot of the speed up against the prediction accuracy of Office-31 datasetusing cross-entropy loss..
In all our experiments, we use 0 .
3 fraction of the source data as the subsetand see > 2 .
5 × speed up.
Figure 3 presents the convergence curves of the Office-Home datasetusing cross-entropy loss.
It is evident from these charts that our proposed approach O RIENT canachieve better or comparable performance to the Full training in significantly less amount of time,indicating better efficiency.
This analysis helps us answer our Q1.
O RIENT reduces the training timesubstantially, the speed up achieved is reciprocal to the fraction of the subset used, without tradingthe performance.
Rather, in some combinations, we observe that O RIENT outperforms Full.
Figure 4 presents the bar plots of the speed up when O RIENT (FLMI) is used in conjunction withd -SNE loss function on Office-31 and Office-Home datasets.
Here, we see a consistent 3 × speed upas compared to full model training with d -SNE loss.
These bar plots demonstrate that our proposedapproach O RIENT can augment existing domain adaptation approaches and substantially reducethe training times.
Additionally, Tables 2 and 3 present the test accuracy while using d -SNE and
CCSA loss in conjunction with O RIENT (FLMI) on Office-31 and Office-Home datasets, respectively.
It’s evident that augmenting existing domain adaptation approaches with O RIENT achieves betteror comparable performance to the Full training.
These two observations help us answer our Q2 inthe affirmative.
Precise values of prediction accuracy and training time for all the experiments areprovided in Appendix (A.5).
Further, in Appendix Appendix (A.7) and Appendix (A.8), we analyzethe effect of different subset sizes and subset sampling frequencies.
Figure 5 presents the GradCam [46] class-activation maps of trained models on the Office-Homedataset
( P → R setting) using d -SNE loss for both Full and O RIENT (FLMI).
These activation mapsshow that the model trained with the O RIENT framework learned better class discrimination features
[Figure] [Figure]
than the model trained with Full.
This might explain why the O RIENT framework performs bettersometimes than Full.
Comparison of different SMI functions: Domain adaptation results on Office31 (Figure 2) and
OfficeHome (Figure 3) datasets show that O RIENT (GCMI) performs suboptimally compared to
ORIENT using other SMI functions in terms of target domain accuracy.
Although O RIENT (LDMI)achieves reasonable target domain accuracy, it is computationally expensive and does not achieve thebest performance-speedup trade-off.
In comparison to the O RIENT using remaining SMI functions,i.e., O RIENT (GM), O RIENT (G), and O RIENT (FLMI), O RIENT (FLMI) consistently achieves the bestperformance versus speed-up trade-off.
We further present synthetic experiments to provide intuitionson how different SMI functions selects data subsets in Appendix A.
To summarize, our experiments on two adaptation datasets suggest that our proposed approach O RIENT can substantially reduce the training time while maintaining comparable performance to trainingon complete source data.
Additionally, our experiments using d -SNE and CCSA loss functionssuggest that O RIENT can be used in conjunction with existing SDA methods to achieve significantspeed ups in training time, while maintaining or improving the prediction accuracy.
In particular,two instantiations of our proposed approach, O RIENT (FLMI) and O RIENT (GM) consistently achievecomparable or better performance compared to Full training while being ∼ 3 × faster to train.
Conclusion, Limitations, and Broader Impact
We introduce O RIENT , a subset selection framework based on SMI functions for supervised domainadaptation.
The submodularity of SMI functions allows us to use scalable greedy algorithms to selectthe data subsets efficiently.
In addition, we demonstrate how O RIENT is a unified framework thatintegrates previous approaches based on subset selection for robust learning.
Empirically, we showthat O RIENT is very effective for SDA.
Specifically, it achieves ∼ 3 × speed up over existing SDAapproaches like d -SNE and CCSA while achieving comparable or better performance.
Our findingsconfirm that O RIENT has a significant social impact by making existing SDA algorithms significantlyfaster and more energy-efficient, reducing CO2 emissions and energy consumption incurred duringtraining, thus contributing to a Green AI [45].
The main limitation of our work is that although
O RIENT significantly reduces the training time, it requires more memory to store the similarity kernelrequired for subset selection.
This makes running O RIENT harder on devices with low memory.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See Section 3 for theoretical results and Section 4 for
empirical evaluation of our claims.(b)
Did you describe the limitations of your work?
[Yes] We discuss limitations of ourproposed approach in Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
We
discuss the potential societal impact of our approach in Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] We’ve ensured that our paper conforms to the guidelines to the best ofour knowledge.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Sections
3 for theoretical results and A.3 for details
(b) Did you include complete proofs of all theoretical results?
[Yes] See A.3.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See C.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please refer to Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Please refer to A.5 for precise results for all
our
experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please refer to Section A.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please refer to
dataset and model architecture citations.
(b) Did you mention the license of the assets?
[Yes] Please refer to Section D.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Please refer to Section C for our code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Please refer to Section D for licenses for the datasets used.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
Our data does not use personally identifiable
information or offensive content.
Please refer to Section D and Section A.5 for details
regarding the data used.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
Our work does not use research involving human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
Our work does not use research involving
human subjects.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
Our work does not use research involving
human subjects.