CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The increasing size and complexity of modern ML systems has improved theirpredictive capabilities but made their behavior harder to explain.
Many techniquesfor model explanation have been developed in response, but we lack clear criteriafor assessing these techniques.
In this paper, we cast model explanation as thecausal inference problem of estimating causal effects of real-world concepts on theoutput behavior of ML models given actual input data.
We introduce CEBaB, a newbenchmark dataset for assessing concept-based explanation methods in Natural
Language Processing (NLP).
CEBaB consists of short restaurant reviews withhuman-generated counterfactual reviews in which an aspect (food, noise, ambiance,service) of the dining experience was modiﬁed.
Original and counterfactual reviewsare annotated with multiply-validated sentiment ratings at the aspect-level andreview-level.
The rich structure of CEBaB allows us to go beyond input featuresto study the effects of abstract, real-world concepts on model behavior.
We use
CEBaB to compare the quality of a range of concept-based explanation methodscovering different assumptions and conceptions of the problem, and we seek toestablish natural metrics for comparative assessments of these methods.
Introduction
Explaining model behavior has emerged as a central goal within ML.
In NLP, models have grown insize and complexity and while they have become increasingly successful, they have also becomemore opaque [28,36], raising concerns about trust [18,23], safety [1,34] and fairness [16,19].
Theseconcerns will persist if these models remain “black-boxes”.
Seeking to open the black-box, researchers have developed methods that try to explain model behavior[2,11,13,30,41].
However, there is no consensus about how to evaluate such methods to allow robustcomparisons.
This is not surprising, since such evaluations require very rich empirical data.
Intuitively,we would like to (1) intervene on model inputs, to modify speciﬁc concepts without changing othercorrelated information, (2) observe the effects this has on model predictions, and, ﬁnally, (3) assessexplanation methods for their ability to accurately predict these effects.
The absence of interventional data, or even an agreed-upon non-interventional benchmark, hascreated an environment in which explanation methods are often evaluated individually, and withoutcomparison to alternatives.
Attempts have been made to conduct comparative evaluations [11,17,37],but only with synthetic, simpliﬁed datasets.
Furthermore, these attempts do not deﬁne a uniﬁedevaluation approach, nor do they seek to contribute benchmark datasets that support such evaluations.
In this paper, we seek to overcome this obstacle by introducing CEBaB ( C ausal E
stimation-
Ba sed
B enchmark).
Table 1 summarizes the structure of CEBaB with a toy example: beginning with areview text from the OpenTable website, we crowdsourced edits of the original text that are designed
[Table]
[Table]
to meet a speciﬁc goal, such as changing the food rating in the original text to negative or unknown.
All of the resulting edits were validated by ﬁve crowdworkers and each full text was evaluated by ﬁvecrowdworkers for its overall sentiment.
CEBaB is grounded in 2,299 original reviews, which wereexpanded via this editing procedure to a total of 15,089 texts, targeting four different aspect-levelconcepts (food, service, ambiance, noise) with three potential labels (positive, negative, and unknown,i.e., not expressed in the review), and each full text was labeled on a ﬁve-star scale.
We focus on using CEBaB to compare concept-based explanation methods.
This allows us to gobeyond the effect of individual tokens to study how more abstract concepts (in our case, aspect-levelsentiment) contribute to model predictions (about the overall sentiment of the text).
Our proposedmetrics center around assessing concept-based explanation methods for their ability to accuratelyestimate causal concept effects [17], allowing us to isolate the effect of individual concepts.
More speciﬁcally, we use CEBaB to measure the causal effects of particular variables in a causalgraph, and we cast each explanation method as a causal estimator of these measurements.
Forexample, suppose our causal graph of the data says that all four of our aspect-level categories willaffect a reviewer’s overall rating.
To estimate the effect of positive food quality on the predictedoverall rating from a classiﬁer, we need to compare examples with high food quality to those withlow quality, holding all other aspects constant.
Such pairs of examples are normally not observed, butthis is precisely what CEBaB provides.
With CEBaB, we can directly compare the actual change inmodel predictions with the change that a concept-based explanation method predicts.
In our experiments, we evaluate ﬁve leading concept-based explanation methods: CONEXP [17],
TCAV [26], ConceptSHAP [57], INLP [40], CausaLM [11], and S-Learner [27].
These methodsmake a wide range of different assumptions about how much access we have to the model’s internalstructure, and they also diverge in the degree to which they account for the causal nature of theconcept effect estimation problem.
Remarkably, CEBaB reveals that most methods cannot beat asimple baseline.
Indeed, this negative result emphasizes the value in our primary contribution ofproviding the data and metrics that enables a direct comparison of explanation methods.
Previous Work
Benchmarks for Explanation Methods Benchmark datasets have propelled ML forward bycreating shared metrics that predictive models can be evaluated on [22,25,54,55].
Unfortunately,benchmarks that are suitable for assessing the quality of model explanations are still uncommon [10,21].
Previous work on comparing explanation methods has generally only correlated the performanceof a given explainability method with others, without ground-truth comparisons [8,20,21,43].
[Figure]
Other works that do compare to some ground-truth either employ a non-causal evaluation scheme[26], use causal evaluation metrics which do not capture performance on individual examples [52],evaluate on synthetic counterfactuals and rule-based augmentations [11,52], or are tailored for aspeciﬁc explanation method and hard to generalize [57].
To the best of our knowledge, CEBaB is the Explanation Methods and Causality Probing is a relatively new technique for understandingwhat model internal representations encode.
In probing, a small supervised [6,51] or unsupervised[5,32,44] model is used to estimate whether speciﬁc concepts are encoded at speciﬁc places in anetwork.
While probes have helped illuminate what models (especially pretrained ones) have learnedfrom data, Geiger et al.
[15] show with simple analytic examples that probes cannot reliably providecausal explanations for model behavior.
Feature importance methods can also be seen as explanation methods [33].
Many methods in thisspace are restricted to input features, but gradient-based methods can often quantify the relativeimportance of hidden states as well [3,46,48,58].
The Integrated Gradients method of Sundararajanet al.
[50] has a natural causal interpretation stemming from its exploration of baseline (counterfactual)inputs [15].
However, even where these methods can focus on internal states, it remains difﬁcult toconnect their analyses with real-world concepts that do not reduce to simple properties of inputs.
Intervention-based methods involve modifying inputs or internal representations and studyingthe effects that this has on model behavior [30, 41].
Recent methods perturb input or hiddenrepresentations to create counterfactual states that can then be used to estimate causal effects[9,12,47,53,15].
However, these methods are prone to generating implausible inputs or network statesunless the interventions are carefully controlled [14].
Generating counterfactual texts automatically remains challenging and is still a work-in-progress [4].
To overcome this problem, another class of approaches proposes to manipulate the representation ofthe text with respect to some concept, rather than the text itself [9,11,40].
These methods fall into thecategory of concept-based explanations and we discuss two of them extensively in §3.
Estimating Concept Effects with CEBaB
We now deﬁne the core metrics that we use to evaluate different explanation methods.
Figure 1provides a high-level view of the causal process we are envisioning.
The process begins with anexogenous variable U representing a state of the world.
For CEBaB, we can imagine that the valueof U is a state of affairs u of a person evaluating a restaurant in a particular way.
u contributes toa review variable X , with the value x of X mediated by u and by mediating concepts C 1 , . . .
C k ,which correspond to the four aspect-level categories in CEBaB (food, service, ambiance, and noise),each of which can have values c 2 { positive , negative , unknown } .
The review x is processed by amodel that outputs a vector of scores over classes (sentiment labels in CEBaB).
Core Metric Our central goal is to use CEBaB to evaluate explanation methods themselves.
CEBaBsupports many approaches to such evaluation.
In this paper, we adopt an approach based on individuallevel rather than average effects.
This makes very rich use of the counterfactual text and associatedlabels provided by CEBaB. The starting point for this metric is the Individual Causal Concept Effect:
Deﬁnition 1 (Individual Causal Concept Effect; ICaCE) .
For a neural network N and featurefunction  , the individual causal concept effect of changing the value of concept C from c to c 0 forstate of affairs u in an underlying data generation process G is
[Equation]
ICaCE is a theoretical quantity.
In practice, we use the Empirical Individual Causal Concept Effect.
Deﬁnition 2 (Empirical Individual Causal Concept Effect; \ ICaCE) .
For a neural network N andfeature function  , the empirical individual causal concept effect of changing the value of concept Cfrom c to c 0 for state of affairs u is
[Equation]
where ( x C =
c u , x C =
c 0 u ) is a tuple of inputs originating from u with the concept C set to the values cand c 0 , respectively.
The \ ICaCE N  for a pair of examples ( x C =
c u , x C =
c 0 u ) is simply the difference between the output scorevectors for the two cases.
With CEBaB, we can easily calculate these values because we have clustersof examples that are tied to the same reviewing situation u and express different concept values.
For assessing an explanation method E , we compare ICaCE values with those returned by E .
Ourcore metric is the ICaCE-Error:
Deﬁnition 3 (ICaCE-Error) .
For a neural network N , feature function  and distance metric Dist ,the ICaCE-Error of an explanation method E for changing the value of concept C from c to c 0 is:
[Equation]
We present results for three choices of Dist which vary in their ability to model the direction andmagnitude of effects.
These choices give subtly different but largely converging results, as detailed in
Section 6 and reported more fully in Appendix D.
Aggregating Individual Causal Concept Effect It is often useful to also have a direct estimate ofa model’s ability to capture concept-level causal effects.
For this, we employ an aggregating versionof \ ICaCE, the Empirical Causal Concept Effect:
Deﬁnition 4 (Empirical Causal Concept Effect; \ CaCE) .
For a neural network N and feature function , the empirical causal concept effect of changing the value of concept C from c to c 0 in dataset D is
[Equation]
This is an empirical estimator of the Causal Concept Effect (CaCE) of Goyal et al.
It estimates,in general, how the classiﬁer predictions change for a given concept and intervention direction.
Estimating Real-World Causal Effect of Aspect Sentiment on Overall Sentiment We can alsoestimate ground truth causal effects in CEBaB by simply using its labels directly.
There are again avariety of ways that this could be done.
We opt for the one that makes the richest use of the structuresafforded by CEBaB. For perspicuity, in parallel to the neural network-based \ ICaCE (Deﬁnition 2),we deﬁne the Empirical Individual Treatment Effect for our dataset:
Deﬁnition 5 (Empirical Individual Treatment Effects in CEBaB; d
ITE) .
The empirical individualtreatment effect of changing the value of concept C from c to c 0 in CEBaB isd
[Equation]
where f is a simple look-up procedure that retrieves the overall sentiment labels for CEBaB examples.
We aggregate over these values by taking their average, in parallel to what we do for networkpredictions (Deﬁnition 4).
This yields the Empirical Average Treatment Effect ( d
ATE) for CEBaB.
[Table] Alternative Metrics In Appendix A in our supplementary materials, we consider alternativeformulations of the core metrics with causal concept effects and absolute causal concept effects ,relating them to the different questions they engage with.
We opt for the individual causal concepteffect in our central metric (Deﬁnition 3), taking the central question to be what caused an ML modelto produce an output for an actual input created from a real-world process [Table]
Evaluated Explanation Methods
We compare several model explanation methods that share three main characteristics.
First, they are allsuitable for NLP models and have been used in the literature for generating model explanations in theform of estimated effects on model predictions.
Second, they all provide concept-level explanations,for a pre-deﬁned list of human-interpretable concepts (e.g., how sensitive a restaurant review ratingclassiﬁer is to language related to food quality).
This approach is also forward-looking, allowingmore researchers to construct new hypotheses (i.e., concepts we have not collected labels for) andestimate their effect on the predictor.
Third, all of the tested methods are model-agnostic, meaningthat they separate the explanation from the model.
At the same time, these methods differ in ﬁveimportant ways, as summarized Table 2.
We now turn to reviewing the explanation methods that we later compare on CEBaB (§6).
In ourmathematical formulas, we employ a uniﬁed notation for all methods, to make the deﬁnitions moreaccessible and easier to integrate into our experimental set-up.
Assume we have a classiﬁer N (whichoutputs a probability vector) and feature function  , and we want to compute the effect on N  ( x C
= c u )of changing the value of concept C from c to c 0 using an unseen test set ( D , Y ) . Approximate Counterfactuals The gold labels of CEBaB are the difference between the logitsfor some original review x C = c u and ground-truth counterfactual x C
= c 0 u . As a baseline, we samplean original review x C = c uwith the same aspect-labels as the x C =
c
u and use it as an approximatecounterfactual:
[Equation]
We do this sampling using predicted aspect labels from the aspect-level sentiment analysis modelsdescribed in Appendix C.
Conditional Expectation (CONEXP) Goyal et al.
[17] propose a baseline where the effect of aconcept C is the average difference in predictions on examples with different values of C . [Equation] where
Conditional Expectation Learner (S-Learner)
We adapt S-Learner , a popular method forestimating the Conditional Average Treatment Effect (CATE) [27].
To estimate causal concepteffects, our S-Learner trains a logistic regression model E to predict N (  ( x )) using the values of allthe labeled concepts of example x , denoted by x 0 .
Then, during inference, we compute an individualeffect for example pair ( x C
= c u , x C =
c 0 u ) by comparing the output of the model E x on this pair:
[Equation]
Learner
[Equation] C [Equation] At inference time, S-Learner assumes access to all aspect-level labels x 0 , which might not always beavailable.
To alleviate this issue, we instead predict the aspect-level labels x 0 from the original text xusing models described in Appendix C.
TCAV Kim et al.
[26] use Concept Activation Vectors (CAVs), which are semantically meaningfuldirections in the embedding space of  .
Our adapted version of Testing with CAVs (TCAV) outputs avector measuring the sensitivity of each output class k to changes towards the direction of a conceptv C at the point of the embedded input.
It is computed as:
[Equation]
where K is the number of classes and v C is a linear separator learned to separate concept C in theembedding space of  .
ConceptSHAP Yeh et al.
[57] propose this expansion to SHAP [30], to generate concept-basedexplanation based on Shapley values [45].
Given a complete (i.e., such that the accuracy it achieves ona test set is higher than some threshold  ) set of m concepts { C 1 , . . .
, C m } , ConceptSHAP calculatesthe contribution of each concept to the ﬁnal prediction.
Our adapted version outputs a vector foreach C 2 { C 1 , . . .
, C m } and x .
We justify this modiﬁcation and provide implementation details in Appendix H.
CausaLM Feder et al.
[11] estimate the causal effect of a binary concept C on the model’spredictions by adding auxiliary adversarial tasks to the language representation model in order tolearn a counterfactual representation  CF C ( x ) , while keeping essential information about potentialconfounders (control concepts).
Their method outputs the text representation-based individualtreatment effect (TReITE), which is computed as:
[Equation]
where  CF C denotes the learned counterfactual representation, where the information about concept Cis not present, and N 0 is a classiﬁer trained on this counterfactual representation.
A key feature of
CausaLM is its ability to control for confounding concepts (if modeled).
An inherent drawback of thistechnique is that it can only estimate interventions well for c
= Unknown, since the counterfactualrepresentation is only trained to remove a concept C .
Iterative Nullspace Projection (INLP)
Ravfogel et al.
[40] remove a concept from a representationvector by repeatedly training linear classiﬁers that aim to predict that attribute from the representationsand projecting the learned representations on their null-space.
Similar to CausaLM, INLP alsoestimates the TReATE (Equation 10) and can only estimate interventions for c
= Unknown.
The CEBaB Dataset
Table 1 provides an intuitive overview of the structure of CEBaB. In the editing phase of datasetcreation, crowdworkers modiﬁed an existing OpenTable review in an effort to achieve a speciﬁcaspect-level goal while holding all other properties of the original text constant.
Our aspect-levelcategories are food, ambiance, service, and noise.
In the validation phrase, crowdworkers labeled eachexample relative to each aspect as ‘Positive’, ‘Negative’, or ‘Can’t tell’ (Unknown).
Having ﬁve labels [Table] per example allows us to infer a majority label or reason in terms of the full label distributions.
In therating phase, each full text was labeled using a common ﬁve-star scale, again by ﬁve crowdworkers.
We began with 2,299 original reviews from OpenTable (related to 1,084 restaurants) and expandedthem, via the above editing procedure, into a total of 15,089 texts.
The distribution of normalizededit distances has peaks around 0.28 and 0.77, showing that workers made non-trivial changes tothe originals, and even often had to make substantial changes to achieve the editing goal.
(See
Appendix B for the full distribution.)
Table 3 summarizes the resulting label distributions, where an example has label y if at least 3 ofthe 5 labelers chose y , otherwise it is in the ‘no majority’ category.
99% of aspect-level edits have amajority label that corresponds to the editing goal, and 88% of the texts have a review-level majoritylabel on the ﬁve-star scale.
Overall, these percentages show that workers were extremely successfulin achieving their editing goals and that edits have systematic effects on overall sentiment.
The central goal of CEBaB is to create edit pairs : pairs of examples that come from the same originaltext and differ only in their labels for a particular aspect.
For example, in Table 1, the ﬁrst two ‘foodedit’ cases form an edit pair, since they come from the same original text and differ only in their foodlabel.
Original texts can also contribute to edit pairs; the original text in Table 1 forms an edit pairwith each of the texts it is related to by edits.
Table 3c summarizes the distribution of edit pairs, and
Table 3d reports the ground-truth d
ATE values (§3).
We release the dataset with ﬁxed train/dev/test splits.
In creating these splits, we enforce two highlevel constraints.
The ﬁrst is our ‘grouped’ requirement: for each original review t , all texts that arerelated to t via editing occur in the same split as t .
This ensures that models are not evaluated onexamples that are related by editing to those they have seen in training.
Second, if any text t in agroup received a ‘no majority’ label, then the entire group containing t is put in the train set.
Thisensures that there is no ambiguity about how to evaluate models on dev and test examples.
Once these high-level conditions were imposed, the examples were sampled randomly to createthe splits.
This allows that individual workers can contribute edited texts across splits.
This minorcompromise was necessary to ensure that we could have large dev and test splits.
Appendix C in oursupplementary materials shows that worker identity has negligible predictive power.
There are two versions of the train set: inclusive and exclusive .
The inclusive train set contains alloriginal and edited non-dev/test texts (11,728 texts).
The exclusive version samples exactly one traintext from each set of texts that are related by editing (1,755 examples).
The rationale is that models [Table] trained with an original review as well as its edited counterparts may explicitly learn causal effectstrivially by aggregating learning signals across inputs.
Our exclusive train split prevents this, whichhelps facilitate fair comparisons between explanation methods and better resembles a real-worldsetting.
Our dataset is released publicly in JSON format and is available in the Hugging Face datasets library.
It includes restaurant metadata, full rating distributions, and anonymized worker ids.
Appendix B inour supplementary materials provides additional details on the dataset construction, including theprompts used by the crowdworkers, the number of workers per task, worker compensation and asample of examples with ratings to help convey the nature of workers’ edits and the overall quality ofthe resulting texts and labels.
In addition, Appendix C reports on a wide range of classiﬁer experimentsat the aspect-level and text-level that show that models perform well on CEBaB classiﬁcation tasks,which bolsters the claim that CEBaB is a reliable tool for assessing explanation methods.
Experiments and Results
For each experiment, we ﬁne-tune a pretrained language model to predict the overall sentiment ofall restaurant reviews from our exclusive OpenTable train set.
Since the goal of our work is not toachieve state-of-the-art performance, but rather to compare explanation methods and demonstratethe usage of CEBaB, we test the ability of methods to explain commonly used models, trained withstandard experimental conﬁgurations.
In the main text, we report results for bert-base-uncased ﬁne-tuned as a ﬁve-way classiﬁer.
Appendix D includes results for GPT-2 , RoBERTa , and an LSTM , ﬁne-tuned on binary, 3-way and5-way versions of the sentiment task.
All results, including the ground-truth effect that depends onthe speciﬁc instance of a model, are averaged across 5 seeds.
To evaluate the intrinsic capacity of a model to capture causal effects, we report the \ CaCE values,as in Deﬁnition 4.
The results for bert-base-uncased are given in Table 4.
They are intuitive andwell-aligned with the d
ATE estimates in Table 3d, indicating that the model has captured the real-worldeffects. Our primary assessment of the evaluation methods is given in Figure 2, again focusing on a ﬁve-way 271 bert-base-uncased model as representative of our results.
We provide values based on cosine , L2 , and normdiff as the value of Dist in Deﬁnition 3. The cosine -distance metric measures if theestimated and observed effect have the same direction but does not take the magnitudes of the effectsinto account. The L2 -distance measures the Euclidian norm of the difference of the observed andestimated effect.
Both the direction and magnitude of the effects inﬂuence this metric. To onlycompare the magnitudes, we use the normdiff -distance, which computes the absolute differencebetween the Euclidean norms of the observed and estimated effects, thus completely ignoring thedirections of both effects.
Remarkably, our approximate counterfactual baseline proves to be the best method at capturing boththe direction and magnitude of the effects.
The fact that a simple baseline method beats almost allother methods indicates that we need better explanation methods if we are going to capture evenrelatively simple causal effects like those given by CEBaB.
Recall from Table 2 that the compared methods require different levels of access to concept labels atinference time.
Approximate counterfactuals and S-Learner have access to both the direction of theintervention and the predicted test-time aspect labels, enabling them to outperform CONEXP, whichhas access to only the direction of the intervention, and TCAV, ConceptSHAP, and CausaLM, whichhave access to neither the intervention direction nor test-time aspect labels.
The INLP method ties with the best method for the cosine metric, despite having access to neitherintervention directions nor test-time aspect labels.
Perhaps this method could be extended to make useof this additional information and decisively improve upon our approximate counterfactual baseline.
While CausaLM and INLP both estimate the effect of removing a concept from an input, INLP useslinear probes to guide interventions on the original model, while CausaLM trains an entirely newmodel with an auxiliary adversarial objective.
The direct use of the original model is something INLPshares with the approximate counterfactual baseline; it seems that a tight connection to the originalmodel may underlie success on CEBaB.
Conclusion
Our main contributions in this paper are twofold.
First, we introduced CEBaB, the ﬁrst benchmarkdataset to support comparing different explanation methods against a single ground-truth withhuman-created counterfactual texts and multiply-validated concept labels for aspect-level and overallsentiment.
Using this resource, one can isolate the true causal concept effect of aspect-level sentimenton any trained overall sentiment classiﬁer.
CEBaB provides a level playing ﬁeld on which we cancompare a variety of explanation methods that differ in their assumptions about their access to themodel, their computational demands, their access to ground-truth concept labels at inference time,and their overall conception of the explanation problem.
Furthermore, the evaluated methods makeabsolutely no use of CEBaB’s counterfactual train set.
In turn, we hope that CEBaB will facilitate thedevelopment of explanation methods that can take advantage of the very rich counterfactual structure CEBaB provides across all its splits.
Second, we have provided an in-depth experimental analysis of how well multiple model explanationmethods are able to capture the true concept effect.
A naive baseline that approximates counterfactualsthrough sampling achieves the best performance, with INLP and S-Learner being the only othermethods that achieves state-of-the art on any metric.
While CEBaB is only grounded in one task,sentiment analysis alone is enough to produce starkly negative results that should serve as a call toaction for NLP researchers aiming to explain their models.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations that we found mostsigniﬁcant are discussed in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work?
[No] The work
does not treat a speciﬁc task where we see an immediate risk for negative societal
impact.
Though the methods described in this paper might be used by practitioners,
and we suggest doing so with care since the suggested methods need to be studied
further in controlled settings before being used in such domains.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] The paper conforms with the provided guidelines.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[Yes] Code
and Data for reproducing all of our results are submitted as part of the supplementary
material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 5 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes] All results are averaged across 5 runs
(d) Did you include the total amount of compute and the type of resources used (e g., type
of GPUs, internal cluster or cloud provider)?
[Yes See supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section
(b) Did you mention the license of the assets?
[Yes] See supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] All assets we use are allowed for public use for the purposes of
scientiﬁc research.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] The data we use is taken from publicly available
datasets, where all personally identiﬁable information have been removed.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Section 5 for a high-level description of the crowdsourced data
and the supplementary material for the exact set of instructions given to crowd workers.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See supplementary material for worker pay
and costs breakdown.