Information-Theoretic Evaluation of Free-Text Rationales with Conditional V -Information
Abstract
Free-text rationales are a promising step towards explainable AI, yet their evaluation remains an open research problem.
While existing metrics have mostly focused on measuring the direct association between the rationale and a given label, we argue that an ideal metric should also be able to focus on the new information uniquely provided in the rationale that is otherwise not provided in the input or the label.
We investigate this research problem from an information-theoretic perspective using the conditional V -information [Hewitt et al., 2021].
More concretely, we propose a metric called R EV (Rationale Evaluation with conditional
V -information), that can quantify the new information in a rationale supporting a given label beyond the information already available in the input or the label.
Experiments on reasoning tasks across four benchmarks demonstrate the effectiveness of R EV in evaluating different types of rationale-label pairs, compared to existing metrics.
Furthermore, R EV is consistent with human judgments on rationale evaluations.
Overall, when used alongside traditional performance metrics,
R EV provides deeper insights into a models’ reasoning and prediction processes.
Introduction
Model explanations have been indispensable for trust and interpretability in AI [Lipton, 2018, DoshiVelez and Kim, 2017, Kim et al., 2016, Alvarez Melis and Jaakkola, 2018].
Free-text rationales, which explain a model prediction in natural language, have been especially appealing due to their flexibility in eliciting the reasoning process behind the model’s decision making [Camburu et al., 2018, Narang et al., 2020, Rajani et al., 2019, Kumar and Talukdar, 2020b, Brahman et al., 2021], making them closer to human explanations.
However, current automatic evaluation of free-text rationales remains narrowly focused.
Existing metrics primarily measure the extent to which a rationale can help a (proxy) model predict the label it explains (i.e., accuracy based)
[Hase - effe et al., 2021].
Yet, these metrics offer little understanding of the new information contained in the rationale, as added to the original input, that could explain why the label is selected —the very purpose a rationale is designed to serve.
For instance, the two rationales r ∗ 1 and ˆ r 1 ,a in Fig.
1 would be considered equally valuable under existing metrics, even though they supply different amount of novel and relevant information.
In this paper, we overcome this shortcoming by introducing an automatic evaluation for free-text rationales along two dimensions: (1) whether the rationale supports (i.e., is predictive of) the in-
People usually go hiking to enjoy nature.
Why do people go hiking?
Hiking means the activity of going for long walks especially across country, or in the nature.
People who go hiking enjoy nature.
̂ r 1, a r *̂ r 1, b ̂ r
People go hiking might end up getting lost.
Hiking is a form of exercise.
People go hiking to lose weight.
enjoy nature lose weight
People go hiking to enjoy nature.
People go hiking to lose weight.
Vacuous Baseline Rationales bb
Evaluator
High Low f REV ( x , y , r ) = log f
( y | r ) − log f
( y | r , b )
[Figure]
b = x || y where tended label, and (2) how much new information does it provide to justify the label, beyond what is contained in the input. For example, rationale ˆ r 1 ,b in Fig. 1 violates (1) because it is not predictive of the label, “enjoy nature”. Rationale ˆ r 1 ,a does support the label but contains no new information that justifies it, beyond what is stated in the input x ; thus, it violates (2).
Rationale r ∗ 1 is satisfied along both dimensions: it supports the label and does so by providing new and relevant information, beyond what is in the input.
Our proposed evaluation is designed to penalize both ˆ r 1 ,a and ˆ r 1 ,b , while rewarding rationales like r ∗ 1 .
We introduce a R EV 2 , which adapts an information-theoretic framework from Xu et al.
[2019] for evaluating free-text rationales along the two dimensions mentioned above.
Specifically, R EV is based on conditional V -information [Hewitt et al., 2021], which quantifies the degree of information contained in a representation, beyond another (baseline) representation, accessible to a model family,
V .
As our baseline representation, we consider any vacuous rationale which simply combines an input with a given label, without providing any new information relevant to answering why the label was chosen.
R EV adapts conditional V -information to evaluate rationales, where the representation is obtained via an evaluator model trained to produce a label given the rationale.
Other metrics do not take into consideration vacuous rationales, and are hence unable to measure new, label-relevant information in rationales, beyond a vacuous baseline.
Our experiments present evaluations with R EV for rationales under two reasoning tasks, commonsense question-answering (CQA; Talmor et al., 2018) and natural language inference (NLI; Bowman et al., 2015), across four benchmarks.
Several quantitative evaluations demonstrate the capabilities of R EV in providing evaluations along new dimensions for free-text rationales, while being more consistent with human judgements compared to existing metrics.
R EV : Information-Theoretic Evaluation of Rationales
We introduce a new metric, R EV , Rationale Evaluation with conditional V -information, for evaluation of free-text rationales on the proposed dimensions (§2.2), based on the information-theoretic framework of conditional V -information (§2.1).
We consider the setting where we have input X ∈ X , label Y ∈ Y , and free-text rationale R ∈ R generated for label Y .
A common strategy to evaluate rationales R is through an evaluator f ∈ V based on how much R helps f predict Y given X .
The evaluator f ∶ Z → Y maps a variable
Z to a label distribution.
The definition of Z depends on the evaluation framework; e.g., Z can be a concatenation of X and R .
The evaluator f is trained on a set of input, output and rationale triples D train = {( x j , y j , r j )} , and applied to D test =
{( x i , y i , r i )} for evaluation.
The utility of R is formulated as the difference between the performance of the evaluator on predicting Y with R , and 2 For Rationale Evaluation with conditional V -information. without it, i.e.
[Equation]
A larger performance gap indicates a better rationale.
Existing metrics [Hase et al., 2020, Wiegreffe et al., 2021] compute the performance gap based on prediction accuracies, measuring how much R can help the evaluator correctly predict Y given X .
However, accuracy-based evaluation can only indicate whether or not a rationale is predictive of a label, but cannot quantify how much new information the rationale provides to justify the label .
Fig. 1 illustrates this issue via an example.
Accuracy-based evaluation can distinguish between ˆ r 1 ,a and ˆ r 1 ,b since ˆ r 1 ,a supports y 1 and ˆ r 1 ,b does not.
However, it is unable to distinguish between r ∗and ˆ r 1 ,a (since both are predictive of y 1 ), despite the fact that ˆ r 1 ,a does not provide any unique and relevant information to answer why the label should be y 1 .
In practice, vacuous rationales such as ˆ r 1 ,a are commonly seen in model generations [Sun et al., 2022, Wiegreffe and Marasovi´c, 2021].
This calls for an evaluation metric which is able to identify and penalize such vacuous rationales.
2.1 An Information-Theoretic Perspective on Rationale Evaluation
The key quantity of interest for our evaluation of rationales R is the amount of new information expressed in R (e.g., background knowledge, reasoning process) that can justify a label Y .
The mutual information between R and Y , I ( Y ; R ) can be helpful for evaluating this quantity.
However, we are not interested in the information that is already captured in the input X .
A vacuous rationale, such as ˆ r 1 ,a in Fig.
1, which simply combines the input X and the label, Y , captures all the information in X and Y without specifying any new information to help understand why Y has been chosen for
X ; let us denote such rationales as B ∈ B .
Thus, we argue that a good evaluation metric must be able to measure the amount of relevant, new information contained in a rationale beyond what is contained in any vacuous rationale, B , that leads to the prediction of Y .
Then the new information in R beyond what is available in B can be grounded with conditional mutual information [Shannon, 1948] as follows,
[Equation]
where the difference of two information quantities demonstrates the performance gap in Equation 1.
Directly computing mutual information, however, is challenging because true distributions of random variables are usually unknown, and we do not have unbounded computation.
A recently introduced information-theoretic framework called V -information circumvents this by restricting the computation to certain predictive model families, V [Xu et al., 2019].
Our approach to evaluate rationales extends this framework, following [Hewitt et al., 2021], as described below.
Conditional V -information Given a model family V that maps two random variables R and Y ,
V -information defines the usable information that can be extracted from R by models in V to predict
Y , i.e. I V ( R → Y ) .
If V generalizes to the set of all possible functions, then V -information is mutual information [Shannon, 1948].
In practice, it is feasible to estimate the usable information from R about Y by selecting any neural model without frozen parameters as V .[-
-]
Following conditional mutual information in information theory [Cover and Thomas, 2006], Vinformation has been extended to conditional V -information (CVI; Hewitt et al., 2021).
CVI quantifies the V -usable information in R about Y conditioned on a variable B , i.e.
[Equation]
Here B is any vacuous rationale that leads to the prediction of Y .
In this work, we consider B simply as the concatenation of X and Y .
We leave analyzing how different baseline construction impacts our metric to future work.
H V ( ⋅ ∣ ⋅ ) is the conditional V -entropy
[Xu et al., 2019, Hewitt et al., 2021, Ethayarajh et al., 2022], defined as
[Equation]
where f [ b ] and f [ r, b ] produce a probability distribution over the labels given b and [ r, b ] as inputs respectively.
Further, we consider pointwise CVI for evaluating individual samples, ( r, y, b ) as
[Equation]
2.2 Computing R EV for Rationale Evaluation
Building on the framework of CVI, we propose a new metric R EV , for Rationale Evaluation with conditional V -information.
We compute R EV over a given test set, D test = {( x i , y i , r i )} , by estimating CVI over the set with an evaluator f ∈ V .
For a test example ( x, y, r ) , the R EV score denoted as R EV ( x, y, r ) is computed based on Equation 5, where b is constructed by combining x and y . [Equation] x
[Equation]
The R EV score for the test corpus D test , is given by the average pointwise R EV score:
[Equation]
The higher the R EV score, the more additional ( new and relevant ) information the rationale r contains to explain the label beyond the baseline rationale b .
R EV ( x, y, r ) can take positive, negative, or zero values.
When R EV ( x, y, r )) > 0 , the rationale supplies additional information for supporting the label (e.g., r ∗ 1 in Fig. 1); when R EV ( x i , y i , r i ) = 0 , the rationale provides no additional information beyond the baseline (e.g., r 1 ,a in Fig. 1); and when R EV ( x i , y i , r i ) < 0 , the rationale contains additional information which does not support the label (e.g., r 1 ,b in Fig. 1).
R EV can assign a positive score to a rationale for an incorrect prediction as long as the rationale supports it and provides additional information beyond a vacuous baseline rationale.
Thus, R EV cannot be seen as a replacement for prediction accuracy, but rather as an orthogonal metric to interpret the usefulness of a generated rationale for the model decision.
Constructing a Baseline with Vacuous Rationales Given an input x and a label y , we construct a baseline rationale b by converting x and y into a declarative sentence.
For the CQA task, we adopt a pre-trained T5-3B model fine-tuned on a set of ( question , answer , declarative sentence ) tuples [Chen et al., 2021] annotated by Demszky et al.
For the NLI task, we use a template to convert ( premise , hypothesis , label ) tuple into a baseline rationale: “ premise implies / contradicts / is not related to hypothesis ”.
Table 2 shows some examples of constructed vacuous rationales.
Training the evaluator, f
We select a generative model f ∈ V as the evaluator to learn the mapping f ∶ [ r, b ] → y , where the input [ r, b ] is the concatenation of r and b . {+
+}
In particular, we use pre-trained language models [e.g., T5; Raffel et al., 2020] and fine-tune them on the training set D train =
{( x, y ∗ , r ∗ )} , where { y ∗ } and { r ∗ } are gold labels and human-annotated rationales, respectively.
We construct baseline rationales { b ∗ } based on {( x, y ∗ )} .
The objective is to maximize the log-likelihood of y ∗ given r ∗ and b ∗ .
After training, the evaluator can be applied to evaluate a given rationale-label pair ( y, r ) w.r.t.
an input x .
The rationale-label pair ( y, r ) can be model-generated and the label may not be groundtruth (e.g., y 2 in Fig. 1), while R EV is still able to provide an assessment on the rationale along the two dimensions (§1), e.g., R EV ( x, y 2 , ˆ r 2 )
= 0 .
6 in Fig.
Experimental Setup
We outline our experimental setup by describing the reasoning tasks and datasets (§3.1), followed by the task and evaluation models (§3.2), and the baseline metrics for comparison (§3.3).
Additional details on the setup are provided in Appendix C.
3.1 Datasets
We explore CommonsenseQA (CQA) across 3 datasets, all containing human-annotated free-text rationales.
For CQA task, we use ECQA [Aggarwal et al., 2021], CoS-E (v1.11; Rajani et al., 2019) and QuaRTz
[Tafjord et al., 2019].
ECQA contains higher quality human-written rationales compared to CoS-E [Aggarwal et al., 2021] More details of the datasets are in Appendix C.2.
3.2 Task and Evaluation Models
Task models We choose T5 Large [Raffel et al., 2020] as the task model (finetuned on ground truth labels and rationales) to produce generated rationale-label pairs under three settings:
• XY ∗ → R:
Given an input text and the gold label, generate a rationale.
• X → YR: Given an input text, generate a label followed by a rationale.
Since T5 decodes tokens sequentially, each R is generated conditioned on the predicted Y. • X → RY: Given an input text, generate a rationale followed by a label.
Here, we compute a likelihood for each candidate Y conditioned on R, and then select the most probable candidate.
This operation can improve the model prediction accuracy, while weakening the consistency and relevance between the generated rationales and predicted labels.
After training, we collect three types of rationale-label pairs by applying the three task models on the test set of each dataset.
In addition to these three settings, we also evaluate ground-truth labels paired with crowd-sourced rationales (Y ∗ ;R ∗ ).
Evaluators Our evaluator, f
(see Equation 6 in §2) is also based on T5 Large trained on gold rationale-label pairs of the respective dataset.
We refer readers to the Appendix D.1 for results using
T5 Base, BART Large [Lewis et al., 2020] and GPT-2 Large [Radford et al., 2019] as the evaluator.
3.3 Other Metrics for Rationale Evaluation
We compare with two existing automatic metrics for free-text rationale evaluation: LAS [Hase et al., 2020] and RQ
[Wiegreffe et al., 2021].
Analogous to our evaluator, f , both approaches use proxy models; we use the same architecture (T5 Large) across metrics in our reported results.
LAS computes the difference between the prediction accuracy of a model proxy on the predicted label when the rationale is included into the input vs. when it is not, 1 [ ˆ y ∣ x, ˆ r ] − 1 [ ˆ y ∣ x ] , averaged over examples grouped based on whether they leak labels or not.
Differently, RQ uses gold labels.
The quality of a rationale ˆ r is measured as 1 [ y ∗ ∣ x, ˆ r ] − 1 [ y ∗ ∣ x ] , where y ∗ is the gold label.
Similarly,
RQ is the average score over all test examples.
Experiments
We compare R EV with existing metrics (§4.1) and human judgments (§4.2) on the ECQA dataset, as well as show R EV on other CQA and NLI benchmarks.
4.1 Comparison Between Evaluation Metrics
We compare R EV to LAS and RQ, in evaluating different rationale-label pairs on the ECQA dataset.
In addition to Y ∗ ;R ∗ , XY ∗ → R, X → YR and X → RY, we also explore the evaluation on vacuous baseline rationales (Y ∗ ;B), which simply combine inputs and labels with no additional information.
Note that the scores obtained from different metrics are not directly comparable due to different comparison scales and criteria (e.g., log-probability vs. accuracy).
We mainly focus on the ranking over different types of rationale-label pairs.
The results averaged over 4 random seeds are shown in the left part of Fig.
Qualitative results are provided in Table 5 in Appendix D.2.
All three metrics agree that the crowdsourced rationales (Y ∗ ;R ∗ ) in the ECQA have the highest quality.
While by definition, R EV for vacuous rationales is low, both LAS and RQ scores for these rationales are quite high, showing that these metrics are incapable of measuring the amount of additional information in rationales.
Intuitively, we expect weaker rationale-label consistency in
X → RY setting compared to X → YR, as the labels are forcefully selected among the candidates as opposed to being freely generated by the task model (§3.2).
While R EV is able to capture this intuition and rank X → YR higher than X → RY, LAS and RQ have a different ranking.
[Table]
Next, we apply R EV to evaluate crowdsourced and model generated rationale-label pairs (Y ∗ ;R ∗ , XY ∗ → R, X → YR, X → RY) across different datasets.
For each dataset, the evaluator is trained on the training set with gold labels and crowdsourced rationales.
The results are shown in Table 1.
We observe that the gold rationales in the ECQA dataset achieve higher
R EV score than those in CoS-E. This observation is in line with the known quality issues of crowdsourced rationales in CoS-E [Aggarwal et al., 2021, Sun et al., 2022].
Moreover, training the evaluator with CoS-E results in lower R EV for all models, compared to training with ECQA.
Interestingly, model-generated rationales (XY ∗ → R) have higher R EV scores than crowdsourced rationales for CoS-E, and similar R EV scores for ECQA,
QuaRTz and e-SNLI.
For QuaRTz, the model generated rationales seem to not contain much new and relevant information over a vacuous baseline.
In the case of e-SNLI, the problem is even severer as most of the crowdsourced or generated rationales do not provide reasoning but rather follow a label-specific template e.g., A implies (that) B [Kumar and Talukdar, 2020a, Brahman et al., 2021].
4.2 Human Evaluation
To understand how R EV correlates with human judgments of rationales, we conduct a crowdsourcing experiment via Amazon Mechanical Turk.
We randomly sample 230 examples from the ECQA test set and ask workers to evaluate the four types of rationale-label pairs (Y ∗ ;R ∗ , XY ∗ → R, X → YR,
X → RY) for each example.
We present workers with a question (input text), an answer (label) and an explanation (rationale), and ask them whether the explanation justifies the answer ( yes/no ).
If they answer yes , we further ask them to evaluate the amount of additional information supplied by the explanation that explains why the answer might have been chosen for the question.
The workers choose from none / little / some / enough , corresponding to a 4-point Likert-scale.
We collect 3 annotations per instance and use majority vote to decide whether the rationale can justify the label.
If yes , we take the average over the 3 human-annotated scores as the amount of information.
Otherwise, we give a score of -1.
More details of human evaluation are in Appendix D.3.
The results are shown in the right part of Fig.
2, where the ranking of the four types of rationalelabel pairs is Y ∗ ;R ∗ > XY ∗ → R > X
→ YR > X → RY.
While LAS and RQ rank X → RY better than
X → YR (see the left part of Fig. 2), the ranking from R EV is more consistent with human judgments, suggesting its effectiveness in evaluating rationale quality.
Related Work
Self-rationalized models serve interpretability by providing rationales for their predictions.
The rationales broadly fall into two categories: extractive rationales and free-text rationales.
Evaluations on extractive rationales have been well studied, generally from two perspectives — faithfulness and plausibility [DeYoung et al., 2020, Pruthi et al., 2022, Chan et al., 2022b].
Faithfulness measures to which extent rationales reflect the true reasoning process of models, while plausibility evaluates how convincing rationales are to humans [Jacovi and Goldberg, 2020].
However, the faithfulness evaluation metrics (e.g., sufficiency, comprehensiveness) proposed for extractive rationales are not applicable to free-text rationales because they are not a part of inputs.
Existing automatic metrics for free-text rationales focus on rationale-label association, and measure the utility of a rationale based on how much it helps a model proxy predict the given label [Hase et al., 2020] or the gold label [Wiegreffe et al., 2021] given the input.
Chan et al.
[2022a] further propose a framework to evaluate the automatic metrics.
However, none of them consider measuring the amount of additional information in free-text rationales.
Sun et al.
[2022] conduct a human study on the additional knowledge provided by free-text rationales.
This work is the first that proposes an automatic metric to quantify the additional information in free-text rationales.
Conclusion
In this paper, we propose an information-theoretic metric, R EV , to evaluate free-text rationale.
R EV measures if a rationale contains new information that is relevant for the label of interest, beyond the information contained in the input.
We show the advantage of R EV in evaluating different types of rationale-label pairs compared to existing metrics.
We demonstrate that the evaluation of free-text rationales with R EV is consistent with human judgments.
Future work might explore evaluation that penalizes rationales which support incorrect predictions, thus bridging together predictive performance with interpretability metrics.
A Properties of Conditional V -information
As proved by Hewitt et al.
[2021], CVI has several useful properties:
Non-Negativity : I V ( R → Y ∣ B ) ≥ 0 . 2.
Independence : If Y and B are jointly independent of R , then I V ( R → Y ∣ B ) = 0 .
Monotonicity :
If U ⊆ V , then H V ( Y ∣ B ) ≤ H U
( Y ∣ B ) .
An implication from Monotonicity is complex models (e.g., pre-trained language models) can do better than simpler ones (e.g., linear models) in estimating V -usable information.
Since CVI measures the additional V -usable information in R about Y beyond what’s already extracted from B by models in V , it grounds the goal of the proposed metric R EV .
B Computing R EV for Rationale Evaluation
Algorithm 1 shows the process of computing both pointwise and aggregate R EV scores.
C Supplement of Experimental Setup C.1 Constructing a Baseline with Vacuous Rationales
[Table]
C.2 Datasets
[Table]
For CQA task, we use ECQA [Aggarwal et al., 2021],
CoS-E (v1.11) [-
-]6 [Rajani et al., 2019] and QuaRTz [Tafjord et al., 2019].
Both ECQA and CoS-E originate from the CommonsenseQA dataset [Talmor et al., 2018], where each commonsense question is paired with 5 candidate choices and the task is to select an answer from the candidates.
ECQA contains higher quality free-text rationales compared to CoS-E, in terms of comprehensiveness, coherence, non-redundancy, etc.
[Aggarwal et al., 2021,
Sun et al., 2022].
QuaRTz is an open-domain reasoning task about textual qualitative relationships.
Each instance contains a situated qualitative question, two answer options and a knowledge statement.
The task is to select an answer from the two options to the question based on the textual qualitative knowledge.
We use the knowledge statement as a free-text rationale since it explains why the answer is to the question.
For NLI task, we use e-SNLI [Camburu et al., 2018] which is an extension of SNLI [Bowman et al., 2015] with augmented free-text human-written rationales.
The task is to predict the entailment relationship between a premise and a hypothesis.
Table 3 shows the summary statistics of the four datasets.
[Table]
C.3 Models
We use Huggingface Transformers
[Wolf et al., 2020] to access all task models and evaluators.
We train each model for up to 30 epochs with a learning rate 5 e − 6 and a batch size 8 .
All experiments were performed on a single NVIDIA RTX 8000 GPU.
Table 4 shows input-output formattings of different task models for different tasks
[Table]
D Supplement of Experiments D.1 Comparison Between Evaluator Architectures
We apply R EV to evaluate different types of free-text rationales w.r.t.
labels on the ECQA dataset.
Figure 3 shows R EV scores of the four types of rationale-label pairs evaluated by the four evaluators.
The ranking of the four groups of rationale-label pairs is consistent across the four evaluators, i [Table] e.
Y ∗ ;R ∗ > XY ∗ → R > X
→ YR > X → RY.
This ranking is also consistent with human evaluation in §4.2.
Since ECQA contains high-quality crowdsourced rationales [Aggarwal et al., 2021], it is expected that the R EV of gold rationale-label pairs (Y ∗ ;R ∗ ) is the highest.
The R EV of XY ∗ → R is close to that of Y ∗ ;R ∗ , indicating the task model (T5 Large) can produce good quality rationales when it is prompted with ground-truth labels.
All four evaluators agree that the generated rationales of X → YR contain more additional background information for explaining the predicted labels than those of X → RY.
This is consistent with our design of the X → RY in §3.3, where the generated rationales and labels have weakened relevance.
For each type of rationale-label pairs, the four evaluators capture different amount of conditional V -information, while T5 Large consistently outperforms other three models.
In the reported experiments §4, we use T5 Large as the evaluator.
D.2 Qualitative Analysis of Different Metrics
Table 5 shows the qualitative analysis of different metrics on the four types of rationale-label pairs (Y ∗ ;R ∗ , XY ∗ → R, X → YR, X → RY) on the ECQA dataset.
R EV provides more accurate evaluations on those examples than LAS and RQ.
D.3 Human Evaluation Details
We randomly select 230 examples from the ECQA test set and conduct human evaluation on the four types of rationale-label pairs (Y ∗ ;R ∗ , XY ∗ → R, X → YR, X → RY) w.r.t.
each example through the Amazon Mechanical Turk (AMT).
Each instance is assessed by 3 workers.
We pay the workers $ 0 . 08 for assessing each instance.
Figure 4 shows the instructions we provide to workers.
In Figure 5, we show three examples, illustrating when the explanation (rationale) does not justify the answer (label), when the explanation supports the answer while not supplying additional information, and when the explanation supports the answer and provides additional information.
Figure 6 shows the interface of the actual hit for human evaluation.
For each instance, we provide a question (input), an answer (label), and an explanation (rationale), and ask the workers to answer the following two questions:
Does the Explanation justify the given Answer?
(yes or no)
The question is to ask workers to judge whether the rationale supports the label or not.
If yes, how much additional information does the Explanation have to justify the Answer beyond just reiterating what is stated in Question and Answer?
(No additional info, Little additional info, Some additional info, Enough additional info)
We only ask this question if the workers choose “yes” for the first question.
We design this question to ask workers to evaluate the extent to which the rationale provides additional information for justifying the label beyond repeating it w.r.t.
the input.
[Table]