Explaining Why: How Instructions and User Interfaces Impact Annotator Rationales When Labeling Text Data (2022)

Abstract: In the context of data labeling, NLP researchers are increasingly interested in having humans select rationales, a subset of input tokens relevant to the chosen label. We conducted a 332-participant online user study to understand how humans select rationales, especially how different instructions and user interface affordances impact the rationales chosen. Participants labeled ten movie reviews as positive or negative, selecting words and phrases supporting their label as rationales. We varied the instructions given, the rationale-selection task, and the user interface. Participants often selected about 12% of input tokens as rationales, but selected fewer if unable to drag over multiple tokens at once. Whereas participants were near unanimous in their data labels, they were far less consistent in their rationales. The user interface affordances and task greatly impacted the types of rationales chosen. We also observed large variance across participants.

## introduction there is a growing effort in nlp to collect humanprovided justifications in addition to labels during annotation @xcite . these justifications can take many forms, ranging from subsets of input tokens (henceforth rationales) to natural language. they can provide additional information on how the label is derived from the input and are hypothesized to improve the robustness of models by correcting spurious correlations and enhancing the transparency of models for human-ai collaboration. @xcite points out that diverse instructions can be used to collect even simple forms @xcite : "to justify why a review is positive @xcite , highlight the most important words and phrases that would tell someone @xcite to see the movie." @xcite : "select all words and phrases in the review that would suggest the review is positive @xcite ." it remains an open question how these divergent instructions and other human-centered design decisions affect the collection of rationales. in this work, we conduct the first study of how to collect rationales by investigating how these factors affect the process and the rationales ultimately collected. we use sentiment analysis as a testbed. rather than building a large dataset, we study how different design choices impact the collection of rationales. we thus ask all participants to label the sentiment of the same ten imdb movie reviews and provide rationales for their selections, varying the instructions and interface in a between-subjects experiment. our experiment had three phases of data collection, with 332 participants total across phases. first, we assigned participants to one of six sets of instructions explaining what rationales are. second, we tested a variation in which participants annotated rationales not just in support of the bi-nary classification label they chose, but in support of the opposite label, which we term two-sided rationales. finally, we varied both a key interface affordance-the ability to drag to select multiple words at once-and whether participants were asked to select "words and phrases" or just "words." superficially, one might expect the dragging affordance not to affect rationales since annotators can always click all relevant words individually. these three phases of our between-subjects experiment let us answer four key research questions: • rq1: how do instructions and interfaces impact the fraction of words selected as rationales? • rq2: how consistently do different annotators select rationales? do instructions and interfaces impact inter-annotator consistency? • rq3: how do instructions and interfaces shape which types of words are selected as rationales? • rq4: how long does selecting rationales take? when instructed to select "words and phrases" as rationales and given an interface where they could click individual words or drag over multiple words, the median participant selected about 12% of words as rationales. this fraction did not vary significantly based on instructions. however, we observed high variance across participants; some selected only a few words, while others selected one-third of them. without the dragging affordance, participants instead selected only about 5% of words. participants (with dragging) selected 4% of words as rationales for the label not chosen. participants were near unanimous in the labels chosen (krippendorff's α close to 1.0). in contrast, rationales were fairly inconsistent (α around 0.3) even when keeping the instructions and interface constant. as such, rationales collected from only a few annotators can be highly variable, impacting their downstream usage. the instructions and interface minimally impacted inter-annotator reliability. both the availability of the dragging affordance and the task (selecting "words and phrases" or just "words") greatly impacted which words were selected as rationales. in part, these factors impacted how much context participants included in rationales. given the dragging affordance, participants tended to select full phrases like "worst movie i have seen" as rationales. without this affordance, they tended to select only "worst movie." asked to select "words" (vs. "words and phrases"), they tended to select single adjectives, such as "worst." in contrast, instructions had only a small impact. a few words in each review were labeled as rationales by nearly all participants regardless of their assigned condition. however, participants varied greatly from each other in whether they labeled parts of reviews expressing ambiguous and indirect thoughts as rationales, as well as how they handled negation and complex sentences. finally, we found that providing one-sided rationales took about 2.5× as long as providing only labels. thus, researchers may consider rationales worthwhile to collect if they provide greater benefit than having 2.5× as much data with only labels. we conclude by discussing lessons for collecting rationales. ## related work our work is primarily related to two strands of prior work: the construction of explanation datasets and understanding the quality of annotated rationales. explanation datasets: driven by growing interest in interpretable ml @xcite , many researchers @xcite @xcite @xcite @xcite @xcite have collected datasets of human explanations. see @xcite for a survey. despite this interest, to our knowledge we are the first researchers to focus on the hci design space of rationale collection. in particular, we focus on a particular type of explanation: highlighting parts of the input that suggest the chosen label without further explanations of the mechanism. this approach is sometimes also termed feature feedback. aiming to improve the user experience of data labeling, @xcite used attention-based models to automatically highlight potentially important words. they also enabled crowdworkers to highlight missing important words (rationales) to improve their attention model, but did not study the design of rationale-selection interfaces. quality of existing rationales: further motivating the study of the rationale-annotation process, carton et al. (2020) found substantial variance in the quality of existing rationale datasets. they measured the sufficiency and comprehensiveness of human rationales based on machine learning models. many studies have also found that learning from existing rationale datasets can hurt model performance, or that any improvement is marginal or limited to secondary goals like explanation quality @xcite @xcite . recent work in the hci community has shown that unclear or imprecise requester instructions can lead to differences between (label) annotations @xcite , partially inspiring our examination of the instructions provided for annotating rationales. by investigating the annotation process, our work can shed light on how the shortcomings of rationale datasets may arise. ## method in this section, we present the design of our experiment and its detailed procedures. ## results we present our key results organized by the four research questions described in section 1. ## concluding discussion we conducted the first study of how to collect rationales. our results have implications for ui design for this task, as well as for how rationales are used. affordances: interface affordances mattered more than instruction wordings, suggesting that hci factors overlooked in prior work critically impact the characteristics of rationales collected from humans. it is critical that future work carefully consider, and also carefully report, these design factors to allow for replicable studies and effective comparisons. we also encourage researchers to open-source interfaces used for data annotation. low inter-annotator agreement: the low interannotator agreement we observed draws into question approaches in past work @xcite of using rationale annotation as ground truth to evaluate machine-generated rationales. it also suggests the need for more nuanced ways to combine rationale annotations from humans, rather than simply treating rationales as another binary classification. to this end, it may be important to embrace these exhibited ambiguities and develop novel algorithms that can learn from the distributions of rationale annotations. the role of phrases highlights the importance of moving beyond bag-of-words approaches in incorporating rationales. in contrast, if researchers want to emphasize collecting consistent (if semantically limited) rationales from annotators, it may be best not to provide a drag affordance. two-sided rationales: while prior work focuses on one-sided rationales, our results show that a non-trivial fraction of words imply the opposite sentiment. this finding is especially important for sentiment analysis, and it echoes recent studies showing that negative reviews nonetheless contain positive words @xcite . however, two-sided rationale collection took our participants far more time than one-sided, suggesting that future work investigate ux strategies to reduce this time. limitations: our work focused on a single task, sentiment analysis, and was based on ten curated reviews. furthermore, we explored only one of many possible workflows (e.g., selecting rationales at the same time as selecting the sentiment label). experiments with diverse datasets and tasks, as well as alternative workflows, can verify the robustness of our findings. however, we believe our work represents a valuable contribution toward understanding how hci factors impact rationale annotation. risks: annotation seeking to maximize rationale consistency may exclude diverse opinions. availability: we have made data and code available in both the online data supplement and on github @xcite . our data supplement contains heatmaps analogous to figure 4 for all ten reviews, as well as the full data (rationales, detailed timing/click information, survey responses) for the 92% of participants who granted us optional permission to release their data. our opensource code consists of our rationale-annotation ui, which was written in the react framework. ethics: our study was determined to be exempt by uchicago's irb. we built on our institution's model consent form for online studies. participants checked boxes to affirm consent. we did not collect any pii beyond prolific ids. our compensation (@xmath018 usd hourly wage.