Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design (2023)

Abstract: Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and the annotation frameworks. Here, we propose to analyze another source of bias—task design bias, which has a particularly strong impact on crowdsourced linguistic annotations where natural language is used to elicit the interpretation of lay annotators. For this purpose we look at implicit discourse relation annotation, a task that has repeatedly been shown to be difficult due to the relations’ ambiguity. We compare the annotations of 1,200 discourse relations obtained using two distinct annotation tasks and quantify the biases of both methods across four different domains. Both methods are natural language annotation tasks designed for crowdsourcing. We show that the task design can push annotators towards certain relations and that some discourse relation senses can be better elicited with one or the other annotation approach. We also conclude that this type of bias should be taken into account when training and testing models.

## introduction crowdsourcing has become a popular method for data collection. it not only allows researchers to collect large amounts of annotated data in a shorter amount of time, but also captures human inference in natural language, which should be the goal of benchmark nlp tasks @xcite . in order to obtain reliable annotations, the crowdsourced labels are traditionally aggregated to a single label per item, using simple majority voting or annotation models that reduce noise from the data based on the disagreement among the annotators @xcite . however, there is increasing consensus that disagreement in annotation cannot be generally discarded as noise in a range of nlp tasks, such as natural language inferences @xcite @xcite , word sense disambiguation @xcite , question answering @xcite , anaphora resolution @xcite , sentiment analysis @xcite , and stance classification @xcite . label distributions are proposed to replace categorical labels in order to represent the label ambiguity @xcite @xcite . there are various reasons behind the ambiguity of linguistic annotations @xcite . @xcite summarize the sources of ambiguity into three categories: the text, the annotators, and the annotation scheme. in downstream nlp tasks, it would be helpful if models could detect possible alternative interpretations of ambiguous texts, or predict a distribution of interpretations by a population. in addition to the existing work on the disagreement due to annotators' bias, the effect of annotation frameworks has also been studied, such as the discussion on whether entailment should include pragmatic inferences @xcite , the effect of the granularity of the collected labels @xcite , or the system of labels that categorize the linguistic phenomenon @xcite . in this work, we examine the effect of task design bias, which is independent of the annotation framework, on the quality of crowdsourced annotations. specifically, s1 and s2 ) and the dc and qa annotation in the middle. implicit dr annotation is arguably the hardest task in discourse parsing. discourse coherence is a feature of the mental representation that readers form of a text, rather than of the linguistic material itself @xcite . discourse annotation thus relies on annotators' interpretation of a text. further, relations can often be interpreted in various ways @xcite , with multiple valid readings holding at the same time. these factors make discourse relation annotation, especially for implicit relations, a particularly difficult task. we collect 10 different annotations per dr, thereby focusing on distributional representations, which are more informative than categorical labels. since dr annotation labels are often abstract terms that are not easily understood by lay individuals, we focus on ''natural language'' task designs. decomposing and simplifying an annotation task, where the dr labels can be obtained indirectly from the natural language annotations, has been shown to work well for crowdsourcing @xcite @xcite . crowdsourcing with natural language has become increasingly popular. this includes tasks such as nli @xcite , srl @xcite , and qa @xcite . this trend is further visible in modeling approaches that cast traditional structured prediction tasks into nl tasks, such as for co-reference @xcite , discourse comprehension @xcite , or bridging anaphora @xcite . it is therefore of interest to the broader research community to see how task design biases can arise, even when the tasks are more accessible to the lay public. we examine two distinct natural language crowdsourcing discourse relation annotation tasks (figure 1 ) @xcite derive relation labels from discourse connectives (dcs) that crowd workers insert; @xcite derive labels from question answer (qa) pairs that crowd workers write. both task designs employ natural language annotations instead of labels from a taxonomy. the two task designs, dc and qa, are used to annotate 1,200 implicit discourse relations in 4 different domains. this allows us to explore how the task design impacts the obtained annotations, as well as the biases that are inherent to each method. to do so we showcase the difference of various inter-annotator agreement metrics on annotations with distributional and aggregated labels. we find that both methods have strengths and weaknesses in identifying certain types of relations. we further see that these biases are also affected by the domain. in a series of discourse relation classification experiments, we demonstrate the benefits of collecting annotations with mixed methodologies, we show that training with a soft loss with distributions as targets improves model performance, and we find that cross-task generalization is harder than crossdomain generalization. the outline of the paper is as follows. we introduce the notion of task design bias and analyze its effect on crowdsourcing implicit drs, using two different task designs (section 3-4). next, we quantify strengths and weaknesses of each method using the obtained annotations, and suggest ways to reduce task bias (section 5). then we look at genre-specific task bias (section 6). lastly, we demonstrate the task bias effect on dr classification performance (section 7). ## results we first compare the iaa between the two crowdsourced annotations, then we discuss iaa between dc/qa and the reference annotations, and lastly we perform an analysis based on annotation uncertainty. here, ''sub-labels'' of an instance means all relations that have received more than one annotation; and ''label distribution'' is the distribution of the votes of the sub-labels. ## sources of the method bias in this section, we analyze method bias in terms of the sense labels collected by each method. we also examine the potential limitations of the methods which could have contributed to the bias and demonstrate how we can utilize information on method bias to crowdsource more reliable labels. lastly, we provide a cross-domain analysis. ## analysis by genre for each of the four genres (novel, wikipedia, europarl, and wsj) we have ∼300 implicit drs annotated by both dc and qa. scholman et al. (2022a) showed, based on the dc method, that in discogem, conjunction is prevalent in the wikipedia domain, precedence in literature and result in europarl. the qa annotations replicate this finding, as displayed in figure 4 . it appears more difficult to obtain agreement with the majority labels in europarl than in other genres, which is reflected in the average entropy (see in the latter domain, the dc method results in more causal relations: 36% of the conjunc-tions labelled by qa are labelled as result in dc. 11 manual inspection of these dc annotations reveals that workers chose considering this frequently only in the europarl subset. this connective phrase is typically used to mark a pragmatic result relation, where the result reading comes from the belief of the speaker (ex. (4)). this type of relation is expected to be more frequent in speech and argumentative contexts and is labelled as result-belief in pdtb3. qa does not have a question prefix available that could capture result-belief senses. the result labels obtained by dc are therefore a better fit with the pdtb3 framework than qa's conjunctions. concession is generally more prevalent with the dc method, especially in europarl, with 9% compared to 3% for qa. contrast, on the other hand, seems to be favored by the qa method, of which most (6%) figure 4 : level-2 sublabel counts of all the annotated labels of both methods, split by domain. contrast relations are found in wikipedia, compared to 3% for dc. figure 4 also highlights that for the qa approach, annotators tend to choose a wider variety of senses which are rarely ever annotated by dc, such as purpose, condition, and manner. we conclude that encyclopedic and literary texts are the most suitable to be annotated using either dc or qa, as they show higher inter-method agreement (and for wikipedia also higher agreement with gold). spoken-language and argumentative domains on the other hand are trickier to annotate as they contain more pragmatic readings of the relations. ## case studies: effect of task design on dr classification models analysis of the crowdsourced annotations reveals that the two methods have different biases and different correlations with domains and the style (and possibly function) of the language used in the domains. we now investigate the effect of task design bias on automatic prediction of implicit discourse relations. specifically, we carry out two case studies to demonstrate the effect that task design and the resulting label distributions have on discourse parsing models. task and setup we formulate the task of predicting implicit discourse relations as follows. the input to the model are two sequences s 1 and s 2 , which represent the arguments of a discourse relation. the targets are pdtb 3.0 sense types (including level-3). this model architecture is similar to the model for implicit dr prediction by @xcite . we experiment with two different losses and targets: a cross-entropy loss where the target is a single majority label and a soft cross-entropy loss where the target is a probability distribution over the annotated labels. using the 10 annotations per instance, we obtain label distributions for each relation, which we use as soft targets. training with a soft loss has been shown to improve generalization in vision and nlp tasks @xcite . as suggested in @xcite , we normalize the sense-distribution over the 30 possible labels foot_9 with a softmax. assume one has a relation with the following annotations: 4 result, 3 conjunction, 2 succession, 1 arg1-as-detail. for the hard loss, the target would be the majority label: result. for the soft loss we normalize the counts (every label with no annotation has a count of 0) using a softmax, for a smoother distribution without zeros. we fine-tune deberta (deberta-base) @xcite in a sequence classification setup using the huggingface checkpoint @xcite . the model trains for 30 epochs with early stopping and a batch size of 8. ## discussion and conclusion dr annotation is a notoriously difficult task with low iaa. annotations are not only subject to the interpretation of the coder (spooren and degand, the results show that both methods have unique biases, but also that both methods are valid, as similar sets of labels are produced. further, the methods seem to be complementary: both methods show higher agreement with the reference label than with each other. this indicates that the methods capture different sense types. the results further show that the textual domain can push each method towards different label distributions. lastly, we simulated how aggregating annotations based on method bias improves agreement. we suggest several modifications to both methods for future work. for qa, we recommend to replace question prefix options which start with a connective, such as ''after what''. the revised options should ideally start with a wh-question word, for example, ''what happens after..''. this would make the questions sound more natural and help to prevent confusion with respect to level-3 sense distinctions. for dc, an improved interface that allows workers to highlight argument spans could serve as a screen that confirms the relation is between the two consecutive sentences. syntactic constraints making it difficult to insert certain rare connectives could also be mitigated if the workers are allowed to make minor edits to the texts. considering that both methods show benefits and possible downsides, it could be interesting to combine them for future crowdsourcing efforts. given that obtaining dc annotations is cheaper and quicker, it could make sense to collect dc annotations on a larger scale and then use the qa method for a specific subset that shows high label entropy. another option would be to merge both methods, by first letting the crowdworkers insert a connective and then use qas for the second connective-disambiguation step. lastly, since we showed that often more than one relation sense can hold, it would make sense to allow annotators to write multiple qa pairs or insert multiple possible connectives for a given relation. the dr classification experiments revealed that generalization across data from different task designs is hard, in the dc and qa case even harder than cross-domain generalization. additionally, we found that merging data distributions coming from different task designs can help boost performance on data coming from a third source (traditional annotations). lastly, we confirmed that soft modeling approaches using label distributions can improve discourse classification performance. task design bias has been identified as one source of annotation bias and acknowledged as an artifact of the dataset in other linguistic tasks as well @xcite . our findings show that the effect of this type of bias can be reduced by training with data collected by multiple methods. this could be the same for other nlp tasks, especially those cast in natural language, and comparing their task designs could be an interesting future research direction. we therefore encourage researchers to be more conscious about the biases crowdsourcing task design introduces.