Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems (2024)

Abstract: Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems (TDSs). Obtaining high-quality and consistent ground-truth labels from annotators presents challenges. When evaluating a TDS, annotators must fully comprehend the dialogue before providing judgments. Previous studies suggest using only a portion of the dialogue context in the annotation process. However, the impact of this limitation on label quality remains unexplored. This study investigates the influence of dialogue context on annotation quality, considering the truncated context for relevance and usefulness labeling. We further propose to use large language models ( LLMs) to summarize the dialogue context to provide a rich and short description of the dialogue context and study the impact of doing so on the annotator’s performance. Reducing context leads to more positive ratings. Conversely, providing the entire dialogue context yields higher-quality relevance ratings but introduces ambiguity in usefulness ratings. Using the first user utterance as context leads to consistent ratings, akin to those obtained using the entire dialogue, with significantly reduced annotation effort. Our findings show how task design, particularly the availability of dialogue context, affects the quality and consistency of crowdsourced evaluation labels.

## introduction with recent advances in pre-trained language models and large language models (llms), task-oriented dialogue systems (tdss) have redefined how people seek information, presenting a more natural approach for users to engage with information sources @xcite . as tdss become increasingly integral to information-seeking processes, the question of how to accurately and effectively evaluate their performance becomes critical. due to the poor correlation of automatic metrics with human-generated labels @xcite , evaluation of tdss has shifted towards relying on user ratings or crowdsourced labels as ground-truth measures @xcite . various crowdsourcing techniques have been employed to collect ground-truth labels, such as sequential labeling @xcite , where the annotators go through each utterance and annotate them one by one. this approach introduces certain risks in the annotation process, such as annotators' fatigue and high cognitive load in extra-long dialogues, requiring them to remember and track the state of the dialogue as they annotate the utterances @xcite . while following and understanding the dialogue context is crucial and can influence the annotators' ratings, reading and understanding very long dialogues can lead to degraded performance. to address this issue, another line of research proposes to randomly sample only a few utterances in each dialogue to be annotated @xcite @xcite . while addressing the high cognitive load and fatigue, limiting annotators' understanding of the dialogue poses obvious risks, such as unreliable and biased labels @xcite . in particular, the amount of dialogue context can lead to biases. for example, annotators who lack rich context may unintentionally lean towards positive or negative ratings, neglecting the broader quality of the response. thus, offering annotators too little context risks misleading judgments, potentially leading to inaccurate or inconsistent labels. conversely, flooding annotators with excessive information can overwhelm them, which can lead to lower returns in terms of label quality. prior work has investigated factors that affect the quality and consistency of crowdsourced evaluation labels, including annotator characteristics, task design, cognitive load, and evaluation protocols (see, e.g., @xcite @xcite . however, no previous work studies the effect of random sampling and the number of sampled utterances on the annotation quality. in this study, we aim to address this research gap by investigating how different amounts of contextual information impact the quality and consistency of crowdsourced labels for tdss, contributing to understanding of the impact of such design choices. we experiment with crowdsourcing labels for two major evaluation aspects, namely, relevance and usefulness under different conditions, where we compare the annotation quality under different dialogue context truncation strategies. addressing the challenge of insufficient context at the turn level, we propose to use heuristic methods and llms to generate the user's information need and dialogue summary. llms can play the role of annotation assistants @xcite by summarizing the dialogue history, facilitating a more efficient and effective understanding of the dialogue context before annotating an utterance. to this aim, we use gpt-4 for dialogue context summarization and compare the performance of annotators' under different conditions, as well as different context sizes. through these experiments, we answer two main questions: (rq1) how does varying the amount of dialogue context affect the crowdsourced evaluation of tdss? (rq2) can the consistency of crowdsourced labels be improved with automatically generated supplementary context? our findings reveal that the availability of previous dialogue context significantly influences annotators' ratings, with a noticeable impact on their quality. without prior context, annotators tend to assign more positive ratings to system responses, possibly due to insufficient evidence for penalization, introducing a positivity bias. in contrast, presenting the entire dialogue context yields higher relevance ratings. as for usefulness, presenting the entire dialogue context introduces ambiguity and slightly lowers annotator agreement. this highlights the delicate balance in contextual information provided for evaluations. the inclusion of automatically generated dialogue context enhances annotator agreement in the no-context (c 0 ) condition while reducing annotation time compared to the full-context (c 7 ) condition, presenting an ideal balance between annotator effort and performance. our findings extend to other task-oriented conversational tasks like conversational search and preference elicitation, both relying on crowdsourced experiments to assess system performance. ## methodology we examine how contextual information about a dialogue affects the consistency of crowdsourced judgments regarding relevance and usefulness of a dialogue response. here, contextual information refers to the information or conversation that precedes a specific response. we carry out experiments in two phases. phase 1 involves varying the amount of dialogue context for annotators to answer rq1. in phase 2, we vary the type of previous contextual information available to annotators to address rq2. ## results and analysis we address (rq1) and (rq2) by providing an overview of the results and in-depth analysis of our crowdsourcing experiments. we first describe the key data statistics. ## discussion and implications our findings reveal intriguing insights into the impact of context size and type on crowdsourced relevance and usefulness labels for tds. expanding the dialogue context from c 0 to c 7 significantly improves agreement among annotators, indicating that annotators rely on comprehensive context to make more accurate assessments. this trend does not hold for usefulness, where we notice a decrease in agreement when all previous dialogue context is available. the optimal amount of context required for reliable labels relies on the aspect evaluated. consistent with prior work @xcite , we observe an inconsistency in relevance labels across variations, with the same system response being rated differently depending on the context provided. given the lack of label consistency across variations, future studies should carefully tailor their annotation task design and test various settings to ensure high-quality and consistent labels. additionally, much care should be taken when comparing the performance of a system across several datasets when labels are crowdsourced with a different strategy to ensure a fair comparison as models similar to humans can be sensitive to the annotation strategy @xcite . we also analyzed data from the open-ended question asking annotators about their experience with the annotation task. annotators note that dialogue summaries fail to convey a user's emotion, limiting their annotation process. additionally, lower accuracy of the context generated by an llm may lead to low agreement among annotators. this signifies the importance of carefully considering the quality and accuracy of generated content in the evaluation process. we provide examples in section a.5 in the appendix. while there may be constraints in presenting user information need and dialogue summary as dialogue context, one key consideration to take into account is the cognitive load of annota-tors. providing a shorter, focused context reduces the cognitive burden on annotators, allowing them to devote more attention to actually evaluating a response. this not only streamlines the annotation process but also helps maintain high-quality results. reducing the amount of content to be assessed may lead to faster annotation times without compromising the quality of ratings @xcite . another approach to using llms in annotation, is for researchers to consider co-annotation @xcite between humans and llms. optimal context varies by the aspect under evaluation, challenging the idea of a universal strategy. the consistent reliability of automatic methods suggests their potential as dependable tools for evaluation. this implies their use in generating supplementary context, eliminating the need for manual determination of context amounts. this streamlines evaluation, enhancing efficiency in contextdriven evaluations for tds. for data lacking topic or preference shifts, heuristics perform effectively. however, llms are recommended for shifting conditions, showcasing adaptability not easily discernible with heuristics. while our primary focus was limited to relevance and usefulness, the proposed experimental design can be extended to other aspects of tdss evaluation. moreover, our findings may be taskor dataset-specific, prompting the need for further investigation into their generalizability. as to future work, we aspire to enhance the robustness of our findings by conducting studies on larger-scale datasets. in addition following previous work by @xcite , we would also want to understand the effect of annotator background: experience of interacting with conversational system or prior experience in doing the annotation task on label consistency for tdss. ## related work we review related work not covered in the paper so far. several user-centric dialogue evaluation metrics @xcite @xcite have been proposed. for tdss, high-level dimensions such as user satisfaction @xcite and fine-grained metrics such as relevance and interestingness @xcite have gained interest. due to the ineffectiveness of standard evaluation metrics such as rouge @xcite , bleu @xcite , which show poor correlation with human judgments @xcite , a significant amount of research on these metrics relies on crowdsourcing dialogue evaluation labels to improve correlation with actual user ratings. crowdsourcing ground-truth labels has gained momentum in information retrieval (ir) for tasks like search relevance evaluation @xcite and measuring user satisfaction in tds. a major challenge is ensuring quality and consistency of crowdsourced labels. task design and annotators' behavioral features and demographics can affect the quality of the collected labels @xcite @xcite . @xcite examine how effort and incentive influence the quality of labels provided by assessors when making relevance judgments. other factors such as judgment scale @xcite , annotator background @xcite , and annotators' demographics @xcite have also been studied. most studies focus on search systems, not dialogue systems. closer to our work, @xcite study the effect of cognitive bias in the evaluation of dialogue systems. providing an anchor to annotators introduces anchoring bias, where annotators' ratings are close to the anchor's numerical value. like @xcite , we focus on the effect of task design on the evaluation of tdss. in particular, we investigate how the amount and type of dialogue context provided to annotators affect the quality and consistency of evaluation labels and the annotator experience during the evaluation task. ## conclusion in this work, we investigated the impact of varying the dialogue context size and type on crowdsourced evaluation labels. in particular we crowdsourced evaluation labels for two aspects: relevance and usefulness. our findings reveal that optimal context is dependent on the aspect under evaluation. for relevance annotators tend to agree more on a label when they have access to the whole dialogue context. however this does not hold for the usefulness aspect where we witness high annotator agreement when partial context is available. we show that a simple approach like providing an automatically generated user need through heuristics without revealing the entire dialogue can consistently increase annotator agreement across the two aspects. this implies that we can rely on auto-matic methods such as the use of llms to improve the productivity of the crowdworkers by reducing the amount of dialogue they have to read before evaluating the current response. this study contributes towards how llms can be integrated in the annotation process to ensure quality labels from the crowdworkers. in this work we used gpt-4 api which is not open source. for future work we will explore the use of open-source llms, like llama-chat @xcite , to facilitate a more transparent and reproducible experimental framework.