A Study on Annotation Interfaces for Summary Comparison (2023)

Abstract: The task of summarisation is notoriously difficult to evaluate, with agreement even between expert raters unlikely to be perfect. One technique for summary evaluation relies on collecting comparison data by presenting annotators with generated summaries and tasking them with selecting the best one. This paradigm is currently being exploited in reinforcement learning using human feedback, whereby a reward function is trained using pairwise choice data. Comparisons are an easier way to elicit human feedback for summarisation, however, such decisions can be bottle necked by the usability of the annotator interface. In this paper, we present the results of a pilot study exploring how the user interface impacts annotator agreement when judging summary quality.

## introduction as language models become more powerful, training and evaluation are increasingly limited by the data and metrics used for a particular task @xcite . human evaluation has traditionally been used in the field of summarisation as a gold standard when assessing the quality of model outputs and for corroborating automated evaluation techniques. however, ensuring high quality evaluation with human annotators is difficult due to the subjective and task-dependent paradigm of summarisation. as model refinement will increasingly rely on human feedback it is important to consider how to best elicit high quality signal from human annotators. one technique to judge the quality of summaries is the use of human preferences via comparison or ranking. such rankings can also be used to improve summary quality by training models in a reinforcement learning paradigm. for instance, @xcite show that human preference data can be used to improve the capability of large language models (llms) for summarisation via a technique referred to as reinforcement learning from human feedback (rlhf). however, human feedback does not always provide a gold standard for summarisation when the task is not clearly defined. it has been established that linguistically trained, expert raters, provide the gold standard in summarisation evaluation and the reliability of non-experts has been repeatedly questioned @xcite . for instance, it has been found that crowd workers should not be used to evaluate summary quality because of a non-correlation with experts @xcite . furthermore, even for expert annotations mediation meetings are necessary to assure reliability @xcite . in short, evaluating the quality of a summary is not an easy or straightforward task. the use of rlhf to train llms is becoming increasingly common. however, training a model from human feedback relies on the collection of data via user interfaces for the chosen task. increasingly then, natural language processing applications are heavily influenced by the human computer interaction that takes place when collecting preference data. recent work in rlhf for summarisation overlooks how critical the user interface is in this process, with little to no discussion of the design decisions made. in this paper, we present the findings from a pilot study introducing a novel user interface for summary comparisons. we document how the introduction of the new interface impacts annotator engagement as well as investigate the following research questions: it is widely understood that machine learning systems are limited by the quality of the labelled training data @xcite . one approach to improving the performance of such systems is to treat the human labeller(s) as a source of noise @xcite who can be modelled statistically @xcite in order to more accurately identify an underlying ground truth. noise estimation can be improved if multiple labels are obtained for each item in the training set in order to model inconsistency @xcite , or if a distribution of label values can be used as a basis for rejecting outliers @xcite . more recent approaches have relied on probabilistic methods for training deep classifiers under input-dependent label noise @xcite . however, these approaches focus on dealing with noise post-annotation, whereas it is known that the quality and clarity of the user interface itself, as well as the task formulation has large implications for the annotator agreement. for instance, several of the human factors can be addressed through the use of pairwise comparison, where labellers make relative judgments to compare training items, rather that attempting to characterize each item independently against an abstract conceptual category, for which they are expected to have a stable definition and associated membership criteria. in the context of labelling, comparative judgments are used to compare how well the training items correspond to the required concept. @xcite demonstrate that this method can facilitate judgments for information retrieval applications. comparative judgments have also been used in gamified labelling @xcite , where cooperating players reduce the set of alternative items until agreement is reached. recent work has looked into the application of comparative judgments to labelling as opposed to assignment of categorical values or scores on a scale @xcite @xcite . @xcite note that comparative judgments are suitable for abstract linguistic properties, whose nature can cause inconsistencies in the assigned numerical scores. ## experimental design our study compares the use of a baseline interface for summary comparisons with an novel interface designed in conversation with annotators. we investigate how the use of the new interface impacts both annotator engagement and agreement using specially trained annotators for the task of summary comparison. ## results the following results section presents the findings related to annotator behavior and agreement in the decision-making tasks. the initial analysis focuses on measuring the time taken by annotators to make decisions in two task settings. additionally, we examine annotators' engagement with the highlighting tool in the second interface, assessing the overlap between annotators and the tool's usage patterns. lastly, we evaluate the level of agreement among annotators when selecting the best summary in both scenarios. these results provide insights into annotator decision-making processes, tool engagement, and the consensus achieved, contributing to a better understanding of the task dynamics and effectiveness in each setting. ## discussion after receiving written feedback from annotators following the adoption of the new user interface, it was noted that all annotators stated the highlighting feature was useful. however, when analysing annotator behaviours not all annotators are using the tool. this presents an interesting issue of misaligned incentives, where annotators may feel the need to praise the new interface to maintain their employment status, even if they don't actually find it useful. while it's beneficial to have a consistent pool of annotators for engagement purposes, it's challenging to eliminate the power dynamic that arises from employing them directly. therefore, performing an interaction-based analysis is valuable as it shows the true nature of tool adoption by annotators. it is possible that the lack of adoption among some annotators is due to unfamiliarity rather than a lack of utility, which may change over time. in the second setting, we observed a reduction in 2 using r formula notation, the model is: agreement ∼ uncertainty + (1|participant) + (1|task). tests were performed using the lme4 and lmertest r packages by @xcite . the new interface has a significant advantage in that it enables us to use annotation behaviour to gain a better understanding of the task of summary comparison. for instance, we have observed that annotators who use the highlight option tend to prefer nouns as their preferred token type to search for. furthermore, we have found that the stability of annotator choices during annotation (i.e. the frequency of deselecting an option) is a reliable indicator of annotator uncertainty and is strongly correlated with the level of agreement among annotators. these behaviours are statistically significant and can be used to predict the likelihood of achieving high agreement. ## limitations and future work the authors would like to emphasise that this paper presents an initial pilot study aimed at documenting the process of updating an internal annotation tool. our main contribution lies in emphasizing the influence of task conceptualization and interface design on annotator agreement. additionally, we draw attention to the significant impact of the interface used for annotating summaries in the current human feedback reinforcement learning paradigm, which is often overlooked. while there is a distinction between binary selection of the best summary and n-ary ranking, it is still the case that both scenarios involve selecting a preferred top candidate. therefore, the substantial difference in agreement rates raises questions about the stability of the task and how the experimental setting can affect annotators' perception of summary quality, even among experienced and trained individuals. it is important to acknowledge that due to the variations in experiment settings and interfaces between the two task formats, it is difficult to draw definitive conclusions about the primary factor impacting annotator agreement. however, as an initial exploratory pilot study, our focus was primarily on assessing the tool's robustness and comparing the relative times taken in the different scenarios as well as measuring the annotator's usage of the new tool. in future work, we will investigate how annotator behaviors can provide insights into the task difficulty and likelihood of agreement. this will involve analysing the interactions with the new interface, such as the time taken to complete the task, the frequency of selecting tokens, and the number of summary selections. by gaining a better understanding of the cognitive processes involved in annotation and how they affect annotator agreement, we can improve the development of annotation tools and methodologies for more accurate reward models. ## conclusion the results of this pilot study emphasise how subtle variations in an annotation task can impact annotator agreement. even highly experienced annotators can experience fluctuations in agreement as a result of interface changes. to aid in the annotation of summary comparison, we developed a new interface that allows tokens to be selected and displayed across resulting summaries and observed patterns in the types of tokens highlighted by annotators. moving forward, we plan to conduct additional studies to explore the use of implicit interaction signals in predicting annotator agreement.