Tools Impact on the Quality of Annotations for Chat Untangling (2021)

Abstract: The quality of the annotated data directly influences in the success of supervised NLP models. However, creating annotated datasets is often time-consuming and expensive. Although the annotation tool takes an important role, we know little about how it influences annotation quality. We compare the quality of annotations for the task of chat-untangling made by non-experts annotators using two different tools. The first is SLATE, an existing command-line based tool, and the second is Parlay, a new tool we developed that integrates mouse interaction and visual links. Our experimental results indicate that, while both tools perform similarly in terms of annotation quality, Parlay offers a significantly better user experience.

## introduction human linguistic annotation is essential for many natural language processing tasks. however, the construction of these datasets is extremely expensive, both in terms of annotator hours and financial cost @xcite . we know that the performance of many natural language processing tasks is limited by the quantity and quality of the data available to them @xcite . many of the studies focus on the number of annotators and their experience in improving annotation quality. however, little has been studied about the role annotation tools play in producing quality data. we study the effect of annotation tools in chatuntangling task. chat-tangling occurs when simultaneous conversations arise in chat with multiple participants @xcite . the goal of chat-untangling is to identify the conversations in a chat-thread. in order to perform this task, annotators must maintain a complex mental representation of the ongoing conversations. hence, a tool should minimize the task load and facilitate the annotation process. to the best of our knowledge, there is only one public dataset sufficiently large for training modern nlp architectures for this task published by @xcite . this dataset was annotated using slate @xcite , a terminal based annotation tool. in slate, interactions are marked by keyboard commands, messages are shown as raw-text, and annotations are presented by color coding. we argue that these characteristics may influence in quality of annotations for non-experts. this paper presents parlay, a new annotation tool for the chat-untangling task. the main difference between parlay and slate is that it integrates mouse interaction and visual links into the annotation representation. we conducted a controlled experiment with 12 non-expert participants, in which each participant was first introduced to the annotation task and then asked to use each tool to annotate 100 messages. the data to be annotated are selected from gold standard adjudicated dataset presented by @xcite . next, each annotation tool is evaluated in terms of usability (sus), task load (nasa/tlx), performance and annotation time. we define annotator performance by the quality of its annotations calculated by comparing them to the gold standard (expert) annotations @xcite ) using cohen's kappa coefficient @xcite . ## background chat-untangling. chat-untangling is an nlp task that aims to find existing conversations within a chat-log with two or more participants @xcite @xcite @xcite . conversations are represented by a connected graph in which the vertices are messages and the edges are relationships between them (e.g., an answer to a question) @xcite @xcite . annotation process. manual text annotation is the process of assigning some tags to the whole text or to fragments of it. a corpus is a collection of texts on a particular topic. annotators tag parts of the text in the corpus with labels that represent the structure or semantics of interest. the annotated data then goes through a curation process in which a curator manually resolves discrepancies and inconsistencies. curation is primarily performed to ensure data quality @xcite . one important step in curation is adjudication @xcite . in this step the curator merges the annotations from different annotators though agreement and resolves discrepancies to produce a gold standard annotation. the annotation process concludes with the release of the curated corpus to the community. annotation quality. to a large extent, the final quality of the annotation process depends on how human error is reduced and how discrepancies are consolidated without biasing the annotator's judgment @xcite . it is demonstrated that the annotation quality is related to the annotator expertise @xcite , number of annotators @xcite and the process of learning the annotation task @xcite . on the other hand, annotation tools are used to assist the annotation process (e.g., file loading, user annotation interaction, data curation) @xcite @xcite . although there is evidence that user-friendly annotation tools can benefit the annotation process and reduce the annotation time @xcite @xcite , little is know how can they can influence to the annotator's performance. to the best of our knowledge, grosman et al. ( 2020 ) is the only study that report changes in the inter-agreement evaluating different tools for annotation. however, there is no further understanding how this occurs at annotation time. ## methodology in this section we introduce parlay, a new chatuntangling annotation tool, and slate as baseline. finally, we provide the evaluation methodology, criteria and our hypotheses during this study. ## results in this section we present the results of our methodology of evaluation for i) the task load, ii) usability, iii) performance and iv) annotation time. task load. usability annotation time. slate reported less annotation time in minutes in our participants (@xmath0.083, @xmath1.714) compared to parlay (@xmath2.5, @xmath3.211). lastly the second evaluation session reported less annotation time in our participants (@xmath4.417, @xmath5.249) compared to parlay (@xmath6.167, @xmath7.643). performance. we assess our hypotheses by calculating the performance (κ) by two criteria: i) the tools used in the annotation task (h1) and ii) the session in which the participants annotate (h2). for h1 there was a non-significant difference in the scores for parlay and slate with p-@xmath8.1583. for h2 there was a significant difference in the scores for the first evaluation session and second evaluation session with p-@xmath9.007636. therefore, the quality of annotations increases as annotators gain experience. finally, the annotation tool used does not influence the quality of annotations. ## conclusion in this paper we evaluate the influence of the annotation tool for non-expert users in terms of data quality and user experience in the chat-untangling task. to achieve our purpose we introduce a new annotation tool named parlay and establish slate as our baseline. subsequently, we conducted a controlled experiment with 12 non-expert annotators in which each participant annotated 100 messages on each tool in turn. each tool was evaluated under i) usability, ii) task load, iii) annotation time and iv) annotation performance. lastly, we establish that the quality of annotations is measured by calculating the agreement (κ) between participants' annotations and the gold annotated data @xcite . the results indicate that the tools did not show significant differences in the annotators' performance outcome on the chat-untangling task. on the other hand, participants showed better performance in the second session, presumably due to a gain in experience. the annotation time is also lower in slate. complementary to these results, we also report that parlay scored better in usability and task load. where participants highlighted that i) link representation between annotated messages and ii) mouse interaction where the main characteristics that made parlay. in conclusion, parlay offers a better user experience while achieving comparable annotation performance. the study of annotation quality is an important issue for future development of nlp models. tools remain as an important factor in the development of high-quality training datasets for nlp tasks @xcite . despite the results presented in this study, further work is required to get a thorough understanding of how the annotation tool affects the quality of chat-untangling data. this study contributes to the area by introducing parlay, a new tool for the chat-untangling task. we believe that an important contribution of our work is the methodology we propose, which allows us to compare annotation tools according to both data quality and user experience. for future work we aim to study the inter-annotator agreement of each tool. as well as, the discrepancies between expert an non-expert annotations.