Chat Disentanglement: Data for New Domains and Methods for More Accurate Annotation (2023)

Abstract: Conversation disentanglement is the task of taking a log of intertwined conversations from a shared channel and breaking the log into individual conversations. The standard datasets for disentanglement are in a single domain and were annotated by linguistics experts with careful training for the task. In this paper, we introduce the first multi-domain dataset and a study of annotation by people without linguistics expertise or extensive training. We experiment with several variations in interfaces, conducting user studies with domain experts and crowd workers. We also test a hypothesis from prior work that link-based annotation is more accurate, finding that it actually has comparable accuracy to set-based annotation. Our new dataset will support the development of more useful systems for this task, and our experimental findings suggest that users are capable of improving the usefulness of these systems by accurately annotating their own data.

## introduction rapid synchronous chat involving a large group often leads to overlapping conversations. the challenge of disentangling these conversations has been studied for over a decade, but the main datasets are expert annotated and based on discussion of linux @xcite and ubuntu @xcite . recent work has considered scripts from movies @xcite , but there is still the need for data from additional sources to measure the generalizability of methods. a range of methods have been proposed to avoid expensive expert annotation in nlp, e.g., crowd work @xcite , games with a purpose @xcite and user feedback @xcite . various annotation methods have been used for disentanglement, but all focused on experts and only one study has compared annotation tools @xcite . this work takes two key steps to expand this task to new domains: (1) we created a new, multidomain, gold-standard dataset, and (2) we explored annotation methods to see if domain experts and crowd workers can do the task. our dataset includes several important variations not seen in existing datasets: (a) new types of conversations (e.g., meetings), (b) new types of user relationships (e.g., business-customer), and (c) a range of internet relay chat (irc) networks. we annotated 600 messages from each channel, which is enough to evaluate out-of-domain ccuracy. it is impossible to collect expert labels for every domain. however, if we can develop the right tools, owners and users of channels may be able to improve models by annotating some of their own data. we conducted a user study with domain experts and crowd workers, exploring two types of variation in user interfaces: (1) whether annotators receive automatic guidance, and (2) what structure is annotated. prior work has speculated that link annotation foot_0 is more accurate than set annotation foot_1 @xcite , but our work is the first controlled comparison. we found that domain experts can effectively annotate data, and improve with automatic guidance. crowd workers struggled with the task, doing worse than an automatic model. set-based and linkbased annotation are actually comparable in accuracy. we recommend link annotation as it provides the internal structure of conversations. the dataset we release foot_2 will support the development of more generalizable models, and our findings show how to help domain experts annotate effectively. together, these results will enable progress on this challenge in new domains, making conversations easier to follow for everyone online. ## related work all prior annotation for conversation disentanglement has been done by trained experts, like many tasks in nlp @xcite . early work on the task asked annotators to form sets of messages @xcite , but they speculated that annotators may be more consistent at annotating reply-to links. subsequent work took the link approach @xcite @xcite . this work is the first controlled comparison of the two. @xcite compared a command-line ui and gui, finding that annotators preferred the gui, but accuracy was the same, and using the gui was slower. our study complements theirs by considering: (1) variation in who annotators are, (2) variation in the form of annotation, and (3) guidance. crowd work can be cheaper and more scalable than expert annotation @xcite . effective crowd annotation user interfaces and workflows have been developed for a range of tasks (e.g., @xcite @xcite , but there has been no prior work for disentanglement. guiding annotators using an automatic system has improved speed for other tasks @xcite . recent work has applied similar ideas to crowd work @xcite . we apply this idea to conversation disentanglement for the first time. ## data in new domains when multiple synchronous conversations are happening in the same channel they can be difficult to understand. 4 conversation disentanglement is the 4 some services, e.g., slack, webex, and microsoft teams, have the ability to split a conversation starting at a message, but that only solves the problem if the split is created as soon as a new topic is started and the conversation remains on topic. task of identifying separate conversations, to make them understandable and useful. there are hundreds of active internet relay chat (irc) foot_3 channels, but only two have disentanglement annotations: #ubuntu @xcite and #linux @xcite . to create a realistic out-of-domain setting, we annotated data from four diverse channels, described in for each channel, we used three random samples, each 1,200 messages long (200 to annotate, 1,000 for context). this leads to a total of 2,400 annotated messages and a further 12,000 context messages. our data is in the same format as @xcite to enable easy evaluation. this is the first work to annotate multi-domain data, enabling outof-domain evaluation. expert annotation to make a gold-standard reference, two of the authors labeled each file, then one of the authors adjudicated disagreements. to match the annotations of @xcite as closely as possible, we labeled reply-to links using their tool, slate @xcite , and the same annotation guidelines. conversations are the connected components in the reply-to graph. ## improving annotation to go beyond expert-annotated resources, we need effective annotation methods for either users (e.g., domain experts who run a channel and are willing to annotate data for their own use) or crowd workers (who can be recruited at larger scale). we perform the first experiments in annotation with both of these groups, exploring several variations in tool support for them. ## results ## limitations there are three main limitations of this work. first, the study participants are an approximation of domain experts, rather than being actual users of the irc channels we consider. we believe computer science students are a reasonable proxy, given their knowledge of the subjects discussed in these channels, but it is possible that they are unaware of community-specific conventions or jargon. second, we only considered online communities writing in english. it is possible that communities writing in other languages use significantly different conventions that make this task easier or harder. third, our sample size is only large enough to make strong claims about some of the variations in results. it's possible that other variations in ## conclusion this work makes two key contributions. first, the new dataset we are releasing expands the scope of multi-domain evaluation of conversation disentanglement models. second, our user study of variations in annotation tools shows that domain experts can effectively annotate, particularly when given automatic guidance. together, these contributions show how better models and systems can be created that give domain-expert users the ability to improve systems. that will enable the use of this technology in a wide variety of new domains.