CHAMP : Efficient Annotation and Consolidation of Cluster Hierarchies (2023)

Abstract: Various NLP tasks require a complex hierarchical structure over nodes, where each node is a cluster of items. Examples include generating entailment graphs, hierarchical cross-document coreference resolution, annotating event and subevent relations, etc. To enable efficient annotation of such hierarchical structures, we release CHAMP, an open source tool allowing to incrementally construct both clusters and hierarchy simultaneously over any type of texts. This incremental approach significantly reduces annotation time compared to the common pairwise annotation approach and also guarantees maintaining transitivity at the cluster and hierarchy levels. Furthermore, CHAMP includes a consolidation mode, where an adjudicator can easily compare multiple cluster hierarchy annotations and resolve disagreements.

## introduction in numerous annotation tasks, the annotator needs to perform individual and independent decisions. such tasks include named entity recognition (ner), text categorization and part-of-speech tagging, among others @xcite @xcite @xcite . however, certain annotation tasks are more demanding because they involve the construction of a complex structure that must satisfy global constraints. one such complex structure is clustering, where annotated clusters must respect the equivalence relation. specifically, if items a and b belong to the same cluster, and items b and c also belong to the same cluster, then a and c must belong to the same cluster as well. another prominent example of a global structure is hierarchy, where typically, if a is an ancestor of b and b is an ancestor of c, then a must also be an ancestor of c. figure 1 : example of hierarchy of clusters from thinkp @xcite . nodes group similar statements together and arrows represent child-parent relations, relating specific statements to more general ones. in this work, we focus on annotating a hierarchy of clusters, a global structure that combines the constraints of both clustering and hierarchy, thereby posing further challenges. in this hierarchy, nodes are clusters of (text) items, where each node can have at most a single parent, as illustrated in figure 1 . annotating a hierarchy of clusters is relevant for a multitude of tasks, such as hierarchical cross-document coreference resolution @xcite , structured summarization as a hierarchy of key points @xcite , entailment graph construction @xcite and event-subevent relations detection ( @xcite . while there are some annotation tools for annotating either clustering or a hierarchy ( §2.1), to the best of our knowledge there is no available tool allowing to annotate a hierarchy of clusters simultaneously within the same tool. to address this need, we introduce champ (cluster hierarchy annotation for multiple participants), an intuitive and efficient tool for annotating a hierarchy of clusters in a globally consistent manner, supporting multiple annotators ( §3). specifically, annotators are presented with input text spans one by one and form incrementally and simultaneously the clusters and their hierarchy ( §3.1). additionally to the annotation process, we develop an adjudication mode for easily comparing multiple annotated hierarchies of clusters ( §3.2). this mode can be used either by an adjudicator, which is typically a more reliable annotator, or by the original annotators during discussions to resolve conflicts. indeed, adjudication is crucial to ensure quality in general @xcite , and particularly important for our structure, requiring a more challenging global annotation. we demonstrate the use of champ in two notably different use-cases, both involving annotating hierarchies of clusters: hierarchical crossdocument coreference resolution @xcite and key point hierarchy @xcite . in both settings, champ is significantly more efficient than a pairwise annotation approach, in which the relation between each pair of items is annotated independently. moreover, our consolidation phase enhances the annotation quality, yielding an improvement of 5-6 f1 points @xcite . champ was implemented on top of corefi @xcite , which was initially designed for coreference, and allowed only standard (non-hierarchical) annotation. champ includes a webcomponent, which can easily be embedded into any html page, including popular crowdsourcing platforms such as amazon mechanical turk. we also develop an annotation portal (the link appears in our github repository), allowing users to perform online the annotation task and dataset developers to effortlessly compute inter-annotator agreement. overall, champ is an intuitive tool for efficiently annotating and adjudicating hierarchies of clusters. we believe that champ will remove barriers when annotating such challenging global tasks and will facilitate future dataset creation. ## champ we present champ, a new tool for annotating a hierarchy of clusters. to annotate such a structure, the annotators are provided with a list of input spans, denoted as @xmath0, ..., s n }, that they need to group into disjoint clusters of semantically equivalent spans @xmath1, ..., c k }. in addition, annotators need to form a directed forest @xmath2, e), constituting a directed acyclic graph (dag) in which every node-representing the cluster c ihas no more than one parent. within this structure, each edge e ij represents a hierarchical relation between clusters c i -→ c j , signifying that c i is a child of c j . considering the example in figure 1 , the cluster {starts up very quickly, no waiting for long boot-ups} is more specific than the cluster {very fast for a laptop, amazingly fast device}. importantly, input spans can be standalone spans (as in figure 1 ) or appear within a surrounding context. for the remainder of this section, we will focus on demonstrating champ using standalone spans, while an example featuring spans within context is provided in appendix a. we next describe the core annotation interface ( §3.1), and then present the adjudication mode, which allows to effectively compare multiple annotations and build a consolidated hierarchy of clusters ( §3.2). ## applications we used champ for annotating datasets for two different tasks that require annotating of hierarchy of clusters: despite the different nature of these tasks and their unit of annotation (i.e., standalone state,emts vs. concept spans in context), we seamlessly leveraged champ for both with minimal effort (using a simple json configuration schema), as both tasks involve annotating a hierarchy of clusters. in our experiments, we observed that annotating or consolidating a hierarchy of clusters for fifty statements takes approximately one hour @xcite . in contrast, collecting annotations for all possible pairs, as commonly done in prior datasets for entailment graphs @xcite , would have been much more expensive since it would require at least 1225 decisions on average for our data, which would obviously take much more than one hour. furthermore, unlike the pairwise annotation approach, our incremental method for constructing a hierarchy of clusters guarantees that the resulting annotation will respect the global constraint of transitivity. finally, our experiments also revealed that the consolidation mode significantly enhances human performance, yielding a gain of 5-6 f1 points @xcite . ## implementation details and release we implement champ on top of corefi @xcite , using the vue.js framework, that we open source under the permissive mit license. following corefi, we release champ as a webcomponent, which can easily be embedded into any html page, including popular crowdsourcing platforms such as amazon mechanical turk. both the annotation and consolidation processes share the same interface and are easily configurable using a straightforward json schema. we also develop an annotation portal where users can upload a configuration file (either for annotation or adjudication), perform the annotation task and download it upon completion. this portal also provides the capability to upload multiple annotation files from various annotators and to compute the inter-annotator agreement. as such, champ is not only easy-to-use for annotators, but it is also easy to setup and manage for dataset developers. ## conclusion this paper aims to foster research on global annotation tasks by introducing champ, an efficient tool designed for annotating a hierarchy of clusters. this annotation tool also incorporates an adjudication mode that conveniently supports identification and consolidation of annotators' disagreements. as champ enables efficient and high-quality annotation, we believe that it will facilitate the creation of datasets for various tasks involving this complex structure, and will inspire tool development for other global annotation tasks. demonstrations, pages 31-36, melbourne, australia. association for computational linguistics. seid muhie yimam, iryna gurevych, richard eckart de castilho, and chris biemann. 2013. webanno: a flexible, web-based and visually supported system for distributed annotations. in proceedings of the 51st annual meeting of the association for computational linguistics: system demonstrations, pages 1-6, sofia, bulgaria. association for computational linguistics.