C ode A nno: Extending W eb A nno with Hierarchical Document Level Annotation and Automation (2023)

Abstract: WebAnno is one of the most popular annotation tools that supports generic annotation types and distributive annotation with multiple user roles. However, WebAnno focuses on annotating span-level mentions and relations among them, making document-level annotation complicated. When it comes to the annotation and analysis of social science materials, it usually involves the creation of codes to categorize a given document. The codes, which are known as codebooks, are typically hierarchical, which enables to code the document either with a general category or more fine-grained subcategories. CodeAnno is forked from WebAnno and designed to solve the coding problems faced by many social science researchers with the following main functionalities. 1) Creation of hierarchical codebooks, with functionality to move and sort categories in the hierarchy 2) an interactive UI for codebook annotation 3) import and export of annotations in CSV format, hence being compatible with existing annotations conducted using spreadsheet applications 4) integration of an external automation component to facilitate coding using machine learning 5) project templating that allows duplicating a project structure without copying the actual documents. We present different use-cases to demonstrate the capability of CodeAnno. A shot demonstration video of the system is available here: https://www.youtube.com/watch?v=RmCdTghBe-s

## introduction when webanno was initiated, the main purpose was to address the missing functionalities from the annotation tools, particularly adding the distributive, web-based, generic, and customizable annotation layers behaviors @xcite . since then, webanno has been improved a lot, including the support for semantic annotation @xcite , support for automation functionality @xcite , and adapted to the integrating knowledge-supported search 'incep-tion' platform @xcite . the span annotation type, which allows annotating tokens, sub-tokens, and phrases as well as the relation type that connects two-span annotation types with relation value in webanno covers most of the linguistic and machine learning annotation tasks. the document level annotation type is one of the most sought annotation types needed in webanno, which is also missed functionality from several annotation tools @xcite . one of the workarounds in webanno was the zero-width spans, which is not attached to any span but only to a specific sentence in a document. however, this still could not fully support document-level annotation, where the task is to label or classify a document based on a predefined category. the document-level annotation is particularly important for annotations in social science and digital humanities research. a typical example is document coding, where a researcher in communication science prepares a list of codes or tags known as codebooks. journalists and media analysts prepare the codes for a given codebook but the annotation of content involves a lot of distraction, such as 1) preparing the code in a spreadsheet, 2) coding or annotating an article or media which is not linked with the codebook in the spreadsheet, 3) analyzing and interpreting the results from the coding. codeanno is a fork and an extension to we-banno, which enables coding based on hierarchical and fully customizable document-level annotations within webanno. codeanno facilitates the creation of codebooks and coding using the existing functionality from webanno such as curation, coding agreement, distributed annotation (annotating a single document by multiple coders or users), webbased annotation, and so on. as codeanno supports all existing functionalities of webanno, for example, span and relation annotation, one can easily connect the coding with the built-in webanno annotation support that can help to draw some conclusions. the following are the main contributions of this work: 1) support codebook creation and annotation (hierarchical and document-level annotation), 2) importing and exporting annotations in commaseparated format, hence compatible with existing annotations conducted using spreadsheet applications, 3) integration of an external automation component to facilitate media coding using machine learning models, 4) project templating that allows duplicating a project structure without copying the actual documents. in the remainder of this paper, we will first discuss the related works of document-level annotation in section 2. in section 3 and 4, the main components of codeanno and the automation functionalities are presented. some use-cases showing the codebook annotation and automation components are presented in section 5 while section 6 highlighted the main contribution of this work. ## related work most of the annotation tools support linguistic annotations such as pos tagging, dependency relation, named entity recognition and so on @xcite . brat @xcite , @xcite , docanno @xcite , and lighttag @xcite are some popular annotation tools. the survey by @xcite indicated that only 5 annotation tools out 78 have some kind of document level annotation supports. ac-tiveanno @xcite is one of the latest annotation tools that support annotation and automation for document-level annotation tasks. activeanno focuses on five central design goals, namely, efficiently creating annotations of high quality, supporting a broad range of use cases, responsive web application, open-source and apis integration. while activeanno has great support for document-level annotation, it does not natively support hierarchical annotation using codebooks. it also lacks the extensive generic annotation support from webanno, which can be used to link the document level annotation to span level annotations. inception is another annotation platform, which integrates all the functionalities of we-banno and focuses on corpus creation, annotation, and knowledge management. while incep-tion extends the functionality of webanno to more advanced functionality, including the external recommender system and knowledge supported search, codeanno particularly focuses on the creation of document-level annotation with the hi-erarchical codebooks. furthermore, codeanno uniquely supports importing/exporting documents in a spreadsheet format, which is the de facto annotation format in social science document coding. project templating, external automation service, and document-level codebook agreements are additional specific functionalities in codeanno. in social science, coding refers to the process of assigning descriptive or inferential labels to documents or parts of the document, that could help in developing a new concept or theory @xcite . researchers usually take a sample of the data and apply manual coding. as the size of data increases, manually coding the entire dataset in detail is not feasible for social science researchers. building a representative machine learning model is also becoming challenging @xcite . to address this issue, we have integrated an external automation component that could provide an initial suggestion of codes. the process is iterative and adaptive, where small portions of the document are labeled to train the external predictive model. ## codeanno features codeanno extends webanno by introducing new functionalities in webanno. in webanno, "layers" are used for span, chain, or relation annotations on token, sentence, or paragraph level. codeanno introduces document-level annotations, referred to as codebooks. the new functionality is seamlessly integrated into webanno, i.e., the backend is also based on apache uima foot_0 while the frontend is based on apache wicket foot_1 . besides the support for codebooks, another main contribution of codeanno is the integration of an external machine-learning-based service to automate codebook annotations. codebooks are hierarchical document-level annotations consisting of a name, optional parent and child codebooks, and a list of codebook tags. the resulting codebook structure is therefore best represented as a tree. codebook tags are the value of the respective codebook annotation and can be predefined by the project organization or, if allowed, created ad-hoc by the annotator. further, codebooks and codebook tags can have a textual description so that the annotators know what they stand for and when or how to use them to ensure high coding quality. before the actual coding process starts, the project manager has to set up the codebook structure that the coders will use to annotate the documents. this is done in the project settings with an intuitive and easy-to-use codebook editor shown in figure 1 . in the left part of the figure, a codebook structure is displayed as a directory tree. the "expand all" and "collapse all" buttons are used to show or hide nodes in the codebook tree. codebooks can be moved up or down on the same level with the arrow buttons on the right. during creating a new codebook, the project manager enters the codebook properties in the middle part of figure 1 . to create a codebook hierarchy, the parent codebook can be selected from the drop-down list. if the codebook is a root, none is selected. it is also possible to change the parent codebook afterwards to update the tree structure. further, if annotators should be allowed to enter custom codebook tags, the respective checkbox must be selected. otherwise, only predefined codebook tags shown on the right side of figure 1 can be chosen during the coding process. in the "tag details" panel, that appears when a new codebook tag is created, its name, its description, and its parent tag will be recorded. the sort, move-up, and move-down buttons can be used to sort the list of codebook tags alphabetically or manually. when deleting a codebook that is not a leaf node in the structure, all codebooks in the subtree with their codebook tags also get deleted. moreover, it is possible to import or export codebook structures in human-readable json format. this is especially helpful when multiple coding projects with codeanno are planned. ## codebook automation (cba) another major component of codeanno is the codebook automation (cba) extension. as the name suggests, this functionality enables automatic annotation of codebooks leveraging state-of-theart machine learning technology to train and evaluate generic, user-specific classification models. the component consists of three parts: the cba backend, the cba webapp, and the cba integration in codeanno. an overview of the different parts is shown in figure 4 . the backend, which contains all logic and manages data and models, is accessible via a rest api. the cba webapp is a web-based user interface that consumes this api so that users can conveniently manage datasets, and create, train and test user-specific codebook classifiers. both parts are separated from codeanno and run independently. the third part is the cba integration in codeanno to apply trained classifiers to annotate, i.e., predict documents with codebooks automatically. ## conclusion in this paper, we discussed main functionalities of codeanno, an extension of webanno that supports coding of social science documents using hierarchical codebooks. social science researchers usually employ traditional annotation tools, for example spreadsheet applications to code and analyze a text. this annotation process is cumbersome and hinders the development of predictive machine learning models. codeanno supports the creation of codebooks, coding or annotating documents, and integration of custom machine learning models. further, it eases the analysis of annotated documents and enables to import and export documents in different formats. since codeanno intrinsically supports the annotation of entities and relations from webanno, we plan to integrate a functionality that provides automatic coding recommendations using the span-level entity and relation annotations. this facilitates the explainability of codebook annotations by linking back to the rationale in the document.