HUMAN : Hierarchical Universal Modular AN notator (2020)

Abstract: A lot of real-world phenomena are complex and cannot be captured by single task annotations. This causes a need for subsequent annotations, with interdependent questions and answers describing the nature of the subject at hand. Even in the case a phenomenon is easily captured by a single task, the high specialisation of most annotation tools can result in having to switch to another tool if the task only slightly changes. We introduce HUMAN, a novel web-based annotation tool that addresses the above problems by a) covering a variety of annotation tasks on both textual and image data, and b) the usage of an internal deterministic state machine, allowing the researcher to chain different annotation tasks in an interdependent manner. Further, the modular nature of the tool makes it easy to define new annotation tasks and integrate machine learning algorithms e.g., for active learning. HUMAN comes with an easy-to-use graphical user interface that simplifies the annotation task and management.

## introduction access to suitable annotated data constitutes a fundamental prerequisite for r&d of machine learning algorithms and models. the demand for new annotated data is growing as new data is collected or existing data is being re-annotated from a new perspective. in the area of natural language processing (nlp) alone, there is a large variety of tasks that require different types of annotations to be covered by a tool. this includes named-entity recognition or part-of-speech tagging, which require a tool to cover sequence labeling @xcite , coreference and dependency parsing requiring relational annotations @xcite @xcite , or any since most annotation tools cover only one or two annotation types, a change in the annotation task can easily require a change in the annotation tool itself. commercial platforms such as light-tag foot_0 or prodigy foot_1 cover a larger array of tasks to choose from. however, none of these are able to chain a mixture of different tasks (e.g., documentlevel classification followed by finer-grained sequence labeling) to be performed on a single annotation instance. one tool that comes close to achieving this, is angrist foot_2 , however, its lack of modularity makes it difficult to adapt to new annotation scenarios. hierarchical universal modular annotator (hu-man) follows a highly modular concept, which makes it easy to adapt to a specific annotation scenario. it uses an internal deterministic state machine (dsm) to guide the annotator through the pre-defined annotation task(s). this usage of a dsm allows annotation tasks to be chained in any order needed and makes it easy to implement entirely new annotation tasks and custom features in the future. this modular nature is especially useful when single task annotations do not capture the reality of a problem or when several dependencies exist in the annotations. one example being hate speech corpora @xcite , where the target of hate is only supposed to be annotated if a comment has been previously annotated as hateful. further, human covers a variety of annotation tasks, including the often lacking multi-label annotations and document-level classification, but also sequence labeling on textual data as well as image labeling; a pursuit towards universality. an example involving both document-level annotation as well as sequence labeling is multi-lingual named entity recognition (ner), where the annotator has to identify the language on the document level and then annotate the named entities on the sequence-level. moreover, when the annotation need is single-task, the fact that many tasks are covered by human makes it easy to re-use previous installations of human for a new scenario, even if the task at hand changes. lastly, human makes the annotation of hierarchical data possible. that is, if an annotation instance is embedded in a context of previous content (e.g., comments in a forum) this context can be shown to the annotators. the remainder of this paper is organized as follows. in section 2, we explain the structure of human, starting with the architecture and followed by the internal deterministic state machine, annotation protocol, api, database, and graphical user interface. in section 3, we demonstrate the application of human for a real-life use case. this is followed by a discussion (section 4) and conclusion (section 5). ## system description the human annotation tool is primarily designed to run on a web server. as such its architecture follows a basic client-server model (figure 1 ). clients and servers exchange messages in a request-response pattern, where the client sends a request to which the server responds. the server, consisting of the database and the api, serves the code for the client. the database (section 2.4) is used for sending new annotation instances to the client or saving finished annotations which are sent by the client. the client is controlled by a dsm (section 2.1) to show an annotation task in the gui (section 2.5). for this it requests new content and sends finished annotations to the server when an annotation instance is completed. the annotators interact with the gui to solve annotation tasks. during setup of the human system, administrators design the annotation protocol (ap), which is a json-style definition of the annotation task(s) at hand. this is then used to generate both the database and the dsm. further, it is possible for the administrator to write a custom api on the server and call its functions through the dsm (section 2.3. annotations can be given as arguments to these functions making it easy to realize active learning or similar tasks. the server is implemented using flask (grinberg, 2018), a common web framework for python. the client is written in typescript and transpiled to javascript. the client was tested on chrome/chromium (v85.0) and firefox (v80.0.1). ## use-case in the case of the interdisciplinary project m-phasis, where computer and communication scientists are collaborating to study online hate speech in user-generated content, an annotation tool was needed that can implement multiple facets of hate speech in the ap. annotating in communication science means following a complex ap which consists of different theoretically deduced categories to answer previously defined research questions or hypotheses @xcite . often there are multiple levels of analysis units which are hierarchical @xcite , e.g., news articles with corresponding comments which can be also divided into various statements. due to the analytical splitting of comments into different statements, it was crucial to have a tool that allows answering specific questions more than once for the same comment (loops). due to collaboration across disciplines it was necessary to combine the different types of select and label questions (section 2.2). to study media content, communication scholars apply an empirical method called quantitative content analysis, by which they try to systematically categorize content and formal characteristics of messages @xcite . the computational linguists, on the other hand, also required sequence labels to identify specific entities in the texts, e.g., targets of hate speech. for m-phasis there are different questions which depend on the type of text input: news articles (3 select questions, 1 label question, 1 boolean question) and comments. for comments, the questions vary according to type: moderating comment (1 select question) or user comment (10 label questions, up to 30 questions of another type). for the user comments, there are six different thematic blocks in the ap. some questions must be answered for all comments, other (follow-up) questions must only be answered when certain conditions apply to the particular comment (branching). this includes, inter alia, the question of whether statements within a comment contain a positive or negative evaluation of a target or an action recommendation of how to deal with a target, for example when the adaption of a specific behavior is demanded or a threat of physical violence is expressed. depending on the annotator's decision, follow-up questions are shown, e.g., regarding the characteristics of a negative evaluation. if required, the tool loops through a block of questions as many times as needed, for example when various evaluations of different targets are expressed in one comment. the flexible structure of human allows one to change the extent of the tool in consideration of the actual content. this enables complex annotations when necessary, but also makes it possible to shorten the ap and therefore the time spent per annotation -something no other tool has been able to accomplish until now. ## discussion human strongly follows its concept of modularity and allows for the design and implementation of complex annotation protocols. and while it is currently already able to handle a variety of tasks on textual data as well as pdfs and images, many tasks are still uncovered. two examples here being relationship annotations or asking open answer questions. in order to truly reach universality, we envision that the modular nature of the code will invite anyone interested to add new and custom features and annotation types to this open-source tool. ## conclusion and future work we have described human, a modular annotation tool that covers a variety of annotation tasks, ranging from document-level annotation over sequence labeling to image annotations. its usage of a deterministic state machine, also accommodates different annotation tasks to be chained in such a way that annotation decisions of the annotator can be followed by different subsequent questions (branching) or the revisions of previous questions (loops). its context and content fields make it possible to perform hierarchical annotations, i.e., annotating an instance together with the context it was embedded in. this is, as far as we know, the only annotation tool capable of covering such complex annotation needs. this is of use not only for disciplines that require multi-task annotation protocols, but also for various single-task scenarios where users do not want change the tool every time they have a new annotation need with a slightly different task. while human is already fully functional and has been used for a real-life annotation scenario, it is a work in progress. possible new annotation tasks could be e.g., annotations of relationships as in brat @xcite , of wave signals, similar to praat @xcite or even videos as in nova @xcite . in order to improve accessibility of the tool in the future, we plan to implement a drag-and-drop gui for the creation of the ap, as well as a visualization of the internally generated dsm to improve debugging. automatic calculation of statistics such as the inter annotator agreement and average time spent on an annotation instance are planned. to further ease the database management, administrators should have direct insight on each annotation instance in the database, which can then be added, removed or edited in the gui without the need of sqlite commands on the server. the code foot_5 is published under a gpl-3 licence together with a wiki with detailed instructions on how to setup the server and define an ap. it also explains how to write custom annotation states and api calls. two functioning demos of the human annotation page on two different aps are published on our homepage foot_6 .