A ctive A nno: General-Purpose Document-Level Annotation Tool with Active Learning Integration (2021)

Abstract: ActiveAnno is an annotation tool focused on document-level annotation tasks developed both for industry and research settings. It is designed to be a general-purpose tool with a wide variety of use cases. It features a modern and responsive web UI for creating annotation projects, conducting annotations, adjudicating disagreements, and analyzing annotation results. ActiveAnno embeds a highly configurable and interactive user interface. The tool also integrates a RESTful API that enables integration into other software systems, including an API for machine learning integration. ActiveAnno is built with extensible design and easy deployment in mind, all to enable users to perform annotation tasks with high efficiency and high-quality annotation results.

## introduction lots of tasks in industry and research require the manual annotation of a potentially large number of documents, for example in content analysis research or supervised machine learning. existing tools, such as webanno @xcite , allow for span-level annotation tasks like ner and pos tagging. as @xcite point out in their review of annotation tools, document-level annotation is a feature missing in most existing tools. activeanno is a novel tool created to fill this void. it was created based on five central design goals: 1) creating annotations of high quality in an efficient manner (quality). 2) support a broad range of use cases without any additional programming effort (configurability). 3) provide a good user experience through a modern, responsive web application to be more attractive than specialized prototypes of annotation software (responsiveness). 4) publicly available to extend for future use cases (open source). 5) provide apis for easy integration into existing software systems and easy deployment to minimize the upfront effort for using activeanno (extensibility). through this approach, we would like to put forward activeanno as a candidate for the default option for document-level annotation in industry and research. according to @xcite , webanno is the best tool fitting their criteria, however, it does not support document-level annotations. webanno is a notable example of an annotation tool which is open-source, widely used and featurerich. we adapt some of the functionalities into our activeanno document-level annotation tool. another annotation tool with limited support for document-level annotations is doccano foot_4 @xcite , though it is not mentioned in the evaluation of neves and ševa. the opensource tool currently supports three distinct annotation tasks: text classification, sequence labeling, and sequence to sequence tasks. activeanno is a client-server application as depicted in figure 1 . on the server-side, the main component is a backend service written in kotlin using the ktor framework. the backend service is stateless, allowing for horizontal scaling for advanced performance requirements. it is connected to a mongodb database that is used to store documents and annotation projects. there is also a minimal authentication service written in python and django, which provides a simple user management ui and the authentication mechanism of json web token (jwt) that is used by the frontend and backend components. the authentication part is extracted into this service such that it can be exchanged for an existing or a more sophisticated user management service, if needed. finally, the frontend is written in javascript using react, which is served by an nginx. all backend components are deployed as docker containers through existing docker compose files. the communication between the client and server is done through http requests via the json format. activeanno documents: a document is represented as a json object with a textual field con-taining the document text, for example the text of a tweet or the abstract of a paper. it can optionally have additional fields with meta data that can also be displayed to support the annotation process. metadata examples include the date and username for tweets, the authors and publication for papers, or the number of stars for an app review. documents are imported through the ui or api in the json format and can be annotated for multiple annotation projects afterward. annotation definitions: annotation projects define an annotation task for one or more annotation definitions. an annotation definition describes the required type and form of the annotation that should be created. the typical example is a tag set annotation, where a document is annotated with one or more predefined tags such as spam/no spam, the sentiment or the topic classification of a text. activeanno also supports boolean, number, and text annotations with various configuration options such as minimum/maximum values. annotations and annotation results: for an annotation project, every document can be annotated for every annotation definition of the annotation project, resulting in one annotation per definition. together, all annotations for a document from a single annotator form an annotation result. every annotation of the annotation result has a different value structure depending on the type of annotation definition. additionally, every annotation value can have an associated probability used in the context of automatically generated annotations through the machine learning integration. annotation process: activeanno has a twostep annotation process. in the first step, annotation results are created. they can be created either by annotators, annotation generators (see section 4), or they can be imported through the import annotation api (see section 3.2). after every new annotation result, a finalization policy logic is applied to the document and its annotation results. this logic decides if the annotation process is finished for the document and a final annotation result exists. the logic is dependent on the project configuration. the manager of a project can set the number of annotators per document. this is the minimum number of different annotators required for each document until any additional logic is applied. for example, if the number of annotators is set to three, every document will be shown to the annotators until the document is annotated by three users. one annotator can only annotate every document once. after three annotation results were created, the finalization policy is applied. depending on the selected policy, the annotation results will either be exported individually, or the annotations are merged based on a majority calculation. if no majority is observed, the document can either be presented to an additional annotator until a majority is reached, or to a curator to decide on the correct annotation result. it is also possible to always require a curator for use cases with very high-quality requirements. ## annotation generator the integration of machine learning and the ability to automate parts of the annotation process are important to increase the efficiency of the annotation process. the basis for automation is the ability to automatically generate annotations without a human annotator. to generalize the concept and to allow for other ways to automatically create annotations, activeanno defines the concept of annotation generators. an annotation generator is anything capable of creating an annotation for a specific annotation definition given a document and potentially other previously generated annotations for other annotation definitions. an annotation generator is defined through an abstract class with a generateannotation method that every generator needs to implement. every annotation generator also has to define what is the input to use for the actual annotation generation. it can be any field from the original document, or it can be any value from another created annotation that is part of the same annotation schema, allowing for chaining or conditional connecting of annotation generators. currently, activeanno has three inbuilt annotation generator implementations. the first one automatically detects the language of the generator input using the language detection library lingua foot_5 . this is an example of a statistical and rule-based annotation generator as compared to a machine learning-based generator. the second annotation generator allows calling an external machine learning service through a url for tag set annotation definitions, which will take the response and map it into the tag set options from the annotation definition. this can be used when an already trained machine learning model exists. the model would have to be wrapped by an http api to comply with the api definition of activeanno for this annotation generator. the api is structured to allow for multiple documents to be predicted at once. the third annotation generator is similar to the second one, but also supports automatically updating the external machine learning model by sending an http request with the training data in the body. to support this functionality, the concept of an updatable annotation generator exists. this kind of generator extends the base annotation generator, but also requires its subclasses to implement an update method, where the training data will be aggregated and used to train or update the annotation generator. for this, updatable annotation generators also need to define a threshold when to start the training and when to update an existing model. for example, the first model should be trained after 100 training samples are generated, and then it should be updated for every 25 new training samples. an updatable annotation generator is versioned with a version number and an update state, to ensure the version is actually usable to generate new annotations. annotation generators can be triggered to generate/re-generate annotations for a project when appropriate, and they can be triggered to update themselves if they are updatable annotation generators and enough new training data is created to allow for a model update. ## use cases activeanno is designed to be applicable in many settings, including industry and research; for small projects on a local machine and large projects with many users on the internet; for use cases with and without machine learning components. we have used activeanno in two industry setups, to test its suitability to collect real-world annotation requirements. it was directly deployed on the service cluster of a company, configured via environment variables, and connected to the existing system via its api with some small additions to the existing software system to get documents into activeanno. for the machine learning integration, a small python service wrapping fasttext @xcite models was employed. the data for the experiments were around 250,000 ger-man textual feedback comments obtained from a retail context. the first experiment was set up to analyze the effects of pre-annotations on annotation quality and efficiency. three annotation definitions, namely spam/not spam, sentiment, and topic classification were annotated by two annotators employed at the company. one condition was provided without pre-annotations while the other one was provided with pre-annotations for all three annotation definitions. the annotations were created by pre-trained machine learning models that are integrated into activeanno through the annotation generator and the machine learning-related apis. through the analyze functionality of activeanno, the inter-annotator agreement (iaa), annotation duration, and the accuracy of annotators as well as the machine learning models are compared inside the tool. the final result showed a 28% faster annotation without any change of annotation accuracy or annotator agreement for the condition with preannotations. activeanno enabled the research project itself while the annotators reported a positive user experience using the tool. they reported that the tool is easy, fast, and convenient to use. the second experiment explored the incremental and active learning capabilities of activeanno. a new annotation definition about the utility (or information quality) of a comment with three classes (useful, okay, spam) was created as it was a new business requirement to create such annotations for further aggregation and processing. during the experiment, multiple new machine learning models for different conditions were created by annotating new comments inside activeanno and triggering the automatic annotation generator updating and re-annotating functionality. by additionally enabling pre-annotations, annotators had reduced effort for selecting the correct annotation option, once the model had an acceptable level of accuracy. selecting the best performing model based on the experiment conditions then enables and improves the regular annotation process of the company. once the model performs well enough while being used for pre-annotation, it could then be used to partly automate the process as described by simply editing the project configuration. finally, as both experiments were conducted on non-publicly available data, we created another example project based on the offenseval 2019 shared task @xcite , specifically sub-task a. the task is directly integrated as an example project in activeanno, including a machine learning component that can be updated through activeanno. please refer to the demo 7 and video 8 to see this use case in action. ## conclusion and future work activeanno supports several mechanisms to produce high-quality annotations with an efficient annotation process, like the number of annotators per document, a configurable agreement logic, curators, machine learning integration through annotation generators, pre-annotations, treating annotation generators as annotators, partly or fully automating annotations, and updating annotation generators with incremental or active learning. a fully configurable annotation schema with annotation definition types like tag sets, numbers and texts, a modern and responsive web ui, as well as flexible user management allows activeanno to be adaptable to many different kinds of annotation process requirements. the restful api and webhooks allow for easy integration with other software components. future work planned is to add in-app feedback between curators and annotators for improving annotator performance, adding more in-built annotation generators for other types of annotation definitions, ui improvements (layout editor, better display of very long documents) and span-level annotations as well as hybrid-level annotations which can be defined either on a span or document level. most importantly, activeanno was designed with extensibility and flexibility in mind. it is available as open-source software under the mit license.