GATE Teamware 2: An open-source tool for collaborative document classification annotation (2023)

Abstract: We present GATE Teamware 2: an open-source web-based platform for managing teams of annotators working on document classification tasks. GATE Teamware 2 is an entirely re-engineered successor to GATE Teamware, using contemporary web frameworks. The software allows the management of teams of multiple annotators, project managers and administrators - including the management of annotators - across multiple projects. Projects can be configured to control and monitor the annotation statistics and have a highly flexible JSON-configurable annotation display which can include arbitrary HTML. Optionally, documents can be uploaded with pre-existing annotations and documents are served to annotators in a random order by default to reduce bias. Crucially, annotators can be trained on applying the annotation guidelines correctly and then screened for quality assurance purposes, prior to being cleared for independent annotation. GATE Teamware 2 can be self-deployed, including in container orchestration environments, or provided as private, hosted cloud instances.GATE Teamware 2 is an open-source software and can be downloaded from https://github.com/GATENLP/gate-teamware.A demonstration video of the system has also been made available at https://youtu.be/KoXkuhc4fmM .

## introduction machine learning models are an important element of modern natural language processing @xcite . these models need to be trained and evaluated on gold-standard human-annotated datasets. depending on the purpose of the model, there are two types of annotations: classification and chunking. chunking tasks (e.g. named entity recognition) require the annotators to identify the subset (or span) of the given text, while classification tasks (such as sentiment and stance classification) require annotators to assign labels to the given text. gate teamware 2 focuses on classification annotation tasks specifically. depending on text length or specific machine learning tasks, the classification annotation could vary from word level to document level, or singlelabel to multi-label annotation. in addition, other information (e.g. confidence score for the annotations) may also be needed. this requires annotation tools to be configurable and flexible to adapt to different classification annotation tasks. furthermore, machine learning model performance is highly dependent on annotation quality and as such, large annotation tasks often hire multiple annotators to collaborate to remedy this. personal biases may be unknowingly introduced into annotations, so being able to collaborate with numerous annotators is a way to reduce the effect of this @xcite . here we introduce gate teamware 2 @xcite , a free open-source software tool for the management of teams of human annotators engaged in annotating datasets for document level classification, referred to here simply as document classification tasks. gate teamware 2 provides a highly flexible annotation configuration mechanism. annotators can participate in more than one project. in order to ensure high quality human annotations, gate teamware 2 also provides functionality for training and testing annotators on a subset of example documents, prior to qualifying them to perform the given annotation task independently. gate teamware 2 is the spiritual successor to the popular gate teamware @xcite , but has been entirely re-implemented in modern web frameworks with a containerised deployment architecture, as well as extended with new functionalities. teamware 2 has a number of unique features for an application specifically targeted at creating datasets for document classification tasks, includ-ing: as a demonstration of its utility, teamware 2 has already been used successfully for the annotation of several datasets consisting of tens of thousands of documents by numerous annotators per project on research topics such as covid vaccine hesitancy on twitter, stance detection in tweets (figure 1 ) and emotion recognition. ## related work there is an abundance of software tools facilitating document annotation serving a variety of concerns in this area. neves and ševa (2020) collated a searchable directory of over 90 such tools and their features. in this section, the focus will be on the review of tools which can perform document classification while also offering collaborative features that allow multiple users to annotate the same set of documents. according to @xcite and additional web searches, four annotation tools are available, which can: these tools are doccano @xcite , tagtog @xcite , universal data tool @xcite , and label studio @xcite . these four tools all allow self-hosting of the software, enable multiple annotators to annotate documents through a web interface on a single hosted instance, and support a wide variety of annotation tasks out of the box in addition to document classification. however, they vary widely in availability of collaborative features and handling of document classification tasks. in terms of suitability for document classification, while tagtog @xcite can perform document classification as it allows document-wide labels to be captured, its focus is on sequence labelling tasks, and this is reflected in the user interface where these document-wide labels are only located in the sidebar, making it unsuitable for use especially when document classification is the only annotation task. both universal data tool @xcite and doccano @xcite have recipes for performing document classification, but both are only able to capture a single category; expect a fixed document layout; and have no options to customise the presentation of the text to be annotated by the user. label studio @xcite is perhaps the most similar tool to gate teamware 2 in that it provides a fine-grained way to customise the number of labels, their layout, and how the text to be annotated can be formatted and presented to the annotator. in terms of collaborative features, universal data tool @xcite , unlike the other tools, does not have user accounts and collaborators can join an annotation project through a generated link which would make the tool more suitable for smaller and shorter annotation projects. the free version of label studio @xcite does not allow multiple users with clearlydefined user roles -all users are able to add and modify documents and annotations freely, though this may be different under a paid subscription. finally, tagtog @xcite and doccano @xcite are the only two applications that have clear definition of user roles and limit annotators to performing annotation tasks, preventing them from modifying annotation project settings. with these four tools, the expectation is for annotators to annotate every document in the project. with larger annotation projects that can contain thousands of documents, in order to limit the number of documents that have to be annotated by each individual annotator, it becomes necessary to split the corpus into multiple projects, recruiting different sets of annotators to each one. gate teamware 2 solves this problem by allowing limits to be placed on the number of annotations a document requires and the maximum proportion of the corpus a single annotator can annotate. this means annotation tasks can instead be automati-cally distributed to annotators within the project whilst avoiding the problem of annotators having too many annotation tasks. ## gate teamware 2 overview gate teamware 2 provides a database-backed web application for managing groups of annotators and providing document classification annotations. it is written in the modern web frameworks django and vue.js with a postgresql database. a teamware 2 instance is set up by an administrator for a set group of users who can act as project managers and/or annotators. project managers can create projects, which represent annotation tasks and are configured for the annotation of a single corpus of documents. annotators can then be added to a project to create annotations according to the guidelines and configuration for that project. ## user roles there are 3 user roles in teamware 2: annotator: can only annotate via the annotation view, when assigned to a project with annotation tasks available to them. once a task is complete, they must contact a manager to be added to any new projects. this is the default role assigned to newly registered users. see section 6. manager: can annotate as the annotator above. also has the ability to create new projects and edit all existing projects on the instance. see section 5. admin: as roles above; also has total responsibility over an entire teamware 2 instance, including the ability to promote annotator users to manager, and manage all users. ## annotation projects annotation in gate teamware 2 is organised into 'projects', each of which has a manager, who owns the project. managers create projects simply through a web interface which populates the configuration form with default options. alternatively, an existing project can be 'cloned' to create a new project with no documents or annotators, but an identical configuration. projects are configured to include a description and annotator guidelines, both of which can contain text formatted as markdown or html and can be previewed by the project manager. annotation parameters are set at this stage to include: the total number of annotations required per document; the maximum proportion of documents in the corpus that any one user can annotate; and how long an annotator assigned a document has to annotate it before it is returned to the pool of documents. in the interface, each of these options has concise inline help text to remove ambiguity and streamline the project setup process. ## annotating with gate teamware 2 once logged in, annotators are only able to view the annotation view, which presents them with the next annotation task assigned to them. depending on the project configuration, there are several options made available; such as rejecting an annotation, or adding a training or test stage to the project. once there are no more documents available, they are instructed to contact the project manager to be assigned to a new project (if applicable). during an annotation task, annotators are able to go back to the previous annotation to make a correction. annotations can also be changed at any stage by visiting the 'my annotations' view which displays all of an annotator's annotations. a complete history of all annotations and changes made is kept for posterity. ## software quality assurance quality and consistency of gate teamware 2's source code is ensured via extensive software testing using pytest for the back end unit tests, jest for the front end unit tests and cypress for integration testing. testing currently covers 85% of the codebase. the software is under active development and uses a continuous integration approach using github actions to run tests on all changes to the source code, therefore reducing breaking changes to functionality. ## conclusions & future work gate teamware 2 is a fast and simple to use platform for facilitating collaborative document annotation, and includes a number of important features such as pre-annotation upload, flexible annotation configuration and multiple user management. the software is under active development and a number of valuable features will be coming to the software soon. for instance, in addition to document classification, sequence labelling is planned to be integrated in the future, allowing it to be used in an even greater range of nlp annotation tasks. other planned future features include: calculation of inter-annotator agreement metrics, ability to define hierarchical/dependent annotations, exposing an annotation management api for integration with active learning algorithms, control over project visibility and ownership, and availability of the platform as a cloud service.