Paladin: an annotation tool based on active and proactive learning (2021)

Abstract: In this paper, we present Paladin, an open-source web-based annotation tool for creating high-quality multi-label document-level datasets. By integrating active learning and proactive learning to the annotation task, Paladin makes the task less time-consuming and requiring less human effort. Although Paladin is designed for multi-label settings, the system is flexible and can be adapted to other tasks in single-label settings.

## introduction labelled data is essential in many nlp tasks based on machine learning. manually annotating such data is time-consuming, and require a lot of human effort. active learning has been used to ease this process by choosing the data points for annotation instead of annotating all instances of the unlabeled data @xcite . some recent research has also utilized proactive learning, in which the system is allowed to assign specific unlabeled instances to specific annotators @xcite . the annotators, in these scenarios, only have to annotate a small set of representative and informative data which they can provide reliable labels. it helps reduce the labelling effort and at the same time makes the best use of available annotators. to date, there are many tools available for active learning, such as the texnlp @xcite , the active-learning-scala (santos and carvalho, 2014), the jclal @xcite , the libact @xcite libraries, the vowpal wabbit 1 . these tools, however, focus only on the active learning algorithms and provide no user interface thus making it difficult to use for the end-users. on the other hand, several tools have been made with user-friendly interface such 1 http://hunch.net/ ˜vw/ as brat @xcite , webanno @xcite , pubannotation @xcite , doccano foot_0 . some of the tools offer active/proactive learning such as aplenty @xcite , dualist @xcite , alpacatag @xcite , discrete active learning coref @xcite . currently, these tools support sequence labelling/coreference resolution tasks but not document classification tasks. to the best of our knowledge, there is no such tool for document classification which supports active/proactive learning. prodigy foot_1 supports active learning for both sequence labelling and document classification tasks but it is a commercial product. to compensate for the lack of available document-level annotation tool, we develop paladin (proactive learning annotator for document instances), an open-source web-based system for creating labelled data using active/proactive learning 4 . the main innovation of paladin is the combination of a user-friendly annotation tool with active/proactive learning. specifically: 1. active/proactive learning integration: paladin makes annotation easy, time-efficient, and require less human effort by offering active and proactive learning. the remainder of this paper is organized as follows. section 2 presents details of paladin. section 3 describes a case study of using paladin for a multi-label document annotation task. section 4 concludes the paper and points to avenues for future work. ## system descriptions paladin is a web-based tool implemented in python using django web framework and vue.js. the main user interface consists of a project management page and an annotation page. below, this section describes paladin in detail. ## use cases the typical use cases of paladin are as following: with this option, the model will try to select more data from the potential minority classes for annotation. ## conclusion we introduced paladin, a web-based open environment for constructing multi-label document-level datasets using active and proactive learning. paladin can support the quick development of highquality labelled data needed to train and evaluate nlp tools for different applications. considerably more work will need to be done to further enhance paladin to work with other active/proactive learning algorithms. besides that, a natural progression of this work is to evaluate paladin in a large scale annotation project.