FITA nnotator: A Flexible and Intelligent Text Annotation System (2021)

Abstract: In this paper, we introduce FITAnnotator, a generic web-based tool for efficient text annotation. Benefiting from the fully modular architecture design, FITAnnotator provides a systematic solution for the annotation of a variety of natural language processing tasks, including classification, sequence tagging and semantic role annotation, regardless of the language. Three kinds of interfaces are developed to annotate instances, evaluate annotation quality and manage the annotation task for annotators, reviewers and managers, respectively. FITAnnotator also gives intelligent annotations by introducing task-specific assistant to support and guide the annotators based on active learning and incremental learning strategies. This assistant is able to effectively update from the annotator feedbacks and easily handle the incremental labeling scenarios.

## introduction manually-labeled gold standard annotations are the first prerequisite for the training and evaluation of modern natural language processing (nlp) methods. with the development of deep learning, neural networks have achieved state-of-the-art performance in a variety of nlp fields. these impressive achievements rely on large-scale training data for supervised training. however, building annotation requires a significant amount of human effort and incurs high costs, and can place heavy demands on human annotators for maintaining annotation quality and consistency. to improve annotation productivity and reduce the financial cost of annotation, many text annotation softwares are developed by constraining user actions and providing an effective interface. in the early days, platforms for linguistic annotations such as @xcite , bart @xcite , @xcite mainly focused on providing a visual interface for user labeling process, making annotation accessible to non-expert users. recently, integrating active learning into annotation systems for providing suggestions to user has became mainstream (textpro @xcite , @xcite , active dop (van cranenburgh, 2018), in-ception @xcite , etc), but most of these works focus on english text and rarely consider the multi-lingual setting, which is necessary due to the growing demand for annotation in other languages. in addition to the interface and efficiency, incremental annotation is also necessary in realworld scenarios since the pre-defined annotation standards and rules cannot handle rapidly emerging novel classes in the real world, while being less addressed in existing annotation tools. to address the challenges above, we propose fitannotator, a generic web-based tool for text annotation, which fulfills the following requirements: data human model s c h e m a a d a p t e d active learning incremental learning crowdsourcing s c h e m a d e s i g n human-in-ai-loop figure 1 : the interaction of the three major elements of the intelligent annotation system for annotating from sketch and expansion. fi-tannotator sets up an independent data loader and data provider, which can continuously import data to the corpus in bulk. the flexible data provider also brings new problems, such as dynamic labeling schema, which should be solved by incremental learning. figure 1 reflects our design philosophy and comprehension of the interaction between the three major elements in our annotation system. ## related works in recent years, the nlp community has developed several annotation tools @xcite . yedda @xcite provides an easy-to-use and lightweight gui software for collaborative text annotation, and provides certain administrator analysis for evaluating multi-annotators. flat foot_0 introduces generalised paradigm and well-defined annotation format defined in folia @xcite , and provides web-based annotation interface. doccano @xcite is an open-source, web-based text annotation tool that provides collaboration, intelligent recommendation functions, and includes a user-friendly annotation interface. inception @xcite is a comprehensive text annotation system, which is also web-based and open-source, integrates active learning algorithms and provides various interfaces for different annotation tasks, and it is developing for more tasks @xcite , more convenient @xcite and low-resource scenarios @xcite . in addition, commercial annotation tools such as prodigy foot_1 , tagtog foot_2 , lighttag foot_3 also provide powerful active learning support, team-collaboration functions, efficient user interfaces, and provide more related commercial solutions, which have gained appreciable business achievement. all of these intelligent text annotation tools have several common features: supporting active learning and a rich variety of tasks. and commercial annotation tools pay more attention to user experience and collaboration. ## architecture the architecture of fitannotator is influenced by the ideas of functional programming and, in particular, by the desire to combine functional with object-oriented programming. the adherence to the programming principles such as immutability and modularity, fitannotator is developed by hybrid programming language python. an overview of our system is shown in figure 2 , which has four main modules: ## intelligent annotation creating high-quality annotated corpora is a laborious process and requires experts who are highly familiar with the annotation schemes and standards. to accelerate the annotation process, we introduce the intelligent assistant that incorporates task-specific neural networks which actively assist and guide annotators. the cores of intelligent annotation are two adaptive learning mechanisms: active learning and incremental learning.