D o TAT : A Domain-oriented Text Annotation Tool (2022)

Abstract: We propose DoTAT, a domain-oriented text annotation tool. The tool designs and implements functions heavily in need in domain-oriented information extraction. Firstly, the tool supports a multi-person collaborative process with automatically merging and review, which can greatly improve the annotation accuracy. Secondly, the tool provides annotation of events, nested event and nested entity, which are frequently required in domain-related text structuring tasks. Finally, DoTAT provides visual annotation specification definition, automatic batch annotation and iterative annotation to improve annotation efficiency. Experiments on the ACE2005 dataset show that DoTAT can reduce the event annotation time by 19.7% compared with existing annotation tools. The accuracy without review is 84.09%, 1.35% higher than Brat and 2.59% higher than Webanno. The accuracy of DoTAT even reaches 93.76% with review. The demonstration video can be accessed from https://ecust-nlp-docker.oss-cn-shanghai.aliyuncs.com/dotat_demo.mp4 . A live demo website is available at https://github.com/FXLP/MarkTool .

## introduction a high-quality corpus is a prerequisite in supervised machine learning, especially for most neural natural language processing (nlp) systems. however, annotation is also one of the most timeconsuming and costly components of many nlp research work, and the quality of the annotation results greatly affects the effect of the trained model. currently more and more domain-oriented information extraction tasks @xcite @xcite are proposed, therefore annotation tools should be redesigned to meet the new requirements: however, the existing annotation tools only support one or two of the above requirements. only brat @xcite , webanno @xcite and inception @xcite support event annotation, but they do not design event annotation as a core function and do not contain enough features for specification management and quality improvement. to address the challenges above, we propose dotat, a domainoriented text annotation tool for complex event annotation tasks. specifically, it satisfies the abovementioned new requirements through the following methods which even support iterative annotation and automatic batch annotation: ## related work there are various text annotation tools for different scenarios, but most of them do not support event annotation, including @xcite , wordfreak @xcite , anafora @xcite , atomic @xcite , gate teamware @xcite , doccano and yedda @xcite . each tool has their own special features, e.g., word-freak supports constituent parse structure and dependent annotations as well as ace named-entity and coreference annotation. doccano and yedda support the use of shortcut keys for entity annotation, and yedda can perform batch annotation through the command line. currently only brat @xcite , webanno @xcite and inception @xcite support event annotation. however, it is difficult for them to annotate nested event. the method used by them for event annotation is to connect multiple entities through directed arcs. if the number of entities is numerous or the distance between entities is far, abundant arcs and intersections will appear on the whole page, resulting in an inferior visualization effect. except for wordfreak, anafora and atomic, most tools declare to support multi-person collaborative annotation. gate teamware provides the adjudication interfaces to compare annotations. however, only webanno and inception provide the curation with automatic merging function. inception is partially based on webanno (eckart de castilho et al., 2016). compared to these tools, event annotation in dotat is much easier to perform. furthermore dotat designs an iterative process from specification definition to merging and review, which can help the annotation team gradually increase the quality of annotated corpus. ## dotat dotat is a web-based multilingual text annotation tool. the raw texts that need to be annotated can be in chinese or in any other language. there are three types of user roles: administrator, annotator, and reviewer. the fundamental annotation types include entity annotation, relation annotation, event annotation, and text classification. as shown in figure 1 , a typical annotation process using dotat may include the following five steps: it is recommended that two annotators and one reviewer are assigned to each task. ## implementation dotat is a web-based text annotation tool with the software license apache-2.0. we used the vue.js and element ui to build the user interface. the core of vue.js is a responsive data binding framework, which makes it pretty easy to synchronize data with the dom (document object model). therefore, vue.js is particularly suitable for real-time visualization of text annotations. the server side utilizes the python-based open-source django framework to build restful web services. mysql database is adopted to organize, store and manage data. the code is available at the github repository https: //github.com/fxlp/marktool , which also contains a live demo website. ## experiments we compared dotat with the other two text annotation tools (brat and webanno) for annotation time (see section 5.1) and annotation result (see section 5.2) on the event annotation task. ## case study dotat has been used in the annotation projects of three different domains. the details in the public security and medical domains are shown in ## conclusions the demands for annotation corpus in different domains are rapidly increasing with the development of deep learning. we propose a web-based text annotation tool, dotat, which is suitable for domain-oriented complex event annotation. we demonstrate the powerfulness of our tool with experiments and real-world scenarios. we find that the pre-annotation and reviewing are critical steps to improve the quality of corpus. in the future, we plan to integrate the active learning algorithm into dotat to reduce the manual annotation work.