Erase and Rewind: Manual Correction of NLP Output through a Web Interface (2021)

Abstract: In this paper, we present Tintful, an NLP annotation software that can be used both to manually annotate texts and to fix mistakes in NLP pipelines, such as Stanford CoreNLP. Using a paradigm similar to wiki-like systems, a user who notices some wrong annotation can easily fix it and submit the resulting (and right) entry back to the tool developers. Moreover, Tintful can be used to easily annotate data from scratch. The input documents do not need to be in a particular format: starting from the plain text, the sentences are first annotated with CoreNLP, then the user can edit the annotations and submit everything back through a user-friendly interface.

## introduction in the last years, nlp tools are being more and more used in tasks such as textual inference, machine translation, hate speech detection @xcite . most of these tasks rely on machine learning systems trained on large amounts of data, which have been manually labeled by annotators, often domain experts. in particular, recent deep learning algorithms are more accurate, but they need more data for training, making the data collection a major challenge for the nlp community. when the annotation task does not require a specialised competence, one can use a platform such as amazon mechanical turk (amt), foot_1 that enables the distribution of low-skill but difficult-to-automate tasks to a network of humans, who could work in parallel, when and where they prefer. however, not all nlp assignments can be solved by non-experts because they may require background knowledge or linguistic expertise. for these tasks, expert annotators should be hired and receive a specific training, which is time-consuming and can become costly. a similar problem arises when a known task has to be ported to another domain, and existing tools turn out to have a poor accuracy, because the original training data does not include annotated instances from that domain @xcite . for example, the performance of standard nlp tools (part-of-speech taggers, dependency parsers, and so on) is severely degraded on tweets for this very reason @xcite . one of the solutions to this set of problems would be to make nlp tools more similar to wiki-like systems, where a user who notices some wrong annotation can easily fix it and submit the resulting (and right) entry back to the tool developers, so that they can add the instance to the training examples and re-generate the model. this is basically the paradigm already used for active learning @xcite which uses "humans in the loop" to increase the accuracy of a system, by including in the workflow a targeted correction of instances that are misclassified @xcite . as someone said way back in 1969: "computers are incredibly fast, accurate and stupid. on the other hand, a well trained operator as compared with a computer is incredibly slow, inaccurate and brilliant". @xcite to obtain a seamless integration between automatic classification and human correction, we need to develop nlp tools that are accessible through a user-friendly interface @xcite , easing the interaction between non-technical persons and the underlying technology. in this paper, we present tintful, a working example of the described paradigm, an interface that combines the output of stanford corenlp @xcite with a newly created annotation tool that allows the user to edit and fix the output data, in a wiki-like style. the user (registered or not) can edit the tokens, the lemmas, the parts of speech, the dependency trees and the named entities @xcite . thanks to an api released with the web interface, the resulting annotation can be stored for later use (for example, the software retraining), so that users can contribute to improving the system performance on specific tasks or domains of interest. compared to other similar tools (see section 2), tintful has some main strengths: finally, the whole tool is released open source and available on github (see section 7). ## related work some of the available programs for the manual annotation of texts are generic and can be configured and used for a great variety of tasks. these are usually powerful but need some work for configuration. some other, on the contrary, have been developed for a particular purpose, and usually are easier to launch and configure. webanno foot_3 (eckart de castilho et al., 2016) is a general purpose web-based annotation tool for a wide range of linguistic annotations and belongs to the former category. it is multi-user and supports different roles to guarantee a quality check of the annotated data. inception foot_4 @xcite is an opensource and multi-user text annotation platform developed at the technische universität darmstadt. it is general-purpose and can be configured to perform a number of annotation tasks. similarly, doccano foot_5 @xcite provides annotation features for text classification, sequence labeling and sequence to sequence tasks. regarding specifically the annotation of dependency graphs, there several tools that help researchers to manage complex output formats such as conll-u. 7 for instance, conllueditor foot_7 (heinecke, 2019) is an actively maintained tool which facilitates the editing of syntactic relations and morphological features of files in conll-u format. similarly, ud-annotatrix foot_8 @xcite is a language-independent tool for editing dependency trees according to the guidelines established by the universal dependencies project. truducer foot_9 (hennig and köhn, 2017) is a software for transforming dependency treebanks from one schema to another, especially to conll-u. finally, arborator foot_10 is an annotation tool for dependency trees that allows users to perform collaborative work. the tool has a specific focus on hci aspects, since most of the actions can be done using mouse drag and drop. in this paper, we propose a tool that is different from all the ones described above. with tintful the user does not need the data to be in a particular format: starting from the plain text, the sentences are first annotated with stanford corenlp, then the user can edit the annotations and submit everything back to the server. ## existing components tintful is mainly based on two existing pieces of software: stanford corenlp and brat. ## tintful architecture the architecture of tintful is represented in figure 1 . ## user experience in developing tintful, a particular attention has been paid to the interface, given that annotators performing linguistic tasks need to be focused and typically spend a lot of time interacting with the tool. ## tintful release the web interface of tintful is written using vuejs. 16 and the structure of the website is built with tailwind css. 17 the api is written in php and needs a machine with at least version 7 of the interpreter and mysql server installed. it must be configured to work in a web server (such as apache or nginx). the whole tintful package is available on github foot_17 and released under the apache license. ## conclusions and future work in this paper, we present tintful, an nlp annotation software that can be used both to annotate texts from scratch and to fix mistakes in nlp pipelines. differently from other similar tools, data do not need to be in a particular format: starting from plain text, the sentences are first annotated with stanford corenlp, then the user can edit the annotations and submit everything back to the server. in the future, we will extend the tool to accept more input formats, so that tintful can work with software different from corenlp, such as spacy @xcite and udpipe @xcite . in addition, we want to improve the user management part, by creating an admin interface to simplify the user creation. we also want to add the login through external services, such as google, github, facebook, and so on. for registered users, this means that they do not need to remember the password. for casual users, this allows them to review already submitted annotations. finally, we are integrating tintful with the corenlp scripts that perform the training of the models (in particular for part-of-speech, dependency parsing and named-entities recognition), to obtain a fullly automatic incremental learning pipeline.