MIC : A Multi-task Interactive Curation Tool (2022)

Abstract: This paper introduces MIC, a Multi-task Interactive Curation tool, a human-machine collaborative curation tool for multiple NLP tasks. The tool aims to borrow recent advances in literature to solve pain-points in real NLP tasks. Firstly, it supports multiple projects with multiple users which enables collaborative annotations. Secondly, MIC allows easy integration of pre-trained models, rules, and dictionaries to auto label the text and speed up the labeling process. Thirdly, MIC supports annotation at different scales (span of characters and words, tokens and lines, or document) and different types (free text, sentence labels, entity labels, and relationship triplets) with easy GUI operations.

## introduction with the recent advances in many frontiers, highquality annotations are essential to the success of nlp applications. numerous organizations have accumulated vast amounts of unlabeled text data that they want to utilize in nlp applications. however, for many of these tasks (text summarization, relation extraction, named-entity recognition), aquiring labels can be very costly and susceptible to error. furthermore, domain adaptation @xcite , which is the common approach of fine-tuning gigantic domain agnostic nlp models on a small amount of domain-specific labeled data, commonly has difficulty on new emerging / specific domains that lack similar labeled datasets and still requires annotations from scratch. meanwhile, adoption of accelerated ml solutions have shown to reduce the workload and budget required for manual labeling. example techniques include active learning, weak supervision, data augmentation, and many others. to concur the timeconsuming, labor-intensive, and expensive annotation challenges, recent trends in annotation tool de- * all authors contributed equally velopment @xcite focus on cost-effective and human-machine collaborative mechanisms, which leverage the processing power of state-of-the-art models pre-trained on large corpora and high-accuracy human intelligence on rare ambiguous incidents. please visit www.textmic.com using username ds and password demouser123 to visit the demo system. a screencast video is at https://youtu.be/ phxt5k_mlvw . the github repo of mic is at https: //github.com/cyberyu/textmic. ## related work in the past decade, there were about 30 popular annotation tools published. among them, there are well-known tools like brat @xcite , which supports a wide variety of nlp tasks, including entity recognition, event extraction, and pos (part-of-speech) tagging. gate teamware @xcite is both a desktop application and a web tool that focuses on user management and supports multi-user roles. yedda @xcite is a recent tool built on python that offers auto-labeling via machine learning and provides both command line and web-based interfaces. santo @xcite , which is designed primarily for slot-filling tasks, enables the formation of relational structures from an ontology. it also visualizes the annotations of every user at once to help project owners monitor the quality of annotations. talen @xcite specialises in the annotation of rare entities. easytree @xcite is specifically designed for the annotation of dependency trees and is integrated with the amazon mechanical turk crowdsourcing platform. alpacatag @xcite and lean-life @xcite ) leverage machine learning models, active learning, and weak supervision, respectively, to provide annotation recommendations to reduce annotation costs. system annotation type adjud -ication intelligent interactive annotation external dependencies programming language mic classify, link yes pre-trained models, crowd -sourcing weak-supervision mongodb, postgresql python,django,vue.js redcoat (stewart et al., 2019) classify, link yes hierarchical entities mongodb javascript, python santo @xcite link -ontology-driven apache,mysql php talen @xcite classify yes entity propagation, internet search -unknown easytree @xcite classify,link yes crowd-sourcing amazon turk java servlet, javascript alpacatag @xcite classify yes recommendation, crowd-sourcing -python, django lean-life @xcite classify, link yes crowd-sourcing, weak-supervision -python slate @xcite classify, link yes terminal-based annotation -python brat @xcite classify, link yes -apache python,javascript gate @xcite classify, link yes --java yedda @xcite classify yes --python wat-sl @xcite classify yes -apache java sawt @xcite classify ---python, php graphanno @xcite classify, link ---ruby cora @xcite classify ---php,javascript webanno @xcite classify, link ---java anafora @xcite classify, link yes --python analec @xcite classify, link --java labelstudio labelstud.io classify, link yes text, images, video, audio commercial react, mst, python prodigy https://prodi.gy/ classify, link yes active learning in annotation commercial python tagtog tagtog.com classify, link yes support annotations in pdf commercial java, python lighttag lighttag.io classify, link yes support inter-annotator, project management commercial python ## system description and key features rather than specifying a task such as named entity recognition or sentence labeling, mic is designed to flexibly support any annotations that can be formulated as one of three annotation types: sentence-level labeling, word/phrase-level labeling, or entity-relation-entity triplet labeling. these can be applied to items that are single documents, lines, phrases, tokens, or token combinations. furthermore, mic is able to manage annotation tasks associated with interactions among various annotation types. for example, one can restrict the annotation task on named entities among sentences having positive sentiment score, or limit the findings of re-lationship triplets that contain the entity olympics with the entity type as event. one major novelty of mic is its support of human-machine interactive annotation via a flexible user interface. for each annotation task, the human annotator can be empowered by a set of pretrained ml models to quickly generate machineannotated labels. these pre-trained models are either built-in models from popular data science packages or novel open-source implementations from github. pre-trained models can be configured by administrators via the mic backend interface and each annotation project can associate with multiple models. pre-trained models are grouped in three categories: sentence labeling, ner labeling and relationship labeling. all pre-trained models are hosted as restful api endpoints, and for each annotation category the input/output parameters of all endpoints are required to follow the same standards so various models can be interchanged easily. though machine generated labels cannot be directly considered as ground-truth annotations, the advantages here are two fold. first, instead of requiring the annotator to write everything from scratch, they start from the most likely machinegenerated outputs, and mic supports quick and intuitive editing operations. this helps the human annotator to spend effort effectively and focus on correcting difficult examples. second, if the human annotator load and apply multiple machinegenerated outputs on the same text, those machine outputs can be exported as noisy labels to train a consensus model using weak supervision. finally, mic has been designed to manage annotation tasks for multiple projects and multiple users. multi-project setting allows mic to be configured flexibly to support diverse annotation tasks. for each project, new textual data can be loaded to seek curations at sentence level, named entity level or relationship level, or any combinations of them. from mic's backend user interface, one can associate a number of relevant ml models/dictionaries/rules to a project to allow quick generation of machine labels. the textual data can be fully unlabeled, partially labeled, or integrated with ground truth labels. in cases where the data is partially or fully labeled with ground truths, the administrator can setup a built-in validation process to monitor performance of annotations as the task continues. annotation performance can be evaluated by comparing ground truth labels with human/machine generated labels, or comparing ground truth labels against consensus labels learned by weak supervision. the multi-user setting allows mic to involve multiple parties in the annotation pipeline. each project can allow users with different roles such as administrator, curator, data scientist, reviewer, etc., and their operational accesses are categorized and limited by roles to ensure the integrity of the annotation task. in conclusion, the main novelties of mic are (1) extendable framework to integrate customized annotation models; (2) multi-project and multi-user management; (3) support of multi-layer annotations from sentences to entities and relations; (4) state-of-art user interface design for annotations. ## evaluation of annotation efficiency and accuracy we conducted a benchmark study to investigate the efficiency and accuracy of mic in real annotations. three different data sets were used for task preparation: (1) connl2003 @xcite ; (2) nyt open relation extraction benchmark @xcite ; (3) proprietary fintech customer support call transcripts. the first and second data sets are publicly accessible and widely used as ner and relation extraction benchmarks. the third data is a proprietary data set and the goal is to obtain three levels of annotations. the first level is sentence tagging: the annotator needs to extract the main customer complaint sentences from the call transcript if the complaint is related to buy/sell financial product (stocks/funds), denoted as a buy/sell relevant sentence. all other sentences are irrelevant. the second level is ner: among the relevant sentences, tag any mentioned financial products (stock/fund tickers, bank names, and others) as named entities. the third level is to annotate any unary relationship, if related to buy or sell, of annotated entities if mentioned in the same sentence. for example, sell apple stock, buy nvda, exchange money market funds, and so on. we compared four annotation tools, including free version of prodigy, yedda @xcite , gate @xcite master internship program were trained to perform annotations. each annotator spent about 2 hours on each tool using labeled data to get familiar with a tool's specific annotation mechanism. then, a random sample of 50-sentence corpora from data sets ( 1 ) and ( 2 ), and 20 random transcriptions of data set (3) were assigned for annotation. for each annotator, sentences/transcriptions were stratified samplings by different annotation tools, so there was no occurrence of seen sentences across different tools. in this setting, each task had four samples, assigned to four annotators in parallel and all annotated once using the same tool. because data in all tasks comes with ground truth labels, we measure annotation performance in this step through evaluating the micro-entity precision, recall, and then calculate the f1 scores of the annotated labels against the ground truth labels. the average annotation time and f1-scores of four annotators spent on this task-tool combination were recorded and compared in we compared mic in two configurations: (1) mic-nm only allowed manual annotations, so no pre-trained model was used. (2) mic-m included pre-trained annotation models so annotators could confirm final labels using auto-annotations. in the mic-m setting, mic included four ner models (flair, finbert hmm, en core web md, snips), one mama re model @xcite , and a proprietary intent classification model to classify sentences. the average annotation time of four annotators, and the f1-score of their annotated results evaluated by ground truths, are reported in ## scalability and deployment for budget reasons, the demo system of mic hosted at www.textmic.com is deployed on a single aws t3.xlarge instance. however, restful apis can be distributed to different physical instances for better performance and richer model capacity. thus mic could host a wide range of large-scale pre-trained models in its library and allow easy adaption of relevant models in specific annotation tasks. ## future development our roadmap to enhance mic for the future lies ahead in several directions. we are interested in connecting mic to advanced processes cloudbased apis such as zero-shot learning, few-shot learning, and textual entailment models to provide annotators access to more sota nlp models. additionally, we'll implement automated training pipelines for several weak supervision algorithms including @xcite @xcite to allow automatic denoising of conflicting human or machine labels.