Q uick G raph: A Rapid Annotation Tool for Knowledge Graph Extraction from Technical Text (2022)

Abstract: Acquiring high-quality annotated corpora for complex multi-task information extraction (MT-IE) is an arduous and costly process for human-annotators. Adoption of unsupervised techniques for automated annotation have thus become popular. However, these techniques rely heavily on dictionaries, gazetteers, and knowledge bases. While such resources are abundant for general domains, they are scarce for specialised technical domains. To tackle this challenge, we present QuickGraph, the first collaborative MT-IE annotation tool built with indirect weak supervision and clustering to maximise annotator productivity. QuickGraph’s main contribution is a set of novel features that enable knowledge graph extraction through rapid and consistent complex multi-task entity and relation annotation. In this paper, we discuss these key features and qualitatively compare QuickGraph to existing annotation tools.

## introduction hand-labelling is still the most reliable means to obtain quality training data to support deep learning applications; however, it is timeconsuming and resource-intensive @xcite . unsupervised approaches such as weak/distant supervision @xcite and data programming @xcite thus are usually attractive alternatives or starting points to human-annotation. leveraging unsupervised techniques, however, is predicated on the availability of relevant external resources such as semantically aligned knowledge bases, and a priori knowledge of phenomena/concepts in the corpus of interest. in general domains, these are widely available, e.g. yago @xcite , freebase @xcite , wikidata @xcite , dbpedia @xcite , whereas they remain scarce for emerging and specialised domains, such as engineering, industrial, medical, biological, law enforcement @xcite which we refer to as technical domains. due to their close real-world applications, technical domains are often more impactful and likely to have formal ontologies of engineered knowledge. consequently, human-annotation remains critical and essential for obtaining quality training data for technical information extraction and instance population. numerous annotation tools exist, supporting many nlp tasks @xcite . however, few tools support large-scale, hierarchical, multi-task, multi-label, entity and relation annotation that is required for translating nlp research to real-world industry applications in technical domains @xcite @xcite @xcite @xcite . moreover, these tools lack features to optimise annotator productivity and return-on-time-invested. such features are particularly essential for technical domains, as annotators are frequently subject matter experts, who are typically time-poor and costly. weak/distant supervision @xcite is a powerful paradigm for large-scale (potentially lowquality) automatic annotation. surprisingly, the integration of this paradigm into annotation tools to accelerate labelled sample acquisition for deep learning applications remains unexplored. to fill these gaps, we introduce quickgraph, the first collaborative annotation tool for multitask ie that is designed to be: ## related work many nlp annotation tools have been developed in recent years @xcite , however few support entity and relation annotation as a single integrated task, nor do they contain purposely designed features to enhance annotator productivity. here we discuss the most notable entity and relation annotation tools. historically, brat @xcite has been the most popular but routinely receives criticism for it's antiquated technology and setup difficulty @xcite . similarly, webanno (yimam et al., 2013), inception (klie et al., 2018) and tex-tannotator (abrami et al., 2019) are featurerich and multi-purpose tools, but are challenging to use. slate (kummerfeld, 2019) is a command-line-based tool, but lacks multitask functionality and is restricted to technical end users because of its command-line design. teamtat (islamaj et al., 2020) is a powerful tool, but is oriented for small-batch complex annotation of large documents such as scholarly articles. redcoat (stewart et al., 2019) is an feature-rich entity typing tool and has been demonstrated to support mt-ie (stewart and liu, 2020), but is not purpose-built for relation annotation. salkg (tang et al., 2020) is a unique knowledge-graph annotation tool, yet lacks features for collaborative annotation and adjudication. despite this, each of the aforementioned tools contain powerful elements, but universally lack features to support rapid largescale, complex, annotation. recent tools enhance annotator productivity using active and proactive learning, including aplenty (nghiem and ananiadou, 2018), paladin @xcite , fitannotator @xcite and activeanno @xcite . inadequately, these tools cannot perform multi-task entity and relation annotation. moreover, their performance using large ontologies remains unproven @xcite . reliance on 2 when both entity and relation annotation is performed. ## 3 person there was nothing so very remarkable in that; nor did alice think it so very much out of the way to hear the rabbit say to itself, "oh dear! oh dear! i shall be late!" (when she thought it over afterwards, it occurred to her that she ought to have wondered at this, but at the time it all seemed quite natural); but when the rabbit actually took a watch out of its waistcoat-pocket, and looked at it, and then hurried on, alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with either a waistcoat-pocket, or a watch to take out of it, and burning with curiosity, she ran across the field after it, and fortunately was just in time to see it pop down a large rabbit-hole under the hedge. either the well was very deep, or she fell very slowly, for she had plenty of time as she went down to look about her and to wonder what was going to happen next. first, she tried to look down and make out what she was coming to, but it was too dark to see anything; then she looked at the sides of the well, and noticed that they were filled with cupboards and book-shelves; here and there she saw maps and pictures hung upon pegs. she took down a jar from one of the shelves as she passed; it was labelled "orange marmalade", but to her great disappointment it was empty: she did not like to drop the jar for fear of killing somebody underneath, so managed to put it into one of the cupboards as she fell past it. down, down, down. would the fall never come to an end? "i wonder how many miles i've fallen by this time?" she said aloud. "i must be getting somewhere near the centre of the earth. let me see: that would be four thousand miles down, i think-" (for, you see, alice ## system architecture quickgraph is a multi-user tool built using the modern full-stack framework mern foot_6 , python and docker. our tool consists of four containerised components (figure 5 ): i) web client, ii) nosql database, iii) server, and iv) cluster server. using docker, our tool can be built and ready to annotate in minutes foot_7 . tool annotation type multilabel preannotation autoannotation relation constraints annotation propagation document clustering quickgraph annotation type: 75% of the reviewed tools support entity annotation, with most also allowing relation annotation. only redcoat (and quick-graph) permit hierarchical entity annotation. multi-label: 75% of the reviewed tools support multi-label annotation. of these, activeanno and paladin permit multi-labels, but are restricted to document classification tasks. pre-annotation: 50% of the reviewed tools allow pre-annotation of corpora prior to manuallabelling. these tools are limited to entities, while quickgraph also supports relations through triples. ## conclusion and future work we introduced quickgraph, a collaborative annotation tool for multi-task information extraction that accelerates annotator productivity. distinguishing features of quickgraph are its support for diverse information extraction tasks through hierarchical entity and open/closed relation annotation, annotation propagation, and semantic clustering. whilst quickgraph is ready to use, there are features still under development, including: i) expanding available semantic embedding and clustering options, ii) improving annotation propagation processes, iii) relaxing interannotator agreement metrics, and iv) adding support for cross-document annotation. in proceedings of the 51st annual meeting of the association for computational linguistics: system demonstrations, pages 1-6, sofia, bulgaria. association for computational linguistics.