Taxonomy Builder: a Data-driven and User-centric Tool for Streamlining Taxonomy Construction (2022)

Abstract: An existing domain taxonomy for normalizing content is often assumed when discussing approaches to information extraction, yet often in real-world scenarios there is none. When one does exist, as the information needs shift, it must be continually extended. This is a slow and tedious task, and one which does not scale well. Here we propose an interactive tool that allows a taxonomy to be built or extended rapidly and with a human in the loop to control precision. We apply insights from text summarization and information extraction to reduce the search space dramatically, then leverage modern pretrained language models to perform contextualized clustering of the remaining concepts to yield candidate nodes for the user to review. We show this allows a user to consider as many as 200 taxonomy concept candidates an hour, to quickly build or extend a taxonomy to better fit information needs.

## introduction information extraction (ie), or the extraction of structured information from free text, is a sub-field of natural language processing (nlp) that is of keen interest to those outside of the nlp community. practitioners who desire to mine information from text often set this up as a two-part task: (1) extracting the structured information from each document, and then (2) normalizing the extractions to be able to aggregate across a given corpus. for this second step, often referred to as grounding, there are several approaches, but in industry and many domain-specific applications, the standard method for grounding is linking to a relevant ontology or taxonomy @xcite @xcite , i.e., a set of domain concepts and their hierarchical organization (and additional relations in the case of an ontology). however, the task of creating or maintaining these resources is laborious and never complete; as new documents are added or the use case shifts, there are concepts that are not well covered by the current taxonomy. as a result, typically a user must manually add new, relevant concepts to the taxonomy -a process which is both time-consuming and expensive. here we present a tool that is designed to streamline this process by using automatically gleaned text summarization analytics from the corpus itself, coupled with the power and expressivity of recent contextualized embeddings, to suggest candidate concepts to a human user during an interactive session. the user can accept, reject, or manipulate the suggested concept, then determine where it belongs in relation to other existing nodes. taxonomies can be persisted for downstream use or a subsequent editing session. our contributions are: ## related work there are several large, ontological resources, e.g., wordnet @xcite , cyc (lenat, 1995), umls @xcite , and snomed ct @xcite . however, resources such as these, even in aggregate, do not have coverage of all domains; instead, they need to be perpetually extended to cover new concepts @xcite . to address the human cost of creating or maintaining taxonomies, many proposed approaches either rely on supervised data @xcite @xcite or perform unsupervised term extraction and clustering @xcite . as supervised training cannot be assumed in real-world settings, our approach is more similar to the latter; we use unsupervised methods for concept discovery, ranking, and clustering. however, we keep the human in the loop to guide the process, critical for sensitive use cases such as military events, medical emergencies, etc. @xcite similarly propose a tool that allows a user to guide a semi-automatic ontology creation process. however, our approach uses techniques from text summarization to filter candidate concepts and modern contextualized embeddings to more robustly handle multiple word senses and multi-word phrases to minimize the work that must be done by the user. ## architecture our approach for rapid data-driven taxonomy generation combines insights from several layers of text understanding to suggest highly relevant candidate concepts to a user, who decides how to use them (or not) to build the taxonomy they need. the architecture is shown in figure 1 . specifically, given a corpus of documents and optionally an existing taxonomy, we assign salience scores to each sentence based on keyword occurrence (appendix a.1). from sentences with a sufficiently high salience, we extract multi-word expressions (noun and verb phrases) as potential phrases of interest. these are ranked with respect to each other using an extension of the textrank algorithm @xcite ). the top-ranked concepts are encoded with contextualized word embeddings and clustered (appendix b). resulting clusters are presented to the user as suggested novel concepts; the phrases in the clusters can be thought of as examples of that concept. to ensure interactivity, these computationally expensive steps are done ahead of time, and the user need only load the pre-computed initial clusters. once loaded, clusters are exposed to the user through our interactive web application. the user can then decide between a series of actions (section 4); specifically, they can accept, edit, skip, or discard the node. accepted nodes are then inserted into the current working taxonomy. the user continues to work through the system's suggestions until they are satisfied with the taxonomy. ## taxonomy builder workflow once phrases are ranked and clustered, they are passed to the user-facing interface, which implements a two-part workflow: (a) cluster curation and (b) concept organization. ## usage scenarios we evaluate our tool through two use cases, both of which are motivated by darpa's world modelers program: foot_0 food insecurity and regional security. we detail both evaluations in appendix c. the take-home messages from this evaluation were: (a) the users were able to process 200 candidates for taxonomy concepts per hour using the tool; (b) the updated taxonomies yielded increased grounding confidence scores, which indicate that the new taxonomies fit the data better, and (c) the new grounding confidence scores have increased correlation with grounding correctness. ## conclusions we introduced a tool to streamline taxonomy construction and extension. using techniques from text summarization alongside the benefits of modern pretrained language models, we automatically glean cohesive taxonomy node suggestions from the corpus itself. the user interface then allows users to quickly review suggestions and decide whether to accept or reject part or all of each. through two case studies, we showed that this tool can be used to rapidly extend an existing taxonomy in a matter of hours, and further that the resulting taxonomy is a better fit to the domain of interest. the software is included in the dart framework at: george a miller. 1995. wordnet: a lexical database for english. communications of the acm, 38(11):39-41. lawrence page, sergey brin, rajeev motwani, and terry winograd. 1999. the pagerank citation ranking: bringing order to the web. technical report, stanford infolab. patrick pantel and dekang lin. 2002. discovering word senses from text. in proceedings of the eighth acm sigkdd international conference on knowledge discovery and data mining (sigkdd-02), pages 613-619. jeffrey pennington, richard socher, and christopher d. manning. 2014. glove: global vectors for word representation. in empirical methods in natural language processing (emnlp), pages 1532-1543. tammy powell, suresh srinivasan, stuart j nelson, william t hole, laura roth, and vladimir olenichev. 2002. tracking meaning over time in the umls metathesaurus. in proceedings of the amia symposium, page 622. american medical informatics association. victor sanh, lysandre debut, julien chaumond, and thomas wolf. 2020. distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. hinrich schütze, christopher d manning, and prabhakar raghavan. 2008. introduction to information retrieval, volume 39. cambridge university press cambridge. ozge sevgili, artem shelmanov, mikhail arkhipov, alexander panchenko, and chris biemann. 2020. neural entity linking: a survey of models based on deep learning. arxiv preprint arxiv:2006.00575. rebecca sharp, adarsh pyarelal, benjamin gyori, keith alcock, egoitz laparra, marco a valenzuela-escárcega, ajay nagesh, vikas yadav, john bachman, zheng tang, et al. 2019. eidos, indra, & delphi: from free text to executable causal models. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics (demonstrations). wei shen, jianyong wang, and jiawei han. 2014. entity linking with a knowledge base: issues, techniques, and solutions. ieee transactions on knowledge and data engineering, 27(2):443-460. suresh srinivasan, thomas c rindflesch, william t hole, alan r aronson, and james g mork. 2002. finding umls metathesaurus concepts in medline. in proceedings of the amia symposium, page 727. american medical informatics association. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, remi louf, morgan funtowicz, joe davison, sam shleifer, patrick von platen, clara ma, yacine jernite, julien plu, canwen xu, teven le scao, sylvain gugger, mariama drame, quentin lhoest, and alexander rush. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, online. association for computational linguistics.