ITAKE : Interactive Unstructured Text Annotation and Knowledge Extraction System with LLM s and M odel O ps (2024)

Abstract: Extracting structured knowledge from unstructured text data has a wide range of application prospects, and a pervasive trend is to develop text annotation tools to help extraction. However, they often encounter issues such as single scenario usage, lack of effective human-machine collaboration, insufficient model supervision, and suboptimal utilization of Large Language Models (LLMs). We introduces an interactive unstructured text annotation and knowledge extraction system that synergistically integrates LLMs and ModelOps to alleviate these issues. The system leverages LLMs for enhanced performance in low-resource contexts, employs a ModelOps platform to monitor models throughout their lifecycle, and amalgamates interactive annotation methods with online machine learning and active learning. The demo video and website are now publicly available.

## introduction unstructured text data contains a large amount of valuable knowledge, from which structured knowledge such as entities, relationships and attributes can be extracted to help the construction of knowledge graphs, and can also support downstream tasks, which has a wide range of application prospects. however, real-world text exists multilanguage, a mixture of short and long text, and complex terminological references, etc. unstructured text knowledge extraction methods based solely on machine intelligence are far from meeting the needs of actual business. for example, on the publicly available datasets wnut-17 @xcite , docred @xcite , the highest f1score for named entity recognition and relation extraction are only 60.45% @xcite and 67.53% @xcite . besides, the cost of relying only on human annotation is very expensive. currently, there are many open-source text annotation tools dedicated to solving the above challenges, but they have some problems resulting in a not-so-perfect process. first of all, some of the tools are used in a single scenario, targeting a fixed application domain, ontology and language (challenge c1). for example, medcat @xcite only supports english and is limited to medical data annotation. secondly, most of the tools lack the organic combination of human and machine, resulting in too much user participation to increase the cost @xcite or lack of user feedback leading to poor modeling accuracy @xcite especially in low resource situation (challenge c2). in addition, even if models are involved in the extraction process of some tools @xcite , there is a lack of model supervision and state analysis in the process of using them, and the reuse support capability for models and datasets is weak, which prevents the rapid development and deployment of models for specific domain requirements (challenge c3). finally, after the popularity of llms @xcite @xcite , many extraction tools intergrated llms to assist extraction @xcite . however, although llms are more effective than traditional knowledge extraction state-of-the-art model (hereinafter referred to as the extraction model) in low resource situation because of their strong generalization ability, the improvement effect of llms is not obvious after the increase of training data, and when they reaches a certain threshold, their effect is far worse than that of well-trained extraction model @xcite . at the same time, llms are conversational generative models, which lead to slower inference speed and are difficult to meet the real-time demand (challenge c4). aiming at the above problem, we developed itake (an interactive unstructured text annotation and knowledge extraction system) that integrates llms and modelops @xcite . specifically, (1) addressing challenge c1 and challenge c3, we adopt modelops platform to integrate different models and monitor whole lifecycle of them. (2) addressing challenge c2, we combine the interactive annotation methods for online machine learning @xcite and active learning @xcite . (3) addressing challenge c4, we integrate llms under low resources situation and use extraction models for well-labeled situation. ## conclusion and future work we developed itake, a knowledge extraction system that combines llms and modelops. its usability and cost reduction have been fully demonstrated through real case study. in the future, we hope to add events and multi-modal extraction, and add the llms self-feedback mechanism, so as to reduce human cost more effectively.