C og KTR : A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding (2022)

Abstract: As the first step of modern natural language processing, text representation encodes discrete texts as continuous embeddings. Pre-trained language models (PLMs) have demonstrated strong ability in text representation and significantly promoted the development of natural language understanding (NLU). However, existing PLMs represent a text solely by its context, which is not enough to support knowledge-intensive NLU tasks. Knowledge is power, and fusing external knowledge explicitly into PLMs can provide knowledgeable text representations. Since previous knowledge-enhanced methods differ in many aspects, making it difficult for us to reproduce previous methods, implement new methods, and transfer between different methods. It is highly desirable to have a unified paradigm to encompass all kinds of methods in one framework. In this paper, we propose CogKTR, a knowledge-enhanced text representation toolkit for natural language understanding. According to our proposed Unified Knowledge-Enhanced Paradigm (UniKEP), CogKTR consists of four key stages, including knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. CogKTR currently supports easy-to-use knowledge acquisition interfaces, multi-source knowledge embeddings, diverse knowledge-enhanced models, and various knowledge-intensive NLU tasks. Our unified, knowledgeable and modular toolkit is publicly available at GitHub, with an online system and a short instruction video.

## introduction in modern natural language processing (nlp), texts need to be represented into a machine-readable form. many work has shown that pre-trained lan-guage models (plms) @xcite can provide powerful distributed representations for natural language texts, leading to great successes on various natural language understanding (nlu) @xcite tasks. recently, some studies @xcite @xcite have shown that specific knowledge is implicitly stored in the parameters of plms. this implicit knowledge is vague so that it is hard to dynamically update this knowledge to satisfy the needs of realworld applications @xcite . existing plms @xcite represent and understand a text solely by its context, which is insufficient to solve knowledge-intensive nlu tasks. these tasks are highly dependent on background knowledge. it is necessary to leverage external knowledge to enhance the text representations explicitly. for word sense disambiguation, synonyms, sense definitions, and other linguistic knowledge play an essential role in identifying the meaning of ambiguous words. for commonsense question answering, commonsense knowledge like structured knowledge graph (kg) triples can enhance the models' reasoning capacity. as illustrated above, knowledge-enhanced text representations are essential for nlu tasks, meanwhile, many methods @xcite @xcite have been proposed. however, previous methods differ in many aspects, especially in knowledge acquisition procedure, knowledge representation form, and knowledge fusion approach. these differences make it challenging to reproduce previous methods, implement new methods, and transfer between different methods. so we need a unified paradigm to implement various knowledge-enhanced methods in the same framework. therefore, designing the framework should consider the following key principles. first, the process of knowledge acquisition is laborious and complex, including knowledge tag-ging (e.g., named entity recognition and semantic role labeling), knowledge grounding (e.g., entity linking) and knowledge retrieving (e.g., regular expression matching and sparql query). a good framework should let users pay more attention to the details in the models rather than tedious data processing. second, different knowledge embeddings vary in knowledge sources (e.g., wikidata @xcite and conceptnet @xcite ) and knowledge representation algorithms (e.g., transe @xcite and wikipedia2vec @xcite ). to make rigorous comparisons between them, it is highly desirable to have a toolkit that provides built-in knowledge embeddings. third, although a lot of knowledge fusion approaches have been proposed, there is still a lack of a comprehensive framework to encompass them. such a framework should provide knowledgeable text representations which can be directly used in numerous downstream tasks. to this end, we propose , a knowledge-enhanced text representation toolkit for natural language understanding. cogktr is built on the unified knowledge-enhanced paradigm (unikep), which can be formalized in four stages, including knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. first, knowledge acquisition aims to identify structured information from unstructured texts, then ground them in knowledge sources. then, knowledge representation can transform knowledge from discrete form to continuous form. next, knowledge injection, as the most critical stage, combines raw texts and external knowledge for knowledgeable text representation. in the end, knowledge application verifies the effectiveness of knowledge-enhanced methods in downstream tasks. in detail, cogktr has the following functions. first, our toolkit provides user-friendly knowledge acquisition interfaces. users can use our toolkit to enhance the given texts with one click. and we also implement plenty of knowledge-enhanced methods so researchers can quickly reproduce these models. moreover, cogktr supports many built-in nlu tasks to evaluate the effectiveness of knowledgeenhanced methods. in our paradigm, users can easily conduct their research via a pipeline. besides the toolkit, we also release an online cogktr demo to show the process of knowledge acquisition and the effect of knowledge enhancement. in summary, the main features and contributions are as follows: as mentioned above, it is vital to propose a paradigm that can formalize the knowledgeenhanced process. as shown in figure 1 , our proposed unified knowledge-enhanced paradigm (unikep) consists of four key stages: knowledge acquisition, knowledge representation, knowledge injection and knowledge application. below are the detailed descriptions of the four stages. ## system design and architecture according to the paradigm mentioned above, we divide cogktr architecture into four modules. for knowledge acquisition and representation, cogktr modularizes them as the enhancer module. to implement knowledge injection and application, we build the model module to integrate knowledge into models. considering that the development process is time-consuming, we also design two basic modules, namely data module and core module, to accelerate the data processing procedure and improve training efficiency. an overview of cogktr architecture is shown in figure 2 . in the following, we will introduce these four modules. ## system usage in this section, we will give detailed guidelines on how to use cogktr toolkit and online demo. ## evaluation cogktr aims to support various nlu tasks under a unified paradigm. to demonstrate the effectiveness of knowledge-enhanced methods, we implement several baselines and evaluate them on the corresponding tasks. the evaluation tasks include commonsenseqa @xcite and openbookqa @xcite for commonsense reasoning; lama @xcite for knowledge probing; squad2.0 @xcite for reading comprehension; qnli and sst-b @xcite ) for text entailment; conll2003 @xcite for sequence labeling; sst-2 and sst-5 @xcite for sentiment analysis; semcor @xcite and @xcite for word sense disambiguation. reader and processor classes of these datasets have already been integrated into cogktr. the experimental results are available at our github foot_0 . ## conclusion and future work in this paper, we propose cogktr, a knowledgeenhanced text representation toolkit for natural language understanding. cogktr is built on our unified knowledge-enhanced paradigm, which is composed of four stages: knowledge acquisition, knowledge representation, knowledge injection, and knowledge application. in cogktr, we provide easy-to-use knowledge acquisition interface, off-the-shelf knowledge embeddings, builtin knowledge-enhanced models, and knowledgeintensive nlu tasks. besides the toolkit, we also release an online demo system. in the future, more knowledge sources, benchmark datasets, and models will be incorporated into cogktr.