Cross-domain NER with Generated Task-Oriented Knowledge: An Empirical Study from Information Density Perspective (2024)

Abstract: Cross-domain Named Entity Recognition (CDNER) is crucial for Knowledge Graph (KG) construction and natural language processing (NLP), enabling learning from source to target domains with limited data. Previous studies often rely on manually collected entity-relevant sentences from the web or attempt to bridge the gap between tokens and entity labels across domains. These approaches are time-consuming and inefficient, as these data are often weakly correlated with the target task and require extensive pre-training.To address these issues, we propose automatically generating task-oriented knowledge (GTOK) using large language models (LLMs), focusing on the reasoning process of entity extraction. Then, we employ task-oriented pre-training (TOPT) to facilitate domain adaptation. Additionally, current cross-domain NER methods often lack explicit explanations for their effectiveness. Therefore, we introduce the concept of information density to better evaluate the model’s effectiveness before performing entity recognition.We conduct systematic experiments and analyses to demonstrate the effectiveness of our proposed approach and the validity of using information density for model evaluation.

## introduction cross-domain named entity recognition (cd-ner) involves identifying and classifying named entities (e.g., people, organizations, locations) in text from different domains. traditional ner systems @xcite , typically trained on domain-specific data, often perform poorly on text from other domains @xcite . while, cdner ad-target text: to allow for multiple entities , a separate hinge loss is computed for each capsule. entity and type: (hinge loss, metrics) the hinge loss is used for "maximum-margin" classification, most notably for support vector machines (svms). the term max(0, 1 -y , f(x)) is the hinge loss used by support vector machines; the quadratically smoothed hinge loss is a generalization of mathl. the hinge loss is a measure of the difference between the predicted output of a capsule and the actual output. by computing a separate hinge loss for each capsule, the model can learn to distinguish between different entities and improve its accuracy. figure 1 : dapt corpus based on retrieval denotes the manual collected knowledge related to target domain entity from web @xcite . while, our gtok corpus based on generation is automatically generated from a fundamental large language model (llm), which is strongly related to the target domain entity and the recognition process. previous cdner studies mainly adopt two paradigms: 1) capturing domain differences @xcite @xcite , such as linking tokens to domain-specific entity types to enhance generalization @xcite . despite their success, these methods have limitations: 1) manual collection: collecting large-scale external knowledge is time-consuming and laborintensive. automating this process could save considerable time. 2) relevance: much of the collected entity knowledge is only relevant to the entity but not closely related to the cdner task. for example, figure 1 shows that sentences about "hinge loss" in the dapt corpus are mere definitions, irrelevant to the ner task, which requires identifying all possible entity spans and types in the text. the automatically extracted logical reasoning processes of ner, as shown in the gtok corpus, could more effectively help models generalize. 3) validation strategies: current works mostly use post-analysis methods like ner performance comparison implicitly to validate their approaches. employing quantitative pre-analysis methods, such as estimating the impact of external knowledge explicitly before the ner task, would mark significant progress. to tackle these issues, we propose a novel generative framework with ner task-oriented pretraining on generated knowledge, namely topt. our framework comprises generating task-oriented knowledge, task-oriented pre-training with masked span modeling, fine-tuning the ner model, and inferring on the target domain. inspired by the strong emergence and reasoning capabilities of large language models (llms, 7b level), we first use an llm to generate a small-scale task-oriented knowledge corpus (gtok corpus), illustrating the entity recognition reasoning flow, as in figure 1 . next, we employ masked span language modeling (mslm) to pre-train the ner model on the gtok corpus, guiding the model to understand the entity recognition task. we then fine-tune the model with labeled samples from both source and target domains. finally, the fine-tuned model infers entity spans and labels in the target test set. note that information density is introduced to evaluate the model potential ability with external knowledge to perform cdner. in summary, our contributions are: ## methodology in this section, we first present the detailed modules of our topt: task-oriented knowledge generation, masked span modeling for pre-training, text-to-text generation for cdner. then, we introduce how to employ the uid to explain why our approach with generative task-oriented knowledge (gtok) outperforms sota with other manual large-scale corpus. problem definition. given a n-token sentence @xmath0, • • • , x @xmath1, • • • , t k >, the object of ner task is to extract all entities e @xmath2, where e @xmath3:end , t) denotes the i-th entity of x and @xmath4. x start:end refers to a continues word @xmath5, • • • , x @xmath6, where start and end refers to the entity boundary indexes respectively. given dataset d of the source domain and dataset t of the target domain, the object of the cross-domain ner task is to acquire targetrelated knowledge from d to enhance model's performance on t . to be accordant with real-world applications, d is supposed to contain a single source as well as a combined multiple sources. ## conclusion we propose a novel approach for cross-domain ner tasks, namely topt. we first apply llms to automatically generate a task-oriented knowledge corpus and pre-train the model on the generated corpus to enhance domain-adaptation and ner task sensitivity, thus, improving the model's performance on cross-domain ner. employing these comprehensive experiments, our approach achieves a better performance than previous sota crossdomain ner approaches. besides, we reformulate the ner task as "text-to-text" generation, which avoids unique settings for separated domains and makes real-world applications easier. moreover, we introduce uniform information density theory to analyze the effectiveness of our approach and explain why the generated corpus is better. in the future, we will attempt to mine more taskoriented knowledge for cdner, and investigate more domain to verify our approach. moreover, we plan to apply our task-oriented pre-training strategies into other areas to motivate their further development in nlp. algorithm 1 transfer from d to t input: domain d, t (contain sentence with labels ( predictions ŷj = argmax(o j ) 6: update corresponding parameter θ by minimizing: to this end, we first theoretically present the rationality. in shannon information theory, language can be regarded as a communication system and each linguistic unit of the language carries several information. the amount of information can be quantified with surprisal (degree of surprise, @xcite ). suppose a linguistic signal: that is, the smaller the probability of occurrence of a linguistic unit, the more information it contains. we can plainly assume that the cognitive load of the entire linguistic signal u derives from the sum of each linguistic unit in it: to simplify the calculations, we leverage bi-gram language model for approximate uid: in addition to uid hypothesis, shannon information entropy is also a common method to quantify the information of texts. the elementary definition of information entropy h is: based on the above rationale, we can conclude that if information density of one corpus for pretraining distributes more uniformly than that of another corpus, the former corpus involves more effective information for subsequent ner task @xcite . then, we empirically present the rationality of our hypothesis through corresponding results as section 4.4, also including the calculation of information entropy in different corpus for domain adaptation.