Named Entity Recognition Under Domain Shift via Metric Learning for Life Sciences (2024)

Abstract: Named entity recognition is a key component of Information Extraction (IE), particularly in scientific domains such as biomedicine and chemistry, where large language models (LLMs), e.g., ChatGPT, fall short. We investigate the applicability of transfer learning for enhancing a named entity recognition model trained in the biomedical domain (the source domain) to be used in the chemical domain (the target domain). A common practice for training such a model in a few-shot learning setting is to pretrain the model on the labeled source data, and then, to finetune it on a hand-full of labeled target examples. In our experiments, we observed that such a model is prone to mislabeling the source entities, which can often appear in the text, as the target entities. To alleviate this problem, we propose a model to transfer the knowledge from the source domain to the target domain, but, at the same time, to project the source entities and target entities into separate regions of the feature space. This diminishes the risk of mislabeling the source entities as the target entities. Our model consists of two stages: 1) entity grouping in the source domain, which incorporates knowledge from annotated events to establish relations between entities, and 2) entity discrimination in the target domain, which relies on pseudo labeling and contrastive learning to enhance discrimination between the entities in the two domains. We conduct our extensive experiments across three source and three target datasets, demonstrating that our method outperforms the baselines by up to 5% absolute value. Code, data, and resources are publicly available for research purposes: https://github.com/Lhtie/Bio-Domain-Transfer .

## introduction named entity recognition is a crucial step in ie tasks. existing models have achieved remarkable performance in the general domain @xcite 1 code, data, and resources are publicly available for research purposes: https://github.com/lhtie/ bio-domain-transfer . the catechol and hydroquinone metabolites, induced apoptosis in hl60 and hbmp cells in a time-and concentration dependent manner, while the phenols, ncr181, fla873, and fla797, and the derivatives formed by oxidation of the pyrrolidine ring had no effect. the entities in red are mislabeled by a few-shot model as chemical entities. @xcite @xcite . however, in the scientific domains, e.g., medical or chemical domains, these models usually struggle due to the extremely large quantity of concepts, the wide presence of multi-token entities, and the ambiguity in detecting entity boundaries. large language models (llms) show an impressive performance on various nlp tasks such as question answering or text summarization @xcite . models such as chatgpt @xcite can achieve outstanding results given just a few training examples @xcite . @xcite recently report that the performance of these models is proportional to the number of relevant documents present in their pretraining corpus. thus, one can expect that their performance fluctuates across domains. this is particularly expected to occur across certain scientific subjects, where the data may be scarce. given the already existing challenges of the named entity recognition task in the scientific domainmentioned earlier-this factor can further exacerbate the problem. for instance, in our early fewshot learning experiments, we observed that the results of chatgpt in the chemical domain named entity recognition task are significantly worse than 1 those in the general domain. in the present work, we employ transfer learning @xcite to alleviate this problem. transfer learning methods exploit the label data from one domain (called the source domain) to minimize the prediction error in another domain (namely the target domain). we particularly focus on a realistic setting, where given a large set of labeled data from the source domain and a small set of labeled data from the target domain, the goal is to develop a model for the target domain 3 . there are more resources in the biomedical domain than in the chemical domain due to funding priorities and bionlp workshops. therefore, as a case study, we take the biomedical domain as the source and the chemical domain as the target. figure 1 shows the challenges a named entity recognition model can face in the chemical domain. the model is trained in a few-shot learning setting. thus, it is trained on labeled biomedical data, and then, further finetuned on a few labeled examples from the chemical domain. the task is to recognize only the chemical entities-ignoring other types of entities. blue indicators represent chemical entities, while red and orange indicators denote biomedical entities. the model categorizes the entities marked with blue and red as chemical entities. the first observation is that, to an inexpert human, it is difficult to perform the task because the entities are highly domain-specific. the second observation is that the model wrongfully labels some entities from the source domain as the target entities. therefore, engineers developing such a model face a dilemma: while not using the source data dramatically deteriorates performance, foot_2 simply pretraining with the source data increases the false positive rate. the third, and perhaps the most important, observation is that the examples from the source and target domains can co-occur in the same document. this problem setting contradicts the regular transfer learning setting, where the examples from the source and the target domains are fully disjoint @xcite . the latter property poses difficulties in the applicability of the traditional transfer learning methods in this setting. our core idea is to train a named entity recognition model such that it is able to project the representations of the source and target entities into separate regions of the feature space. such a model can potentially transfer knowledge from the source domain to the target domain by constructing a shared feature space between the two domains. furthermore, it reduces the similarity between the representations of the entities in the two domains, and consequently, it can potentially minimize the number of source entities that are mislabeled as target entities. to achieve this, our model consists of a pretraining stage on the source data, and a finetuning stage on the target data. in the pretraining stage, we propose two methods to enrich the feature space with auxiliary data. the auxiliary data is extracted from the event mentions that the entities participate in. additionally, during this stage, we propose to employ the multisimilarity loss term @xcite , which enables us to partition (or group) the source entities. our empirical analysis shows that constructing such a feature space during the pretraining stage facilitates our projection step during the finetuning stage. in the finetuning stage, we detect the potentially false positive entities by pseudo-labeling them. then, we aim to construct a feature space that projects the pseudo-labeled entities and the target entities into separate regions. we achieve this by employing the multi-similarity loss again. we evaluate our method across twelve use cases and show it outperforms the baselines in most experiments, with improvements of up to 5% in absolute value. we also empirically analyze our method and show that each proposed technique is individually effective. our contributions are threefold: ## proposed method figure 2 illustrates our framework. it consists of two stages, pretraining on the source data, and then, finetuning on the target data. in the source domain, we employ external knowledge to construct ## conclusions we proposed a named entity recognition task for transferring knowledge from the biomedical domain to the chemical domain. our core idea is to train a shared feature space between the two domains to facilitate the knowledge transfer, and, at the same time, to project the source and target entities into separate regions of the feature space to reduce the false negative rate. we achieve this in a few steps. we begin by enriching the source feature space with information about events, then train a named entity recognition model to cluster similar entities into groups. we then use the trained model to label the entities that may belong to the source domain, and use these entities in a multi similarity loss function to achieve our goal. our experiments across three source and three target datasets signify the effectiveness of our method.