Reducing Knowledge Noise for Improved Semantic Analysis in Biomedical Natural Language Processing Applications (2023)

Abstract: Graph-based techniques have gained traction for representing and analyzing data in various natural language processing (NLP) tasks. Knowledge graph-based language representation models have shown promising results in leveraging domain-specific knowledge for NLP tasks, particularly in the biomedical NLP field. However, such models have limitations, including knowledge noise and neglect of contextual relationships, leading to potential semantic errors and reduced accuracy. To address these issues, this paper proposes two novel methods. The first method combines knowledge graph-based language model with nearest-neighbor models to incorporate semantic and category information from neighboring instances. The second method involves integrating knowledge graph-based language model with graph neural networks (GNNs) to leverage feature information from neighboring nodes in the graph. Experiments on relation extraction (RE) and classification tasks in English and Chinese language datasets demonstrate significant performance improvements with both methods, highlighting their potential for enhancing the performance of language models and improving NLP applications in the biomedical domain.

## introduction language models (lm) have become increasingly popular in a wide range of applications @xcite . lms have also shown great promise in assisting clinical decisionmaking. lms for clinical decision-making are based on massive amounts of text data, including medical literature, clinical notes, and patient records @xcite . by analyzing medical records and identifying patterns, language models can help doctors identify potential health risks and recommend appropriate treatment options @xcite . additionally, lms can be used to assist with drug discovery by analyzing vast amounts of scientific literature and identifying potential drug targets @xcite . lms can also improve patient outcomes by assisting with patient education. lms can analyze patient records and suggest personalized educational resources, such as videos or articles, to help patients better understand their conditions and treatments. likewise, language models can help clinicians communicate more effectively with patients by providing real-time translation services for patients who speak different languages. however, these language models are not without their shortcomings. for example, the cross-entropy loss used in fine-tuning language models can lead to poor generalization of performance, as pointed out by @xcite . similarly, lms can be prone to overfitting as the predictions are made through linear classifiers added directly to the top of pretrained lms @xcite . moreover, they may neglect the relationship between textual contexts, impacting performance. given the importance of the medical field, errors in language models can have significant consequences. thus, efforts to improve the performance of language models are being made. a possible scope for improvement can be using knowledge from nearest neighbors. @xcite employed a k-nearest neighbor (knn) approach to enhance the performance of lms. the knn-lm approach uses nearest-neighbor models to improve language modeling by explicitly memorizing rare patterns and improving performance, indicating that the representation learning problem is easier than the prediction problem. similarly, @xcite approached an open-domain question-answering problem with knn-bert, a combination of bert's prediction for a given question with a knn search. the authors demonstrated through their experimental results and evaluation that incorporating knn into the bert-based question-answering model was effective in retriev-ing accurate factual information. this model particularly excelled in providing answers to less frequent and difficult questions and was able to handle even recent events that were not included in the training data of the bert model. unlike other studies that use knn to generate augmented samples based on pre-trained language models, @xcite utilized a knn classifier as the decision maker. it was demonstrated that incorporating knns with traditional fine-tuning of bert-like models can significantly improve accuracy in both rich-source and few-shot settings and improve robustness against adversarial attacks. nearest neighbors have shown immense success in improving the interpretability of models as well @xcite . the use of knn in explaining the model behavior in language models is a topic that has been attracting the research community @xcite . interpretable machine learning is key to building trustworthiness in ai systems used in healthcare. interpretability can provide clinicians and patients with insights into the reasoning behind a particular decision made by a machine learning model, making it easier to understand and trust. this can help to improve patient outcomes and increase the acceptance of ai systems in healthcare. additionally, interpretability can help identify and address biases in machine learning models. while lms like bert @xcite have limited ability to capture global information, graph neural networks (gnns) have proved to be better at it @xcite . language models like bert are good at capturing contextual information. in order to make the best out of a language model, it needs to have global information as well as proper contextual information. utilizing the strengths of both graph neural networks and language models, @xcite proposed vocabulary graph convolutional network (vgcn)-bert. the motivation for combining vgcn and bert is to allow them to collectively build an optimal representation while performing tasks such as classification. vgcn models the relationships between words in a text by representing them as nodes in a graph, while bert is a state-of-the-art language model that can effectively capture the contextual information of text data. this enhanced representation led to improved performance in text classification tasks, as demonstrated by the results of the study. global knowledge is very important in the medical domain because medical data is inherently com-plex and interconnected. in order to effectively analyze medical data, it is essential to capture the relationships between different medical concepts and understand their context within the larger medical knowledge network @xcite . for example, in the field of medical diagnosis, a patient's symptoms and medical history need to be considered in the context of the larger medical knowledge network to accurately diagnose their condition @xcite . by leveraging global knowledge, medical professionals can better understand the relationships between different medical concepts and make more well-informed decisions regarding patient care. furthermore, medical research often involves analyzing large datasets containing vast medical information @xcite . by utilizing global knowledge, researchers can better identify patterns and relationships within the data, leading to more accurate and insightful findings. thus, there is a pressing need to build representation comprising contextual and global information. pre-trained language models are mostly generic in nature and lack domain-specific knowledge @xcite . this can be a problem for tasks that require access to domain-specific knowledge, such as medical text classification and medical relation extraction @xcite . to mitigate this problem, @xcite proposed k-bert, a knowledge-enabled language representation model that addresses this problem by incorporating knowledge from a knowledge graph into its language representation. this knowledge incorporation allows k-bert to perform better on tasks that require access to domain-specific knowledge. k-bert allows the triplets to be injected into sentences as knowledge, making it useful for domain-specific tasks. thus, leveraging k-bert for domain knowledge and knn and gnn for global knowledge, we present methods for integrating (i) k-bert with gnn and (ii) k-bert with knn. our contributions are as follows: ## methodology as discussed above, our aim is to utilize the information of the neighbors in the dataset to help improve the performance of the language model in various downstream tasks. we use k-bert as a language model because of its flexibility to adapt to domain knowledge. ## results and discussion we use accuracy as a measure to evaluate the performance of our algorithms for both downstream tasks (relation extraction and text classification). from the results show that the base model k-bert improved performance by integrating supplementary information. the classification and relation extraction tasks are important in the medical domain. the ability of our proposed methodology to get improved performance can also be adapted to other tasks in the medical domain. ## conclusion in this paper, we propose a methodology for integrating additional information, such as neighboring nodes and instances, to improve the performance of a language model in the medical domain. the additional information also provides models with the ability to learn beyond contextual information. the methodology proposed in our work is generic and can be adapted to multiple tasks and domains. the work can be extended to solving other critical problems in the medical domain like report generation, medical dialogue generation, etc. it would also be interesting to integrate both nearest neighbor and graph information into the language model and evaluate the model performance. another important future direction can be exploring how adding information in our framework contributes to the explainability of the model. overall, our proposed methodology shows promising results and has the potential to enhance the performance and interpretability of language models in various domains.