Incorporating Medical Knowledge to Transformer-based Language Models for Medical Dialogue Generation (2022)

Abstract: Medical dialogue systems have the potential to assist doctors in expanding access to medical care, improving the quality of patient experiences, and lowering medical expenses. The computational methods are still in their early stages and are not ready for widespread application despite their great potential. Existing transformer-based language models have shown promising results but lack domain-specific knowledge. However, to diagnose like doctors, an automatic medical diagnosis necessitates more stringent requirements for the rationality of the dialogue in the context of relevant knowledge. In this study, we propose a new method that addresses the challenges of medical dialogue generation by incorporating medical knowledge into transformer-based language models. We present a method that leverages an external medical knowledge graph and injects triples as domain knowledge into the utterances. Automatic and human evaluation on a publicly available dataset demonstrates that incorporating medical knowledge outperforms several state-of-the-art baseline methods.

## introduction medical dialogue systems, which have gained increasing attention, aim to communicate with patients to enquire about diseases beyond their selfreported and make an automatic diagnosis @xcite @xcite . it has the potential to substantially automate the diagnostic process while also lowering the cost of gathering information from patients @xcite . in addition, preliminary diagnosis findings that are generated by a medical dialogue system may help doctors make a diagnosis more quickly. because of these advantages, researchers work on addressing sub-problems in a medical dialogue system, such as natural language understanding @xcite . however, the dialogue system for medical diagnosis, on the other hand, has specific requirements for dialogue reasoning in the context of medical knowledge. the diagnosis elicited by the dialogue system should be associated with the underlying medical condition and coherent with medical knowledge. in the absence of medical knowledge, traditional generative dialogue models frequently use neural sequence modelling @xcite and cannot be directly applied to the medical dialogue scenario. recently, transformer-based language models (lms) @xcite @xcite are fine-tuned for medical dialogue tasks. @xcite collected a meddialog dataset and fine-tuned various transformer-based lms which includes a vanilla transformer @xcite , gpt @xcite and bert-gpt @xcite for medical dialogue generation task. @xcite , in another study, presented a coviddialog dataset and then train dialogue generation models based on transformer, gpt-based model, and bart @xcite and bert-gpt for medical dialogue generation tasks. these lms are trained on huge corpus but may not provide a good representation of specific domains @xcite and need an adequate amount of task-specific data @xcite in order to establish correlations between diseases and symptoms (see figure 1 ). instead of using publicly available models, we can pre-train a model that emphasizes domain-specificity. on the other hand, pre-training is time-intensive and computationally costly, making it unavailable for most users. furthermore, while it is possible to inject domain-specific knowledge into lms during pretraining, this method of acquiring knowledge can be expensive and inefficient. for instance, pretraining data must contain many occurrences of the words "panadol" and "headache" occurring together for the model to learn that "panadol" can treat headaches. what other options do we have to make the model an expert in its field besides this one? the knowledge graph (kg), also known as an ontology, was a good solution in the early stages of research. snomed-ct @xcite , in the medical field, and hownet @xcite , in the field of chinese conception, are two examples of kgs developed as knowledge was distilled into a structured form. if kg can be incorporated into the lm, it will provide domain knowledge to the computational method, enhancing its effectiveness on domain-specific tasks while significantly lowering the expense of pre-training. to address the limitations mentioned above, this article describes a method for incorporating domain-specific external knowledge into transformer-based lms for medical dialogue generation tasks. our contributions are as follows: ## conclusion we present a method for enabling lms with kgs to achieve domain knowledge like doctors. the proposed method transforms an utterance into a knowledge-enriched utterance tree by injecting medical knowledge from kg. the embedding layer converts the utterance tree into an embedding fed to the masked self-attention of a transformer, followed by the decoder to generate the response using medical dialogue history. experimental results demonstrated that our method outperforms stateof-the-art lms trained on general data. further, through human evaluation, we conclude that generated responses are informative and doctor-like. in future, we aim to expand this work to other tasks and datasets.