Clinical Text Classification to SNOMED CT Codes Using Transformers Trained on Linked Open Medical Ontologies (2023)

Abstract: We present an approach for medical text coding with SNOMED CT. Our approach uses publicly available linked open data from terminologies and ontologies as training data for the algorithms. We claim that even small training corpora made of short text snippets can be used to train models for the given task. We propose a method based on transformers enhanced with clustering and filtering of the candidates. Further, we adopt a classical machine learning approach - support vector classification (SVC) using transformer embeddings. The resulting approach proves to be more accurate than the predictions given by Large Language Models. We evaluate on a dataset generated from linked open data for SNOMED codes related to morphology and topography for four use cases. Our transformers-based approach achieves an F1-score of 0.82 for morphology and 0.99 for topography codes. Further, we validate the applicability of our approach in a clinical context using labelled real clinical data that are not used for model training.

## introduction despite being widely applicable in healthcare, medical insurance and medical research, medical coding remains an under-automated process. this is mainly due to the huge amount of codes in medical ontologies on one hand and the very limited access to medical texts for training natural language processing systems on the other. we are presenting research on the clinical text classification task using snomed ct 1 codes as target values. although the recent advances in artificial intelligence (ai) show significant improvement in transformer-based models' performance on various natural language processing (nlp) tasks, medical coding remains challenging due to the large number of classes in 1 https://www.snomed.org/ snomed (about 350k). moreover, such systems need to be precise and reliable, hence they are usually integrated in hospital information systems or used in health insurance companies. thus we propose ml-based approach that is developed on publicly available data. in addition, we compare our system to domain-specific large language models (llms). ## related work as manual annotation in the biomedical domain is insufficient, there's a rise in the adoption of ml approaches that leverage clinical text data for task automation, predictive modelling, and knowledge discovery @xcite . however, as free-text clinical notes are unstructured, and contain spelling errors, abbreviations, and domain-specific terminology @xcite , the problem of correct information extraction from clinical free-text remains a bottleneck to be properly addressed. the limited scope of available data leads to a limited range of models that can be employed and, consequently, to poorer results. this problem can be partially alleviated by using an english-centric multilingual approach that can leverage larger sets of data available in english for applications intended for other languages @xcite . the other way of coping with the lack of annotated training data is leveraging large language models (llms). as those models are trained on vast amounts of data, they can perform quite well on simple classification tasks in zero-shot setting @xcite . despite the fact that llms are quite powerful for common-domain nlp tasks, their efficiency in medicine is yet to be explored. @xcite present impressive results for llm application in clinical domain, evaluating performance over sev-eral benchmark datasets for question answering and named entity recognition. however, au @xcite argue that such models are not ready for application in real clinical practice. due to its widespread adoption the systematized nomenclature of medicine clinical terms (snomed ct) has been employed in clinical text processing for a range of tasks. @xcite , however, concluded that the majority of the applied approaches are rule-based. we present a method for semi-automated annotation through the classification of machinetranslated histopathology reports to snomed ct codes corresponding to relevant morphology or topography terms. we examine the performance of our model on four data sets composed of diagnostic and/or synoptic reports for cervical cancer, colon cancer, lung cancer and celiac disease use cases. we address additional problems such as small sample sizes and class distribution imbalance and compare our approach with domain-specific llms. ## data our data collection and preparation approach is: ## method following the data preparation is the model training and application. our proposed approach is composed of the following steps: ## experiments and results as described in section 4.4 we split our dataset into train and test sets. as shown in ## conclusion we have demonstrated an approach for extracting snomed ct concepts from clinical texts in multiple languages. employing a combination of machine translation, linked open data (both general resources, as wikidata, and narrower, as specific medical ontologies), transformers and more, we are able to leverage the rich resources available in english for classification of texts in languages with limited corpora available. while we apply our approach to the clinical field, more specifically histopathology texts, we believe the same approach can be tailored to another task or another discipline with similar success, as long as both pre-trained domain-specific models (or, alternatively, enough data and computational resources for pre-training) and linked open domainspecific ontologies and terminologies are available (or could be rather easily developed). our model is pre-trained and fine-tuned on open data only. as such, it can be further tailored towards a specific task where richer proprietary data is also available to fine-tune the model. one drawback of our approach is employing machine translation tools that are not domain-specific and cannot be fine-tuned. while not included in the present study, we expect that using relevant parallel corpora in the narrow fine-tuning step (or following it) could allow for sufficient transfer of embedded knowledge from the context-rich english corpora to the context-poor other language and allow for classification directly on the untranslated text. obtaining such parallel corpora, however, is likely to be an even bigger obstacle. comparison of the two approaches, where such corpora is available, would be an interesting direction for future study.