Classification of Syncope Cases in N orwegian Medical Records (2020)

Abstract: Loss of consciousness, so-called syncope, is a commonly occurring symptom associated with worse prognosis for a number of heart-related diseases. We present a comparison of methods for a diagnosis classification task in Norwegian clinical notes, targeting syncope, i.e. fainting cases. We find that an often neglected baseline with keyword matching constitutes a rather strong basis, but more advanced methods do offer some improvement in classification performance, especially a convolutional neural network model. The developed pipeline is planned to be used for quantifying unregistered syncope cases in Norway.

## introduction neural methods have revolutionized the field of nlp, including the clinical domain in recent years. the amount of performance gain, however, may not always be proportional to the increased complexity and decreased transparency that their use might entail, especially in data-sparse domains and target languages. the limited availability of data and its linguistic characteristics, i.e. a high density of terminology, repetitions, abbreviations and misspellings @xcite , are aspects that influence greatly the efficiency of the nlp methods applied. these have been compared to some extent in previous work @xcite @xcite , however, they are often evaluated on the same (and often limited) openly available datasets @xcite . the real-world utility of various approaches in clinical text processing, especially for languages other than english, however, remains still to be investigated @xcite . moreover, comparison to a simple rule-based baseline is often missing, leaving some uncertainty around the advantage of more advanced methods. starting from a close collaboration with akershus university hospital, we re-examine the question of the optimal methodological choice in the context of diagnosis coding in norwegian clinical notes. diagnosis codes are standard alpha-numeric codes representing a disease, a widely adopted scheme being @xcite . icd-10 codes are used for a variety of purposes, including hospital billing and reimbursement, population health statistics, and clinical research. additionally, the re-use of structured health data in clinical decision support and risk assessment has also been suggested. icd-10 coding is used as a most relevant classification of the reason for contact, underlying conditions or procedures related to the stay. a host of signs, events and observations are not coded. syncope, or similar signs, may be regarded as secondary or irrelevant for a certain patient, and thus only mentioned but not coded. clearly, accurate coding is important, but as a human process prone to error and biases, the quality of icd-10 codes has been questioned. this is the case for syncope -a transient loss of conciousness typically due to insufficient blood flow to the brain -which was chosen as the use case for our study. a large study of danish medical records @xcite found that around a third of actual syncope records did not have the appropriate icd-10 code. since syncope can be an important sign of heart disease and a marker of elevated risk of death in certain conditions such as hypertrophic cardiomyopathy @xcite , being able to retrieve information about patient's syncope events even when an icd-10 code is not present, is crucial for better risk assessment. also, this work constitutes a first step in the direction of an automatic diagnosis coding system for norwegian, which is currently not available. the research questions we investigate in this context are: (i) how do linear and neural models compare to a simple keyword matching baseline for binary automatic diagnosis code classification?; and (ii) how useful are pretrained embeddings for this task? in what follows, we first describe our health record data and our pre-processing steps. we then compare three types of methods for syncope classification: a rule-based one relying on keyword matching, linear machine learning models and neural models. besides estimating the amount of unregistered syncope cases in norway, our processing and classification pipeline can also easily be re-used to train more generic diagnosis code classifiers. ## background since medical language is rather terminologyheavy, rule-based methods can often go a long way in clinical nlp tasks and are, therefore, still rather wide-spread @xcite . statistical approaches handle better linguistic phenomena such as synonyms, code-switching and negation, however, they are computationally more expensive, require resources and, in particular neural ones, are often less interpretable @xcite . moreover, neural methods substantially alleviate the burden of feature-engineering, but are considerably more challenging in terms of hyper-parameter tuning. incorporating such models into clinical data processing pipelines is thus an advantage only if they can demonstrate a clear advantage over their simpler counterparts. @xcite developed linear classifiers with manually and automatically selected n-grams as features for classifying syncope in italian medical records. a frequent target of investigations has been the 2007 computational medicine challenge (cmc) dataset, focusing on automatic icd coding in radiology reports. both rule-based @xcite and statistical methods @xcite including neural ones @xcite , have been tested and sometimes compared on this data. @xcite reported that the performance of a support vector machine (svm) with term frequency-inverse document frequency (tf-idf) bag-of-words (bow) features remained considerably below the results of a convolutional neural network (cnn) with dynamic in-domain pre-trained word2vec embeddings with f1 scores of .65 and .81 respectively. a direct comparison across these works, however, is difficult given differences in the evaluation and data subset used. more recently, using another dataset, mimic-iii @xcite , experiments presented by @xcite indicated that neural methods outperform linear models for the same type of multi-class classification of icd codes, although not always by a large margin. @xcite also described a comparison between linear and neural models, but for different clinical binary classification tasks (e.g. status and negation prediction) and showed that recurrent neural networks tuned for their task performed on par with the more recent, transformer models @xcite . rule-based baselines were often not included in these recent studies @xcite @xcite , the practical advantage of different approaches therefore remains somewhat unclear compared to methods based on heuristics. ## dataset our data consisted of de-identified discharge summaries from akershus university hospital hospital. half of the notes were diagnosed syncope cases (syn), the other half were notes with a variety of diagnosis codes for patients with no recorded and coded history of syncope (nons). the documents were authored between 2005-2016. foot_0 while patients in syn were from a variety of departments, all nons patients were from the cardiology department. moreover, only patients who @xmath0. ## experimental setup the first pre-processing step consisted of tokenization with udpipe @xcite . diagnosis information reflecting the labels used for classification (syn vs. nons) was then removed from the documents using: (i) lexical matching for section title identification; and (ii) udpipe paragraph information for determining section boundaries. we divided our data into three stratified splits: 70% of it reserved for training, 15% used as validation data for hyper-parameter tuning and the remaining 15% was set aside for testing. we compared a keyword matching baseline to two linear classifiers, a logistic regression (lr) classifier and an svm, and to neural models, namely cnns. these learning algorithms have been commonly and successfully used in previous nlp studies, including the clinical domain @xcite . baseline with lexical matching (lexm) we computed a baseline consisting of a simple lexical matching applied to the pre-processed documents using the term synkope 'syncope', which would find both its baseform and other derived forms without additional lemmatization. whenever a document contained this term at least once, it was classified as belonging to the syn class, and otherwise as nons. linear models for training the linear models, we use scikit-learn @xcite , and we employ keras with tensorflow @xcite as backend for the neural models. for both svm and lr, we use bow features extracted with a tf-idf vectorizer. we perform a grid search for finding the optimal hyper-parameters on the validation data. neural models for the cnn, word2vec @xcite embeddings were used as input representation to capture contextual similarity between words. we adopted a common cnn architecture @xcite consisting of an input layer of 100 dimensions, a convolutional layer concatenating 100 filters of sizes 3 to 5, with rectified linear units, max pooling and a dropout of 0.5, followed by a fully connected softmax layer. we used binary cross-entropy loss, the adam optimizer, a learning rate of 0.001 and a batch size of 32. we trained for 10 epochs with early stopping based on validation accuracy and a patience of 2 epochs. we experimented with different embedding initializations, inspired by @xcite : a randomly initialized one (w2v-r) and two where weights were based on pre-trained embeddings. in one case, weights were not trainable during the learning process (static) and in the other, we continued training these weights (dynamic). this type of transfer learning consisting of fine-tuning pre-trained embeddings for a specific task is often beneficial when the size of the available training data is small @xcite . pre-trained embeddings in the absence of pretrained clinical embeddings for norwegian, we compared two other types of pre-trained embeddings, both 100 dimensional word2vec skip-gram models trained with gensim ( řehůřek and sojka, 2010): (i) general language embeddings w2v-g trained on ocr-ed books, news and web corpora, namely model nr. 100 from the nlpl repository foot_2 @xcite ; and (ii) domain-related embeddings w2v-m, which we trained on data from the norsk legemiddelhåndbok foot_3 'norwegian drug manual'. the medical vocabulary of these disease and drug descriptions was closely connected to the clinical domain. we used default parameters for training w2v-m, but lowered minimum word count to 1 given the small data size. ## model comparison and error analysis in lexical matching provided a rather high baseline, namely .80 accuracy, which suggests that similar terminology matching methods are worth testing and comparing to in terminology-rich domains such as the clinical one. although we started from a strong baseline, we found that, with increasing computational complexity, performance improved somewhat. lr proved to be the best linear model (.86 accuracy) with l1 penalty, @xmath1 method features init acc f1 prec r-sens r-avg lexm n/a n/a .80 .80 .80 .79 .80 svm bow n/a .85 .85 .85 .87 .85 lr bow n/a .86 .86 .86 .89 .86 cnn w2v-r random .92 (±.01) .92 (±.01) .91 (±.01) .93 (±.01) .92 (±.01) w2v-g static .87 (±.01) .87 (±.01) .87 (±.01) .85 (±.03) .87 (±.01) dynamic .89 (±.00) .89 (±.01) .89 (±.01) .88 (±.01) .89 (±.01) w2v-m static .68 (±.04) .67 (±.05) .70 (±.02) .80 (±.07) .68 (±.04) dynamic .77 (±.02) .77(±.02) .78 (±.02) .78 (±.03) .77 (±.02) for neural models, initializing embeddings randomly worked best. the number of in-embedding words was rather low in fact for both w2v-g and w2v-m, namely 51% and 27.5% respectively. in addition, w2v-g results might be influenced by a difference in domains. models with w2v-m produced not only lower scores, but also more instability as standard deviation shows, likely due to the small vocabulary size (50k) and few in-embedding words. dynamic embeddings showed improvements over static ones, especially for w2v-m, in line with previous findings @xcite . we compared our methods also with mcnemar's test @xcite 4 and found statistically significant difference in the misclassifications at α = 0.05 only between the baseline and cnn-w2v-r (@xmath2.003), but not between the other two model pairs, namely lr vs. baseline (@xmath3.163) and lr vs. cnn-w2v-r (@xmath4.077). figure 1 shows the receiver-operating characteristic (roc) curve for lr and cnn-w2v-r on the test set, which also shows a rather similar performance. to gain a better understanding into what the best performing linear and neural models, lr and cnn-w2v-r respectively have learned, we inspected the 30 words which received the highest weights after training. these included for both models nearsynonyms such as svimmel 'dizziness' and bevissthetstap 'unconsciousness' and even the english 4 with binomial distribution given the small sample size. translation of the term (syncope). yet another group of informative features described typical circumstances of syncope (e.g. gulvet 'floor'). lr also captured inflectional variants like synkopert, 'syncopated'. both models' decisions relied thus on factors relevant to the target medical phenomenon. our error analysis revealed that around half of the nons instances missclassified by both lr and cnn-w2v-r as syn did contain mentions of 'syncope', but sometimes either as part of a patient's previous history of illnesses or with negation (aldri synkopert 'never syncoped'). slightly more (60%) of misclassifications occurred for nons texts, however, 23% (lr) and 38% (cnn-w2v-r) of these appeared to be unregistered syncope cases. manually re-diagnosed data might therefore improve performance. ## conclusions we described a set of experiments using keywordmatching as well as machine learning methods for the classification of syncope cases in norwe-gian clinical notes. our results indicate that neural methods provide some advantage over a keyword baseline, but the latter performs surprisingly well, which indicates that terminological cues can be easily leveraged for such binary clinical text classification tasks in the absence of access to training data. this type of baseline constitutes thus a valuable starting and reference point for comparison to more advanced methods. future work includes hyper-parameter tuning of the neural models and comparing the generalizability of our models to new data, including different note types. we plan to use the developed models for quantifying the amount of unregistered syncope cases in norway and to extend them to classify a variety of diagnostic codes. embeddings trained on large norwegian clinical data would be valuable to boost performance for both this and other tasks. this work showcases a fruitful collaboration between an nlp research environment and a hospital. aligning clinical data processing interests and needs is particularly important for smaller languages without publicly available data for both moving the clinical nlp research front forward and to bring findings closer to the clinical practice.