Benchmark for Evaluation of D anish Clinical Word Embeddings (2023)

Abstract: In natural language processing, benchmarks are used to track progress and identify useful models. Currently, no benchmark for Danish clinical word embeddings exists. This paper describes the development of a Danish benchmark for clinical word embeddings. The clinical benchmark consists of ten datasets: eight intrinsic and two extrinsic. Moreover, we evaluate word embeddings trained on text from the clinical domain, general practitioner domain and general domain on the established benchmark. All the intrinsic tasks of the benchmark are publicly available.

## introduction word embeddings are real-valued vectors that are trained to represent words based on the context in which they appear. based on the distributional hypothesis @xcite , which suggests that words with similar contexts have similar meaning, embeddings of semantically similar words are expected to appear close to each other in vector space. since their introduction, word embeddings have been ubiquitous in natural language processing (nlp) due to their ability to represent word meaning. typically, word embeddings are trained on a general text corpus such as wikipedia. afterwards, word embeddings are used as stand-alone features or as input to neural networks to perform a wide variety of nlp tasks such as text classification, named entity recognition (ner) and machine translation. in specialized domains, such as the clinical, word embeddings are also widely used to e.g. extract information from electronic health records (ehrs). however, the text in clinical ehrs differs significantly from the general domain. clinical ehrs include rare words, domain specific abbreviations and a mix of languages (for example @xcite . the text is often non-narrative and very concise, free of syntactic rules, sometimes consisting of a sequence of keywords. moreover, it contains many spelling errors, and the se-mantic meaning of words can differ from that of the general domain @xcite . in the clinical domain, word embeddings are, therefore, often trained on an in-domain corpus to better capture the vocabulary and the semantic meaning of words. after being trained on an in-domain corpus, they are used for e.g. clinical ner, international classification of diseases coding, clinical event detection, de-identification and patient similarity estimation with improved performance over general word embeddings @xcite @xcite . for evaluating word embeddings, two different methods are typically used: intrinsic and extrinsic evaluation @xcite . in intrinsic evaluation, word embeddings are evaluated based on their inherent information, e.g. by exploring the syntactic or semantic relationship between words. in extrinsic evaluation, word embeddings are evaluated based on their ability to solve a downstream task, e.g. by using them as input to a neural network. while word embeddings can be evaluated using extrinsic benchmarks by holding the network architecture fixed while varying the set of word embeddings, intrinsic benchmarks provide an intermediate evaluation of the embeddings' properties before being used as input to a larger system. this supports the need for intrinsic evaluation. word embeddings for the general domain are publicly available in many languages @xcite . however, publicly available embeddings for the clinical domain are scarce @xcite . this is most likely due to strict regulations around clinical data which contain sensitive information making them unsuitable for sharing. therefore, researchers in clinical nlp are often forced to create their own word embeddings in order not to expose sensitive information @xcite . clinical intrinsic benchmark datasets do not necessarily contain sensitive information and can, in that case, be shared openly, benefitting researchers producing clinical word embeddings. for the english language, both intrinsic and extrinsic benchmarks exist, e.g. university of minnesota medical residents similarity / relatedness set (umnsrs) @xcite for word similarity and relatedness, and blue @xcite , which includes both clinical and biomedical datasets, for extrinsic evaluation. for danish, though, no clinical benchmark exists. in this paper, we introduce a clinical word embedding benchmark for the danish language. moreover, we produce clinical word embeddings and use the benchmark to compare them to embeddings trained on the general domain and embeddings trained on the general practitioner (gp) domain. the benchmark is specifically constructed to evaluate static word embeddings such as glove @xcite , continuous bag-of-words @xcite , skip-gram @xcite and fasttext @xcite . it is therefore not suitable for evaluation of contextual word embeddings produced by transformer models like bert @xcite , roberta @xcite and electra @xcite . although transformer models achieve state-of-theart results @xcite , static word embeddings are still useful as input to nlp pipelines. some advantages are that they require less compute to train and at inference time, and they work better on limited data @xcite . this is relevant for research within specialized domains, such as clinical nlp, where researchers must often train their own word embeddings on limited data and the hardware to train and run a transformer model is not necessarily available. static word embeddings are also relevant in time-critical tasks in clinical practice such as expanding single-word searches in the ehr using the nearest neighbors of the search term. expanding singleword searches is especially relevant in the clinical domain where many different terms can be used about the same basic symptom or disease. another advantage is their ease of use for medical doctors (mds) and clinical researchers who are not machine learning scientists compared to contextual word embeddings. the remainder of this paper first introduces the benchmark including the methods for creating each intrinsic and extrinsic dataset. it then describes the train-ing methods of the produced danish clinical word embeddings and those from the general and gp domains which they will be benchmarked against. finally, the benchmark results are presented and discussed. ## establishing benchmark the benchmark consists of an intrinsic and extrinsic part. in this paper, intrinsic performance is evaluated based on the quality of the semantic and syntactic inherent information using analogy and similarity tasks. we produce datasets for three different intrinsic evaluation methods: analogy tasks, similarity and relatedness tasks, and an equality task. analogy tasks, introduced by @xcite , take tuples of four words (a, b, c and d) and evaluate 'what is to c as b is to a' by selecting the nearest neighbour to the calculated vector in the embedding space, excluding the words forming the analogy: if the nearest neighbor to the calculated vector is d, the analogy task is correct. the task is evaluated on the percentage of correct predictions in the dataset. similar to @xcite , the similarity tasks take a tuple of two words and their similarity score, in our case, produced by one or more mds. the similarity score of a word pair is compared to the cosine similarity of the pair's word embeddings. the cosine similarity is calculated as: the correlation between md scores and cosine similarities for the dataset of word pairs is evaluated using the spearman's rank correlation coefficient. relatedness tasks are identical to similarity tasks except the mds produce a relatedness score instead of a similarity score. relatedness refers to one word calling to mind another word (e.g., needle-thread), while similarity reflects the degree of semantic feature overlap between words (e.g., whale-dolphin) @xcite . equality tasks take a tuple of two terms with the exact same meaning. as the similarity score of a pair is 1 for a perfect match, the objective is maximization of the cosine similarity between terms. the task is evaluated as the mean of the cosine similarities for all pairs in the dataset. the extrinsic part consists of two different text classification tasks in the clinical domain with the word embeddings as input. the quality of the word embeddings is evaluated based on the evaluation metric of the classification task. an overview of all datasets can be seen in ## word embedding evaluation this section describes an evaluation of word embedding models, trained on data from different domains, using the established benchmark. we make a clinical-general domain comparison using a fasttext @xcite model as it has the best performance on danish text according to benchmark results @xcite . we make a clinical-gp domain comparison using a glove @xcite model as it is the only available type of embeddings trained on danish gp data. we describe how the benchmark can be used to show strengths and weaknesses of different word embeddings. we trained two sets of clinical word embeddings using the fasttext and glove methods. the embeddings were trained on 299,718 danish ehrs from odense university hospital. the text was preprocessed by lowercasing and removing headlines, subheadings, phone numbers, social security numbers, emails, urls, dates and time stamps. samples were defined as text from the same subheading. after removal of duplicates and samples with less than 3 words, the corpus consisted of 1.4 billion tokens. for the clinical-general domain comparison, the clinical fasttext embeddings were trained with the default settings from the fasttext api ( www.fasttext.cc ) except from a vector size of 300, 10 negative samples and 10 epochs. the hyperparameters were chosen to be able to compare the produced embeddings with the fasttext word embeddings from @xcite pretrained on a general domain, specifically wikipedia and common crawl. the fasttext models can generate outof-vocabulary (oov) words from subwords which e.g. makes it capable of representing unknown spelling errors. for clarity, only the results without oov generation are reported here while the results with oov generation are found in appendix a. for the clinical-gp domain comparison, the clinical glove embeddings are 100-dimensional embeddings trained with the default settings from the code and paper by @xcite except for a min-count of 3. the hyperparameters were chosen to be able to compare with the glove word embeddings from @xcite trained on 323,122 gp ehrs. the word embedding models are benchmarked on the established intrinsic and extrinsic datasets. for each intrinsic task, we show the performance of the embeddings on the part of the evaluation dataset which is in-vocabulary (iv), ignoring the word pairs or analogies containing oov words. we also produce the iv rate as the proportion of word pairs or analogies which are in the vocabulary of the embeddings. additionally, appendix b contains the iv intersection results which show the performance of the embeddings on the intersection of all embeddings' iv dataset for that task. for the extrinsic tasks, the word embeddings are used as input to a recurrent neural network which is initialized and trained three times with the same set of standard hyperparameters. no hyperparameter tuning is performed. a bidirectional gated recurrent unit @xcite with 128 units followed by a dropout layer with probability 0.3 is trained with the adam optimizer with a learning rate of 5e-4 for a maximum of 100 epochs using early stopping. the best model, based on the validation loss, is evaluated on the test set. the test set accuracy is reported as the evaluation result. ## discussion in this paper, we have presented the first benchmark for evaluating danish clinical word embeddings. although the clinical word embeddings cannot be shared due to privacy concerns, having a publicly available benchmark will allow researchers to compare and evaluate locally available clinical word embeddings. below, we discuss the capability of the benchmark to compare word embedding performance in the clinical domain. as the intrinsic benchmark tasks consist of words which are typically, and in some cases, exclusively, used in the clinical domain, we expected higher iv rates from clinical domain embeddings. in concurrence, the results show that the clinical word embeddings, in general, have higher iv rates than those trained on the gp and general domain. an exception is that the general fasttext embeddings have the highest iv rate for the adjective inflection analogy task. one explanation could be that clinical written language does not use as many inflections of adjectives as the general. interestingly, when comparing the gp and general word em3 : syntactic benchmark results on the in-vocabulary (iv) dataset for each task by model type (fasttext, glove) and domain (clinical, general, general practitioner (gp)). the accuracy metric is the accuracy on the dataset. iv rates are reported in parenthesis. we underline the best results per task by model type. beddings on the semantic tasks, the gp embeddings, in four out of five tasks, have higher iv rates but lower accuracy. this result shows that the gp embeddings have seen more clinical domain words than the general embeddings during training, but the general embeddings capture higher quality information for the words that it has seen. this could be due to the size and quality of the dataset, differences between model types or the dimensionality of the embeddings. future work should investigate these claims further. the benchmark shows that the clinical embeddings surpass the general and gp embeddings in all semantic tasks except for the clinical analogy task where the general fasttext embeddings performed better than the clinical fasttext embeddings. this discrepancy may be caused by the clinical analogy dataset only containing 164 analogies of which only 54% are iv for the general fasttext model. the general embeddings surpass the clinical embeddings on the syntactic tasks which shows that it has captured higher quality syntactic information for the words that it has seen during training. this is most likely due to wikipedia and common crawl, which it was trained on, containing a higher quality of syntactic information than clinical ehrs. the fact that the general embeddings achieve the highest iv rate on the adjective inflection task suggests that the task consists of more inflections specific to the general domain than our clinical dataset. on the contrary, clinical domain embeddings achieve the highest iv rates on the verb and noun inflection tasks which suggests that these syntactic tasks do contain inflections specific to the clinical domain. similar to earlier work @xcite @xcite , we found that the clinical word embeddings perform better than the gp and general domain embeddings on extrinsic tasks. it is notable that for the extrinsic tasks, the gp glove embeddings are closer to the performance of the clinical glove embeddings than the general fasttext embeddings are to ). we report accuracies on the class-balanced bleeding and department classification tasks using the word embeddings as input. we underline the best results per task by model type. considering that the general embeddings perform well on syntactic tasks and clinical embeddings perform well on semantic and extrinsic tasks, future work should explore training word embeddings from the general fasttext checkpoint on clinical data. this might provide word embeddings that better capture both clinical and general syntactic and semantic properties. ## conclusion in this paper, we presented a benchmark for danish clinical word embeddings. the benchmark consists of two extrinsic tasks, five intrinsic semantic tasks and three intrinsic syntactic tasks. we developed clinical word embeddings and compared them with word embeddings trained on a general and general practitioner domain. the benchmark showed that the word embeddings trained on clinical data performed better on the extrinsic and semantic tasks, except for the clinical analogy task. on the syntactic tasks, the fasttext word embeddings trained on a general domain performed better than those trained on a clinical domain.