Biomedical Concept Relatedness – A large EHR -based benchmark (2020)

Abstract: A promising application of AI to healthcare is the retrieval of information from electronic health records (EHRs), e.g. to aid clinicians in finding relevant information for a consultation or to recruit suitable patients for a study. This requires search capabilities far beyond simple string matching, including the retrieval of concepts (diagnoses, symptoms, medications, etc.) related to the one in question. The suitability of AI methods for such applications is tested by predicting the relatedness of concepts with known relatedness scores. However, all existing biomedical concept relatedness datasets are notoriously small and consist of hand-picked concept pairs. We open-source a novel concept relatedness benchmark overcoming these issues: it is six times larger than existing datasets and concept pairs are chosen based on co-occurrence in EHRs, ensuring their relevance for the application of interest. We present an in-depth analysis of our new dataset and compare it to existing ones, highlighting that it is not only larger but also complements existing datasets in terms of the types of concepts included. Initial experiments with state-of-the-art embedding methods show that our dataset is a challenging new benchmark for testing concept relatedness models.

## introduction the adoption of electronic health records (ehrs) facilitates interoperability, meaning that more and more information from different sources is being stored about a patient. this makes it increasingly challenging for doctors to efficiently filter a patient's record for relevant information during a consultation without missing anything. this is particularly problematic since consultations are time-constrained. in the uk for example, general practitioner (gp) doctors usually have less than 10 minutes to consult a patient @xcite . ehrs consist not only of free-text records but are furthermore tagged by doctors with medical concept codes. this coding is aimed at standardising health records to enable, e.g., the seamless transfer of patient information between practices and the analysis of health data from different practices @xcite . in addition, coded ehrs allow for the search of concept codes in an ehr. however, retrieving not only the exact concept in question but also related ones, as done by doctors when reading a patient's record, is less straight-forward. for a patient with potential liver failure, related information of interest to a doctor in the patient's history are for example 'alcohol abuse', as it is a risk factor of liver failure, and 'jaundice', a symptom of liver failure. other risk factors, symptoms, treatments, conditions, or tests associated with liver failure would also be considered relevant. concept representation models, such as embeddings or ontology-based methods @xcite @xcite @xcite , have been developed to tackle the task of identifying and retrieving related concepts. these methods have the potential to aid doctors in finding related information in a patient's ehr, which can increase the quality of medical outcomes by improving efficiency, as consequently alleviating time pressure, and ensuring that doctors do not miss important information. it is however unclear how well these methods would perform in real-world ehr concept retrieval settings as they have so far only been tested on very small datasets, as pointed out by @xcite . we address this issue by constructing a novel open-source foot_0 biomedical concept relatedness dataset consisting of 3630 concept pairs -six times more than the largest existing dataset. instead of manually selecting and pairing concepts as done in previous work, our dataset is sampled from ehrs to ensure concepts are relevant for the ehr concept retrieval task. the relatedness scores assigned to concept pairs in our dataset are of high quality, as shown by good inter-annotator agreement and reliability metrics. a detailed analysis of the concepts in our novel dataset reveals a far larger coverage compared to existing datasets. we furthermore report the results of initial experiments with state-of-the-art embeddings, illustrating that our dataset constitutes a challenging new benchmark. ## related work relatedness and similarity are not to be confused, even though embedding models are often tested on both types of relations @xcite @xcite . semantic similarity is a specific type of semantic relatedness @xcite , meaning that similar concepts are generally related but not vice versa. as an example, 'liver failure' and 'alcohol abuse' are medically related but not semantically similar. we are here concerned with relatedness. @xcite hand-picked 120 pairs of medical concepts from umls @xcite ) that were expected to have a balanced distribution across four categories: closely related, somewhat related, somewhat unrelated and completely unrelated. the pairs were then rated by 13 medical coders on a 1-10 relatedness scale. coders were not given a definition of the scale and were instructed to use their intuition. since the coders' agreement was low, the 29 concept pairs with highest agreement were chosen and annotated again by nine medical coders and three physicians as synonyms (4), related (3), marginally related (2), or unrelated (1), resulting in the minimayosrs dataset. @xcite selected a subset of 101 pairs from the original 120, excluding duplicates and ambiguous pairs, and analysed the coders' ratings in more detail. this subset is available as the mayosrs dataset. based on their observations, they also proposed a framework for the future creation of concept relatedness datasets, which we closely follow. minimayosrs and mayosrs both lack size and coverage. they are thus not suitable for testing concept relatedness models with the purpose of selecting the best one for real-world applications. @xcite introduced the umnsrs-sim and umnsrs-rel datasets, consisting of manually chosen pairs of umls concepts rated on a continuous scale of 0-1600 regarding their similarity and relatedness, respectively. the rating was performed on a touch screen, where the continuous scale corresponds to pixels on the screen. raters had only 4 seconds to rate each pair and were not given any definition of the similarity/relatedness scale. out of 724 given concept pairs, four medical coders rated 566 of them regarding similarity and another four coders 587 of them regarding relatedness. @xcite presented 36 pairs of mesh terms with a similarity score of 0-1. chiu et al. (2018) created the bio-simlex and bio-simverb datasets of, respectively, 988 pairs of nouns and 1000 pairs of verbs that frequently occur in pubmed. since both works involve similarity rather than relatedness and terms are not linked to concepts in any biomedical ontology, they are omitted from our comparison of existing datasets with our novel benchmark. ## a new concept relatedness benchmark to enable the development of reliable models for searching ehrs and biomedical literature, appropriate benchmark datasets for testing are essential. existing datasets (see section 2) have various shortcomings, which we address in the construction of our novel benchmark: 1) size: an appropriate test set needs to be of sufficient size to allow for the generalisability of performance results. our novel benchmark is 6 times larger than the biggest existing dataset. ## dataset analysis some concept pairs in ehr-rela and ehr-relb were not rated as the meaning of some concepts was unclear. excluding these pairs, ehr-rela consists of 111 concept pairs and ehr-relb of 3630. ## concept coverage clearly our new ehr-relb dataset is larger than existing ones in terms of number of concept pairs and unique concepts. in this section, we further investigate the types of concepts in ehr-relb compared to existing datasets. since umnsrs-rel and umnsrs-sim consist of nearly the same concept pairs, their concept coverage is very similar. we thus only present results for the relatedness dataset umnsrs-rel. ## experiments with sota embeddings as an initial experimental evaluation on our dataset, we evaluate the 13 state-of-the-art open-source biomedical word embeddings tested by schulz and juric (2020) on existing datasets: pmc, pm, pp, and ppw by @xcite 2018 ), (mesh) extr and intr by @xcite , and mim(ic) and its m(odel) version by @xcite . we do not consider the sentence embeddings tested by schulz and juric (2020) as they showed poor performance on existing datasets. compared to existing datasets, the performance of embeddings on ehr-relb is lower. the best of the 13 embeddings on mayosrs yields a spearman's correlation of 0.57 and on umnsrs-rel 0.59 @xcite . to investigate possible reasons for the lower performance, we measure the performance of embeddings on a further subset of 2978 concept pairs, excluding concept pairs with high disagreement between annotators (3 different scores assigned). however, this only marginally improves performance, indicating that low agreement concept pairs are not a source of the lower model performance. a possible explanation for the lower performance is that ehr-relb consists of 89% multi-word concepts, whereas mayosrs has only 47% and umnsrs-rel 0%. representing multi-word concepts with word embeddings is likely to induce noise, whereas representing single-word concepts does not. the human upper bound (hub), i.e. the maximum spearman's correlation achieved by any annotator with the mean rating, is 0.88. note that this is a slightly biased metric as the mean rating will include the annotator's rating. if the hub is computed by comparing an annotator's rating with the mean rating of the other annotators, it is slightly lower at 0.70. the hub shows that there is large scope for improvement of relatedness models, but that it should not be expected to achieve performance scores of 0.9 or higher. our initial experiments show that ehr-relb is a challenging new benchmark for the performance analysis of concept relatedness models. ## conclusions we presented a novel biomedical concept relatedness dataset sampled from ehr data, thus ensuring its relevance to ehr retrieval tasks. it is six times bigger than existing datasets, has high quality annotations, and complements existing datasets in terms of concept coverage. initial experiments showed that it is a challenging new benchmark for state-of-the-art biomedical word embedding models. despite our benchmark being much larger than existing datasets, we hope that this work inspires others to use our methodology for building even larger ones. as explained, our dataset covers around 3000 unique concepts, whereas snomed-ct consists of close to 350,000 concepts. we here focused on the most frequently co-occurring concept pairs. it would be interesting to expand this to less frequent pairs in future work. this could also involve focusing on specific areas of medicine. since our new benchmark consists of concept pairs expressed as 1) biomedical terms, 2) snomed ids, and 3) umls cuis, it can be used as a test bed for a large variety of concept representation models. in our initial experiments, we only considered word embedding models based on terms. in future work, it will be interesting to evaluate umls concept embeddings @xcite @xcite as well as graph-embeddings @xcite . our work was motivated by the retrieval of information in ehrs related to a patient's presenting complaint. however, the usage of this benchmark goes far beyond this motivation. coding in ehrs is not always perfect. for example, doctors do not always code both symptoms and diagnosis. enabling the search for related information is thus crucial to overcome the challenges associated with missing data.