R u CC o N : Clinical Concept Normalization in R ussian (2022)

Abstract: We present RuCCoN, a new dataset for clinical concept normalization in Russian manually annotated by medical professionals. It contains over 16,028 entity mentions manually linked to over 2,409 unique concepts from the Russian language part of the UMLS ontology. We provide train/test splits for different settings (stratified, zero-shot, and CUI-less) and present strong baselines obtained with state-of-the-art models such as SapBERT. At present, Russian medical NLP is lacking in both datasets and trained models, and we view this work as an important step towards filling this gap. Our dataset and annotation guidelines are available at https://github.com/AIRI-Institute/RuCCoN .

## introduction electronic health records and other clinical texts contain patient histories through the progression of diseases and represent a treasure trove of information for medical specialists. this information is often unstructured, concealed in freeform text, which leads to the need for natural language processing on medical texts. mentions of diseases, symptoms, drugs, and other concepts are highly variable, and since the medical vocabulary is very large, entity linking and concept normalization become hard and important problems. state-of-the-art models are increasingly successful in high-resource languages such as english or spanish, where labeled datasets include share/clef ehealth 2013 task 1 @xcite , @xcite , semeval-2015 task 14 @xcite , mcn @xcite , cantemist @xcite , codiesp @xcite , and others. however, little has been done for medical entity linking in many languages that are highresource in other regards. one example is russian: it is among top 10 languages in the world and has many nlp datasets and resources, but the medical part of russian nlp is underdeveloped. the russian umls includes translations of medical dictionary for regulatory activities (meddra) @xcite , logical observation identifiers names and codes (loinc) @xcite , and medical subject headings (mesh) @xcite , but it still only amounts to 1.8% of the english umls in vocabulary and 1.36% in source counts @xcite . in this work, we present ruccon (russian clinical concept normalization), a new labeled dataset for clinical concept normalization in russian. we have employed medical professionals to label the dataset based on concepts from the russian umls (section 2). moreover, we present several types of test sets for various settings, including stratified, zero-shot, and cui-less settings (section 2.4). we evaluate several state-of-the-art models on ruccon, including various fine-tuning variations, and check whether labeled data in russian is necessary (spoiler alert: it is) by testing cross-lingual concept normalization from english (section 3). our results can serve as baselines for ruccon and cross-lingual concept normalization. ## evaluation for entity linking, we use ranking based on embeddings of a mention and a possible concept. each entity mention and concept name are first passed through a model that produces their embeddings and then through a pooling layer that yields a fixedsized vector. the inference task is then reduced to finding the closest concept name representation to the entity mention representation in a common embedding space, where the euclidean distance can be used as the metric. nearest concept names are chosen as top-k concepts for entities. for ranking, we use the publicly available code foot_1 from @xcite . we compare ranking models based on several different embeddings: additionally, we compare several variations of fine-tuning on datasets with training sets via synonym marginalization as suggested by the authors of biosyn @xcite ): 1. sapbert+ruccon, with fine-tuning on our target train set of ehrs; 2. sapbert+mcn, with tuning on the mcn set; 3. sapbert+wrn, on the dataset extracted from the medical part of the ruwordnet thesaurus; 4. sapbert+xl-bel, on the the russian part of xl-bel; for training, we have used the publicly available code provided by the authors at https:// github.com/dmis-lab/biosyn with the following parameters: the number of top candidates k is 20, the mini-batch size is 16, the learning rate is 1e-5, the dense ratio for candidate retrieval is 0.5, the number of epochs is 5. to deal with nil prediction, we apply the strategy from @xcite ; a mention is out of kb if the nearest candidate is further than a threshold in terms of weighted average of two distances: minimum distance of false positives and maximum distance of true positives, as computed on the train set. following previous works on entity linking @xcite @xcite @xcite @xcite , we use top-k accuracy as the evaluation metric: acc@@xmath0, otherwise acc@@xmath1. ## error analysis to better understand the quality of our best model, we analyzed its erroneous predictions. for analysis, we randomly selected 100 erroneous predictions, which were then analyzed by an expert annotator with ph.d. in medicine. as can be seen from ## conclusion in this work, we have presented ruccon, a new clinical concept normalization dataset in russian, labeled by medical professionals and accompanied with several train/test splits for fair evaluation in various settings. we make ruccon publicly available for research purposes, and we hope that future works will make use of ruccon as a training and evaluation resource.