Detecting Idiomatic Multiword Expressions in Clinical Terminology using Definition-Based Representation Learning (2023)

Abstract: This paper shines a light on the potential of definition-based semantic models for detecting idiomatic and semi-idiomatic multiword expressions (MWEs) in clinical terminology. Our study focuses on biomedical entities defined in the UMLS ontology and aims to help prioritize the translation efforts of these entities. In particular, we develop an effective tool for scoring the idiomaticity of biomedical MWEs based on the degree of similarity between the semantic representations of those MWEs and a weighted average of the representation of their constituents. We achieve this using a biomedical language model trained to produce similar representations for entity names and their definitions, called BioLORD. The importance of this definition-based approach is highlighted by comparing the BioLORD model to two other state-of-the-art biomedical language models based on Transformer: SapBERT and CODER. Our results show that the BioLORD model has a strong ability to identify idiomatic MWEs, not replicated in other models. Our corpus-free idiomaticity estimation helps ontology translators to focus on more challenging MWEs.

## introduction translation in the biomedical domain remains particularly challenging due to the large number of specific and ad-hoc usage of terminology @xcite . some medical ontologies such as umls @xcite contain more than 4 million entities. out of these, only a fraction has already been labelled in languages other than english. while large efforts to translate some medical ontologies such as snomedct @xcite can be noted, few if any of these efforts have yet to yield full coverage of the ontology in their target language @xcite . popularity is of course one factor motivating the prioritization of the expert translation of some entity names over others, as translating popular entities makes the ontology usable to a large number of practitioners at a lower cost. but, with the rise of automatic translation tools, another factor worth considering in the prioritization is the translation difficulty of the entities being passed on to medical translation experts. their efforts should indeed better be directed to cases where automatic translation does not provide good results. in this context, idiomaticity has a key role to play. indeed, the automatic translation of idiomatic foot_0 mwes poses a significant challenge, as juxtaposing the translation of each individual constituent often results in a loss of meaning that can, in some cases, be catastrophic. this difficulty has been noted by prominent researchers such as @xcite and evjen (2018). as a result, identifying such idiomatic mwes would therefore immensely benefit the prioritization of translation efforts of medical ontologies. while many strategies for identifying mwes have been presented in the past @xcite @xcite , we found that applying them to the medical domain (and especially its clinical counterpart) was challenging due to the extreme corpus size that would be required to produce statistically significant results for the long tail of medical entities. in this paper, we investigate another approach relying on an ontological representation learning strategy based on definitions, and the empirical properties of semantic latent spaces, described by @xcite and @xcite . in particular, we investigate whether semantic models trained from ontological definitions perform better than other semantic models for the task of identifying idiomatic mwes without relying on their usage in context, using a novel self-explainability score which will be introduced in section 2. our semantic model, being based on a transformer + mean pooling pipeline (see figure 1 ), produces its representations by averaging the representation of the tokens it is provided as an input (after taking their interactions into account): to isolate the effect of these interactions, we compute a weighted average of the independent representations of the constituents of the mwe (with weights α i ) as a generalization of the above: our novel self-explainability score for mwes corresponds to the degree of similarity between their latent semantic representation (r σ ) and the best 2 weighted average of the independent representations of their constituents ( α i r i ). only strong inter-constituent interactions should be able to explain low self-explainability scores. 2 we determine the optimal weights αi in appendix a. based on this insight, we hypothesize that low self-explainability scores identify the mwes that the semantic model treats as idiomatic. to validate our hypothesis, we will demonstrate that there is indeed a statistically significant difference in selfexplainability scores between idiomatic and nonidiomatic mwes, among a chosen population. for our analysis, we construct a set of two-words mwes obtained from umls foot_1 , which were then subsequently divided into two groups by our annotators foot_2 : those which were "perceived as idiomatic or semi-idiomatic" and those which were "perceived as self-explanatory". we also hypothesize that a definition-based pretraining is essential for this analysis to produce good results. however, as the proposed analysis could be applied to any contextual text representation model, we set out to evaluate the benefits of the definition-based pretraining of the biolord model @xcite by comparing its results with two strong alternatives: sapbert @xcite and coder @xcite . these two state-of-the-art biomedical language models were also trained using contrastive learning and umls, but not using definitions as a semantic anchor. ## experimental results we start our analysis by plotting the empirical distribution of self-explainability scores for all considered umls entities. we report this empirical distribution as a histogram in figure 2 . interestingly, this distribution is unimodal, which seems to give weight to the hypothesis that mwes exist on a spectrum of idiomaticity, as described by @xcite , and do not form clearly distinct idiomaticity classes. based on our annotations, we evaluate the proportion of idiomatic mwes present in a subset of 10 bins of self-explainability scores (see figure 4 ). this enables us to estimate the full distribution of idiomatic mwes by multiplying these ratios with the population counts (see figure 3 ). these two distributions have very different means (0.850 vs 0.697), indicating that our selfexplainability score is indeed significantly lower for idiomatic mwes than for non-idiomatic ones. we determined based on our annotations that about 2.6% of the mwes in our dataset appeared idiomatic or semi-idiomatic in nature. to evaluate how effectively our self-explainability score can help identifying idiomatic mwes, we determined the threshold score enabling a recall of about 50% of idiomatic mwes in our dataset. this corresponds to about 4000 mwes featuring a similarity below 0.714, consisting of the outliers at or below the 95% percentile of our self-explainability scores. to confirm this, we annotated more extensively the mwes of our dataset falling into these 5 outlier percentiles. we find that about 23% of these mwes appear idiomatic to our annotators, which is in line with our population-based estimates of 26% (2.6% of idiomatic mwes * 50% @xmath0.3% of idiomatic mwes out of these 5% of outliers, yielding an expected precision of 26%). of course, a threshold of 0.714 represents only one of the possible operating points of our model. by varying this threshold, we compute the receiver operating characteristic (roc) of our classifier, and plot it in figure 5 (green curve). we find that our model shows an area under curve (auc) of 93%. repeating this analysis for other semantic biomedical models demonstrates the importance of biolord's definition-based training. indeed, both sapbert (orange curve) and coder (red curve) fail to provide a classifier that is as effective as biolord for this task, with auc scores of 0.84 and 0.65 respectively. see also figure 6 . to enable a more qualitative appreciation of the results, we also report the mwes featuring the lowest self-explainability scores, for each of the considered models (see ## conclusion in this paper, we investigated the suitability of definition-based semantic models for detecting idiomatic mwes in the terminology of a domain. we were able to demonstrate that our proposed selfexplainability score can indeed serves as a proxy for idiomaticity, and observed that the biolord model indeed displays strong ability to perform this evaluation in the biomedical domain. the corpus-free idiomaticity estimation thereby developed is powerful enough to help ontology translators to focus on more challenging mwes, with about half of the idiomatic mwes contained in the 5% of self-explainability score outliers. finally, we were also able to show that biomedical models which were not trained using a definition-based strategy perform significantly worse than our chosen definition-based model, showing the importance of a definition-based pretraining strategy in the development of reliable semantic representations for idiomatic mwes.