Generalizing over Long Tail Concepts for Medical Term Normalization (2022)

Abstract: Medical term normalization consists in mapping a piece of text to a large number of output classes.Given the small size of the annotated datasets and the extremely long tail distribution of the concepts, it is of utmost importance to develop models that are capable to generalize to scarce or unseen concepts.An important attribute of most target ontologies is their hierarchical structure. In this paper we introduce a simple and effective learning strategy that leverages such information to enhance the generalizability of both discriminative and generative models.The evaluation shows that the proposed strategy produces state-of-the-art performance on seen concepts and consistent improvements on unseen ones, allowing also for efficient zero-shot knowledge transfer across text typologies and datasets.

## introduction term normalization is the task of mapping a variety of natural language expressions to specific concepts in a dictionary or an ontology. it is a key component for information processing systems, and it is extensively used in the medical domain. in this context, term normalization is often used to map reported adverse events (aes) related to a drug to a medical ontology, such as meddra @xcite . this is a challenging task, due to the high variability of natural language input (i.e., from the informality of social media and conversational transcripts to the formality of medical and legal reports) and the high cardinality and long tail distribution of the output concepts. aes are usually mappable to different levels of the same ontology: low-level concepts, which are closer to layman terms, and higher level concepts, which encompass the meaning of multiple low-level concepts. in meddra, foot_1 these two sets of concepts are called lowest level terms (llt), and preferred terms (pt) respectively; both of them have a very high cardinality (48,713 for llt and 24,571 for pt, in meddra version 23.1). the following are examples of aes, with their corresponding llts and pts: ae llt pt feel like crap feeling unwell malaise weak knees weakness asthenia zap me of all energy loss of energy asthenia currently this problem is addressed with large pretrained language models @xcite , finetuned on medical term normalization datasets, such as smm4h @xcite or cadec @xcite . however, these datasets contain maximum 5,000 samples, distributed on a few pt/llt classes, and with a long tail distribution (see figure 1 ). due to the size and distribution of these datasets, the resulting models usually perform well on examples that are seen in the training, but struggle to generalize on rare or unseen samples. to improve the generalization capabilities of the models on long tail concepts, in this paper, we suggest to leverage the hierarchical nature of the medical ontology to enrich the large language models with domain knowledge before finetuning on a given training set. extensive experimental evaluation on three different datasets shows that the proposed strategy can be successfully applied to various model typologies, and it consistently outperforms other mainstream learning strategies, showing generalization capabilities not only across the long tail distribution, but also across text typologies and datasets. the code and resources needed to replicate our experiments and test our learning strategy are publicly available foot_2 . ## related work medical term normalization is generally regarded as either a classification or a ranking problem @xcite . in the former case, a neural architecture encodes the term into a hidden representation and outputs a distribution over the classes @xcite @xcite , but this is difficult to scale to ontologies containing thousands of concepts, due to the absence of comprehensive datasets. in the ranking approach, on the other hand, the goal is to rank concepts by their similarity to the input term @xcite @xcite : a system is trained on binary classification, where terms and matching concepts are the positive samples, while terms and non-matching concepts are the negative ones. the raw output of the model is then used to rank the concepts. recent work successfully combined the two approaches. @xcite presented a system mixing a bert-based classifier @xcite and a zero/few-shot learning method to incorporate label semantics in the input instances @xcite , showing improved performance in single model and in ensemble settings. finally, systems like coder @xcite and sapbert @xcite introduced novel contrastive pretraining strategies that leverage umls to improve the medical embeddings of bert-based models. while sapbert leverages self-alignment methods, coder maximizes the similarities between positive term-term pairs and term-relation-term triples and it claimed state-ofthe-art results on several tasks, including zero-shot term normalization. another recent work by @xcite introduced an even more extensive pretraining procedure, based on self-supervision and a combination of the traditional masked language modelling with contrastive losses . the strategy proved to be extremely effective for medical entity linking, a kind of term normalization which makes use of the full original context (instead of using only the ae). ## proposed learning strategy: op+ft let's consider a target ontology (e.g. meddra v23.1) containing two sets of concepts @xmath0. the ontology is structured so that every ℓ i has only one parent p j : parent(ℓ i ) = p j , but each p j can be parent of many ℓ i . given a set of adverse events @xmath1, every a i can be univocally mapped to a p j : norm(a i ) = p j . our objective is to train a large language model m to encode norm: given a sample (a i , p j ), such that norm(a i ) = p j , we want m(a i ) = p j . we propose a learning strategy based on the hierarchical structure of the ontology, composed of two steps: ontology pretraining and finetuning. during the first step, we expose the language model m to all possible output classes p j by leveraging the intrinsic hierarchical relation between llt and pt. specifically, we use the parent relation to create a new set of training samples from the ontology, pairing each ℓ i to its parent concept p j . in the case of meddra, the new set of samples contains 48,713 (ℓ i , p j ) pairs, where each p j appears multiple times, associated with different ℓ i . for example the pt "asthenia" will appear in the samples (weakness, asthenia), and (loss of energy, asthenia). as llts are more informal than pts, the language model m can be pretrained on this new set of data to gain general knowledge about all the target classes. this pretraining set is highly similar to our target dataset (m(a i ) = p j ), increasing the model transfer capability. we call this process "ontology pretraining" (op). the second step consists in finetuning (ft) an op model on a specific term normalization dataset, which maps every ae a i to the corresponding pt p j . this step is crucial because the op model lacks specific knowledge about the natural language style of real-world samples. finetuning will also exploit the dataset sample distribution to boost the model's accuracy on the specific set of p j in the training set. note that the ft step can also be applied to a regular model without op, resulting in a regular finetuning. we hypothesize that the combination of op+ft to a discriminative or a generative language model m will improve its performance on seen concepts in the training set, while making it more generalizable to long tail and unseen concepts. ## experimental results in an ablation-study fashion, we compare the op+ft learning strategy with its two components: op and ft. figure 2 contains the results for all the tested models and training strategies, and is organized as follows. we display a plot for each dataset, reporting the accuracy of the models on in samples (•), out samples (⋄) and the whole test set (•). the first column shows the performance of a basic coder and sapbert model without any additional training. we consider their accuracy on out (⋄) as our generalization goal, and plot them as solid lines across the chart. the following three columns display the performance of all the models, trained with one of the learning strategies (ft, op and op+ft respectively). for tabular results, see appendix d. coder and sapbert on their own proved to be strong baselines across the three datasets. looking at the first column, they reach 40-50% accuracy on cadec and smm4h (overall, in and out, see solid lines), and around 15-20% overall accuracy (•) on prop. all learning strategies seemed to be ineffective on coder: its performance (gray markers) remains roughly the same across all strategies (ft, op or op+ft). a possible explanation for this behaviour is that coder embeddings are already in an optimal state according to the training objectives, as they have been trained on a very similar 3 coder: https://github.com/ganjinzero/coder sapbert: https://github.com/cambridgeltl/sapbert task. in fact, coder generates predictions using the similarity between the embeddings, and the stable performance indicates that there were no drastic changes in the structure of the embedding space. a clearer effect of the training strategies can be seen on sapbert (lilac markers), although it is still limited when compared with the other models. sapbert embeddings are probably more subject to adjustments compared to coder because the latter was trained for significantly more steps and using more objective functions, leading to lessadaptable embeddings. the following observations apply the other three models: pmb, gpt-2 and sci5. coder sapb 0.0 0.2 0.4 0.6 0.8 1.0 accuracy@1 coder sapb pmb gpt2 sci5 ft coder sapb pmb gpt2 sci5 cadec op coder sapb pmb gpt2 sci5 op+ft coder sapb 0.0 0.2 0.4 0.6 0.8 1.0 accuracy@1 coder sapb pmb gpt2 sci5 ft coder sapb pmb gpt2 sci5 smm4h op coder sapb pmb gpt2 sci5 op+ft coder sapb 0.0 0.2 0.4 0.6 0.8 1.0 accuracy@1 coder sapb pmb gpt2 sci5 ft coder sapb pmb gpt2 sci5 prop op coder sapb pmb gpt2 sci5 op+ft figure 2: accuracy of all models on the three datasets on in (•), out (⋄) and all (•) samples. the ft strategy (second column), as expected, works really well for in (•) samples: on cadec the in accuracy of all models is over to 80%, while it is close to 50% for the other two datasets. however, the out accuracy (⋄) is lower than 20% in all cases (significantly lower than the solid line), and reaches 0% for pmb, showing that finetuning alone is not sufficient for classifiers to generalize on out samples in this setting. the op strategy (third column), brings the out accuracy of all models on pair with the coder/sapbert baselines, while the in/overall accuracy matches or surpasses them. comparing op with ft, we see that the overall accuracy (•) of the model is generally lower for op. however, the performance on out (⋄) samples doubles for generative models, and jumps from 0 to 40% for pmb. this shows that the first step of our proposed learning strategy has the desired effect, as it improves the models' understanding of all the output classes. finally, looking at the models trained with the op+ft strategy (fourth column), we see that they outperform the ft ones on overall and in accuracy. the effect is particularly strong on the smm4h dataset (cf. pmb ft, 44% and pmb op+ft, 70%). at the same time, the performance on out (⋄) samples remains similar to the op models and close to the coder baseline (gray solid line). the only exception is cadec, where the performance on out is in-between the baseline and the accuracy with ft only. this shows that the proposed op+ft learning strategy can successfully improve the generalization capabilities of various language models, while also improving their overall performance. we further test the generalization of op+ft models in zero-shot, cross-dataset term normalization, normalizing the terms of each dataset with models that have been trained on one of the other two. figure 3 shows the accuracy of gpt-2, with a plot for each test dataset, different training datasets on the x-axis, and one column for each learning strategy (ft or op+ft). the behaviour of the other models is similar (see appendix d). in all columns, we observe a drop in overall accuracy (•) between the first data point and the following ones (e.g., cf. cadec trained on cadec and cadec trained on smm4h). however, this drop is larger for ft models than op+ft ones (e.g., 35 vs. 20 points on cadec). in addition, the out accuracy of op+ft models remains high regardless of the training set. this shows that op+ft models generalize better than ft models across-dataset. note that generalization is still challenging when moving from a dataset with highly informal language to a formal one (see prop trained on smm4h). ## conclusions in this paper, we shed some light on the importance of generalization for medical term normalization models. we showed that ae normalization models trained with traditional finetuning, despite showing high accuracy on leaderboards, have low generalization capabilities due to the long tail distribution of the target labels. our proposed training strategy (op+ft), which leverages the hierarchical structure of the ontology, outperforms traditional models, while also obtaining state-of-the-art results in generalization on out samples. this was also demonstrated in a zero-shot normalization setting. op+ft showed improvements on discriminative and generative language models, while it seems to be less effective on models trained with contrastive losses. this promising technique could also be applied to other tasks with massive output spaces organized in a hierarchical manner.