Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification? (2024)

Abstract: The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in—chief of which is a model’s ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model’s output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.

## introduction deep-learning models are trained with datadriven approaches to maximize prediction accuracy @xcite . this entails several welldocumented pitfalls, ranging from closed-domain limitations @xcite to social systemic biases @xcite . these limitations compound to a severe deterioration of model performances in out-ofdomain (ood) scenarios @xcite . this has led to engineering efforts towards developing models tailored to specific domains, ranging from the legal @xcite to the biomedical @xcite ones. domain-specific models, while useful, are rarely considered as a definitive answer. crucially, in the biomedical domain, experts require more reliability from these models-in particular, insofar as accounting for uncertainty in prediction is concerned. for example, in the case of a risk scoring model used to rank patients for live transplant, uncertainty-awareness becomes critical. the lack of uncertainty-aware models may lead to improper allocation of medical resources (steyerberg et al., @xcite uncertaintyaware ## related work recently, uncertainty quantification has gained attention from the nlp community @xcite @xcite particularly in mission critical settings, such as in the medical domain @xcite . in parallel, compared to domain adaptation approaches @xcite for the medical domain, there is a growing interest in domainspecific language models starting from biobert @xcite to the recent medpalm @xcite . @xcite presented an elaborate study of uncertainty paradigm for generaldomain plms. while uncertainty modeling has been applied to biomedical data previously (e.g., @xcite , surprisingly little has been done for biomedical textual data. therefore, our study precisely focuses on the interaction between the two paradigms for medical domain nlp tasks. we address this gap by focusing specifically on predictive entropy @xcite . ## methodology datasets. we conduct experiments on six standard biomedical datasets: three english datasets, viz. medabs @xcite , mednli (romanov and shivade) and smoking @xcite ; as well as three french datasets, viz. morfitt @xcite , pxslu @xcite and medmcqa @xcite . for medabs, smoking, pxslu, and medmcqa, we do not perform any special pre-processing. for medmcqa, we perform task 2, i.e., predicting the number of possible responses (ranging from 1-5) for the input multi choice question. for mednli, we concatenate the statement and hypothesis using the [sep] token and use it as an input converting it to a multi-class task. for morfitt, which is originally a multi-label classification task, we use the first label for each sample to convert it to a multi-class problem. the descriptive statistics of these datasets are listed in models. we derive classifiers from languagespecific plms: for english datasets, we use bert @xcite and biobert @xcite ; for french, we use camembert @xcite and camembert-bio @xcite . we compare two types of models, frequentist deep learning models (dnn) and bayesian deep learning models (bnns). the dnn model comprises of a plm-based encoder, a dropout unit along with 1-layer classifier. the bnn models are likewise based on a plm encoder, along with a bayesian module applied over the classification layer. we also experimented with mc-dropout models @xcite , dropconnect @xcite , and variational inference @xcite models. we focus foot_0 on the dropconnect architecture which comprises a plm encoder along a dropconnect dense classification layer. this approach infuses stochasticity into a deterministic model by randomly zeroing out classifier weights with probability 1 -p. this allows us to sample multiple outputs for a given input, thus enabling to aggregate the predictions and to produce estimates of uncertainty. for simplicity, we note domain-specific models as +d (and general models -d); uncertainty aware models are referred to as +u (with frequen-tist models noted -u). we replicate training across 10 seeds per model and dataset; further implementation details can be found in appendix a.2. evaluation setup. we evaluate classifiers on two aspects: task performance and uncertainty awareness. for text classification, we report macro-f1 and accuracy. for uncertainty quantification we report brier score (bs; @xcite , static calibration error (sce; @xcite , negative log likelihood (nll), coverage (cov%) and entropy (h). see appendix a.3 for definitions. figure 2: performances for empirically best models (selected metrics), z-normalized per dataset. see ## results performance. all results are listed in relative importance. to interpret results in figure 2 more rigorously, we rely on shap (lundberg and @xcite . shap is an algorithm to compute heuristics for shapley values @xcite , viz. a game theoretical additive and fair distribution of a given variable to be explained across predetermined factors of interest. here, we analyze the scores obtained by individual classifiers on all 8 metrics, and try to attribute their values (z-normalized per dataset) to domain specificity (±d), uncertainty awareness (±u) and the dataset one observation corresponds to (ds.). results are displayed in figure 3 ; specific points correspond to weights assigned to one of the factors for one of the datapoints, factors are sorted from most to least impactful from top to bottom. we can see that which of domain specificity and uncertainty awareness has the strongest impact depends strictly on the metrics: cases @xmath0. another import trend is that effects tied to +d are also often attested for +u: if domain specificity is useful, then uncertainty awareness is as well. 2 lastly, weights assigned to @xmath1, showcasing that these trends are often overpowered by the specifics of the task at hand. ## discussion & conclusion we can now answer our initial research questions. rq1: are the benefits of uncertainty-awareness and domain-specificity orthogonal? we have seen in rq2: should medical practitioners prioritize domain-specificity or uncertainty-awareness? shap attributions in figure 3 strongly suggest that the evaluation metric dictates the strategy to follow. as one would expect, accuracy is better captured with domain-specific models, whereas uncertaintyaware models tend to be better calibrated. we also found significant evidence throughout our experiments that the exact classification task at hand weighs in much more strongly than the design of the classifier. this extraneous factor necessar-ily complicates the relationship between domainspecificity and uncertainty-awareness: in a handful of cases in figure 2 , we observe classifiers that are neither uncertainty-aware nor domain specific faring best among all the models we survey-and conversely domain-specific uncertainty-aware classifiers can also rank dead last. this is also related to the often limited quantitative difference between best and worst models, which for instance can be as low @xmath2.3% for f1 on medabs (cf. overall, our experiments suggest a very nuanced conclusion. domain-specificity and uncertaintyawareness do appear to shape classifiers' distributions and their entropy in distinct but compatible ways, but they have a lesser impact than the task itself. hence, while we can often combine uncertainty-awareness and domain-specificity, there are no out-of-the-box solutions, and optimal performances require careful application designs. yuxin xiao, paul pu liang, umang bhatt, willie neiswanger, ruslan salakhutdinov, and louis-philippe morency. 2022. uncertainty quantification with pre-trained language models: a large-scale empirical analysis. arxiv preprint arxiv:2210.04714. sihao yu, jiafeng guo, ruqing zhang, yixing fan, zizhen wang, and xueqi cheng. 2022. a rebalancing strategy for class-imbalanced classification based on instance difficulty. in proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 70-79.