Uncertainty Quantification of Text Classification in a Multi-Label Setting for Risk-Sensitive Systems (2023)

Abstract: This paper addresses the challenge of uncertainty quantification in text classification for medical purposes and provides a three-fold approach to support robust and trustworthy decision-making by medical practitioners. Also, we address the challenge of imbalanced datasets in the medical domain by utilizing the Mondrian Conformal Predictor with a Naïve Bayes classifier.

## introduction this paper focuses on developing a novel method based on a robust conformal framework for a confidence-based classification for better decisionmaking. our project aims to develop methods for uncertainty quantification in text classification for risk-sensitive systems. using medical transcription data from kaggle, we assign patients to specific labels based on their medical history. with a better understanding of the uncertainty associated with our predictions, we aim to enable more reliable and robust decision-making in the medical domain. to address the limitations of traditional nlp techniques in the medical domain, our paper proposes a novel framework for uncertainty quantification in text classification for risk-sensitive systems. we highlight the existing problems in text classification and why uncertainty quantification is essential for evaluating the models. we review the previous works on uncertainty quantification in ml and emphasize the need for a reliable decision-making framework. we propose a three-step methodology that involves training and testing data sets, calibration sets, and classification engines. in the first step, we use a medical transcription data set and obtain a confusion matrix using the naïve bayes classifier. in the second step, we use conformal prediction with a calibration set and create another confusion matrix to observe a decrease in error rate in most cases. we assign p -values to labels based on the confusion matrix output, which gives us the confidence level and credibility score, decided by the p -value. our main novelty is the integration of existing conformal prediction with text similarity. our proposed framework gives a classification and provides two evaluation metrics, confidence and credibility, which offer helpful insights instead of just giving binary classification labels. in conclusion, our proposed framework can be used for reliable decision-making in risk-sensitive systems such as the medical domain. ## related work text classification has been widely explored in the field of nlp, and it has found applications in various domains such as finance @xcite , military @xcite , and medical @xcite , among others. most of the research in this field has focused on developing algorithms that can improve accuracy while keeping the computational cost low @xcite . however, achieving high accuracy alone cannot ensure a reliable system in risk-sensitive domains like medical applications. a framework is required to address the uncertainty associated with the predictions made by ml models to enable trustworthy decision-making @xcite . recently, there has been growing interest in designing novel metrics for the medical applications of artificial intelligence (ai) @xcite . however, we still see a gap in the practical realization and the applicability of the metrics for confident decision-making for a text classification system. @xcite suggests a technique called "calibrated regression" to estimate uncer-tainty in deep learning models accurately. the method involves training a regression model to predict the variance of the model's output given the input data. the regression model is trained on a validation set to ensure it is well-calibrated, meaning that the predicted variance values accurately measure the model's uncertainty. they show that their approach can accurately estimate uncertainty in various deep learning models, including those used for image classification and nlp. another proposed method for estimating predictive uncertainty in deep neural networks is called "deep ensembles", where multiple networks with the same architecture but different random initializations are trained to estimate uncertainty. the authors demonstrate that their approach is simple, scalable, and effective in estimating uncertainty in various benchmark datasets, which can be utilized to detect out-of-distribution examples and improve model calibration @xcite . in this paper, we overcome the above limitations, and with the proposed method, we conclude multifold benefits. we provide complementary metrics to quantify the uncertainty and provide the outcome to the decision maker to make a robust and trustworthy decision. ## methodology the methodology used in this study complements the existing ml classification algorithms for nlp techniques by incorporating conformal prediction (cp) as an uncertainty quantifier to reduce the false discovery rate and make the model robust and reliable. traditionally, classification algorithms for nlp use descriptive text data (x) as input data to predict the output label (y), such as positive or negative sentiment. this prediction is made by feeding x into a function f (x), which returns a label (y) based on the features in x. in this paper, we take a step further by incorporating cp into our approach. ## experimental framework this section shows the experimental results of the medical transcriptions dataset from kaggle. the experimental results with source code and dataset are provided on github foot_0