RECAL : Sample-Relation Guided Confidence Calibration over Tabular Data (2023)

Abstract: Tabular-format data is widely adopted in various real-world applications. Various machine learning models have achieved remarkable success in both industrial applications and data-science competitions. Despite these successes, most current machine learning methods for tabular data lack accurate confidence estimation, which is needed by some high-risk sensitive applications such as credit modeling and financial fraud detection. In this paper, we study the confidence estimation of machine learning models applied to tabular data. The key finding of our paper is that a real-world tabular dataset typically contains implicit sample relations, and this can further help to obtain a more accurate estimation. To this end, we introduce a general post-training confidence calibration framework named RECAL to calibrate the predictive confidence of current machine learning models by employing graph neural networks to model the relations between different samples. We perform extensive experiments on tabular datasets with both implicit and explicit graph structures and show that RECAL can significantly improve the calibration quality compared to the conventional method without considering the sample relations.

## introduction tabular-format data widely exists in various realworld applications such as healthcare @xcite , credit modeling @xcite , and finical fraud detection @xcite . various machine learning models @xcite are proposed for modeling tabularformat data, ranging from the traditional model such as @xcite to advanced deep models such as transformer @xcite . these methods have achieved remarkable progress in both industrial applications @xcite and data science competitions @xcite . despite these progresses, the reliability of current models trained on tabular data still remains largely unexplored. specifically, the predictive confidence of the model is the key aspect of reliability, which measures the probability that predictions will be correct. a high-quality confidence estimator is desired to deploy these ml models to risk-sensitive scenarios such as finance and healthcare @xcite . a natural way to construct the predictive confidence is based on the maximum value of the output probability vector @xcite . unfortunately, current ml models are prone to output inaccurate predictive distributions, which further lead low-quality confidence estimation. for example, as prior work shows @xcite , transformer models typically produce over-confident predictions @xcite . besides, even for traditional ml models such as gbdt and random forests, we also show that there is room for improvement in their predictive confidence quality. to provide accurate predictive confidence estimation of ml models, numerous post-hoc calibration techniques are presented for improving the confidence estimation in the post-training stage without modifying the original training procedure @xcite . the core idea of these techniques is to transform the original predictive probability into the calibrated one with some learnable transformation parameters. one common method is temperature scaling @xcite , which has been widely adopted for calibrating both deep and traditional ml models @xcite . however, most of these methods are designed for general settings without considering the unique properties of tabular-data format. this paper aims to improve previous calibration techniques over tabular-format data by considering the implicit relations between different samples. the motivation behind our work is based on the natural observation that real-world data tables contain implicit sample relationships, which can often be reconstructed by mining feature similarities. figure 1 shows a real-world tabular dataset used for house price forecasting. in general, the prices of houses with similar latitude and longitude are always correlated. in other words, home prices are more closely related to their location. these sample relationships can be further used to improve the model confidence. based on this observation, we introduce a general calibration method for ml models -recal, which provides a confidence calibration model with the ability to aggregate potentially correlated data across the table as clues for calibration. to integrate the implicit sample relation with the confidence, we build an undirected graph on top of them, where nodes are the samples in the table and edges represent the connections between them. a graph neural network (gnn) is employed to aggregate logits from different samples, calibrating confidence. recal learns a unique temperature t for each sample for scaling, hence retaining the accuracy of the basic model. by incorporating these implicit sample relations into the calibration process, we empirically show that the quality of estimated confidence has been consistently improved. moreover, recal demonstrates remarkable robustness and stability against data noise and distribution shift. in a summary, the main contribution of this paper is as follows: confidence calibration. confidence calibration has been gaining more and more attention recently. given a class prediction ŷ and a data set x, the model being perfectly calibrated should satisfy: thus, confidence calibration changes the predictive distribution (also called logits) of the original model. for the tree-based model, @xcite uses stochastic gradient boosting (sgb) or stochastic gradient langevin boosting (sglb) to generate ensembles of gbdt models. ngboost @xcite is based on natural gradients and uses the parameters of the conditional distribution as the target for multi-parameter boosting. all of the above methods require to retraining the basic model from scratch. in deep learning, order-preserving functions @xcite train the calibration function on the basic models using a post-hoc method through a held-out dataset. scaling is the commonly used post-hoc confidence calibration technique. on the binary model, platt scaling @xcite transforms logits by predicting two scalar parameters. temperature scaling and matrix scaling @xcite are multiclassification extensions of platt scaling. cagcn @xcite uses neural networks as nonlinear posthoc calibration functions. however, how to find implicit sample relations from the tabular data and construct a generic calibration model for tabular data independent of the basic model remains to be studied. ## methodology in this section, we introduce a post-hoc calibration method called recal to calibrate the model applied to the tabular data. assume in a tabular classification task, we have a basic model m, which can classify each row into k classes. yet m is not well-calibrated, indicating its confidence cannot accurately reflect its performance. our purpose is post-calibration, which indicates that the training data is unavailable at this stage. only a small held-out dataset can be used @xcite . therefore, we aim to train another calibration model, i.e. recal c, with the held-out dataset @xmath0, r 2 , ..., r n }, where n indicates the number of samples in r. recal achieves calibration for each sample by leveraging the graph structure built from implicit sample relations. specifically, as depicted in figure 3 , recal first pre-trains a feature selection model and constructs a graph structure implicit in the table based on the selected feature ( §3.1). then we use gnn as a non-linear calibration function to characterize the topology on the graph ( §3.2). finally recal outputs a unique temperature t for each sample and calibrates samples using a scaling-based approach ( §3.3). ## conclusion in this paper, we exploit the implicit sample relation on tabular data and propose a post-hoc calibration framework recal, which improves the confidence of the model while maintaining classification accuracy. it is worth noting that our approach is general and applicable to both deep learning models and traditional machine learning models. recal uses the gcn as a nonlinear calibration function so that the confidence can be propagated along the graph between the sample. in addition, recal generates a unique temperature for each sample to be calibrated using the scaling method, which has the accuracy-preserving property. we perform detailed experiments on binary and multi-classified tabular datasets with implicit and explicit graph. empirically results demonstrate that recal has a better performance compared to traditional calibration models. in addition, experiments demonstrate that recal has remarkable robustness against data noise and distribution shift. an interesting future research work would be to use embedding of samples to guide the discovery of graph g. this extension would enable the application of recal not only to tabular data but also to textual and image data.