Confidence Calibration in Large Language Model-Based Entity Matching (2025)

Abstract: This research aims to explore the intersection of Large Language Models and confidence calibration in Entity Matching. To this end, we perform an empirical study to compare baseline RoBERTa confidences for an Entity Matching task against confidences that are calibrated using Temperature Scaling, Monte Carlo Dropout and Ensembles. We use the Abt-Buy, DBLP-ACM, iTunes-Amazon and Company datasets. The findings indicate that the proposed modified RoBERTa model exhibits a slight overconfidence, with Expected Calibration Error scores ranging from 0.0043 to 0.0552 across datasets. We find that this overconfidence can be mitigated using Temperature Scaling, reducing Expected Calibration Error scores by up to 23.83%.

## introduction entity resolution (er) can be defined as the task of determining which data entries across different data sources refer to the same real-world entity. a key sub-task of er is entity matching (em), which specifically addresses the binary classification problem of determining whether pairs of data entries from different sources refer to the same entity @xcite . in today's data-driven era, em plays a critical role in various domains, including the medical field @xcite , where accurate matching can improve patient care; the reconstruction of historical populations by linking birth, marriage, and death records @xcite ; and law enforcement, where matching data entries is vital for investigations and crime prevention @xcite . the state-of-the-art methods for performing em utilize transformer-based architectures @xcite , pre-trained large language models (llms) @xcite @xcite , in addition to classifying each entry pair as a 'match' or 'no match,' the model also generates a score that should reflect the model's confidence in its prediction. @xcite @xcite , such as roberta @xcite and @xcite . however, while these models are successful, they have, in other natural language processing tasks, shown to struggle to accurately express their confidence in predictions and can exhibit overconfidence @xcite . ideally, a model provides information about its certainty alongside its predictions. for example, in a binary em task, a model would output a 'match' or 'no match' prediction label alongside a probability, or confidence score, that is reliable. refining models' predicted confidence scores to ensure that they accurately reflect the true likelihood of the predictions is called confidence calibration. while the topic of confidence calibration on llms has been explored @xcite @xcite , the intersection of confidence calibration, llms, and the application of em has not yet been researched. yet, confidence calibration is important as it provides transparency over models' results @xcite . for example, the distribution of confidence in a model's em predictions can give the user insights into the model's overall reliability in the task. precise confidence scores can also play a crucial role in guiding subsequent tasks. moreover, confidence scores can be used to help researchers better understand a model's inner workings. finally, they can help in improving a model: when it is clear in what specific cases a model is uncertain, it is easier to see a model's weak points and with that, possible areas for improvement. contributions. this paper aims to explore the confidence calibration performance of llms in em and benchmark confidence calibration methods to enhance their performance. we focus on pretrained roberta @xcite as the llm of interest as it has a competitive performance among llm models for em @xcite @xcite . in contrast to other state-ofthe-earth models for em, roberta is open-source and lightweight. our study assesses the confidence calibration performance in em using the expected calibration error (ece) as the primary metric. we evaluate fine-tuned roberta model's ece scores both with and without the use of confidence calibration methods and investigate which methods yield the greatest improvement. since confidence calibration methods may influence the model's predictions, we additionally examine their effect on the f 1 score to ensure that improved confidence calibration does not come at the cost of classification performance. furthermore, we analyze confidence histograms, reliability diagrams, the maximum calibration error (mce) and the root mean square calibration error (rmsce). the confidence calibration methods tested are temperature scaling @xcite , monte carlo dropout @xcite , and ensembles @xcite . we use the abt-buy, dblp-acm (dirty and structured) @xcite , itunes-amazon (dirty and structured), and company @xcite datasets, ensuring diversity in terms of data content, size and structure. figure 1 presents an overview of the proposed modified roberta model used in this research. as shown, the goal is to obtain confidence scores that accurately reflect the model's confidence in its em predictions. confidence calibration methods can help improve these scores. ## confidence calibration we say that a model is well-calibrated if its prediction's confidence scores accurately reflect the probability of those predictions being correct. for em, for example, all pairs that are predicted to match with around 0.5 to 0.6 confidence should be actual matching pairs 50 to 60% of the time. this is also referred to as the alignment between the 'predicted probability' (the confidence) and the 'empirical probability' @xcite @xcite . generally, for a binary classification task such as em, the 'confidence' signifies the confidence of a prediction belonging to the positive class (in the case of em: a 'match'). the predicted probability of the positive class then needs to align with the empirical probability of the positive class. 'high confidences', in this context, generally denote predicted probabilities close to either 0 or 1, while 'low confidences' denote predicted probabilities close to 0.5. the confidence calibration of models has been evaluated by plotting confidence histograms and reliability diagrams, and by measuring the expected calibration error (ece) @xcite or similar metrics such as the maximum calibration error (mce) @xcite and root mean square calibration error (rmsce) @xcite . intuitively, these scores measure the difference between the predicted probability and the empirical probability, and should therefore be minimized to optimize the confidence calibration. compared to the ece, the mce is useful in production settings where reliable confidence measures are absolutely necessary due to high risks. this is due to its measure of the worst-case deviation between the predicted probabilities and the empirical probabilities. when comparing the ece to the rmsce, the latter places a greater emphasis on larger errors. ## results appendix a presents a more detailed overview of the performance of the baseline roberta model in terms of f 1 score, precision, recall and inference time. ## conclusions we compare the confidence calibration of baseline roberta probability predictions without any use of confidence calibration methods, to the confidence calibration using temperature scaling, monte carlo dropout and ensembles as confidence calibration methods for em. we find that the ece performance and overall confidence calibration performance for roberta's performance on em, without using any confidence calibration methods, is reasonable, but often overconfident, with ece scores ranging from 0.0043 to 0.0552, leaving room for improvement. we find temperature scaling to work best, compared to monte carlo dropout and ensembles, in improving a roberta model's eces for em, reducing ece scores by up to 23.83%. this is a simple method that can easily be implemented in practical settings. we find that neither temperature scaling, monte carlo dropout, nor ensembles have consistently significant effects on the f 1 scores of the the roberta em model. overall, the eces reported for the baseline roberta em model results are slightly higher than those reported for roberta by @xcite and slightly lower to those reported for roberta by xiao and colleagues @xcite . both studies applied the model to natural language processing tasks other than em. it would be interesting for future research to investigate the cause of these differences in metric values. another avenue for future research is to combine confidence calibration methods for em. for example, @xcite found that using ensembles, and applying temperature scaling to the averaged sigmoid scores can reduce ece scores by half compared to just using ensembles, on image classification tasks. temperature scaling could be combined with monte carlo dropout in the same way. additionally, future work could leverage the individual variances in the sigmoid scores produced by monte carlo dropout and ensembles. if these variances are high, the confidence levels can be lowered accordingly, potentially improving calibration. by incorporating variance-based adjustments, it might be possible to create more reliable confidence estimates and further enhance the overall performance of the roberta pipeline. additionally, entry pairs with large variances in their sigmoid scores can be more closely analyzed to gain deeper insights into the pipeline's prediction patterns.