Evaluating Large Language Models for Confidence-based Check Set Selection (2025)

Abstract: Large Language Models (LLMs) have shown promise in automating high-labor data tasks, but the adoption of LLMs in high-stake scenarios faces two key challenges: their tendency to answer despite uncertainty and their difficulty handling long input contexts robustly.We investigate commonly used off-the-shelf LLMs’ ability to identify low-confidence outputs for human review through “check set selection”–a process where LLMs prioritize information needing human judgment.Using a case study on social media monitoring for disaster risk management,we define the “check set” as a list of tweets escalated to the disaster manager when the LLM has the least confidence, enabling human oversight within budgeted effort.We test two strategies for LLM check set selection: *individual confidence elicitation* – LLMs assesses confidence for each tweet classification individually, requiring more prompts with shorter contexts, and *direct set confidence elicitation* – LLM evaluates confidence for a list of tweet classifications at once, using less prompts but longer contexts.Our results reveal that set selection via individual probabilities is more reliable but that direct set confidence merits further investigation.Direct set selection challenges include inconsistent outputs, incorrect check set size, and low inter-annotator agreement. Despite these challenges, our approach improves collaborative disaster tweet classification by outperforming random-sample check set selection, demonstrating the potential of human-LLM collaboration.

## introduction large language models (llms) have significantly advanced the field of natural language processing (nlp) and made it possible to automate a wide range of nlp tasks such as classification, information retrieval, summarization, and many more @xcite @xcite . llms can perform these tasks by following prompts, where the enduser provides task details and input data, and the model generates a text response. however, studies show that end users tend to struggle to identify incorrect llm responses, a problem that can escalate as larger and more complex llms are less likely to refrain answering questions @xcite . the adoption of llms in high-stakes scenarios continues to be a challenge, as assuming llmgenerated responses to be always correct can have severe consequences, i.e., if incorrect outputs influence decision-making processes. previous studies evaluated llms' ability to express uncertainty which we refer to as confidence elicitation @xcite @xcite . confidence elicitation methods have shown that uncertainty estimates are closely correlated with the accuracy of the prediction @xcite . while llm's output is challenging to evaluate automatically in high-stakes scenarios, we investigate if we can surface llm incorrectness using confidence elicitation techniques. we introduce the check set for the human-llm collaboration pipeline. the check set is a list of potentially misclassified predictions by the llm needing review by the end users. while prior research has evaluated the quality of llm-generated output for escalation to human review, such efforts have typically relied on separate verifier models @xcite , task-specific fine-tuning @xcite , or probing the model @xcite . in contrast, this study introduces a novel approach in which the check set is directly selected by the off-the-shelf llm itself. in this paper, we investigate the llms' check set selection capability with a case study in the field of disaster risk management. for this use case, the check set is a list of tweets escalated to the disaster manager when the llm has the least confidence, enabling human oversight within a budgeted timeframe. llms have the potential to assist disaster managers in sifting through massive amounts of online social media data for relevant, critical, and actionable information during disaster events. with the goal of helping disaster managers, we are focused on commonly available llms that allow the disaster managers independence from a complex pipeline and the maintenance it implies. we present two methods for check set selection as seen in figure 1: (1) individual confidence elicitation: llm assesses confidence of each tweet classification separately using individual probabilities, requiring more prompts with shorter contexts and (2) direct set confidence elicitation: llm evaluates confidence for a list of tweet classifications at once which allows for comparison within the list, using fewer prompts but longer contexts. these two approaches attempt to mitigate two underlying problems of llms in high-stakes use cases, llms refusing to refrain from answering questions they may not know the answers to @xcite and llms being unable to robustly make use of information in long input contexts @xcite . previous work on selection from long-context lists @xcite @xcite , has not required the llm to identify a subset of specific items within a longer list provided in the prompt and has not explored the influence of the referencing method used for the input. intuitively, more input data and longer contexts provide llms more information i.e., the more classifications, the more comparisons llms can make to determine the potential incorrect classifications. however, recent studies show that llms struggle with long-context tasks where performance is influenced by the input order and context size @xcite @xcite . we ran our experiments using both closed and open-sourced off-the-shelf llms: gpt-4o-mini (openai, 2024a), gpt-4o (openai, 2024b), llama 3.1 8b-instruct (llama team, 2024), mistral 7b-instruct v0.3 @xcite across check set selection from predictions on two classification tasks: (1) humanitarian aid vs. not humanitarian aid and our key contributions are as follows: our results show that llms have the ability of check set selection using confidence elicitation techniques by outperforming random check set selection. individual confidence elicitation is found to be more reliable compared to direct set confidence selection. this demonstrates that the current off-the-shelf llms are not sufficiently developed for the direct set selection method. this is evidenced in section 4 demonstrating the issues in the direct set method such as providing incorrect list sizes, inconsistent outputs across different listreferencing methods, and low inter-annotator agreement. the direct set selection capabilities ought to be further explored as llms improve, as the issues pointed out need to be solved, especially when llms are to be applied in high stakes scenarios. ## method the study investigates llm's ability to select a useful check set from long-context input using confidence elicitation. first, we present the motivation of our approach and how we use llms as our disaster tweet classifiers. then, we demonstrate the two set selection methods. lastly, we deep dive on the llms direct set selection ability from long context input. problem definition. llms have been very effective in various natural language tasks. however, adoption of llms in high-stake scenarios continues to be a challenge due to two main issues: the larger and more complex the llms the less likely they are to refrain from answering questions they do not know the answer to @xcite and llms struggle with long-context tasks @xcite . we aim to mitigate these problems using check set selection by allowing llms to utilize their confidence estimates of their initial predictions to prioritize information needing human review. llm as disaster tweet classifier we test the performance of llms as disaster tweet classifiers using two classification tasks: task (1) humanitarian aid vs. not humanitarian aid -asking llms if the tweet is useful for humanitarian aid or not and task (2) humanitarian aid information classification -asking llms to classify the tweet based on the type of humanitarian aid information it contains. we ran our experiments on sixteen (16) different disaster events, task (1) with 6 disaster events with 500 tweets per event and task (2) with 10 disaster events with 300 tweets per event. more details are found in section 3.1. the selected check sets are from the initially classified list by these classifiers. set selection using individual confidence elicitation. we make use of an llm to predict the probability of the initial tweet classification from our disaster tweet classifier to be correct with a value between 0.0 and 1.0, referring to one of the methods by @xcite on confidence elicitation. we select the check set by using the tweet classifications with the lowest probabilities of being correct at the lowest 20% of the tweet classifications. the chosen check set size of 20% corresponds to the estimated effort the disaster managers have budget for, i.e., time and people to review check set. we chose a fixed check set size because it standardizes the effort done by the end users and allows us to compare across different check set selection strategies. for cut-off tweets with the same probabilities, we use random selection. ## discussion we discovered from our experiments that although we set llms to their most deterministic setting, when we do direct check set selection, changing the order of the input context (list of tweets) leads to different check set selections and can even return invalid responses. invalid responses include providing incorrect number of items from the output list (both more or less than asked), repeating items in the output, editing short-uuid's where charac- we did a quick exploration on inference time vs. effective accuracy across our two strategies. the disaster manager wants to keep the use of the model cheap, which is probably related to computational/inference time, but more practically it is related to output tokens. here, we assume that direct set selection has a clear advantage. in terms of inference time a rough estimate on a 100-tweet sample for llama to select a checks size of 20, for example takes 18.4 seconds inference time on individual confidence (100 prompts), 3 seconds on direct set (d-100), 3.33 seconds on d-50, and 3.65 ## conclusion in this paper, we investigate the ability of llms to identify low-confidence outputs for human review through check set creation, the process of utilizing llms to prioritize information needing human review. we run our experiments using a case study for social media monitoring in disaster risk management. we tested two strategies for check set selection: individual confidence elicitation by assessing confidence for each tweet classification and direct set confidence elicitation by evaluating confidence for a list of tweet classifications at once. furthermore, we examined the impact of context length, input order and referencing methods for direct set selection. our results show that llms struggle in direct set selection as they cannot consistently provide valid prompt responses, being influenced by all the three factors mentioned. hence, we say that individual confidence set selection is more reliable than direct set selection for our particular setting. however, we observe that the direct set method has potential and could be explored and evaluated further as llms continue to improve. despite these challenges, our approach improves collaborative disaster tweet classification, demonstrating the potential of human-llm collaboration. such collaboration is crucial for high stake scenarios where we want the end-user in control of the final decisions.