A Conformal Risk Control Framework for Granular Word Assessment and Uncertainty Calibration of CLIPS core Quality Estimates (2025)

Abstract: This study explores current limitations of learned image captioning evaluation metrics, specifically the lack of granular assessments for errors within captions, and the reliance on single-point quality estimates without considering uncertainty. To address the limitations, we propose a simple yet effective strategy for generating and calibrating distributions of CLIPScore values. Leveraging a model-agnostic conformal risk control framework, we calibrate CLIPScore values for task-specific control variables, tackling the aforementioned limitations. Experimental results demonstrate that using conformal risk control, over score distributions produced with simple methods such as input masking, can achieve competitive performance compared to more complex approaches. Our method effectively detects erroneous words, while providing formal guarantees aligned with desired risk levels. It also improves the correlation between uncertainty estimations and prediction errors, thus enhancing the overall reliability of caption evaluation metrics.

## introduction image captioning (ic) evaluation is a crucial task in vision-and-language research, aiming to assess how accurately textual descriptions represent visual contents. reference-free metrics such as clip-score @xcite , which measure caption quality by computing the cosine similarity between image and text embeddings, have been shown to correlate strongly with human judgments. however, simply scoring captions is often insufficient, as these quality assessments can be hard to interpret or unreliable. in many cases, effective evaluation requires not only an overall score of caption quality, but also the detection of specific errors within the caption. without this granular information, the assessment can seem incomplete or less useful. beyond the lack of granularity, existing metrics provide ic quality assessments relying on single-point estimates, in the sense that a single value is used as an evaluation score, without incorporating any indication of confidence over the results. this absence of uncertainty quantification can give way to misleading scores, reducing user trust. to address these challenges, we propose a conformal risk control framework, aiming at taskspecific calibrated predictions, in conjunction with a simple yet effective strategy for generating distributions over clipscore predictions. this provides us with a principled way to adapt ic evaluation both to fine-grained analysis for each caption, and to a broader view of performance over a dataset, allowing for user-defined criteria to determine risk. first, we improve interpretability by detecting foils, which are words in the caption that are not associated with the image. second, we overcome the limitations of single-point evaluation by introducing well-calibrated intervals, providing a trustworthy measure of caption reliability. experimental findings demonstrate that using conformal risk control, over the distributions produced with simple methods for expressing uncertainty, such as masking parts of the input, can achieve competitive performance on foil detection compared to more complex and specialized approaches. conformal risk control can also provide improvements in the correlation between uncertainty estimations and prediction errors, enhancing the overall reliability of caption evaluation metrics. furthermore, we emphasize that other existing state-of-the-art methods can also benefit from our conformal calibration framework, gaining formal guarantees over their results. the proposed methodology is model-agnostic, and our work underscores the adaptability and broad applicability of risk control, offering a compelling case for its integration into vision and language research. the code is available in a public github repository foot_0 . ## related work this section reviews prior work relevant to our research, organized into three key areas: uncertainty quantification, conformal risk control, and image captioning evaluation. ## from point estimates to distributions techniques like deep ensembles or monte carlo (mc) dropout, commonly used to construct output distributions from instance regressor systems, are not fully model-agnostic approaches, nor are they suited to our specific objectives. deep ensembles are unsuitable since we aim to measure the uncertainty of scores produced with individual publicly available clip models, without further training. in turn, mc dropout is impractical since clip models generally lack dropout layers. we propose an alternative strategy for producing score distributions that express uncertainty, leveraging attention masks over the clip vision and text encoders to generate output distributions, by randomly masking portions of the input data. the top part from figure 1 illustrates this procedure. we create i samples for images by randomly masking ξ i % of the image patches from the computations associated to the self-attention operations. for captions, we generate t samples by randomly masking ξ t % of the tokens, corresponding to words of specific parts of speech, namely nouns, proper nouns, numerals, verbs, adjectives, and adverbs. this strategy allows us to produce i image embeddings and t text embeddings, which can be combined to compute i × t different clipscore values, following the procedure outlined in appendix a. the clip model we used relies on a robertabased text encoder, which may split a word into multiple subword tokens, while our pos tagging two dogs run through a field a little boy that is holding a hot dog a group of people that are standing next to each other model foot_1 operates at the word level. to reconcile word-level pos tagging with subword tokenization, we first map each subword token to its corresponding word. during sampling, we randomly select the words we want to mask and then identify the subword tokens associated with each sampled word. the attention for all the corresponding tokens is then masked accordingly. figure 2 presents violin plots of the clipscore distributions produced according to our method. ## conformal detection of caption errors in this section, we describe the application of conformal risk control for detecting foil words in image-caption pairs. the overall approach is illustrated in the bottom-left part of figure 1 . leveraging the attention mask sampling method that was described in section 3, we can calibrate a control variable λ that acts as a threshold to identify foil words in the caption. empirical results show that this method provides good performance across several well-established benchmarks in the field @xcite @xcite . furthermore, we compare the results of our simple, yet robust and well-calibrated method, against more complex, specialized, and state-of-the-art approaches, underscoring its advantages and overall effectiveness. ## conformalized intervals for clipscore we now test a second application of risk control over clipscore values, to address the limitations of single-point evaluation metrics in image captioning (ic) assessments, aiming to get reliable and interpretable confidence intervals for each ic score. leveraging the uncertainty quantification method described in section 3, we fit a truncated gaussian distribution to construct intervals. these intervals help quantify model uncertainty more effectively, providing a nuanced and trustworthy assessment of caption quality. the overall approach is illustrated in the bottom-right part of figure 1 . the choice of using truncated gaussian distributions is motivated by clipscore being inherently bounded, as the metric is defined as a modified cosine similarity. in addition, it allows us to define a more meaningful rescaling of initially estimated uncertainties, effectively reordering confidence intervals to align with the deviation from ground truth, as described in the following sections. ## conclusions we proposed methods for producing and calibrating distributions on clipscore assessments, enabling granular caption evaluation and uncertainty representation. we leverage conformal risk control to address word-level foil detection and confidence estimation, allowing for flexible, task-specific riskcontrol with formal guarantees. the experimental results demonstrate competitive performance against more complex models on several wellestablished benchmarks. the proposed methods achieve controllable and trustworthy performance in detecting foil words, and improved correlation between uncertainty estimates and prediction errors, without compromising human rating alignment. our work highlights the potential of conformal calibration in enhancing the robustness and reliability of vision-and-language evaluation metrics.