FESTA : Functionally Equivalent Sampling for Trust Assessment of Multimodal LLM s (2025)

Abstract: The accurate trust assessment of multimodal large language models (MLLMs) generated predictions, which can enable selective prediction and improve user confidence, is challenging due to the diverse multi-modal input paradigms. We propose \textbf{F} unctionally \textbf{E} quivalent \textbf{S} ampling for \textbf{T} rust \textbf{A} ssessment (FESTA), a multimodal input sampling technique for MLLMs, that generates an uncertainty measure based on the equivalent and complementary input samplings. The proposed task-preserving sampling approach for uncertainty quantification expands the input space to probe the consistency (through equivalent samples) and sensitivity (through complementary samples) of the model. FESTA uses only input-output access of the model (black-box), and does not require ground truth (unsupervised). The experiments are conducted with various off-the-shelf multi-modal LLMs, on both visual and audio reasoning tasks. The proposed FESTA uncertainty estimate achieves significant improvement (33.3% relative improvement for vision-LLMs and 29.6% relative improvement for audio-LLMs) in selective prediction performance, based on area-under-receiver-operating-characteristic curve (AUROC) metric in detecting mispredictions. The code implementation is open-sourced.

## introduction large language models (llms) have achieved remarkable performance across a wide array of natural language processing tasks @xcite @xcite . however, their performance in integration with other modalities-called, multimodal llms (mllms), is often inferior compared to text-only llms @xcite . ## problem statement while festa is applicable to general multi-modal outputs, we make a few assumptions and restrictions. we restrict this paper to the purely textual output case, in a multi-choice question-answering (mcqa) setting. further, it is also assumed that the mllms under study are instruction-tuned well enough to follow instructions to generate singletoken mcqa outputs. now, we define the problem statement, followed by formulating the festa algorithm. a summary of the notations used is present in appendix problem statement: given multi-modal input x and model output ŷ generated from inherent predictive distribution q(y|x), estimate the predictive uncertainty of ŷ without access to q(y|x) in a black-box setting. we resort to the following directions to develop the uncertainty estimator, • sampling fes to estimate the epistemic uncertainty (consistency). ## festa uncertainty estimator in this section, we describe the sampling processes, and the computation of festa uncertainty score. ## conclusion the proposed multimodal uncertainty estimation algorithm, festa, is presented as a principled and formally grounded framework for selective prediction by mllms. firstly, it introduces a novel input sampling paradigm based on functional equivalence (fes) and complementarity (fcs), a previously unexplored space, and demonstrates its effectiveness for abstention in the unsupervised and black-box settings. secondly, it proposes the kldistance from a hypothetically ideal model as an uncertainty score over the widely used entropy. festa enables accurate abstention from incorrect predictions in both audio-llms and vision-llms, elicited by improved auroc values. given the limited grounding of current mllms (especially lightweight mllms), they tend to produce biased, low-uncertainty hallucinations. a key contribution of festa is its ability to detect and abstain from such hallucinations. this acts as the core reason behind the substantial improvements in selective prediction performance across models. building on these findings, we plan to extend festa to support open-ended generation and multimodal outputs in mllms, beyond mcqa tasks.