From Observation to Understanding: Front-Door Adjustments with Uncertainty Calibration for Enhancing Egocentric Reasoning in LVLM s (2025)

Abstract: Recent progress in large vision-language models (LVLMs) has shown substantial potential across a broad spectrum of third-person tasks. However, adapting these LVLMs to egocentric scenarios remains challenging due to their third-person training bias. Existing methods that adapt LVLMs for first-person tasks often overlook critical agent-environment interactions, limiting their ability to perform egocentric reasoning. To address these challenges, we propose a novel zero-shot paradigm termed Front-Door Adjustments with Uncertainty Calibration (FRUIT) to enhance the egocentric reasoning abilities of LVLMs by simulating human causal reasoning. Specifically, the FRUIT operates in two stages: observation and understanding. Unlike conventional prompting techniques, we formalize egocentric reasoning using a structural causal model. Then, we ground interaction regions and expand them into hierarchical visual cues, augmented with corresponding captions, to form the initial observations. To reduce noise in these observations, we employ uncertainty calibration to filter out unreliable information. These refined observations as mediators are then incorporated into the prompt template, guiding the model to understand semantics from a first-person perspective. Extensive experiments conducted on the EgoThink benchmark demonstrate that our FRUIT method consistently enhances the performance of existing LVLMs on six distinct tasks. Our code is available at https://github.com/Mrshenshen/FRUIT.

## introduction recently, large vision-language models (lvlms) @xcite have made significant progress in third-person scene understanding tasks @xcite . meanwhile, the growing research interest in egocentric tasks has driven the expansion of lvlms into domains such ## proposed method preliminary overview. our primary goal is to enhance the egocentric reasoning of lvlms to generate precise responses in first-person contexts. given an input prompt x including an image i and text t , we first employ a grounding model to derive hierarchical visual egocentric cues h r and textual cues h c , which can be considered as mediated observations. after filtering noise in the observation, we incorporate observations into the prompt template following the front-door criterion to obtain a set of answer candidates a. finally, we compute the clip score between each candidate in a and the input image, selecting the candidate with the highest score as the final output o. ## conclusion in this work, we introduced the front-door adjustments with uncertainty calibration (fruit) framework to enhance the egocentric reasoning capabilities of large vision-language models (lvlms). by incorporating egocentric front-door adjustments, fruit guide the model focus on interaction objects, while the uncertainty calibration mechanism filters out unreliable information. in future work, we will explore additional strategies to further enhance lvlm performance in first-person contexts.