Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions (2024)

Abstract: Visual grounding (VG) aims at locating the foreground entities that match the given natural language expression. Previous datasets and methods for classic VG task mainly rely on the prior assumption that the given expression must literally refer to the target object, which greatly impedes the practical deployment of agents in real-world scenarios. Since users usually prefer to provide the intention-based expressions for the desired object instead of covering all the details, it is necessary for the agents to interpret the intention-driven instructions. Thus, in this work, we take a step further to the intention-driven visual-language (V-L) understanding. To promote classic VG towards human intention interpretation, we propose a new intention-driven visual grounding (IVG) task and build a largest-scale IVG dataset named IntentionVG with free-form intention expressions. Considering that practical agents need to move and find specific targets among various scenarios to realize the grounding task, our IVG task and IntentionVG dataset have taken the crucial properties of both multi-scenario perception and egocentric view into consideration. Besides, various types of models are set up as the baselines to realize our IVG task. Extensive experiments on our IntentionVG dataset and baselines demonstrate the necessity and efficacy of our method for the V-L field. To foster future research in this direction, our newly built dataset and baselines will be publicly available at https://github.com/Rubics-Xuan/IVG.

## introduction recently, the research community has witnessed the rapid advancement of multimodal embodied intelligence @xcite @xcite @xcite . for an intelligent agent, the capability of locating the target objects in the unpredictable open-world scenarios based on natural language expressions is crucial, underscoring the importance of visual grounding (vg) task within the broader context. notably, instructions provided by users often encapsulate their genuine needs through nuanced intention-driven expressions, which are usually not literal or explicit with much details as classic vg task. this nuance brings to light the critical role of vg based on user intention expressions, where challenge lies in interpreting and responding to user commands in a way that truly reflect their underlying desires, transcending surface-level expressions to foster more flexible and understanding human-machine interactions. contrary to the classic vg task, realizing intention-based grounding involves several unique aspects that require consideration. 1) intention-driven descriptions: since humans tend to provide intention-based expressions to get the desired objects rather than detailing every aspect, it is imperative for intelligent agents to interpret these intention-driven instructions and act accordingly, focusing on the semantic core of the requests rather than their literal descriptions. however, previous studies in this field have primarily concentrated on literal textual descriptions, with scant attention to understanding user intentions. 2) egocentric perspective: as explored in prior studies @xcite @xcite , the practical agents actually receive all visual information from a first-person view. however, most classic vg datasets are predominantly collected from third-person perspective, which greatly deviate from the application contexts of an embodied agent. 3) multi-scene perception: given that agents are expected to navigate and identify specific targets across diverse scenarios in the real world, the capability to perceive and interact within dynamic multi-scene environments is crucial for accurately accomplishing visual grounding task. yet, most prior research on vg has overlooked this therefore, in this work, we attempt to fill this important blank space that has been neglected before and move towards intention-oriented visual grounding. specifically, based on classic vg, we propose a new intention-driven visual grounding (ivg) task to push towards intention-oriented vision-language understanding, which requires the models to identify the corresponding scene and target object that match the intent expressions from the given multiscene input. to solve the data scarcity problem of intention-oriented grounding task, we build a largescale grounding dataset termed intentionvg which is also the first grounding dataset to support freeform intention expressions. besides, we also con- • we build an intention-oriented grounding benchmark named intentionvg, which to the best of our knowledge is the first large-scale intention-driven grounding dataset that supports free-form intention-based vision-language annotations. vision-language complex reasoning aims at understanding intricate textual-visual input information and accomplishing the vision-language (v-l) tasks based on reasoning, in which broader knowledge and strong expression comprehension ability are essential thus posing a greater challenge compared with conventional v-l tasks. due to the reasoning ability and rich prior knowledge in large language models (llms), numerous methods such as @xcite @xcite @xcite leverage llms to understand complex instructions. however, prior works mainly focus on literal description reasoning, falling short in understanding potential user intentions. recently, lisa @xcite introduces a challenging res benchmark that incorporates complex expressions, and rio @xcite proposes a new iood dataset that includes the specific affordance of the objects. the format of rio's sentences is "something can be used...", which fails to describe the intentions directly from the user's perspective but simply describes the affordance of the target objects. therefore, in this work, starting from a perspective that aligns more closely with real-world scenarios, we for the first time integrate egocentric viewpoint of data collection, multi-scene perception, and free-form expression of human intentions to construct a new ivg task with a corresponding intentionvg benchmark dataset, thereby enhancing llms' reasoning capabilities to better understand human intentions. ## ivg task & intentionvg dataset in this section, we first introduce our ivg task's definition (sec. 3.1) and present collection pipeline for intentionvg data (sec. 3.2). then, the specific details and evaluation metrics about our inten-tionvg are provided (sec. 3.3 and sec. 3.4). ## baseline construction to realize our ivg task based on multi-scene perception and egocentric viewpoint, we formulate two different kinds of baseline models below. more illustrations of the baseline structures can be found in appendix sec. a.3 for better understanding. ## experiments to evaluate the effectiveness and the designing rationale of our data and baseline models, comprehensive experiments are conducted on our built intentionvg dataset for the new ivg task. ## conclusion and broader impact in this paper, we move beyond previous works that focused solely on literal description based grounding and take a step further to intention-driven v-l understanding. specifically, by considering that the practical agents need to move and search for expected targets among different scenarios, we put forward a new ivg task. the ivg task requires agents to interpret user intentions and locate specific targets based on egocentric view and multiscene perception. to enable existing models to better accomplish our ivg task and assess their capabilities on it, we build the first also the largestscale intention-driven grounding dataset termed in-tentionvg with free-form expressions and develop a series of baseline to accomplish the ivg task. comprehensive experiments conducted on our intention dataset demonstrate that most previous methods struggle to directly understand users' nonliteral intent expressions and locate the expected targets. with the aid of our data, there is a significant enhancement in the ability to comprehend intentions for ivg task, but there remains substantial room for improvement, warranting further investigations by the research community. aspiring to foster future research into intention-oriented grounding and inspire new research in this direction, we plan to release our newly built intentionvg dataset and baseline models to the community.