Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge (2023)

Abstract: The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually. One important step towards this goal is to localize and track key active objects that undergo major state change as a consequence of human actions/interactions to the environment without being told exactly what/where to ground (e.g., localizing and tracking the ‘sponge‘ in video from the instruction “Dip the sponge into the bucket.”). While existing works approach this problem from a pure vision perspective, we investigate to which extent the textual modality (i.e., task instructions) and their interaction with visual modality can be beneficial. Specifically, we propose to improve phrase grounding models’ ability on localizing the active objects by: (1) learning the role of ‘objects undergoing change‘ and extracting them accurately from the instructions, (2) leveraging pre- and post-conditions of the objects during actions, and (3) recognizing the objects more robustly with descriptional knowledge. We leverage large language models (LLMs) to extract the aforementioned action-object knowledge, and design a per-object aggregation masking technique to effectively perform joint inference on object phrases and symbolic knowledge. We evaluate our framework on Ego4D and Epic-Kitchens datasets. Extensive experiments demonstrate the effectiveness of our proposed framework, which leads to>54% improvements in all standard metrics on the TREK-150-OPE-Det localization + tracking task, >7% improvements in all standard metrics on the TREK-150-OPE tracking task, and >3% improvements in average precision (AP) on the Ego4D SCOD task.

## introduction recent technological advancements in smart glasses (and headsets) from industry leaders such * the authors contribute equally, alphabetical order. ## experiments and analysis we adopt the glip-l variant (and its pretrained weights) for all of our experiments, where its visual encoder uses the swin-l transformer @xcite . we train the glip-l with our framework primarily on the scod dataset, and perform a zeroshot transfer to the trek-150 task. ## conclusions in this work, we approach the active object grounding task leveraging two narrated egocentric video datasets, ego4d and epic-kitchens. we propose a carefully designed prompting scheme to obtain useful action-object knowledge from llms (gpt), with specific focuses on object pre-/post-conditions during an action and its attributional descriptions. enriching the glip model with the aforementioned knowledge as well as the proposed perobject knowledge aggregation technique, our models outperforms various strong baselines in both active object localization and tracking tasks.