A lana VLM : A Multimodal Embodied AI Foundation Model for Egocentric Video Understanding (2024)

Abstract: AI personal assistants deployed via robots or wearables require embodied understanding to collaborate with humans effectively. However, current Vision-Language Models (VLMs) primarily focus on third-person view videos, neglecting the richness of egocentric perceptual experience. To address this gap, we propose three key contributions. First, we introduce the Egocentric Video Understanding Dataset (EVUD) for training VLMs on video captioning and question answering tasks specific to egocentric videos. Second, we present , a 7B parameter VLM trained using parameter-efficient methods on EVUD. Finally, we evaluate ‘s capabilities on OpenEQA, a challenging benchmark for embodied video question answering. Our model achieves state-of-the-art performance, outperforming open-source models including strong Socratic models using GPT-4 as a planner by 3.6%.Additionally, we outperform Claude 3 and Gemini Pro Vision 1.0 and showcase competitive results compared to Gemini Pro 1.5 and GPT-4V, even surpassing the latter in spatial reasoning. This research paves the way for building efficient VLMs that can be deployed in robots or wearables, leveraging embodied video understanding to collaborate seamlessly with humans in everyday tasks, contributing to the advancement of next-generation Embodied AI.

## introduction embodied cognition posits that our understanding of the world is fundamentally shaped by our physical bodies and their interaction with the environment @xcite . humans leverage this embodied understanding to intuitively grasp physical tasks, anticipate actions, and communicate effectively through nonverbal cues. for robots and ai systems to become true collaborators, they too must develop a similar understanding. egocentric understanding of video data also has key applications in areas such as wearable computing, vr and ar, and video game technology. in the scenario of an embedded artificial personal assistant, e.g., smart glasses (or a wearable camera for blind and partially sighted people), that can support the user in providing responses to visual queries, we want to build ai systems that can understand videos of the user's activities and of their visual-spatial environment. for this task, it becomes essential that the model is able to receive as input a sequence of frames before generating an answer. recently, by leveraging pre-trained powerful large language models (llms), vision-language models (vlms) have been proposed by using adapters that fuse representations generated by visual experts with textual tokens that can be manipulated by text-only language models (e.g., @xcite . following this approach, vlms have been extended to handle video understanding tasks as well (e.g., @xcite . however, most of these models have been developed using datasets that include a majority of third-person view videos only ignoring the importance of modelling egocentric videos (e.g., @xcite @xcite . as demonstrated by @xcite , modelling both perspectives is challenging, and dedicated data creation efforts are required to distil this capability into vlms. in this paper, we provide a recipe for building vlms that can solve tasks involving egocentric videos by extending existing video-based vlms which are trained only on third-person view videos. concretely, we present three main contributions: 1) we introduce the egocentric video understanding dataset (evud), a collection of egocentric video caption generation and video questionanswering tasks that can be used for instructiontuning video-based vlms, which underwent a rigorous human evaluation, 2) we leverage parameterefficient training to extend existing vlms and train alanavlm using a limited computational budget; 3) we extensively evaluate different model variants on openeqa @xcite , a challenging real-world benchmark for embodied video question-answering, and achieve state-of-the-art results compared to similarly-sized open-source models and competitive performance with much larger, proprietary variants. we also conducted rigorous human evaluation and quality control of a large portion of evud and elicited an error analysis on our system outputs that we hope will inform the next generation of egocentric video-based vlms. ## evud: egocentric video understanding dataset we developed the egocentric video understanding dataset (evud) to train vlms for egocentric video question-answering tasks. this dataset includes 29,477 examples and its components are described below (see figure 2 for an overview). ## model training we build alanavlm by fine-tuning chat-univi (jin et al., 2024) -a vision & language foundation model equipped with video understanding capabilities -on evud. this fine-tuning step is essential for injecting the egocentric video understanding skills that are unique to alanavlm. we decided to build our model starting from chat-univi for several reasons. first, it is an open-source model whose code and weights are publicly available. second, it is designed for handling language, images, and videos, taking an arbitrary number of frames into account. third, it outperforms other opensource vision and language foundation models in classic video understanding tasks @xcite . in the following, we describe the fine-tuning recipe that we used to build our model, trying to preserve the original capabilities that were distilled during the instruction tuning stage. we mitigate the forgetting of previously learned skills by leveraging rehearsal @xcite , which consists in the retraining of the model on a small percent-age of the previously learned information as the model is trained on new information. we fine-tune our model using low-rank adaptation (lora; hu et al., 2021), which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. in all our experiments, we fine-tune alanavlm on rehearsal data and evud. see appendix h for training details. considering that we are using parameterefficient methods such as lora, they do not require a large dataset due to the smaller number of tunable parameters. however, our gemini dataset generation procedure is generic enough to be used for generating much larger datasets, which we make publicly available on github and could be used by others in the future to make larger datasets if desired. ## evaluation & results for our evaluation, we use the episodic memory use case of the openeqa benchmark @xcite . openeqa uses gpt-4 to rank the appropriateness of the generated answers concerning the ground-truth answers. to favour reproducibility, we use the highly capable open-weight model llama-3 70b (ai@meta, 2024). it's important to note that openeqa is used for testing only. at training time there is no overfitting happening because the model sees completely different videos. we only use the openeqa categories (which are standard visual understanding tasks) to narrow down the type of questions that gemini can generate to make it more controllable, whilst using the ego4d videos as input. additionally, we chose not to evaluate our model on the egoschema benchmark @xcite as it was derived from ego4d videos, which were used by our model for training. additionally, egoschema is composed of multiple choice questions, instead of free-form answers, which is what we trained on. to derive alanavlm's best configuration, we compare several dataset mixtures to assess the importance of each dataset subset in evud as well as the importance of different model parameters. we report additional details (including error statistics) in appendix i and table e in jin et al. 2024). we notice that most models perform better on sn than on hm3d, probably because of its higher quality. adding hm3d to the alanavlm training doesn't help either, maybe because its descriptions are not fine-grained enough. finally, we note that the most recent version of gemini surpasses all other frontier models in this task presumably due to its ability to encode higher-resolution video frames leveraging its 1m context length. error analysis to gain further insights into alanavlm predictions, we perform an error analysis based on 98 qa pairs and derive a categoriza-tion of the errors. we find that 60% of answers are incorrect. we notice that the pearson correlation between human and llm ratings was 0.76. moreover, in roughly 7% of cases, the llm said that the answers were wrong even though humans noticed that both answers applied. as shown in figure 3 , most errors concern visual attributes (e.g., object colours) or hallucinations (e.g., missing objects). additionally, we find that alanavlm struggles with spatial reasoning which is required to understand the relationships between objects (error @xmath0. we note that in a few cases alanavlm generates answers that are not aligned with the camera wearer's egocentric point of view (error @xmath1. this highlights the need for more robust visual encoders for vlms that can capture fine-grained details of the visual scenes when trained with egocentric vision perception @xcite . finally, we also highlight the problem of current vlms being overpowered by the original llm probability distribution which produces not only hallucinations but answers that are not visually grounded or which indirectly answer the question @xcite . ## conclusions in this work, we focused on egocentric video understanding to contribute to the development of the next generation of wearables and robots that can effectively help humans in their daily tasks. our work offers three main contributions. first, we created a comprehensive dataset for egocentric video understanding including video question answering and video captioning pairs from egocentric videos. second, we show how existing vlms can be extended to deal with egocentric videos through parameter-efficient training on evud. in this way, even with limited computational resources, it is possible to extend vlms to handle egocentric videos. third, we evaluated different model variants using openeqa, a benchmark for embodied video question answering including a variety of household environments. despite having only 7b parameters, the best-performing alanavlm not only achieved state-of-the-art results compared to opensource models but also demonstrated competitive performance against much larger, proprietary models. including video caption generation and video question-answering.