OSC a R : Object State Captioning and State Change Representation (2024)

Abstract: The capability of intelligent models to extrapolate and comprehend changes in object states is a crucial yet demanding aspect of AI research, particularly through the lens of human interaction in real-world settings. This task involves describing complex visual environments, identifying active objects, and interpreting their changes as conveyed through language. Traditional methods, which isolate object captioning and state change detection, offer a limited view of dynamic environments. Moreover, relying on a small set of symbolic words to represent changes has restricted the expressiveness of language. To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark. OSCaR consists of 14,084 annotated video segments with nearly 1,000 unique objects from various egocentric video collections. It sets a new testbed for evaluating Multimodal Large Language Models (MLLMs). Our experiments demonstrate that while MLLMs show some skill, they lack a full understanding of object state changes. The benchmark includes a fine-tuned model that, despite initial capabilities, requires significant improvements in accuracy and generalization ability for effective understanding of these changes. Our code and dataset are available at https://github.com/nguyennm1024/OSCaR.

## introduction the field of natural language processing (nlp) has evolved beyond mere text interpretation and generation, advancing into realms where understanding and interacting with the physical world becomes imperative. from studying causal reasoning @xcite to building a world model for cause-effect prediction @xcite , researchers have been working on the problem of causation in the physical world. in this paper, we investigate the very basic causal relations between a concrete action and the change our method achieves near parity with gpt-4v ratings across helpfulness, accuracy, reasoning, and other key metrics. imagining a scenario where artificial agents collaborate with humans in the physical world, they will need to understand the physical action effect to reason, learn, and assist humans @xcite . to empower machines with such capabilities, this paper introduces a novel benchmark focusing on understanding object state changes from egocentric visual inputs, which has the advantage of the lens of human eyes. understanding object state change is not only a complex task but also practical and foundational question: what does the sequence of movements of the pear slice across the cutting board indicate about the method of preparation? question: what does the positioning of the pear slice near the edge of the cutting board, and the trail it has left, suggest about its handling and the preparation process? question: describe state of the pearl in detail way. question: what details about the pear slice's appearance give insight into the preparation method and the fruit's natural texture? answer: a slice of pear rests on a white cutting board, its surface a palimpsest of knife marks that speak to the act of peeling and cutting. the pear's flesh is a creamy white with a slightly granular texture, typical of this fruit, and it has a subtle sheen, suggesting juiciness. the edges where the skin has been removed are slightly ragged, revealing the methodical but imperfect passage of a blade. the pear's shape is irregular, with curves that follow the natural form of the fruit, and the remnants of the stem cavity indicate it has been sliced from top to bottom. ## the oscar dataset this section outlines our pipeline for creating visual instructions on object states. we begin with the process of collecting diverse visual data from public sources, detailed in section 3.1. following this, section 3.2 describes our approach to enhancing data quality using simple human annotations across various tasks, facilitating a deeper understanding of object states. our method enables the generation of detailed captions, visual question answering, and visual dialogue. ## data statistics in order to help models generate concise and informative answers, we have defined short answers as those with less than ten words and long answers as those with more than ten words. short answers provide brevity, while long answers offer detailed and informative information. the distribution of these two types of answers can be seen in figure 3 . the average answer length in the dataset is 47.06 words. long answers make up about 75% of the data, with an average length of 63 words, while short answers account for about 25% of the data, with an average length of 3.32 words. by splitting the data accordingly, future models can provide short, direct, and informative answers with explanations. to showcase the uniqueness of our oscar dataset, we have presented a comparison between oscar and other related datasets in in section 4.2, we discussed two types of openworld datasets for object state understanding: indomain cooking and open domains. although we trained on videos with object state changes, in open-world evaluation, we tested the models on both types of videos, with and without object state changes, to ensure their generalizability. the in-domain evaluation set consists of 2,485 videos with 1,024 novel objects extracted from epic-kitchens. we have extracted an open-domain evaluation set from the ego4d dataset. the top 10 most frequent domains in the open-world testing set are shown in figure 4 . this evaluation set from 51 different domains, contains annotations for domain, action, object name, and action narrations extracted from annotations of ego4d. the set includes 43,367 open-world videos for which we know their domains and 56,231 videos of unknown domains, but we still have information about their object names and action narrations. thus, this set can be utilized not only for open-world evaluation but also for the advancement of general domain object-state understanding in the future when applying our method to generate labels. this set of data hasn't been annotated, but the data we extracted from ego4d are ready to use our pipeline to scale up the data generation. ## experiments in this section, we will discuss the experimental design we used and how we trained our model. our fine-tuning process will be described in section 6.1. additionally, we included other vision language models such as blip @xcite , blip2 @xcite , lavila @xcite , and video-llama @xcite for comparison purposes. firstly, we will evaluate our model's performance in the cooking domain in section 6.3. after that, we will also evaluate its performance in an open-world setting in section 6.4. ## conclusion this paper presents a new task for comprehending the state of objects and their changes using natural language. we also propose a data generation pipeline that utilizes the capabilities of gpt-4v to tackle this task. furthermore, we introduce oscar, a dataset that includes training data and a benchmark with various protocols. our comprehensive experiments not only demonstrate the superiority of our methods in comparison to previous state-ofthe-art open-source solutions but also examine the limitations of gpt-4v in addressing this challenge.