i MOVE : Instance-Motion-Aware Video Understanding (2025)

Abstract: Enhancing the fine-grained instance spatiotemporal motion perception capabilities of Video Large Language Models is crucial for improving their temporal and general video understanding. However, current models struggle to perceive detailed and complex instance motions. To address these challenges, we have made improvements from both data and model perspectives. In terms of data, we have meticulously curated iMOVE-IT, the first large-scale instance-motion-aware video instruction-tuning dataset. This dataset is enriched with comprehensive instance motion annotations and spatiotemporal mutual-supervision tasks, providing extensive training for the model’s instance-motion-awareness. Building on this foundation, we introduce iMOVE, an instance-motion-aware video foundation model that utilizes Event-aware Spatiotemporal Efficient Modeling to retain informative instance spatiotemporal motion details while maintaining computational efficiency. It also incorporates Relative Spatiotemporal Position Tokens to ensure awareness of instance spatiotemporal positions. Evaluations indicate that iMOVE excels not only in video temporal understanding and general video understanding but also demonstrates significant advantages in long-term video understanding. We will release the data, code, and model weights after acceptance.

## introduction recent advancements in video large language models @xcite @xcite @xcite , i.e., video-llms, have led to rapid development, significantly enhancing the capture of overall video semantics and achieving remarkable performance in general video understanding tasks @xcite . furthermore, some works @xcite @xcite have begun to explore how does the dog's position change relative to the man as they walk, and at what timestamp does the dog change its position? ## method in this section, we outline the construction of imove as follows: collect the dataset imove-it with an automated pipline in sec 3.1 and build the model imove in sec 3.2. ## conclusion in this paper, we enhanced the fine-grained instance spatiotemporal motion perception in video-llms by making improvements from both data and model perspectives, thereby boosting their capabilities in temporal and general video understanding. datawise, we propose imove-it, a large instancemotion-aware video dataset with spatiotemporal mutual-supervision tasks to enhance instance spatiotemporal motion learning. model-wise, we develop imove featuring event-aware spatiotemporal efficient modeling for efficiently preserving motion details, and relative spatiotemporal position tokens for accurate spatial-temporal positioning. evaluations indicate that imove excels in video temporal understanding and general video understanding, with significant advantages in long-term video understanding, offering valuable insights for video understanding research.