SA - CLIP : Language Guided Image Spatial and Action Feature Learning (2025)

Abstract: We observed that Contrastive Language-Image Pretraining (CLIP) models struggle with real-world downstream tasks such as road traffic anomaly detection, due to their inability to effectively capture spatial and action relationships between objects within images. To address this, we compile and curate a dataset with 1M samples of images using language supervision provided by the common image caption dataset, in which each image is paired with subject-relationship-object descriptions emphasizing spatial and action interactions, and train a S patial and A ction relationship aware CLIP ( SA-CLIP ) model. We evaluated the proposed model on the Visual Spatial Reasoning (VSR) dataset and further verified its effectiveness on the Detection-of-Traffic-Anomaly (DoTA) dataset. Experiment results show that the proposed SA-CLIP demonstrates strong abilities in understanding spatial relationships while achieving good zero-shot performance on the traffic anomaly detection task.

## introduction vision-language models have demonstrated strong potential in real-world tasks and in producing explainable results @xcite @xcite , among which the clip @xcite based models gain particular attention @xcite @xcite . clip is pretrained to align image and text representations by minimizing their distance, and shows strong generalization ability for real-world image-text matching tasks. such ability makes clip particularly attractive for tasks requiring a high degree of interpretability, as it allows the model to explain its decisions in humanunderstandable terms by linking specific visual elements with descriptive language, while being com-the bumper of {car} pressed against the bumper of {car}. the side door of {car} smashed by the front end of {car}. the side door of {car} smashed by the front end of {car} . the side door of {car} is away from the front end of {car} . the bumper of {car} is alongside the bumper of {car} . ## methodology to improve clip's ability to understand spatial and action relationships among objects in scenes, we focus on the subject-verb-object (svo) structure in both image descriptions and scene graphs. the svo structure is a fundamental linguistic pattern that describes interactions (verbs) between subjects and objects, and we extend the verbal references to include prepositional phrases (pps) that indicate spatial relationships. we train clip to map the svo structures from language to visual representations. due to the scarcity of datasets with diverse and detailed spatial and action annotations, we mine such information from abundant image-caption datasets and pretrain the proposed sa-clip on the curated data. ## conclusion in this work, we address clip's limitations in spatial and action understanding by introducing sa-clip, pretrained on a curated dataset focused on these relationships in both images and text. sa-clip outperforms vanilla clip in spatial action understanding and also proves effective in the realworld task of road traffic anomaly detection.