Interesting Culture: Social Relation Recognition from Videos via Culture De-confounding (2025)

Abstract: Social relationship recognition, as one of the fundamental tasks in video understanding, contributes to the construction and application of multi-modal knowledge graph. Previous works have mainly focused on two aspects: generating character graphs and multi-modal fusion. However, they often overlook the impact of cultural differences on relationship recognition. Specifically, relationship recognition models are susceptible to being misled by training data from a specific cultural context. This can result in the learning of culture-specific spurious correlations, ultimately restricting the ability to recognize social relationships in different cultures. Therefore, we employ a customized causal graph to analyze the confounding effects of culture in the relationship recognition task. We propose a Cultural Causal Intervention (CCI) model that mitigates the influence of culture as a confounding factor in the visual and textual modalities. Importantly, we also construct a novel video social relation recognition (CVSR) dataset to facilitate discussion and research on cultural factors in video tasks. Extensive experiments conducted on several datasets demonstrate that the proposed model surpasses state-of-the-art methods.

## introduction social relationship recognition task initially focused on extracting relationship categories between characters in text and still images @xcite . however, due to the widespread use of social media, there has been a significant growth trend in video data. the temporal sequence information and multi-modal semantic information present in videos make relation recognition more challenging, but provide more reliable evidence. in previous research works, researchers focused on multi-modal fusion and character graphs construction @xcite . but they have overlooked the impact of cultural differences @xcite on model performance. culture is a set of attitudes, values, beliefs, and behaviors shared by a group of people and communicated from one generation to the next @xcite . cultural bias is universal, which means that subjective opinions from a particular culture may offend or misinterpret other cultures. in the task of relationship recognition, due to the influence of different cultures, individuals with the same social relationships may engage in significantly different interactions and language communication, which can even lead to confusion in relationship discrimination. we extracted the cultural background of the visr dataset @xcite , operationalizing it into country or region reflected in the video data @xcite . nationality or region is not always coextensive with cultural background, but it was typically the only clue of cultural belonging in the video data @xcite . simple statistical analysis revealed significant imbalance in the distribution of cultural categories within the visr dataset. figure 1 illustrates how cultural bias affect predictions. more intrigued, we conducted an experiment us-group train test map balance a, c, i, j a, c, i, j 0.464 imbalance a, c, i j 0.296 imbalance a, i, j c 0. 342 based on the observations mentioned above, we attempted to improve the relation recognition model by applying causal intervention @xcite rather than aiming to beat them. however, the implementation of causal intervention in video relation recognition task is challenging, such as the definition and extraction of culturally biased features, and lack of cultural annotations in existing datasets. therefore, we propose a cultural causal intervention model (cci) to mitigate the impact of cultural differences by incorporating visual and textual modalities. to facilitate the study of cultural factors in video tasks, we construct a new dataset for video social relation recognition. and conduct cultural annotations on multiple datasets based on the consensus. we evaluated the effectiveness of the cci framework on the several datasets. extensive experiments demonstrate that cci can effectively mitigate the impact of cultural bias on model performance and achieve sota performance. we summarize our contributions as follows: ## conclusion this paper is the first to present a analysis and identification of cultural bias in the video social relationship recognition task from a causal perspective. it proposes a cultural causal intervention model (cci) to mitigate the adverse effects of cultural bias as a confounding factor by utilizing causal graphs and backdoor adjustment techniques. extensive experiments conducted on several datasets demonstrate that cci effectively alleviates the impact of bias on model performance and enhances the accuracy of relationship classification. in the future, we will focus on the separation of causal and biased features under self-supervision and design more general frameworks for multimodal bias.