Cross-Lingual and Cross-Cultural Variation in Image Descriptions (2025)

Abstract: Do speakers of different languages talk differently about what they see? Behavioural and cognitive studies report cultural effects on perception; however, these are mostly limited in scope and hard to replicate. In this work, we conduct the first large-scale empirical study of cross-lingual variation in image descriptions. Using a multimodal dataset with 31 languages and images from diverse locations, we develop a method to accurately identify entities mentioned in captions and present in the images, then measure how they vary across languages. Our analysis reveals that pairs of languages that are geographically or genetically closer tend to mention the same entities more frequently. We also identify entity categories whose saliency is universally high (such as animate beings), low (clothing accessories) or displaying high variance across languages (landscape). In a case study, we measure the differences in a specific language pair (e.g., Japanese mentions clothing far more frequently than English). Furthermore, our method corroborates previous small-scale studies, including 1) Rosch et al. (1976)’s theory of basic-level categories, demonstrating a preference for entities that are neither too generic nor too specific, and 2) Miyamoto et al. (2006)’s hypothesis that environments afford patterns of perception, such as entity counts. Overall, our work reveals the presence of both universal and culture-specific patterns in entity mentions.

## introduction do speakers of different languages talk differently about what they see? this question is important for both machine learning and cognitive science. from a computational perspective, the answer can guide the creation of more diverse vision-and-language datasets @xcite . from a cognitive science 1 we provide our code at github.com/uriberger/cross_lingual_diff_in_descs and a web interface for our dataset at tinyurl.com/5ekabrb5. perspective, it can help us study cultural effects on perception and language. to investigate their cross-lingual and crosscultural variation, we can compare the descriptions of the same images by different communities of speakers. in this setting, differences in semantic content must stem from linguistic rather than perceptual differences. 2 crucially, describing the same image takes into account cultural effects (e.g., the saliency of an entity) while excluding environmental ones (e.g., the presence of an entity) @xcite . previous work investigated which cultures draw more attention to the foreground or the context of a visual scene @xcite , and how often each culture uses negation or certain bigrams at the start of the descriptions @xcite . nevertheless, these studies were conducted on a small scale, either with artificial images or focused on just a few related languages. to induce more solid conclusions, a large dataset of image descriptions in a variety of typologically and areally diverse languages is needed @xcite . we use crossmodal3600 (xm3600, @xcite , which is both relatively large in scale (3600 images) compared with previous controlled studies and more linguistically diverse (36 languages) compared with any previous imagedescription dataset. we develop automatic tools for accurately identifying phrases mentioning entities in image descriptions and map these phrases into wordnet @xcite synsets, which function as a proxy for entity categories. we further annotate images with the categories they contain, independent of the descriptions. finally, we study how entity mentions vary across different languages (see figure 1 for an example). using this annotation as quantitative evidence, we are able to address the following questions: our method can be used both as an exploration and a verification tool for cultural effects on object saliency. for the former, we present new findings (sections 4.2, 4.3, and 4.4). future work may propose small-scale, controlled studies to verify these findings. for the latter, we demonstrate how our method can validate previous findings from cognitive studies. first, annotators typically select object cate-gories that fall in the middle of the hierarchy, with more general (lower) and more specific (higher) categories being less frequent, supporting @xcite 's hypothesis of basic-level categories (section 4.5). second, we put to the test @xcite 's claim that images taken in east asian locations tend to include more entities compared to western locations (section 4.6). to summarise, our contributions are as follows. first, we present new insights into cross-lingual variation in entity saliency and confirm previous findings on a wider and more diverse set of languages. second, we release our entity mention dataset to support future research on cross-lingual and cross-cultural variation of image descriptions. ## background and related work our study falls under the umbrella of works that study cultural effects on perception and language. early studies relied on small-scale, controlled experiments, while the recent rise in the amount of publicly available visual data with human annotations has prompted large-scale and real-world studies harnessing computational methods to investigate these datasets. ## methods to identify the entities mentioned in a caption, we use wordnet's synsets as a proxy for entity categories. we translate all captions to english (section 3.1), pre-define a target list of synsets (section 3.2), extract noun phrases that correspond to one of these synsets (section 3.3) and filter the resulting synset lists to remove synsets corresponding to entities that are not in the image (section 3.4). ## experiments in this section we experiment on our entity mentions corpus, extracted from xm3600 captions. ## conclusion we presented the first large-scale empirical study of how the saliency of entities in image descriptions varies across languages. starting from an existing dataset with culturally diverse images and typologically diverse languages, crossmodal3600, we developed an automated method to extract entity mentions from multilingual captions and we annotated images with the corresponding entities. we then conducted a series of experiments on the resulting dataset, finding that: 1. languages in the same family or area have a mild tendency to mention the same entities, whereas no such effect occurs for featural similarity between languages. although we tested our method solely on the xm3600 dataset, which is notable for its cultural diversity, we anticipate that the findings will generalise to other datasets such as mscoco, which is more anglocentric. this expectation is supported by our analysis in section 4.6, where we observed consistent trends across languages in the number of entities mentioned, regardless of the annotator's familiarity. our method provides an alternative to smallscale, highly controlled experiments, but is meant to complement them, rather than replace them. in fact, while carefully designed experiments require significant effort in recruiting and training participants, our method is more cost-effective and faster. therefore, it can serve as an exploratory tool to identify interesting phenomena for later thorough study, and as a large-scale verification for previous findings. we demonstrated both use cases in this work.