Reading between the Lines: Can LLM s Identify Cross-Cultural Communication Gaps? (2025)

Abstract: In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: https://github.com/sougata-ub/reading_between_lines.

## introduction whether performing outdoor activities, household chores, or surfing the internet, our perception and understanding of the world are varied. each of us has a distinct worldview, which causes us to understand and internalize information distinctly. although our knowledge, and hence our worldview, are individual-specific @xcite @xcite , our culture does contribute to the variation by shaping our worldview and understandability @xcite . this effect of culture is also evident in the online world, where we might find text from distinct online sources hard to understand due to a lack of common ground between us and the writer of the text @xcite . for example, people unfamiliar with the arabic culture might not understand the meaning of the dishes "machboos" and "luqaimat" from the review text "the machboos was perfectly spiced, * both authors contributed equally to this paper. and the luqaimat was a real treat." or, someone unfamiliar with the us culture and not exposed to fast food might not understand that "golden arches" refers to "mcdonald's" in the text "let's go to the golden arches for a quick bite.". the recent advancements in ai and nlp make us question whether and how large language models (llms) can help overcome such barriers in online cross-cultural communication. here, we consider a use case where llms act as cultural mediators by identifying, categorizing, and explaining text spans (culture-specific items) from online reviews that users from distinct cultures might find hard to understand. although numerous use cases of llms in cross-cultural communication are conceivable @xcite , the following fundamental and overarching questions need answering before implementing them across any domain. firstly, although studies discuss the importance of cross-cultural communication @xcite , the need for llms and ai assistants in general for cross-cultural communication has not been studied. secondly, the cross-cultural performance parity of llms in such settings is unclear and unmeasured. although studies have shown llms to exhibit anglo-centric biases @xcite , this does not mean they perform well in those cultures and poorly in others. we refer to this as the equitability, defined as a model's capability to maintain the parity between its performance and the user's need across cultures. through our specific use-case of llms as a cultural mediator, we answer these fundamental questions, a prerequisite of using them to overcome cross-cultural communication barriers. the contributions of this paper are as follows: ## culture: a primer culture is a complex concept, and is defined as the "way of life of a collective group of people distinguishing them from other groups" @xcite @xcite , making it experiential @xcite . it provides a common ground to groups through shared experiences and creates distinct "worldviews" between people from different cultures @xcite @xcite @xcite by shaping their cognition @xcite @xcite . this difference in perception due to culture affects all aspects of life, where literary works composed by individuals from one culture might not be perceived similarly by individuals from a different culture and hamper cross-cultural communication @xcite . in nlp, these variations can broadly be due to differences in linguistic form and style, common ground, aboutness, and values @xcite . among demographies, these differences vary across dimensions or semantic proxies, ranging from physical items such as food and materials to abstract constructs like emotions and values @xcite . @xcite terms such concepts as culture-specific items (csis), identifying and addressing which is crucial for cross-cultural communication. thus, to technologically facilitate cross-cultural communication, it is essential to identify and gauge the extent of the variation of such csis, which can ## user study to measure the extent of the difficulty in comprehension due to culture and establish a need for cross-cultural tools, we conducted an online user study where we showed participants book reviews written in english and asked them to highlight text spans they did not understand. the length of the spans was unrestricted and could be individual words or even multiple sentences. our goal was to quantify how much of things people do not understand and, out of that, how much is cultural (csis). measuring this cultural knowledge gap can serve as an evaluation benchmark for tools that facilitate cross-cultural communication. ## discussion although our study encompasses goodreads book reviews, we believe the findings apply to other domains. since the books and their reviews were random, they do not pertain to any single domain and often comment on different aspects of life. our studies suggest that although understandability varies by person, csis are an agreed set of items that people do not understand within a culture. furthermore, the csis are distinct across cultures. this finding can benefit any domain and application that spans cultures, such as translation, education, intercultural training, content personalization, etc. in a rapidly globalizing world, where the definition of culture has shifted from a traditional nationality or ethnicity-centric definition to include digital communities @xcite , our study establishes the need for tools that enable cross-cultural communication and demonstrates where current state-of-the-art llms, such as gpt-4o, are lacking. recent tools such as culturally yours @xcite , an ai-based reading assistant that enables cross-cultural communication and personalizes to each user, further attest to the practical implications of our study. ## related work communicating across cultures is crucial in today's global world. although several theories and frameworks exist that define the nature of crosscultural communication @xcite @xcite , practical applications implementing such concepts are still nascent. computationally, a considerable amount of work has been done in hci in defining the considerations of cross-cultural tools and systems @xcite @xcite . cultural adaptation is also extensively studied in machine translation @xcite @xcite , aixelá (1996) introduced the term "culture-specific items" (csis), and newmark ( 2003 advances in llms have garnered studies that detect biases in llms @xcite @xcite . using curated cultural datasets, most methods probe llms and test their knowledge and reasoning capabilities in culture-specific settings @xcite @xcite @xcite ; tanmay et al. @xcite @xcite . some methods @xcite @xcite also analyze the model-generated responses along theoretical frameworks such as hofstede's cultural dimensions @xcite and measure their proximity with cultures, where high proximity indicates better value alignment between the nearby cultures and the values portrayed by the model's response. most of these methods necessitate constructing cultural-specific test beds @xcite @xcite @xcite @xcite @xcite @xcite . ## conclusion although cultural knowledge is necessary for llms to operate in cross-cultural settings, its impact on practical cross-cultural settings is unmeasured. hence, determining llms' cross-cultural equitability primarily based on their factual knowledge about cultures is limiting. to address this gap, here we take a distinct approach where we perform a thorough user study of the cross-cultural understandability of goodreads reviews to understand the need for cross-cultural ai tools. we then evaluate gpt-4o as a cultural reading assistant that can identify and adapt culture-specific items to the culture of a user. our evaluations reveal that there are indeed lots of cultural things in goodreads reviews that might not be comprehensible to people from distinct cultures. also, gpt-4o performs equitably across the studied cultures. however, with overall low precision and recall scores, the amount of alignment is low, indicating room for improvement.