Do Large Language Models Know Folktales? A Case Study of Yokai in J apanese Folktales (2025)

Abstract: Although Large Language Models (LLMs) have demonstrated strong language understanding and generation abilities across various languages, their cultural knowledge is often limited to English-speaking communities, which can marginalize the cultures of non-English communities. To address the problem, the evaluation of the cultural awareness of the LLMs and the methods to develop culturally aware LLMs have been investigated. In this study, we focus on evaluating knowledge of folktales, a key medium for conveying and circulating culture. In particular, we focus on Japanese folktales, specifically on knowledge of Yokai. Yokai are supernatural creatures originating from Japanese folktales that continue to be popular motifs in art and entertainment today. Yokai have long served as a medium for cultural expression, making them an ideal subject for assessing the cultural awareness of LLMs. We introduce YokaiEval, a benchmark dataset consisting of 809 multiple-choice questions (each with four options) designed to probe knowledge about yokai. We evaluate the performance of 31 Japanese and multilingual LLMs on this dataset. The results show that models trained with Japanese language resources achieve higher accuracy than English-centric models, with those that underwent continued pretraining in Japanese, particularly those based on Llama-3, performing especially well.

## introduction large language models (llm) have shown remarkable performance in language understanding and generation tasks @xcite @xcite . despite many llms being predominantly trained in english, their generalization capabilities allow them to transfer knowledge across languages, achieving decent performance even in resource-limited languages @xcite @xcite . while there is evidence for strong cross-lingual transfer capability, cross-cultural transfer is known to be challenging for llms @xcite @xcite @xcite @xcite . prior work shows that llms tend to be biased toward the values and opinions of certain communities, rather than representing the diversity of human values @xcite . to address this issue, many studies have investigated methods to evaluate the cultural awareness of llms @xcite @xcite . simultaneously, approaches to develop culturally aware llms using the language resources of target communities are being explored @xcite @xcite @xcite @xcite . while many prior studies have investigated differences in the values and opinions of communities @xcite @xcite @xcite , the medium that conveys these values and opinions has received less attention. specifically, there is less focus on folktales, which are important for creating, receiving, and circulating a community's traditions, values, opinions, and culture @xcite . the goal of the study is to investigate how much llms know about folktales. specifically, we focus on yokai (figure 1 ), supernatural phenomena or entities believed to cause such phenomena, which originated from japanese folktales @xcite . yokai have been popular motifs in art and entertainment from the edo period to the present, appearing in ukiyo-e (a genre of japanese paintings), kabuki (a traditional japanese theater featuring performance and dance), toys, manga, and anime @xcite . in this paper, we present yokaieval, a benchmark dataset designed to evaluate knowledge about yokai. yokaieval is created using wikipedia articles, with gpt-4o generating the question-answer pairs (qas). these qas are then curated using gpt-4omini to filter out inconsistent questions, followed by manual verification to ensure their correctness, appropriateness, and the presence of references. we use yokaieval to evaluate 31 llms, including japanese-centric, englishcentric, and other multilingual models. the results show that japanese-centric models achieve comparatively higher scores than the others, particularly those models that have been continually pretrained from highly capable english-centric models (e.g., llama-3). this study suggests that using language resources from the community is crucial for acquiring regional folktale knowledge and that continual pretraining is an effective method for training llms with these resources. ## yokai: from traditional folktales to today's art and entertainment yokai are supernatural phenomena or entities believed to cause such phenomena @xcite . traditionally, yokai are described in oral lore and rumors as superstitions, but they have also become popular characters in contemporary art and entertainment @xcite . ## related work we describe prior work on evaluating and training culturally aware llms. ## constructing yokaieval we first describe the procedure for generating yokaieval in section 4.1. then, in section 4.2, we analyze the characteristics of the generated dataset to evaluate if it is suitable for measuring the knowledge of yokai in llms. ## evaluating japanese and multilingual llms on yokaieval with yokaieval generated, we run experiments to evaluate the performance of 31 publically available llms, including japanese-centric, english-centric, and chinese-centric models. we then further look into the performance of japanse-centric models to investigate which training procedure (pretraining, sft, and pl) is important to acquiring the knowledge of yokai. ## conclusions in this paper, we introduce yokaieval, a benchmark dataset designed to evaluate llms' knowledge of japanese folktales. the dataset is generated by extracting qas from wikipedia articles using gpt-4o, followed by curation with gpt-4omini and human verification. we evaluate 31 llms using yokaieval, showing that japanese llms outperform the others, with models continually pretrained on japanese language resources achieving the highest scores. the result implies the importance of using communitysourced resources to develop llms knowledgeable of folktales.