No Language Data Left Behind: A Cross-Cultural Study of CJK Language Datasets in the Hugging Face Ecosystem (2025)

Abstract: Recent advances in Natural Language Processing (NLP) have underscored the crucial role of high-quality datasets in building large language models (LLMs). However, while extensive resources and analyses exist for English, the landscape for East Asian languages, particularly Chinese, Japanese, and Korean (CJK), remains fragmented and underexplored, despite these languages serving over 1.6 billion speakers. To address this gap, we investigate the HuggingFace ecosystem from a cross-linguistic perspective, focusing on how cultural norms, research environments, and institutional practices shape dataset availability and quality. Drawing on more than 3,300 datasets, we employ quantitative and qualitative methods to examine how these factors drive distinct creation and curation patterns across Chinese, Japanese, and Korean NLP communities. Our findings highlight the large-scale and often institution-driven nature of Chinese datasets, grassroots community-led development in Korean NLP, and an entertainment and subculture-focused emphasis on Japanese collections. By uncovering these patterns, we reveal practical strategies for enhancing dataset documentation, licensing clarity, and cross-lingual resource sharing, guiding more effective and culturally attuned LLM development in East Asia. We conclude by discussing best practices for future dataset curation and collaboration, aiming to strengthen resource development across all three languages.

## introduction with the emergence of large language models (llms) transforming the field of natural language processing (nlp) @xcite @xcite , the importance of high-quality datasets in model development has become increasingly critical. for datasets to be valuable in this context, they must meet both * work done during internship at saionic ai. ## discussion our analysis reveals distinct characteristics across cjk dataset ecosystems: chinese datasets show strong institutional backing but inconsistent documentation; korean datasets demonstrate community-driven development but face licensing gaps; and japanese datasets emphasize subcultural content while dealing with copyright constraints. three practical issues warrant attention. first, licensing diversity (particularly "unknown" licenses in korean datasets and restricted licenses in japanese resources) complicates collaborative projects. more consistent adherence to guidelines like "datasheets for datasets" @xcite could enhance reusability. second, domain clustering (medical for chinese, subcultural for japanese, and moderation for korean) may underserve other areas needed for general-purpose llm development. third, culturally specific content requires transparent documentation, as simple translations miss nuanced cultural meanings. despite these differences, strong synergy potential exists. joint benchmarks could facilitate crosslingual comparisons, while unified documentation frameworks could standardize metadata and licensing. our findings underscore both the richness and fragmentation of cjk resources, suggesting that clearer practices and cross-lingual collaboration can foster a robust ecosystem for east asian llm development. ## limitations our analysis primarily focused on datasets with relatively high download counts, which may have led us to overlook smaller or emerging resources that could shed light on niche trends or specialized applications. furthermore, we limited our scope to the hugging face platform; investigating additional repositories (e.g., github, kaggle, or papers with code) could reveal a broader range of dataset characteristics and host factors. although we manually requested permission to access certain private or restricted datasets, some ultimately remained inaccessible, thereby constraining the representativeness of our findings. in addition, while korean and japanese datasets were examined with input from language experts, our review of chinese data relied solely on documentation, potentially affecting the depth of our analysis. finally, we chose to focus on three major east asian languages, excluding many lowresource languages and dialects, whose inclusion could further expand and enrich our findings. ## conclusion this study presents a comparative analysis of over 3,300 chinese, japanese, and korean datasets on huggingface, revealing distinct ecosystem characteristics-chinese datasets show strong institutional involvement, korean resources are community-driven, and japanese datasets emphasize subcultural content-highlighting that documentation, licensing, and ownership must be addressed in cultural context to guide inclusive east asian language technologies.