SEAC rowd: A Multilingual Multimodal Data Hub and Benchmark Suite for S outheast A sian Languages (2024)

Abstract: Southeast Asia (SEA) is a region rich in linguistic diversity and cultural variety, with over 1,300 indigenous languages and a population of 671 million people. However, prevailing AI models suffer from a significant lack of representation of texts, images, and audio datasets from SEA, compromising the quality of AI models for SEA languages. Evaluating models for SEA languages is challenging due to the scarcity of high-quality datasets, compounded by the dominance of English training data, raising concerns about potential cultural misrepresentation. To address these challenges, through a collaborative movement, we introduce SEACrowd, a comprehensive resource center that fills the resource gap by providing standardized corpora in nearly 1,000 SEA languages across three modalities. Through our SEACrowd benchmarks, we assess the quality of AI models on 36 indigenous languages across 13 tasks, offering valuable insights into the current AI landscape in SEA. Furthermore, we propose strategies to facilitate greater AI advancements, maximizing potential utility and resource equity for the future of AI in Southeast Asia.

## introduction despite southeast asia (sea) being home to 1,300 indigenous languages (18% of the world's languages) and 671 million people (8.75% of the world's population), the representation of texts, images, and audio datasets from this region is significantly lacking in machine learning models. this deficiency adversely affects the model quality for sea languages. the language coverage of sea languages in two common pre-training resources, common crawl foot_0 and c4 @xcite , is extremely limited, with only 2.36% (in 11 languages) and 10.62% (in 11 languages), respectively. in modalities beyond text, the representation is even more limited. for instance, common voice, one of the largest multilingual speech corpora, includes six sea indigenous languages @xcite , and laion-5b, one of the largest multilingual vision-language (vl) corpora, includes 12 sea indigenous languages @xcite . datasets for other sea indigenous languages exist, but are often scattered, insufficiently documented, or varied in quality and formatting, thereby making access and usage challenging @xcite @xcite . in terms of evaluation, the sparse availability of high-quality test sets for these languages also complicates evaluating models for sea languages. despite there being 1,300+ languages in the sea region, prior works @xcite @xcite @xcite @xcite have only evaluated fewer than 10 sea languages collectively. the actual performance of current models on most sea languages remains largely unknown. moreover, the dominance of anglocentric training data may result in cultural bias when generating texts, images, or audio in underrepresented sea languages @xcite . @xcite ; @xcite have shown that the learned representations in large language models (llms) often fail to reflect local cultural values in sea @xcite @xcite . this raises concerns about the ability of current llms to generate natural, high-quality texts for this region. in addition, the discrepancy in language support creates language barriers in technological access and risks marginalizing minority groups who do not speak the dominant language. in this work, we investigate the current ai progress for sea languages by addressing the challenges of resources, evaluation, and generation quality. our contributions are three-fold: ## seacrowd seacrowd represents the first comprehensive ai dataset collection initiative for sea, developed through a collaborative effort among researchers and engineers primarily based in the sea region. as addressed in §1, resource scarcity and the scattered nature of the data are crucial challenges in sea. seacrowd addresses these issues through two primary contributions: 1) consolidating datasheets to enhance data discoverability; and 2) standardizing dataloaders for easier use, especially in multiple dataset loading. we also follow data provenance practices @xcite to preserve the proprietary rights of dataset owners.