F ormosan B ench: Benchmarking Low-Resource A ustronesian Languages in the Era of Large Language Models (2025)

Abstract: While large language models (LLMs) have demonstrated impressive performance across a wide range of natural language processing (NLP) tasks in high-resource languages, their capabilities in low-resource and minority languages remain significantly underexplored. Formosan languages—a subgroup of Austronesian languages spoken in Taiwan—are both linguistically rich and endangered, largely due to the sociolinguistic dominance of Mandarin. In this work, we introduce FormosanBench, the first benchmark for evaluating LLMs on low-resource Austronesian languages. It covers three endangered Formosan languages: Atayal, Amis, and Paiwan, across three core NLP tasks: machine translation, automatic speech recognition (ASR), and text summarization. We assess model performance in zero-shot, 10-shot, and fine-tuned settings using FormosanBench. Our results reveal a substantial performance gap between high-resource and Formosan languages. Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. These findings underscore the urgent need for more inclusive NLP technologies that can effectively support endangered and underrepresented languages. We release our datasets and code to facilitate future research in this direction :https://anonymous.4open.science/r/FormosanBench-DB43/

## introduction large language models (llms) have achieved remarkable success across a wide range of natural language processing (nlp) tasks @xcite , particularly for highresource languages @xcite . this success is largely driven by pretraining on massive datasets dominated by majority languages such as english, mandarin, and spanish @xcite . in contrast, the capabilities of llms in low-resource language settings remain significantly underexplored and underdeveloped @xcite . bridging this gap is critical for ensuring that nlp technologies are inclusive and beneficial to linguistically underrepresented communities. recent work has begun to address this disparity by developing benchmarks and datasets for lowresource languages. for example, @xcite introduced datasets for yorùbá and its dialects, while @xcite released irokobench, a multilingual benchmark for 17 typologically diverse african languages. these efforts highlight the growing recognition that evaluation frameworks must extend beyond the high-resource paradigm. formosan languages-a subgroup of austronesian languages spoken in taiwan-represent one such underrepresented group. with fewer than 200,000 speakers in total, foot_0 these languages are both linguistically rich and severely endangered. nlp development for formosan languages faces unique challenges, particularly the scarcity of high-quality, annotated data @xcite . in this paper, we address these gaps by introducing formosanbench, the first multi-task benchmark for formosan languages, covering three endangered languages-amis (iso-ami), atayal (iso-tay), and paiwan (iso-pwn)-across three core nlp tasks: machine translation, automatic speech recognition (asr), and text summarization. with formosanbench, we systematically evaluate the zero-shot performance of state-of-the-art large language models (llms) and explore strategies for improving their performance through 10shot in-context learning and small-scale fine-tuning. experimental results reveal a significant performance gap for existing llms on formosan languages, underscoring the need for further research. our main contributions are as follows: the austronesian language family is one of the world's largest and most geographically dispersed, extending from madagascar in the west to hawaii and new zealand in the east. taiwan is widely considered the linguistic homeland of austronesian, a hypothesis supported by the high degree of linguistic diversity found among its indigenous languages-collectively known as the formosan languages. this diversity includes significant phonological, lexical, and syntactic variation, distinguishing formosan languages from other austronesian branches @xcite @xcite . formosan languages pose considerable challenges for natural language processing (nlp), especially for large language models (llms). like many other austronesian languages, they feature verb-initial (vso or vos) syntax and a complex voice system that is typologically rare. for instance, as illustrated in figure 1 , paiwan allows multiple syntactic realizations of a sentence with essentially a similar meaning-e.g., 'zepulj eats sweet potatoes'-depending on how the verb is marked to highlight the roles of the noun phrases. the verb carries an agent-voice or patient-voice marker, and noun phrases change case marking accordingly and can switch positions @xcite . this flexibility, governed by a morphosyntactic voice system, is uncommon in the world's languages and adds complexity for modeling. further examples in atayal and amis (figure 1 ) demonstrate similar typological features. these three languages-atayal, amis, and paiwan-belong to early-diverging branches of the austronesian family and are genealogically distinct from each other as well as from more widely studied malayo-polynesian languages such as hawaiian and tagalog (see figure 3 ). their linguistic distinctiveness, combined with the scarcity of pretraining corpora, makes them ideal candidates for studying llm performance in low-resource and typologically diverse settings. early austronesian atayal amis paiwan malayo-polynesian etc. hawaiian tagalog figure 3: simplified austronesian family tree. ## related work low-resource languages are those with limited linguistic data, tools, and representation in nlp research. an estimated 80-90% of the world's languages fall into this category @xcite . low-resource nlp: recent advances in lowresource nlp have centered around three major areas: multilingual modeling, data preprocessing pipelines, and the development of benchmark datasets. multilingual models such as those proposed by @xcite al. (2019) and conneau et al. (2020) demonstrate that training on large-scale corpora from over 100 languages can improve performance on tasks like machine translation, crosslingual natural language inference, and general language understanding benchmarks. to improve data quality, wenzek et al. (2020) introduced ccnet, a scalable pipeline for cleaning noisy web-crawled text-an essential resource for expanding datasets in low-resource settings. in task-specific applications, mutsaddi and choudhary (2025) enhanced plagiarism detection in the low-resource indian language marathi by combining bert embeddings with tf-idf, outperforming both methods used independently and achieving over 82% accuracy. more recently, researchers have introduced benchmark datasets to support a broader range of typologically diverse low-resource languages. @xcite developed benchmark datasets for yorùbá and its dialects, while @xcite introduced atlas-chat, a multi-task benchmark for moroccan arabic (darija). they evaluated several state-of-the-art llms and demonstrated that fewshot and fine-tuned models significantly improved performance. these works highlight the growing focus on evaluation, benchmarking, and adaptation techniques for underrepresented languages in nlp. nlp in formosan languages: despite growing interest in digital preservation, computational research on formosan languages remains sparse. among the few studies in this area, @xcite proposed a transformer-based machine translation system for three atayalic languages-atayal, seediq, and truku-using training data from online dictionaries and educational materials, and testing on manually curated book content. similarly, zheng et al. (2024) explored translation techniques across 16 formosan languages by incorporating lexical resources and generating pseudo-parallel data through lexicon-based substitutions. while these approaches represent valuable steps forward, translation performance remains limited, reflecting both the complexity of formosan languages and the scarcity of high-quality data. notably, existing work has focused narrowly on machine translation and conventional sequence models. no prior research has systematically evaluated the capabilities of state-of-the-art llms across a broader range of nlp tasks in these languages. this gap underscores the importance of developing comprehensive benchmarks and assessing the adaptability of llms in truly low-resource and typologically diverse language settings. ## formosanbench we present formosanbench, the first benchmark to support three endangered formosan languages-atayal, amis, and paiwan-across three core nlp tasks: machine translation, automatic speech recognition, and text summarization. these languages were selected based on three criteria: 1) a combined speaker population exceeding 100,000; 2) the availability of existing digital resources; and 3) their linguistic diversity within the austronesian language family. the choice of nlp tasks was guided primarily by data availability. figure 4 illustrates examples of each task. machine translation our mt dataset comprises approximately 5,000 sentence pairs for each of three formosan languages-atayal, amis, and paiwan-paired with mandarin. the data was extracted from the taiwan indigenous languages e-dictionary foot_1 and underwent rigorous preprocessing as documented by formosanbank contributors (2024) foot_2 .the preprocessing included text normalization, punctuation correction, removal of encoding artifacts, and xml structure validation to ensure data quality across sentence-level units. we extracted the formosan sentences and their mandarin counterparts from the preprocessed xml files to form the final parallel corpora. asr we collected the asr data from formosan-bank. the data were sourced from the klokā digital platform for indigenous languages foot_3 , which hosts digitized textbooks for 16 formosan languages aimed at learners of all ages. each textbook section consists of written text in a formosan language together with audio recordings produced by native speakers who read the content aloud. the materials cover typical languagelearning themes-such as reading and writing, picture books, daily conversations, short texts, situational language use, illustrated stories, and cultural topics-offering a range of speech styles, vocabulary, and sentence complexity grounded in culturally specific content. for this study, we selected the dialects coastal amis, squliq atayal, and central paiwan respectively, due to their relative representativeness and greater accessibility within available resources. the final compiled datasets consist of paired sentence-level audio recordings with their corresponding transcriptions. summarization following @xcite , we collected wikipedia articles written in formosan languages. these articles often focus on culturally significant topics such as indigenous traditions, historical narratives, and regional geography @xcite . notably, the articles follow a relatively consistent structure: an initial introductory section summarizing the topic, followed by a series of subsections with more detailed expository content. this natural hierarchy of information lends itself well to the single-document summarization task, where the introduction can serve as a gold-standard summary and the remainder of the article as the input document @xcite . based on this structure, we curated the dataset by pairing each article's introductory section with the corresponding main body content. we manually reviewed the articles to ensure coherence between the summary and body, and excluded entries that lacked sufficient structure or were too short to support meaningful summarization. quality control to ensure the reliability and usefulness of our datasets, we applied task-specific quality control procedures. for the mt dataset, we apply rule-based filter to remove duplicate sentence pairs to avoid redundancy and evaluation bias. for the asr dataset, we reviewed the xml metadata associated with the selected formosan languages from formosanbank. we excluded entries that met any of the following criteria: 1) instances containing only a single word, 2) entries lacking corresponding audio file links sourced from the associated textbooks, and 3) duplicate entries. the summarization datasets underwent additional filtering to address quality issues in wikipedia articles written in formosan languages. due to content sparsity, many entries contained only introductory paragraphs without substantial body text. we implemented a two-stage filtering process: first removing datapoints where article bodies consisted solely of section titles but with-out accompanying content, then excluding entries where the input text (article body) was less than 1.5 times the length of the summary (introduction). this ensured that remaining samples provided meaningful training and evaluation material with sufficient information density for effective summarization model development. benchmark statistics detailed statistics for each task in formosanbench are presented in we used consistent prompting strategies for both mt and summarization tasks across all models. example prompts are provided in appendix. automatic speech recognition for the asr task, we evaluated a set of state-of-the-art models with demonstrated performance in multilingual and low-resource contexts @xcite ): it is well-regarded for robustness in noisy and low-resource scenarios. ## conclusion in this paper, we present evaluation results from several large language models (llms) across three formosan languages. our findings demonstrate that low-resource languages, such as those in the formosan family, remain a significant blind spot for current state-of-the-art llms. the models perform poorly across all three evaluated nlp tasks-machine translation (mt), automatic speech recognition (asr), and text summarization. in particular, results from mt and summarization indicate that the models struggle to exhibit meaningful understanding of the target languages, often producing outputs that are entirely unrelated to the input. while asr performance is relatively better, it still falls short of practical usability. notably, our adaptation efforts led to a substantial reduction in word error rate (wer) for asr and a notable increase in rouge scores for gpt-4o in summarization. however, improvements in bleu scores for mt were marginal, and most models showed a decline in rouge scores on the summarization task. overall, these benchmarking results underscore the urgent need for targeted research and model development tailored to low-resource languages such as those in the formosan family.