ATAIGI : An AI -Powered Multimodal Learning App Leveraging Generative Models for Low-Resource T aiwanese Hokkien (2025)

Abstract: Many endangered languages are at risk of extinction due to barriers in communication and generational gaps that hinder their preservation. A cause for languages becoming endangered is the lack of language educational tools and artificial intelligence (AI) models for these low-resource languages. To address this, we propose the ATAIGI learning app designed with AI-powered models leveraging multimodal generative techniques. Our app offers users a comprehensive learning experience by providing translated phrases and definitions, example sentences, illustrative images, romanized pronunciation, and audio speech to accelerate language learning. ATAIGI is built on five AI models that are rigorously benchmarked individually, with our Transliteration Model achieving state-of-the-art results for Taiwanese Hokkien transliteration. ATAIGI is available for all to learn the endangered language of Taiwanese Hokkien, an endangered language spoken in Taiwan. A human evaluation conducted demonstrates the effectiveness of ATAIGI in improving language proficiency and cultural understanding, supporting its potential for the preservation and education of endangered languages like the Taiwanese Hokkien.

## introduction natural language processing tools and models used for language learning are commonly applied to high-resource languages (hrls) rather than lowresource languages (lrls) @xcite . recent language learning tools showed they could be effective in helping users learn a language @xcite @xcite . existing research @xcite investigates the feasibility of gpt @xcite as a learning tool for a second language; however even a model as powerful as gpt does not perform well for lrls @xcite @xcite . despite the abundance of software tools and literature on hrls available, lrls are often overlooked due to the scarcity of data and models required to create learning tools. taiwanese hokkien is an endangered lrl that represents a cultural heritage on the brink of extinction @xcite . a generational linguistic gap exists between the elderly who predominantly communicate in taiwanese hokkien, and the younger generation who communicate in hrls such as mandarin and english. although useful tools such as the translation tool itaigi 3 and the text-to-speech (tts) tool tauphahji foot_3 are available, but the scattered and unintegrated nature of these resources makes it difficult for users to master the lrl. when using specific vocabulary, learners often face difficulties recalling suitable terms and examples quickly. we created ataigi, which pronounces "learning taiwanese hokkien" in its native tongue, to serve as a learning framework for teaching the endangered language of taiwanese hokkien. based on the mentioned shortcomings, the ataigi learning app makes the following contributions: we have developed a user-friendly learning application through these ai-powered generative models to bridge generational communication gaps for taiwanese hokkien. a comprehensive user study validated ataigi's efficacy, demonstrating significant improvements in taiwanese hokkien proficiency across diverse participants. high engagement and satisfaction ratings affirmed ataigi's potential for endangered language preservation. ataigi will carry the mission of cultural preservation and serve as a bridge for communications between generations. a video demonstration is available at https://youtu.be/fc409q7jl2c . ## ataigi language application system to use ataigi (figure 1 ), the user first inputs an english or chinese phrase, which is entered into the taiwanese hokkien dictionary to retrieve its definition. then the phrase is translated into taiwanese hokkien by the taiwanese hokkien translation model (sec. 2.1.2). meanwhile, the output of the translation model and context generation model (sec. 2.1.3) is romanized by the hokkien transliteration model (sec. 2.1.5) and then converted into audible speech by the text2speech model (sec. 2.1.6) to assist users in correctly pronouncing and articulating phrases and sentences. ## evaluation comprehensive evaluations were conducted for each model to ensure effective contribution and acceptable model performance. the evaluation is crucial for the correctness of each of the models by verifying that it: 1) functioned as intended, 2) generated the intended data, and 3) contributed effectively to the overarching objective. context generation model to ensure the context generation model generates appropriate sentences, we evaluated its fluency, semantic correctness and sentence length. we utilize the state-ofthe-art language model gpt-4 to assess these aspects @xcite . customized prompts included the phrase, its corresponding sentence in hokkien and its definition (appendix d). we used gpt-4 to rate the fluency and semantic consistency between the generated sentence and the definition of the hokkien phrase on a 10-point scale. collection field description sentence sentences stores sentences in traditional and hokkien chinese, and english. translationlist a list of unique document ids in the translations collection. imageurl the url of a pre-generated image (figure 1 ) from the input phrase. translation chineseinput the input phrase in simplified or traditional chinese. definitions hokkien explanations of the phrase. englishdefinitions english explanations of the phrase englishinput the input phrase in english. hokkientranslation the translated hokkien phrase. sentence a unique document id in the sentence collection. due to gpt-4 's limited performance with lowresource languages, we initially employ our taiwanese hokkien translation model to translate the generated sentences and definitions from hokkien to english. following this translation, we proceed with the evaluation. the scores rated by gpt-4 were then aggregated and averaged, resulting in a fluency score of 8.19 and a meaning score of 7.71. these scores reflect that context generation model has strong performance, indicating that our context generation model can produce sentences that are both linguistically fluent and semantically coherent. this evaluation method, using pre-eminent language models such as gpt-4, provides a standardized, objective and cost-effective approach to measuring the capabilities of our context generation model. more importantly, the high fluency and meaning scores generated demonstrate the effectiveness of the context generation model, which highlights its robustness in our framework. text2image model in order to evaluate the images, we make use of the heim evaluation method @xcite , where we selected two metrics from the heim evaluation method , aesthetics and alignment, supplemented by a human evaluation, to holistically assess the quality of the image generated in our system ( for aesthetics evaluation, we employed the laion aesthetics metric @xcite , to quantify visual appeal. compared to a stable diffusion xl model (sdxl 1.0), our chosen model has obtained an aesthetics score higher than the baseline sdxl model. this makes it a better choice compared to sdxl to output more aesthetically appealing images that complement our system. to evaluate the alignment between the generated images and their corresponding sentence, we utilize the clip score @xcite , indicating the degree of alignment between the visual content and the textual descriptions. our chosen model has obtained a better clip score than a standard sdxl model, making it a better choice than sdxl to output images that align better with the given context. in addition to these automated metrics, we conducted a human evaluation on the text2image model alongside the user study for our system. this human assessment provides a holistic judgement that complements the machine-derived scores mentioned previously. in the user study, our chosen model received a high rating of 4.6/5 on average, across all image component questions, demonstrating its effectiveness. details on the user study will be elaborated in the qualitative human evaluation section. this study employs a multifaceted evaluation approach to comprehensively analyze the model's image generation performance. based on the scores shown in hokkien transliteration model we selected 450 sentences from the icorpus dataset to evaluate whether the hokkien transliteration model can accurately translate the input sentences to its romanized phonetic representation. we employed three evaluation metrics, bleu score foot_7 @xcite , chrf++ @xcite , and word error rate (wer). two distinct evaluation strategies were used. the first strategy involved adding spaces before and after hyphens to prevent multisyllable words from being inaccurately considered as a single word in the bleu score metric. the second strategy involved removing hyphens to examine the performance of our model on individual words. based on these metrics and evaluation methods ( qualitative human evaluation the efficacy of ataigi was evaluated through a comprehensive study 9 involving 25 participants with diverse ages and proficiency levels. the assessment encompassed multiple dimensions, including usage patterns, functionality, learning outcomes, and user satisfaction ( ## conclusion our ataigi is designed to teach taiwanese hokkien through phrases, example sentences, pictures, audio speech, and translations. the system is composed of five ai models, the most notable of which include an innovative context generation model and high-quality hokkien transliteration model. this system aims to prevent the extinction of low-resource languages and bridge the gap between different languages and generations. empirical evaluation with heterogeneous participants substantiated ataigi's efficacy in enhancing hokkien proficiency, confirming its potential for endangered language preservation. in the future, speakers of low-resource languages could use this system to overcome cross-language barriers and preserve their cultural heritage.