Teaching Large Language Models an Unseen Language on the Fly (2024)

Abstract: Existing large language models struggle to support numerous low-resource languages, particularly the extremely low-resource ones, for which there is minimal training data available for effective parameter updating. We thus investigate whether LLMs can learn a new language on the fly solely through prompting. To study this question, we collect a research suite for Zhuang, a language supported by no LLMs currently. We introduce DiPMT++, a framework for adapting LLMs to unseen languages by in-context learning. Using a dictionary and 5K parallel sentences only, DiPMT++ significantly enhances the performance of GPT-4 from 0 to 16 BLEU for Chinese-to-Zhuang translation and achieves 32 BLEU for Zhuang-to-Chinese translation. We also validate the effectiveness of our framework on Kalamang, another unseen language. Furthermore, we demonstrate the practical utility of DiPMT++ in aiding humans in translating completely unseen languages, which could contribute to the preservation of linguistic diversity.

## introduction existing large language models (llms) provide robust support for many high-resource languages, but their support for numerous low-resource languages is limited @xcite . to adapt llms to low-resource languages, continual pre-training or adaptors are commonly employed @xcite . however, a corpus of only a few thousand sentences is insufficient to update the model parameters effectively for learning extremely low-resource languages @xcite . considering the inductive and mimicking capabilities of llms, an interesting research question arises: can llms learn a new low-resource language on the fly solely through prompting? this learning paradigm could enable more efficient utilization of limited resources and holds significant translate the sentence: song mehlwg itcig byaij. "song" means "two"; "meh" means "mother"; "lwg" means "child"; "itcig" means "continuously"; "byaij" means "walk". the translation result is: llm the mother and her child kept walking. potential in the preservation and education of underrepresented languages. ## dataset: zhuangbench we present zhuangbench, the first nlp research suite for zhuang, consisting of a zhuang-chinese dictionary, a zhuang-chinese parallel corpus, and a zhuang-chinese translation test set. it can be used for various nlp tasks, such as word sense disambiguation, cross-lingual retrieval, and machine translation. here, we especially focus on the task of performing zhuang-chinese translation using the dictionary and parallel corpus. ## discussion here we delve into two important questions when adapting llms to an unseen language. one is how to improve the coverage of lexical explanations given limited or incomplete resources. we analyze the three strategies used in dipmt++ to alleviate the out-of-dictionary problem. the other is how to teach llms syntactic rules solely through prompting. we go beyond dipmt++ and investigate more potential strategies to help llms learn syntax implicitly or explicitly. ## assisting human translation here we discuss the potential of applying dipmt++ to realistic scenarios. through a user study, we show that we can use llms to assist humans in understanding an extremely low-resource language, even if both the llms and the humans have no prior knowledge about this language. ## conclusions in this paper, we investigate whether llms can learn a completely new language on the fly. our ex-periments on the zhuang language show that llms can rapidly grasp an unseen language through proper icl. however, challenges persist in analyzing more intricate morphological phenomena and achieving a more comprehensive and precise understanding of syntax. we hope that zhuangbench and dipmt++ can encourage more research efforts on efficient methods for underrepresented languages.