Hire a Linguist!: Learning Endangered Languages in LLM s with In-Context Linguistic Descriptions (2024)

Abstract: How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LingoLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM’s prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LingoLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LingoLLM elevates translation capability from GPT-4’s 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations will be released to the public. Our data, code, and model generations can be found at https://github.com/LLiLab/llm4endangeredlang .

## introduction large language models (llms) are already powerful in many language understanding and generation tasks @xcite . their language processing capabilities rely on very large amounts of training data @xcite . for example, a recent llm llama-2 uses a pre-training dataset with 2 trillion tokens @xcite . while languages such as english or spanish enjoy abundant accessible data, the majority of the world's 7000 languages lack a rich corpus, including most endangered languages recognized by @xcite . existing llms such as llama @xcite and gpt-4 show poor performance figure 1 : among the world's ∼7000 languages, 95% don't have enough data (>100k sentences) for training llms @xcite , while most have a grammar book (60%) or dictionary (75%) @xcite , including many endangered languages @xcite . therefore, we utilize these linguistic descriptions to bring llms to endangered languages. on languages that may not occur in pre-training @xcite . we believe that speakers of endangered languages deserve equitable access to nlp technologies including llms. how can we enable an llm with language processing capabilities on unseen and endangered languages? we are motivated by how human linguists analyze utterances in a language they don't knowthey use existing grammar books and dictionaries. fortunately, thanks to the efforts of generations of linguists over the years, many endangered languages have published dictionaries and descriptive grammar. compared to llms' training corpora, which mostly consist of unstructured text, these linguistic descriptions have two major differences. first, they are instructional. though they are much smaller than typical training sets, they contain explicit grammar rules of a language that can be used as instructions for both llms and humans. second, linguistic descriptions have much broader coverage. as shown in figure 1 , very few languages have training corpora, but most have documented grammar or dictionary. however, directly using these linguistic descriptions in an llm's prompt is infeasible. a grammar book and a dictionary are often too large to fill in the prompt of an llm. in this paper, we propose lingollm, an efficient approach to enable an llm to process and translate unseen languages that never occur in its pre-training. our key insight is to properly exploit linguistic description of an unseen language, including a dictionary, a grammar book, and morphologically analyzed input text. lingollm first preprocesses input text in an endangered language via a morphological analyzer and a dictionary, both from linguistic descriptions of the language. the inputs, annotated with grammar features and wordlevel translations, are passed to an llm along with the grammar book. the llm then translates the endangered language inputs to a high-resource language like english to process them. lingollm is training-free as it only requires the underlying llm to be instruction-tuned. lingollm can adapt to languages according to the availability of different types of linguistic descriptions. we implement lingollm on top of two models, gpt-4 and mixtral. our experiments consist of a total of 5 tasks (including translation from/to english, mathematical reasoning, response selection, word reordering, and keyword-to-text) in 8 endangered/low-resource languages that are geographically and typologically diverse. as shown in figure 2 , lingollm significantly improves gpt-4's performance on all 5 tasks by a large margin. noticeably the translation quality increases from an incomprehensible 0.5 to 10 bleu points. our contributions are: • we propose lingollm, an approach to integrate linguistic descriptions to process and translate text in endangered languages. ## related work various recent studies explore the possibility of llms for low-resource languages on machine translation @xcite and other nlp tasks @xcite . their scope of evaluation mostly covers languages whose resources are low but still exist. hence llms can still have a non-zero translation ability. we go beyond their scope towards languages that are truly extinct where llms' zero-shot translation of them is near zero. moreover, our method relies on external linguistic descriptions rather than internal knowledge of llms, focusing on how llms can utilize information they don't know instead of how they can "recall" information they have seen in training. ## the lingollm approach since we do not have enough data to fine-tune an llm, lingollm equips an llm with linguistic knowledge to process text in an endangered language (figure 3 ). we obtain a morphological analyzer, a dictionary, and a grammar description for each target language. our method consists of four steps: 1) given an utterance in an endangered language, we first use a morphological analyzer to split each word into morphemes; 2) we search for the closest matches from a dictionary for each morpheme to obtain an annotated gloss; 3) we prompt an instruction-tuned llm with the annotated gloss and a grammar description to get the translation in a high-resource language such as english; and 4) we further process the translated text using a same llm for the downstream task. to adapt lingollm to a new language, we first collect linguistic descriptions with the help from linguists. we use three types of linguistic descriptions in lingollm -morphological analyzers, dictionaries, and grammar books. for all languages studied in this paper, we look for references to these descriptions on glottolog foot_0 and then collect them via web interfaces or ebooks. for one type of description, we build a universal interface to use them despite different underlying formats. we list the collected linguistic descriptions in appendix b. ## ablation and analysis we conduct ablation studies and qualitative analysis to show how helpful each component of lin-gollm is and explore the best ways in which they can be used. note that some of our ablation experiments depend on extra annotations such as oracle dictionary mappings and oracle glosses, which only exist for a subset of the languages. ## lessons for further extending lingollm we ran into many obstacles and caveats while collecting linguistic descriptions and building lin-gollm. since some of these caveats are common to other endangered languages, we record them here for readers interested in extending lin-gollm or similar approaches to more languages. many linguistic descriptions are scanned from physical books that involve typewritten and handwritten parts. converting them to plain text that can be tokenized is not easy, especially when there are non-latin scripts or infrequent alphabets. things get even more complicated when there are complex hierarchies in the organization of grammar books or when the examples and grammar descriptions are interleaved in the book. we had to give up adapting lingollm to several languages simply because of the digitization difficulty. ## conclusion in this paper, we introduced lingollm, a novel approach for enabling llms to process endangered languages. lingollm integrates linguistic descriptions such as grammar books and dictionaries, a critical resource that is often more available for endangered languages than extensive corpora. lin-gollm has demonstrated remarkable improvements on multiple tasks across many languages. our work with lingollm highlights the potential of existing linguistic resources in the era of advanced llms and how they might make endangered languages more accessible in modern technological contexts.