Back to School: Translation Using Grammar Books (2024)

Abstract: Machine translation systems for high resource languages perform exceptionally well and produce high quality translations. Unfortunately, the vast majority of languages are not considered high resource and lack the quantity of parallel sentences needed to train such systems. These under-represented languages are not without resources, however, and bilingual dictionaries and grammar books are available as linguistic reference material. With current large language models (LLMs) supporting near book-length contexts, we can begin to use the available material to ensure advancements are shared among all of the world’s languages. In this paper, we demonstrate incorporating grammar books in the prompt of GPT-4 to improve machine translation and evaluate the performance on 16 topologically diverse low-resource languages, using a combination of reference material to show that the machine translation performance of LLMs can be improved using this method.

## introduction machine translation systems have progressed remarkably, but they require massive amounts of parallel sentences @xcite . more recently, instruction-tuned large language models (llms) have also proven capable of performing machine translation. however, their performance is best when translating among high-resource languages that were most likely seen during training. current transformer-based state-of-the-art large language models and mutlilingual translation models are trained on huge web-scraped corpora, with data in the order of trillions of tokens. while the web is a vast resource of good training data, 2 the web is also mainly comprised of just a handful of languages. there are an estimated 7000 languages in the world, but just 10 languages cover 84% of the web content, with english covering more than 50%. therefore, low-resource languages are not well-represented in the training 1 code and data to reproduce our experiments are here: https://github.com/jonathanhus/back-to-school . one approach to enabling machine translation for low-resource languages is to collect many parallel sentences. however, this is laborious, expensive, and time-consuming, requiring the skills of linguists and native speakers. another approach would be to incorporate language reference material into the translation process of the llm. the advantage of this approach is that a good number of dictionaries and grammar books have been created over decades @xcite and require little additional effort to use them. in this work, we push the frontier using the latter approach to improve on the ability of llms to perform machine translation of low-resource languages by utilizing available linguistic reference materials. we incorporate dictionaries, grammar books, and a small number of parallel sentences into the prompt of a state-of-the-art llm. we evaluate on 16 typologically diverse low-resource languages, performing analyses using different combinations of reference materials. ## related work while tens of high-resource languages have enjoyed the recent advances in machine translation, many of the world's 7000+ languages have been unable to partake in the success. the current state of the art in multilingual and low-resource translation is the no language left behind model @xcite , relying on a mined and curated corpus of parallel sentences for 200 languages, including many low-resource ones. a large multilingual encoder-decoder translation model was then trained on this data to create a machine translation system for these languages. on the other end of the spectrum, tanzer et al. (2023) incorporated dictionaries, sentences, and grammar books to perform machine translation in a zero-shot setting, i.e., in a language without any other data available, akin to how a documentary linguist or any second-language learner might learn a new language ("machine translation from one book (mtob)"). this paper inspired our own work, as it provides a framework for using llms to perform translation of resource-scarce languages. however, they were limited in the size of the context for the models they chose, and therefore, were only able to extract smaller chunks of the grammar book for inclusion. here, we explore this paradigm in a much larger scale, with 15 more languages, performing additional necessary analyses. last, @xcite explored a similar path utilizing grammar books. they were also limited by the size of the model context, but they additionally used a morphological analyzer on the grammar books to extract linguistic features to assist in translation. such tools are unfortunately unavailable for all languages, making this approach not feasible for scaling to thousands of languages. ## preliminaries and problem definition a traditional neural mt system models p mt (y|x), learned over source-target sentence pairs ⟨x, y⟩. at inference time, given a new source sentence, we sample a high-probability output from the learned distributions. a sota llm, however, is first pre-trained to model p lm (x) and then instructiontuned on p lm-ins (y|π) over prompt-target text pairs ⟨π, y⟩ covering multiple downstream tasks (often including mt). at inference time, with a similar prompt we sample outputs from the final model. a translation prompt π(•) at a minimum needs to include the task definition t (e.g. "please translate the following sentence to french:") and the source sentence x: π(x, t). for learning to translate an entirely unseen language, @xcite crafted prompts π(x, t, d, s, g) that additionally included: • word-level translations d obtained from a bilingual dictionary d, selected for their similarity to the words of the given source sentence, • a few parallel sentence examples s, selected from a small collection of parallel sentences s for their similarity to the given source sentence, and • excerpts g from a grammar book g, also selected for similarity to the source sentence using longest common substring distance. dictionaries we obtain dictionaries from pan-lex foot_0 for all our languages. note that, in cases where the number of words in the dictionary was less than 100 we do not include them in the prompt. the size of each dictionary is included in appendix b. parallel sentences for the parallel sentences that are part of the prompts as translation examples, we use the dev portion of the flores-200 dataset. 4 gitksan and natugu are not represented in flores and instead we use the data that @xcite provided. grammar books the dream corpus @xcite ) contains digitized versions of thousands of linguistic documents, including grammar books and sketches, for many languages. the source of these documents is often in paper format, and due to the scanning/ocr quality, the digitized versions often contain scanning artifacts. we select one grammar document for each of our languages (concrete details in appendix b). we perform slight manual cleanup to remove some items (e.g., scanning artifacts, table of contents) and to ensure that the grammar would fit in the llm's context size. ## results we report two comparison models: baseline corresponds to 0-shot llm translation performance i.e., only with prompt π(x, t), and the "skyline" performance of nllb, the current sota multilingual mt model. we also report results by adding words (w: π(x, t, d)), sentences (w+s: π(x, t, d, s), and grammars (w+s+g: π(x, t, d, s, g)) to the prompt. for each language and direction, we have four systems that we compare. we compute all evaluation metrics using sacrebleu foot_2 @xcite and we also report statistical significance using paired bootstrap resampling, comparing our best-performing system to the other systems. in most cases, we find that the difference is statistically significant, indicating that the translation performance is dependent on the selected prompt content. ## conclusion in this paper, we showed that utilizing reference material such as dictionaries and grammar books in the prompt of an llm can improve the performance of machine translation for low-resource languages. we evaluated the performance on 16 languages and showed that the improvement is especially pronounced for languages that have minimal presence on the web. our work shows that this approach has the potential to address the gap for extremely low-resource languages and identifies a concrete path for improving mt for more than 2,000 languages.