C hat GPT is not a good indigenous translator (2023)

Abstract: This report investigates the continuous challenges of Machine Translation (MT) systems on indigenous and extremely low-resource language pairs. Despite the notable achievements of Large Language Models (LLMs) that excel in various tasks, their applicability to low-resource languages remains questionable. In this study, we leveraged the AmericasNLP competition to evaluate the translation performance of different systems for Spanish to 11 indigenous languages from South America. Our team, LTLAmsterdam, submitted a total of four systems including GPT-4, a bilingual model, fine-tuned M2M100, and a combination of fine-tuned M2M100 with $k$NN-MT. We found that even large language models like GPT-4 are not well-suited for extremely low-resource languages. Our results suggest that fine-tuning M2M100 models can offer significantly better performance for extremely low-resource translation.

## introduction this paper presents the participation of the language technology lab (ltl) from the university of amsterdam in the americasnlp 2023 shared task, which aims to develop machine translation (mt) systems for indigenous languages of the americas. we submitted translation results for spanish into all indigenous languages: hñähñu (oto), wixarika (hch), nahuatl (nah), guaraní (gn), bribri (bzd), rarámuri (tar), quechua (quy), aymara (aym), shipibo-konibo (shp), asháninka (cni), and chatino (czn). in the face of limited parallel and monolingual data, our approaches focus on maximizing the potential of available resources and models. specifically, our objectives include: 1) evaluating the performance of gpt-4, a state-ofthe-art language model, in extremely low-resource settings; 2) utilizing a carefully optimized transformer setting for low-resource nmt @xcite ; 3) exploring the effectiveness of a fine-tuned version of the multilingual m2m100 @xcite model; and 4) investigating the potential of augmenting a neural model with a k-nearest-neighbor machine translation (knn-mt) @xcite component. our experiments show that, while gpt-4 demonstrates comparable performance to bilingual models trained from scratch, it significantly lags behind the fine-tuned m2m100 models. notably, our m2m100+knn-mt system emerged as our top-performing approach. ## results we report chrf++ scores @xcite in our knn approach yields best results for 10/11 language directions, and the fine-tuned m2m100 is the best model that does not encode the development set. compared to other submissions, our knn model ranks first for spanish-bribri, spanish-asháninka, and spanish-nahuatl, but we decided to withdraw this model (see section 3.4). ## conclusion in this paper, we describe our submissions to the americasnlp 2023 shared task on machine translation into indigenous languages. we submitted translations for all 11 languages. our best system is the result of finetuning m2m100 on an unseen indigenous language, and augmenting this model with a k-nearest-neighbor datastore based on the development set. this model ranked first in the spanish-bribri, spanish-asháninka, and spanish-nahuatl language pairs in the competition. however, we have made the decision to withdraw this model due to its operation in a grey area with respect to the competition rules. the uncertainty surrounding its compliance raises concerns about fairness among all participants, prompting us to take this action after discussion with the organizers.