C hat GPT MT : Competitive for High- (but Not Low-) Resource Languages (2023)

Abstract: Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs’ MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world’s diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language’s resource level is the most important feature in determining ChatGPT’s relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.

## introduction despite the majority of the world's languages being low-resource, current mt systems still perform poorly on them or do not include them at all. some commercial systems like google translate foot_0 support a number of lrls, but many systems do not support any, and in either case the majority of lrls are largely neglected in language technologies. in recent years, generative llms have shown increasingly impressive translation abilities @xcite . even more recently, llm tools like chatgpt have become popular and accessible to end users. this marks an important shift, since a majority of llm users are now consumers rather than researchers. the prospect of llm translation is exciting, since theoretically, generative llms could support more languages than commercial systems like google's. 2 but only beginning steps have been made to test this hypothesis. while some studies outlined in §4 have evaluated mt with recent llms, evaluation is still lacking for many languages. this brings up important questions, such as: can end users in need of mt for a variety of languages use chatgpt? are chatgpt and other llms reliable translators? for which languages are they reliable? initially we hypothesize that llms translate hrls better than lrls. but due to limited information about the training data and methods for powerful llms like chatgpt (gpt-3.5 and variants) and gpt-4, hypotheses like this must be experimentally verified. we significantly expand experimental verification for such hypotheses by testing chatgpt's performance on the flores-200 benchmark @xcite , containing 204 language varieties. we emphasize that, rather than optimizing llm mt for a few languages, we focus on helping end users of various language communities know how and when to use llm mt. we expect that our contributions may benefit both direct end users, such as lrl speakers in need of translation, and indirect users, such as researchers of lrl translation considering chatgpt to enhance specialized mt systems. in summary, we contribute: our experiments are motivated by the interests of llm users speaking a variety of languages. in addition to evaluating a large language set ( §3), we chose to analyse language features ( §3.4), to draw generalizations for even more lrl speakers. we compare mt costs because they impact end users ( §3.7). we keep chatgpt central to our analyses because of its current popularity among consumers. ## methodology we used data for 204 language varieties from flores-200 @xcite . we used the 1012 devtest sentences for our main experiments and the 997 dev sentences for follow-up experiments. we queried the openai api foot_2 to translate our test set from english into the target languages. we explored @xmath0. thus openai's gpt models were likely trained on those exact english sentences, making fair @xmath1. ## related work we are not the first researchers to explore llm mt. however, most existing studies do not provide benchmarks for a large number or languages. wang et al. (2023) ## conclusion we provide benchmarks for llm @xmath2, with comparisons to state-of-the-art commercial and opensource mt models. for many hrls, llms like chatgpt perform competitively with these traditional models. but for lrls, traditional mt remains dominant, despite llms' increased parameter size. our decision-tree analysis reveals language features that predict chatgpt's translation effectiveness relative to nllb, finding that chatgpt is especially disadvantaged for lrls and african languages, and that the number of wikipedia pages a language has is a strong predictor of chatgpt's effectiveness in it. we present evidence that few-shot learning offers generally marginal improvements for @xmath3, which may not justify its additional cost. we provide mt users with scores and cost estimates for four llm and traditional mt systems, to help them determine which to use for their languages. future work may include more translation directions (@xmath4, document-level mt, and human evaluation of llm outputs to reveal trends along fluency and accuracy dimensions. we open-source software and outputs of the models we evaluated on our repository.