Fumbling in B abel: An Investigation into C hat GPT ’s Language Identification Ability (2024)

Abstract: ChatGPT has recently emerged as a powerful NLP tool that can carry out a variety of tasks. However, the range of languages ChatGPT can handle remains largely a mystery. To uncover which languages ChatGPT ‘knows’, we investigate its language identification (LID) abilities. For this purpose, we compile Babel-670, a benchmark comprising 670 languages representing 23 language families spoken in five continents. Languages in Babel-670 run the gamut from the very high-resource to the very low-resource. We then study ChatGPT’s (both GPT-3.5 and GPT-4) ability to (i) identify language names and language codes (ii) under zero- and few-shot conditions (iii) with and without provision of a label set. When compared to smaller finetuned LID tools, we find that ChatGPT lags behind. For example, it has poor performance on African languages. We conclude that current large language models would benefit from further development before they can sufficiently serve diverse communities.

## related work traditional approaches to language identification involved rule-based methods @xcite , statistical models @xcite , and handcrafted features such as character combination cooccurrence (van der @xcite @xcite and feature smoothing @xcite . recently, deep learning has revolutionized language identification techniques by showing superior performance @xcite . the efficacy of deep learning methods in addressing lid for high-resource languages has generally been established as a resolved issue @xcite . however, the domain of lid for low-resource languages remains significantly under served @xcite . chatgpt is a general-purpose language model that is capable of performing a variety of language tasks @xcite . it has been claimed to have the ability to solve any kind of task without taskspecific training or in few-shot settings @xcite . in this work, we eval-uate the performance of chatgpt on lid on both low-resource and high-resource languages. to the best of our knowledge, there are no prior works that evaluate chatgpt on lid. we now introduce bable-670, our dataset for this work. ## dataset we curate babel-670, a dataset for our lid task compiled from three different datasets that cover a total 670 languages from 24 language families (shown in appendix table a.1) written in 30 different scripts (shown in appendix table a.2). a full list of languages included in babel-670 can be seen in appendix d. ## methodology to explore chatgpt's ability to identify languages, we design two major types of prompts: lan-guage name prompt (lnp) and language code prompt (lcp), each encompassing three numbers of shots and three different difficulty levels (see section 4.2). an overview of our data pipeline is shown in figure 2 . lnp asks chatgpt to predict language names while lcp asks it to produce threedigit is0-693 language codes. although most language identification research use language code as labels, we decide to also prompt chatgpt to predict language name because (1) it will be a very strong assumption that chatgpt knows all iso language codes and (2) we hypothesize chatgpt is more likely to be fed language names during pretraining. in fact, through analysis, we observe that chatgpt predicts language names better than language codes given the same set of test examples (see section 5.1.1). ## conclusion to investigate chatgpt's ability to identify human languages, we curate babel-670, which is a dataset that covers 670 languages spoken in five continents, belonging to 24 language families and are written in 30 different scripts. we prompt two versions of chatgpt to produce language names and language codes, each with a different number of exemplars with and without provision of a label set. we conduct comprehensive analyses focusing on errors, geographic distribution, and script variations on the results retrieved with our proposed novel evaluation method that takes language aliases and dialects into consideration. we find that chatgpt has an uneven ability at identifying languages. the model is able to identify one hundred languages @xmath0. geographically, among the five continents, african languages have the least support by chatgpt. the investigation demonstrates that chatgpt is still a considerable distance away from serving wide and diverse communities adequately.