C hat GPT Beyond E nglish: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023)

Abstract: Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. In particular, we evaluate ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. Compared to the performance of previous models, our extensive experiments demonstrate the worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.

## introduction since the introduction of word embeddings @xcite and deep learning architectures @xcite , natural language processing (nlp) has witnessed significant breakthroughs that fundamentally transform research and applications in various areas. starting with the creation of word2vec @xcite , the major milestones in nlp involve the presentation of the seq2seq or encoder-decoder framework @xcite , the proposal of the attention mechanism @xcite , the development of transformer architecture @xcite , the notion of uncontextualized word embeddings from language models in elmo @xcite , and the pre-trained transformerbased language models, e.g., bert @xcite , gpt @xcite , t5 @xcite , and bart @xcite . the recent advances in nlp feature large language models (llms) that have parameter sizes over a hundred billion and are pre-trained on massive data, e.g., @xcite , megatron @xcite , gpt-jurassic @xcite , opt-175b @xcite , and multilingual bloom @xcite . although still relying on the transformer architecture, the unprecedented scales of model size and training data have allowed new emergent abilities to change the landscape and practices in nlp @xcite . an important emergent skill involves prompt-based learning that facilities the probing of information from llms with prompts by sampling the learned language distributions @xcite . in this way, the models demonstrate strong generalization in few-shot and zero-shot learning while avoiding parameter updates for the underlying architectures. to this end, chatgpt foot_0 is one of the latest developments in nlp. in the first two months of its launch, chatgpt has attracted 100 million users @xcite . as the next iteration of instructgpt @xcite , chatgpt is optimized on top of a gpt-3.5 series model using reinforcement learning from human feedback (rlhf) @xcite . in contrast to pre-vious llms, chatgpt and instructgpt leverage human demonstrations of desired outputs for input prompts to train supervised models, while human rankings of generated outputs are obtained to train a reward model to further optimize the llms with reinforcement learning. compared to instructgpt, chatgpt is trained with conversational data to allow follow-up questions. in this way, chatgpt is able to interact with humans in multi-turn conversations to generate more aligned outputs with human interests, thus being more natural and accessible to users. in addition, due to the deployment of public apis to facilitate general users, there have been multiple reports on the successes of chatgpt in solving challenging tasks in various areas, e.g., passing the united states medical licensing examination @xcite and real exams in a law school @xcite , performing competitively with commercial translation services for some high-resource languages @xcite , and even producing code from natural language instructions. nonetheless, the communities also express concerns about long-term implications of chatgpt and llms for society, citing issues on plagiarism, privacy, misinformation, and security @xcite . similar to other llms, chatgpt is trained on a mix of training data from multiple languages. although english is the majority, the combination of multilingual data contributes to chatgpt's abilities to accept inputs and generate responses in different languages, making it accessible and widely adopted by people around the world. however, given the recency of the technology, chatgpt has been mainly evaluated over english data. the community is lacking a comprehensive, public, and independent evaluation of chatgpt over various non-english languages for diverse nlp tasks to provide proper perspectives for future research and applications. given chatgpt's transformative potentials, associated long-term risks, huge cost for training, and limited transparency, a fundamental question is whether multilingual llms such as chatgpt can also be reliably adopted for different languages or it is necessary to develop languagespecific llms/other technologies to solve nlp problems for non-english languages. to address the multilingual concerns for chat-gpt, a few recent studies have investigated chatgpt's performance and responses for non-english languages. however, the considered tasks/languages/settings and scale of evaluation data in existing multilingual evaluations are still limited, which is unable to show a comprehensive picture of the potentials/performance of the technology on a diversity of other languages. for instance, @xcite evaluates the multilingual performance of chatgpt on three tasks of language identification, sentiment analysis, and machine translation; however, only a few languages are selected for each task and the number of evaluation samples for each language does not exceed 50. beyond english, the analysis of chatgpt's responses for input questions in @xcite is only done for chinese, while the results of the medical licensing examinations for chatgpt are only shown for japanese in @xcite . in addition, @xcite and @xcite explores chatgpt in three languages english, chinese, and german; however, the studies only focus on grammatical error correction or cross-lingual summarization. to this end, our paper aims to perform a more thorough evaluation of chatgpt for its performance on multiple languages over different nlp tasks. our experiments consider 37 diverse languages, characterizing high-, medium-, low-, and extremely low-resource languages, to better highlight chatgpt's potentials and limitations. to our knowledge, this is one of the largest sets of languages evaluated for chatgpt in a public study to date. in addition to natural language inference (nli), question answering, and common sense reasoning, our current work will examine the tasks of part-of-speech (pos) tagging, named entity recognition (ner), relation extraction, and summarization, which are not covered in previous multilingual evaluations for chatgpt. to improve the reproducibility of the evaluations and better reflect the approach of general users, our current work will focus on the zero-shot learning setting for chatgpt where no human-provided examples are presented to the model. importantly, due to the scale of available languages/tasks/datasets/models and the growing nature of multilingual learning research in nlp, we will use this work as an ongoing and public effort to evaluate chatgpt and other llms for multiple languages, emphasizing on understudied languages to measure robustness and democratize impacts of the technologies. despite some potential updates with future experiments, our current experiments suggest the following tendencies: ## related work since the release of chatgpt in november 2022 with impressive language abilities, there has been a growing interest in evaluating chatgpt for different aspects of natural language understanding. the first line of work concerns the performance comparison of chatgpt and state-of-the-art systems for important tasks in nlp such as text summarization @xcite , machine translation @xcite @xcite , question answering @xcite @xcite , information extraction @xcite ), text classification @xcite , grammatical error detection @xcite , and stance detection @xcite . along this line, several recent studies have attempted to examine the performance of chatgpt more comprehensively on multiple datasets @xcite @xcite . the second direction for chatgpt evaluation focuses on the robustness/reliability of the model against possible variants of input texts. for example, @xcite explores the robustness of chatgpt under the adversarial and out-of-domain learning settings while @xcite examines the logical prediction consistency of chatgpt for inputs with semantic equivalence, logical negation, or symmetricity. finally, the third dimension for chatgpt evaluation discusses the potential impacts and risks of the technology for the broader society, e.g., in education @xcite , law @xcite , medical @xcite , ethnics @xcite , human-computer collaboration (lanzi and loiacono, 2023), and cognition @xcite . however, to our knowledge, none of existing work has conducted large-scale evaluations of chatgpt for multiple and diverse languages/tasks as we do. ## methodology the goal of our research is to evaluate the performance of chatgpt and llms for nlp tasks in different languages. given the large numbers of nlp datasets/tasks/languages and the growing developments of llms, our work will be an ongoing effort to include additional experiments to be more comprehensive along the way. in the current version of the paper, we will evaluate chatgpt on seven diverse nlp tasks, i.e., part-of-speech (pos) tagging, named entity recognition (ner), relation classification, natural language inference (nli), question answering (qa), common sense reasoning (csr), and summarization. over different tasks, our experiments will cover 34 diverse languages, characterizing high-, medium-, low-, and extremely low-resource languages to provide broader perspectives. following @xcite , we employ the ratio of the data for each language in the commoncrawl corpus foot_1 , i.e., the main data to pre-train gpt-3, to classify the resource levels. in particular, a language will be considered as high-, medium-, low-, and extremely low-resource if its data ratio is greater than 1% (> 1%), between 0.1% and 1% (> 0.1%), between 0.01% and 0.1% (> 0.01%), and smaller than 0.01% (< 0.01%) respectively. different from some previous work that exploits two-stage prompting to adopt a zero-shot chain of thoughts @xcite , we directly utilize single-stage prompting that only adds the task description d into each input x to simulate the common approach of general users for chatgpt. other prompting strategies can be explored in future work. as such, in the current version, we aim to design simple task descriptions d while ensuring necessary information to indicate the task and facilitate the parsing of responses to produce accurate outputs y . in addition, for tasks in a non-english target language, we will evaluate task descriptions in both english and target-specific languages to shed light on the best approach to prompt chatgpt in multilingual settings. to facilitate the experiments, all non-english task descriptions are obtained using the automatic translation tool google translate foot_2 to translate the designed english descriptions for each task. finally, all of the responses from chatgpt in this work are obtained between march 1 and april 5. this is right after chatgpt is made available in openai apis to enable large-scale requests from the public for comprehensive evaluations. to improve reproducibility, we clear the conversations in chatgpt for each query to remove any previous context. in the following, due to the space constraint, we will only describe the tasks, datasets, and chatgpt's performance. the designed prompts for each task will be provided in the appendix. ## part-of-speech tagging part-of-speech (pos) tagging is a coarse-grained word classification task whose goal is to label the syntactic information of the words in a sentence. we evaluate chatgpt for its multilingual pos tagging abilities over the xglue-pos dataset @xcite , which covers 18 languages and includes labels derived from the universal dependencies (ud) treebanks (v2.5) @xcite . in the experiments, we utilize the xglue-pos dataset from huggingface datasets foot_3 that only includes 17 languages (e.g., excluding portuguese). as such, we use the test sets of xglue-pos with more than 15k samples for the selected languages in the evaluation. appendix a provides details for our pos tagging prompt for chatgpt. results: ## named entity recognition named entity recognition (ner) is an important task in nlp @xcite , aiming to identify spans and semantic types of names (e.g., person, organization) in text. ner is usually formulated as a sequence tagging problem where a label is assigned to each word in a sentence to indicate names. the bio annotation schema is often leveraged to form the labels to capture both span and type information @xcite . for multilingual ner evaluation of chatgpt, we employ the datasets from the recent shared task multiconer @xcite that seeks to build ner systems for 11 languages following the wnut 2017 taxonomy for entity types @xcite . there are 6 entity types in multi-coner, i.e., per (person), loc (location), corp (corporation), cw (creative work), grp (group of people), and prod (product). we utilize the test sets of the language in multiconer for evaluation. our prompt for the ner task is described in appendix b. results: results: ## natural language inference natural language inference (nli) aims to predict the entailment/contradiction relations between two input sentences, i.e., a premise and a hypothesis. to evaluate chatgpt for multilingual nli, we utilize the xnli dataset @xcite that provides annotated data for english and 14 other languages with three categories, i.e., entailment, contradiction, and neutral. as such, the data in non-english languages is obtained by translating english data for xnli. xnli provides development and test data to facilitate development and evaluation. however, as the labels for the test data are not publicly available, we utilize the development data of xnli in this experiment. the chat-gpt prompt for nli is described in appendix e. results: ## question answering given a context passage and a question, a question answering (qa) model needs to return the answer for the question, which should be a span of text in the input passage. to this end, we utilize the xquad dataset @xcite to evaluate chatgpt in multiple languages for qa. xquad involves 240 paragraphs and 1190 question-answer pairs in english and their translations into ten other languages for evaluation. we describe our chat-gpt prompt for qa in appendix f. given the responses from chatgpt for our qa prompts for the examples, we remove the period characters in the end and directly evaluate remaining responses using the squad's scorer 5 , which is suggested by the original paper of xquad @xcite results: ## common sense reasoning common sense reasoning (csr) evaluates the reasoning of the models via multiple-choice questions. the inputs for the models involve a question and a few choices for the answer, and the models need to select one of the choices. to evaluate chatgpt's multilingual abilities for csr, we leverage two datasets: (i) x-csqa @xcite , which involves english data and its translations to 15 other languages, and (ii) wikipedia cloze qa from indicnlpsuite @xcite , which covers 11 low-and extremely-low-resource indian languages. we evaluate the models on the dev set of x-csqa with 1,000 samples for each language, while the wiki cloze qa dataset from indicnlpsuite contains 62,314 samples for all languages. appendix g presents our chatgpt prompt for csr. results: for indicnlpsuite, finally, our chatgpt evaluation for multilingual summarization is included in appendix h. ## discussion the most important findings from our experiment results is that chatgpt exhibits significantly worse performance than state-of-the-art supervised models for most of considered nlp tasks in different languages. given the huge costs to train chatgpt and similar llms as well as the necessity of paid apis to run large amounts of requests with ope-nai, it seems more reasonable to build smaller task-specific models for nlp problems (or at least for the considered tasks) in different languages that can be hosted locally to serve at lower costs. in addition, we notice an exception for the pos tagging task where chatgpt can achieve competitive or even better performance than the supervised learning models (especially with english prompts) over different languages. for instance, chatgpt has significantly better pos tagging accuracy for thai, vietnamese, bulgarian, hindi, and urdu, which are medium-and low-resource languages. as such, in contrast to other considered tasks which require some level of semantic reasoning, pos tagging focuses on low-level syntactic analysis. we thus hypothesize that chatgpt possesses high-level skills in grammar and low-level abilities of semantic reasoning to generate seemingly fluent texts for multiple languages. however, for more complicated semantic analysis, chatgpt might find it more challenging to perform accurate predictions and generations. regarding the classification of high-, medium-, low-, and extremely low-resource languages, our work currently relies on data ratios for the languages in the commoncrawl corpus. according to our experiments, it is interesting that the performance of chatgpt for low-and extremely-lowresource languages in some tasks is better or comparable to those for high-or medium-resource languages. for instance, for pos tagging in compared to language-specific prompts, the superior performance of chatgpt with english task descriptions over a majority of problems and languages suggests that chatgpt might better understand/analyze the tasks with english prompts to lead to improved abilities to generate responses with accurate outputs. in addition, the inclusion of english task descriptions for non-english inputs can be seen as an approach to shift the representations of language-specific inputs toward the english space that can be better processed by chatgpt due to the domination of english in its training data. however, we also note some recent work that reveals a rather different findings, suggesting that chatgpt can perform competitively or even better with language-specific prompts for nlp tasks in target languages @xcite . a reason for those different findings might come from potentially different versions of chat-gpt at different times that are used to conduct the studies. it thus highlights the importance of better transparency for llms, e.g., with respect to training data @xcite , to allow accurate and deeper investigation of the models. finally, the better performance with english prompts also raises an interesting question on whether english is the optimal language to prompt chatgpt or it is better to employ other languages for this purpose for different target languages. ## conclusion toward a more comprehensive understanding of chatgpt and llms on their multilingual learning abilities for nlp, our work conducts an evaluation for chatgpt on 7 different tasks, i.e., part-of-speech tagging, named entity recognition, relation extraction, natural language inference, question answering, common sense reasoning, and summarization. using 37 diverse languages with high-, medium-, low-, and extremely low resources for the experiments, our results reveal the less optimal performance of chatgpt in the zero-shot learning setting for nlp tasks in different languages, advocating for task-specific models to secure best performance. as an ongoing research, we plan to extend the experiments to include more languages, tasks, models, criteria, and settings in future work to obtain broader and deeper insights.