Don’t Trust C hat GPT when your Question is not in E nglish: A Study of Multilingual Abilities and Types of LLM s (2023)

Abstract: Large language models (LLMs) have demonstrated exceptional natural language understanding abilities, and have excelled in a variety of natural language processing (NLP) tasks. Despite the fact that most LLMs are trained predominantly on English, multiple studies have demonstrated their capabilities in a variety of languages. However, fundamental questions persist regarding how LLMs acquire their multilingual abilities and how performance varies across different languages. These inquiries are crucial for the study of LLMs since users and researchers often come from diverse language backgrounds, potentially influencing how they use LLMs and interpret their output. In this work, we propose a systematic way of qualitatively and quantitatively evaluating the multilingual capabilities of LLMs. We investigate the phenomenon of cross-language generalization in LLMs, wherein limited multilingual training data leads to advanced multilingual capabilities. To accomplish this, we employ a novel prompt back-translation method. The results demonstrate that LLMs, such as GPT, can effectively transfer learned knowledge across different languages, yielding relatively consistent results in translation-equivariant tasks, in which the correct output does not depend on the language of the input. However, LLMs struggle to provide accurate results in translation-variant tasks, which lack this property, requiring careful user judgment to evaluate the answers.

## introduction the study of bilingualism has long been a topic of interest among linguists @xcite , as it provides insight into the mechanisms of language acquisition and processing. furthermore, research on multilingualism has contributed to the development of more effective machine learning models, such as neural translation systems @xcite . with the rise of large language models (llms), researchers have discovered many emergent properties @xcite in these models, and have used them for a variety of purposes @xcite . however, the multilingual ability of these models has not been extensively studied. previous research has shown that large language models, such as gpt, are capable of performing a wide variety of language tasks when the task is presented in english @xcite . however, investigations into the multilingual language abilities of these models have been limited. @xcite explore this topic by applying the models to multilingual datasets, and measuring performance differences across languages. however, they do not explore the underlying mechanisms of how llms perform different tasks, nor how this affects the results. moreover, most llms @xcite are trained on datasets that are heavily skewed towards english, which leaves open the question of how multilingual abilities in such models are acquired. in this study, we present a systematic approach to analyzing the multilingual capabilities of llms. to facilitate a comprehensive analysis, we propose categorizing language-dependent abilities into three distinct categories which vary in the impact of language choice on the performance: reasoning (least impact), knowledge access, and articulation (most impact). we investigate a carefully selected set of tasks from these three categories by evalu-ating the multilingual abilities of an llm using a novel prompting method which we call response back-translation (rbt). by comparing the generated answers, we can both measure multilingual performance of the llm, but also determine the type of multilinguality they exhibit. for example, we examine the capabilities of llms on pun detection, a highly language-dependent task. the results of our experiments show that the popular llm "gpt": (1) achieves higher performance when the task is presented in english; (2) achieves higher performance on tasks that can be translated without altering the correct output; and (3) exhibits a mixture of coordinate and subordinate bilingualism. our main contributions 1 are: ## background linguists categorize bilingual individuals into three groups: compound, coordinate, and subordinate bilinguals (d' @xcite . figure 1 illustrates this categorization, showing how individuals with different types of english-french bilingualism might internally represent the concept of "knife". compound bilingualism mostly emerges among individuals who learn two languages simultaneously from birth. in this case, both languages are equally dominant and integrated, blurring any clear distinction between them and giving the impression of a single unified language @xcite . compound bilingualism entails a shared mental representation of lexicons across both languages they acquire, and compound bilinguals are the most flexible in their use of multiple languages, exhibiting the ability to switch between languages without losing consistency in linguistic tasks (de @xcite . in contrast, individuals exhibiting coordinate bilingualism maintain separate mental representations for the lexicon of each language they 1 our data is publicly available at github. learn. this separation leads to differences when tasks are performed under different language settings @xcite . finally, subordinate bilingualism is characterized by a "translator" behaviour @xcite . this type of bilingualism is characterized by a single lexicon representation that is linked to their dominant language @xcite . when performing tasks in languages other than their dominant one, subordinate bilinguals tend to rely on translating the task into their dominant language, formulating an answer in the dominant language, and then translating that answer back into the language of the task. as a result, subordinate bilinguals may experience lower proficiency in communicating and completing tasks in the second, subordinate language. despite the demonstration in prior work of consistent multilingual performance in many large language models @xcite , it remains unclear how the multilingualism of llms should be categorized. it is an open question whether the llms exhibit a representation of knowledge shared across both languages (compound), separate representations for each language (coordinate), or whether they rely on a translation processes (subordinate). we develop an experimental framework aimed at using performance on various natural language processing tasks to determine how the multilingual abilities of llms relate to these categories. ## categorizing language-dependent tasks language ability is a multifaceted concept encompassing various tasks and aspects @xcite . it is therefore difficult to assess a model's capabilities with respect to a given language. to facilitate such assessment, researchers have often classified tasks into distinct categories @xcite , such as parsing and summarization. however, the delineation of such categories often lacks systematic criteria, particularly in the context of multilingual analysis. in this section, we propose a novel approach to categorizing nlp tasks, which is better suited to analysis of multilingual abilities. the categorization is two dimensional: one dimension is based on the linguistic knowledge necessary to complete the task (section 3.1), the other on how the task is impacted by the language in which it is presented (section 3.2). ## methods in this section, we present our approach to analyzing the multilingual ability of llms. our methods involve prompt translation (pt) and response backtranslation (rbt). they are designed to measure performance of an llm, and its consistency across languages. in our experiments, we apply these methods to both te and tv tasks, with the aim of determining the type of bilingualism (compound, coordinate, or subordinate) exhibited by an llm. ## experiments we apply the methodology proposed in section 4 to te and tv tasks. as our llm, we use chat-gpt, via the official web application foot_0 , due to its availability. ## discussion and future directions our research provides robust support for the notion that llms have not achieved the ideal behaviour of compound multilingualism. even if the quality and quantity of training data in various languages were held constant, we speculate that compound multilingualism would still not be achieved, due to the inherent limitations of current data collection methods and training techniques. drawing a parallel to human multi-modal learning offers an intuitive understanding of why this could be the case. consider how humans acquire concepts related to vision and language: a child grows by consistently pairing visual stimuli with linguistic cues, intertwining the two modalities over time. consequently, it is rare to observe a mismatch between visual and linguistic perceptions. in this context, humans exhibit a highly integrated understanding of vision and their native languages. however, unless raised in a perfectly bilingual environment, individuals seldom showcase equivalent proficiency in two languages. indeed, bilingual individuals often demonstrate cognitive variations depending on which language is in use. a rudimentary multi-modal system can be likened to a crude fusion of a vision model trained on image data and a language model trained on text. these systems possess minimal, if any, shared representations or information overlap. beneath the facade of a system that seemingly excels at both visual and language tasks, lie two distinct networks. nevertheless, recent advancements in multi-modal studies, combined with the availability of extensively captioned image data, have given rise to more sophisticated systems. these systems bridge the gap between the two modalities, moving the field closer to human-like integration. acquiring aligned multilingual data is a significant challenge, with the exception of some translation datasets. the majority of online articles and posts are monolingual and cannot be easily paired. therefore, training on these multilingual corpora results in models that essentially act as an amalgamation of several independent language-specific models, with minimal information interchange, pri-marily anchored by the translation datasets which comprise a relatively small portion of the corpus. when corpora are disproportionately comprised of some language or set of languages, the models tend to become predominantly subordinate, with minimal coordination arising from monolingual datasets. moving forward, our objective is to narrow the divide between languages within a multilingual system and to cultivate language models that lean more towards a compound archetype. this will require both crafting highly parallel paired data across languages and innovating training methodologies that promote the learning of compound representations for universal concepts irrespective of the language used to express them. for the former, we intend to delve into ontology linkages. for the latter, we plan to leverage recent advancements in model training, such as contrastive learning. our goal is to create multilingual models that are both technically sophisticated and universally adept. ## conclusion we have proposed a systematic approach to analyzing multilingual abilities of large language models. our experiments provide new evidence for a subordinate multilingualism in gpt-3.5, with english functioning as the model's native language. our experimental results, supplemented by the analysis of specific examples and case studies, demonstrate that such subordinate multilingualism can limit performance even in language-independent tasks. we postulate that explicit inclusion of additional multilingual parallel corpora and multimodal datasets into the training data of llms could ameliorate this issue.