Analyzing Multilingual Competency of LLM s in Multi-Turn Instruction Following: A Case Study of A rabic (2023)

Abstract: While significant progress has been made in benchmarking Large Language Models (LLMs) across various tasks, there is a lack of comprehensive evaluation of their abilities in responding to multi-turn instructions in less-commonly tested languages like Arabic. Our paper offers a detailed examination of the proficiency of open LLMs in such scenarios in Arabic. Utilizing a customized Arabic translation of the MT-Bench benchmark suite, we employ GPT-4 as a uniform evaluator for both English and Arabic queries to assess and compare the performance of the LLMs on various open-ended tasks. Our findings reveal variations in model responses on different task categories, e.g., logic vs. literacy, when instructed in English or Arabic. We find that fine-tuned base models using multilingual and multi-turn datasets could be competitive to models trained from scratch on multilingual data. Finally, we hypothesize that an ensemble of small, open LLMs could perform competitively to proprietary LLMs on the benchmark.

## introduction recently, large language models (llms) have brought about significant disruptions across various domains in both research and industry. llms have shown strong capability in solving and generalizing across diverse and complex tasks in natural language processing (nlp) and beyond. moreover, their success in engaging in conversations and accurately following human instructions has been particularly noteworthy. the recent surge in the availability of llms necessitates extensive benchmarking and evaluation. in this work, we analyze the competency of publicly-available, open llms when prompted with open-ended, multi-turn instructions in a language different than english. we compare the quality of these responses to the ones generated from equivalent instructions in english in order to iden- figure 1: performance scores per category for selected llms on the original mt-bench @xcite for english. the model responses are evaluated by gpt-4 and scored on a scale of 1 to 10 using criteria of helpfulness, relevance, accuracy, depth, creativity, and level of detail. we start by a brief overview of the llm benchmarking effort in section 2. we introduce arabic mt-bench in section 3 as an analysis tool for multilingual instruction following. then, we attempt to answer the proposed questions through a number of analyses in section 4. finally, we conclude in section 5 with some insights and recommendations for pushing forward the competency of arabic llms. ## llm benchmarking llms have shown capabilities that go far beyond traditional nlp tasks, such as text classification or multi-choice question answering in some target natural language. their ability to generate human-like text and engage in long conversations in any topic have opened up a multitude of novel opportunities and horizons that transcend tasks and languages. however, many existing benchmarks for llms are still anchored in the conventional nlp paradigm or support english only. consequently, these benchmarks exhibit limitations when it comes to evaluating the proficiency of llms in open-ended generation, multi-turn tasks, or in languages other than english. ## arabic mt-bench in this work, we develop an arabic version of mt-bench. first, we auto-translated the original benchmarking questions using google translate. a thorough manual curation of the translations is then performed. this step is essential to ensure the quality of the question set and hence the responses and the judgment. for example, all people names in the questions were changed to arabic names, and questions about correcting english grammatical errors were re-written. see in addition to the questions, the benchmark provides reference answers for reasoning, math and code questions that are passed to the llm judge to aid in the judgment. one option to get these reference answers in arabic is to prompt gpt-4 with the translated arabic questions directly, but we decided instead to translate the original reference answers from english to ensure that the arabic scores for these three categories stay as close as possible to the english mt-bench scores. finally, our initial evaluation showed that some llms tend to respond in english despite the question being in arabic. hence, we decided to add at the end of each question a clear instruction to the llm to respond in arabic ( ). we observed that, without having to modify the original judgment prompt, gpt-4, acting as an arabic judge, has taken into consideration that instruction and scored lower responses in english. ## conclusion in this paper, we propose a framework for analyzing the effect of multilinguality on llm performance in open-ended tasks. in particular, we assessed the interaction between language, dialog and instruction following in arabic and english for small open llms. we employ an llm as a judge following the paradigm of mt-bench. we show the effects of language on different categories of tasks and suggest ways to ensemble small llms to achieve better performance on the benchmark. in future work, we plan to extend the benchmark and analysis with more models and tasks, and investigate the viability of llm ensemble models.