TARJAMAT : Evaluation of Bard and C hat GPT on Machine Translation of Ten A rabic Varieties (2023)

Abstract: Despite the purported multilingual proficiency of instruction-finetuned large language models (LLMs) such as ChatGPT and Bard, the linguistic inclusivity of these models remains insufficiently explored. Considering this constraint, we present a thorough assessment of Bard and ChatGPT (encompassing both GPT-3.5 and GPT-4) regarding their machine translation proficiencies across ten varieties of Arabic. Our evaluation covers diverse Arabic varieties such as Classical Arabic (CA), Modern Standard Arabic (MSA), and several country-level dialectal variants. Our analysis indicates that LLMs may encounter challenges with dialects for which minimal public datasets exist, but on average are better translators of dialects than existing commercial systems. On CA and MSA, instruction-tuned LLMs, however, trail behind commercial systems such as Google Translate. Finally, we undertake a human-centric study to scrutinize the efficacy of the relatively recent model, Bard, in following human instructions during translation tasks. Our analysis reveals a circumscribed capability of Bard in aligning with human instructions in translation contexts. Collectively, our findings underscore that prevailing LLMs remain far from inclusive, with only limited ability to cater for the linguistic and cultural intricacies of diverse communities.

## introduction large language models (llms) finetuned to follow instructions @xcite @xcite have recently emerged as powerful systems for handling a wide range of nlp tasks. in accordance with the scaling law (i.e., pretraining larger models will continue to result in better performance) @xcite , a number of llms such as gpt-3 @xcite , chinchilla @xcite , claude (an- ## results and discussion we evaluate all models on x-english translation direction where x is an arabic variety (msa and ca). as mentioned earlier, we evaluate llms (chatgpt, gpt-4, and bard) in n-shot settings. we report bleu, comet, and chrf++ in is gpt-4 better than chatgpt? in most cases, yes. gpt-4 consistently outperforms chatgpt on many dialects and varieties. however, for jor and uae, chatgpt 0-shot performs better than 0-shot gpt-4. overall, on average, gpt-4 0shot outperforms chatgpt 0-shot by 1 ∼ 3 points on all metrics. additionally, gpt4 in 0-shot setting is on par with chatgpt in the 5-shot setting. when comparing chatgpt with gpt-4 under 5shot setting, we observe that chatgpt substantially closes the performance gap, even outperforming gpt-4 in 6 out of 10 varieties in terms of bleu score. although gpt-4 marginally outperforms chatgpt on average bleu score, this result shows that by providing few-shot examples, it is possible for chatgpt to achieve comparable performance to gpt-4 on arabic mt. is chatgpt/gpt4 better than bard? in most cases, yes. for fairness, we compare bard, chat-gpt, and gpt-4 only under the 0-shot condition. in the majority of the varieties, either chatgpt or gpt-4 outperforms the best bard draft (i.e., draft 1). our results show that bard is better than both of these models in only three cases (i.e., ca, egy and jor). overall, gpt-4 ranks best (bleu score at 23.12), followed by chatgpt (21.77 bleu points), which in turn is followed by bard (20.47 bleu points). is chatgpt/gpt4 better than commercial systems? yes, but only on dialects. we evaluate three commercial translation systems, namely, amazon, microsoft, and google translate. among commercial systems, we find google translate to outperform other commercial systems across all varieties except yem. the average score for google translate is 22.29/64.89/43.11 (bleu/comet/chrf++) compared to 18.80/63.68/41.55 and 17.77/62.85/39.76 for microsoft and amazon systems, respectively. from our evaluation results in is chatgpt/gpt-4 better than the supervised baseline? yes, it is. we evaluate nllb @xcite as the supervised baseline, finding both chatgpt and gpt-4 able to outperform this baseline in the 0-shot setting. the average bleu score for nllb is 12.97 compared to 21.77 and 23.12 of chatgpt and gpt-4 under 0-shot settings, respectively. similar to the commercial systems, the supervised baseline (nllb) does well on msa and is on par with chatgpt and gpt-4. however, both chatgpt and gpt-4 outperform it is nllb with dialects as source better than vanilla nllb? yes, it mostly is when the dialects match. our supervised baseline, nllb, takes the dialects of the source into consideration. for example, both jor and pal dialects in nllb can be defined as south levantine, i.e., (jor, pal)→south levantine. in addition, source dialects like egy and mor can be defined in their actual forms, while yem can be defined as taizzi. the column nllb (dia) in is bard a good instruction following model? not always. we evaluate bard for our translation using the web interface 7 . we find that bard can fail to follow the instructions we prompt it with. we further discuss and describe this in section 6. bard often provides the main translation output within double 7 https://bard.google.com/ quotes (""), which we extract semi-automatically. 8 additionally, bard provides three different drafts. we report results for each draft independently, as well as the average of all three drafts in our results. are instruction following models better at dialect translation? in most cases? yes. in order to clearly see performance on dialects, we exclude ca and msa results and report the average performance of the models on the various dialects as reported in do diacritics affect translation? yes, in most cases they do. although in most real-world use, native speakers do not usually employ diacritics, some arabic texts (especially those written in ca) do make use of diacritic markers. we were inquisitive about the effect of diacritics on the translation task across the different systems and so carry out a limited study of any such effect. to this end, we collect and manually translate 50 new ca sentences that are fully diacritized. the sentences conform to the identical selection criteria as those utilized within the study, specifically with regard to their length and as they originate from the first and second centuries ah books. we make a copy of this set and remove diacritics, and then independently feed both the diacritized and undiacritized versions to all the systems that we evaluate in this work. as shown in robustness. we also run a series of bootstrapping experiments that confirm the robustness of the results we acquire from the different models. we describe these experiments in appendix 3.2. 12 and 13 ). our experience working with bard reveals that the model does not always follow human instructions. for this reason, we decided to carry out a human study to assess bard's helpfulness. we define helpfulness here simply as the model's ability to follow human instructions. for each variety of arabic, we task two native speakers of arabic with familiarity with the dialects to assign one tags from the set {wrong_lang, no_translation, degeneration, content_filtering} to the model responses. we develop this tagset based on a bottom-up approach where we let the categories emerge from the data. although this tagset may not be exhaustive, we find it to reasonably capture errors we identify with model responsiveness to instructions. each of the two annotators manually label each draft, independently, with one tag from the set of our helpfulness error tags. the annotators meet and discuss differences, reaching 100% agreement which indicates that the categories are clear and independent. ## conclusion we evaluate bard, chatgpt, and gpt-4 on mt of ten diverse varieties of arabic, comparing to three commercial systems and a supervised model to juxtapose the performance of these llms under varying conditions. to assess the capacity of the llms on truly unseen data, we manually create a multi-dialectal arabic dataset for mt evaluation. we find that although llms can do well on some of the varieties we consider, they struggle especially on varieties on the more scarce public data end. as such, these llms suffer from not being quite inclusive of the different varieties of even languages they are claimed to perform well on such as arabic. a rigorous human investigation also underscores a palpable scope for enhancement in bard's adherence to instructions in the context of mt. our future work includes evaluating the performance of bard and other llms on more arabic varieties.