CIDAR : Culturally Relevant Instruction Dataset For A rabic (2024)

Abstract: Instruction tuning has emerged as a prominent methodology for teaching Large Language Models (LLMs) to follow instructions. However, current instruction datasets predominantly cater to English or are derived from English-dominated LLMs, leading to inherent biases toward Western culture. This bias negatively impacts non-English languages such as Arabic and the unique culture of the Arab region. This paper addresses this limitation by introducing CIDAR, the first open Arabic instruction-tuning dataset culturally aligned by native Arabic speakers. CIDAR contains 10,000 instruction and output pairs that represent the Arab region. We discuss the cultural relevance of CIDAR via the analysis and comparison to a few models fine-tuned on other datasets. Our experiments indicate that models fine-tuned on CIDAR achieve better cultural alignment compared to those fine-tuned on 30x more data.

## introduction the need for natural language processing (nlp) applications has exploded in an era of unprecedented linguistic interaction between humans and machines. as these applications strive for greater inclusivity and effectiveness across diverse linguistic landscapes, the need for datasets that reflect the cultural differences and linguistic peculiarities of specific regions becomes increasingly important. in the context of arabic language understanding, the challenge lies not only in linguistic complexity but also in capturing the rich cultural fabric that shapes communication in the arab world. ## issues of arabic instruction datasets two main issues currently exist in the literature, as addressed in section 6, in creating arabic instruction-tuning datasets: the full translation of both instruction-response pairs using machine translation tools (mts) and the translation of instructions, then generating responses using llms like gpt-4 @xcite . next, we highlight the drawbacks of such approaches. ## cidar we introduce cidar, a dataset that has 10,000 instruction and output pairs. cidar was constructed using two sources. first, we used the alpaga-sus dataset 1 by @xcite , which is a high-quality dataset filtered from the stanford alpaca dataset @xcite . alpagasus contains more than 9k instruction, input, and output triplets. we translate 9,109 of the data to arabic using chatgpt (gpt-3.5 turbo). then, we append it with around 891 questions and answers about the arabic language and grammar crawled from ask-theteacher website 2 . figure 2 highlights the main ## dataset analysis we, in this section, compare between cidar and the initial translated alpagasus to emphasize the importance of manual revision and cultural alignment of machine-generated or translated data. ## evaluation we, in this section, shed light on the performance of llms after being fine-tuned on cidar dataset. ## related work in the literature, there are many english instruction datasets, whether generated by llms like alpaca @xcite , dolly @xcite , and self-instruct @xcite , or human-generated with templates like flan collections @xcite , p3 @xcite , and natural instructions @xcite . ## conclusion in this work, we present cidar, the first open arabic instruction-tuning dataset that is culturally aligned by native arabic reviewers to address the drawbacks of the conventional approach of finetuning llms on machine-generated or machinetranslated datasets. additionally, we introduce two datasets, cidar-eval-100 and cidar-mcq-100, for evaluating llms on cultural relevance for arabic. using such benchmarks and via thorough analyses, we demonstrate that cidar is useful for enriching research efforts in culturally aligning llms with the arabic culture. the experiments conducted validate our datasets' cultural relevance and highlight their potential to enhance the performance and understanding of llms within the rich arabic linguistic and cultural context.