A ce GPT , Localizing Large Language Models in A rabic (2024)

Abstract: This paper is devoted to the development of a localized Large Language Model (LLM) specifically for Arabic, a language imbued with unique cultural characteristics inadequately addressed by current mainstream models. Significant concerns emerge when addressing cultural sensitivity and local values. To address this, the paper proposes a comprehensive solution that includes further pre-training with Arabic texts, Supervised Fine-Tuning (SFT) utilizing native Arabic instructions, and GPT-4 responses in Arabic, alongside Reinforcement Learning with AI Feedback (RLAIF) employing a reward model attuned to local culture and values. The goal is to cultivate culturally cognizant and value-aligned Arabic LLMs capable of accommodating the diverse, application-specific needs of Arabic-speaking communities. Comprehensive evaluations reveal that the resulting model, dubbed ‘AceGPT’, sets the state-of-the-art standard for open Arabic LLMs across various benchmarks. Codes, data, and models are in https://github.com/FreedomIntelligence/AceGPT.

## introduction llms like gpt-3.5 turbo and gpt-4 have been shaping the current landscape of natural language understanding and generation @xcite . in contrast to the proprietary nature of gpt-3.5 turbo and gpt-4, there has been a trend towards developing open-source large language models capable of instruction-following @xcite and fluent conversations @xcite , a phenomenon termed as 'democratization of chat-gpt' @xcite . while these models have shown great promise in understanding and producing content in various * benyou wang is the corresponding author. † indicates that the three authors contributed to this work equally. languages, they might fail to align with local values and cultural norms in non-english environments @xcite ; we call it the 'localization issue'. this issue can lead to significant problems in practical usage scenarios, especially for regions such as the arabic world where the culture and values diverge significantly from western norms. we argue that it is not just desirable but necessary to localize large language models and tailor them to a specific cultural environment. ## conclusion acegpt addresses the 'localization issue' in large language models by specifically catering to the distinct linguistic and cultural contexts of arabic environments, leveraging further pre-training, instruction tuning, and rlaif. it excels in multiple benchmarks including instruction-following and natural language understanding, setting a new standard among arabic llms. we contribute high-quality datasets and evaluation resources, highlighting the need for localizing large language models and introducing acegpt as a pioneering solution for arabic linguistic and cultural adaptation.