Dolphin: A Challenging and Diverse Benchmark for A rabic NLG (2023)

Abstract: We present Dolphin, a novel benchmark that addresses the need for a natural language generation (NLG) evaluation framework dedicated to the wide collection of Arabic languages and varieties. The proposed benchmark encompasses a broad range of 13 different NLG tasks, including dialogue generation, question answering, machine translation, summarization, among others. Dolphin comprises a substantial corpus of 40 diverse and representative public datasets across 50 test splits, carefully curated to reflect real-world scenarios and the linguistic richness of Arabic. It sets a new standard for evaluating the performance and generalization capabilities of Arabic and multilingual models, promising to enable researchers to push the boundaries of current methodologies. We provide an extensive analysis of Dolphin, highlighting its diversity and identifying gaps in current Arabic NLG research. We also offer a public leaderboard that is both interactive and modular and evaluate several Arabic and multilingual models on our benchmark, allowing us to set strong baselines against which researchers can compare.

## introduction natural language generation (nlg) systems attempt to produce coherent, contextually appropriate, and linguistically accurate human-like language. these systems have a wide range of applications in everyday life, including in recreation, education, health, etc. the recent rise of generative models has transformed these nlg systems, making them more relevant and engaging than before. crucial to measuring the performance of nlg systems are high-quality benchmarks. in particular, they provide standardized frameworks for comparing and quantitatively assessing different algorithms, models, and techniques. for nlg, benchmarks define specific criteria and metrics for evaluating performance, allowing for objectively gauging the strengths and limitations of different approaches and encouraging healthy competition. nlg benchmarks can also facilitate reproducibility and promote transparency across different studies, acting as a catalyst for advancement in the field. despite of this significance, efforts for developing nuanced nlg benchmarks that can allow us to track and guide performance on particular languages remain limited. for arabic, a wide collection of languages and diverse varieties, there is currently no sizeable benchmark that caters to the needs of the community. in this work, we present a large benchmark for arabic, dubbed dolphin, to bridge this gap. our novel benchmark is carefully curated to represent real-world usage of arabic at scale. dolphin covers classical arabic (ca), a premodern standardized form of arabic used for old poetry and religious discourse that continues to be employed for literary expression and oration, modern standard arabic (msa), a modern descendent of ca used in formal settings and in pan-arab media, dialectal arabic (da), such as varieties used in everyday communication in the different arab countries. dolphin also encompasses text written in both arabic and latin scripts, the latter usually referred to as arabizi. the benchmark is comprised of 13 different generation tasks based on 40 different datasets across 50 test splits, making it by far the largest arabic nlg benchmark to date and among the largest for any group of languages. we build dolphin on top of exclusively public datasets, adding a number of newly developed datasets of our creation. this makes dolphin accessible and easy to use. our benchmark is accompanied by a modular leaderboard with a unified evaluation metric, i.e., a dolphin score. the leaderboard is designed to serve as a central hub for tracking and showcasing the performance of nlg systems. it functions as a dynamic and transparent platform where users can submit their models to compare their results against the state-of-the-art approaches. it also encourages a culture of transparency and detailed model description. overall, we make the following contributions: (1) we introduce a novel benchmark for arabic nlg that is large, public, diverse, and inclusive. ## related works existing nlg benchmarks can be classified into three distinct categories: arabic-specific, x-specific (where x refers to languages other than arabic, such as english, chinese, etc.), and multilingual benchmarks. in this section, we provide a brief overview of each category, highlighting their respective characteristics and scope. we offer more details on target languages, dataset sizes, and the breadth of tasks dolphin covers in appendix a. ## dolphin benchmark our objective is to provide a comprehensive and challenging benchmark for natural language generation that enables the assessment of language models and the tracking of progress in arabic. to attain this objective, we develop dolphin , considering several design principles that we will now elucidate. ## model evaluation on dolphin in order to establish a conducive environment for meaningful comparisons on dolphin, we offer a number of strong baselines for both finetuning and k-shot settings as described next. ## conclusion we presented dolphin, a large and diverse benchmark for arabic nlg composed of 40 datasets 11 bloomz is finetuned on multiple tasks in 46 languages, including ∼ 1% arabic. 12 we evaluate the version existing on march 1st, 2023. 13 we only exclude the data-to-text task. that are arranged in 13 tasks. dolphin is designed to facilitate meaningful comparisons and encourage healthy competition in arabic. we also provide an interactive leaderboard with a range of useful tools and detailed metadata to help situate future research in a rich context of information sharing. dolphin datasets are all publicly available, which should facilitate the adoption and further development of the benchmark. in the future, we intend to build on top of dolphin by extending it to more tasks and arabic varieties.