Gazelle: An Instruction Dataset for A rabic Writing Assistance (2024)

Abstract: Writing has long been considered a hallmark of human intelligence and remains a pinnacle task for artificial intelligence (AI) due to the intricate cognitive processes involved. Recently, rapid advancements in generative AI, particularly through the development of Large Language Models (LLMs), have significantly transformed the landscape of writing assistance. However, underrepresented languages like Arabic encounter significant challenges in the development of advanced AI writing tools, largely due to the limited availability of data. This scarcity constrains the training of effective models, impeding the creation of sophisticated writing assistance technologies. To address these issues, we present *Gazelle*, a comprehensive dataset for Arabic writing assistance. In addition, we offer an evaluation framework designed to enhance Arabic writing assistance tools. Our human evaluation of leading LLMs, including GPT-**4**, GPT-**4o**, Cohere Command R+, and Gemini **1.5** Pro, highlights their respective strengths and limitations in addressing the challenges of Arabic writing. Our findings underscore the need for continuous model training and dataset enrichment to manage the complexities of Arabic language processing, paving the way for more effective AI-powered Arabic writing tools

## introduction writing is an essential skill for both individuals and professionals, requiring a long-term commitment to mastery that can span years of continuous learning and refinement @xcite . the distinct cognitive processes involved in writing underscore the significant importance and complexity of writing as a task. consequently, in the fields of ai and natural language generation (nlg), writing is considered a hallmark of human intelligence, and researchers have long viewed it as a pinnacle task for ⇤ these authors contributed equally. their studies @xcite @xcite . in recent years, the landscape of writing assistance has been transformed by rapid advances in generative ai, particularly through the development of llms @xcite . these neural models, trained on the intricacies of language by processing vast amounts of text, have led to the creation of ai-powered tools designed to provide comprehensive writing assistance across a wide range of tasks @xcite . models like gpt-3 and gpt-4 @xcite have demonstrated impressive capabilities in both understanding and generating text @xcite . these llms have significantly expanded the potential for "human-ai co-writing," where ai systems provide flexible support that extends far beyond simple grammar and spell-checking to polishing texts and generating completely new content, significantly enhanc-ing writing assistance for diverse writing tasks @xcite . however, underrepresented languages like arabic often face substantial challenges in developing advanced ai tools, including writing assistance, largely due to the scarcity of relevant training data. in this work, we seek to bridge this gap by introducing gazelle, a dataset specifically curated for arabic writing, designed to advance the development of ai-powered writing tools for underrepresented languages. gazelle is a manually curated instruction-style dataset crafted in both english and arabic and organized around two primary themes. the first theme, text rewriting, aims at supporting learners in rewriting their texts across different linguistic levels. this encompasses (i) grammatical error correction (gec) at various levels such as orthographic, morphological, syntactic, and semantic with catgeories beyond those proposed in the arabic learner corpus (alc) @xcite adopted in previous works. text rewriting also involves corrections of deeper semantic levels beyond traditional gec that cover (ii) metaphors and multi-word expressions (mwes) and (iii) text refinement, which focuses on improving texts by replacing dialectal sequences with their equivalent standard forms and fixing poor language. the second theme, writing advice, covers (iv) rule explanation and definitions as well as (v) i'rab. in arabic, i'rab or declension refers to the variation in a word's form that indicates its grammatical case and role within a sentence. in gazelle, the i'rab category guides learners through the different grammatical roles within a text, such as identifying when a word takes on a nominal case due to its role as a subject in a sentence. across its various categories, gazelle also provides explanations for the suggested corrections. figure 1 outlines the main tasks covered in gazelle, and figure 2 illustrates each of these tasks. our contributions can be summarized as follows: (1) we introduce a comprehensive, manually curated dataset specifically for arabic, covering two different writing themes across five distinct tasks. (2) we expand the arabic learner corpus (alc) @xcite error taxonomy by developing a fine-grained classification, including adding sub-classes and detailed sub-subclasses under the main error categories. furthermore, we generate synthetic data from the arabic tree bank (atb) @xcite ) that aligns with our new classification and sub-classes for specific error types in the alc error taxonomy. (3) we conduct a comparative evaluation of current leading llms -gpt-4 @xcite , gpt-4o @xcite , cohere command r+, foot_0 and gemini 1.5 pro @xcite )-focusing on their capabilities in different arabic writing tasks. the rest of this paper is organized as follows: in section 2, we provide an overview of related work. section 3 introduces our manually curated datasets for arabic writing assistance. in section 4, we provide a comparison of gazelle with other gec datasets. section 5 presents our evaluation of the frontier llms on arabic writing tasks. we also detail our human evaluation methodology and results. we conclude in section 6. ## related work llms for writing assistance. the advent of llms and their enhancement through data augmentation and instruction tuning have significantly advanced automated writing assistance tools @xcite @xcite . models trained on extensive datasets of human-written instructions, using methods like instruction fine-tuning, have proven to enhance their ability to generalize across various tasks by leveraging large, diverse sets of instructional data @xcite . models like flan-t5 @xcite and gpt-4 @xcite @xcite demonstrate the progress and capabilities of incorporating a wide array of tasks during fine-tuning to achieve robust performance and adaptability in various writing applications @xcite . we provide detailed examples of recent advancements in instruction-tuned models and their application on writing assistance in appendix a. progress in arabic writing tasks. arabic writing tasks are complex due to its linguistic and morphological diversity, compounded by orthographic ambiguities from optional diacritics and dialectal variations @xcite . despite these challenges, significant progress has been made in terms of model development and dataset developments across different writing tasks such as gec @xcite @xcite , table d.1 2023 ), text simplification @xcite ) and text summarization (al-maleh and desouki, 2020; @xcite . further details about the progress in arabic writing tasks are provided in appendix a. ## gazelle as mentioned in section 1, gazelle comprises two main themes, text rewriting and writing advice, across five distinct types of writing assistance instructions that we manually curate. gazelle encompasses parallel data with instructions crafted in both arabic and english, covering both input and output, allowing users to fine-tune models for these various tasks bilingually. except for the i'rab and arabic grammatical rules and definitions datasets, we manually translate the arabic explanations and instructions into english. to ensure thorough coverage, we curate instructions for each of these tasks separately from several publicly available online sources. 2 our proposed taxonomy introduces two more sub-classes to the original alc error categories: closed class error under syntax and special expression error under semantic, bringing the total to 31 error tags. closed class errors address issues with closed class items, such as the five nouns, the five verbs, and pronouns. special expression errors covers the use of collocations, multi-word expressions, and other commonly misused terms additionally, each of our new error categories include several sub-subclasses. for example, the sub-classs number under syntax which pertains to the correct use of singular, dual, and plural forms in nouns, verbs, and pronouns is further divided into several sub-subclasses. our newly-developed categorization scheme incorporates extensive knowledge of arabic grammar, aiming to provide users with not only the category of grammatical error but also its sub-and sub-subclasses. table d presents a detailed overview of all the newly introduced categories. it comprises 7 classes, each with an average of 9 sub-classes. 4 gec instructions. we develop a set of 3,382 bilingual instructions for gec, based on our newly extended error taxonomy. this comprehensive taxonomy includes a wide range of error sub-classes and sub-subclasses, aiding identification of specific errors. we manually curate these sentences from various arabic learning online sources to reflect the detailed error categories in our extended taxonomy. each sentence pair includes grammatical errors along with their corrected versions and explanations, as shown in table d.1. examples of these detailed error sub-classes and their coverage are provided in appendix c. ## gazelle in comparison the qalb dataset comprises manually corrected arabic texts, including msa commentaries from aljazeera by native speakers (l1) and texts by non-native learners (l2) @xcite . the zaebuc corpus, on the other hand, contains essays written by first-year students at zayed university in the uae @xcite . while both the qalb and zaebuc corpora include multiple errors per sentence or paragraph, facilitating automatic evaluation, our datasets are curated with a different focus. we manually curate the gazelle dataset to target specific error types within the alc error taxonomy by extending the existing error categories and incorporating our own extended categories. each sentence, sourced from various arabic learning online sources, represents a specific subclass or sub-subclass of our extended error taxonomy and contains only one type of error along with explanation of this type of error. this targeted approach makes our dataset particularly useful for learners aiming to understand and correct specific types of errors. we provide a comparison of gazelle dataset to the qalb and zaebuc corpora in appendix f.1 ## evaluating leading llms on gazelle we evaluate gazelle on current leading llms, namely gpt-4, gpt-4o, gemini 1.5 pro, and cohere command r+, to assess their capabilities across our proposed tasks in arabic. we test the models under a zero-shot setting @xcite , allowing us to assess the models' inherent capabilities in handling our proposed writing tasks. for each task, we initially tested the prompts and iteratively refined a zero-shot prompt tailored to each subtasks. further details on our prompt design process, along with examples used for evaluation, are provided in appendix f. ## conclusion in this work, we present gazelle a comprehensive dataset and evaluation framework to enhance arabic writing assistance tools. our manually finegrained dataset incorporates five writing subtasks under two main themes. our analysis uncovers both strengths and limitations of the most leading llms in tackling the unique challenges of arabic writing. these insights highlight the need for continuous model training and dataset enrichment to address the complexities of arabic language processing. our study showcases the varied capabilities of models in handling arabic grammar and stylistic nuances, providing a foundational step towards more advanced 'human-ai co-writing' tools for arabic writing. future research may include on expanding the dataset and integrating diverse linguistic features to develop robust and reliable models for mastering arabic writing. overall, our work serves as a stepping stone towards creating more effective arabic writing assistance tools and advancing the field significantly.