Octopus: A Multitask Model and Toolkit for A rabic Natural Language Generation (2023)

Abstract: Understanding Arabic text and generating human-like responses is a challenging task. While many researchers have proposed models and solutions for individual problems, there is an acute shortage of a comprehensive Arabic natural language generation toolkit that is capable of handling a wide range of tasks. In this work, we present a robust Arabic text-to-text Transformer model, namely AraT5v2, methodically trained on extensive and diverse data, utilizing an extended sequence length of 2,048 tokens. We explore various pretraining strategies including unsupervised, supervised, and joint pertaining, under both single and multitask settings. Our models outperform competitive baselines with large margins. We take our work one step further by developing and publicly releasing OCTOPUS, a Python-based package and command-line toolkit tailored for eight Arabic generation tasks all exploiting a single model. We provide a link to the models and the toolkit through our public repository.

## introduction natural language generation (nlg) is a fundamental component of natural language processing that aims to generate human-like, coherent, contextually fitting, and linguistically precise text from structured data or various other input formats. nlg systems find applications in various aspects of daily life, including education, healthcare, business, and more. the recent emergence of generative models has significantly impacted the field of nlg. while important progress has been made in nlg research, the majority of existing tools, systems, and models are primarily focused on english @xcite @xcite , leaving behind many languages, including arabic. although it is one of the most widely spoken languages in the world, and one with a rich linguistic structure and diverse dialects, arabic remains underrepresented in nlg. one reason is the complex morphology and syntax of arabic. hence, the primary focus of our research here is to develop an advanced tool capable of performing several key arabic nlg tasks. for example, we target tasks such as text summarization, question answering, question generation, news headline generation, and paraphrasing. these are tasks that necessitate a deep understanding of semantics, syntax, and pragmatics of arabic. we also focus on tasks that require an understanding of both the syntax and morphology such as diacritization, transliteration, and grammatical error correction. our main contributions are as follows: in the following section, we offer a concise overview of publicly available arabic nlu and nlg tools, along with the arabic and multilingual sequence-to-sequence (s2s) language models that we employ in this work. other tools, such as mazajek (farha and magdy, 2019), and aranet (abdul-mageed et al., 2019), are dedicated to social meaning tasks such as sentiment analysis, emotion detection, age and gender prediction, and sarcasm detection. nlg tools. regarding arabic nlg, as far as we know, the only publicly available tools are primarily focused on many-to-arabic machine translation (mt). these include open-mt (tiedemann and thottingal, 2020), nllb (costa-jussà et al., 2022), and turjuman (nagoudi et al., 2022d). ## arat5 v2 in this section, we present a novel version of arat5, the arabic-specific sequence-to-sequence model. we refer to this novel version as arat5 v2 . this new version represents a substantial evolution of the original arat5 v1 model, foot_1 marked by notable improvements. these include (1) training on an expanded dataset comprising both labeled and unlabeled data, (2) larger sequence length of 2, 048 tokens, and (3) diverse training strategies that integrate a combination of unsupervised and supervised training techniques. pretraining data. as we mentioned previously, our pretraining (unlabeled and labeled) dataset is linguistically diverse, covering all categories of arabic (i.e., ca, da, and msa). as we will now describe. ## training strategies in this section, we describe the different strategies we use to pretrain and finetune arat5 v2 . ## conclusion we introduced a suite of powerful arabic text-totext transformer models trained on large and diverse datasets, with an extended sequence length of up to 2, 048. we also explored various pretraining strategies, including unsupervised and joint pertaining, using both single and multitask settings. our models outperform competitive baselines, demonstrating their effectiveness. furthermore, we introduced octopus, a publicly available pythonbased package and command-line toolkit tailored for eight arabic natural language generation tasks. octopus is designed to be extensible, and we plan to expand its capabilities by adding more tasks and increasing the capacity of our back-end model.