A rabic Topic Classification in the Generative and A uto ML Era (2023)

Abstract: Most recent models for Arabic topic classification leveraged fine-tuning existing pre-trained transformer models and targeted a limited number of categories. More recently, advances in automated ML and generative models introduced novel potentials for the task. While these approaches work for English, it is a question of whether they perform well for low-resourced languages; Arabic in particular. This paper presents (i) ArBoNeClass; a novel Arabic dataset with an extended 14-topic class set covering modern books from social sciences and humanities along with newspaper articles, and (ii) a set of topic classifiers built from it. We finetuned an open LLM model to build ArGTClass. We compared its performance against the best models built with Vertex AI (Google), AutoML(H2O), and AutoTrain(HuggingFace). ArGTClass outperformed the VertexAi and AutoML models and was reasonably similar to the AutoTrain model.

## related work the use of transformer-based models is quickly covering all nlp tasks. it started with bertarchitecture models @xcite . transformer models take advantage of their abilities to represent contextual relationships between concepts through contextual embeddings. arabic text categorization research also emerged recently (alammary, 2022). several arabic nlp tasks followed after the inception of multilingual bert. however, studies reported better results using monolingual models, specifically for low to mid-resource languages @xcite . arabert, trained on arabic wikipedia and newspaper, was applied to several downstream tasks @xcite . the model performed well on multi-class tasks (zahra @xcite . training on a mixture of dialect-arabic through social media datasets, and modern standard arabic (msa) data, has resulted in better encoder models, like qarib @xcite ). text categorization attempts on iterations of qarib have proven successful, through fine-tuning on classified msa and dialect datasets, with 6 to 12 labeling classes used @xcite . apart from bert models, considerable progress has been made using gpt and t5based models. for instance, arat5, a finetuned version of multilingual t5 on the arabic language @xcite , is achieving close to state-of-the-art performances on a variety of tasks, including categorization @xcite . gpt-3.5/4 base models augmented with arabic data, are also performing exceptionally well on text and sequence classification @xcite . however, due to openai's business model, fine-tuned versions of gpt are only available for use through the paid api. also important to mention that with the release of the bloom family, including the bloomz model, which is our choice of foundation, bigscience has also deployed multiple inference heads on top, with specific configurations, for different tasks. this includes sequence classification, question answering, and generation. ## fine-tuning data and model we selected the 7-billion parameter, publicly available, bloomz-7b-mt @xcite model as our base model. it belongs to the bloomz and mt0 family resulting from fine-tuning bloom on the multilingual xp3mt dataset @xcite . we fine-tuned it in two phases: (i) for arabic text generation, and then (ii) for arabic text classification. to realize this objective we fine-tuned bloomz-7b-mt on an additional 58, 682 tokens taken from books and newspapers datasets written in modern standard arabic. the resultant model encompasses a comprehensive spectrum of fourteen distinct subjects that are religion, finance and economics, politics, medical, culture, sports, science and technology, anthropology and sociology, art and literature, education, history, language and linguistics, law, as well as philosophy. the generated model was further fine-tuned on a bigger and labeled dataset that comprises 833,642 tokens. we call the resulting model argtc. argtc is designed to categorize input text, determining its alignment with one of the 14 predefined topics. ## results following the fine-tuning and upon testing the argtc on arbntopic labeled with the 14 categories, it scored 83% accuracy, 81% precision, and recall ( while the best h2o model scored lower than argtc on all three measurements, the model produced using autotrain scored 84% for accuracy, precision, and recall. in addition, the fourth classification model we trained using google vertex ai scored 77% precision, 76% recall, and 76% accuracy. hence, the autotrain model scored the highest on all measurements, accuracy, precision, and recall, followed closely by argtc, then the h2o model followed by the model obtained through google vertex ai training. it is worth noting that while h2o is chargefree, autotrain is free only for datasets up to 3,000 samples. for which contained 19,784 samples, extra charges amount to @xmath022. fine-tuning time for argtc was 4-5 hours, which amounts to a cost of around $9.3. consequently, the argtc model optimally balances performance against costs. ## conclusion this paper introduced argtc, an arabic text classification model with 14 categories. utilizing transfer learning techniques, the model was fine-tuned in two stages from bloomz-7b-mt, achieving 83% accuracy, 81% precision, and recall and surpassing the best models trained using vertexai and automl. it performed comparably to the best model we trained with au-totrain with a small margin.