Generative Approach for Gender-Rewriting Task with A rabic T 5 (2022)

Abstract: Addressing the correct gender in generative tasks (e.g., Machine Translation) has been an overlooked issue in the Arabic NLP. However, the recent introduction of the Arabic Parallel Gender Corpus (APGC) dataset has established new baselines for the Arabic Gender Rewriting task. To address the Gender Rewriting task, we first pre-train our new Seq2Seq ArabicT5 model on a 17GB of Arabic Corpora. Then, we continue pre-training our ArabicT5 model on the APGC dataset using a newly proposed method. Our evaluation shows that our ArabicT5 model, when trained on the APGC dataset, achieved competitive results against existing state-of-the-art methods. In addition, our ArabicT5 model shows better results on the APGC dataset compared to other Arabic and multilingual T5 models.

## introduction in many generative downstream tasks in arabic nlp, such as machine translation and chatbot applications, addressing the correct gender is crucial to increase the quality of the generated text to reach human-level performance. this also leads to having a generated text that is less biased and discriminating against specific gender. moreover, when used in translation and chatbot applications, generative models such as t5 @xcite , and bart @xcite ) may adopt a gender bias, which they learn from the pre-training corpora. thus, the gender rewriting downstream task has recently received more attention in arabic nlp. this attention can be seen with the introduction of the arabic parallel gender corpus (apgc) dataset @xcite . current state-of-the-art methods to address the gender rewriting task uses a multi-stage model consisting of rule-based, morphological analyzer, and encoder-decoder gru model @xcite . however, one issue with using a multistage model is that it increases the complexity of the model. this motivates us to seek a more simple alternative approach. in this work, we hypothesize that generative models such as t5 and bart could address the gender rewriting problem when trained on the apgc dataset. thus, in this work, we introduce a novel method to address the gender-rewriting task through our arabict5 model, a model that we pre-trained on a collection of arabic corpora. thus, our contributions in this work can be summarized in the following points: ## background in this section, we will first explain the apgc dataset. then, we will have an overview of the current state-of-the-art model; the multi-step gender rewriting model @xcite . then we will explain the t5, mt5 @xcite , and arat5 models @xcite . ## method in this section, we will first explain how we build our new t5 model. next, we will explain our method to address the gender rewriting task and the details of our environmental and evaluation setup. ## results and discussion in the multi-step model by @xcite . we explain the multi-step model in detail in section 2.2. this evaluation in however, observe the close gap in blue score between all models in on the other hand, the evaluation comparison between t5-based models, including mt5, arat5, and our arabict5, shows how pre-training corpora significantly affect the performance in the gender-rewriting task. our arabict5, pre-trained on modern classical arabic corpora (arabic encyclopedias and arabic news articles), shows superiority against other models that use arabic website collection and twitter datasets. we use our best-performing model arabict5 xlarge to submit our prediction for the blind test dataset of gender rewriting task @xcite at the seventh arabic natural language processing workshop (wanlp 2022). ## conclusion in this paper, we introduced a new arabic t5 model pre-trained on 17gb of arabic corpora. also, we illustrate how our arabict5 model shows a competitive evaluation performance against the current state-of-the-art model and other seq2seq t5 models. for future work, we plan to add further stages to our arabict5 model to improve the evaluation performance on the gender-rewriting task.