A ra BART : a Pretrained A rabic Sequence-to-Sequence Model for Abstractive Summarization (2022)

Abstract: Like most natural language understanding and generation tasks, state-of-the-art models for summarization are transformer-based sequence-to-sequence architectures that are pretrained on large corpora. While most existing models focus on English, Arabic remains understudied. In this paper we propose AraBART, the first Arabic model in which the encoder and the decoder are pretrained end-to-end, based on BART. We show that AraBART achieves the best performance on multiple abstractive summarization datasets, outperforming strong baselines including a pretrained Arabic BERT-based model, multilingual BART, Arabic T5, and a multilingual T5 model. AraBART is publicly available.

## introduction summarization is the task of transforming a text into a shorter representation of its essential meaning in natural language. extractive approaches @xcite @xcite identify informative spans in the original text and stitch them together to generate the summary. abstractive approaches on the other hand are not restricted to the input @xcite @xcite . while the vast majority of published models in both categories focus on english, some tackle other languages including chinese @xcite and french @xcite , while arabic remains understudied. in fact, most arabic summarization models are extractive @xcite . they generate explainable and factual summaries but tend to be verbose and lack fluency. addressing this problem, abstractive models are flexible in their word choices, resorting to paraphrasing and generalization to obtain more fluent and coherent summaries. sequence-tosequence (seq2seq) is the architecture of choice for abstractive models. al-maleh and desouki (2020), for instance, apply the pointer-generator network @xcite to @xcite propose a more generic rnn-based model. there are, however, two main issues with abstractive models as applied to arabic. first, they are trained and evaluated either on extractive datasets such as kalimat @xcite and @xcite , or on headline generation datasets such as ahs @xcite , which only contains short and rather extractive headlines. second, despite their state-of-the-art performance, abstractive models frequently generate content that is nonfactual or unfaithful to the original text. @xcite showed that english models that are based on the transformer architecture such as bert2bert @xcite efficiently mitigate this phenomenon thanks to pretraining on large corpora. therefore, @xcite finetuned a pretrained bert using the encoderdecoder architecture of bertsum @xcite . however, only the encoder is pretrained, the decoder and the connection weights between the encoder and the decoder are initialized randomly which is suboptimal. to address these two problems, we propose arabart, the first sequence-to-sequence arabic model in which the encoder, the decoder and their connection weights are pretrained end-to-end using bart's denoising autoencoder objective @xcite . while the encoder is bidirectional, the decoder is auto-regressive and thus more suitable for summarization than bert-based models. we finetuned and evaluated our model on two abstractive datasets. the first is arabic gigaword @xcite , a newswire headlinegeneration dataset, not previously exploited in ara-bic abstractive summarization; the second is xl-sum, a multilingual text summarization dataset for 44 languages including arabic @xcite . we evaluate our model and the other baselines using both automatic and manual evaluation. in the former we use rouge @xcite and bertscore @xcite , while in the latter we collect human annotations assessing the quality and the faithfulness of the individual summaries generated by different systems. arabart achieves state-of-the-art results outperforming pretrained bert-based models, t5-based models @xcite , as well as a much larger model, mbart25 @xcite , a multilingual denoising auto-encoder pretrained on 25 different languages using the bart objective. this improvement is observed in both automatic and manual evaluation. in section 2, we present the architecture and the pretraining settings of arabart. in section 3, we conduct an automatic evaluation of arabart against four strong baselines on a wide range of abstractive summarization datasets. in section 4, we present a detailed human evaluation using quality and faithfulness assessments. finally, we discuss related work in section 5. ## arabart arabart follows the architecture of bart base @xcite , which has 6 encoder and 6 decoder layers and 768 hidden dimensions. in total arabart has 139m parameters. we add one additional layer-normalization layer on top of the encoder and the decoder to stabilize training at fp16 precision, following @xcite . we use sentencepiece @xcite to create the vocabulary of arabart. we train the sentencepiece model on a randomly sampled subset of the pretraining corpus (without any preprocessing) with size 20gb. we fix the vocabulary size to 50k tokens and the character coverage to 99.99% to avoid a high rate of unknown tokens. ## experiments although arabart can be adapted to be finetuned on different nlp tasks, our main focus in this work is abstractive summarization. our motivation is that other tasks (e.g., text classification, named entity recognition, etc.) can be performed using other existing pretrained models with bert-like architectures. however, when it comes to generative tasks, these models underperform and cannot be easily adapted. ## human evaluation to validate the automatic evaluation results, we conducted a detailed manual evaluation that covers two aspects: quality and faithfulness. we considered 100 documents randomly sampled from the test set along with their respective candidate summaries. the systems included in the manual evaluation are: arabart, mbart25, mt base and camelbert2camelbert (c2c). 7 in addition to the generated summaries, we include the reference summaries following @xcite ; kamal @xcite . the annotations were carried out by 14 arabic native speaker volunteers. to guarantee a better quality assessments, each example was annotated by two volunteers separately. the guidelines provided to the annotators are presented in figure 2 . ## related work arabic summarization the overwhelming majority of past arabic models are extractive @xcite @xcite @xcite @xcite . recently, seq2seq abstractive models for arabic have been proposed in the literature @xcite @xcite , but none of them used pretraining. fine-tuning transformer-based language models like bert @xcite has been shown to help arabic abstractive @xcite and extractive @xcite summarization, but unlike arabart, not all components of the model are pretrained. readily-available multilingual pretrained seq2seq models have been applied to arabic summarization. @xcite uses mbart25 @xcite in cross-lingual transfer setup on an unpublished dataset, while @xcite experiment with mt5 @xcite on xl-sum. our model, tailored specifically for arabic, outperforms mbart25 and mt5 for almost all datasets despite having a smaller architecture with less parameters. arabic datasets most available datasets for arabic are extractive @xcite , use short headlines that are designed to attract the reader @xcite , or contain machine-generated (el-haj and koulali, 2013) or translated @xcite summaries. notable exceptions we choose for our experiments are gigaword @xcite and xl-sum @xcite because they cover both headline and summary generation, contain multiple sources, and manifest variable levels of abstractiveness as shown in pretrained seq2seq models bart-based models have been developed for multiple language including english @xcite and chinese @xcite in addition to multilingual models @xcite . while they can be finetuned to perform any language understanding or generation tasks, we focus on summarization in this work. ## conclusion and future work we release arabart, the first sequence-tosequence pretrained arabic model. we evaluated our model on a set of abstractive summarization tasks, with different level of abstractiveness. we compared arabart to a number of state-of-the-art models and we showed that it outperforms them almost everywhere despite the fact that it is smaller in terms of parameters. in future work, we are planning to extend the model to multitask setups to take advantage of availability of both titles and summaries in some datasets including xl-sum, and use external knowledge sources to improve faithfulness. we will also explore new directions for automatic summarization evaluation on morphologically rich languages like arabic. we would like to use arabart in other text transformation and generation tasks, such as spelling and grammar correction.