Cross-lingual Fine-tuning for Abstractive A rabic Text Summarization (2021)

Abstract: While abstractive summarization in certain languages, like English, has already reached fairly good results due to the availability of trend-setting resources, like the CNN/Daily Mail dataset, and considerable progress in generative neural models, progress in abstractive summarization for Arabic, the fifth most-spoken language globally, is still in baby shoes. While some resources for extractive summarization have been available for some time, in this paper, we present the first corpus of human-written abstractive news summaries in Arabic, hoping to lay the foundation of this line of research for this important language. The dataset consists of more than 21 thousand items. We used this dataset to train a set of neural abstractive summarization systems for Arabic by fine-tuning pre-trained language models such as multilingual BERT, AraBERT, and multilingual BART-50. As the Arabic dataset is much smaller than e.g. the CNN/Daily Mail dataset, we also applied cross-lingual knowledge transfer to significantly improve the performance of our baseline systems. The setups included two M-BERT-based summarization models originally trained for Hungarian/English and a similar system based on M-BART-50 originally trained for Russian that were further fine-tuned for Arabic. Evaluation of the models was performed in terms of ROUGE, and a manual evaluation of fluency and adequacy of the models was also performed.

## introduction when we talk about text summarization that practically means that using certain algorithms, we teach a machine to subtract information from an extensive text and provide a significantly shorter overview of it. and just like in the case of human beings, like a number of cases in our own school experiences, there are two ways of doing that. the first way is called extractive summarization @xcite @xcite . in this method the idea is to practically highlight, take out certain keywords, phrases, or sentences from the text and put them together. therefore, the result, the summary will use the exact same words, terms, and sentences as the original, and almost certainly even in the same order. it is practically the same method, when we humans glance through a massive text, like a thick script, or a book in just a few minutes. the machine would, just like our mind, focus on the first few words, paragraph, or page, then pick up the most commonly occurring words, and extract complete sentences with them, without changing anything in the sentence. the second way is abstractive summarization @xcite @xcite . once again, this is not something new for our brain. like in so many of our school studies, when a certain assignment was given to us and then later asked about in school, we probably did not use the same words and phrases of the original reading material, but we had a general idea about it, which we explained in our own plain words. and since our limited capability to remember and the inherent tendency for laziness, our summary was usually short, only an essence reflecting the original passage. quintessentially that is abstractive summarization. creative reconstruction of a textual message built on its comprehension. we chose to focus on the arabic language because it presents us challenges, which once surpassed can open up new fields of research. arabic has many features. one of the advantages of the arabic language is that besides its huge variety of dialects and spoken versions -which is a naturally occurring phenomenon -its formal written version (fusha), a practically dead language, is highly standardized and lacks major regional vari-ety. but it still has a massive amount of primary users -native arabic speakers -and text providers, also a big amount of secondary users, meaning people who are not native speakers themselves but in their engagements with arab people, or organizations create new texts. this sort of lingual stability is rare among "big languages". so that would be the good side that we have a fairly standard massive corpus with minimal presence of dialectical variables complicating the learning process. on the other hand, arabic is a language in which short vowels are not written. though possible, usually not even marked in texts, therefore it is required for the reader to have extensive knowledge of the language, otherwise, the reader would not comprehend the message of the letters, even if he/she knows them. for better comprehension let us see an example. in english, a given word, like "wish" is always written and read the same way. it might be understood as a verb, or an identically written noun, so there is a number of variables we can attach to the word but given the surrounding text that is easy to process. with some words, like "read", which can be either past or present, and the application of idioms, this number of variables grows, but not significantly. and the number of words presenting such a feature is also fairly limited, as these are rather highlightable exceptions than rules. in arabic, however, a three-letter word k-t-b, can automatically present 3 distinct forms, namely "he wrote", "it was written" and "books". with minimal alteration, the number of possible solutions goes up to 20, or so, and that is a base rule with very few exceptions. and here we are still not talking about an agglutinative language, like hungarian and turkish, where the suffixes with a big number of variables, but cutting them the core stays fairly the same. with arabic, we face a massive amount of inherent variables, all to be taken into consideration upon processing. the problem, however, presents an opportunity, as we can exploit this phenomenon to achieve bigger coherence in the text digestion. arabic also has another advantage, both scientifically and in the sense of application. it is one of the only 6 u.n. official languages, along with english, french, spanish, russian, and chinese. that means that we have a massive resource of scientific and checked texts, beyond the usual quantity, upon which can be built, and which can be a potential test ground for further development. meaning that we not only have a large quantity of informal, or semiformal text between native speakers, but we also have a huge reviewed linguistically double-checked text. given it is a u.n. language, massive amount of texts, which otherwise would have necessarily not concerned the arab world, are translated into it and linguistically checked by professionals. with the inclusion of the political and economic value of the region, and the amount of politically sensitive and important material to be assessed, the value of a text summarization tool for arabic cannot be overstated. the main contributions presented in this paper include a) presenting the first corpus for abstractive arabic text summarization, b) several neural models to perform abstractive news summarization for arabic, and c) evaluation of the performance of these models. in addition to leveraging linguistic knowledge embodied in pretrained neural language models (using multilingual bert, a transformer encoder trained on 104 wikipedia languages including arabic, and arabert, a monolingual bert model trained specifically for arabic), we also apply cross-lingual transfer to improve our results. we use summarization models based on multilingual neural language models (multilingual bert and multilingual bart-50, a pre-trained sequenceto-sequence model for 50 languages including arabic) that were originally fine-tuned to do summarization in another language, and further fine-tune them for the arabic summarization task. we thus also leverage the knowledge of the original models concerning the summarization task they learned from resources in other languages. the rest of the paper is structured as follows. section 2 presents related work published on arabic summarization. section 3 describes the methodology and the experiments that we have done. section 4 describes the results of automatic and manual evaluation. the last section concludes the paper. ## related work the work for arabic summarization is limited. most existing systems use the extractive approach. lakhas (douzidia and lapalme, 2004) is considered the first extractive arabic summarization system that was evaluated and compared with systems processing english input. the system produces a 10-word summary and translates it to english and then it is evaluated using the rouge measure @xcite . another arabic text summarization approach based on fuzzy logic was proposed by @xcite . this model is based on a new noun extraction method and fuzzy logic. yet another arabic text summarization tool, sumsat, (lakhdar and chéragui, 2019) adopts an extractive approach using a hybrid of three techniques: a) contextual exploration which allows access to the semantic content of a text, without the need for deep syntactic analysis; b) identification of indicative expressions offering the possibility of generating a summary in a general topic or a specific domain by selecting sentences that contain specific indicators, and c) the graph method which generates the summary by selecting the most representative phrases of the source text. the evaluation process differs between these systems, as well as the datasets used for evaluation. lakhas (the first extractive arabic summarization system) was evaluated cross-lingually. it generates a summary, translates it to english, and then it evaluates the english summary using the rouge-n measure. for that aim 240 documents and their corresponding summaries were produced and used as a dataset. in the case of the sumsat tool, performance was evaluated in terms of precision and recall of the discursive annotation generated by sumsat and manual reference annotations (i.e. it was not summaries per se that were evaluated). for the evaluation, they constructed a dataset composed of 25 documents and their corresponding annotation. in contrast with the above, @xcite evaluated the summaries using rouge-n (@xmath0, which contains 153 arabic articles and 765 human-generated extractive summaries of those articles created using mechanical turk. an rst-based foot_0 automatic summarization technique for arabic texts was presented by @xcite which was implemented through the arstremue system. they created a corpus of arabic texts from a newspaper website, and they claimed that the arstresume evaluation showed encouraging results based on 50 texts. some recently published work also aims to address abstractive arabic text summarization. azmi and altmami (2018) proposed a four-phase abstractive summarizer for arabic where the core need a big enough training and evaluation dataset consisting of arabic texts and their abstractive summaries. so, the first step we took was to compile a reliable arabic abstractive news summary corpus. ## results measuring the performance of a summarization system can be done through either automatic or manual evaluation. we evaluated our experiments using the rouge automatic metric and compared them to other abstractive arabic summarization results. we also evaluated our results manually since the reliability of automatic metrics is often perceived as insufficient. ## conclusion in this paper, we introduced the first corpus for abstractive arabic text summarization, which we compiled with our own effort. based on this corpus,