G OUD . MA : A N EWS A RTICLE D ATASET FOR S UMMARIZATION IN M OROCCAN D ARIJA
Cognitive
A BSTRACT
Moroccan Darija is a vernacular spoken by over 30 million people primarily in
Morocco.
Despite a high number of speakers, it remains a low-resource language.
In this paper we introduce G OUD .
MA : a dataset of over 158k news articles for automatic summarization in code-switched Moroccan Darija.
We analyze the dataset and find that it requires a high level of abstractive reasoning.
We fine-tune the
Arabic-language BERT (AraBERT) and the language models for the Moroccan (DarijaBERT) and Algerian (DziriBERT) national vernaculars for summarization on G OUD .
MA .
The results show that G OUD .
MA is a challenging summarization benchmark dataset.
We release our dataset publicly in an effort to encourage the diversity of evaluation tasks to improve language modeling in Moroccan Darija.
I NTRODUCTION
Moroccan Darija is the form of Arabic spoken in Morocco.
According to the 2014 census, 91% of people in Morocco – over 30 million people – speak Moroccan Darija.
However, Moroccan
Darija remains a low-resource language with very few existing text datasets (Voss et al., 2014;
Mrini & Bond, 2017; Outchakoucht & Es-Samaali, 2021), similarly to Arabic dialects and African vernaculars.
The emergence of language models, most notably BERT Devlin et al.
(2019), has pushed the state of the art in various NLP tasks, mostly in English and other high-resource languages.
There has been growing interest in offering the same level of language understanding performance and NLP services to speakers of North African dialects of Arabic.
Tunisia’s TunBERT (Messaoudi et al., 2021) is an R&D collaborative effort.
Abdaoui et al.
(2021) introduce DziriBERT for Algerian
Darja.
DarijaBERT is the language model trained for Moroccan Darija 2 .
All three language models are trained on social media data, amid a common scarcity in training datasets from other domains.
In this paper, we introduce G OUD .
MA : a dataset of 158k news articles for abstractive summarization in code-switched Moroccan Darija.
To the best of our knowledge, our dataset is the first summarization benchmark in Moroccan Darija.
We analyze our dataset and show that it requires a high level of abstractive reasoning compared to a popular summarization benchmark dataset in English.
We propose to train two summarization heuristics and train three language models on this task.
The results show that the language models have not yet achieved a high level of abstractive reasoning over the G OUD .
MA dataset, and that this dataset offers a challenging summarization problem.
We opensource this dataset in an effort to encourage the diversity of evaluation tasks for language modeling in Moroccan Darija.
R ELATED W ORK
Moroccan News Article Datasets.
Previous work has proposed news article datasets from Moroccan sources, but in Modern Standard Arabic and not in Moroccan Darija.
Jbene et al.
(2021) propose the Moroccan News Article Dataset (MNAD): a dataset of 418k news articles.
The authors collect the articles from four major Moroccan online news sources:
Akhbarona.ma, Hespress.ma, Hibapress.com, and Le360.ma.
They also collect 19 different categories for the news articles, and propose to perform text categorization on their dataset.
Boukil et al.
(2018) propose a CNN-based model for text categorization for news articles from three
Moroccan online sources: Hespress, Akhbarona, and Assabah.
The total number of articles they collect is 112k.
Likewise, these news articles are in Arabic and not in Moroccan Darija.
Moroccan Darija Datasets.
There is a scarcity of text datasets in Moroccan Darija.
These datasets can be divided into two categories: word-level bilingual resources, and free text datasets.
Many word-level bilingual resources for Moroccan Darija focus on word-level and context-free translation.
Tachicart et al.
(2014) introduce the Moroccan Dialect Electronic Dictionary (MDED): a word-level Moroccan Darija-Standard Arabic bilingual dictionary.
Mrini & Bond (2017) build the
Moroccan Darija WordNet (MDW): a Moroccan Darija extension of the Open Multilingual WordNet (Bond & Foster, 2013).
Both the MDW and MDED were used to compute similarities of single words in Moroccan Darija to their translations in Modern Standard Arabic (Tachicart et al., 2014) and other languages (Mrini & Bond, 2018).
Outchakoucht & Es-Samaali (2021) propose DODa: the
Darija Open Dataset, a collaborative dataset of words in Darija and their equivalent in English.
Free text datasets in Moroccan Darija are more frequently used for deep learning applications.
Tratz et al.
(2013) present an annotated corpus of mixed-script and code-switched Moroccan Darija tweets.
Samih & Maier (2016) study code-switching detection in Moroccan Darija using their own dataset.
Mihi et al.
(2020) introduce a dataset for sentiment analysis over tweets in Moroccan Darija.
There has been significant effort in Arabic dialect identification involving text data in Moroccan Darija (Nour-Eddine & Abdelkader, 2015; Tachicart et al., 2017; Abdelali et al., 2020; Issa et al., 2021).
Moroccan Darija has recently obtained its own Wikipedia, where the standardization of writing has been a hot topic for debate (Sedrati et al., 2020).
It is not yet feasible to train an entire language model on the Moroccan Darija Wikipedia alone, as it remains relatively small.
D ATASET C ONSTRUCTION 3. G OUD . MA
G OUD .
MA 3 is a Moroccan news website, created in 2011 by Ahmed Najim and edited by the Goud
Media company.
G OUD .
MA is a free, ad-supported online media.
“ Goud ” means “straight-forward” in Moroccan Darija.
All articles on G OUD .
MA are written in the Arabic script.
All headlines are in Moroccan Darija, whereas articles may be in Moroccan Darija, in Modern Standard Arabic, or a mix of both (codeswitched Moroccan Darija).
This generally represents the use of Moroccan Darija when written in the Arabic script, as code-switching with Modern Standard Arabic is frequent.
3. O BTAINING G OUD . MA A RTICLES
The G OUD .
MA website is built on top of Wordpress, therefore we rely on their .json Rest API to access their data, which is retrieved in .json format from each page.
A single page containsarticles and their headlines, and other information such as the link to the article on the G OUD .
MA website and the date it was published.
We loop through these pages and get the articles, headlines and article categories.
Since the articles come with HTML tags and with characters that appear in their Unicode encodings, we rely on BeautifulSoup to parse the HTML and extract only the content of the HTML paragraph tag ( <p> ).
Most of articles on G OUD .
MA start with the name of the author and sometimes the city where the event that is being described in the article has happened, thus, we remove this part.
We also removed website links, and some HTML tags that were still in the articles.
Some articles and headlines are very short, we only keep entries where the number of words in the article and the headline are higher than 30 and 3 respectively.
The only cleaning we needed to apply to the titles is parsing them using BeautifulSoup to decode the Unicode encodings back to their string format.
[Table] [Table]
D ATASET A NALYSIS AND S TATISTICS 4. O VERVIEW
After cleaning the dataset
[Table]
we end up with 158,282 articles and their headlines.
Table 1 shows simple statistical features of the dataset, some features are directly affected by the cleaning we applied, like the minimum number of words.
We split our dataset randomly into 90% training, 5% validation and 5% test.
One article can be tagged with up to four categories on G OUD .
MA , but most articles have two categories.
Most of the time, articles are tagged as (cid:16)(cid:233)(cid:74)(cid:10)(cid:130)(cid:28)(cid:10)(cid:13)(cid:75)(cid:81)(cid:203)(cid:64)
(Main page) and another category.
Tableshows the number of articles of each category and its translation to English.
The diversity of these categories shows the diversity of topics covered in the G OUD .
MA website.
4. S UMMARY A NALYSIS
Summarization datasets differ in their levels of abstractiveness and extractiveness.
Grusky et al.
(2018) define two measures to quantify the extractiveness of summaries: extractive fragment coverage and extractive fragment density.
Given an article A = ⟨ a 1 , a 2 , ..., a n ⟩ and its summary
S = ⟨ s 1 , s 2 , ..., s n ⟩ , the fragments F ( A, S ) are the set of tokens that are shared between A and
S identified greedily.
Coverage is the percentage of summary words that are part of the fragments, while density is defined as the average length of the fragment of each word in the summary.
More formally, we compute coverage and density as follows:
[Equation]
In Figure 1, we show the distribution of coverage and density of our dataset in comparison to the
CNN-Daily Mail summarization benchmark dataset.
The CNN-Daily Mail dataset was introduced as a question answering dataset (Hermann et al., 2015), but it is currently widely used as a summarization dataset (See et al., 2017), where article highlights that existed as bullet points are concatenated to form summaries.
From Figure 1 and as was noted in Grusky et al.
(2018), CNN-Daily Mail is skewed towards extractiveness, and in comparison, our G OUD .
MA dataset is more on the abstractive side, as it is characterized by low coverage and density.
E XPERIMENTS
In this section, we show how one can use the G OUD .
MA dataset for abstractive summarization.
We propose the following task definition: we train our model to summarize news articles, with the headlines as the target text.
The goal of our experiments is to establish baselines for Moroccan Darija summarization on the G OUD .
MA dataset, and thus encourage further development of summarization systems in this low-resource language.
5. S ETUP
To measure the performance of our models, we use the F1 variant of ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which is a popular metric for text summarization.
In our case, we only consider ROUGE-1, ROUGE-2 and ROUGE-L variants.
ROUGE-1 measures the overlap of unigrams between the reference summary and the candidate, while ROUGE-2 considers bigrams instead.
We also compute ROUGE-L metric which computes the longest common subsequence.
Unlike ROUGE- n , ROUGE-L takes into consideration the order of words in the sentence, and doesn’t require a predefined n -gram length.
5. M ETHODS
We propose to use two widely used summarization heuristics and three deep learning methods based on fine-tuned language models.
The two summarization heuristics are:
• Extractive Oracle Fragments: This is the result of concatenating the fragments in F ( A, S )
(see §4) in the order they appear in the summary, and represents the best possible performance of an extractive system.
Lead-1: This strategy simply considers the first sentence of an article as a candidate summary.
Since the first few sentences introduce the content of an article, this can be a competitive baseline.
[Table]
We fine-tune three language models for this abstractive summarization task:
• AraBERT 4 :
Antoun et al.
(2020) introduce a BERT model (Devlin et al., 2019) trained for the Arabic language.
AraBERT was trained on the Arabic-language Wikipedia, and on two additional Arabic-language news corpora of a cumulative size of 2.5 billion tokens.
• DarijaBERT 5 : this language model was developed and made available by AIOX Labs, an AI company based in Rabat, Morocco.
DarijaBERT is a language model for Moroccan Darija written in the Arabic script.
It is pre-trained on stories written in Darija, comments from 40 Moroccan Youtube channels, and tweets crawled using a keyword list.
No paper was published detailing the training setup.
• DziriBERT 6 : Abdaoui et al.
(2021) introduce a language model based on BERT for Algerian Darja.
We use this model as Algeria’s national vernacular is largely mutually intelligible with Moroccan Darija.
DziriBERT is trained on 1.2 million tweets emanating from major Algerian cities, but the authors consider this to be on the smaller side of language modeling training dataset sizes.
All three language models are available on HuggingFace (Wolf et al., 2020).
5. T RAINING D ETAILS
For the language models, we build encoder-decoder models that are warm-started by leveraging pretrained BERT checkpoints, this was introduced in Rothe et al.
and was shown to improve the results in multiple tasks including abstractive summarization.
While in Rothe et al.
they investigate multiple combinations, we use the BERT setup that achieved the best results in abstractive summarization, which is to warm-start both the encoder and the decoder, and tie their weights, this also helps reduce the memory footprint.
While BERT Models can handle a maximum of 512 tokens, we truncate the articles to a maximum length 256, and the headlines to 32.
We train our models for 20 epochs, with a per device batch size of 8, and 4 gradient accumulation steps, this results in a total batch size of 128.
We specify a weight decay of 0.01 and set the number of warmup steps to 1000.
Half-precision was used to speed up the training.
During text generation, we use Beam Search with 5 as the number of beams, we enable early stopping and prevent the model from repeating bigrams.
The minimum and maximum length of the sequence to be generated was set to 4 and 32 respectively to match the length distribution of the summaries in our dataset.
5. R ESULTS AND D ISCUSSION
We show the test results of our summarization experiments on the G OUD .
MA dataset in Table 3.
The results of the language models are fairly close to the Lead-1 baseline: this shows that this abstractive summarization task is difficult for the existing language models for Modern Standard
Arabic and North African Arabic dialects.
We notice that AraBERT performs the best, and that DarijaBERT and DziriBERT are close to each other.
A number of factors can explain this performance.
[Table]
First, AraBERT has the largest training dataset size of the three models.
Second, many articles contain Modern Standard Arabic.
Finally, AraBERT is the only language model in our list to be trained on news corpora, whereas DarijaBERT and DziriBERT are trained on social media text data.
The scores of the language models are well bellow the score of the Extractive Oracle Fragments.
This means that they are not yet able to perform abstractive reasoning on the G OUD .
MA dataset.
Therefore our dataset presents a reasonable challenge for abstractive summarization in Moroccan Darija.
The G OUD .
MA dataset can serve to evaluate improvements in North African language models in yet another task.
We show an example of a news article from G OUD .
MA in Table 4, along with the headline (reference summary) and the generated summaries from all three language models.
This news article is mainly written in Moroccan Darija, with a few code-switched expressions in French and Modern Standard
Arabic.
Whereas the reference summary is abstractive, the generated summaries are more extractive.
We notice AraBERT has learned words from Moroccan Darija ( (cid:16)(cid:72)(cid:65)(cid:146)(cid:9)(cid:29)(cid:241)(cid:203) ), whereas DziriBERT picks up on click-bait patterns ( (cid:201)(cid:74)(cid:10)(cid:147)(cid:65) (cid:9)(cid:174)(cid:16)(cid:74)(cid:203)(cid:64) (cid:65)(cid:235)(cid:240) – “ and here are the details ”).
DarijaBERT has no repetitions ( (cid:65)(cid:146)(cid:9)(cid:29)(cid:241)(cid:203) (cid:161)(cid:74)(cid:10)(cid:186)(cid:9)(cid:75)(cid:241)(cid:203) ) or hallucinations ( (cid:200)(cid:241)(cid:186)(cid:75)(cid:10)(cid:81)(cid:9)(cid:175)(cid:13)(cid:64) ) unlike the other two language models.
C ONCLUSIONS
In this paper we introduce G OUD .
MA : a dataset of over 158k news articles for abstractive summarization in code-switched Moroccan Darija.
To the best of our knowledge, it is the first summarization benchmark for Morocco’s national vernacular.
We show that summarization on our dataset requires a high level of abstractive reasoning compared to CNN-Daily Mail, a popular benchmark for summarization in English.
We train three language models for our proposed task.
The results show that G OUD .
MA is a challenging summarization dataset, and that it can serve as an additional axis of evaluation to further improve language modeling for Moroccan Darija