I GBO S UM 1500 - I NTRODUCING THE I GBO T EXT S UMMARIZATION D ATASET
A BSTRACT
Igbo, like many low-resource African languages, faces the same challenge of inadequate or complete lack of resources - dataset and methods - to support the research and the development of even basic NLP tools for its over 30 millions users.
One major gap in the IgboNLP research is the absence of a text summarization tool for Igbo.
In this paper, we report our on-going effort in the creation of the IgboSum1500 dataset, the first standard Igbo text summarization dataset, which will serve as a fundamental precursor to development of the Igbo text summarization resources as well as the expansion of the Igbo and African NLP.
B ACKGROUND
Igbo 1 , along with Hausa and Yor ` ub ´ a, is one of the three prominent indigenous Nigerian languages.
It is spoken by the Igbos of southeastern Nigeria with over 30 million speakers resident in Nigeria and many more abroad.
In NLP terms, Igbo is still considered to be acutely under-resourced and ‘scraping-by’ according to Joshi et al.
Currently, efforts are on-going in developing
IgboNLP
e.g. part-of-speech tagging (Onyenwe et al., 2019), diacritic restoration (Ezeani et al., 2016), embedding based analogy and similarity (Ezeani et al., 2018), machine translation (Ezeani et al., 2020), (Nekoto et al., 2020), and named-entity recognition (Adelani et al., 2021).
However, these efforts need to be sustained by creating more resources and expanding the scope of coverage of common downstream NLP tasks in Igbo, and one of such tasks is text summarization.
O VERVIEW OF T EXT S UMMARIZATION
The growth the internet and web with the accompanying information overload has, among other things, necessitated the quest for tools that can summarise large volumes of texts (Gambhir & Gupta, 2017).
The task of text summarization, therefore involves compressing a large body of text into its shorter version which contains only the relevant information in the text (Allahyari et al., 2017).
In the text summarization study, a test dataset is needed to evaluate the performance of any proposed method.
Some of the publicly available datasets for the English language summarization tasks include the CNN-Daily Mail dataset (See et al., 2017).
Automatic text summarizers are computer programs that can identify relevant parts of a text document and put them together in coherent and readable way.
Common techniques to building automatic text summarizers can be categorized into two key approaches:
• extractive summarization (Nallapati et al., 2017; Liu, 2019; Mihalcea, 2005) where the summary contains the exact words and phrases in the original text, and
[Table]
part of the Igbo version of the UN’s Universal Declaration of Human Right showing an example of a reference summary. • abstractive summarization (Gupta & Gupta, 2019; Paulus et al., 2017; Gehrmann et al., 2018) where new words could used, as done by humans, to create the summary Works on combining the two approached in some hybrid form have also been reported (Chen et al., 2019;
Jin et al., 2020; Hsu et al., 2018).
The task of text summarization, which is a form of language understanding and generation, is a complex one.
This is because it is hard for the machine to extract the actual meaning of words or phrases in context, inferential interpretation, and generate correct and relevant sentences for the summaries.
M ETHODOLOGY FOR C REATING ‘I GBO S UM 1500’
Given the nature of the source of our data and the time and resources available to the authors, we consider this paper an extended proposal.
It details our approach to the core work which is still in progress at the time of submitting this, as well as early evaluations results of the Igbo text summarization systems built.
As shown in Figure 1 we adopted a simple 6-step process for bootstrapping the process of creating the IgboSum1500 dataset as well as baseline Igbo text summarizers which will be discussed in the sections below.
3. D ATASET C OLLECTION AND A NALYSIS
The main source of our dataset is the website of the Anambra Broadcasting Service 2 - a radio and television station based in one of the major southeastern states Anambra.
The choice of this station is mainly because it is the most accessible to the main author who has also secured the permission to use their content.
Although this is a good website for relevant and contemporary local contents across multiple genres, these contents are unfortunately in English.
This is challenging given that we aim at building the
Igbo text summarization dataset.
However, it also provides the opportunity to leverage existing and relatively more developed NLP tools (summarization and translation) for English language in our pipeline for bootstrapping the dataset creation process.
For the purpose of this work we randomly extracted 1500 articles uploaded on the website between the month of May 2021 and February 2022.
Figure 2a shows that majority of the articles we collected - over 65% - were published November 2021 and January 2022.
We did not investigate whether this is by pure sampling chance or due the electioneering activities happening in the region about the same time.
In total, there are 51 unique authors for all the 1500 articles.
However, some of the authors were more prolific than the others leading to only 15 writing over 50% (850) articles.
Figure 2b shows an anonymized 3 plot of the counts of articles by each of the top authors.
Looking at categories of the articles as documented on the website, we observed that there were ten unique categories: Entertainment, Business, News, Sports, Health, Politics, Columnist, Nigeria and
State .
Each article belongs to at least one of these categories but some articles belong to more than one category.
Figure 2c but shows that State clearly dominates all the other categories by a large margin.
Another statistic we looked at (Figure 2d) is the number of sentences per article.
While there are a lot of articles that are quite long (up to 80 sentences), majority of the articles have between 5 andsentences.
3. G ENERATING R EFERENCE S UMMARIES
The evaluation of text summarization tools is often done by comparing their summaries with goldstandard or ‘reference’ summaries which are often provided by human summarizers.
In order to efficientize this process and given that our original data was in English, we defined a three-stage process ( summarize-evaluate-correct ) that leverages a summarization tool for English as well as Igbo language speakers to create our reference summaries.
The summarize stage employs a version of the state-of-the-art BART based (Lewis et al., 2019) summarisation model trained on the cnn-dailymail dataset (Nallapati et al., 2016) which is available on HuggingFace 4 .
This model was applied to the English version of the articles and then passed on to three language speakers were asked to evaluate the quality of the summaries based on some defined criteria:
• 5: Very clear expression and very readable style.
Very few language errors.
Relevant knowledge and a good understanding of the article; without significant gaps.
• 4: Clear expression and legible style.
Small number of language errors.
Relevant knowledge and a good understanding of the article, with some gaps.
• 3: Generally clear expression, and legible style.
Number of language errors.
The knowledge and understanding of the article is sufficient, although there are several omissions and several errors.
• 2: Expression is generally clear but sometimes unclear.
Significant number of language errors.
The knowledge and understanding of the article is sufficient for an elementary summary, but there are a number of omissions and errors.
• 1: Expression is often difficult to understand.
Defective style.
Persistently serious language errors.
The information is inadequate for summary purposes.
Obvious deficiencies in understanding the article.
The final stage - the evaluation - was actually going on simultaneously with the correction stage i.e. the evaluators were instructed to fix errors as they encountered them thereby creating high-quality summaries after the process.
3. T RANSLATING A RTICLES AND S UMMARIES
As stated in Section 3.1, our original dataset was in English and so were the generated summaries.
However our aim in this work was to build the Igbo text summarization dataset.
So having created and corrected the English versions summaries as described in Section 3.2, we proceeded with translating both the articles and their summaries using a standard English-Igbo summarization tool, the
GoogleTranslate API 5 .
Although this approach was useful for facilitating the process, the quality of the translation was not very good.
To improve that, we adopted a similar human-in-the-loop approach to the one in Section 3.2 where language speakers were asked to correct the translated articles and summaries.
3. B UILDING S UMMARIZATION S YSTEMS
To obtain some initial results, we then built some basic extractive summarisation systems - which will serve as baseline models - and investigated their performances.
We built and compared two common extractive summarisation systems - TextRank (Mihalcea & Tarau, 2004) and LexRank (Erkan & Radev, 2004).
Both systems use versions of existing ranking algorithms such as PageRank to determine the importance of parts of a text e.g. sentences or phrases.
We used a naive baseline that uses the title of each article as a quasi summary.
Some previous works, especially in topic modelling, have noted that similar to the first sentence of a document or the key words, the title of a document does contain some meaningful information about the document (Radev et al., 2004).
3. E VALUATION AND D ISCUSSION
The system summaries were evaluated by comparing them with reference summarisers using the four commonly used versions of the ROUGE 6 metrics as implemented by Ganesan (2018).
ROUGE-N (where N= 1 or 2) i.e. unigrams and bigrams; ROUGE-L the longest common subsequence in both system and reference summaries that retains the word order; and ROUGE-SU : a version of ROUGES 7 that includes unigrams.
ROUGE typically present three key metric scores precision, recall and
F1-score as described below.
[Equation]
where the value of β is used to control the relative importance of precision and recall .
Larger β values give more weight to recall while β values less than 1 give preference to precision .
In the this work, β is set to 1 making it equivalent to the harmonic mean between precision and recall .
The term ‘ units ’ as used in the equation refers to either words or n-grams.
Typically, summarization systems aim at improving the recall score i.e. the fraction of the reference summary it is able to get.
Figure 3 shows that the baseline system performed poorly in that regard such that, depsite its high score, the f1 score remained low.
This is not surprising given the those summaries were significantly smaller in size that the reference summaries they are comparing against.
The other baseline models did much better with significantly higher recall scores mainly because they produced summaries that were of comparable lengths with the reference summaries.
Also the higher n-gram scores were generally poor and this is possibly because of their extractive approach to summarisation which is the different from the abstractive and human approach used in creating the reference summaries thereby reducing the chance of longer ngram overlap.
TextRank appears to have done slightly better overall given that the f1 scores are higher
C ONCLUSION AND F UTURE WORK
In this work, we present the first standard, high quality and publicly available Igbo summarisation dataset - IgboSum1500.
This is a major contribution to Igbo and AfricanNLP in particular and lowresource NLP in general especially in the natural language understanding and text generation space.
We are quite aware of the possible limitations of this work.
Using existing tools in other languages may be helpful but may sometimes propagate the errors and biases along the pipeline.
Also, the use of basic extractive does not give room for exploration of the challenges text summarisation.
Work is currently ongoing on packaging and releasing the dataset.
Future work will focus on experimenting with creating or finetuning state-of-the-art neural models for the Igbo summarisation task focusing on abstractive approach.