A C OMPARISON OF T OPIC M ODELING AND C LASSIFICATION M ACHINE L EARNING A LGORITHMS ON L UGANDA D ATA
A BSTRACT
Extracting functional themes and topics from a large text corpus manually is usually infeasible.
There is a need to build text mining techniques such as topic modeling, which provide a mechanism to infer topics from a corpus of text automatically.
This paper discusses topic modeling and topic classiﬁcation models on Luganda text data.
For topic modeling, we considered a Non-negative matrix factorization (NMF) which is an unsupervised machine learning algorithm that extracts hidden patterns from unlabeled text data to create latent topics, and for topic classiﬁcation, we considered classic approaches, neural networks and pretrained algorithms.
The Bidirectional Encoder Representations from Transformers(BERT), a pretrained model that uses an attention mechanism that learns contextual relations between words (or sub-words) in a text, and a Support Vector Machine (SVM) algorithm, a classic model which analyzes particular properties of learning within text data, record the best results for topic classiﬁcation.
Our results indicate that topic modeling and topic classiﬁcation algorithms produce relatively similar results when topic classiﬁcation algorithms are trained on a balanced dataset.
I NTRODUCTION
In this information age, vast amounts of text data are continuously generated from social media platforms and the world wide web.
The extraction of meaningful information from text data requires that data be categorized into various themes and topics.
The topics are essential to understand subject domains, generate secondary data for casual discoveries, and in the end, design tools that facilitate decision-making processes.
Social media platforms like Twitter, a microblogging platform, provide a way of topic mining using its ‘Hashtag’ and ‘mention’ feature.
Although these features also facilitate the discovery of most trending topics on the platform in real-time, this is not sufﬁcient as most trending topics are represented using either a name of an individual, hashtags, or words in other languages.
In most cases, it is not easy to understand what the trending topics are about (Lee et al., 2011).
Modeling or classifying these topics into general categories with high accuracy and precision remains an important research area in Natural Language Processing (NLP) as it facilitates better information retrieval.
Topic modeling in NLP is an unsupervised machine learning technique that involves detecting word and phrase patterns within a set of sentences or documents and automatically clustering these word groups and similar expressions into topics (Zheng et al., 2021).
On the other hand, text classiﬁcation is a supervised machine learning technique that involves the extraction of features from the data and the use of the extracted features to assign a set of pre-deﬁned labels to open-ended text(Hamed et al., 2020).
Previous research in Hamed et al.
; Lee et al.
(2011) has shown that various efforts have been made in building topic classiﬁcation and topic modeling models for the English language.
However, there has been less focus on low-resourced languages like Luganda, a Bantu language spoken in the African Great Lakes region by more than ﬁfteen million people (UBOS, 2016).
This research compares topic modeling and topic classiﬁcation machine learning algorithms on our unique Luganda dataset.
Our contribution is threefold: (a) We develop FastText, Glove, Paragram, and word2vec
Luganda word embeddings of 50 dimensions, which other NLP researchers can use to train models to understand the characteristics of the Luganda Language.
(b)We develop a Luganda topic model using the NMF algorithm to uncover latent topics under study within the Luganda dataset.
Topics like Education, Land, COVID-19 and Security emerge with the highest weights and thus can be easily discovered by our Luganda topic model.
(c) We develop Luganda topic classiﬁcation models using classic approaches like neural and pre-trained algorithms.
BERT, a pretrained algorithm, performs better than the other classiﬁcation algorithms with a precision of 0.
and a recall of 0.996 on the Luganda text data.
The rest of the paper is organised as follows: Section 2 discusses related work; Section 3 discusses the methodology which contains a description of the data, steps taken to preprocess the data, topic modeling and classiﬁcation approaches used in this study and the evaluation metrics used to assess the performance of the models.
Section 4 discusses the results from both the topic modelling and topic classiﬁcation algorithms.
Section 5 concludes the paper.
R ELATED W ORK
Machine Learning approaches have been used in various Natural Language Processing (NLP) tasks including sentiment classiﬁcation Yoon (2014),statistical machine translation Jacob et al.
(2014), text classiﬁcation Rubungo Andre et al.
and
topic modeling(Hamed et al., 2020).
Hamed et al.
(2020) present an automated extraction of COVID-19 related discussions from social media which leverages topic modeling approaches to uncover various issues related to COVID-19 from public opinions.
Lee et al.
(2011) present a two scheme based approach: text-based and networkbased classiﬁcation for classifying trending topics in twitter data.
The experimental results suggest that Naive Bayes Multinomial classiﬁer using text from trend deﬁnition, 100 tweets, and a maximum of 1000 word tokens per category gives the best accuracy of 65.36%.
They also discover that some topics could fall under more than one category for example
actor’s biography would fall under TV, movies and books categories.
Recently, there has been research carried out in NLP for African languages.
For example the creating of open source data sets like the: Agriculture keyword dataset for building speech keyword spotter models for Luganda (Mukiibi et al., 2020).
The authors in Vukosi & Tshephisho (2020) created a news classiﬁcation dataset that was limited to headlines.
The researchers created a benchmark for research in Setswana and Sepidi, two Bantu South African languages.
The authors in
Rubungo Andre et al.
(2020) created the KINNEWS and KIRNEWS dataset in Kinyarwanda and
Kirundi respectively.
The dataset was created for multi-class classiﬁcation on the news articles.
These datasets have been used to be used to create benchmark experiments for topic classiﬁcation (Vukosi & Tshephisho, 2020; Rubungo Andre et al., 2020).
2. T OPIC MODELING AND T OPIC CLASSIFICATION ON LOW - RESOURCED LANGUAGES
Vukosi & Tshephisho (2020) discuss work where four machine learning models (Logistic Regression, Support Vector Classiﬁcation, XGBoost, and MLP Neural network) were used to classify news in Setswana and Sepidi.
The XGBoost model provided the best results for Sepedi news headline classiﬁcation.
Rubungo Andre et al.
(2020) present benchmark experiments on Kinyarwanda and Kirundi news articles using different classic and neural approaches.
The classic approaches experimented on the Multinomial Naive Bayes, Logistic Regression, and Support Vector Machine with SGD.
The neural model approaches that were used in the study included the Character-level Convolutional Neural Networks, Convolutional Neural Network, and Bidirectional Gated Recur- rent Unit(Bi-GRU). Rubungo Andre et al.
(2020), shows that Support Vector Machine(SVM) and Bidirectional Gated Recurrent Unit(Bi-GRU) yielded the best results in the classical and neural approaches respectively.
Zahara (2020) focuses on constructing a viable focused-analysis topic model which learns topics from Twitter data written in a closely related group of non-standardized varieties in Arabic widely spoken in the Levant region.
Results of the baseline model and a targeted topic model are compared and designed to serve the purpose of focused analysis.
Nasim (2021) combines deep-learningbased language modeling with classic topic modeling techniques to produce interpretable topics for a given set of documents in Urdu, a low resource language.
They propose building an accurate Part of Speech tagger for the Urdu language, using semantically rich feature extraction approaches like word2Vec and BERT, they experiment with different clustering and topic modeling techniques to produce a list of potential topics from a given set of documents.
Although the focus on previous work in topic classiﬁcation and modelling is on low-resourced languages, these languages have different dialects, syntax and lexical semantics.
Therefore, it is important to build a machine learning model that can be trained on speciﬁc natural language corpora to capture speciﬁc language features.
Currently, there is hardly any NLP research involving the Luganda language in classifying and modeling topics from a dataset or a set of documents.
In an effort to close this gap, this research explores the application of various machine learning approaches both supervised (classic, neural and pretrained models) and unsupervised(NMF) on the Luganda dataset.
M ETHODOLOGY 3. D ATA
A Corpus of 15000 English sentences was retrieved from websites and newspapers that publish articles in Uganda.
This English text data was sub divided into sets of 100 sentences and distributed for translation among a team of Luganda Linguists from the Department of African Languages Makerere University.
While exploring the data, a word cloud was generated to provide a visual representation of the data.
District names like Nebbi, Yumbe, Moyo, Koboko were identiﬁed as some of the most common words in the data as depicted in Figure 1, which shows that the data was about the local regions in Uganda.
3. D ATA A NNOTATION
We developed annotation guidelines which were followed by annotators who are Luganda Linguists to review and label each sentence with the corresponding topic code based on the sentence subject.
Annotators categorized sentences into 14 topics that have been captured in both English and Luganda as depicted in Table1. [Table]
The distribution of the Luganda corpora used in this study across the topics is as depicted in Figure [Table]
3. D ATA P REPROCESSING
Luganda is a low-resourced language which implies that the available resources for NLP researchers such as lemmatizers, stemmers, a rich set of stop-words are limited and not available.
Data preprocessing involved identifying and removing stop words from the corpora and generating word embeddings.
3.3. L UGANDA S TOP WORDS
Together with the linguists
[Table]
we created a list of Luganda stop words as depicted in Table 2.
The list was compiled from the Luganda corpora
[Table]
and this is a resource that other NLP researchers can use as a benchmark while dealing with Luganda data and/or code-mixed.
3. T OPIC M ODELING
We used the Non-negative Matrix Factorization(NMF)
topic modeling technique to detect and discover meaningful topics from the unlabeled monolingual Luganda corpora.
In the NMF model, [Table] each sentence was considered as a data point.
The NMF model operates by decomposing high dimensional vectors into lower-dimensional representations, and these lower-dimensional vectors are non-negative, which means that their coefﬁcients are non-negative as well (Hyun Ah & Soo-Young, 2013).
Using the original matrix (A) , NMF gives two matrices (W and H) where W is the topics the model found, H is the coefﬁcient(weight) of a topic.
Equivalent to this is; A matrix holds records by words, H matrix holds records by topics and W is a representation of topics by words.
The NMF model was trained on features extracted from the TF-IDF vectorizer.
The TF-IDF vectors were considered as high dimensional vectors to enable the model to modify the initial values of W and H so that the product approaches A until either approximation error converges or the maximum iterations are reached.
Most of the hyper-parameters in the NMF model were set to their default values.
However, some parameters were changed such as solver which was set to ”mu” value; maxinterations which was set to 1000 , alpha set to 0 . 01 , and l1-ratio set to 0 .
3. T OPIC CLASSIFICATION
After annotating the data, the data was unevenly distributed across the different topics as depicted in
Figure 2.
The “business” topic was the majority class with 1423 data points whereas the “environment” was the minority class with 350 sentences.
Standard classiﬁers usually get biased towards the majority class (Cristian & Mihaela Elena, 2019).
Random oversampling was applied to the training dataset to create a balanced set across all the topics.
With random oversampling, samples of the data are taken from the minority classes(business class) randomly and duplicate instances are created so that the minority class reaches a size comparable with the majority class (Cristian & Mihaela Elena, 2019).
After resampling the size of the corpus increased by 30% to make a dataset of 20922 sentences.
However, experiments were done on both the imbalanced set and re-sampled set and the results are presented in the subsequent sections.
3.5. W ORD E MBEDDINGS
Word embeddings are commonly used in NLP research as real-valued representations because of their ability to capture lexical semantics from the natural language corpora they are trained on (Amir, 2018).
According to recent studies, the transfer learning approach has been used for text classiﬁcation in low-resource languages.
This approach involves using the features of high-resource languages, which are learned by pre-trained word embeddings to train models on low-resource languages.
However, this technique might not be practical or even applicable in a low-resource setting due to insufﬁcient parallel corpus for both the low-resource and high-resource languages
Rubungo Andre et al.
On the other hand, these existing pre-trained models such as Glove,
Fast text, Paragram, and word2vec are not applicable for Luganda since they were exclusively trained on high resource languages.
We have developed Luganda word embeddings in this research, i.e., Fasttext, Glove, Paragram, and
Word2vec Embeddings on a 50-dimensional input feature from the 15,000 monolingual Luganda dataset.
These were trained using the Gensim 1 and the Glove framework 2 .
3.5. T OPIC CLASSIFICATION METHODS
We utilized different classiﬁcation methods to learn topic classiﬁcation on the Luganda dataset.
These included: baseline models Logistic Regression, Support vector machine, multi-layer perceptron, XGBoost model, neural network models, and pre-trained models.
Baseline Models For the baseline models, the data was transformed into tokens using a count vectorizer.
A count vectorizer provides a simple way to tokenize a collection of sentences, build a vocabulary of known words, and also encode new documents using that vocabulary.
All the baseline models above were implemented with the help of the scikit-learn framework 3 .
The dataset was split into: training set and testing set, with a ratio of 9:1.
Neural Models For all the proposed neural models, As input into the neural network, every sentence was tokenized to produce a vector and then the input sequence was truncated and padded.
The padded sentences generated for every sentence in the data were then taken as input into the model to output the different class topics.
The dataset was split into: training set, validation set and testing set, with a ratio of 8:1:1.
B IDIRECTIONAL LSTM WITH 2D M AX POOLING
We used Bidirectional LSTM for text classiﬁcation as proposed by Peng et al.
(2016) with default hyperparameters.
However, we changed the original feature map to 250 and a min-batch to 64 since we were training the model on a small dataset.
This model was trained with and without word embeddings in the embedding layer.
While training with word embeddings, we used three different word embeddings i.e., Glove, fasttext and word2vec embeddings of the same dimensions (50 dimensions).
G ATED R ECURRENT U NIT (GRU)
We explored a GRU for text classiﬁcation proposed by Junyoung et al.
(2014) with default hyperparameters.
An embedding layer was used to input wordembeddings and in another experiment word-embeddings were not used in the ﬁrst layer, the second layer was the Spatial 1 Dimension of Dropout(SpatialDropout1D) layer with a dropout of 0.2.
The GRU layer was used as the fourth layer followed by the ﬁfth layer as the output dense layer.
Pre-trained Models We used the BERT and RoBERTa transformer models which are pretrained on English data for text classiﬁcation from the Hugging Face platform 4 .
As proposed by Ashish et al.
we retrained both the BERT and RoBERTa models using a Luganda dataset to implement Luganda text classiﬁcation using the above mentioned models. The dataset was split into: training set, validation set and testing set, with a ratio of 8:1:1.
3. E VALUATION METRICS 3.6. C OHERENCE S CORE
The performance of the NMF model was measured using coherence score.
Coherence score measures the relative distance between words in topics.
Mainly there are two kinds of coherence scores: CV Coherence and UMass.
CV Coherence, creates content vectors of words using their co-occurrences and, after that, calculates the score using normalized pointwise mutual information (NPMI) and the cosine similarity. UMass calculates how often two words appear together in the corpus.
For this study, we used CV Coherence which ranges between 0 and 1 with 1 being the best value where the topics are perfectly coherent.
3.6. M ODEL RESULTS
We measured and assessed the performance of the topic classiﬁcation models based on different evaluation metrics such as F1 score, Precision, and Recall These were computed using the values in the confusion matrix that is: True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN), as depicted in Figures 3 and 4.
R ESULTS
Figure 5 in the Appendix section show the results from the NMF model.
The results include the meaningful topics generated from the model.
Additionally, the top 10 words under each topic are captured in Table 3.
We achieved a coherence score of 0.56 on evaluating the NMF model. Topic Topic Topic
Topic
Topic
Topicakawuka ettaka poliisi ttaka abayizi ttiimu kolona enkaayana emisango obukuubagano ssomero omupiira abalwadde ensonga omusango enkaayana abasomesa abawagizi abalina ssentebe abateeberezebwa nkaayana akamalirizo abazannyi abalala lyabwe okunoonyereza bwannannyini bubi baayo okusaasaana zirina okwekalakaasa obwannannyini masomero mpaka kawuka ekyalo kaduukulu bungi ensoma ebyemizannyo omuwendo lisobola ekitundu lyabwe bajja ebigere abasawo okuwandiisa okukuuma okugonjoola omusomesa amaanyi obulwadde mingi ekwata mateeka abali omutendesi Topic Topic Topic
Topic
Topic
Topicabalimi kkooti okulonda kitundu enjawulo enguudo ebirime emisango omwaka obutebenkevu obuwangwa embi beetaaga enkulu ogujja obutali ebifo obubenje abalunzi mingi pulezidenti mugaso amawanga zirina ebibiina
musango bwenkanya emirembe bulina nguudo okulima yasindikiddwa ennaku obuwangwa enzikiriza ennungi balima guno omuntu amagye byobuwangwa nnyingi katale ensala kwaliwo butebenkevu ngeri mbeera abasinga obujulizi okuba entambula ebintu ensimbi obwegassi mateeka kujja ebyokwerinda bingi entambula
Topic
Topic
Topic
Topic
Topic
Topicemiti katonda
abaana essomero bangi ssente obutonde kkanisa okusoma omukulu ekirwadde okukola ensi omuntu abazadde ebibiina ssenyiga bizinensi okutema obulamu ssomero bbaasi omukambwe okufuna okusimba alina masomero abasomesa abavubuka pulojekiti okukuuma abakrisito bateekeddwa abalala obulamu nnyingi ensigo okubeera beetaaga okuzimba bafudde oluguudo butebenkevu ngeri amasomero omukulembeze ggwanga emirimu kyonoona kisa engeri ebisale bwabwe ekitundu batema okuweereza abato amazzi ababbi kkampuni
We also present results from the topic classiﬁcation experiments using the evaluation metrics i.e.,
F1 score, precision and recall for all the classiﬁers used in this study on both the unsampled dataset and resampled dataset as depicted in Tables 4 and 5 respectively.
In the groups of classic models, SVM emerges the best performer on both the unsampled and resampled datasets whereas for the pre-trained models BERT performed better than the RoBERTa model.
4. D ISCUSSION OF R ESULTS
As depicted in Figure 5, the model clusters topics: COVID (“Ssenyinga omukambwe”), Land (“Ebyettaka”), Security (“Ebyokwerinda”), and Education (“Ebyenjigirza”) under more than one cluster.
This shows the weights of these topics over the other topics but it also shows us more words that are associated with that topic as stored in the W matrix.
The Luganda topic model was not able to linearly separate the Health topic and this could be attributed to the fact that some of the keywords generated under the COVID topic such as “ekirwadde”, “obulwadde”, “abalwadde”, “obulamu”, “akawuka” are also used while discussing or referring to the Health topic but also given the fact that this data was collected during the COVID-19 pandemic, [Table] [Table] there were more co-occurrences of these words with words like “senyiga”, “okusasana” which were commonly used to refer to COVID-19.
On resampling the training data for topic classiﬁcation [Table] the performance for all the models increased as depicted in Table 5.
There was a signiﬁcant difference in the performance of the models between the two datasets(unsampled and resampled) and thus the sampling technique worked well for topic classiﬁcation of the Luganda dataset.
The results of the neural models with the Luganda word embeddings that were developed from the dataset using word2vec models are not presented in this paper as they are slightly lower than the results obtained from the default layers of the neural network.
This could be attributed to the size of the data on which the embeddings were developed. ENGLISH SENTENCE LUGANDA SENTENCE ACTUAL
PREDICTED LABEL
LABEL
BERT SVM NMF Youth are urged to have medical check-ups early enough
Abavubuka bakubiribiwa okwekebeza nga bukyali
H
H
H
Other
A police ofﬁcer has been sentenced to jail for six months
Omusirikale wa
Poliisi asindikiddwa mu kkomera okumala emyezi omukaaga
L
L
L
SE
A lawyer has been sued by the court for three cases
KKOOTI eggudde emisango esatu ku Munnamateeka
L
L
L
L
The police has warned people who abuse other people’s rights while pretending to exercise there rights
Poliisi erabudde abeerimbika mu ddembe lyabwe okuvvoola erya abalala
SE
SE
SE
SE
Youth are urged not to sell off land
Abavubuka basabiddwa obuteetundako ttaka
Land
Land
Land
Land
What comprise of the deal to send bodaboda riders out of the city?
Ebiri mu ddiiru egoba takisi ne boda boda mu Kibuga
T
T
T
B
Lets go and play football
Ka tugende tuzannye omupiira
S
S
S
S
The teacher reached earlier in class than us
Omusomesa yetusoose mukibiina
Educ
Educ
Educ
Educ
He urged the public to go for corona vaccination
Yakubirizza abantu okugenda bagemebwe akawuka ka
Kkolona.
Covid
Covid
Covid
Covid
He went to his garden to dig Yagenze mu nnimiro ye [Table] To better assess the models both the Luganda topic classiﬁcation models and the Luganda topic model, but also to understand the error set (points that are off the main diagonal in the confusion matrices shown in ﬁgure 3 and ﬁgure 4) we used a sample set of ten new Luganda sentences to test and compare the results of the deployed best Luganda topic classiﬁcation models and the Luganda topic model, as presented in Table 6.
From that comparison, we observed that for some instances the Luganda topic model prediction deviates from that of the Luganda topic classiﬁcation models and also the actual label.
This was attributed to the issue of duality, a scenario where a sentence can belong to more than one topic.
A sentence such as “Omusirikale wa Poliisi asindikiddwa mu kkomera okumala emyezi mukaaga.” translated as “A police ofﬁcer has been sentenced to jail for six months.”
in English can belong to either the Legal topic or Security topic . Since the models were trained on a single topic classiﬁcation problem, in a scenario where the provided sentence belongs to more than one topic, the models can only provide one of the topics as the result.
C ONCLUSION AND F UTURE W ORK
This research presents the ﬁrst topic classiﬁcation and modeling benchmark for Luganda, a low resource language and the most common native language in Uganda.
The performance of the models builds a conﬁdence that we can develop downstream models in low-resourced languages which can be used in different applications.
Considering the gap between high resource languages and low resource languages, our future work will look at enriching the dataset to address the issue of duality in topic classiﬁcation and also use the data to develop better word embeddings for the language. A CKNOWLEDGMENTS
This work was carried out with support from Lacuna Fund, an initiative cofounded by The Rockefeller Foundation, Google.org, and Canada’s International Development Research Centre
A PPENDIX
[Figure]