I si Z ulu noun classification based on replicating the ensemble approach for R unyankore (2025)

Abstract: A noun’s class is a crucial component in NLP, because it governs agreement across the sentence in Niger Congo B (NCB) languages, among others. The phenomenon is ill-documented in most NCB languages, or in a non-reusable format, such as a printed dictionary subject to copyright restrictions. A promising approach by Byamugisha (2022) used a data-driven approach for Runyankore that combined syntax and semantics. The code and data are inaccessible however, and it remains to be seen whether it is suitable for other NCB languages. We aimed to reproduce Byamugisha’s experiment, but then for isiZulu. We conducted this as two independent experiments, so that we also could subject it to a meta-analysis. Results showed that it was reproducible only in part, mainly due to imprecision in the original description, and the current impossibility to generate the same kind of source data set generated from an existing grammar. The different choices made in attempting to reproduce the pipeline as well as differences in choice of training and test data had a large effect on the eventual accuracy of noun class disambiguation but could produce accuracies in the same range as for Runyankore: 80-85%.

## introduction there has been a resurgence of research focusing on dialogue systems and interfaces in recent years, through efforts focusing on building virtual assistants @xcite , large language model (llm) powered systems that are used in industry @xcite , and social robots (e.g., @xcite ). this trend can also be seen in african natural language processing (nlp) research; where there have been efforts to build ontology verbalisers @xcite @xcite , determine the requirements for building social robots @xcite , and data collection aimed at building speech recognition systems @xcite . within this context, noun class identification/classification has not received a lot of attention in several african languages despite its importance to these existing efforts. we demonstrate the importance of computational methods for noun class/category identification using an example of a hypothetical kitchen robot that communicates in isizulu and french. since french nouns are categorised by gender, one needs the capability to determine the gender of each noun; hence, the robot will need to be able to select or generate the right option from la casserole 'the f eminine pot' vs. le casserole 'the masculine pot'. for a kitchen robot to respond in isizulu, it may need to generate amazambane athambile 'the potatoes are soft' by choosing the appropriate prefix (underlined) using automatically detected class of the subject noun, instead of amazambane lithambile 'the potatoes is soft' or any other prefix. for isizulu, however, there is no computational noun class identification model that such a robot has to rely on. broadening the scope of languages, the work by @xcite produced the most promising results on this particular task, obtaining best results using a noun's morphological and semantic information to identify its class, and outperforming other approaches by a large margin. however, it focused only on three languages belonging to guthrie zone j-thousands of kilometres apart from zone s that isizulu is part of-and produced no open source and re-usable tool for use with other languages. thus, there is a need to investigate the extent to which its findings hold for languages outside guthrie zone j. the first, and key, task is to determine whether the procedure of @xcite is reproducible, serving as a candidate for a method also useful for other niger-congo b (ncb) languages, and if it is not, what the (in)surmountable impediments are. second, if the approach is reproducible, then to ascertain whether the performance of noun class detection is in the same range of success as for runyankore that byamugisha (2022) focussed on. third, the aim is to make as much data and software available as legally possible. to investigate the former as a reproducibility study, two co-authors gave the task to two other authors without their knowledge, one of whom had the sole focus of teasing out reproducibility issues and the other as a 'quickly reproduce it' for it to serve as a baseline for designing a better algorithm to outperform said baseline. all used the most wellresourced among the low-resourced languages of south africa, namely, isizulu. the best performing models, emanating from the 'quickly reproduce it' group, affirm @xcite , since the model that combines syntax, morphology, and semantics has an accuracy of 83% while the prefix-only model has accuracy of 59%. however, the differences in the choice of training and test data, as well as the design of the module that combines the different linguistic aspects have a significant impact on performance. in the remainder of the paper, we first describe pertinent details about the ncb noun class system (section 2) and related work (section 3). this is followed by the main sections on reproducibility experiment design (section 4) and results and discussion (section 5). we close with conclusions in section 6. ## noun complexity in ncb languages prior to discussing computational approaches for noun class identification, we first present the complexity of ncb nouns. we do not limit our discussion of noun complexity to only languages in guthrie zone j or s since there are features shared across all ncb languages. in ncb languages, nouns are categorised into one of 23 classes. for instance, in isizulu the noun umuthi 'tree' belongs to noun class 3 and its plural imithi 'trees' belongs to noun class 4. for each ncb language, nouns are not necessarily categorised into all the 23 noun classes since some of the classes may be obsolete, or the word is not classified as a noun, such as ekhaya 'at home' in isizulu as the locative of ikhaya 'home' versus eka 'at home' in noun class 23 in luganda. we now turn to our running example of a kitchen/social robot to show the impact of the noun class on the text: unlike the french case where the robot can only generate two forms of the text, the noun class of the subject can lead to many more variations of the text; e.g., zithambile, ihambile, bathambile, lithambile, athambile, sithambile, luthambile, buthambile, and kuthambile (all meaning 'soft'), where the underlined parts denote the subject concord values that are dependent on the noun class of the subject. the task of retrieving a concord can be solved easily; however, the question of how to computationally identify the class of a noun still needs to be resolved. while early linguistic literature argued that the noun classes are not a semantically motivated categorisation @xcite , a number of authors have attempted to disprove that for proto-bantu 1 (denny and creider, 1986) or individual ncb languages such as @xcite , swahili (contini-morava, 1997), shona @xcite , sesotho, setswana, isizulu @xcite , and siswati @xcite . despite their proposals, however, there is no consensus on the matter and most authors rely on the rough guide as summarised in the semantic features shown in the ## existing computational models and their usability to the best of our knowledge, there is limited work focusing on noun class identification for ncb languages. as such, we cast the net wider to also include the computational modelling of nouns. most existing computational models created for ncb nouns can be classified into: noun pluralization (e.g., @xcite ), lemmatization, stemming and segmentation (e.g., @xcite @xcite ), morphological analysis (e.g., @xcite ), and pos tagging (e.g., @xcite ). when considering work for languages from outside the african continent, we see that there are semantic and gender classifiers (gagliardi et al., 2012; falk 1 the hypothetical ancestor of all ncb languages byamugisha (2022) created a three-module classifier, whose architecture is provided in figure 1 , using three datasets made up of 2803 runyankore, 153 luganda, and 70 kinyarwanda nouns and split them 70% for training, 20% for validation, and 10% for testing. following that, they then created three main model variants for identifying the class when provided with a noun (i.e., morphological, semantic, or morphological + syntax + semantic). the morphological model uses "morphological rules" to match a prefix to one of the noun classes, should the class's prefix be unique. the semantic model uses a pretrained fasttext embedding model to embed the input noun and then relies on two al-ternative algorithms to identify between 10-1000 semantically related words. the properties of the neighbouring words are then used to predict noun class of the input noun. the third type is ensemble that starts by using the morphological model and in cases where it fails, it then feeds the noun to the semantic model. while the work concluded that the combination of morphology, syntax, and semantics yields the best performance, the generalisability of this finding across different guthrie zones is unclear since the author only focused on zone j. for instance, a key notion with the semantic model is the reliance on the uniqueness of grammatical information that is used to label each input's semantic neighbours. however, it is unclear whether such an assumption holds for ncb languages in other guthrie zones as well. there is also no open access source code version of their tool that can be used to investigate generalisability across the different guthrie zones for languages that have the appropriate datasets. ## experiment design while the work done by @xcite shows some promising results when combining syntax and semantics, there are some details that were not clear in the associated publications (i.e., (byamugisha, 2020, 2022)). for instance, they use a dataset of sentences to create static word embeddings; however, the nature of the model used to obtain the embeddings is not specified (i.e., skipgram or continuous bag-of-words (cbow)). similarly, they state that a dataset is enriched with syntax and morphological information and then used to train a classifier to predict parts-of-speech and morphological information. however, it is not specified whether the dataset is labelled manually or whether each label is assigned to a sentence or the individual words. owing to this, the work was independently replicated by two groups for isizulu using different datasets and we confirmed some details with the author. ## results and discussion the accuracy of the resulting models are reported in the models that classify nouns based on only their prefix differ by 23% between the two groups. since group 1's model does not make any predictions for prefixes that are ambiguous and group 2 does so by averaging over multiple runs, this suggests that there is utility in randomly choosing a class amongst possible classes, on average over multiple uses. nonetheless, byamugisha's prefix-only model outperforms groups 2's model by 10%. it is unclear whether the two models are directly comparable because byamugisha's work lacks clarify regarding how this model processes nouns where usage of the prefix, by itself, to determine the noun class is insufficient. it is likely that it ignores such nouns similar to group 1 or employs some other strategy. there was also a large gap in performance in the classifiers that rely on the prefix, syntax, and semantics. the largest gap in accuracy between group 1 and 2's models was 17%. there are numerous reasons why there could be differences in performance between the models, especially since there were multiple differences taken by the two groups due to interpretation and approach to replication. of note, while group 1 relied on a similar sized dataset to @xcite for training the semantic classifier while the second group did not, the group 1's model performs far worse than byamugisha's model. this may indicate that ones' possible access to the same kind of resources, and not just similarly sized ones, has an impact on the accuracy-in addition to their interpretation of the original work. more generally, the two groups identified the following areas for which the original work differs: there is no equivalent cfg for isizulu; hence, the groups used classifiers that are trained on similarly sized datasets and crafted labelled datasets. notably, group 1's careful design of the labelled dataset vs. group 2's use of ad hoc rules did not yield better performance overall. 3. both groups were unable to identify the precise filtering strategy employed by byamugisha hence considered multiple variations. these replication challenges and differences in interpretation highlight that there is still a need to (1) conduct a comprehensive exploration of key functionalities, such as candidate prediction filtering and quantifying the accuracy of the prefix-only model while ensuring transparency regarding its handling of nouns that cannot classified using the prefix only, and (2) investigate more diverse models that integrate semantics, syntax, and morphology, ## conclusions aiming to replicate byamugisha (2022)'s work on a combined syntactic and semantic approach to classifying nouns in their noun class for isizulu, our results showed that our best performing models combined syntax, morphology, and semantics yield better performance (83%) compared to relying only on the prefix (59%), which is in line with the findings for runyankore. however, the differences in the choice of training and test data, as well the design of the models that make up the module that combines syntax, morphology, and semantics, had a substantial effect on the eventual accuracy of noun class disambiguation. specifically, while group 1's model with prefix-semanticssyntax outperformed their prefix-only models, their performance was not in the same range as the runyankore models. possible causes for the difference in accuracy is the lack of a sufficient cfg to create training data in isizulu and the imprecision with which candidate filtering was implemented in the original work. future work includes investigating the impact of differences in the number nouns that cannot be classified by prefix alone per language and how that effects transferability of the use of semantics, syntax, and morphology and investigating more diverse models (e.g., include pretrained language models) that integrate such knowledge while ensuring that the data is auditable.