Deep learning and low-resource languages: How much data is enough? A case study of three linguistically distinct S outh A frican languages (2023)

Abstract: In this paper we present a case study for three under-resourced linguistically distinct South African languages (Afrikaans, isiZulu, and Sesotho sa Leboa) to investigate the influence of data size and linguistic nature of a language on the performance of different embedding types. Our experimental setup consists of training embeddings on increasing amounts of data and then evaluating the impact of data size for the downstream task of part of speech tagging. We find that relatively little data can produce useful representations for this specific task for all three languages. Our analysis also shows that the influence of linguistic and orthographic differences between languages should not be underestimated: morphologically complex, conjunctively written languages (isiZulu in our case) need substantially more data to achieve good results, while disjunctively written languages require substantially less data. This is not only the case with regard to the data for training the embedding model, but also annotated training material for the task at hand. It is therefore imperative to know the characteristics of the language you are working on to make linguistically informed choices about the amount of data and the type of embeddings to use.

## introduction over the last decade vectorised word representations and the use of deep learning have become de facto standards in natural language processing (nlp) @xcite . there has also been a push to broaden the linguistic diversity in nlp research @xcite . both learning vectorised representations, a.k.a. embeddings, and deep learning are inherently data-driven procedures where models are trained from vast amounts of data to either represent language numerically or learn some downstream task. including a bigger variety of languages than mainstream languages, such as english, spanish, ger-man, japanese, etc., to achieve more linguistic diversity typically means studying low-resource or under-resourced languages. this, however, leads to a dichotomy: highperforming deep learning models, like bert, have been trained on billions of words. when developing models for languages other than english, lesser resourced languages (such as the south african languages) get left behind because there is very little available data. also, evaluation of existing techniques is only partially applied to under-resourced languages and researchers typically assume that the generalisations achieved with training on a lot of data will mostly hold true with less data. more recently there have been efforts to extend the usefulness of embeddings trained on wellresourced languages with languages that have substantially less data in so-called multi-lingual models, such as xlm-r @xcite and mbert @xcite . these models generate representations that are a combination of language specific information as well as information learned across all of the languages included in the model. typically they are trained exclusively on web data like common crawl or wikipedia, which have some inherent limitations (as discussed later in this paper), but also have limited availability for south african languages. for instance, isindebele has no wikipedia data and is therefore not even present in common crawl. consequently, most of the south african languages are not included in these multilingual models. doing nlp research on south african languages in this day and age then leads to the question of what the implications of working with very little data is on current standard techniques like embeddings and neural language models. or in other words: how much data is needed for learning useful vector representations? the underlying assumption is that learning from less data will yield less representative and thus less useful models. it re-mains to be seen, however, if this is truly the case. from a linguistic diversity point of view, it is also relevant to know how the embedding models vary from each other for structurally different languages and how the amount of available data influences the learned representations for typographically different languages. in this paper, we present a case study attempting to answer these questions. our setup consists of training embeddings for three linguistically distinct south african languages (afrikaans, isizulu, and sesotho sa leboa) to evaluate the impact of embeddings trained on increasing amounts of data for a part of speech (pos) tagging downstream task. the goal is to determine the influence of data size on the performance of different embedding types and to describe the effects observed for different languages. the results of our experiments show that even relatively little data can be useful in some scenarios and that morphologically complex and conjunctively written languages require substantially more data, both for training the embeddings and the downstream task, especially when using full/sub word representations. @xcite have a number of linguistic characteristics that make them substantially different from most indo-european languages: all of them are tone languages; they use an elaborate system of noun classes with up to 21 classes; and their nominal and verbal morphology is highly agglutinative and very productive, which can result in a large vocabulary for those languages that follow the conjunctive writing system. for historic reasons, the south african ncb languages adopted two different writing systems, either conjunctive or disjunctive, where a distinction is generally made between linguistic words and orthographic words. for conjunctively written languages one orthographic word (token) corresponds to one or more linguistic words, whereas for the disjunctively written languages several orthographic words can correspond to one linguistic word @xcite . the four nguni languages, isindebele, isixhosa, isizulu, and siswati, are written conjunctively, while the three sotho languages, sesotho, sesotho sa leboa (also known as sepedi), and setswana, tshivend ˆa, a venda language, as well as xitsonga, a tswa-ronga language, are disjunctively written. this is a marked difference from the two germanic languages present in south africa, afrikaans and english, where mostly a linguistic word and an orthographic word coincide. the implication of these different writing systems is that multiple tokens in disjunctively written languages can correspond to a single token in the conjunctively written languages. this leads to sparse token frequency for the conjunctively written languages, while the opposite is true for the disjunctive languages. as an illustration, the parallel equivalents of a 50,000 word english corpus will have approximately 43,000 words for the conjunctively written languages, while the disjunctive languages will have approximately 60,000 tokens. this difference and its implications are discussed in more detail in section 3. one of the objectives of this paper is to investigate the influence of linguistic and orthographic differences in south african languages on using embedding models in nlp tasks, specifically pos tagging. to that purpose we have chosen one language from each family for our experiments: afrikaans (germanic), isizulu (conjunctively written nguni) and sesotho sa leboa (disjunctively written sotho). ## data the major prerequisite for training embeddings and language models for any language is the availability of large amounts of text data. although there have been several efforts to create such corpora for the south african languages @xcite @xcite , there is still relatively little data available for most of them. the data collected for this study is a combination of various open data sets (mostly cc-by and cc-nc licenses), as well as some data only available to the authors with copyright restrictions prohibiting the distribution of the full corpora. the data included in the training corpora for the isizulu and sesotho sa leboa embeddings are primarily from the nchlt text corpora @xcite although the data in both the leipzig and common crawl corpora are language identified, an initial investigation showed that a substantial amount of the data is incorrectly attributed to one of the languages. this is primarily due to the fact that all three languages in this study have related languages that share similar orthographic features which leads to misclassification of the language data, specifically: consequently, all of the data from the leipzig and common crawl corpora were further cleaned with the nchlt language identifier @xcite a summary of the data available for training embeddings is presented in the pos data used in this study is the nchlt annotated corpora for afrikaans and sesotho sa leboa @xcite , and the linguistically enriched corpora for conjunctively written south african languages for isizulu @xcite . each annotated corpus consists of approximately 50,000 tokens for the training set, and a separate test set of approximately 5,000 tokens. although the data is annotated on very fine-grained pos tag sets (typically consisting of 90+ tags), for this investigation we reduced the tag sets to between 12 and 17 tags by e.g. excluding class information and using only main pos classes. this makes the results between languages more compa-rable, but does not obscure the functional differences a conversion to upos foot_2 would. an overview of the pos data and tags is presented in ## experimental design in order to determine the impact of different embedding architectures and morphosyntactic attributes on the usefulness of embeddings in low-resource environments, we perform a set of experiments to establish how these attributes in combination with data size affect the quality of a single downstream task, namely pos tagging. the first step in the process is generating embeddings in each of the chosen architectures -fast-text, glove, and flair -with different data set sizes. for each language a random selection of paragraphs from the available corpus is made in iteratively larger sizes, starting with 10,000 paragraphs and doubling the amount of data randomly for each iteration. for isizulu and sesotho sa leboa this process is repeated up to the full available corpus (292,600 and 838,000 paragraphs respectively), while for afrikaans we only select data up to one increment above the largest of the other two languages (1,280,000 paragraphs). 4 based on each data iteration, embeddings for all three architectures, including their different flavours, are trained. to make the comparison of models as consequent as possible, the hyperparameters for each of the architectures are kept the same (typically the default settings, see the quality of the embeddings trained on the different corpus sizes for the three languages is evaluated using a vanilla bidirectional lstm-crf pos tagger implemented as part of the flair framework foot_4 @xcite . a separate pos tagger is trained for each of the embedding models, on the same gpu hardware used to train the embeddings. the main reason for using the flair framework is the fact that it has built-in support for all of the different embedding architectures, thus ensuring that the results of the respective taggers can reasonably be compared. as with the embedding models, the hyperparameters are kept to the default settings (hidden size: 256, learning rate: 0.1, drop out: 0.05, epochs: 40). it should also be noted that because the pos training sets are relatively small, fine-tuning of the embeddings during the pos tagger training is not carried out. all taggers are evaluated using the accuracy metric, and compared to the nchlt web services foot_5 pos taggers @xcite as a baseline for each language. with such a large set of taggers (24 in total), the pos taggers were not trained multiple times with averaged scores. doing so would once again substantially increase the required gpu hours (currently 82) required for the experiment, and cannot ## results and discussion we will now discuss the performance of the various embedding models with different data sizes for afrikaans, isizulu and sesotho sa leboa. a graphic representation of the performances on the pos tagging task can be found in figures 1 (for afrikaans), 2 (for isizulu) and 3 (for sesotho sa leboa), with the full numerical results available in the first notable conclusion that can be drawn from inspecting the results is that even embeddings trained on very small corpora can benefit the quality of relatively simple downstream tasks, such as pos tagging, when compared to baseline systems. specifically, the flair contextual character embeddings produce downstream results that are surprisingly good, even for the conjunctively written isizulu. it was expected that the characterbased models would perform best with very small amounts of data, but the models trained on the smallest data sets actually perform comparably to the best results for any of the other embeddings. conversely, although the flair models perform best when trained with the largest data sets, there is not nearly the same level of improvement as there is for the other two architectures. one implication of these findings is that much smaller and faster models may perform well enough for certain purposes, if not necessarily attaining state-of-the-art results. this allows researchers and developers with limited hardware capacity to also benefit from using these types of embeddings. as expected, glove embeddings consistently perform the worst of all the embedding types, especially so with the first couple of iterations of very small corpora, for two main reasons. firstly, as these embeddings only generate representations for words in the vocabulary, all words in the tagging task that are not part of the vocabulary are represented by the same vector, and therefore do not have any distinctive representations. secondly, since many words will only appear a small number of times in the training data, learning complex representations is difficult when a word is only seen in a small number of contexts. these problems are exacerbated for isizulu where the conjunctive writing style causes a large number of distinctive co-occurrences for all words, especially less frequent words, and learning meaningful representations is almost impossible. for the disjunctively written sesotho sa leboa, however, the vocabulary is relatively representative even with a small corpus, and glove embeddings perform only slightly worse than the other embedding types. the fasttext models perform substantially better than the glove models with very small data sets, while still performing worse than the flair embeddings. with the largest data sets, however, the fasttext cbow models perform either very similarly or better than the flair models. interestingly, with very small data sets the skipgram models outperform the cbow models for the first two or three data iterations, after which the cbow models consistently perform better across all languages. it is not immediately obvious why this would be the case, but the fact that this occurs across all three languages definitively shows that with very small data sets skipgrams are preferable over cbow, whereas for any data set with more than 500,000 or a million tokens (corresponding to about 40,000 paragraphs in our data), the cbow models generate better representations measured on the pos tagging task. this contradicts the initial findings of @xcite , but is in line with the latest released fasttext models 7 which have also switched to cbow models by default, as opposed to previously released models @xcite . our results also clearly show that the writing system of the language plays a major part in how much data is required to train embeddings that are useful to any degree. for conjunctively written languages, as the token-level morphological complexity of the language increases, so does the amount of data required to create meaningful representations. in the two extreme cases of isizulu and sesotho sa leboa, even the embeddings from the largest available corpora for isizulu perform substantially worse than the embeddings based on the smallest sesotho sa leboa corpus, with accuracies between 4.28% and 15.19% lower depending on the model. afrikaans, which is slightly more morphologically complex on token level than sesotho sa leboa, but not nearly as complex as isizulu, also performs somewhere between the two languages when considering the 7 https://fasttext.cc/docs/en/crawl-vectors. html different data sizes and embeddings. apart from the general findings presented in the previous paragraphs, there are also certain language specific aspects of the results that warrant discussion. we include some broad linguistic error analysis for each of the languages to determine where the main focus of errors are for the best models for each language. 8 for afrikaans, the fasttext cbow model performs the worst of all models on the smallest data set, but shows the largest degree of improvement as the data size increases, to the point where it is the best performing of all models on the largest data set. also, the afrikaans flair forward model performs at almost an identical level to the fasttext cbow model, while the flair backward model is slightly worse. when considering the tag error differences between the embeddings trained on the smallest and largest corpora, it becomes clear that the main source of improvements for both fasttext and glove embeddings is the size of the data. as more words are included in the vocabulary, the relative error rates for nouns, verbs, adjectives, and adverbs are reduced by between 35% and 45%. the relative error rate reductions for the flair models are not as uniform across all open word classes. as an example, the flair forward model reduces the percentage of errors for adjective and verbs by 43% and 72% respectively, while increasing the number of errors for nouns or adverbs. the flair backward model shows improvements for adjectives (25%) and verbs (61%) as well, but also substantial improvements for nouns (64%) and adverbs (48%). in the case of isizulu, the flair backward model performs best overall, although the flair forward performs comparably. this differentiation with afrikaans is likely due to the fact that isizulu uses prefixation more productively than suffixation, and processing data from the end of the text to the beginning leads to a slightly more informative model. the glove model for isizulu is significantly worse than any of the other models trained in this investigation and is definitely a consequence of data sparsity during training as well as previously unseen words in the tagging task. this problem is less prevalent for the fasttext models: since the ngrams of previously unseen words can still generate a representation, and although not as informative to the task, these "subword" representations obviously have a substantial impact on the quality of the results. the cbow and glove models show the largest error rate reductions across the major pos classes of between 17% and 77%, particularly for possessives and adverbs. the flair models on the other hand do not show large improvements for any of the categories, and the improvements are counteracted by regressions in other classes, to which end the results remain relatively stable between the models trained on the smallest and largest corpora. even though the flair embeddings perform best for isizulu, there is very little improvement for these models as the size of the data increases. there are two possible, and related, reasons why this may be the case. firstly, since most of the affixation in isizulu is fairly regular, most of the morphological structure of the language may be encoded well with small amounts of data. this is supported by the fact that the most productive word classes (i.e. nouns, verbs, adverbs, possessives, and relatives) are tagged more accurately with a flair model trained on the smallest amount of data, than for any of the other embedding models. the fast-text cbow model does perform similarly on these classes with the largest training set, and may possibly outperform the flair models if more data is made available. the second possible reason is that the annotated pos training data is just too small for further improvements to be possible, and a substantially larger set is required to attain results comparable to those of afrikaans or sesotho sa leboa. all of the embedding models for sesotho sa leboa, with its disjunctive writing style, clearly already perform well with very small data sets. the improvements on the pos tagging task with larger data sets is also not nearly as large as for the other two languages. as with the other languages, the flair models perform the best with very little data, while the fasttext cbow models generate the best overall results with more data. surprisingly, the fasttext skipgram models do not show much improvement between the smallest and largest data sets, and there does not seem to be an easily identifiable reason for this result. as with both other languages, the glove embeddings generally show improvements with each iteration of larger data, and are likely to keep improving if more data were available to be included. for the glove and flair models, the improvements in tag classes are much more moderate, between 8% and 46% for the noun, verb, concord, and adjective classes. both of the flair models also regress on the adverb class. the fasttext models do show more substantial improvements for some of the classes, but for the skipgram model the error reduction in one class is counteracted by an increase in errors in another class. for example, the noun class errors are reduced by 40%, but the adjective and concordial classes increase their errors by more than 40%, resulting in accuracies that are very similar to the model trained on the smallest data sets. generalizing our findings for the type of embeddings to use with little data, the takeaway is that flair models will produce decent results, especially with very little data. with slightly more data, fasttext cbow embeddings will also perform adequately. glove, however, needs large amounts of data to reach enough generalization power to be applied successfully to a morpho-syntactic downstream task. our analysis also shows that the influence of linguistic and orthographic differences between languages should not be underestimated. a language such as isizulu with a complex morphology and large vocabulary (and consequently more data sparseness) will need more data to train representative language models. but a better language model alone is not sufficient. more task related annotated data is also needed to substantially increase the pos accuracy -again an effect of trying to learn from sparse data. it is important to acknowledge the influence of data sparseness in both the learned representations and the actual task to be learned on the final tagging results. ## conclusion and future work in this paper, we investigated how the amount of available training data and the linguistic attributes of a language influence the quality of learned embeddings. our case study consisted of training three different embedding architectures on varying amounts of data, and evaluating the embeddings extrinsically on the downstream task of pos tagging for three linguistically distinct south african languages (afrikaans, isizulu and sesotho sa leboa). our results indicate that under certain conditions even relatively little data can produce useful representations for a specific task. we explicitly show that with very little data (approximately 300,000 tokens) flair embeddings generate representations that perform comparably to any of the other architectures trained on the largest data sets, irre-spective of the morphological complexity of the language. the flair models do not generally show the same level of improvements as the other embedding types when larger data sets are available, and in some cases are out-preformed by the fasttext cbow embeddings with the largest available training sets. the results further reinforce the knowledge that for morphologically complex, conjunctively written languages, substantially more data is needed to achieve good results, not only unannotated text for training the language model, but also annotated training material for the task at hand. overall we conclude that it is imperative to know the characteristics of the language you are working on to make linguistically informed choices about the amount of data and the type of embeddings to use. although these results are encouraging for the relatively simple task of pos tagging, the same may not be true for other, more complex tasks, especially where semantic attributes are of interest. we do however expect that the shortcomings apparent in the fasttext skipgram and glove models will remain in under-resourced settings regardless of the task they are applied to. with this in mind there are two areas for future investigation. firstly, these embedding models should be applied to different tasks that may require different linguistic attributes. secondly, a comparable experimental design should be applied to transformer models, such as roberta, and to fine-tuning multi-lingual language models (e.g. mbert, xlm-r) to determine whether similar encouraging results are possible with these more complex architectures.