  Language models are the foundation of current neural network-based models for
natural language understanding and generation. However, research on the
intrinsic performance of language models on African languages has been
extremely limited, which is made more challenging by the lack of large or
standardised training and evaluation sets that exist for English and other
high-resource languages. In this paper, we evaluate the performance of
open-vocabulary language models on low-resource South African languages, using
byte-pair encoding to handle the rich morphology of these languages. We
evaluate different variants of n-gram models, feedforward neural networks,
recurrent neural networks (RNNs), and Transformers on small-scale datasets.
Overall, well-regularized RNNs give the best performance across two isiZulu and
one Sepedi datasets. Multilingual training further improves performance on
these datasets. We hope that this research will open new avenues for research
into multilingual and low-resource language modelling for African languages.
