Automatic Spell Checker and Correction for Under-represented Spoken Languages: Case Study on W olof (2023)

Abstract: This paper presents a spell checker and correction tool specifically designed for Wolof, an under-represented spoken language in Africa. The proposed spell checker leverages a combination of a trie data structure, dynamic programming, and the weighted Levenshtein distance to generate suggestions for misspelled words. We created novel linguistic resources for Wolof, such as a lexicon and a corpus of misspelled words, using a semi-automatic approach that combines manual and automatic annotation methods. Despite the limited data available for the Wolof language, the spell checker’s performance showed a predictive accuracy of 98.31% and a suggestion accuracy of 93.33%.Our primary focus remains the revitalization and preservation of Wolof as an Indigenous and spoken language in Africa, providing our efforts to develop novel linguistic resources. This work represents a valuable contribution to the growth of computational tools and resources for the Wolof language and provides a strong foundation for future studies in the automatic spell checking and correction field.

## introduction linguistic diversity in natural language processing (nlp) is essential to enable communication between different users and thus the development of linguistic tools that will serve the inclusion of diverse communities. several research studies for low-resource languages have emerged; however spoken indigenous and endangered languages in africa have been neglected, even though the cultural and linguistic richness they contain is inestimable. the wolof language, a popular language in west africa spoken by almost 10 million individuals worldwide, is an immensely popular lingua franca in countries on the african continent, such as senegal, gambia and mauritania. it serves as the pri-mary dialect of senegal @xcite , hailing from the senegambian branch of niger-congo's expansive language family. furthermore, the language has been officially acknowledged in west africa @xcite . it is therefore not surprising that the intensive use of wolof within the region has allowed it to be recognized as being of paramount importance. like several indigenous and spoken languages of africa, wolof presents many challenges and issues, among which is the lack of linguistic resources and tools. moreover, it is distinguished by its distinct tonal system which uses nasal vowels. the wolof script is comprised of a total of 45 consonant phonemes, which are further subdivided into categories @xcite . furthermore, the wolof writing system integrates a set of 17 vowel phonemes @xcite complementing the already existing 45 consonant phonemes. as writing becomes increasingly important due to our digital age, automatic spell checking plays a vital role in making sure written communications are both efficient and accurate. despite the lack of standardization in their orthography, there has vowels short long a, à, ã, i, o, ó, u, e, ë, é ii, uu, éé, óó, ee, oo, aa our main contribution in this paper, is the development of new resources for the wolof language. specifically, we have created a spell checker for the autocorrection of wolof text, as well as a corpus of misspelled words that will enable researchers to evaluate the performance of future autocorrection systems. additionally, we have developed a wolof lexicon that can be leveraged for a range of tasks beyond autocorrection, such as neural machine translation, automatic speech recognition, etc. the resources that have been developed over the course of this study are made publicly accessible on github foot_0 , thereby enabling wider dissemination and facilitating the reproducibility of the research findings. the remainder of our paper is structured as follows: in section 2, we conduct a brief literature review and discuss some published studies. in section 3, we describe our proposed methodology and the novel linguistic resources, we developped. in section 4, we present results and evaluations of our study. in section 5, we show the limitations of our system through an error analysis. finally, section 6 concludes the paper and show some promising perspectives for the future. ## background spelling correction consists in suggesting valid words closer to a wrong one. in order to create an automatic spelling correction system, it is imperative to comprehend the root causes of spelling errors @xcite . several studies about spelling errors have been done, with a notable contribution from @xcite who thoroughly analyzed different types of spelling mistakes for english and described methods to construct an automatic spelling correction system. @xcite , on the other hand, presented a survey on documented findings on spelling error patterns and categorized spelling errors into two groups: the causes of spelling errors are diverse and can stem from both cognitive and typographical sources @xcite . cognitive errors arise when an individual lacks the proper understanding of the correct spelling of a word while typographical errors take place when incorrect keystrokes are made when typing. literature in the field of spelling correction has typically approached these error types separately, with various techniques developed specifically to address each type @xcite . despite the significance of language processing, there has been a shortfall of focus on the creation of automatic spelling correction tools for low resource languages especially. while some attempts have been made to apply standard automatic spelling correction techniques to a few african indigenous languages @xcite @xcite , no such efforts have been made for the wolof language. as far as we are aware, the only research solely dedicated to the correction of wolof is @xcite , which provides an overview of the state of the art and outlines potential solutions for developing a tailored orthographic corrector for wolof. this research adopts commonly used approaches in the field and assesses the performance of our system using various known evaluation metrics. ## methodology the system outlined in this study aims to identify and correct non-word errors in wolof language. to achieve this objective, we designed and implemented the flowchart in figure 1 . text input preprocessing rules validator lexicon check trie dictionary lookup correct word misspelled word word transformation generate suggestions trie + wld rank suggestions correct with first suggestion error detection yes yes no no error correction figure 1: flowchart of the spell checker the flowchart in figure 1 provides a visual representation of the various components and processes involved in the proposed spell checker system. the system includes input and output mechanisms, algorithms for error detection and correction, as well as data structures and models that aid in its functionality. the aim of the system is to accurately identify and rectify non-word errors in the wolof language. through the presentation of the system's overall architecture, the reader will be able to comprehend the workings of the system and its design principles aimed at detecting and correcting invalid words in the wolof language. ## evaluations in order to assess the performance of our spell checking system, we first constructed a corpus of misspelled words, then selected the appropriate evaluation metrics, and finally, we implemented the chosen metrics to assess the performance of the system. ## limitations despite the impressive performance and minimal processing time of our spell checker, it is important to acknowledge its limitations. firstly, the spell checker is restricted to the words included in the created wolof lexicon and cannot recognize words outside of it. secondly, the weighted levenshtein distance algorithm used may not always accurately reflect the likelihood of different types of errors, leading to potential inaccuracies in the suggestions. thirdly, the dynamic programming and trie data structures utilized may result in false positive suggestions due to a lack of consideration for the semantic meaning of words. additionally, the computational cost of our approach can be substantial, particularly for larger lexicons or words with numerous possible corrections. finally, the lack of context awareness may result in missed errors or incorrect suggestions. ## conclusion this paper presented a novel spell checker for the wolof language, that has demonstrated its potential, owing to its effective combination of the trie data structure, dynamic programming, and weighted levenshtein distance algorithms. the hybrid approach of manual and automatic annotation enabled the construction of a comprehensive lexicon and a robust misspelled word corpus, allowing for a robust evaluation of the spell checker's potential despite the limited data available for the language. through these efforts, we hope to advance the state of nlp research for the wolof language and contribute to preserving the linguistic heritage of african nations, ensuring that their distinct cultural expressions are protected for future generations. the findings of this research provide compelling evidence of the viability of the spell checker for the wolof language, opening avenues for further improvement and exploration. for future research, it would be of interest to study the effect of increasing the lexicon and misspelled word corpus on the spell checker's performance. furthermore, a comparison of the spell checker's performance with other spell-checking methods used in low-resource languages, such as the indigenous african languages, could provide valuable insights into the strengths and weaknesses of the current approach. the integration of state-of-the-art techniques,taking into consideration the context, such as those based on machine learning and deep neural networks, into the spell checker could also be explored to further enhance its capabilities.