Spellchecker for S anskrit:The Road Less Taken (2022)

Abstract: A spellchecker is essential for any language for producing error-free content. While there exist advanced computational tools for Sanskrit, such as word segmenter, morphological analyser, sentential parser, and machine translation, a fully functional spellchecker is not available. This paper presents a Sanskrit spellchecking dictionary for Hunspell, thereby creating a spellchecker that works across the numerous platforms Hunspell supports. The spellchecking rules are created based on the Paninian grammar, and the dictionary design follows the word-and-paradigm model, thus, making it easily extendible for future improvements. The paper also presents an online spellchecking interface for Sanskrit developed mainly for the platforms where Hunspell integration is not available yet.

## introduction a spellchecker is a program that checks for misspellings in a text and suggests correct alternatives @xcite . it is an essential tool for word processing and document preparation. in the recent decade, digitization of sanskrit manuscripts and using digital technologies for editing and publishing new content in sanskrit have seen a tremendous increase. online sanskrit communities and many individuals are actively using sanskrit in writing blogs, emails and chat messages, while some are designing posters in sanskrit. a spellchecker can help them to communicate clearly and produce error-free content. after the digitization of a manuscript through an ocr (optical character recognition), often there might be errors in the text due to the visual similarity in certain letters such as बव and यथ @xcite . these can be corrected using a spellchecker manually or through a pipeline that performs the ocr post-correction automatically. also, advanced tools like morphological analyser @xcite and sentential parser @xcite might produce incorrect results or no results at all, if the input contains spelling errors @xcite . in such cases, the input can be pipelined through a spellchecker for prechecking and pre-processing. a spellchecker can also sometimes help new learners of sanskrit in learning the correct spellings of words, especially, of those having phonetically similar letters such as शष in "शे ष", एये in "एक", नण in "बाणे न", etc. this paper presents a sanskrit spellchecking dictionary for hunspell based on the wordand-paradigm model and describes its design and implementation. it discusses the suitability of hunspell for a highly inflectional language like sanskrit (section 3) and the format of a hunspell dictionary (section 4). it also describes the framing of spellchecking rules based on paninian grammar (section 5). the paper also presents a web interface for sanskrit spellchecking (section 6). later, it discusses the evaluation of both the spellchecking dictionary and the web interface (section 7). the paper also records the challenges unique to sanskrit in creating a spellchecking dictionary (section 8). ## related works and previous attempts while there exist advanced computational tools for sanskrit such as word segmenter @xcite , morphological analyser @xcite , sentential parser @xcite , machine translation @xcite , automatic speech recognition @xcite , etc., a fully func-tional spellchecker is not available. spellcheckers and grammar checkers are fundamental tools that users need and expect nowadays with the increase in the use of digital technologies for both creating new content as well as digitising existing texts. significant work has been done on spellcheckers for other indian languages like hindi @xcite @xcite , marathi @xcite , odia @xcite , punjabi @xcite , kashmiri @xcite , telugu @xcite , tamil @xcite and @xcite . though there is not much research work available on spellchecker for sanskrit, there were a few attempts in the past to develop one. @xcite proposed and developed a spellchecker based on the morphological rules of sanskrit. it was a standalone application implemented in java. samsādhanī foot_0 developed and hosted a spellchecker web application where their morphological analyser runs on the text provided by a user and highlights incorrect words. the application, however, is no longer maintained foot_1 . patel (2016) built a sanskrit spellchecker that works based on different vowel and consonant patterns. but it was specific to the cologne sanskrit dictionaries, and not for general spellchecking @xcite . @xcite and @xcite independently created sanskrit dictionaries for hunspell. but, as of now, both the dictionaries are not complete and are not maintained. @xcite developed a dictionary add-on for mozilla firefox which contains a sanskrit hunspell dictionary. however, it is also far from complete. other than these, to the best of our knowledge, there has been no significant progress in the development of a sanskrit spellchecking dictionary for hunspell or any other spellchecker for sanskrit. ## why hunspell? hunspell is a free open-source spellchecker and morphological analyser library and also a command-line tool foot_2 . it is the most popular spellchecker that is used in many applications like libreoffice foot_3 , mozilla firefox foot_4 , google chrome foot_5 and adobe indesign foot_6 and has bindings in numerous popular programming languages @xcite . in linux and macos, after installing the dictionaries the users can enjoy system-wide spellchecking. such tight integration helps correct misspellings even as the user is writing instead of having them copy and paste the text into an external program just for spellchecking. therefore, designing a dictionary for hunspell means we can have sanskrit spellchecking on almost all the platforms. sanskrit is a highly inflectional language and highly productive in derivative morphology such as samāsa, kṛdanta and taddhita @xcite . a simple list of all the correct forms would be extremely huge in terms of computer storage. hunspell format allows us to define the base forms and affixes separately which hugely reduces the dictionary size. this division also makes it easier to add new words to the dictionary. ## format of hunspell dictionary files before going into the design and preparation of the hunspell dictionary, the format of the dictionary files is briefly discussed here. hunspell requires two files to define how it should spellcheck for a language and suggest correct alternatives foot_7 -a dictionary file and an affix file. the first line of the dictionary file contains the (approximate) count of the number of entries in the file @xcite the second line onwards, we have the entries, one per line. an entry can be a morpheme or lexeme or can even be a pair of words, and it can be optionally followed by a forward slash ("/") and one or more "flags" which represent its attributes such as suffix, prefix, etc. the definitions of these flags are given in the affix file. here, `sfx 1001 y 17`states that this is a suffix class with paradigm type "1001" which has 17 rules, and these rules can be used even if the entry has prefixes. the fields in the affix rules give the information on the characters to strip from the word, the affix to add to the word and the condition. for example, the first affix rule in in the dictionary file, the affix flag "1001" is assigned to the word "आिशस ् " ( the affix file is not only for defining affix rules. it is also used for containing a lot of options that help improve hunspell's spellchecking process for the language which would be discussed later (sections 5.2 and 5.3). in the following section, we describe the preparation of the dictionary and affix files. ## design and preparation of the dictionary for designing the dictionary, we follow the word and paradigm model, which is computationally easy to work with. we prepare the dictionary entries and affix rules for nouns and verbs using two different methods, as we would describe in the next section (section 5.1). sections 5.2 and 5.3 discuss the use of some of hunspell's options for improving the suggestions and handling optional characters. ## web interface a web interface foot_10 is developed using the spellchecking dictionary and hunspell, mainly, for supporting platforms such as android and iphone where hunspell integration is currently not available. however, it can be used in desktop browsers also. sanskrit text in devanagari script can be typed in or pasted into the editor. the editor marks the incorrect words with a red underline. if an underlined word is clicked, a pop-up window opens next to the word showing a list of correct suggestions (figure 3 ). when a suggestion is clicked, the incorrect word is replaced with the suggested word and the pop-up closes. ## evaluation after the hunspell dictionary was prepared, all the forms of the word paradigms were tested against it with the help of nodehun foot_11 , a node.js binding for the hunspell library, and mocha foot_12 , an automated javascript testing framework. in the test, all the forms were recognised as correct spellings. to evaluate the spellchecking dictionary, a test corpus was prepared from three random ocr-ed pages of śrīmad vālmīki rāmāyaṇa from the sanskrit wikipedia foot_13 . the sandhis and samāsas in the corpus were manually split since the dictionary does not support such words. the corpus consisted of 751 unique words. the dictionary was added to hunspell and, using a script written in python, each word in the corpus was tested with the hunspell spellchecking interface. the hunspell dictionaries prepared by @xcite and @xcite were also tested in a similar manner for comparison. the results were then manually analysed. in the case of (a), all the words declared as correct were also actually correct implying a figure 3 : a screenshot of the web interface. all the words marked with * are incorrect. the editor correctly marks them with a red underline. for the incorrect word "सीतायः", the correct form "सीतायाः" is suggested. 100% precision. a few real-word errors were observed but they were not considered for this evaluation as hunspell cannot handle such errors. of the words reported as incorrect, a large percentage were wrongly considered incorrect (false negatives). the reasons were mainly traced to incomplete vocabulary and incorrect split of sandhi and samāsa. since the dictionary is still in development false negatives due to out-of-vocabulary words are expected. moreover, for better evaluation and for increasing the accuracy of suggestions, we are in the process of creating a larger test corpus. the web interface was also manually tested using random words and sentences. the editor correctly marked the misspellings and also suggested correct and proper alternatives for them (figure 3 ). ## observations some of the observations noted while preparing and evaluating the spellchecking dictionary are discussed here. words with anusvāra (◌ं ) such as सं भमः and असं बाधम ् optionally undergo homo-organic nasalisation and become समः and असबाधम ् , respectively. both the forms are frequently used alternatively in the literature. so both forms had to be added to the dictionary. also, the letter म ् at the end of word is replaced by anusvāra if it is followed by a consonant foot_16 . so, for suffixes ending with म ् ## conclusions and future work the paper proposed and described the design and preparation of a hunspell dictionary for sanskrit. it also discussed the advantages of the paradigm-based approach which was followed for the generation of the dictionary. the proposed dictionary is grammar-based unlike that of @xcite and is general-purpose, that is, it can be used for correcting both typing errors and ocr errors. the paper also presented an online spellchecking interface for sanskrit. the current limitations of the dictionary and future improvements to be made are discussed below.