Beyond A rabic: Software for P erso- A rabic Script Manipulation (2022)

Abstract: This paper presents an open-source software library that provides a set of finite-state transducer (FST) components and corresponding utilities for manipulating the writing systems of languages that use the Perso-Arabic script. The operations include various levels of script normalization, including visual invariance-preserving operations that subsume and go beyond the standard Unicode normalization forms, as well as transformations that modify the visual appearance of characters in accordance with the regional orthographies for eleven contemporary languages from diverse language families. The library also provides simple FST-based romanization and transliteration. We additionally attempt to formalize the typology of Perso-Arabic characters by providing one-to-many mappings from Unicode code points to the languages that use them. While our work focuses on the Arabic script diaspora rather than Arabic itself, this approach could be adopted for any language that uses the Arabic script, thus providing a unified framework for treating a script family used by close to a billion people.

## introduction while originally developed for recording arabic, the perso-arabic script has gradually become one of the most widely used modern scripts. throughout history the script was adapted to record many languages from diverse language families, with scores of adaptations still active today. this flexibility is partly due to the core features of the script itself which over the time evolved from a purely consonantal script to include a productive system of diacritics for representing long vowels and optional marking of short vowels and phonological processes such as gemination @xcite . consequently, many languages productively evolved their own adaptation of the * on contract from optimum solutions, inc. perso-arabic script to better suit their phonology by not only augmenting the set of diacritics but also introducing new consonant shapes. this paper presents an open-source software library designed to deal with the ambiguities and inconsistencies that result from representing various regional perso-arabic adaptations in digital media. some of these issues are due to the unicode standard itself, where a perso-arabic character can often be represented in more than one way (unicode @xcite . others are due to the lack or inadequacies of input methods and the instability of modern orthographies for the languages in question @xcite . such issues percolate through the data available online, such as wikipedia and common crawl @xcite , negatively impacting the quality of nlp models built with such data. the script normalization software described below goes beyond the standard language-agnostic unicode approach for perso-arabic to help alleviate some of these issues. the library design is inspired by and consistent with prior work by @xcite , introduced in §2, who provided a suite of finitestate grammars for various normalization and (reversible) romanization operations for the brahmic family of scripts. foot_0 while the perso-arabic script and the respective set of regional orthographies we support -balochi, kashmiri, kurdish (sorani), malay (jawi), pashto, persian, punjabi (shahmukhi), sindhi, south azerbaijani, urdu and uyghur -is significantly different from those brahmic scripts, we pursue a similar finite-state interpretation, foot_1 as described in §3. implementation details and simple validation are provided in §4. ## related work the approach we take in this paper follows in spirit the work of @xcite and @xcite , who developed a finite-state script normalization framework for brahmic scripts. we adopt their taxonomy and terminology of lowlevel script normalization operations, which consist of three types: unicode-endorsed schemes, such as nfc; further visually-invariant transformations (visual normalization); and transformations that modify a character's shape but preserve pronunciation and the overall word identity (reading normalization). the literature on perso-arabic script normalization for languages we cover in this paper is scarce. the most relevant work was carried out by @xcite for kurdish, who provides a detailed analysis of orthographic issues peculiar to sorani kurdish along with corresponding open-source script normalization software used in downstream nlp applications, such as neural machine translation @xcite . in the context of machine transliteration and spell checking, lehal and saini (2014) included language-agnostic minimal script normalization as a preprocessing step in their open-source n-gram-based transliterator from perso-arabic to brahmic scripts. @xcite introduced a taxonomy of spelling errors for sindhi, including an analysis of mistakes due to visually confusable characters. @xcite provide a good overview of confusable characters for malay jawi orthography. for other languages the regional writing system ambiguities are sometimes mentioned in passing, but do not constitute the main focus of work, as is the case with punjabi shahmukhi @xcite and urdu @xcite . the specific perso-arabic script ambiguities that abound in the online data are often not exhaustively documented, particularly in work focused on multilingual modeling (n. c., 2022; @xcite . as one moves towards lesser-resourced languages, such as kashmiri and uyghur, the nlp literature provides no treatment of script normalization issues and the only reliable sources of information are the proposal and discussion documents from the unicode technical committee (e.g., @xcite @xcite . a forthcoming paper by @xcite covers the writing system differences between these languages in more detail than we can include in this short paper. one area particularly relevant to this study is the work by the internet corporation for assigned names and numbers (icann) towards developing a robust set of standards for representing various internet entities in perso-arabic script, such as domain names in urls. their particular focus is on variants, which are characters that are visually confusable due to identical appearance but different encoding, due to similarity in shape or due to common alternate spellings @xcite . in addition, they developed the first proposal to systematize the available perso-arabic unicode code points along the regional lines @xcite . these studies are particularly important for cybersecurity @xcite @xcite , but also inform this work. this software library is, to the best our knowledge, the first attempt to provide a principled approach to perso-arabic script normalization for multiple languages, for downstream nlp applications and beyond. ## design methodology the core components are implemented as individual fsts that can be efficiently combined together in a single pipeline @xcite . these are shown in as an example of a first type, consider the alef with madda above letter ‫⟩آ⟨‬ that can be composed in two ways: as a single character (u+0622) or by adjoining maddah above to alef ({ u+0627, u+0653 }). the fst n rewrites the adjoined form into its equivalent composed form. the second type of transformation involves the canonical reordering of the arabic combining marks, for example, the sequence of shadda (u+0651) followed by kasra (u+0650) is reversed by n . more complex transformations that combine both compositions and re-orderings are possible. for example, the sequence { alef (u+0627), superscript alef (u+0670), maddah above (u+0653) } normalizes to its equivalent form { alef with madda above (u+0622), superscript alef (u+0670) }. crucially, n is language-agnostic because the nfc standard it implements does not define any transformations that violate the writing system rules of respective languages. visual normalization as mentioned in §2, @xcite introduced the term visual normalization in the context of brahmic scripts to denote visually-invariant transformations that fall outside the scope of nfc. we adopt their definition for perso-arabic, implementing it as a single language-dependent fst v, shown in the second set of visually-invariant transformations, denoted v l , is language-specific and additionally depends on the position within the word. four special cases are distinguished that are represented as fsts: position-independent rewrites (v * l ), isolated-letter rewrites (v i l ), rewrites in the word-final position (v f l ), and finally, rewrites in "non-final" word positions, which include visuallyidentical word-initial and word-medial rewrites ( some examples of these transformations for urdu orthography are shown in reading normalization this type of normalization was introduced for brahmic scripts by @xcite , who noted that regional orthographic conventions or lack thereof, which oftentimes conflict with each other, benefit from normalization to some accepted form. whenever such normalization preserves visual invariance, it falls under the rubric of visual normalization, but other cases belong to reading normalization, denoted r in for efficiency reasons r is stored independently of visual normalization v. at run-time, the reading normalization is applied to an input string s as s ## software details and validation our software library is implemented using pynini, a python library for constructing finite-state grammars and for performing operations on fsts @xcite . each fst is compiled from the collections of individual context-dependent letter rewrite rules @xcite and is available in two versions: over an alphabet of utf-8 encoded bytes and over the integer unicode code points. the fsts are stored uncompressed in binary fst archives (fars) in openfst format @xcite . the summaries of language-agnostic and language-dependent fsts over utf-8 strings are shown in example the use of the library is demonstrated by the following python example that implements a simple command-line utility for performing reading normalization on a single string using pynini apis. the program requires two far files that lang. input output correct output bal ‫دﺋﯿٽ‬ ‫دﺋﯿﺖ‬ teh ckb ‫ﻟەﺷڪﺮ‬ ‫ﻟەﺷﮑﺮ‬ keheh fa ‫ﻣﺆﺳﺴﻪ‬ ‫ﻣﻮﺳﺴﻪ‬ waw ks ‫ﮬۍﺗﮏ‬ ‫ﮬؠﺘﮏ‬ kashmiri yeh pa ‫ﮐﺌﻲ‬ ‫ﮐﺌﯽ‬ farsi yeh sd ‫ﮘﻮﻫﻪ‬ ‫ﮘﻮﮨﮧ‬ heh goal ug ‫ﺳﺎی‬ ‫ﺳﺎي‬ yeh ur ‫ﺻﻮرة‬ ‫ﺻﻮرۃ‬ teh marbuta goal fstoperror as error: raise valueerror(f"error for string `{text}`") def main(argv: sequence[str]) -> none: # ... initializing flags @xmath1.visual_grm, flags.lang) @xmath2.reading_grm, flags.lang) @xmath3.input, [visual_fst, reading_fst]) print(f"=> {out}") if __name__ == "__main__": app.run(main) the visual and reading fsts for a given language are retrieved from the relevant far files using load_fst function. the input string is first converted to a linear fst. the visual and reading normalization fsts are then sequentially composed with the input fst and a shortest path algorithm is applied on the result, which is then converted from a linear fst back to a python string in apply function to yield the final normalized output. some examples of reading normalization pro-duced using the example.py utility above for some of the supported languages are shown in ## conclusion and future work we have presented a flexible fst-based software package for low-level processing of orthographies based on perso-arabic script. we described the main components of the architecture consisting of various script normalization operations, romanization/transliteration, and character-language index. we expect to increase the current language coverage of eleven languages to further relatively well-documented orthographies, but also provide treatment for resource-scarce orthographies, such as the ajami orthographies of sub-saharan africa @xcite .