Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities (2023)

Abstract: The wide accessibility of social media has provided linguistically under-represented communities with an extraordinary opportunity to create content in their native languages. This, however, comes with certain challenges in script normalization, particularly where the speakers of a language in a bilingual community rely on another script or orthography to write their native language. This paper addresses the problem of script normalization for several such languages that are mainly written in a Perso-Arabic script. Using synthetic data with various levels of noise and a transformer-based model, we demonstrate that the problem can be effectively remediated. We conduct a small-scale evaluation of real data as well. Our experiments indicate that script normalization is also beneficial to improve the performance of downstream tasks such as machine translation and language identification.

## introduction with the increasing accessibility of the internet around the globe, from nomad settlements to megacities, we are living in a critical era that gives voice to different language communities to be represented online. text as the essential material of many applications in natural language processing (nlp) may not be consistently created by under-represented communities due to the lack of a writing system and/or tradition, a widely-used orthography, and reasons related to regional and geopolitical conditions. moreover, speakers of a language may opt for another script or prefer to use the script of an administratively dominant language other than the conventional one used in their native language (gathercole and @xcite . this can be due to illiteracy in the native or home language, diglossia, or simply, the lack of adequate support for specific languages and scripts re- despite the fascinating progress in language technology in recent years, there are many fundamental tasks in nlp that are usually deemed trivial yet are ubiquitous, needed, and still unsolved in low-resource settings. one of these tasks is to deal with noisy text which affects the quality and performance of various applications such as text mining @xcite and machine translation @xcite . in practice, some steps are generally taken to deal with inconsistencies in the text, such as unifying character encoding and rectifying orthographic rules -a task broadly referred to as data cleaning or text preprocessing @xcite . however, unconventional and unsystematic writing such as using the script of a language to write in another language presents challenges beyond traditional data cleaning scenarios. in this vein, we address the task of script normalization which aims to normalize a text writ although this task has much in common with data cleaning, transliteration @xcite ), text normalization as defined by @xcite and spelling error correction @xcite , it is particular in a few ways: unconventional writing does not necessarily comply with orthographic rules of the text's language; when writing borrowed or common words from the dominant language, there is an influence of the orthography of the donor language rather than writing in the recipient language's script or orthography; phonemes and graphemes may not be represented according to any specific rule, as in writing /s/ as 'ch', 'sh' or '$' in arabizi @xcite following an erratic or obscure pattern among speakers leading to a huge amount of noisy material. a few examples of unconventional writing in kurdish, kashmiri and gilaki are shown in in this paper, we focus on the script normalization of a few under-resourced languages that use variants of the perso-arabic script, as schematized in figure 1 , and are spoken as minority languages in bilingual communities with a dominant language using a similar but different script. these languages are azeri turkish (azb), mazanderani (mzn), gilaki (glk), sorani kurdish (ckb), kurmanji kurdish (kmr), gorani (hac), kashmiri (kas) and sindhi (snd). although these languages have their own customized scripts with more or less defined orthographies in their communities, they are oftentimes written in the script of the dominant language, notably persian (fas), arabic (arb) and urdu (urd) scripts. furthermore, these languages have been lexically and, to a lesser extent, typologically influenced by the administratively dominant languages. akin to many other multilingual and pluricultural societies, speakers of these languages have faced language conflict and linguistic discrimination in different educational, political, cultural, and communicative domains, and struggle with ethnolinguistic vitality @xcite . appendix a presents a summary of the languages we study. ## related work although some aspects of script normalization have been previously addressed in related nlp tasks, its definition is a rather subjective matter, where a set of intentional or unintentional anoma-lies are "normalized". therefore, script normalization overlaps considerably with more well-defined tasks such as spelling correction, lexical normalization @xcite where an utterance is transformed into its standard form, pattern recognition @xcite , language identification and standardization @xcite . script and text normalization have been proven beneficial in various downstream applications such as dependency parsing @xcite , sentiment analysis @xcite and named-entity recognition @xcite . although in some contexts normalization has been used to refer to basic tasks such as stemming and lemmatization, as in @xcite , those are not within the scope of this paper. text normalization one of the most related tasks to script normalization is text normalization which broadly deals with alternative spellings, typos, abbreviations and non-canonical language and is of importance to text-to-speech systems and for handling micro-blogging data such as tweets @xcite . to this end, a wide range of techniques have been proposed using rules based on annotated corpora @xcite or linguistic information @xcite , edit operations and recurrent neural networks @xcite , machine translation @xcite , supervised learning @xcite , encoder-decoders @xcite and more recently, transformers @xcite . monoise (van der goot and van noord, 2017) is a prominent approach to text normalization where the problem is framed as a domain adaptation one and various steps are taken to generate and rank normalized candidates using spell checkers, word embeddings, dictionaries and n-grams features. ## methodology this section presents our methodology to collect data, create script mapping to generate synthetic data and implement a transformer-based model. source and dominant in this context respectively refer to the language of the original text and that of the dominant one used unconventionally. ## conclusion and discussion this paper discusses the challenge of script normalization of unconventional writing of languages that are spoken in bilingual communities. under the influence of the dominant language of such communities, speakers tend to use scripts or orthographies of the dominant language rather than the one that is conventionally used in their native language. this leads to noisy data that hinder nlp applications and reduce data availability. framing the problem as a machine translation one where noisy data is 'translated' into clean one, we address the task of script normalization for the perso-arabic scripts of several languages, namely azeri turkish, mazanderani, gilaki, sorani and kurmanji kurdish, gorani, kashmiri and sindhi, with dominant languages being arabic, persian and urdu. given that these languages are lessresourced, we rely on synthetic data to create parallel datasets by injecting noise based on script mapping. we then train transformer-based encoderdecoders and show that the models can tackle the problem effectively, even for real data, but differently based on the level of noise. we demonstrate that script normalization can be effectively addressed, particularly where there are many dissimilarities between the source and dominant languages. it can also alleviate the impact of noisy data on downstream tasks, namely language identification and machine translation. in addition to script normalization that can help retrieve texts for the selected under-resourced languages, the trained models along with the collected data, with or without noise, can pave the way for further developments for those languages. limitations one of the limitations of the current study is the lack of annotated data for all languages. this is also the case of machine translation for which data could only be found for kashmiri, sorani and sindhi, while other languages do not have much parallel data yet. on the other hand, the notion of noisy data is limited to the replacement of the missing characters in a script when compared to another one, i.e. that of the dominant language. as an ablation study, injecting other types of noise, beyond those discussed in this paper, may improve the performance of the models to tackle not only script normalization but several related tasks such as spelling error correction and may also increase the robustness of the models for morphologically rich languages or languages with versatile word boundaries using zwnj. although we did our best to filter out code-switched data in the corpora, our datasets may contain data in other languages (in perso-arabic scripts). in future work, we would like to apply our approach to other scripts and languages in bilingual communities. we also suggest evaluating the impact of script normalization on more downstream tasks, especially transliteration and tokenization.