PALI : A Language Identification Benchmark for P erso- A rabic Scripts (2023)

Abstract: The Perso-Arabic scripts are a family of scripts that are widely adopted and used by various linguistic communities around the globe. Identifying various languages using such scripts is crucial to language technologies and challenging in low-resource setups. As such, this paper sheds light on the challenges of detecting languages using Perso-Arabic scripts, especially in bilingual communities where “unconventional” writing is practiced. To address this, we use a set of supervised techniques to classify sentences into their languages. Building on these, we also propose a hierarchical model that targets clusters of languages that are more often confused by the classifiers. Our experiment results indicate the effectiveness of our solutions.

## introduction historically, the territorial expansion of the arab conquests led to various long-lasting changes in the world, particularly from an ethnolinguistic point of view where the local languages of the time faced existential challenges @xcite . with arabic being the language of administration -a reichssprachemany languages were affected and adapted in many ways such as writing or vocabulary. over centuries the persian language extended the arabic script by adding additional graphemes such as ‫>ﭖ<‬ (<p>, u+067e) and ‫>ﮒ<‬ (<g>, u+06af) to conform to the phonology of the language. hence, one of the main extended variants of the classical arabic script is the perso-arabic script which has been gradually adopted by many other languages to our day, mainly in west, @xcite . some of the languages using a perso-arabic script are urdu, kurdish, pashto, azeri turkish, sindhi, and uyghur, along with many others that historically used the script such as ottoman turkish. this said, there are other scripts that were directly adopted from the arabic script without being affected by the persian modifications such as ajami script used in some african languages like swahili and wolof, pegon and jawi scripts used in southern asia and aljamiado historically used for some european languages. language identification is the task of detecting the language of a text at various levels such as document, sentence and sub-sentence. given the importance of this task in natural language processing (nlp) as in machine translation and information retrieval, it has been extensively studied and is shown to be beneficial to various applications such as sentiment analysis and machine translation @xcite . this task is not equally challenging for all setups and languages, as it has been demonstrated that language identification for shorter texts or languages that are closely related, both linguistically and in writing, is more challenging, e.g. farsi vs. dari or varieties of kurdish @xcite . furthermore, some of the less-resourced languages spoken in bilingual communities face various challenges in writing due to a lack of administrative or educational support for their native language or limited technological tools. these result in textual content written unconventionally, i.e. not according to the conventional script or orthography of the language but relying on that of the administratively "dominant" language. for instance, kashmiri or kurdish are sometimes written in the urdu or persian scripts, respectively, rather than using their adopted perso-arabic orthography. this further complicates the identification of those languages, causing confusion due to the resemblance of scripts and hampers data-driven approaches due to the paucity of data. therefore, reliable language identification of languages using perso-arabic scripts remains a challenge to this day, particularly in under-represented languages. language 639-3 wp script type diacritics zwnj dominant as such, we select several languages that use perso-arabic scripts, summarized in contributions this paper sheds light on language identification for languages written in the perso-arabic script or its variants. we describe collecting data and generating synthetically-noisy sentences using script mapping ( §2). we implement a few classification techniques and propose a hierarchical model approach to resolve confusion between clusters of languages. the proposed approach outperforms other techniques with a macroaverage f 1 that ranges from 0.88 to 0.95 for noisy settings ( §3). ## methodology given that the selected languages are mostly lowresourced, collecting data and, more importantly, identifying text written in a conventional and unconventional way is a formidable task. to tackle this, we focus on collecting data from various sources on the web, notably wikipedia. 2 then, we propose an approach to generate synthetic data that can potentially reflect and model various types of noise that occur in unconventional writing. to that end, we use a simple technique that maps characters from the script of a language to that of another one, i.e. the dominant language. and finally, we discuss our efforts to benchmark this task and propose a hierarchical model that resolves confusion between similar languages. ## results in ## related work modeling approaches language identification is generally modeled as a multi-class text classification task and has achieved state-of-the-art performance with straightforward byte, character or word-level n-gram features across languages and language varieties and in limited data settings @xcite . model or classifier choice is highly dependent on the source, domain and quantity of data per language, with simple linear classifiers like support vector machines @xcite and multinomial naive bayes @xcite providing strong baselines with limited data and compute across domains. if large amounts of data are available, aggregated classifiers @xcite and neural models may be used, but have continued to struggle with similar language varieties and dialects and have been prone to overfitting @xcite . in our paper, we propose a hierarchical approach to language identification that identifies commonly-confusable language pairs in noisy settings and resolves such mispredictions with small classification units. such a modeling approach can be used to expand language coverage and improve the performance of the existing pre-trained models without retraining large compute-hungry models. in our case, we noticed statistically significant improvements for noisy data settings. ## conclusion we focus our study on languages written in bilingual communities where an unconventional dominant perso-arabic script is often utilized in place of a conventional and more suitable perso-arabic variant writing system. we discuss challenges unique to this scenario, in both data collection and language identification, and consequent performance issues in state-of-the-art systems when faced with data in such unconventional writing systems. this is highlighted by the 20-point perfor-mance difference in f 1 scores between noisy and clean/mixed settings. our proposed hierarchical approach outperforms a custom-trained fasttext system, simple mnb and mlp and the state-ofthe-art language identification systems of google's cld3, franc and langid.py. we find statistically significant improvements by using a hierarchical model after analyzing a root multilingual model's confusion matrix.