Script-Agnosticism and its Impact on Language Identification for D ravidian Languages (2025)

Abstract: Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets. However, many modern languages, such as Konkani, Kashmiri, Punjabi etc., are synchronically written in several scripts. Moreover, languages with different writing systems do not share significant lexical, semantic, and syntactic properties in neural representation spaces, which is a disadvantage for closely related languages and low-resource languages, especially those from the Indian Subcontinent. To counter this, we propose learning script-agnostic representations using several different experimental strategies (upscaling, flattening, and script mixing) focusing on four major Dravidian languages (Tamil, Telugu, Kannada, and Malayalam). We find that word-level script randomization and exposure to a language written in multiple scripts is extremely valuable for downstream script-agnostic language identification, while also maintaining competitive performance on naturally occurring text.

## introduction in many natural language processing (nlp) tasks or data creation efforts, we often need to first identify the source language of a particular text. for instance, automated translation, part-of-speech (pos) tagging, and web scraping for data collection must typically identify the text's language before performing the given task. the languages involved might occur in non-standard scripts, but as we show in this paper, modern systems are heavily script-dependent in language identification (langid). the result is that most current methods are unable to account for languages written in nonstandard scripts. moreover, script diversity is especially common in low-resource languages. many bilingual communities choose to write their minority language in the region's dominant system (such as those in pakistan, iran, china), instead of their language's traditional writing system @xcite . moreover, some languages simply do not possess one standard script, and are written in multiple writing systems. for instance, the western-indian konkani language is actively written in up to 5 scripts: devanagari, romi, @xcite @xcite . however, most konkani systems only support devanagari and romi scripts, and would not recognize the language if written in the other three. this illustrates the need to have script-agnosticism so we can support scriptdiverse usecases in nlp applications. while previous work has explored romanized langid for indian texts, these are limited to informal contexts, and don't capture the full script-diversity of indian languages. script-agnosticism is expected to be useful for closely related languages that use the same script or with languages that have unique scripts but share lexical information with others -a scenario most common to the indian subcontinent. additionally, it can be useful for applications like large language models that may receive transliterated input in non-traditional scripts. in this paper, we conduct a case study on script-agnosticism, focusing on the langid task, on four languages: tamil, telugu, kannada, and malayalam. we explore three different embedding training methods, evaluate on the langid task across domains, and offer insights for future work. broadly, we attempt to answer the following research questions: ## results we present our results for the baseline, flattening, upscaling, and noisy models here. in general, our script-agnostic models demonstrate good performance above the baselines on the transliterated test sets, and our methods comparable to traditional approaches on clean data. ## discussion the results demonstrate that all of our scriptagnostic language identification models (flattening, noise, and scipt-upscaled) perform well above the baselines on examples that utilize a nonstandard script. in certain cases where data is in native script, our baseline models can surpass some script-agnostic ones; this is likely because the baselines use script as a basis for determining language id. the all-noise model showed very good performance, and we suspect it remains second to the upscaled setting primarily due to the variability of the training data. unlike the upscaled model, it may not see every example transliterated to all scripts, and thus may not become completely agnostic of script. however, it is a strong contender and its performance on other downstream tasks and the quality of its learned representations should be evaluated in future work. in the practical setting, our models, especially script-upscaled, appear to be a reasonable alternative to current language identification systems. additionally, it is likely that had we trained an upscaled model on wikipedia, we would have seen results that matched the wiki baseline on noiseless data. this should be the next step for future research, in addition to exploring transliteration in more indian languages and scripts. since transliteration can be done automatically and cheaply, our final proposal should scale well to other indic languages or languages that use indian writing systems that share high phonemic/graphemic overlap, but researchers should not expect this approach to scale to languages outside the indian subcontinent, as highly overlapping writing systems is a feature unique to indic scripts. for instance, languages such as konkani, tulu, kodava, and badaga in southern india, are all synchronically written in several writing systems by different speaker communities. like most indian scripts, there is a high degree of interchangeability (limited to no loss during transliteration) for these languages' scripts. therefore, our case study could also easily be extended to this real-world setting and to this new set of languages. however, konkani, tulu, kodava, and badaga might have a lot less data to train on, compared to tamil, malayalam, telugu, and kannada, researchers may need to explore upsampling and data augmentation techniques to achieve reasonable downstream task performance. ## related work previous work has demonstrated that script barriers discourage transfer learning from highresource languages into low-resource languages' representation spaces, especially for neural machine translation @xcite . moreover, script diversity negatively impacts low-resource languages disproportionately because their training data is often of poor quality and smaller in size @xcite . consequently, researchers have focused on transliteration, romanization, phonetic representation etc. to reduce vocabulary sizes and allow lexical sharing between languages with different writing systems @xcite . most recently, due to the presence of romanized texts on the web, romanized language identification approaches for indian languages have also been explored with some success @xcite . another common approach relies on existing pre-trained models and finetuning them with different transliterated versions of the originally supported languages @xcite . this is an instance of the common hierarchical pipeline (goutte et al., @xcite or finetuning-based approach for language identification @xcite . identification is also critical in digitization of non-machine-readable documents and can help produce new corpora for researchers at scale @xcite . most recently, moosa et al. ( 2023 ) conducted a study on effects of transliteration on multilingual language modeling, which focused on two kinds of models: a multi-script model with native scripts of each language (matching our baseline setup) and a uni-script model with only one script for all languages (similar to our flatten setup). as a natural extension of their work, we also consider upscale and noise setups for dravidian languages, as described in §3. unlike their work, we do not finetune on downstream tasks, but instead focus on including the transliteration in the original training data to give the model the ability to handle nonnative scripts without losing performance on the original script. moreover, our work is not only motivated from a lexical-sharing and transfer-learning perspective, but is grounded with the aim of supporting synchronic and diachronic digraphia adequately in nlp applications and tasks. ## conclusion we introduce and evaluate three new kinds of language identification models that are scriptagnostic. all of our systems have been shown to outperform the baseline on examples that are not written in the standard script. two of our models (upscaled and all-noise) perform especially well on both clean and transliterated data. our methods may provide a reasonable alternative to training language identifiers that can correctly classify text based on the language used, rather than the script in which it is written. future work should expand to include more languages and scripts, as well as performing thorough intrinsic evaluation on the learned embeddings to determine if these would be effective on other downstream tasks.