Distinguishing R omanized H indi from R omanized U rdu (2023)

Abstract: We examine the task of distinguishing between Hindi and Urdu when those languages are romanized, i.e., written in the Latin script. Both languages are widely informally romanized, and to the extent that they are identified in the Latin script by language identification systems, they are typically conflated. In the absence of large labeled collections of such text, we consider methods for generating training data. Beginning with a small set of seed words, each of which are strongly indicative of one of the languages versus the other, we prompt a pretrained large language model (LLM) to generate romanized text. Treating text generated from an Urdu prompt as one class and text generated from a Hindi prompt as the other class, we build a binary language identification (LangID) classifier. We demonstrate that the resulting classifier distinguishes manually romanized Urdu Wikipedia text from manually romanized Hindi Wikipedia text far better than chance. We use this classifier to estimate the prevalence of Urdu in a large collection of text labeled as romanized Hindi that has been used to train large language models. These techniques can be applied to bootstrap classifiers in other cases where a dataset is known to contain multiple distinct but related classes, such as different dialects of the same language, but for which labels cannot easily be obtained.

## introduction hindi and urdu are considered the two standardized registers of the pluricentric hindustani language. in informal speech, they are highly mutually intelligible, to the point where it can be difficult to immediately assess which one is being spoken @xcite . written (and more formal) hindi and urdu, however, have more noticeable differences. first, outside of the colloquial vocabulary commonly used in speech, they do differ in broad historical influence on the lexicon -hindi making use of more sanskrit-derived words and urdu using more arabic-or persian-derived words. most notably, however, the languages differ in their native scripts: hindi is written in devanagari, a brahmic script, while urdu is written in a perso-arabic script. despite these stark differences, efforts have been made to unify linguistic resources for the languages (e.g., @xcite @xcite . additionally, however, both languages are frequently written informally in the latin script, which is known as romanization @xcite . informal romanization makes text in these languages far more difficult to distinguish than when they are written in their distinct native scripts. despite their overall linguistic similarity, hindi and urdu do represent different cultural contexts, and may have different patterns of expression that are useful to capture correctly. predictive models in service of, for example, romanized text entry -perhaps providing next word prediction and other utilities that should match the user's desired language -will be expected to provide culturally appropriate predictions, which may be difficult if all urdu and hindi data is conflated in the training data. in general, given the larger number of speakers, romanized hindi text may be more prevalent and thus yield degraded performance for urdu speakers in a range of applications that process romanized text if the two languages are conflated. in this preliminary study, we look at leveraging multilingual large language models (llms) that have been pretrained on data that includes (conflated) romanized hindi and urdu text, along with a small seed word list, to gen-erate training data that can capture characteristic differences between romanized urdu and hindi. llms have recently become the backbone of many state-of-the-art nlp systems performing a wide range of tasks, often after some amount of fine-tuning (see, e.g., @xcite for multilingual task benchmarks). in a recent paper, @xcite demonstrated that large language models learn some degree of long-distance sensitivity to spelling convention differences in english -i.e., the t5 llm @xcite is more likely to produce the british spelling of a word following an earlier instance of british spelling than otherwise, even though the english language pretraining data is not labeled with the particular spelling convention. unlike the wellunderstood and conventional set of spelling differences distinguishing us and uk english, romanized hindi and urdu represent a case where (a) there is no fixed orthography, i.e., spelling varies heavily; and (b) as far as we know, there are no documented widely attested differing romanization conventions between the two languages to rely upon. we thus try to exploit any implicit knowledge about such differences that a pretrained llm may contain, as the means to build systems that can distinguish between the two languages. we demonstrate that a simple decision tree classifier using character n-gram features can be profitably trained on llm generated text to distinguish romanized hindi from romanized urdu, even in the face of domainmismatch, at nearly the same accuracy as that classifier's topline (i.e., when trained on domain-and annotator-matched data). interestingly, a more powerful neural classifier, which yields a substantially higher topline accuracy, overfits on the llm generated training data to the point of performing essentially at chance on the validation set, suggesting that the neural classifier relies too heavily on reliable yet spurious differences between the classes in the generated text. we use the resulting decision tree classifiers to estimate the prevalence of urdu in the mc4 corpus, and also examine their most important features, yielding some potentially useful generalizations about the romanization tendencies in the two languages. ## datasets our work makes use of three independent data sources, including a training/validation set derived from wikipedia, a general web-scraped text collection labeled in part as being romanized hindi, and a set of hindi and urdu language-indicating seed words. dakshina while most relevant datasets do not distinguish between romanized hindi and romanized urdu, the dakshina corpus foot_3 @xcite does distinguish between the two. it contains hand-romanized sentences (10k per language) taken from hindi and urdu wikipedia articles. this makes it ideal for evaluating our language id system. we split the hindi and urdu portions of the corpus into training, development, and test sets, foot_4 and we use the development set (965 sentences from each language) to evaluate all versions of our language id system. we also use the training set in one of our baselines (see section 4.1). all of the training and validation sets are balanced between the two languages. seed words consulting with professional linguists who are familiar with both hindi and urdu, we collected 147 hindi/urdu pairs of words that differ between the two languages, but otherwise share the same meaning and are used in mostly the same semantic contexts. these were elicited by asking for words that, if seen in romanized text, would be strongly indicative of either urdu or hindi. the seed words were provided in the native scripts of hindi and urdu along with common romanizations for those words. ## methods in this paper, we focus on comparing different sources of training data for distinguishing romanized hindi and urdu, rather than developing new classification architectures. as such, initial comparisons are done using a straightforward off-the-shelf decision tree classifier @xcite from scikit-learn 6 . this model has the advantage of being highly interpretable, which makes it simple to determine which features are most important 6 https://scikit-learn.org/stable/modules/tree.html for distinguishing hindi from urdu. we train the decision tree with a maximum depth of 5 nodes, and use character 1-through 4-gram features. to see how a more complex neural model behaves, we also finetune foot_5 the same mt5 checkpoint we use for data generation (see section 4.2) to act as a classifier, where the input is a romanized sequence with the added task prefix 'classify_hiur:', and the output is either the string 'hi' or 'ur'. for each of the three sources of classifier training data -dakshina, mc4 and mt5 generated text -we provide the size of the resource and example strings in ## results and discussion in this section, we first determine the language identification classification accuracy of our two classifiers when trained on three different sourcs, before attempting to estimate the amount of urdu in the romanized hindi section of mc4. we additionally examine the most important features of the decision tree model. mc4 extracted sentences, which is a baseline method for making use of the provided seed words; and (c) the mt4 generated data. examining the topline result for each classifier, i.e., training on the well-matched dakshina training set, we can see clearly that the mt5 classifier achieves much higher accuracy (96.7%) than the decision tree classifier (85.6%). the neural classifier is simply more powerful, having access to more than just the local character n-gram features used by the decision tree model, and is able to leverage pretraining effectively. this is exactly why the dakshina training is labeled as a topline evaluation, because strong classifiers can make use of well-matched annotator and/or domain characteristics that permit more effective discrimination between examples in the collection. the decision tree classifier fails to exploit such dependencies, hence its topline performance suffers relative to the neural model. ## conclusion we demonstrate that it is possible to make use of pretrained large language models to generate useful training data for language identification, even if the distinction between the languages was only implicit in the pretraining data. our method only requires a corpus of unlabelled, mixed data from the two language varieties in question and a short list of seed words from each language. it can therefore be applied in cases where only unlabeled textual data exists, including lower-resource language scenarios. interestingly, the combination of a powerful neural llm for generating training data and a relatively simple decision tree classifier making use of local word-level features, yielded the best results. by focusing on local word-form features, the decision tree classifier avoided exploiting more global (but less relevant) cues in the generated strings, and thus was able to learn interesting word-level dependencies that the more powerful model simply ignored. future work will include manual validation and error analysis of classifier performance on a range of texts. further, we intend to examine this method, as suggested earlier, on more clearly documented written language varieties, such as those found with us and uk english spelling differences. we also plan to investigate similar language variety confounds, such as that found between bosnian, croatian and montenegrin in the latin script. this work does not propose a new model or dataset, but rather probes the behavior of existing models. thus novel ethical considerations about model behavior and dataset contents are not directly raised by this work. while not explicitly focused on ethical considerations, this paper's methods hopefully contribute to better understanding model behavior, and could be used to understand the ways in which large language models treat underrepresented and marginalized language varieties.