I ru M ozhi: Automatically classifying diglossia in T amil (2024)

Abstract: Tamil, a Dravidian language of South Asia, is a highly diglossic language with two very different registers in everyday use: Literary Tamil (preferred in writing and formal communication) and Spoken Tamil (confined to speech and informal media). Spoken Tamil is under-studied in modern NLP systems compared to Literary Tamil written in the Tamil script, as evidenced by a lack of datasets explicitly targetting the Spoken variety. In this paper, we release IruMozhi, a human-translated dataset of parallel text in Literary and Spoken Tamil. Using IruMozhi, we train classifiers on the task of identifying which Tamil variety a text belongs to. We use these models to gauge the availability of pretraining data in Spoken Tamil, to audit the composition of existing labelled datasets for Tamil, and to encourage future work on the variety.

## introduction diglossia is a linguistic phenomenon wherein a community maintains two (or more) varieties of their language, with the appropriate variety to use depending on the social context @xcite . prototypically, diglossia manifests as two varieties: a high variety employed in formal contexts and a low variety employed in informal settings. the high variety tends to be standardised and highly preferred in writing and other formal communication (speeches, news broadcasts, etc.), while the low dialect is confined to speech and informal written communication (social media, text messages, etc.) and subject to regional and stylistic variation. diglossia is thus a challenge for modern nlp systems-accessible training data on the internet usually overrepresents the high variety, while the average user may prefer using the low variety to interact with nlp systems. ## dataset to study diglossia in tamil, we created irumozhi, foot_1 a dataset of parallel sentences in literary and spoken tamil. we first collected a high-quality set of 499 sentences randomly sampled from a large corpus of scraped tamil wikipedia articles, written in literary tamil. 3 this initial dataset was then converted to spoken tamil by two native-speaker translators. a few examples of the parallel data are presented in ## experiments using irumozhi, we train models on the task of classifying romanised tamil text as literary or colloquial tamil. after evaluating our models on a held-out test set, we audit existing datasets of romanised tamil text to gauge the amount of data available for the two registers. we train two main types of model: naïve bayes classifiers on n-gram features and xlm-r finetuned for sequence clas- xlm-r is a 279m-parameter masked transformer language model trained on the cc-100 web text corpus of one hundred languages, including romanised tamil @xcite . using the huggingface implementation of xlmrobertaforsequenceclassification, we train a classification head on the first @xmath1. we finetune the entire model for 4 epochs with a learning rate of 2 ⋅ 10 -5 for the adam optimiser, on a single nvidia rtx a6000. ## conclusion we presented irumozhi, a parallel corpus of literary and spoken tamil translated on wikipedia text. we trained models on an augmented version of iru-mozhi for classifying tamil diglossia, and audited the composition of existing labelled datasets and the cc-100 pretraining text in romanised tamil. we found that there are indeed labelled and unlabelled data sources for spoken tamil text, indicating hopeful avenues for future nlp research on the variety. particularly, xlm-r seems to have already been trained on some romanised spoken tamil data. we hope to train style transfer models for the two varieties and study diglossia in other indian languages. our aim is to encourage work on lesserstudied languages and dialects in south asia.