Computational Historical Linguistics and Language Diversity in S outh A sia (2022)

Abstract: South Asia is home to a plethora of languages, many of which severely lack access to new language technologies. This linguistic diversity also results in a research environment conducive to the study of comparative, contact, and historical linguistics–fields which necessitate the gathering of extensive data from many languages. We claim that data scatteredness (rather than scarcity) is the primary obstacle in the development of South Asian language technology, and suggest that the study of language history is uniquely aligned with surmounting this obstacle. We review recent developments in and at the intersection of South Asian NLP and historical-comparative linguistics, describing our and others’ current efforts in this area. We also offer new strategies towards breaking the data barrier.

## introduction south asia foot_0 is home to one-quarter of the world's population and boasts immense linguistic diversity @xcite . with members of at least four top-level major linguistic families foot_1 and several putative linguistic isolates, this region is a fascinating arena for linguistic research. the languages of south asia, moreover, have a long recorded history, and have undergone complex change through genetic descent, sociolinguistic interactions, and contact influence. nevertheless, south asian languages for the most part remain severely underdocumented @xcite , and several languages with even official administrative status (e.g. sindhi) are lowresource (if not data-scarce) for the purposes of all natural language processing tasks @xcite figure 1 : the major language families of south asia @xcite . @xcite ). this data scatteredness persists despite long native traditions of linguistic description, continued language vitality with active use on the internet, and vast numbers of speakers @xcite . we argue that the most basic problem in nlp/cl work on south asian languages is not data scarcity, but data scatteredness. there is much data to be extracted for even the most endangered languages (e.g., burushaski, a language isolate of the northwest), from annotated corpora and grammatical descriptions compiled by linguists, if only one is willing to wrangle idiosyncratic data formats and digitise existing texts. thus far, commercial interests and scientific agencies have only intermittently supported the development of language technology for the region-taking a new approach, we propose a research programme from the perspective of computational historical linguistics, outlining current data gathering initiatives in this discipline and potential benefits to other work across nlp. level languages 4: underdogs hindi 3: rising stars urdu, bengali, tamil 2: hopefuls konkani, marathi, sanskrit, punjabi 1: scraping-bys malayalam, bhojpuri, nepali, doteli, gujarati, newar, dzongkha, maithili, tulu, kannada, odia, kashmiri, romani, pashto, bishnupriya manipuri, divehi, sindhi, tibetan, pali, sinhala, santali, assamese, telugu 0: left-behinds (several hundred languages) narrowing the low-resource category. lowresource languages have recently gained attention in nlp/cl research, both due to the engineering problems of a data-scarce context and also in recognition of the historical focus on english in the field to the detriment of other languages @xcite . this has been accompanied by debate on what languages the label encompasses (e.g. @xcite . in the south asian context, even hindi has been labelled low-resource in some recent work. while it is true that for certain tasks a large institutionallybacked language like hindi can be low-resource, we propose that 'low-resource' languages can be better described with two kinds of situations: the state of nlp in south asia. so far, initiatives for improving language technology in south asia have largely focused on data-scattered (not data-scarce) languages with official status and some degree of standardisation. these include cross-lingual projects such as indicnlpsuite @xcite , the emille corpus @xcite , and inltk @xcite , and workshops like dravidianlangtech @xcite and @xcite . as this is not to say that there are no resources at all for the languages @xcite terms "the left-behinds". linguists, for example, have compiled rudimentary treebanks for many languages, simply waiting to be digitised and converted to a multilingual format like universal dependencies; these include palula @xcite and toda @xcite , which are yet to be the subject of any nlp research work. there are also new treebanks in universal dependencies for kangri, mandeali, bhojpuri (ojha and zeman, 2020), and magahi. historical/comparative linguistics. historical linguistics is concerned with describing change of all kinds (phonological, morphological, syntactic, etc.) in language over time and the factors (social, cognitive, evolutionary) that contribute to that change. comparative linguistics aims to use this historical study to relate languages and reconstruct earlier stages and common ancestors of related languages @xcite . the study of historical and comparative linguistics has a long history in south asia, beginning well before similar threads of inquiry in the western linguistic tradition, with grammarians like pān . ini (c. 5th century bce) and hēmacandra (1088-1173) analysing historical and dialectal language from a comparative perspective. following the recognition by western philologists of an indo-european language family that includes sanskrit, comparative study of the languages of south asia began in earnest. as a result, several comprehensive comparative grammars featuring the dravidian @xcite @xcite and indo-aryan families @xcite @xcite have appeared in the years since. @xcite was the first to posit a south asian zone of language contact and convergence spanning multiple families. subsequent work on micro-areal zones has yielded many insights into the nature of linguistic interactions in the region @xcite @xcite . the sole south asia-wide linguistic data collection effort ever be undertaken was the linguistic survey of india, completed about a century ago @xcite . to date, there has been no comparable centralised data resource on south asian languages of its magnitude-covering typological features, the lexicon, and sociolinguistic phenomena. data in the earliest comparative works was frequently sourced from high-prestige standard varieties like delhi hindi, with progress on studying and collecting data from more localised lects largely proceeding in isolation. compilation of comparative data continued sporadically throughout the 20th century, resulting in works such as the comparative dictionary of the indo-aryan languages @xcite and the dravidian etymology dictionary @xcite which attempt at a more diverse spectrum of language data. meanwhile, progress on documentation and comparative analysis of the austroasiatic @xcite , sino-tibetan, and isolate languages (e.g. burushaski, nihali, kusunda) of south asia is still in its infancy. as a consequence, studies drawing upon their data for purposes such as substrate analysis often lack nuance and familyinternal consistency. ## ongoing work having established the issue of data scatteredness, the mutual benefit inherent to data collection (for historical/comparative linguistic work and other nlp tasks), as well as possible interesting avenues for future research, we present a compilation of our ongoing projects in this direction, most involving languages that have not been studied in nlp before. 2020 )'s level categorization. ## future work the data resources we are in the process of compiling for south asian languages will enable a variety of research to be conducted into language history. we lay out some of the immediate potential pathways for this further research in hopes of stimulating work in this area. ## conclusion in this paper, we gave an overview of the state of nlp in south asia with a special focus on historical-comparative linguistics, a research programme of which we believe will help address the issue of data scatteredness. south asian languages are not obliged to remain low-resource (in the nlp sense), and have plenty of speakers who would like access to and would benefit from language technologies, along with a multitude of raw linguistic resources that can be used to cultivate them. incentives have not been in place to support those demands, however, so we suggest an alternative route founded in linguistic research to gather data. collective efforts have had great success recently in nlp-besides institutional efforts like the stanford center for research on foundation models @xcite and huggingface's big-science workshop, foot_9 there are grassroots organisations like maskhanenlp for african languages @xcite and ai4bharat @xcite that are working towards improving resource availability. our proposals in this paper are the first seeds of a programme similar in spirit, motivated by a dual interest in understanding south asian language history and remedying inequalities in technological availability.