Jambu: A historical linguistic database for S outh A sian languages (2023)

Abstract: We introduce JAMBU, a cognate database of South Asian languages which unifies dozens of previous sources in a structured and accessible format. The database includes nearly 287k lemmata from 602 lects, grouped together in 23k sets of cognates. We outline the data wrangling necessary to compile the dataset and train neural models for reflex prediction on the Indo- Aryan subset of the data. We hope that JAMBU is an invaluable resource for all historical linguists and Indologists, and look towards further improvement and expansion of the database.

## introduction a particular concern of historical linguists is studying relatedness and contact between languages. two languages are related if they share words the arose from a common source, having undergone (potentially different) regular sound changes. 2 for example, the german words schlafen and schiff are cognate to the english words sleep and ship respectively, with the german words having undergone the sound change /p/ → /f/. using evidence like this from all of the germanic languages, historical linguists have reconstructed the historical words that gave rise to these terms: *slāpan and *skip ą @xcite . computational historical linguistics is a growing field that seeks to apply modern computational methods to studying this kind of change @xcite . massive datasets of multilingual cognates are necessary for much of the current research in this area, e.g. on multilingual cognate detection and phoneme-level alignment @xcite and automatic comparative reconstruction 1 the entire dataset is available at https://github.com/ moli-mandala/data , and a web interface for browsing it is at https://neojambu.herokuapp.com/ . cognates. @xcite compiled a cognate database covering 338 languages from wiktionary. they noted that the meaning of cognate varies between research communities-for our purposes as historical linguists, we prefer grouping terms with shared direct etymological sources, while much computational work (e.g. @xcite takes a broader definition which includes loanwords or even all semantic equivalents as cognates. as shown in figure 1, computational historical linguistics has taken on tasks involving cognates such as automatic cognate identification from wordlists @xcite @xcite , cognate/reflex prediction, i.e. predicting the form of a cognate in another language based on concurrent or historical data @xcite @xcite , and reconstruction of the ancestor form of a cognate set @xcite @xcite @xcite . cathcart (2019a,b, 2020) and cathcart and rama (2020) also previously made use of data from turner @xcite @xcite @xcite ## database the jambu database incorporates data from three major language families of south asia: indo-european (the indo-aryan and nuristani subbranches), dravidian, and austroasiatic (the munda subbranch). this comes out to 287k lemmata from 602 lects across 23k cognate sets ( the data is stored in the cldf structured data format. the overall database structure is described in the file wordlist-metadata.json, which includes information about the type of data recorded in each column of each file. the file forms.csv includes all lemmata (word form) and associated etymological and linguistic information. the files parameters.csv and cognates.csv include all cognateset headwords and etymological notes for each. the file languages.csv lists all languages in the database and their geographical location. finally, sources.bib lists all data references in bib-tex format. for each lemma in forms.csv, we store the following information: a unique form id; the language id; the cognate set id, linking it to other cognate lemmata; a normalised representation of the lemma itself, using our transcription scheme; a gloss in english; the spelling of the lemma in the native script; the phonemic ipa representation of the lemma; the unnormalised form of the lemma taken from the original source; a finer-grained cognate set id; notes; and references. for each cognate set, we store a headword, which is usually a common ancestor of the words in that cognateset or a reconstruction of that ancestor if possible. we also store desiderata such as definitions and etymological notes. finally, we take an expansive view of what constitutes a "language" in our database. if a word is ## experiment as a demonstration of the usability of the dataset for computational historical linguistics, we replicate the reflex prediction task of @xcite . we train neural models on the task of reflex prediction in indo-aryan languages, i.e. predicting the descendant of an old indo-aryan word in a given indo-aryan language. rather than being restricted to data from turner @xcite @xcite @xcite , we can draw on all the sources present in jambu. we train on 80% of the data and test on the remaining 20%. we compare two models: a bidirectional gru encoder-decoder with bahdanau attention and a transformer encoder-decoder with learned positional embeddings. the optimised hyperparameters for the gru are a learning rate of 2⋅10 -3 , 4 layers, and embedding and hidden size of 64. the transformer had a learning rate of 1 (using the parameter-based adjustment and warmup/decay schedule from @xcite , 3 layers, 4 attention heads per layer, embedding size of 64, and ffn size of 128. both models were trained for 50 epochs without early stopping with a batch size of 1024 on a single quadro rtx 6000, with a run completing in about 15 minutes. we evaluate bleu and ter on the held-out set using the sacrebleu implementation @xcite , treating a single phoneme as a 'word'. even after comprehensive hyperparameter tuning we find that both models achieve similar performances, per the results in ## conclusion in this paper, we introduced jambu, the largest and most up-to-date cognate database for south asian languages. we are continuing to expand the database, incorporating all lexical data that has so far been unused in comparative linguistic work on the region. we believe that the open questions of south asian historical linguistics cannot be resolved without examining all the information (both synchronic and diachronic) that linguists have collected about language of the region. the old etymological dictionaries are in desparate need of an update. however, much work remains. we briefly discuss some avenues of future work. many sources are yet to be incorporated, especially those recording loanwords from external languages (especially persian, arabic, english, and portuguese) and from local literary languages (particularly sanskrit). we have yet to distentangle cross-lectal interactions and mark lexical isoglosses, which seem necessary to understand the history of language interactions in the region; kalyan et al. ( 2018 )'s wave model of linguistic change has been thought by many scholars to be suited for south asian languages, but it has not been operationalised yet due to a lack of comprehensive data @xcite . another significant task ahead is extending our database structure to support indicating and analysing more complex cross-lingual interactions. for example, the database as it stands does not distinguish between inheritance from the parent language and loaning mediated by a sibling language. we have also been working on a consistent orthography for tonemes in the languages where tones are contrastive, such as the northwestern indo-aryan languages @xcite . older data from these languages either does not notate tone at all (for tonality was not yet recognized, as in gawri and torwali), or represents it indirectly through diachronically correct, but synchronically confusing, spelling systems (as in punjabi and kishtwari). so, our work will also involve analyzing and incorporating new data from tonal languages, both from existing sources and our own fieldwork. finally, we hope to manually improve data quality once the parsing of old sources is stable. this includes fixing known mistakes, reorganising entries to better indicate indirect derivations and crosslectal loans, and etymological notes that summarise the extant literature.