Combining Noisy Semantic Signals with Orthographic Cues: Cognate Induction for the I ndic Dialect Continuum (2022)

Abstract: We present a novel method for unsupervised cognate/borrowing identification from monolingual corpora designed for low and extremely low resource scenarios, based on combining noisy semantic signals from joint bilingual spaces with orthographic cues modelling sound change. We apply our method to the North Indian dialect continuum, containing several dozens of dialects and languages spoken by more than 100 million people. Many of these languages are zero-resource and therefore natural language processing for them is non-existent. We first collect monolingual data for 26 Indic languages, 16 of which were previously zero-resource, and perform exploratory character, lexical and subword cross-lingual alignment experiments for the first time at this scale on this dialect continuum. We create bilingual evaluation lexicons against Hindi for 20 of the languages. We then apply our cognate identification method on the data, and show that our method outperforms both traditional orthography baselines as well as EM-style learnt edit distance matrices. To the best of our knowledge, this is the first work to combine traditional orthographic cues with noisy bilingual embeddings to tackle unsupervised cognate detection in a (truly) low-resource setup, showing that even noisy bilingual embeddings can act as good guides for this task. We release our multilingual dialect corpus, called HinDialect, as well as our scripts for evaluation data collection and cognate induction.

## introduction hindi is listed as one of the 22 official languages of india, with the latest census showing 43.63% of indians as having hindi as their mother tongue. @xcite however, this figure counts speakers of the languages of the whole indic/indo-aryan (ia) dialect continuum, the "hindi belt", that stretches from rajasthan in the west to bihar and jharkhand in the east, and of which modern standard hindi is only a part. @xcite this continuum, spread out over north and central india, contains a wide variety of languages/dialects that may even be mutually incomprehensible, and form subfamilies of their own, e.g. the rajasthani, bihari, or pahari subfamilies. @xcite natural language processing (nlp) resources for these languages are sorely lacking; most of these languages, despite having millions of speakers, have little or no monolingual data, no linguistic resources such as lexicons, grammars, taggers, let alone more elaborate resources such as parallel data or pretrained embeddings. we focus on 26 languages of the hindi belt written in the devanagari script and make the following contributions: (i) we collect the first monolingual resources for many of these languages, and (ii) we develop a novel strategy for cognate lexicon induction in asymmetric truly low-resource scenarios, tackling this problem for the first time with the under-researched indic dialect continuum. cognate induction is an important first step towards obtaining bilingual lexicons, one of the most basic and all-purpose bilingual resources a language can have. bilingual lexicons are especially useful in low-resource scenarios, e.g. for word-by-word translation, bilingual transfer, and as seeds for a variety of tasks; they also have applications in historical linguistics. finally, in the case of severely under-supported languages, they are crucial for building dictionaries for speakers and language learners. in this work, we perform cognate induction for each language against hindi, since hindi is the most well-studied and resource-rich of this set, and therefore the most logical language from which bilingual transfer may be attempted. we crawl monolingual data for the continuum, forming the largest collection (in number of languages) of a dialect continuum as far as we know. this also introduces the first monolingual data for 16 zero-resource ia languages to the nlp community. such a corpus has wide applications for work in transfer, historical linguistics, dialect continua, and building language support for these communities. we probe the resulting multilingual collection at a character, subword and lexical level, finding a general link between relatedness and genealogically and geographically proximal languages. secondly, we use the corpus for cognate/borrowing induction (ci) for each target language with hindi: 6 identifying cognates from monolingual corpora containing fully inflected word forms in a completely unsupervised manner. @xcite we work in an asymmetric data scarcity situation: we have abundant monolingual resources for hindi, but only a few thousands/ten thousands of monolingual tokens for target languages. these constraints set this task apart from most of the previous literature on cognate identification (list, 2014; fourrier et al., 2021; list, 2019; artetxe et al., 2018); however, this setting is realistic when attempting to build resources for truly low-resource languages. we present two simple but novel strategies for cognate identification, evaluating on synthetically created test sets. we experiment with iteratively learning substitution probabilities within an edit distance paradigm, as well as combining noisy semantic signals from a subword embedding space with orthographic distance measures, reporting qualitative improvements over the baseline. ## semantic similarity for cognate induction orthographic matching, even with tailored and learnt substitution matrices for a given pair of languages, may be inherently inadequate, as it pays no heed to the shared semantics of cognates. we use bilingual subword embeddings (be) to address this problem in the following way: we use the semantic space to narrow down possible candidates, and then apply orthographic matching in order to select the top k candidates. this is a two-stage approach that relies mainly on two separate metrics: first, the quality of semantic similarity judgments provided by a semantic embedding space, and second, orthographic similarity judgments provided by the distance/similarity metric we choose to use. sem_jw: be+jw in this approach, we retrieve k nearest neighbours of each source word. these candidates are scored by an interpolation of semantic similarity and orthographic distance, with equal weighting. we use cosine similarity for the former, and jw for the latter. all words that are not within the k nearest neighbours (50 in our experiments) are discarded from consideration. the idea is to mitigate the effect of chance orthographic similarities. for candidates, if e(s) is the embedding vector for string s, we minimize: sem_emt: be+emt we seek to combine the benefits of iteratively learning transformation probabilities with those of semantic spaces. this approach is almost identical to that in section 3.2, except for the fact that only @xmath0, both during training and inference. ## data collection we apply the methods described above to the indic dialect continuum. since these languages cover a range of resource situations, we divide them into three categories, band 1, 2 and 3, based on amount of resources, with band 1 containing the best resourced languages, and band 3 containing (previously) zero-resource languages. see ## discussion and conclusion we analyse the performance of the approaches with respect to the different facets of cognacy. variant inflectional endings: learning the correspondences between inflections in a dialect pair is a crucial task when it comes to cognate identification for fully inflected word forms. in terms of producing the right answer, we see an intuitive split between common and rare words when it comes to other approaches. for common words, sem_jw is likely to perform better than the other approaches because the word is well embedded and the correct word form is likely to be nearby in the semantic space, and subsequently selected by jw. in these cases, especially for short words, ned/jw are likely to be derailed by irrelevant words.