Utilizing Wordnets for Cognate Detection among I ndian Languages (2019)

Abstract: Automatic Cognate Detection (ACD) is a challenging task which has been utilized to help NLP applications like Machine Translation, Information Retrieval and Computational Phylogenetics. Unidentified cognate pairs can pose a challenge to these applications and result in a degradation of performance. In this paper, we detect cognate word pairs among ten Indian languages with Hindi and use deep learning methodologies to predict whether a word pair is cognate or not. We identify IndoWordnet as a potential resource to detect cognate word pairs based on orthographic similarity-based methods and train neural network models using the data obtained from it. We identify parallel corpora as another potential resource and perform the same experiments for them. We also validate the contribution of Wordnets through further experimentation and report improved performance of up to 26%. We discuss the nuances of cognate detection among closely related Indian languages and release the lists of detected cognates as a dataset. We also observe the behaviour of, to an extent, unrelated Indian language pairs and release the lists of detected cognates among them as well.

## introduction cognates are words that have a common etymological origin @xcite . they account for a considerable amount of unique words in many lexical domains, notably technical texts. the orthographic similarity of cognates can be exploited in different tasks involving recognition of translational equivalence between words, such as ma-chine translation and bilingual terminology compilation. for e.g., the german -english cognates, blume -bloom can be identified as cognates with orthographic similarity methods. detection of cognates helps various nlp applications like ir @xcite . @xcite study various cognate detection techniques and provide substantial proof that automatic cognate detection can help infer phylogenetic trees. in many nlp tasks, the orthographic similarity of cognates can compensate for the insufficiency of other kinds of evidence about the translational equivalency of words @xcite . the detection of cognates in compiling bilingual dictionaries has proven to be helpful in machine translation (mt), and information retrieval (ir) tasks @xcite . orthographic similarity-based methods have relied on the lexical similarity of word pairs and have been used extensively to detect cognates @xcite @xcite . these methods, generally, calculate the similarity score between two words and use the result to build training data for further classification. cognate detection can also be performed using phonetic features and researchers have previously used consonant class matching (ccm) @xcite , sound class-based alignment (sca) @xcite etc. to detect cognates in multilingual wordlists. the identification of cognates, here, is based on the comparison of words sound correspondences. semantic similarity methods have also been deployed to detect cognates among word pairs @xcite . the measure of semantic similarity uses the context around both word pairs and helps in the identification of a cognate word pair by looking of similarity among the collected contexts. for our work, we can primarily divide words into four main categories viz. true cognates, false cognates, false friends and non-cognates. in figure 1 , we present this clas-sification with examples from various languages along with their meanings for better understanding. while some false friends are also false cognates, most of them are genuine cognates. our primary goal is to be able to identify true cognates. sanskrit (sa) is known to be the mother of most of the indian languages. hindi (hi), bengali (bn), punjabi (pa), marathi (mr), gujarati (gu), malayalam (ml), tamil (ta) and telugu (te) are known to borrow many words from it. thus, one may observe that words which belong to the same concept in these languages, if orthographically similar, are true cognates. currently, we include loan words in the dataset used for our work and include them as cognates. since, eventually we aim to apply our work to machine translation and other nlp applications, we believe that this would help establish a better correlation among source-target language pairs. also, we do not detect false friends and hence restrict the scope of true cognate detection using this hypothesis to figure 2 . we utilize the synset information from linked wordnets to identify words within the same concept and deploy orthographic similaritybased methods to compute similarity scores between them. this helps us identify words with a high similarity score. in case of most of the indian languages, a sizeable contribution of words/concepts is loaned from the sanskrit language. in linked indowordnet, each concept is aligned to the other based on an 'id' which can be reliably used as a measure to say that the etymological origin is the same, for both the concepts. hence, words with the same orthographic similarity can be said to be 'true cognates'. using this methodology, we detect highly similar words and use them as training data to build models which can predict whether a word pair is cognate or not. the rest of the paper is organized as follows. in section 2 we describe the related work that has been carried out on cognate detection together with some of its practical applications, while in section 3 we present our approach and deal in greater detail with our learning algorithms. once the proposed methodology has been outlined, we step through an evaluation method we devised and report on the results obtained as specified in section 4. section 5 concludes our paper with a brief summary and tackling further challenges in the near future. ## related work one of the most common techniques to find cognates is based on the manual design of rules describing how orthography of a borrowed word should change, once it has been introduced into the other language. @xcite expand a list of english-german cognate words by applying well-established transformation rules. they also noted that the accuracy of their algorithm increased proportionally with the length of the word since the accidental coexistence of two words with the same spelling with different meanings (we identify them as 'false friends') decreases the accuracy. most previous studies on automatic cognate identification do not investigate indian languages. most of the indian languages borrow cognates or "loan words" from sanskrit. indian languages like hindi, bengali, sinhala, oriya and dravidian languages like malayalam, tamil, telugu, and kannada borrow many words from sanskrit. although recently, @xcite perform cognate detection for a few indian languages, but report results with manual verification of their output. identification of cognates for improving ir has already been explored for indian languages @xcite . string similarity-based methods are often used as baseline methods for cognate detection and the most commonly used among them is edit distance based similarity measure. it is used as the baseline in the early cognate detection papers @xcite . essentially, it computes the number of operations required to transform from source to target cognate. research in automatic cognate detection using phonetic aspects involves computation of similarity by decomposing phonetically transcribed words @xcite , acoustic models @xcite , phonetic encodings @xcite , aligned segments of transcribed phonemes @xcite . we study @xcite 's research, which employs a siamese convolutional neural network to learn the phonetic features jointly with language relatedness for cognate identification, which was achieved through phoneme encodings. although it performs well on the accuracy, it shows poor results with mrr. @xcite use svm for phonetic alignment and perform cognate detection for various language families. various works on orthographic cognate detection usually take alignment of substrings within classifiers like svm @xcite or hmm @xcite . we also consider the method of ciobanu and dinu (2014), which employs dynamic programming based methods for sequence alignment. among cognate sets common overlap set measures like set intersection, jaccard @xcite , xdice @xcite or tf-idf @xcite could be used to measure similarities and validate the members of the set. ## datasets and methodology we investigate language pairs for major indian languages namely marathi (mr), gujarati (gu), bengali (bn), punjabi (pa), sanskrit (sa), malayalam (ml), tamil (ta), telugu (te), nepali (ne) and urdu (ur) with hindi (hi). we create two datasets as described below @xmath0. we describe each step in the subsections below. ## results we average the similarity scores obtained using the three methodologies (ned, cos, and jws) described above, for each word pair, and then use these as training labels for cognate detection models. we obtain results using the networks described above and report them in we perform another set of experiments by combining non-redundant word pairs from both datasets. we add wndata in chunks of 20 per cent to pcdata for each language pair and create separate word lists with average similarity scores. we use ffn to train and perform a stratified 5fold evaluation for each language pair after adding each chunk and show the results in only in case of hi-bn, when using the ffn for training, pcdata combined with 80% wndata performs better than 100% data; possibly due to added sparsity of the additional data. our hypothesis that adding wndata to pcdata improves the performance holds for all the other cases, including when trained using rnn. ## conclusion and future work in this paper, we investigate cognate detection for indian language pairs (hi-bn, hi-gu, hi-pa, hi-mr, hi-sa, hi-ml, hi-ta, hi-te, hi-ne, and hi-ur). a pair of words is said to be cognates if they are etymologically related; and true cognates, if they carry the same meaning as well. we know that parallel concepts, bearing the same sense in linked wordnets, are etymologically related. we, then, use the measures of orthographic similarity to find probable cognates among parallel concepts. we perform the same task for a parallel corpus and then train neural network models on this data to perform automatic cognate detection. we compute a list of true cognates and release this data along with the data processed previously. we observe that recurrent neural networks are best suited for this task. we observe that hindi -sanskrit language pair, being the closest, has the highest percentage of cognates among them. we observe that rnn, which treats the words as a sequence of characters, outperforms ffn for all the language pairs and both the datasets. we validate that wordnets can play a crucial role in detecting cognates by combining the datasets for improved performance. we observe a minor, but crucial, increase in the performance of our models when chunks of wordnet data are added to the data generated from the parallel corpora thus confirming that wordnets are a crucial resource for cognate detection task. we also calculate the matches between word pairs from the wordnet data and the word pairs from the parallel corpora to show that wordnet data can form a significant part of parallel corpora and thus can be used in the absence of parallel corpora. in the near future, we would like to use crosslingual word embeddings, include more indian languages, and investigate how semantic similarity could also help in cognate detection. we will also investigate the use of phonetic similarity based methods for cognate detection. we shall also study how our cognate detection techniques can help infer phylogenetic trees for indian lan-guages. we would also like to combine the similarity score by providing them weights based on an empirical evaluation of their outputs and extend our experiments to all the indian languages.