Harnessing Cross-lingual Features to Improve Cognate Detection for Low-resource Languages (2020)

Abstract: Cognates are variants of the same lexical form across different languages; for example “fonema” in Spanish and “phoneme” in English are cognates, both of which mean “a unit of sound”. The task of automatic detection of cognates among any two languages can help downstream NLP tasks such as Cross-lingual Information Retrieval, Computational Phylogenetics, and Machine Translation. In this paper, we demonstrate the use of cross-lingual word embeddings for detecting cognates among fourteen Indian Languages. Our approach introduces the use of context from a knowledge graph to generate improved feature representations for cognate detection. We, then, evaluate the impact of our cognate detection mechanism on neural machine translation (NMT), as a downstream task. We evaluate our methods to detect cognates on a challenging dataset of twelve Indian languages, namely, Sanskrit, Hindi, Assamese, Oriya, Kannada, Gujarati, Tamil, Telugu, Punjabi, Bengali, Marathi, and Malayalam. Additionally, we create evaluation datasets for two more Indian languages, Konkani and Nepali. We observe an improvement of up to 18% points, in terms of F-score, for cognate detection. Furthermore, we observe that cognates extracted using our method help improve NMT quality by up to 2.76 BLEU. We also release our code, newly constructed datasets and cross-lingual models publicly.

## introduction india is a multilingual, multi-script country with 22 scheduled languages and 12 written script forms primarily belonging to 6 different language families. more than a billion people use these languages as their first language. a significant amount of news and information is found on the web in these languages, which is inaccessible to people of other regions within the country. most of the indian language texts found online have several words that have originated from sanskrit, persian, and english. while, in many cases, one might argue that such occurrences do not belong to an indian language, the frequency of such usage indicates a wide acceptance of these foreign language words as indian language words. in numerous cases, these words also are morphologically altered as per the indian language morphological rules to generate new variants of existing words. detection of such variants or 'cognates' across languages helps cross-lingual information retrieval (clir) @xcite , machine translation (mt) @xcite @xcite , and computational phylogenetics @xcite . cognates are etymologically related words across two languages @xcite . however, nlp applications are concerned with the set of cognate words which have similarities in their spelling and their meaning. for example, the french and english word pair, liberté -liberty, reveals itself to be a true cognate through orthographic similarity. in some cases, similar words have a common meaning only in some contexts; such words are called partial cognates. for example, the word "police" in french can translate to "police", "policy" or "font", depending on the context foot_2 . manual detection of such cognate sets requires a human expert with a good linguistic background in multiple languages. moreover, manual annotation of cognate sets is a costly task in terms of time and human effort. the task of cognate detection across languages requires one to detect word pairs which are etymologically related, and carry the same meaning. previous approaches to the task use orhtographic @xcite , phonetic @xcite and semantic @xcite features. however, these methods have a limitation since they do not take into consideration the notion of semantic similarity across languages. a key question that we try to answer in this paper is, "can semantic information be leveraged from cross-lingual models to improve cognate detection amongst low-resource languages?" we hypothesize that utilizing cross-lingual features by employing existing resources such as wordnets and cross-lingual embeddings should help improve cognate detection. in this paper, we utilize the semantic information from cross-lingual word embeddings. cross-lingual word embeddings can be obtained by training monolingual embeddings for individual languages and then projecting them in a shared space using a bilingual dictionary. in the absence of such a bilingual dictionary for low-resource languages, adversarial training can be used over identical words to generate the projections. we build cross-lingual models for thirteen language pairs with hindi as the source (l1) and thirteen target indian languages (l2). we use the context information from a knowledge graph to build the context dictionaries for each pair. the cross-lingual models help us obtain embeddings for the word-pair and the respective context dictionaries, from a shared space. we hypothesize that using this approach should provide a more accurate semantic measure for the detection of cognate pairs. the use of orthographic and phonetic similarity-based methods to perform the same task provides us with baselines for a comparative evaluation. a motivation to investigate this task for low-resource indian languages stems from the fact that most of the indian languages borrow cognates or "loan words" from the sanskrit language. it is, for the most part, considered a historical antecedent of almost all the indian languages. indo-aryan languages like hindi, bengali, gujarati, punjabi borrow from sanskrit. they borrow many lexical forms and language properties from sanskrit. dravidian languages are highly agglutinative and morphologically rich like sanskrit, which makes them tough to parse computationally. marathi and hindi suffer from the same ailment even though hindi is not considered as agglutinative as marathi, but it does exhibit compounding foot_3 which makes it, yet again, difficult to parse for clir and mt systems, and to detect cognates based solely on orthographic similarity. given that clir and mt are usually based on a full-form lexicon, one of the possible issues in the generation of cognates concerns the similarity of words in their root form vs the similarity in their lexical form. for example, the sanskrit word "matra" and the english word "mother" are known cognates from the proto-indo-european language family where the root and the meaning are identical, but the lexical form is considerably different. our approach handles such cases by inculcating the sub-word information while building the embeddings and helps reduce the out-of-vocabulary (oov) words, which have proven to be a challenge for well-established clir systems @xcite . this paper is organized as follows. section 2 briefly describes the previous work done in the area of automatic cognate detection. section 3 describes the dataset source, our additions to it, and the experimental setup. section 4 presents the approaches used in terms of feature sets and classification methodologies. the results obtained are described in section 5 along with a discussion on the qualitative analysis of our output. section 6 concludes this article with possible future work in the area. ## related work the two main existing approaches for the detection of cognates belong to the generative and discriminative paradigms. the first set of approaches is based on the computation of a similarity score between potential candidate pairs. this score can be based on orthographic similarity @xcite @xcite , phonetic similarity @xcite @xcite , or a distance measure with the scores learned from an existing parallel set @xcite . the discriminative paradigm uses standard approaches to machine learning, which are based on (1) extracting features, e.g., character n-grams, and (2) learning to predict the transformations of the source word needed to @xcite . cognate detection has been explored vastly in terms of classification methodologies. previously, rama (2016) employ a siamese convolutional neural network to learn the phonetic features jointly with language relatedness for cognate identification, which was achieved through phoneme encodings. @xcite use svm for phonetic alignment and perform cognate detection for various language families. various works on orthographic cognate detection usually take alignment of substrings within classifiers like svm @xcite or hmm @xcite . ciobanu and dinu (2014) employ dynamic programming based methods for sequence alignment. @xcite perform cognate detection for some indian languages, but a prominent part of their work includes manual verification and segratation of their output into cognates and non-cognates. @xcite utilize recurrent neural networks to harness the character sequence among cognates and non-cognates for indian languages, but employ monolingual embeddings for the task. @xcite show how cross-linguistic similarity of translation equivalents affects bilingual word recognition, even in tasks manually performed by humans. they discuss how the need for recognizing semantic similarity arises for non-identical cognates, based on the reaction time from human annotators. similarly, @xcite show that cross-lingual models exhibit the semantic properties of for bilingual lexicons despite their structural simplicities, which leads us to perform our investigation for low-resource indian languages. @xcite discuss the semantic change in languages by studying the change in cognate words across romance languages using cross-lingual similarity. all of the previous approaches discussed above, lack the use of an appropriate cross-lingual similarity-based measure and do not work well for indian languages as shown in this work. this paper discusses the quantitative and qualitative results using our approach and then, applies our output to different neural machine translation architectures. ## dataset and experimental setup in this section, we describe our primary dataset for the cognate detection task. we also describe the datasets used for building cross-lingual word embedding models, and the parallel corpora used for the neural machine translation (nmt). for our experiments, we use the publicly released challenge dataset @xcite of cognates. this dataset provides labelled cognate and non-cognate pairs for twelve indian languages namely, sanskrit (sa), hindi (hi), assamese (as), oriya (or), kannada (kn), gujarati (gu), tamil (ta), telugu (te), punjabi (pa), bengali (bn), marathi (mr), and malayalam (ml). we reproduce their approach to add two more languages, konkani (ko) and nepali (ne), to this dataset. for building context dictionaries, we use linked indian language wordnets @xcite and concatenate the concept definition and example sentences. we remove stop words from the context dictionaries and append them with their respective word pairs. the lexical overlap between the language pairs ranges from 13% (for hi-te) to only 23% (hi-mr). figure 1 shows an accurate description of the dataset creation process. the cognate dataset statistics are described in ## approaches we use various approaches to perform the cognate detection task viz. baseline cognate detection approaches like orthographic similarity-based, phonetic similarity-based, phonetic vectors with siamese-cnn based proposed by @xcite , and recurrent neural network-based approach proposed by @xcite . we use the same hyperparameters and architectures, as discussed in these papers. we describe each of these feature sets in this section. ## results and discussion from our cross-lingual similarity-based approaches, however, significantly outperform all the baseline approaches. we observe a stark improvement in both precision and recall scores for all the language pairs. the cross-lingual approach, which uses the vectors from vecmap based models, fails to outperform both muse and xlm-r based models. xlm-r model exclusively achieves the best f-score for two language pairs (hi-bn and hi-mr). we believe its performance can be attributed to the closeness of the language pairs as they belong to the same language family (indo-aryan). moreover, xlm-r is a transformer architecture-based model which requires relatively larger corpora sizes and a decent amount of corpus was available to build word embedding models for these target languages ( additionally, we perform an ablation test with our feature sets for further experimentation. we observe that the combination of wls and vectors from the muse model performs even better. an improvement is observed for eight language pairs out of thirteen ranging from 1% point (hi-ko, hi-ml, hi-ta, hi-te) to 5% points (hi-or). it should be noted that this is the only combination where no degradation in performance was observed for any language pair and hence, is reported in the average improvement observed by using our best model (muse + wls) over the strongest baseline approach @xcite is 9% points with the highest being 18% points (hi-ta). over the weakest baseline approach (wls), our best model obtains an average improvement of 50%, peaking at 61% points (hi-or). we present the results of cognate-aware nmt in ## conclusion and future work in this paper, we harness cross-lingual embeddings to improve the task of cognate detection for thirteen indian language pairs. we propose the use of a linked knowledge graph to augment a publicly released cognate dataset with a context dictionary. we reproduce the proposed approach and add two additional language pairs to the same dataset and perform experiments using various approaches for a comparative evaluation. we reproduce the previously proposed approaches @xcite for this task to perform a further evaluation. we obtain monolingual indian language corpora for all the fourteen languages (section 3), from various sources to build monolingual models and use a bilingual dictionary to supervise the task of cross-lingual models generation (muse and vecmap), for thirteen language pairs (hi-mr, hi-ta and so on). we also train a single cross-lingual model using the contextual embedding based approach (xlm-r). our experiments use three different approaches to generate better feature representations for the cognate detection task, and all of them show improvements over previously proposed approaches. we observe consistent improvements in terms of precision, recall and f-scores. we also perform an ablation study and show that augmenting wls baseline feature with muse based features provide us with the best results. over the strongest baseline, this model shows improvements up to 18% points, in terms of f-score. our best f-score is observed for the hi-gu and hi-ne language pairs (0.84) which can still be improved and warrants further investigation into the task. additionally, we use the detected cognate pairs and use a simple approach to inject them into the neural machine translation pipeline. our cognate-aware nmt-bpe results also show a consistent improvement for all the indian language pairs. furthermore, we release this augmented dataset, along with our code and cross-lingual models for further research. in future, we aim further to investigate the performance of contextual embeddings for this task. recent trends show that contextual embeddings based models outperform conventional word embeddings for most tasks. we, however, do not observe this and attribute this primarily due to the dataset size used to train the contextual embeddings. we shall add more data to our monolingual corpora and perform more experiments using xlm-r. future experiments with cognate-aware nmt using the transformer architecture @xcite should further help in showing the importance of our extracted cognate pairs. we also aim to investigate the task of cognate detection for other indian language pairs, along with indo-european language pairs, in the near future.