Cognition-aware Cognate Detection (2021)

Abstract: Automatic detection of cognates helps downstream NLP tasks of Machine Translation, Cross-lingual Information Retrieval, Computational Phylogenetics and Cross-lingual Named Entity Recognition. Previous approaches for the task of cognate detection use orthographic, phonetic and semantic similarity based features sets. In this paper, we propose a novel method for enriching the feature sets, with cognitive features extracted from human readers’ gaze behaviour. We collect gaze behaviour data for a small sample of cognates and show that extracted cognitive features help the task of cognate detection. However, gaze data collection and annotation is a costly task. We use the collected gaze behaviour data to predict cognitive features for a larger sample and show that predicted cognitive features, also, significantly improve the task performance. We report improvements of 10% with the collected gaze features, and 12% using the predicted gaze features, over the previously proposed approaches. Furthermore, we release the collected gaze behaviour data along with our code and cross-lingual models.

## introduction cognates are word pairs, across languages, which have a common etymological origin. for example, the french and english word pair, liberté -liberty, reveals itself to be a cognate through orthographic similarity. the task of automatic cognate detection across languages 1 requires one to detect word pairs which are etymologically related, and carry the same meaning. such word pairs share a formal and/or semantic affinity and facilitate the second language acquisition process, particularly between related languages. although they can accelerate vocabulary acquisition, language learners also need 1 cognates can also exist in the same language. such word pairs/sets are commonly referred to as doublets. to be aware of false friends and partial cognates, at times, leading to unrelated semantic coupling. for example, "gift" in german means "poison", which is a known example of a false friend pair. for an example of a partial cognate, the word "police" can translate to "police", "policy" or "font", in french, depending on the context in which it was used. manual detection of such cognate sets requires a human expert with a good linguistic background in multiple languages. moreover, manual annotation of cognate sets is a costly task in terms of time and human effort. automatic cognate detection (acd) is a well-known task, which has been explored for a range of languages using different methods; and has shown to help cross-lingual information retrieval @xcite , machine translation (mt) @xcite , and computational phylogenetics @xcite . in the traditional approaches to automatic cognate detection, words with similar meanings or forms are used as probable cognates @xcite @xcite . from such pairs, the ones that exhibit a high phonological, lexical and/or semantic similarity, are analyzed in order to find true cognates. @xcite perform an investigation to evaluate the use of crosslingual embeddings and show that these models inherit a lexical structure matching the bilingual lexicon. this study establishes that cross-lingual models can provide an effective similarity score compared to their monolingual counterparts, for both cognates and false friends. however, they do not evaluate machine learning (ml) approaches that distinguish between cognates and false friends. the absence of such an evaluation motivates us to use cross-lingual similarity scores with ml algorithms. our work foot_0 reports whether these scores can provide an adequate distinction or not. inspired from their work, we investigate the use of cross-lingual word embeddings and cognitive features to distinguish between cognates and false friends, for the indian language pair of hindi and marathi. cognitively inspired features have shown to improve various nlp tasks @xcite . however, most of their work involves collecting the gaze behaviour data first on a large sample, and then splitting the data into training and testing data, before performing their experiments. while their work does show significant improvements over baseline approaches, across multiple nlp tasks, collecting gaze behaviour over a large cognate dataset can be costly, both in terms of time and money. our approach tries to reduce annotation cost by predicting gaze behaviour data for a large sample based on the smaller sample of collected gaze data. our investigations use three recently proposed cross-lingual word embeddings based approaches to generate features for the task of cognate detection. we also generate cognitive features from participants' gaze behaviour data and show that gaze behaviour helps the task of cognate detection. additionally, we use the collected gaze behaviour data and predict gaze-based features for a much larger sample. we believe that using gaze features will be more applicable only if gaze features can be predicted for unseen samples. we believe that collection of gaze data cannot be performed in all the scenarios and hence hypothesize that predicting such data if it helps improve the task of cognate detection, should be a viable solution. ## related work current literature which uses gaze behaviour to solve downstream nlp tasks has been applied to the nlp tasks of sentiment analysis @xcite @xcite , sarcasm detection @xcite , grammatical error detection @xcite , hate speech detection @xcite , named entity recognition @xcite , part-of-speech tagging @xcite , sentence simplification @xcite , and readability @xcite . a comprehensive overview is provided by @xcite . the primary motivation of using cognitive features for nlp tasks is derived from the eye-mind hypothesis @xcite , which establishes a direct correlation between a reader's comprehension of the text with the time taken to read the text. this hypothesis has initiated a large body of psycholinguistic research that shows a relationship between text processing and gaze behaviour. @xcite discuss the use of gaze features for the task of anaphora resolution. their findings show that gaze data can substitute the classical text processing approaches along with the fact that human disambiguation process overlaps with the information carried in linguistic features. @xcite use gaze data to automatically identify multiword expressions and observe that gaze features help improve the accuracy of the task when combined with traditional linguistic features used for the task. @xcite describe an approach to scoring essays in a multi-task learning framework automatically. their approach relies on collecting gaze behaviour for essay reading for a small set of essays and then predicting the rest of the dataset's gaze behaviour in a multi-task learning setup. similarly, @xcite use token level averages of cognitive features at run time, to mitigate the need for these features at run time. @xcite and @xcite predict gaze behaviour at the token-level as well. @xcite , gonzález-garduño and søgaard (2018), @xcite , and @xcite , use multi-task learning to solve the primary nlp task, where learning the gaze behaviour is an auxiliary task. orthographic/string similarity-based methods are often used as baseline methods for the cognate detection task, and the most commonly used method amongst them is the edit distance-based similarity measure @xcite . research in automatic cognate detection using various aspects involves the computation of similarity by decomposing phonetically transcribed words @xcite , acoustic models @xcite , clustering based on semantic equivalence @xcite , and aligned segments of transcribed phonemes @xcite . @xcite employs a siamese convolutional neural network to learn the phonetic features jointly with language relatedness for cognate identification. @xcite use svm for phonetic alignment and perform cognate detection for various language families. various works on orthographic cognate detection usually take alignment of substrings within classifiers like svm @xcite or hmm @xcite . ciobanu and dinu (2014) employ dynamic programming based methods for sequence alignment. cognate facilitation in second language learners has previously been explored with the help of eye-tracking studies @xcite @xcite but the task of cognate detection has not been performed with gaze features obtained from the cognitive data in any of the previously available literature. for the task of cognate detection, however, the use of cognitive features has not been established previously. the task of cognate detection is crosslingual, and a reader's cognitive load should vary while trying to comprehend the meaning of concepts, from different languages. our work tries to exploit the difference noted in terms of time taken and eye-movement patterns in cognates vs false friends, to generate additional features for the acd task. moreover, for indian languages such as marathi, where agglutination foot_1 varies the word length, the task becomes tougher, computationally. ## dataset acquisition & analysis we pose the problem of cognate detection as a binary classification task in a supervised setting. we use a recently released challenging dataset @xcite of cognates and false friends. we extract the hindi-marathi cognate and false friend pairs. the number of cognate and false friend pairs released by the paper cited above is 15726, and 5826. we select an equal number of cognates at random to reduce this skew, thus producing a balanced dataset for the classification task. for any further experiments in our paper, we use this artificially class-balanced dataset of 5826 (cognates) + 5826 (false friends) data points. we also augment the complete dataset with context information from the indowordnet @xcite . the conwe extract 100 pairs, at random, from each of the positive and negative labels for collecting gaze behaviour data, to construct what we call "d2". this data, extracted from d1, is used to collect gaze behaviour and annotation. although we have gold labels for the data extracted, we ask the participants to annotate the data by asking them if the concepts shown on the screen mean the same. the annotation screen provides them with contextual clues obtained from wordnet data on the screen, as shown in figure 1 . the complete dataset statistics are shown in ## feature sets for cognate detection in this section, we discuss the various features used for the task of cognate detection. it is to be noted that false friends are spelt similarly across languages but mean differently. using false friends as data points with negative labels restricts us to the use of semantic similarity based features, as orthographic or phonetic similarity-based measures would fail to detect sufficient distinction between them. hence, we use the features proposed by @xcite and @xcite as baseline features for a comparative evaluation. ## the cognate detection task we employ both classical machine learning-based models and a simple feed-forward neural network. to compare our work with the previously proposed approaches, we replicate the best-reported systems from @xcite i.e., siamese convolutional neural network with phonetic vectors as features and also replicate @xcite 's approach which uses a recurrent neural network architecture with a weighted lexical similarity (wls) as the feature set. the input to our classifiers is the feature sets described above for each candidate pair. among the classical machine learning models, we use support vector machines (svm) and logistic regression (lr). we experiment with the use of both linear svms and kernel svms (gaussian and polynomial). we perform a grid-search to find the best hyper-parameter value for c over the range of 0.01 to 1000. we deploy the feed forward neural network (ffnn) with one hidden layer. we perform cross-validation with different activation function settings (tanh, hardtanh, sigmoid and relu) and the hidden layer dimension in the network (30, 50, 100, and 150). we use binary cross-entropy as the optimization algorithm. finally, we choose the hyper-parameter configuration with the best validation accuracy. we train the model with the selected configuration with an initial learning rate of 0.4, and we halve the learning rate when the error on the validation split increases. we stop the training once the learning rate falls below 0.001. we perform 5-fold stratified cross-validation, which divides the data into train and test folds, randomly. initially, we perform our experiments with the feature sets from three different cross-lingual embeddings (muse, xlm, and vecmap) for the dataset d1, then with the smaller dataset d2 and later on the combined dataset d1+d2. we, then, perform the same task for the smaller dataset d2 by combining cognitive features with individual cross-lingual feature sets. we also observe the performance of standalone gaze features for the d2 dataset. finally, we evaluate the predicted gaze features on the combined dataset by combining them with cross-lingual features and a standalone feature set, using the feed-forward neural network. we report the results of the cognate detection task in the next section and discuss them in detail. ## results and dicussion we report the results of the cognate detection task in on the combined dataset with collected gaze features (on d2) and predicted gaze features (on d1), we report our best system [ffnn (d1+d2)] which shown an improvement of 12% over the stronger baseline (wls), and 16% over the weaker baseline (phonetic). this system also outperforms the best reported cross-lingual features-based approach by 3%, as shown in for example, the cognate pair "utpana" (hindi) -"utpaadit" (marathi) (both meaning manufactured) is classified correctly by this system, but incorrectly by both the baselines, and all the cross-lingual systems. furthermore, we also show that gaze features can be predicted based on a small sample data, and improved performance can be attained with the help of cross-lingual features, reported with our work. ## conclusion and future work in this paper, we harness cross-lingual embeddings and gaze-based features to improve the task of cognate detection for the indian language pair of hindi-marathi. we create a novel framework that derives insights from human cognition, that manifests over eye movement patterns. we hypothesize that by augmenting cross-lingual features with features obtained from the gaze data, the task of cognate detection can be improved. we use a linked knowledge graph (indowordnet) to augment a publicly released cognate dataset with contextual clues. we collect the gaze behaviour data from nine participants over 200 samples and perform the task of cognate detection for both our datasets (with gaze data and without gaze data). then, we use a neural network to predict gaze features for unseen samples and perform the task of cognate detection to show improved performance, despite a small sample of collected gaze data. we reproduce the previously proposed baseline approaches and perform experiments using additional features obtained via cross-lingual models for a comparative evaluation. the previously proposed approaches @xcite for this task are shown to be outperformed by cross-lingual features and the combination of these features with the obtained gaze data. our experiments use three different approaches to generate feature representations for the cognate detection task, and all of them show improvements over previously proposed approaches. we observe consistent improvements in terms of precision, recall and f-scores. over the stronger baseline, our best system shows an improvement of 12% points and 16% points over the weaker baseline. this system also outperforms the cross-lingual features based approaches by 3%, over the combined dataset. we release this augmented dataset, along with our code and cross-lingual models for further research. in future, we aim to add more language pairs and leverage contextual information from knowledge graphs using sequence-based neural models. we also aim to collect gaze data and then model the gaze predictions in a multi-task setting. we plan to investigate other multilingual contextual embeddings' performance for this task (e.g., m-bert, indicbert, muril). we also plan to look for a method to differentiate between different interest areas and see if a markup facility is present in the software used to analyze the gaze data. we also aim to investigate the task of cognate detection for the indo-european language family, in the near future.