How do different factors Impact the Inter-language Similarity? A Case Study on I ndian languages (2021)

Abstract: India is one of the most linguistically diverse nations of the world and is culturally very rich. Most of these languages are somewhat similar to each other on account of sharing a common ancestry or being in contact for a long period of time. Nowadays, researchers are constantly putting efforts in utilizing the language relatedness to improve the performance of various NLP systems such as cross lingual semantic search, machine translation, sentiment analysis systems, etc. So in this paper, we performed an extensive case study on similarity involving languages of the Indian subcontinent. Language similarity prediction is defined as the task of measuring how similar the two languages are on the basis of their lexical, morphological and syntactic features. In this study, we concentrate only on the approach to calculate lexical similarity between Indian languages by looking at various factors such as size and type of corpus, similarity algorithms, subword segmentation, etc. The main takeaways from our work are: (i) Relative order of the language similarities largely remain the same, regardless of the factors mentioned above, (ii) Similarity within the same language family is higher, (iii) Languages share more lexical features at the subword level.

## introduction recently, there has been an explosion in information @xcite and a massive amount of natural language data is added daily on the internet. moreover, the human literature in different cultures is digitalized and became available in digital libraries @xcite . a very large amount of this data is formatted in natural language. this makes nlp techniques crucial to make the use of this high amount of data. since most of the nlp techniques either require linguistic knowledge that can only be developed by experts and native speakers of that language or they require a lot of labelled data which is again expensive to generate, nlp tasks become challenging for low resource languages like indian languages. india is a multicultural country, a country with highly religious and ethnically diverse people. people of different races and classes live in different parts of the country, and they speak a variety of languages. most of the indian languages are divided into two main language families namely indo-aryan foot_0 and dravidian foot_1 . underlying the vast diversity in indian languages are many commonalities. because of contact over thousands of years, most of the indian languages have undergone convergence to a large extent @xcite . therefore, exploiting language relatedness becomes very crucial in nlp related tasks for indian languages. @xcite also presents an impressive case study for utilizing language relatedness for machine translation but that study was more inclined toward exploring statistical approaches to mt. prasanna (2018) in his work has explored efficient ways of exploiting relatedness in multilingualism and transfer learning for low resource machine translation. but no such large scale study has been done on exploring different factors that may affect the process of calculating similarity among indian languages. this could really help the future researchers in getting the clear picture while exploiting related languages in nlp related tasks. so, in this work, we performed an extensive case study on the language relatedness involving languages of the indian subcontinent. this case study provides the reader with valuable information about the different methodologies in measuring relatedness between indian languages. in addition, it also compares some popular techniques for measuring sentence-to-sentence similarity. moreover, datasets from different domains and sizes have also been used in comparing the similarity scores to enable the reader to build a complete background in this area. also, these languages share a lot of cognates, that's why we have also compared the similarity among language pairs at both word and subword level. this paper is further divided into 4 sections. section 2 elaborated the methodology behind the different techniques and experiments. section 3 elaborates the experimental details including the dataset preparation and pre-processing. all the results and analysis have been discussed in section 4. section 5 talks about conclusion and possible future work. ## methodology a universal characteristic of indian languages is their complex morphology and their unique characteristics following default sentence structure as subject object verb (sov). thus, we will be using the parallel corpora for calculating the similarity among them. but indian languages are written in different scripts, so in order to calculate the similarity between two languages, one needs to first map every language to a common surface form i.e to a common script. to do so, we are using a very well-known technique of 'unified transliteration'. it is a string homomorphism technique in which every character of the source is replaced with the target language script. following are the steps for calculating the similarity between the two languages: the pipeline of the above discussed procedure is shown in figure 1 . also in computer science, string similarity is an important family of algorithms that try to find a place where one or several strings (also called patterns) are found within a larger string. researchers have already put the efforts and showed that these algorithms effectively calculate the similarity between two strings @xcite @xcite @xcite . some studies have also been done on calculating similarity particularly for indian languages @xcite @xcite @xcite . in this work, we will consider sentences as a string and use some of the above algorithms for calculating the similarity between two languages. ## results & analysis in case 1, we are evaluating the effect of different algorithms on the similarity. figure 2a shows the results corresponding to every language pair using the algorithm mentioned in section 2.1 (token overlap). similarly figure 2b , 2c, 2d represent the results corresponding to other algorithms discussed in section 2.2, 2.3 and 2.4 respectively. here, we observed a small amount of variation in similarity values with the different algorithms. however, more importantly, relative values within the similarity matrix remain almost constant even in the different algorithms, e.g., hindi-urdu has shown the most similarity in all algorithms. our results confirm that this also holds for other language pairs. for case2, we evaluated whether common script selection matters in calculating language similarity or not. to do so, we also performed experiments by calculating similarity for all languages using the abugida script. in these experiments, it can be seen that the scripts do not matter in calculating the similarity. we got similar results with both the devanagari and abugida scripts with a variation of 0.25%. this can be seen by comparing figure 4a with figure 2c and figure 4b with figure 2d . case 3 shows variation in similarity to the number of sentences we are using for calculating it. figure 3a shows variation plots of similarity v/s no. of sentences used. we used overall 2000 sentences and observed that the similarity value gets stable by the end of the curve; in addition to observing that, we also see that the value does not vary much with larger sentences. we also performed experiments with 10k, 20k, 50k sentences; figure 3b , 3c and 3d shows the plot corresponding to each case respectively. it can be seen there is not much fluctuation in the curve, even with the introduction of more sentences after 2k. thus, we can say for calculating similarity, a small parallel data-set of 2k sentences is enough. we can further see in case 4 that the similarity is not dependent on the nature of data, and is thus independent of external factors such as domain. figure 5a and figure 5b show the results corresponding to ilci and pmi corpus, respectively. we observed that the similarity values might vary with the change in data-set, but the overall relative similarity matrix will remain constant. more clearly, if the similarity value of l1-l2 varies by a magnitude of k, then there will be a approximate change of k in the magnitude for the other language pairs. this can be confirmed by observing the results from the above experiments. in the last case 5, we observe from figure 6a and figure 6b that similarity increases drastically for the lexemes of all languages pairs. that is, if we ignore affixes and consider the root form of a word, we can notice that the similarity increases. also from the above experiments, we can conclude some general results as: ## conclusion & future work in this paper, we did an extensive study on similarity involving the languages of indian subcontinent. we explored different factors that may affect the process of calculating similarity. our results led to some interesting conclusions, such as how the relative order of similarity among languages remains same irrespective of the factors we considered, and how the maximum similarity is observed within pairs of the same language family and it increases more with geographic proximity. thus, this study will help future research which focuses on exploiting the language relatedness for nlp tasks. future work along these lines can focus on using semantic similarity alongside lexical similarity to increase accuracy.