Subword-based Cross-lingual Transfer of Embeddings from H indi to M arathi and N epali (2022)

Abstract: Word embeddings are growing to be a crucial resource in the field of NLP for any language. This work introduces a novel technique for static subword embeddings transfer for Indic languages from a relatively higher resource language to a genealogically related low resource language. We primarily work with HindiMarathi, simulating a low-resource scenario for Marathi, and confirm observed trends on Nepali. We demonstrate the consistent benefits of unsupervised morphemic segmentation on both source and target sides over the treatment performed by fastText. Our best-performing approach uses an EM-style approach to learning bilingual subword embeddings; we also show, for the first time, that a trivial “copyand-paste” embeddings transfer based on even perfect bilingual lexicons is inadequate in capturing language-specific relationships. We find that our approach substantially outperforms the fastText baselines for both Marathi and Nepali on the Word Similarity task as well as WordNetBased Synonymy Tests; on the former task, its performance for Marathi is close to that of pretrained fastText embeddings that use three orders of magnitude more Marathi data.

## introduction subword-level embeddings are useful for many tasks, but require large amounts of monolingual data to train. while about 15 indian languages such as hindi, bengali, and marathi have the required magnitudes of data, most indian languages are highly under-resourced; they have very little monolingual data and almost no parallel data, and not much digitization. for example, to the best of our knowledge, marwadi, spoken by 14m people, has no available monolingual corpus; konkani, spoken by about 3m people, has a monolingual corpus containing 3m tokens, and no parallel data. 1 however, many of these languages have very close syntactic, morphological, and lexical connections to surrounding languages including the mentioned high-resource languages. our approach aims to leverage these connections in order to build embeddings for these low-resource languages, in the hope that this will aid further development of other nlp tools for these languages. 2 while there is a growing interest in shifting towards contextual embeddings with bert @xcite , as well as extending them to low-resource languages, static embeddings retain value in being lightweight and less computationally expensive, especially as studies show that they can perform comparably to contextual embeddings in certain settings @xcite and encode similar linguistic information @xcite . thus, an efficient method to develop static embeddings for languages with minimal or no nlp research remains a relevant step to building a basic range of resources in these languages. in this study, we primarily work with hindi-marathi as our genealogically and culturally related language pair, and use asymmetric resources (large data for hindi, artificially small monolingual data for marathi), confirming our final results for nepali. most languages of the indic/indo-aryan family, spoken over most parts of north india, are morphologically rich, including hindi, marathi, and nepali. this means that while related language pairs may have a high number of cognates, these may be "disguised" by surrounding inflectional or derivational morphemes. therefore, even with an identical underlying syntactic structure, lexical correspondences between languages may be obscured or rendered incongruent. further, when working with small data, the corpus frequencies of fully inflected surface forms would be much less reliable than those of stem and affix morphemes, intuitively resulting in a less robust embeddings transfer. these factors add weight to the intuition that many indic languages share morpheme-level correspondences with each other. this motivated us to apply unsupervised morphemic segmentation on both the source and target language data; we demonstrate the benefits of doing so in our evaluations. note that this also makes it natural to work with subword-level embeddings rather than word embeddings; studies show that the former have an advantage over word embeddings especially for morphologically rich languages. @xcite @xcite . the idea of the transfer is to project the lowresource language (lrl) subwords into a shared bilingual space with the high-resource language (hrl). we first attempt a trivial transfer that simply finds the "closest" hrl subword for each lrl subword, and copies its embedding. we demonstrate that this approach, while tempting, is not enough to capture the relationships between even identical words in both languages; embeddings spaces appear to encode more complex information that this approach would suggest. for our best performing approach, we adapt the em-style algorithm described in @xcite to a subwordsetting; the algorithm alternately optimizes the distance between pairs belonging to a bilingual mapping, and generates a bilingual mapping between words from the resulting bilingual embeddings. as far as we know, our work is the first to apply this algorithm in the context of embeddings transfer. we compare the resulting bilingual embeddings to data-intensive fasttext models using the word similarity and wordnet-based synonymy tests for marathi; for nepali, we evaluate on the latter task due to the lack of a word similiarity dataset. ## note on languages hindi, spoken by about 340m people, is related to other large indic languages such as marathi, punjabi, and bangla, and has 48 recognized "dialects" over india, which makes it a good choice for the hrl in this project. hindi is written in the devanagari script, which is also used for over 120 other (often related) languages, including marathi and nepali. hindi, marathi, and nepali share morpho-syntactic properties common within the indic language family, such as (split) ergativity and primarily sov structure with reordering allowed under constraints. for all three languages, (some) nouns inflect for case and number, verbs inflects for tense, number, gender, and person, and adjectives inflect for gender and number, and case in hindi and marathi. some differences are that marathi and nepali exhibit more agglutinative tendencies than hindi, both allowing suffix stacking with certain boundary changes. for example, a marathi token may be a sequence of verb+nominalizing-morpheme+case-marker or noun+postposition+genitive, whereas hindi separates these morphemes into tokens in many cases (while still exhibiting inflectional and some derivational morphology). see figure 1 . 4 data and resources 4.1 training data for hindi, we used 1m sentences containing roughly 18m tokens from the hindmonocorp 0.5 @xcite . for marathi, we used 50k sentences containing 0.8m tokens from the indic-corp marathi monolingual dataset @xcite foot_3 , and for nepali, we use 1.4m tokens from the wortschatz corpus @xcite . we choose these numbers for marathi and nepali because it seems to be the ballpark of the amount of monolingual data collected for newly digitized indic languages. 4 all the above corpora, as well as following resources, are in the devanagari script. ## approach as baseline, we train fasttext models on the available tokenized data (mar-base-0.8m, nep-base-1.4m) for both languages. we work with 300dimensional embeddings for all experiments. 10 ## results: wordnet-based synonymy tests (marathi, nepali) see ## discussion some of the clearer findings of our experiments are as regards segmentation and the benefits of a non-trivial bilingual embeddings transfer. we see repeatedly that segmentation on both sides of the transfer helps the quality of the lrl embeddings. segmenting the marathi data causes a large boost in monolingual performance ( finally, we find that, in agreement with the findings of the papers that investigate subword composition functions @xcite , the bestperforming composition function for subword embeddings seems to be task and data dependent; even discarding everything except the first subword seems to work better sometimes than aggregating all subword embeddings. ## future work this work is intended to be the pilot in a series of similar studies. we hypothesize that we can obtain similar results for other genealogically related lrl-hrl pairs. we intend to repeat these experiments for language pairs (simulating lrl environments) such as punjabi-hindi, assamese-bengali, konkani-marathi, and others. some of the issues we will be working against are different scripts, morphemic segmentation of typologically different languages, and the lack of evaluation data. we would also like to experiment with the integration of parallel data into this approach. finally, we also think it would be interesting to extend our so-21 this is to say even if words a and b occur identically and with the same senses in both languages, the word pair (a, b) may have a different relationship depending on the language. 22 our particular "doubled" dataset actually shows roughly the percentage of shared subwords as before doubling; it is possible that data introducing new subwords will perform better. however, in any case, it is interesting to note that the transfer is not improved by having more hrl data for the same subwords which we might intuitively hope would help the quality of the hrl embeddings and therefore the transfer. ## conclusion embeddings transfer from a high-resource language to a low-resource related language is an important task in today's scenario of data inequality across languages. we target a family of geographically and genealogically related languages, including some high-resource languages and other low-resource languages, possibly undergoing digitization and data collection. we take two indic language pairs, hindi-marathi/nepali, simulating a low-resource scenario for marathi and nepali, and present an approach to embeddings transfer that uses very little monolingual data on the lrl side, and no parallel data. we demonstrate the benefits of unsupervised morphemic segmentation on both source and target sides for subword-level embeddings transfer. our final approach improves substantially over monolingual fasttext baselines for the marathi wordsim task, and the wbst task for marathi and nepali. further, we show that a "copy-and-paste" embeddings transfer fails even with a perfect bilingual dictionary for a closely related language pair, establishing the need for more sophisticated methods of low-resource bilingual transfer.