Morphology-Aware Meta-Embeddings for T amil (2021)

Abstract: In this work, we explore generating morphologically enhanced word embeddings for Tamil, a highly agglutinative South Indian language with rich morphology that remains low-resource with regards to NLP tasks. We present here the first-ever word analogy dataset for Tamil, consisting of 4499 hand-curated word tetrads across 10 semantic and 13 morphological relation types. Using a rules-based segmenter to capture morphology as well as meta-embedding techniques, we train meta-embeddings that outperform existing baselines by 16% on our analogy task and appear to mitigate a previously observed trade-off between semantic and morphological accuracy.

## introduction continuous-space word embedding methods such as word2vec @xcite have proven to be very useful for a wide range of nlp tasks. however, it has been observed that representations that treat each word holistically face inherent limitations when working with morphologically rich languages, and methods have accordingly been designed to incorporate subword information @xcite . among these, the fasttext embeddings remain one of the best-known @xcite , using character n-grams to approximate word-internal structural features. in this work, we focus on producing morphology-aware embeddings for tamil, a dravidian language with over 68 million speakers across india, sri lanka, malaysia, and singapore @xcite . tamil remains a low-resource language for nlp tasks despite its large speaker base, and traditional methods of evaluating word embeddings, for instance the word analogy task @xcite , are almost entirely lacking. thus, to facilitate our work, we present here a * equal contribution. with regard to morphology, tamil is highly agglutinative, encoding grammatical features such as gender, number, and case in single words comprising large sequences of compounded morphemes. approaches such as character n-grams that generically incorporate subword information may be too coarse when working with tamil, due to short morpheme lengths paired with high similarity between morphemes and sandhi across morpheme boundaries. the high frequency of 'false morphemes' or character sequences resembling morphemes in non-productive situations compounds this further. in our work, we attempt to tailor embeddings to tamil morphology with the incorporation of a rules-based morphological segmenter. we present three primary contributions: our dataset, embeddings, and experiments are all publicly available with documentation at our github repository. ## dataset one of the primary contributions of this work is our novel tamil analogy dataset, the first available for the language. the dataset is a hand-crafted set of 426 paired relations between words, and was produced by the authors. these word pairs are split into 10 semantic and 13 morphological relation types (see appendix a). analogy tetrads are then generated in a combinatorial fashion by combining pairs from the same class. given that tamil is a low-resource language, automated construction of analogy dataset is relatively infeasible; lexicons rarely list fully inflected or complex morphological forms, and the use of a segmenter would subject the dataset to limitations similar to those discussed in section 3.4. as such, the decision to produce a human-curated analogy dataset was motivated by the desire to produce a gold-standard analogy task resource. as a result of tamil's morphological richness, even semantic relations often contain pairs with similar morphology. as such, we clustered word pairs within each relation type that shared identical morphology into labelled sub-classes. analogies produced from morphologically identical pairs, along with analogies from morphological relations, were sorted into the subword category of analogies, and semantic tetrads that were produced from morphologically non-identical pairs were placed in the non-subword category. we note that our segmenter does not correctly identify all morphological relations. therefore, morphoseg may not improve overall performance even in the subword category. however, we will see in section 6 that it significantly improves performance on certain morphological relations, and that our final meta-embedding absorbs these strengths to substantially improve overall performance (across relations and categories). ## results and analysis results on the test set are shown in figure 4 . our model was the strongest among those evaluated in both the subword and non-subword categories. first, we examine individual sets of word embeddings in section 6.1 and observe that they differ substantially in their success modes. in particular, we show that the incorporation of the morphological segmenter appears to significantly boost performance on certain morphological relations. secondly, we turn to analyzing our metaembeddings in section 6.2. counterintuitively, we find that meta-embeddings in fact improve their performance when reduced to the same dimension as our original embeddings, seemingly combining the strengths of different representations. ## conclusion and future work in this paper, we investigated directly incorporating morphology into tamil word embeddings using a morphological segmenter, following @xcite in computing representations for subword units. we constructed a word analogy dataset for tamil consisting of 13 types of morphological relations and 10 types of semantic relations to evaluate performance. we combine individual models to obtain more versatile metaembeddings that seem to overcome previously observed trade-offs. it remains for future work to investigate the performance of our techniques on oov words, and the improvements better morphological segmentation might bring. evaluating our embeddings on other tasks for indian languages, such as akhtar et al. (2017)'s tamil word similarity dataset, remains an important direction, as does studying the importance of incorporating morphology for downstream tasks such as pos tagging and nmt @xcite . exploring the applicability of our pipeline of morphological segmentation and meta-embeddings to other morphology-rich languages is another avenue for future work.