Integration of Self-Attention Model with Intralingual Word Embedding for Contextual Semantic Analysis of Thirukkural Text (2024)

Abstract: Thirukkural, one of the ancient works of Tamil Literature, is popular worldwide due to the moral values and practices it teaches to the society. Understanding the verses with meaning, especially context, is important. In this regard, this paper introduces a system designed to generate contextualized word meanings for the couplets of the Thirukkural, tailored to assist school children in understanding the text more effectively. Unlike traditional methods that provide detailed explanations in paragraph form, our method focuses on word-by-word interpretation, based on context through an integrated self-attention model. By combining the self-attention mechanism with FastText embeddings, our approach achieves improved performance over state-of-the-art models such as Word2Vec and standalone FastText. We evaluate the semantic understanding of the Thirukkural text using metrics as manual scoring. Tamil Thirukkural Agarathi serves as the gold-standard dataset for evaluation, demonstrating the effectiveness of our approach in capturing the nuanced semantics of the Thirukkural.

## introduction tamil, an ancient language with a rich cultural heritage, continues to thrive in the modern world. tamil is spoken by approximately 83 million people worldwide, with 79 million first-language speakers (27th edition of ethnologue published). tamil is globally renowned for its rich literature and ancient traditions that spans thousands of years. through classical poetry, art, philosophy and its adaptive nature, tamil continues to inspire and influence global cultures, making it a vital part of the world's linguistic and cultural diversity. thirukkural written by thiruvalluvar, is one of the cornerstones of tamil language, a classic literature, written in 300 bce. it has been widely classified into three sections namely aram, porul and inbam [1] . thirukkural comprises 1330 couplets which are organized into three sections and 133 chapters [2] .each chapter consists of ten kurals under unique topics. each kural consists of two lines, known *both authors contributed equally to this work as couplets, and contains only seven words. it provides all possible solutions to lead a successful and a peaceful life fitting any generation. it has been translated into 82 global languages [3] including 143 different english versions, allowing its profound insights to reach a global audience. the popularity of thirukkural has grown to such an extent that quoting it in speeches, and various other contexts has been happening on various occasions. thirukkural is now part of the school curriculum across all grades in tamil nadu, ensuring that students are exposed to its moral and ethical values from an early age. however, despite its widespread presence, students often struggle to fully grasp the subtle meaning of kural. because, classical tamil used in the thirukkural text differs significantly from contemporary tamil. to address this issue (difficult to understand the classical thirukkural), the couplets are given with paragraph explanations in textbooks. due to the widespread popularity of thirukkural worldwide, computational processing is available in the form of tools, websites and applications. in existing systems, the computational works done on thirukkural vary on different objectives. retrieval of specific kural based on the user query has been done using the multinomial naive bayes classifier @xcite . @xcite focus on retrieving a given english query or keyword using gpt4 and retrieval-augmented generation (website of kanitamil24 conference). use of rhetorical structure theory has been proposed for discourse relation between phrases @xcite . in general the above approaches focus on generating the meaning for the entire couplets. whereas, word-by-word interpretation (in modern tamil) is mainly necessary for a deep understanding of the text especially for children or new readers. ## related work to retrieve word semantics for various downstream tasks, many state-of-the-art systems rely on word embedding models. among these, highly efficient systems include word2vec, introduced by @xcite , is a neural network-based model for learning distributed representations of words. it comes in two architectures: continuous bag-of-words (cbow) and skip-gram. both use a shallow neural network with one hidden layer. word2vec's key feature is its ability to capture semantic and syntactic relationships between words in vector space, where similar words cluster together. next, fasttext architecture introduced by @xcite is an efficient text classification and word representation model that extends word2vec by treating words as bags of character n-grams. fasttext's key innovations are hierarchical softmax and n-gram embeddings. this approach incorporates subword information, enabling the generation of embeddings for out-of-vocabulary words. these features make fasttext particularly effective for languages with rich morphology and texts containing numerous rare words or neologisms, as it can generate meaningful representations for words not encountered during training. however, a key limitation of both word2vec and fasttext is that they extract the meaning of words without considering the context in which the word is used. this results in static word embeddings, where each word has a single vector representation, regardless of the varying meanings it may take in different contexts. this lack of contextual understanding reduces their ability to capture the nuances of meaning that arise from the word's surrounding text. in other direction, vulic and moens (2015) introduce cross-lingual information which is embedded within a single vector space, allowing two different languages to be represented and handled together. this enables the model to map words from different languages into a common semantic space, making it possible to extract meaning across languages and facilitate tasks like cross-lingual information retrieval and translation. by aligning the representations of different languages, the framework allows for more efficient and accurate meaning transfer from one language to another. whereas our work focuses on different varieties of text within the tamil language, specifically classical and modern tamil words. this approach aims to capture the semantic relationships and nuances present in these distinct forms of tamil, allowing for a more comprehensive understanding and representation of the language's rich linguistic heritage. transformer architecture, introduced by @xcite , deals with the attention mechanism. self-attention allows the model to weigh the importance of different parts of the sequence when processing each element. the multi-head attention allows the model to attend to information from different representation subspaces. position encodings are added to input embeddings to retain sequence order information. these methods capture the contextual information of the token. one of the first discourse parsers for thirukkural, utilizing rhetorical structure theory (rst) has been introduced to uncover coherence between text spans through discourse relations @xcite . analyzing thirukkural poses additional challenges due to its irregular syntactic and semantic patterns, rich morphological variations, and unique poetic structure, making it more complex than prose. this approach enhances semantic indexing and retrieval, surpassing simple keyword searches for thirukkural content. the work by ramalingam et al (2022) classified thirukkural into ten new classes named superclasses, further narrowed down by classifying each superclass into two subclasses. multinomial naive bayes classifier has been used to retrieve the meaningful thirukkural for a query in the search system. use of text-to-speech conversion in tamil especially for tirukuural is available @xcite . an e-learning based speech recognition system to recognize and retrieve the meaning of kurals is designed to display meaning with chapter details @xcite and to synthesize the meaning as speech output. the work mentions challenges such as lack of a standard thirukkural speech database and building acoustic models for entire kurals rather than words/phonemes. the system is positioned as a tool to help students, visually challenged people, and others. aalamaram, the largest tree bank in tamil with nearly 10,000 sentences annotated, is available for the tasks of pos tagging, ner, morphological parsing, and dependency parsing @xcite . in the realm of thirukkural literature, various tools and applications have been developed to enhance access to the couplets and their meanings. numerous mobile applications provide the complete text along with translations and interpretations. additionally, several websites showcase the couplets alongside scholarly commentaries, while digital libraries like project madurai and tamil virtual academy offer translations and annotations to enrich users' understanding of this classical work. the aforementioned tools and applications significantly enhance the retrieval of thirukkural couplets and their explanations; however, they often lack a detailed word-by-word analysis, which is essential for a deeper understanding of the text. ## contextualized semantic learning for thirukkural words using unsupervised learning the vocabulary of the thirukkural is rooted in classical tamil, making it challenging for contemporary learners, including both students and people knowing tamil, to grasp its meaning. much of the analytical methodologies for language processing are seen for english language and very few exist towards tamil especially for thirukkural @xcite . the primary objective of this work is to identify modern, easily understandable tamil words that correspond to the original terms in thirukkural, thereby facilitating better comprehension. to achieve this, we employ an unsupervised learning algorithm rather than relying on traditional rule-based approaches or dictionary lookups. several state-of-the-art systems exist for retrieving word meanings , utilizing unsupervised learning algorithms such as word2vec @xcite , glove @xcite and fasttext @xcite . fasttext, in particular, has shown enhanced performance in capturing word semantics when compared to both word2vec and glove. in our proposed approach, we employ fasttext word embeddings for thirukkural word meaning retrieval.