Transformer-based Context Aware Morphological Analyzer for T elugu (2023)

Abstract: This paper addresses the challenges faced by Indian languages in leveraging deep learning for natural language processing (NLP) due to limited resources, annotated datasets, and Transformer-based architectures. We specifically focus on Telugu and aim to construct a Telugu morph analyzer dataset comprising 10,000 sentences. Furthermore, we assess the performance of established multi-lingual Transformer models (m-Bert, XLM-R, IndicBERT) and mono-lingual Transformer models trained from scratch on an extensive Telugu corpus comprising 80,15,588 sentences (BERT-Te). Our findings demonstrate the efficacy of Transformer-based representations pretrained on Telugu data in improving the performance of the Telugu morph analyzer, surpassing existing multi-lingual approaches. This highlights the necessity of developing dedicated corpora, annotated datasets, and machine learning models in a mono-lingual setting. We present benchmark results for the Telugu morph analyzer achieved through simple fine-tuning on our dataset.

## introduction a morphological analyzer is a valuable tool in natural language processing (nlp) that analyzes words by breaking them down into constituent morphemes. it provides crucial grammatical information, including gender, number, person, case markers (gnp), tense-aspectmodal information, and other linguistic features, which are indispensable features for understanding the morphology of a given language @xcite . many agglutinative languages have these grammatical features as part of their words. hence, building a morph analyzer that parses and provides this information is important. this expansion would greatly benefit various nlp tasks and applications tailored to indian languages. this work aims to develop a transformer based context-aware morphological analyzer for telugu. telugu is known for its agglutinative nature, and various affixes were attached to the root word to convey different grammatical meanings. nouns and pronouns in telugu are inflected for gender, number, person and case markers, followed by clitics. verbs exhibit extensive inflections based on tense, aspect, mood and agreement features such as gender, number, and person (gnp). additionally, telugu also uses productive derivational suffixes, where nouns are converted into an adjective by the addition of the suffix -ayna and an adverb by the addition of the suffix -gā and by the addition of the suffix -ad . am allows verb roots to function as gerunds, thereby allowing for noun inflections @xcite . the complexity of telugu morphology necessitates a robust morphological analyzer, which plays a crucial role in various nlp applications such as speech synthesis, information extraction, information retrieval, and machine translation @xcite . a morphological analyzer takes a single word in isolation and provides all possible analysis. although a word may have multiple valid analysis, when considering the context in which the word is used, often only one analysis is appropriate or meaningful. this is because the context helps determine the word's intended meaning, which can help narrow down the possible analysis. traditionally, morphological analyzers for indian languages have been rule-based. still, there is a recent shift towards utilizing machine learning techniques to build computational models with the development of transformer based models from scratch in telugu @xcite on 80,15,588 sentences. this shift leverages advancements in downstream nlp tasks in telugu like named entity recognition @xcite , sentiment analysis, emotion identification, sarcasm detection @xcite , clickbait detection @xcite , and summarization @xcite . also, we see it is important to explore and compare existing multi-lingual transformer based language models like mbert @xcite , indicbert @xcite and xlm @xcite with telugu transfer models (monolingual setting) like bert-te for the low-resource language, telugu. the main contributions of this paper are as follows: the rest of the paper is organized as follows: section 2 discusses the review and comparison of existing approaches. section 3 describes the preparation of the dataset. section 4 provides an overview of the experiment and evaluation of approaches followed using the dataset to train different models on the telugu morphological features. sections 5 and 6 discuss the ethical statement, conclusion, and future work. ## related work this section discusses the related work on building a corpus for morphological analysis focusing on indian languages, existing telugu bert models, and multi-lingual models. we also review the various common approaches used to build morphological analyzers. in the case of many indian languages, morphological analysis has traditionally used a rule-based approach. it can be helpful in linguistic research because they provide a framework for formal analysis and understanding of language structures and patterns. a telugu morphological analyzer @xcite is an example of organizing a linguistic database and employing computing resources effectively. the accuracy and coverage of this morph analyzer is 95-97%. this work is based on the word and paradigm approach @xcite . a set of morpho-phonemically different forms in their inflection and derivation processes are identified. the failure of the presence of a root word in the morphological dictionary decreases the accuracy of the morph analyzer because it cannot analyze the root word. so, it shows issues with the oov (out-of-vocabulary) and is not a contextbased-morphological analyzer. using the paradigm-based inflectional system and finite state systems to represent the language modelling, @xcite developed a highly accurate morphological analyzer for marathi. @xcite based on the unimorph schema or the universal dependency framework with the dataset contains 16527 distinct gujarati inflected words with their morphological segmentation and grammatical feature tagging information is annotated and evaluated using the baseline format. deep neural network-based models have recently been widely employed for building morphological analyzers. @xcite study discusses the malayalam morphological analysis as a characterlevel sequence labeling problem that has been achieved with deep learning architectures such as recurrent neural networks (rnn), long short-term memory (lstm) and gated recurrent units (gru). the model was trained using a 128embedding size. according to their results, gru has the highest accuracy score. @xcite studied the performance of different composite neural models for sanskrit morphological tagging. using neural architecture. @xcite built a lemmatizer for bengali and studied how it performed on the problem of word sense disambiguation. transformer-based language models like bert-te are available in telugu @xcite trained on 80,15,588 sentences. these representations resulted in downstream nlp tasks in telugu like named entity recognition @xcite , sentiment analysis, emotion identification, sarcasm detection @xcite , and clickbait detection @xcite , and summarization @xcite . ## dataset description this section elaborates on the dataset that is used to build our transformer-based morph analyzer. we used a collection of generic telugu corpus of 10,000 sentences as the basis for our work. detailed statistics of lexical categories of this dataset can be found in we used the lt toolbox foot_2 version of telugu morph analyzer, which is developed by @xcite . to identify the pos tags within sentences, we use an existing ilmt pos tagger @xcite . the pos tagger assists in determining the role of each word in the sentence. telugu morphological analyzer generates multiple possible analysis for a given word. to select the most appropriate contextual morph analysis, we used the same technique as mentioned in @xcite wherein we used the pos tagger that selects the relevant pos tag based on the pos category of the word. pos tagger provides the pos tag for the word in context, and then we prune out the multiple analysis in the morph based on the pos tagger's output, if any. for example, if a word has multiple morph analysis with different lexical categories, such as a noun or a verb, the pos tagger selects the noun or verb analysis according to the pruning output. if the pruning module fails, we resort to the pick-one morph strategy that selects the first analysis as the output for the word. however, it should be noted that errors in pos tagging can lead to mistakes in selecting the correct morph, thereby affecting the contextual awareness of the words in a sentence. manual validation is necessary to identify the errors in selecting contextaware morphological analysis. we have discussed specific errors pertaining to wrong gnp marking, incorrect case-marking, sandhi split errors and the like. the following examples explicate the errors: in example (1), pagalu is wrongly tagged as vm (verb main) pagalu 'to break ', instead of a noun (nn) pagalu 'day'. this lead to errors in further stages of processing. in example (2), the subject agreement of cēr-āru 'to join' can be analysed both as 2nd person (exclusive or honorofic pronoun (m īru 'you') and 3rd person. however, in the example (2), the subject vāl . l . u 'they' is the third person pronoun that resolves the ambiguity. it is noted that the morph analyser fails to provide the correct analysis in such cases. in telugu, not all nouns overtly show differences in direct and oblique case marking. one such example, as in (3), is noted in the corpus. the noun amma 'mother' here, when associated with another noun pani leading to a chunk amma pani 'mother's work', does not show any change in form. this leads to an error in the marking of the case for amma 'mother'. it is observed that the noun is marked with a direct case instead of an oblique case. ## methodology we first obtain the sentences with morphological tags for each word in the sentences, and then we feed those sentences to our language model to refine it using this dataset. we segregate the words' properties, such as lexical category, gender, and person, after acquiring the morphological tags for each word in the sentence before feeding them separately to the classifiers. in this section, we develop a comprehensive exploration of the different language models analyzed for the morph tag prediction study, elucidating their configuration in greater detail. in a later section of this article, we present results from language models that were examined in relation to several types of morphological tags. figure 1 depicts the overall process flowchart. ## ethical statement we created a dataset for the telugu morph analyzer and open source the dataset 4 . the codes can be downloaded from here. we reused publicly available telugu transformer models (bert-te) to compare with existing multi-lingual transformers models (indicbert, xlm-r, mbert). ## conclusion and future work understanding the structure of individual words is made easier by morphological analysis. in terms of morph information, we have produced a trustworthy dataset. various nlp tasks can now use this dataset. with the aid of the morph analyser, language models can effectively learn and utilize the additional details provided, enabling them to make more accurate predictions, generate more coherent and contextually appropriate responses, and better comprehend the subtleties of human language. by leveraging the insights from the morph analyser, language models become more efficient at processing and utilizing the available information, leading to improved language processing capabilities and more refined language generation.