Enhancing T elugu Part-of-Speech Tagging with Deep Sequential Models and Multilingual Embeddings (2023)

Abstract: Part-of-speech (POS) tagging is a fundamental task in natural language processing (NLP) that involves assigning grammatical categories to words in a sentence. In this study, we investigate the application of deep sequential models for POS tagging of Telugu, a low-resource Dravidian language with rich morphology. We use the Universal dependencies dataset for this research and explore various deep learning architectures, including Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Units (GRUs), and their stacked variants for POS tagging. Additionally, we utilize multilingual BERT embeddings and indicBERT embeddings to capture contextual information from the input sequences. Our experiments demonstrate that stacked LSTM with multilingual BERT embeddings achieves the highest performance, outperforming other approaches and attaining an F1 score of 0.8812. These findings suggest that deep sequential models, particularly stacked LSTMs with multilingual BERT embeddings, are effective tools for POS tagging in Telugu.

## introduction human communication, a crucial part of everyday interaction, largely hinges on language as a vessel for idea conveyance, emotional expression, and information dissemination. in line with advancements in information and communication technology, there has been a surge in the need for proficient tools to interpret and analyze languages, particularly low-resourced languages. one such language is telugu, a dravidian language predominantly spoken in the indian states of andhra pradesh and telangana, in which the nlp research is still in the infant stages, even for fundamental tasks such as part-of-speech (pos) tagging @xcite . among the various fundamental nlp tasks, pos tagging plays a crucial role @xcite . by at-tributing grammatical categories to words, pos tagging allows computational systems to extract vital syntactic and semantic information from linguistic data. this task catalyzes numerous downstream applications, such as machine translation, text summarization, sentiment analysis, and information retrieval, underscoring the importance of accurate pos tagging @xcite . the significance of pos tagging and the lack of resources and state-of-the-art models in the telugu language motivated us to study the application of deep learning algorithms to build pos tagging for telugu. in this work, we endeavour to enhance the efficiency and preciseness of pos tagging for the telugu language @xcite , bolstered by the deep learning algorithms designed for the sequential data. we used the dataset provided by the universal dependencies foot_0 in this research. the sequential dependencies in the data were captured using sequential deep learning models -recurrent neural network (rnn), long short-term memory (lstm) networks, gated recurrent unit (gru) and their stacked variants. at the core of our methodology is bert (bidirectional encoder representations from transformers) @xcite , a contextual language model. bert was used for generating the input representation for the telugu words, which were fed into the model as input. the advantage of using bert is that it operates by considering the entire context of a word within a phrase, efficiently encapsulating word dependencies. we used bert-base-multilingual-cased embeddings with this idea in mind so that it can improve the accuracy of the pos tagging. in this work, we considered multilingual bert @xcite and in-dicbert @xcite to generate the embeddings for the telugu words. from the experiments, it is observed that the stacked lstm with multilingual bert achieved the highest accuracy and f1 score. the rest of the contents of the paper are explained in the following sections: section 2 presents the related works. dataset description is provided in the section 3. section 4 explains the methodology followed by the results in section 5. the work is concluded in section 6. ## related works while there have been pos taggers developed for indian languages like hindi, bengali, and tamil, there is no publicly available dataset other than the universal dependency dataset, unlike several foreign languages such as arabic, english, and various european languages, which have a more extensive range of pos taggers. it is especially noticeable in the case of low-resource dravidian languages like telugu. the unavailability of a huge gold-standard corps hindered the research in developing computational pos tag models for telugu. in (antony and soman, 2011), antony p j and soman conducted a comprehensive survey focusing on the evolution of various pos tagger systems and pos tagsets for indian languages. their analysis encompassed the existing methodologies employed in developing pos tagger tools. their findings led to the conclusion that the majority of indian language pos tagging systems currently in existence predominantly rely on statistical and hybrid approaches. a malayalam morphological analysis @xcite was conducted using the deep learning models to identify the morphemes automatically which obtained an accuracy of 98.16%. in their study, prabha et al. @xcite described a sequence-to-sequence approach to model the problem using various deep learning algorithms, including recurrent neural networks (rnn), long short-term memory (lstm) networks, gated recurrent units (gru), and their bidirectional variations. it was observed that the bidirectional versions of rnn, lstm, and gru exhibited superior performance, with bidirectional lstm (bi-lstm) surpassing the rest by achieving an impressive accuracy rate of 92.66%. sunil et al. @xcite conducted a comprehensive study on verb generation in malayalam. their research emphasized the importance of morphological analysis and synthesis, adopting a paradigmbased approach that considered various critical mor-figure 1 : a sample of the input word sequence and corresponding tag sequence phological features, including tense, transitivity, intransitivity, causativity, gerund, aspect, modality, voice, causative, transitive, intransitive, and nonfiniteness. the study centred on 55 paradigm categories, effectively categorizing 6,700 verbs and facilitating their synthesis. symbolic mapping rules were also integrated to enhance the synthesis process. the paper @xcite tackles tamil pos tagging, a complex task given the scarcity of resources and agglutinative nature. it serves as a foundational step in nlp. the study deploys deep learning models like rnn, lstm, gru, and bi-lstm at the word level. evaluation employs metrics such as precision, recall, f1score, and accuracy, using a dataset comprising 32 tags and 225,000 tamil words. the findings show that increasing the hidden state parameter enhances model performance, with bi-lstm featuring 64 hidden states achieving the highest accuracy (94%). ## dataset description we utilized the universal dependency (ud) telugu dataset, divided into three segments: training, validation, and testing. the training dataset consists of 1,051 sentences, the testing dataset contains 146 sentences, and the validation dataset contains 131 sentences. the maximum sentence length in the training dataset is 20 words. in the testing dataset, it is 14 words, and in the validation dataset, it is 11 words. the average sentence length across all three datasets is five words. for our first experiment, a text-to-sequences model in which the input sequence is directly passed to the neural network, we combined the training, testing, and validation datasets into a single, large dataset to address out-of-vocabulary (oov) token issues. however, for bert models, we maintained the original dataset separation, as they do not encounter oov token problems due to their pretraining on extensive data. the figure 1 displaces the sample of the dataset used. ## methodology the methodology of this research comprises four main stages: data sourcing and preprocessing, sequence generation, generating the embedding using bert models and application of sequential deep learning networks. figure 2 illustrates our methodology for developing the pos tagger. in the first stage, we collected the ud telugu dataset, primarily developed for dependency tree creation. the dataset consists of dependency relationships and morphological information along with the pos tags. the first process was to collect the words and the corresponding pos tags from the dataset. the dataset was prepared in conll format, and we transformed it to sequence-to-sequence format. this format was chosen because the sequential model can take the input one at a time sequentially and generate the corresponding tag by considering the context into account. in all the experiments, the first task was to tokenize the input sequence into tokens. in the first experiment, the tokens were transformed into an index and the index sequence was fed into the deep sequential models. in the other experiments, the words were converted into their vector representations before feeding them to the deep sequential models. tokenizing the input sequence into tokens was the initial step in every experiment involving the text-to-sequence model. after these tokens were converted into an index, the deep sequential models were given the index sequence. this method makes it possible for our models to comprehend the text's sequential structure, which makes it easier for them to learn about the relationships between words and the pos tags that belong to them. in the subsequent stages, we deploy multiple deep learning models. through complex computations, these models interact with the sequences to decode the linguistic aspects of telugu texts. in parts of speech tagging applications, where word order and context are critical, rnn, lstm, gru and other designs to capture complex temporal correlations in textual data are advantageous. by utilizing the advantages of these various models, we can obtain a more sophisticated understanding of the language, which paves the way for combining the multilingual bert and indicbert models with rnn, gru and other models. bert offers bidirectional context-based insights by understanding words that appear before or after a word to assess the word's entire context. we incorporate in-dicbert, created for indian languages, including telugu, to achieve deep linguistic understanding. this model accurately represents the distinct nuances and intricate grammatical constructions of the telugu language. these models are individually and collectively optimized for pos tagging in telugu texts through experimentation and hyperparameter tuining, resulting in a model that can handle the complex grammar and language structure of telugu with a considerable accuracy. ## experiments, results, and discussion this section discusses the experiments conducted and the results of the various deep learning models to tag telugu words with text-to-sequence, bertbase-multilingual-cased and indic-bert. all algorithms are trained till 100 epochs and implemented early stopping with a patience setting of 3 during the training process. adam optimizer is utilized for optimization purposes and used categorical cross entropy as the loss function. for the stacked algorithms 2 layers of the respective algorithms are used in the architecture, and an embedding dimension of 300 is used. tables 2 and 3 display the results of experiments conducted using bert models. in it can be observed from based on the results obtained, within the group of models including text-to-sequence, bert-basemultilingual-cased, and indic-bert, indic-bert displayed the least effective performance in comparison to the other models. both the text-to-sequence and bert-base-multilingual-cased models demonstrate accurate classification of parts of speech tags. an example output from the bert-base-multilingualcased model is illustrated in figure 3 . ## conclusion in conclusion, the work proposed exemplifies the potential of multilingual models to enhance pos tagging for the telugu language. by addressing the challenges faced by low-resourced languages, we contribute to the broader field of nlp and pave the way for more inclusive and efficient communication technologies. based on the experiments conducted the bert-base-multilingual-cased model has outperformed the other models. as the nlp community continues to evolve, we look forward to further advancements in linguistic diversity and improved accessibility for all. by harnessing advanced deep learning techniques, we aspire to build upon the foundation established by this research, ensuring that linguistic inclusivity remains at the forefront of our shared objectives.