Parts of Speech ( P o S ) and Universal Parts of Speech ( UP o S ) Tagging: A Critical Review with Special Reference to Low Resource Languages (2023)

Abstract: Universal Parts of Speech (UPoS) tags are parts of speech annotations used in Universal Dependencies. Universal Dependency (UD) helps in developing cross-linguistically consistent treebank annotations for multiple languages with a common framework and standard. For various Natural Language Processing (NLP) tasks and research such as semantic parsing, syntactic parsing as well as linguistic parsing, UD treebanks are becoming increasingly important resources. A lot of interest has been seen in adopting UD and UPoS standards and resources for integrating with various NLP techniques, including Machine Translations, Question Answering, Sentiment Analysis etc. Consequently, a wide variety of Artificial Intelligence (AI) and NLP tools are being created with UD and UPoS standards on board. Part of Speech (PoS) tagging is one of the fundamental NLP tasks, which labels a specific sentence or set of words in a paragraph with lexical and grammatical annotations, based on the context of the sentence. Contemporary Machine Learning (ML) and Deep Learning (DL) techniques require god quality tagged resources for training potential tagger models. Low resource languages face serious challenges here. This paper discusses about the UPoS in UD and presents a concise yet inclusive piece of literature regarding UPoS, PoS, and various taggers for multiple languages with special reference to various low resource languages. Already adopted approaches and models developed for different low resource languages are included in this review, considering representations from a wide variety of languages. Also, the study offers a comprehensive classification based on the well-known ML and DL techniques used in the development of part-of-speech taggers. This will serve as a ready-reference for understanding nuances of PoS and UPoS tagging.

## introduction natural language processing (nlp) is a complex procedure of stages of computational modules, and it belongs to the artificial intelligence field. as human languages are inherently ambiguous, it is incredibly challenging for machines to comprehend. therefore, researchers have come up with various tools and techniques both for generation, as well as understanding/processing natural languages both for text and speech forms. nlp research includes various tasks to improve the machine understanding/generation processes such as machine translation, information retrieval, information extraction, questionanswering, speech synthesis and recognition etc. to carry out these tasks, parts-of-speech (pos) tagging or grammatical/lexical annotation is an important and fundamental requirement, as most of the nlp tasks involve rigorous dependencies on syntactic characteristics. the term "parts of speech" refers to a word's grammatical characteristic and how they relate to other words in a sentence. there are eight main parts of speech in the english language, including the noun, pronoun, verb, adjective, adverb, preposition, conjunction, and interjection. with the exception of the word classes "noun" and "verb," which are present in practically every language, there are substantial differences in the parts of speech among different languages. for instance, in assamese, an indo aryan language, there are eight parts of speech given in the standard grammar books namely noun, pronoun, adjective, adverb, verb, preposition, conjunction and interjection, but some other categories are also there such as postposition, auxiliary verb, pronominal, numeral, subordinating, punctuation, symbols, unknown, particles etc. which are required to have a proper hold on the assamese grammar and on the parts of speech. universal parts of speech standard is defined as the core parts of speech categories alongwith a set of universal features used to distinguish additional lexical and grammatical properties of words. there are 17 core tags in upos, shown in ## types of pos taggers and approaches there are two types of pos tagging: supervised and unsupervised. rule-based, stochastic, transformation based, and memory based fall under supervised learning, and neural categories including ml and dl fall under unsupervised learning. i. rule-based tagger: this tagger uses dictionary or lexicon where words are stored alongwith the syntactic information. if a word has more than one possible syntactic categories, the grammar based hand-crafted rules are used to find the correct tag. it is the oldest technique in tagging. ii. statistical tagger: this tagger is also known as stochastic or probabilistic tagger as it selects the most probable sequence of tags from a text corpus, based on past statistical evidences. iii. memory-based tagger: in memory based, a set of cases are stored in memory. each case contains the word, its context and suitable tag. it is a combination of rule based and stochastic method. iv. transformation based tagger: this is also a rule-based tagging method, but rules are applied in multiple cycles. transformation based tagging is also called brill tagging. v. neural tagger-it consist of series of algorithms that recognizes the relationship in the dataset through a process that mimics the way human brain works. ## universal dependencies and upos prior to the 17 pos tags, there were 12 pos tags in the universal parts of speech: noun, verb, adjective, adverb, pronoun, determiner, adposition, numeral, conjunction, and particles. a project called universal dependencies (ud) uses universal parts of speech tags to create treebank annotation that is consistent across many different languages. the annotation scheme is based on an evolution of (universal) stanford dependencies @xcite . they have opted unsupervised approach to evaluate their crosslingual parts-of-speech projection system for six different languages. they have also proposed 12 universal pos; using those tagset and mapping, they compared pos tag accuracies for 25 different treebank to evaluate pos tagging accuracies on a single tagset. secondly, they combine the grammar induction system of @xcite with their cross lingual project pos tagger. sampo pyysalo et al. @xcite have presented universal dependencies (ud) for finnish. they have mapped according to the previously introduced annotation to the ud standard. parsing experiment comparing the performance of a state of-the-art parser trained on a language specific annotation to perform on the corresponding ud annotation was also done. a multilingual multitask model called udify, created by dan @xcite is able to accurately anticipate universal pos. in order to benefit the low resource languages by the cross-linguistic annotation, they evaluated udify for multilingual languages. udify could generate ud annotations automatically for languages that allow ud. it also offers the best results in dependency parsing; however, it gives inaccurate results for the lemmas and universal features. in order to better understand how multilingual training produces ud predictions even for languages for which udify and bert models are not trained, they have also evaluated zero shot learning. they observed a little improvement in the bert model that was trained on 104 languages before being fine-tuned across all datasets. in their study, @xcite tested the effectiveness of anticipated upos tags as inputs for dependency parsers using a low resource universal dependency treebank with varied tree bank sizes. predicted upos tags have been proven helpful for treebanks with few resources, but as data volumes expand, their beneficial effects diminish. they examined @xcite and discovered that low resource parsers performed poorly even when using gold standard pos tagged data. even with cross-lingual approaches and a low resource setting in @xcite , the tagger performance is unsatisfactory. the transitionbased parser for multilingual universal dependence (ud) demonstrates a slight improvement by utilizing universal pos tags (upos). ## taggers for various languages indian languages are morphologically rich, agglutinative and ambiguous, resulting challenges regularly encountered while assigning correct tags to the words as per the sentences, as the words may behave differently in different context. an awngi language part of speech tagger utilizing the hidden markov model was proposed by @xcite . awngi is very low resource ethiopian language. they gathered 94,000 sentences and used 23 individually made tags. performance of the awngi hmm pos tagger was assessed using a tenfold crossvalidation mechanism. the empirical finding demonstrates that uni-gram and bi-gram taggers achieve tagging accuracy of 93.64% and 94.77%, respectively. atmakuri et al. (2018) have discussed different combination of features on different size corpus to develop a crf based pos tagger. they have achieved maximum accuracy of 89% on their largest corpus. they have compared different taggers developed by researchers for kannada language. for kannada, antony p. j and soman k.p. @xcite created a pos tagger with 30 tags using support vector machines (svm). using the 25 tagset pos tags produced by @xcite compared two probabilistic models (hmm and crf) on the emille dataset. crf model was the best, with 84% accuracy, they found. without employing a probabilistic model, m. c. padma and r. j. pratibha (padma and pratibha, 2016) created another pos tagger for kannada. they employed a rule-based strategy and used the bis dravidian tag set. additionally, they ran four small datasets on a tiny fraction of the emille corpus to evaluate their model, and they obtained an average precision of 88.75%. using crf for pos tagging, k.p. pallavi and anitha s. pillai @xcite created a tagger for kannada that achieves a precision of 92.4%. @xcite have developed a unique code-mixed pos-tagged dataset of english and hindi tweets. they have also claimed that their dataset is larger than the previous annotated dataset and closely resemble real world tweets. additionally, they have described how their pos tagged dataset can be used by developing and evaluating an automatic pos-tagging model. a set of self-made features are presented which is used by the model to predict pos tags of a token. they have developed a sequence labelling task pos tagging using conditional random field and lstm recurrent neural networks. the outcome of models are-pos crf achieves f1 score of 90.20% and pos lstm achieves f1 scores of 82.51%. they also observed that above models show low performance for pos tags when it comes to hindi adjectives and adverb, as there are different grammatical rules for hindi language. tham (2020) has developed a hybrid tagger for khasi language integrating hmm (hidden markov model) and crf (conditional random fields). as khasi is an under resourced language without crf integration which incorporates the language features, it is not feasible only with hmm tagger, they observed. while developing the khasi pos tagger, hmm approach was used, and along with that, a tenfold cross validation was also carried out to check the performance. as per available sources on khasi language, two hmm pos taggers were used so far and trained and tested on two different datasets. the first dataset consisting of 86,087 tokens where hmm tagger was trained; it gave accuracy of 95.68% whereas the second one gave accuracy of 76.7% with 7,500 words and custom made tagset of 54 tags. here the corpus consist of 94,651 tokens and while performing the ten-fold cross validation the baseline tagger gave accuracy of 84.05%, the nltk tagger gave accuracy of 87.58% and hmm pos tagger gave 93.39% which is the best among the three. @xcite evaluated different taggers on different data size. the output of all the taggers are compared on the basis of accuracy and total time required for training data. some taggers give good performance in terms of accuracy and some in terms of computation times. straka and straková (2020) have created a system that uses lemmatization and pos tagging and is built on udpipe 2.0. additionally, they evaluated the bert and xlm-roberta, and their impact on the contextualized embedding and treebank encoding, contributing to the eva latin shared job. some of the related task similar to evalatin shared task has also been discussed, such as sigmorphon2019 hard task @xcite worked on 107 different corpora and 66 languages, work by @xcite and multilingual parsing from raw text to universal dependencies shared tasks etc. here, an overview of the employed architecture is covered. the multitask architecture is built on udpipe 2.0. the embedding input words are processed through three shared bidirectional lstm layers, and the output is subsequently processed by softmax classifiers, which produce lemmas and pos tags. all three ud 2.5 latin treebanks are trained for treebank embedding. to compare bert with xlm roberta, an ablation experiment was run. bert embedding demonstrates only a slight improvement in accuracy, although xlm-roberta performs better. @xcite suggested utilizing bi-lstm for pos tagging for arabic gulf dialect. for the goals of sequence modelling, the support vector machine (svm) classifier and bi-directional long short term memory (bi-lstm) machine learning methods are used. with the use of a bi-lstm, the pos tagging model was enhanced from a 75% state-of-the-art accuracy to over 91% accuracy for gulf dialect. additionally, they build a dataset for pos tagging and several sets of characteristics to test the models. @xcite in their paper have discussed about malayalam tweets pos tagging. they have considered 17 coarse tags and 9915 tweets were tagged manually and evaluated by using the methods of deep learning such as recurrent neural network (rnn), gated recurrent units (gru), long short term memory (lstm) and bidirectional lstm (bilstm). theano framework, which is a library of deep learning, was used for the experiment and evaluation. after model training and experiments at word level, gru based deep learning gave the highest f1-measure of 0.9254 at character level, blstm based sequential model gave f1 measure of 0.8739. they have found that with increase in the hidden state there is improvement in the state of the tagger. the authors have mentioned that foregoing task is difficult to understand and to natural language processes such as parsing, pos tagging, named entity recognition seems to be challenging as the task differs in lexical, syntactic and orthographic patterns. initially tweeter pos tagging work in english was done by @xcite where a coarse pos tagset was designed to show the main parts of the speech. @xcite presented a tagger for chhattisgarhi language. the tagger is a hybrid tagger with combination of rule based techniques and statistical techniques. the tools used for database findings are statistical and conditional random field approach. their system has two parts-first searching the word in the database and second is if the word is found then tag it. they have achieved the accuracy of 70% and if the words present in the sentences are also present in the database then the accuracy is higher than 90%. @xcite have developed a pos tagger for nepali language, which belongs to indo-aryan family. their approach was deep learning and the methods like -rnn, lstm, gru and bi-directional variants were used. the work of @xcite shows that high tagging accuracy is required to show effective performance of upos tags as features for neural parsers. the author here also did analysis on the effect of upos accuracy on the parser performance. the impact of pos tag on parsing performance considering different ud treebanks using gold pos tags and also predicted pos tags were evaluated. it has been observed that using gold pos tags results more accuracy in predicting. here the author has considered various ud treebanks such as catalans, ancora, japanese gsd etc. and pos taggers are trained on them using sequence labelling framework ncrf++ @xcite to find different accuracies. also, two dependency parsers were trained with and without predicted pos tags. specifically, deep belief networks and the deep learning methodology are used to create a bengali pos tagger. using the corpus, they have developed a word dictionary for pos tagging. the pos tagging dictionary can reduce the ambiguity of tagging procedures. on the corpus, the deep learning-based bengali pos tagger achieves an accuracy rate of 93.33% @xcite . a dl-based pos tagging approach for bengali @xcite utilized the language's suffixes. an experiment is carried out using a labelled corpus of 2927 words. the accuracy of the suggested dl-based pos tagging model was 93.90%. additionally, the deep learning model outperformed earlier models like rule-based and global linear models in terms of accuracy. additionally, the suggested model is integrated into the free and open-source bengali nlp toolbox written in python. a hmm based tagger was also developed by @xcite and they have used indian language standard pos tagset, and also trained their program using 5000 sentences from the tourism domain. also, another chunk of 4500 sentences were later on combined with the corpus to prepare the system. the hmm could pick the tag for a word by looking the pre and post words. they have achieved accuracy of 85.40%. lohe and pandey (2020) in their paper described the pos tagging processes using rule based approach. the author has discussed the rule-based parts of speech tagger for hindi language. they have also reviewed various papers based on hindi pos taggers. @xcite have designed a pos tagger for marathi language using trigram method of statistical approach. the model provides automatic tagging of marathi words and to derive the best sequence it checks the probability values of the previous two tags. @xcite have discussed about pos tagging for hindi language. their approach was hmm and they have used hindi wordnet dictionary. the performance analysis was done considering the parameters such as precision, recall and f1 measure. they have obtained precision -93.17%, recall-96.46%, fmeasure-90.13%. @xcite have focused on developing a pos tagging system considering hindi and bengali tweets. also, they have discussed about social media text (smt) and the challenges they have faced as smt has different writing practices. a pos tagger for english tweets was developed by @xcite , and then combining it with indian language standard pos tagset (ldc-il) the author have designed coarse grain pos tagset. their corpus consists of 3488 tweets. for annotating their corpus, they opted for the popular and fastest crowd source service of amazon mechanical turk but the outcome was poor, then they tried bootstrapping. they have tried various ml experiments on hindi tweet pos tagging and they also proposed to work for developing pos tagger for code-mix smt. @xcite in their paper presents a pos tagger for hindi language using hybrid approach. initially the authors have tagged around 1 lakh unique class category of words with the help of wordnet dictionary. for some untagged words, rule based approach is used to assign tags to the words. to remove ambiguity, hmm model is used as a statistical approach. also, the corpus is evaluated using seven standard parts of speech tags and the accuracy rate is 92%. the pos tagging identifies the lexical categories of hindi words present. the corpus consists of 1000 sentences and 15000 words. for malayalam, a dl based pos tagger is suggested @xcite . in order to develop the pos tagger, the studies make use of the gated recurrent unit (gru), long short-term memory (lstm), recurrent neural networks (rnn), and bi-directional long short term memory (bilstm). when compared to earlier models, the suggested model performed better. this results in the model's precision, recall, and f-measure being 0.9878, 0.9788, and 0.9832 respectively. turkish deep neural network language models were suggested as a solution to the pos tagging issue. long short-term memory (lstm) and recurrent neural network are used in the experiment (rnn). there is a performance comparison with cutting-edge techniques. according to the experiment's findings, lstm outperforms rnn with an fmeasure metric of 88.7% @xcite . @xcite have used hybrid approach for parts of speech tagging in malayalam language. the rules set consist of 267 rules and is applied with morph analyzer. they have achieved accuracy of 90.5%. since relatively little work has been done on the digitalization of the assamese language corpora, @xcite have created an assamese pos tagged corpus for enhanced language comprehension. the assamese tagset they used consist of 31 tags incorporating bis tagset and 10000 words of assamese text have been tagged manually. the customized tagset consists of 31 tags with 11 top-level categories and corresponding subtypes. in their system, an assamese model was created using a manually tagged dataset as input to the tnt tagger, and an assamese parser was created using cfg and nltk to produce the parse tree. the top-down, recursive-decent parsing method used by the parser is created by a series of recursive procedures. another paper by @xcite have discussed the various approaches for parts of speech tagging used by the researchers and also have given a brief overview of the computational works done by them for the upliftment of the assamese language in terms of language technologies. early's parsing method was used in the work "parsing of parts of speech tagged assamese texts" by @xcite . they have created a method that uses assamese sentence structure analysis to create grammatical rules. bipul et al. discussed their work to develop an effective syntactic analyzer and annotated corpora for the assamese language's rich morphological features. @xcite created an assamese pos tagger using crf++ and fntbl in their study. their accuracy rate when utilizing crf++ was 67.33%, and when using tbl, it was 87.17%. @xcite in their paper have discussed about developing a pos tagger for assamese language, using stochastic approach based on hmm model. the model is trained on 256,690 words, and their system gives accuracy of 89.21%. assamese language pos tagging issues, as well as different pos tagging methodologies and assamese language characteristics, have been examined by @xcite . a comprehensive review is also done by @xcite for different approaches and techniques applied for pos tagging of indo aryan languages. ## summary the majority of researchers have preferred deep learning (dl) methods over the past ten years while constructing pos tagging models, according to the works we have reviewed. it is observed that next frequent solutions use hybrid approaches by fusing machine learning and deep learning algorithms, and rest of the pos tagger models are implemented based on standalone machine learning techniques and conventional rules based and statistical approaches. this review study provides interesting and new researches information with up-to-date knowledge, recent researcher's inclinations, and development of the field by providing a thorough assessment of the part of speech tagging approaches based on deep learning (dl) and machine learning (ml) methods. though for indian languages rule-based, transformation based, supervised and unsupervised approaches gives good performances but it has been observed that the recent state of art models viz. hmm model, deep learning models will be suitable for further research in the upos based tagging. the challenging task, which was encountered while developing a tagger, is the language ambiguity and many languages are still low resource language. our future work is to develop a pos tagger giving better performances for assamese language.