Part of Speech Tagging for a Resource Poor Language : S indhi in D evanagari Script using HMM and CRF (2021)

Abstract: Part of speech tagging is a pre-processing step of various NLP applications. Mainly it is used in Machine Translation. This research proposes two POS taggers, i.e., an HMM-based and CRF based tagger. To develop this tagger, the corpus of manually annotated 30,000 sentences has been prepared with the help of language experts. In this paper, we have developed POS taggers for Sindhi Language (in Devanagari Script), a resource poor language, using HMM (Hidden Markov Model) and Conditional Random Field (CRF).Evaluation results demonstrated the accuracies of 76.60714% and 88.79% in the HMM, and CRF, respectively.

## introduction the main aim of nlp is to create suitable algorithms for computers to understand human language. these language technology tools help the users in translation and understanding of languages. for example, with the help of machine translation, a person can communicate or share information in their native language. although plenty of different language processing tools are available for some languages, still some of the languages have not been able to attract the attention of the research community. the sindhi language is one of them. there are 22 official languages of india. sindhi is one of them. in 1967 it was recognized as an official language of india. the eighth schedules of the constitution of india includes the languages which are resourcepoor and need to preserved and developed. sindhi was included in this schedule in the 21 st amendment. although in india, sindhi is officially recognized, but it is not an official language of any of the states in india. the "international education system" puts a big problem for sindhi people. the upcoming sindhi generation is unable to write and speak the sindhi language, as their parents do not communicate with them in the sindhi language. this may lead to the extinction of the sindhi language. for language preservation, our prime need is to save its speakers by developing some language processing tools which help them to learn sindhi. in the last few years, some work has been done in the sindhi language in arabic script. sindhi devanagari is more resource-poor than sindhi arabic. in this research work we have developed a slp (sindhi language processing) tool i.e., pos part of speech tagger in devanagari script. the words can be classified in various lexical categories, such as nouns, verbs, etc. these categories are also known as parts of speech. parts of speech define their morphological and syntactical behavior. pos tagging is a task of classifying each word in a corpus to a given syntactic class such as noun, verb, etc. a word can belong to more than one lexical category, depending on its use in a sentence. the main objective of the pos tagging process is to remove this ambiguity. tagger uses the contextual information to assign the tag. part of speech tagging is used in various applications of nlp, such as machine translation, information retrieval, information extraction, spelling correction, and word sense disambiguation. this paper presents the development of two automatic taggers using conditional (crf) and hidden markov model(hmm) . ## related work pos tagging is assigning the syntactic or lexical category to a word in a sentence. pos tagging is a ## the approach pos tagging approaches are broadly classified into rule-based, stochastic, and hybrid approaches. in the rule-based approach, handwritten disambiguation linguistic rules are used for tagging. stochastic is also known as a data driven approach, which requires pre tagged corpus for training. hybrid is a combination of rule-based and data driven. to develop a pos tagger, we have chosen the stochastic approach. this is a data driven approach. rule based approach is time consuming and needs language expertise to write the rule. for morphologically rich language, it is impossible to write all the rules stochastic approach is a probability-based approach. we have used two standard algorithms hmm (hidden markov model) and crf (conditional random field) for pos tagging. hmm is an example of a generative model, whereas crf is an example of discriminative model @xcite ). for a particular data set, we cannot predict in advance which model will give the correct results. each type of model is having its own limitations and delimitation. ## corpus annotation the stochastic approach is data driven. this requires the manually annotated corpus for training. the stochastic approach gives better results when the manually annotated corpus is used for training. in this sequence, we have manually annotated the corpus of 30000 sentences by using the guidelines described by @xcite . for tagging, we have used the il tag set. a tag set is a collection of tags used by a tagger. the tag set is described in the following tables: ## pos tagging using hmm for a given input sentence, we can calculate the best tag sequence using the following formula: where c(ti-1,ti) is the counting of how many times tag ti, comes after tag ti-1 (previous tag). to calculate the emission probability: p(w/t) = p (w1/t1) * p(w2/t2) . . . p(wi/ ti) * . . . p(wn/tn) (4) ∏ 𝑖=1 𝑝(𝑤 𝑖 𝑡 𝑖 ) 𝑛 (5) ) ( ) , ( ) / ( i i i i i t c t w c t w @xmath0 where c(wi/ ti) calculate the probabilities , that a given tag ti is associated with given word wi.hmm algorithm chooses the most likely tag sequence with the help of a decoding algorithm, i.e. viterbi algorithm. ## pos tagging using crf crf was introduced by @xcite .crf is a discriminative undirected graphical model that belongs to the family of condition distribution. crf is the most popular method used for structured prediction in the nlp task. in the discriminative approach, for a given input x and output y, the probability is calculated directly p(y|x) whereas in the generative model joint probability p(x,y) is generated. structured means that the output of an algorithm is a structured object such as a tree or a sequence. crf is used for the pos tagging task. this discriminative model x for a given observation sequence @xmath1, o2, o3…. @xmath2, and state sequence @xmath3, s2, s3……@xmath4. conditional probabilities are calculated as: in the above equation, fk is a transition feature function, which is learned via input or observation sequence. the weight of this function is k which defines the weight which is learned in training. the normalization factor z0 is calculated using the following formula: this makes all conditional probabilities sum equal to 0. we have used crf++ 1 for training and testing our tagger. three files are required for crf implementation.1 training file 2. testing file 3. template file. the complete corpus is divided into 80-20 ratio, where 80% is used for training data, and 20 % is used for testing data. crf model is developed in three steps: ## conclusion and future work pos tagging is an important prerequisite task for any nlp research. in this research work we have developed two taggers for a resource poor language sindhi in devanagari script, using stochastic approach. for this we have used 1.hmm and 2.crf. first we have manually annotated the corpus of 30000 sentences. we have got accuracy of 76.60714% and 88.79% for hmm and crf respectively. sindhi is a morphologically, rich language. the various morphological features such as prefix, postfix and word length will help in defining the pos of a word. crf can incorporate all these features in the model, and this is one of the main advantages of crf over hmm. in future to handle the exceptional cases, we could merge the rule-based tagger with the statistical approach. we could use various other machine learning approaches such as svm to develop the pos tagger. in addition we can develop other tools such as named entity recognizer (ner) which will increase the accuracy of tagger. a robust tagger could be developed using an ensemble approach.