NLP Tools for K hasi, a low resource language (2020)

Abstract: Khasi is an Austro Asiatic language spoken by one of the tribes in Meghalaya, and parts of Assam and Bangladesh. The fact that some NLP tools for Khasi are now available online for testing purposes is the culmination of the arduous investment in time and effort. Initially when work for Khasi was initiated, resources for Khasi, such as tagset and annotated corpus or any NLP tools, were nonexistent. As part of the author’s ongoing work for her doctoral program, currently, the resources for Khasi that are in place are the BIS (Bureau of Indian Standards) tagset for Khasi, a 90k annotated corpus, and NLP tools such as POS (parts of speech) taggers and shallow parsers. These mentioned tools are highlighted in this demonstration paper.

## introduction khasi can be categorized as a low resource language from language technology standpoint due to the fact that reported resources available for khasi such as a 90k annotated corpus is comparatively small in size (tham 2020b). however, this did not hinder the development of pos (parts of speech) taggers and shallow parsers for khasi where their performances are at par with performances of other reported taggers and parsers of other indian languages. these tools are now available for testing online in https://medaritham.pythonanywhere.com , where users can enter a sentence in khasi and observe and compare the tagging and parsing performances of various techniques employed in the tools. ## khasi bis tagset and annotated corpus due to the unavailability of any corpus in khasi, the required corpus has to be built from scratch ## pos taggers for khasi initially, a hidden markov model (hmm) pos tagger for khasi was developed and tested only on test data comprising of text from a book not seen during training and achieved an accuracy of 95.68% (tham 2018b). however rigorous testing of the hmm pos tagger was carried out using tenfold cross validation giving an accuracy of 93.39%. in order to address the tagging errors of the hmm pos tagger, a hybrid pos tagger for khasi was developed which reported an accuracy of 95.29% using ten-fold cross validation (tham 2020b). this was possible due to the integration of conditional random fields (crf) which allows the incorporation of language features not possible in an hmm pos tagger. the language features included for khasi are capitalization, prefixes, current word under consideration, previous word, next word, and whether a word begins or ends a sentence. due to the absence of inflection, prefixes nlp tools for khasi, a low resource language medari janai tham department of computer science and engineering assam don bosco university, assam, india medaritham16@gmail.com are prevalent in khasi exhibiting derivational morphology. an additional feature included is the previous tag of a word. both these pos taggers can be observed and compared online in the site mentioned earlier. one distinct difference is the ability of the hybrid tagger to differentiate proper nouns from common nouns a trait where crfs excel. ## shallow parsers for khasi to train a shallow parser for khasi, the annotated corpus has to be further tagged with noun and verb chunks using the bio labelling specified by @xcite where each alphabet symbolizes the following: b-xx: label b for a word starting a chunk of type xx. i-xx: label i for a word inside a chunk of type xx. o: label o for a word outside of any chunk. shallow parsing for khasi has been carried out in the lines of @xcite where they approached shallow parsing as a tagging problem utilizing the standard hmm approach for parsing without changing the training and tagging process. they have put forward a specialized hmm where adjustments have been made in the training corpus while the training and tagging procedure remains intact. this is carried out by embedding only relevant input information into the model and expanding the chunk tags with supplementary details without affecting the learning and tagging process. the parser is the first of its kind for the language, where the training corpus comprises of 24,194 chunks of noun and verb chunks out of a total of 3,983 sentences and 86,087 tokens. the full details of this process is given in (tham 2018a). this specialized hmm for khasi gave an f1 measure of 95.51. this rather optimistic performance is due to the fact that it was tested on gold data that was correctly tagged with pos tags. it was not tested on the output of the pos taggers mentioned earlier. hence, a deep learning approach using bidirectional gated recurrent (bigru) unit was likewise developed for shallow parsing khasi. the performance of this shallow parser on the same gold data gives an f1 measure of 98.91 (tham 2020a). however, to get a picture on how its performance will be affected by the performance of the khasi pos tagger, it was tested on the data tagged by the hmm pos tagger for khasi and gave an f1 measure of 89.91. this result clearly indicates that improving tagging performance will also improve parsing performance because of the dependency of the parser on pos tagged data. therefore, the development of the hybrid pos tagger for khasi is a much needed improvement in this direction. ## conclusion in this demonstration paper the nlp tools for khasi and their performances have been presented. hopefully, with these tools in place it will be rewarding to see their applications in various nlp applications for khasi.