Building a Part-of-Speech Tagged Corpus for Drenjongke (Bhutia) (2020)

Abstract: This research paper reports on the generation of the first Drenjongke corpus based on texts taken from a phrase book for beginners, written in the Tibetan script. A corpus of sentences was created after correcting errors in the text scanned through optical character reading (OCR). A total of 34 Part-of-Speech (PoS) tags were defined based on manual annotation performed by the three authors, one of whom is a native speaker of Drenjongke. The first corpus of the Drenjongke language comprises 275 sentences and 1379 tokens, which we plan to expand with other materials to promote further studies of this language.

## introduction according to @xcite , most of the world's languages cannot benefit from the stateof-the-art methods of natural language processing (nlp) due to a lack of resources. given that the need for language technologies is equally distributed among speakers of each language, the investigation of the construction of language resources is necessary. this paper reports how the first drenjongke machine-readable labeled data were developed. drenjongke (bhutia foot_0 ) is a tibeto-burman language with the status of one of the official languages in the state of sikkim, india. @xcite estimates that the number of drenjongke speakers is 25,000-30,000 and that drenjongke falls into one of the categories of "vulnerable," "definitely endangered," and "severely endangered" within unesco's language vitality and endangerment framework, with a rapid loss of native speakers due to the lack of economic value in speaking drenjongke. the low economic value is due to the widespread use of nepali in sikkim. most official and private business matters are conducted in nepali or english. as a result of community-driven revitalization efforts foot_2 , written materials in drenjongke are being produced, including but not limited to dictionaries, folk songs, and novels. an archive was created resulting from a recent project with drenjongke community members: "phonetics, phonology and new orthographies in roman and indigenous script: helping native language communities in the himalayas @xcite @xcite ." 3 the archive contains spoken materials, including conversations and nursery rhymes. 4 however, the focus of the archive was speech recordings. thus, machine-readable texts were not included. the goal of our project is to build the first digitized resource of drenjongke so as to serve as a basis for expanding nlp resources on drenjongke by suggesting a set of pos tags that can be used for processing other drenjongke texts in the future. moreover, we provide annotated text materials that would promote future computational research. ## related work in other tibeto-burman languages tibeto-burman languages share many characteristics. from nlp perspectives, some similarities include 1) orthographic systems that are not based on the roman alphabet, and 2) agglutinative morphology that creates difficulty in determining the word boundaries often used in nlp systems targeting indo-european languages. even so, annotated text corpora in burmese (myanmar), dzongkha (bhutan), and classical and modern tibetan exist. the largest existing burmese corpus with annotations is the "burmese (myanmar) treebank of asian language treebank project" @xcite . this corpus contains 20,000 sentences that are morphologically annotated with the annotation scheme "nova," in which n stands for nouns, v for verbs, a for adjectives, and o for others. this categorization was created for lowresource, highly analytic languages. subsequently, @xcite propose the conversion procedure of the "nova" tag to the universal part-of-speech (pos) tagset. 5 corpora as well as pos taggers in classical tibetan foot_6 have been developed as part of the project "tibetan in digital communication (2012-2015)" hosted at soas. 7 the project has annotated four documents from tibetan classical literature. 8 as for modern tibetan, @xcite report a collection of tibetan text corpora (cttc) containing 52,041 pos-tagged sentences, which have been used in machine translation and tibetan syntactic parsing. dzongkha also has several nlp resources, including a corpus and tokenizers. @xcite propose a word segmentation method applied to dzongkha using the dictionary-based maximal matching algorithm. @xcite describe the corpus building process for dzongkha, in which they prepared 66 pos tags for annotation. we found that these nlp resources on tibeto-burman languages often lack uniformed annotation guidelines as pointed out in hill and list (2017). as such, when building corpora in a different low-resource tibeto-burman language, namely drenjongke, information about the process of building of existing corpora was not directly transferable. ## corpus construction our corpus contains 275 sentences and 1379 tokens taken from bhutia phrase book @xcite . the corpus has two parts: the sentence level, containing (a) sentence id, (b) the original drenjongke sentence in tibetan script, and (c) an english translation; and the token level, consisting of (a) token, (b) pos tag, (c) romanization of the token in phonological drenjongke (section 4. ## discussion as far as we know, no drenjongke corpus has been available until now. building a drenjongke corpus was not a simple task and various considerations were discussed in the process. we present four points: (a) domain specificity of the corpus, (b) social impact of machine-readable resources, (c) language revitalization, and (d) annotation challenges. ## conclusion and future work this paper reports the building of a drenjongke pos-tagged corpus containing 275 sentences with english translation and romanization in "phonological drenjongke." our future work lies in establishing a reliable word segmentation method as well as in optimizing the pos tagging process while referring to the existing tools for tibeto-burman languages. additionally, the development of the language-specific documentation for the universal pos tagset, as well as dependency relation labelling, remains for future work.