S i POS : A Benchmark Dataset for S indhi Part-of-Speech Tagging (2021)

Abstract: In this paper, we introduce the SiPOS dataset for part-of-speech tagging in the low-resource Sindhi language with quality baselines. The dataset consists of more than 293K tokens annotated with sixteen universal part-of-speech categories. Two experienced native annotators annotated the SiPOS using the Doccano text annotation tool with an inter-annotation agreement of 0.872. We exploit the conditional random field, the popular bidirectional long-short-term memory neural model, and self-attention mechanism with various settings to evaluate the proposed dataset. Besides pre-trained GloVe and fastText representation, the character-level representations are incorporated to extract character-level information using the bidirectional long-short-term memory encoder. The high accuracy of 96.25% is achieved with the task-specific joint word-level and character-level representations. The SiPOS dataset is likely to be a significant resource for the low-resource Sindhi language.

## introduction annotated corpus is an essential resource for developing automatic natural language processing (nlp) systems @xcite . such language resources (lrs) play a significant role in the digital survival of human languages @xcite . the part-of-speech (pos) tagging is a fundamental preprocessing task in various nlp applications @xcite , used to assign appropriate incontext pos tags to each word. one of the main challenges @xcite in pos tagging is ambiguity since one word can take several possible pos labels. another problem is the unspoken or complex pos or words. both of these problems are not rare in natural languages. moreover, there is a lack of benchmark labeled datasets for sindhi pos tagging. to tackle these challenges, we propose a novel benchmark sipos tagset. sindhi is a rich and complex morphological language @xcite . it is a low-resource language @xcite which lacks primary lrs for mature computational processing. sindhi is being written in two famous writing systems of persian-arabic, devanagari, and more recently roman @xcite is also getting popularity. however, persian-arabic is standard script as well as frequently used in literary work, online communication, and journalism. sindhi pos tagging has been previously investigated in various scripts including persian-arabic,devanagari (jamro, 2017), and roman @xcite . however, the low-resource sindhi language lacks a pos labeled dataset for its supervised text classification. in this paper, we introduce a novel benchmark sipos dataset for the low-resource sindhi language. due to the scarcity of open-source pos labeled data, two native experienced annotators performed the pos annotation of sindhi text using the doccano @xcite text annotation tool. to the best of our knowledge, this is the first attempt to address the sindhi pos tagging at a large scale by proposing a new gold-standard sipos dataset foot_0 and exploiting conditional random field (crf), bidirectional long-short-term memory (bilstm) network, and self-attention for its evaluation. our novel contributions are as follows: ## related work the labeling of natural language text with pos tags can be a complicated task, requiring much effort, even for trained annotators @xcite . a large number of lrs are publicly available for high-resource languages such as english (marcus and marcinkiewicz), chinese, indian languages @xcite and others @xcite . unlike rich-resourced languages such as english and chinese with abundant publicly accessible lrs, sindhi is relatively low-resource @xcite , which lacks the pos tagged dataset that can be utilized to train a supervised or statistical algorithm. previously, mahar and memon (2010) proposed a sindhi pos labeled dataset consists of 33k tokens. later, dootio and wagan (2019) published a new dataset containing 6.8k lexicon, which is insufficient to train a robust supervised classification algorithm. more recently, (rahman et al., 2020) annotated 100k words by employing a multi-layer annotation model, which comprises different annotation layers like pos, morphological features, dependency structure, and phrase structure. but their dataset is not publicly available. except for sindhi persian-arabic, the pos tagged datasets in devanagari @xcite , and roman (sodhar et al., 2019) scripts have also been introduced. the pos tagged corpus of sindhi-devanagari consists of 44k tokens, while sindhi-roman @xcite only contains 100 sentences. the review of existing work shows that the low-resource sindhi language lacks a benchmark pos labeled dataset for its supervised text classification. ## corpus acquisition and annotation in this section, we illustrate the opted annotation methodology. we utilized the news corpus @xcite of popular and most circulated sindhi newspapers of kawish and awami-awaz (see ## evaluation methods we evaluate the sipos for the consistency in the dataset by computing the inter-annotator agreement using cohen's kappa @xcite coefficient. we evaluate the proposed sipos dataset by exploiting crf, bilstm and integrating crf and selfattention in the bilstm network for strong baselines. moreover, pre-trained glove, fasttext word representations, and task-specific character-level, and joint w ord character level representations are incorporated to extract word-level and characterlevel information using the bilstm encoder. ## conclusion and future work in this paper, we propose a novel benchmark sipos dataset for the low-resource sindhi language. the dataset consists of more than 293k tokens, annotated using the doccano text annotation tool. we exploit crf as the initial baseline and bilstm model by incorporating crf as a decoder and self-attention mechanism for the performance gain. the bilstm-attention-crf model yields a high accuracy of 96.25% with the joint task-specific character-level and word-level representations. the proposed open-source sipos dataset will be a sophisticated addition to the resources for the sindhi language. in the future, we intend to pre-train bidirectional encoder representations from transformers (bert) and generative pre-trained transformer (gpt) language models for sindhi text classification.