Creating and Evaluating Resources for Sentiment Analysis in the Low-resource Language: S indhi (2021)

Abstract: In this paper, we develop Sindhi subjective lexicon using a merger of existing English resources: NRC lexicon, list of opinion words, SentiWordNet, Sindhi-English bilingual dictionary, and collection of Sindhi modifiers. The positive or negative sentiment score is assigned to each Sindhi opinion word. Afterwards, we determine the coverage of the proposed lexicon with subjectivity analysis. Moreover, we crawl multi-domain tweet corpus of news, sports, and finance. The crawled corpus is annotated by experienced annotators using the Doccano text annotation tool. The sentiment annotated corpus is evaluated by employing support vector machine (SVM), recurrent neural network (RNN) variants, and convolutional neural network (CNN).

## introduction the exponential growth in the online professional and user-generated textual data @xcite , including blog posts, news headlines, product, and book reviews, led to the growth of the sentiment analysis task. the required essential resources for the classification of such opinionated text are the polarity assigned sentiment lexicon @xcite and sentiment annotated corpora @xcite . sophisticated research efforts have been employed for english sentiment analysis @xcite . in the result, a number of resources are available including opinion words @xcite , subjective lexicon @xcite , sentiwordnet (swn) @xcite , nrc lexicon @xcite . the sentiment annotated corpora including financial news (fn) @xcite , sports tweets (st) @xcite , tweet dataset @xcite , and more recently a multi-domain corpora @xcite . among these resources the swn has been widely used for the construction of sentiment lexicon for lowresource languages including urdu @xcite , turkish @xcite , and hindi @xcite . sindhi is an indo-aryan language, spoken by more than 75 million @xcite people. presently, it is being written in two main scripts of @xcite . however, persian-arabic is a popular and standard script @xcite . it is widely used in online communication, mainly in the sindh province of pakistan and some regions of india @xcite . the generated content on social media contains rich information about the interests of individuals. thus, the modeling of such information is essential to analyze where people's opinions are conveyed. the low-resource sindhi language lacks the primary resources for content analysis, such as polarity assigned lexicon and sentiment annotated corpora. in this paper, we create sindhi subjective lexicon using existing english resources. moreover, due to the scarcity of sentiment annotated corpora, we crawl and annotate news headline (nh) tweets, sports tweets (st), and fn tweets, respectively. three native annotators performed the annotation using doccano @xcite text annotation tool with 79.3% inter-annotator agreement. to the best of our knowledge, both datasets foot_0 are the first benchmark for sindhi sentiment analysis (ssa). furthermore, we develop strong baselines of svm @xcite , @xcite and rnn variants of long-short-term memory (lstm) @xcite , bidirectional longshort-term memory (bilstm) @xcite for the evaluation purpose. ## related work the development of polarity assigned sentiment lexicon have largely been investigated for the rich-resource english language, such as bing liu's lexicon @xcite , sentiword-net (swn) @xcite , swn 3.0 @xcite , and nrc lexicon @xcite . these resources have been widely used to create a sentiment lexicon for low-resource languages, mainly by translating the terms into the target languages. the human annotators assigned polarity score to create the sentiment lexicon for south asian languages such as hindi @xcite , bengali, telugu @xcite , tamil @xcite , persian @xcite , urdu @xcite , panjabi @xcite , and sinhala @xcite . @xcite proposed a graph-based wordnet-based approach to develop subjective lexicon by using synonym and antonym relations. das and bandyopadhyay (2010) opted multiple methods such as, dictionary-based, corpus-based or generative approach, and wordnet-based for the construction of sentiment lexicon for indian languages. various resources and tools including bing liu's lexicon, swn, subjectivity lexicon @xcite , afinn-111 lexicon @xcite , and google translate @xcite are utilized to develop sentiment lexicon for persian language. a word-level translation scheme @xcite is proposed to construct urdu lexicon using english resources including bing liu's lexicon, swn 3.0, and english to urdu bilingual dictionary. @xcite proposed sentiment lexicon for low-resource sinhala language by using swn 3.0 using word-level translation scheme. as we mentioned earlier, sindhi stands among the low-resource languages because lack the subjective lexicon and sentiment annotated corpus except a recently ali and wagan 6842 part-of-speech tagged lexicon. hence, their lexicon lack a sentiment intensity score. thus, we propose sindhi subjective lexicons using a merger of existing english resources. moreover, we also crawl multi-domain nh, ft, and st tweets and annotate them using doccano @xcite text annotation tool. many sentiment-annotated corpora have also been created for english in multiple domains, such as news, sports, finance, and products. shamma et al. ( 2009 ) created tweet dataset by crawling u.s. presidential debate in multiple sentiment classes. @xcite proposed sentiment dataset on product reviews, electronics, and kitchen appliances obtained from amazon.com. @xcite manually annotated tweet dataset with +ve and ve sentiments. the annotation of financial blogs and news domain @xcite @xcite have also been investigated at a large scale. @xcite proposed a dataset by crawling sports tweets from twitter using search api. more recently, ekbal et al. (2020) proposed multi-domain tweet corpora, annotated with three sentiment classes. review shows that sindhi lacks the subjective lexicon as well as sentiment annotated corpora for its supervised sentiment analysis that we consider. ## development of subjective lexicon we construct sindhi subjective lexicon by depicting the sentiment polarity score of all english opinion words using bilingual english to sindhi dictionary foot_1 . the construction steps are described as follows: ## evaluation we determine the coverage of the proposed lexicon with subjectivity analysis @xcite . the sentiment annotated corpus is evaluated by employing svm, lstm, bilstm, and cnn models. ## results and analysis the results demonstrate that the bilstm and cnn yield better results than svm and lstm. however, the bilstm network surpasses svm, lstm as well as cnn models. ## conclusion and future work in this paper, we propose sindhi subjective lexicon using various resources and sentiment annotated corpus, which serves as a benchmark for future expansions. the svm and deep neural models are exploited for evaluation purposes. we achieve notable f-value and accuracy of 83.11%, 82.37% with the bilstm network. in the future, the proposed lexicon can be expanded using a corpus-based approach to capture language-specific words. also, it can be used as a seed list, and the corpus can be tagged on the basis of the seed list. moreover, the proposed lexicon consists of positive and negative classes, so a five-point scale can replace this classification in the future.