Resource Creation and Evaluation of Aspect Based Sentiment Analysis in U rdu (2020)

Abstract: Along with the rise of people generated content on social sites, sentiment analysis has gained more importance. Aspect Based Sentiment Analysis (ABSA) is a task of identifying the sentiment at aspect level. It has more importance than sentiment analysis from commercial point of view. To the best of our knowledge, there is very few work on ABSA in Urdu language. Recent work on ABSA has limitations. Only predefined aspects are identified in a specific domain. So our focus is on the creation and evaluation of dataset for ABSA in Urdu language which will support multiple aspects. This dataset will provide a baseline evaluation for ABSA systems.

## introduction we are living in a world where web interaction is increasing. people share their emotions and express their feelings on different platforms through internet. a lot of work has been done to obtain valuable information from these reviews. sentiment analysis is a task of identifying people's emotions, feelings and opinion about a particular object from their reviews @xcite . it is divided into three levels. document level, sentence level and aspect level. "document level" classifies the opinion of a complete document or paragraph into positive, negative, conflict or neutral. "sentence level" classifies the sentiment of a sentence into positive, negative, conflict or neutral. "entity/aspect level" it obtains the sentiment of a sentence at aspect (features) level @xcite . sentiment analysis has obtained importance in different areas like business intelligence and social media monitoring etc. another area of interest is monitoring political data @xcite . there has been very little research in the urdu language since it is a low resourced language @xcite urdu has different morphological structures and linguistic features, so sentiment analysis intended for the english language can't be utilized for this language @xcite , @xcite . many approaches and difficulties are discussed for the english language but very few for the urdu language @xcite . urdu is a mixture of many languages, so one of the challenging tasks is to handle different languages with their orientation @xcite . aspect based sentiment analysis is a process of extracting the opinions expressed about a specific entity. the sentiment analysis task becomes more challenging when there are multiple sentiments in a review on different aspects @xcite . recently, the sentiment analysis research has moved towards having all the more fine-grained approaches thinking about the urdu language aspects. but still, urdu language is struggling at the aspect level. recent work on the urdu language at aspect level is not much more fine-grained as they deal only 4 predefined aspects. @xcite . we aim to improve the current work by dealing the multiple aspects. an aspect can be a single word(e.g., runs) or multi word (e.g., t 20 series). a sentence may have one or more aspects that belongs to the same category. for this purpose clustering is used to group the related aspects into a single cluster. in this way we have a more finegrained work at aspect level. we present a corpus of "cricket" and "football" domain in the urdu language for absa. which will provide a baseline evaluation and can further used for measuring the results of absa systems. the rest of the paper is sorted out as follows. section 2 describes related work. section 3 contains a problem statement. section 4 has a proposed methodology. section 5 contains a description of the data collection and annotation procedure. section 6 provides information on guidelines for annotation. section 7 has a conclusion. ## related work the task of absa was first presented by semeval in 2014 for english language @xcite . they provided data set of restaurant and laptop reviews for training and testing. @xcite a benchmark dataset of arabic language for aspect based sentiment analysis named as "haad" is prepared by them. they manually annotated the data at sentence level. annotators used an online tool (brat) for annotation. this dataset provides the baseline evaluation to the four tasks i.e.; aspect extraction, aspect polarity, aspect category identification, aspect category polarity. two datasets of low resourced languages catalan and basque for aspect level sentiment analysis are introduced by @xcite . they performed tokenization, pos tagging and lemmatization using lxa-pipes. they trained a linear svc classifier for the classification of the polarity of opinion expressions. they trained a crf on the standard features for the extraction of opinion holders, targets and expressions. for evaluation, 10 fold cross validation is used on 80% of data and f1 is used for extraction and classification. @xcite this paper describes the data collection procedure and annotation guidelines. to increase the applicability and comparability of system, semeval-2015 task for english guidelines are used for annotation. they used different systems to identify three types of information (aspect category, opinion target expression and sentiment polarity). the categories returned by a framework are compared with gold annotations and precision, recall and f1 is measured. at layer 1, polarity and subjectivity/objectivity is analyzed. at layer 2, they analyzed word and phrase-level annotation. at layer 3, they focused on the annotation of expression-level. each layer had been analyzed by multiple annotators. to ensure the quality of data, they also calculated the inter-annotator agreement. ## statement of the problem rather than classifying the complete sentiment of a sentence into positive, negative or neutral, absa allows us to associate specific sentiment with different aspects of a product. since urdu language is a low-resourced language, there is very little work on absa, especially in urdu language @xcite . unavailability of urdu dataset for aspect based sentiment analysis leads to constructing a benchmark dataset of a specific domain. we aim to create a benchmark corpus to facilitate the absa systems. annotation will be performed according to the guidelines of semeval and performance is measured by using different machine learning approaches. ## proposed methodology as our proposed methodology can be seen from figure 1 . we crawled urdu tweets of cricket and football domain from twitter using twitter api's. then we performed preprocessing on raw data. in preprocessing, special characters, hashtags, links and punctuation marks are removed using sci kit learn and regular expressions. after preprocessing, we prepared annotation guidelines for annotating the data. these guidelines are according to the standards of semeval. after preparing guidelines for annotation, three annotators will manually annotate the dataset. in the annotation process, four types of information have to identify, i.e., aspect, aspect category, aspect polarity and aspect category polarity. annotators will identify aspects and assign polarity. to ensure the quality of the dataset, we will compute the inter-annotator agreement. for feature extraction, we will apply tf-idf vectorizer and n-gram models. next, we will apply baseline models of machine learning like naive bayes, random forest, knn. then we will evaluate the performance by precision, recall and f1 measure. ## data collection and annotation this dataset is consists of two different domains "cricket" and "football". we collected 7000 tweets of cricket domain and 3000 tweets of football domain from twitter using twitter api's. three annotators will manually annotate the data. they have to identify four types of information, which is discussed in section 4. ## annotation guidelines initially, we have prepared some guidelines to annotate the data. the aim of these guidelines is to identify the aspects and sentiment polarity in sentences. we have to identify four types of information for annotation. ## conclusion in this paper, we have presented a benchmark corpus for absa in the urdu language. this dataset has been prepared to cover different areas of research. it will facilitate researchers because we are covering four tasks of sentiment analysis. which can be further used in the future by adding more domains. we believe that it will be very beneficial for the researcher community because we work in a different language and a specific domain. in future, we aim to improve our work by applying different techniques for feature extraction. we also plan to explore more machine learning algorithms and neural networks for these tasks.