A Dataset and BERT -based Models for Targeted Sentiment Analysis on T urkish Texts (2022)

Abstract: Targeted Sentiment Analysis aims to extract sentiment towards a particular target from a given text. It is a field that is attracting attention due to the increasing accessibility of the Internet, which leads people to generate an enormous amount of data. Sentiment analysis, which in general requires annotated data for training, is a well-researched area for widely studied languages such as English. For low-resource languages such as Turkish, there is a lack of such annotated data. We present an annotated Turkish dataset suitable for targeted sentiment analysis. We also propose BERT-based models with different architectures to accomplish the task of targeted sentiment analysis. The results demonstrate that the proposed models outperform the traditional sentiment analysis models for the targeted sentiment analysis task.

## introduction the increasing availability of the internet and the growing number of online platforms allowed people to easily create online content. because of the value of mining the people's opinions, the sentimental information contained in this online data makes sentiment analysis (sa) an interesting topic. it is an area that is attracting the attention not only of academic researchers, but also of businesses and governments @xcite and has become a rapidly growing field, as evidenced by the number of recent sa papers published @xcite . the problem with traditional sentiment analysis is that it cannot capture the different attitudes toward multiple aspects in a given text. for example, if the given text is "phones from this brand are great, but i don't really like their laptops", the sentiment towards the two targets "phone" and "laptop" are positive and negative, respectively. traditional sentiment analysis methods would not be able to detect this opposing sentiment for "phone" and "laptop", but would assign an overall sentiment for the text. targeted sentiment analysis (tsa) aims to overcome this challenge and extracts sentiment from a given text with respect to a specific target. one of the challenges of tsa is the lack of available datasets. both tsa and sa require labeled datasets. collecting data from various sources and labeling them, which is mostly done manually, is an expensive process. although the number of datasets suitable for sa has recently increased due to new studies in the sa area, not all sa datasets are usable for tsa @xcite . tsa requires more refined datasets. the labels should reflect the sentiment toward targets rather than the overall sentiment of the sentences. english is the most studied language for sentiment analysis @xcite . sa models that perform satisfactorily for english do not seem to always work with similar performance for turkish @xcite . in this work, we create a manually annotated dataset from twitter specifically labeled for both traditional and targeted sentiment analysis in turkish. then, we experiment with different model architectures for the turkish tsa task. experimental results demonstrate that our techniques outperform traditional sentiment analysis models. ## dataset twitter is a commonly used source of sentiment classification dataset in the literature @xcite @xcite . in this study, we also create a twitter dataset with 3952 tweets whose timestamps span a sixmonth period between january 2020 and june 2020. the tweets are collected via the official twitter api by separately searching our 6 targets selected from famous companies and brands. this dataset is manually annotated with three labels, positive, negative, and neutral. two factors are considered in the annotation process, namely sentence sentiment and targeted sentiment. each tweet has the following two labels. the sentence sentiment label expresses the overall sentiment of the sentence, regardless of the target word, as in traditional sentiment analysis techniques. on the other hand, the targeted sentiment label reflects the sentiment for the target in that sentence. the collected tweets are annotated separately by two annotators (one of the authors and a volunteer annotator) who are native turkish speakers. cohen's κ (cohen, 1960) is used to demonstrate inter-annotator agreement and is calculated as 0.855. in case of conflict between annotators, they re-evaluated the conflicting tweets. after re-evaluation, tweets on which the annotators agree are retained and conflicting tweets are removed from the dataset. the dataset contains ungrammatical text, slang, acronyms, as well as special twitter characters. during pre-processing urls and mentions (@) are deleted. hashtag signs (#) are removed, but hashtags are kept for two reasons: hashtags have been shown to express sentiment @xcite and some tweets contain the targets as hashtags. ## methodology baldini @xcite has introduced a novel method with transformer structure in the field of relation extraction. the key idea behind this work is to tag the entities with additional tokens before feeding the bert model with the input. different combinations of input and output types are evaluated. the best results are obtained when entity markers are added to the input and when the output of the starting entity markers are taken as the output from bert. motivated by the results of baldini soares et al.'s work, this paper evaluates several bert architectures with different input and output techniques for the targeted sentiment analysis task. two input representation techniques are investigated. in the standard input representation, the inputs are simply entered into the model without modification. in the second input representation approach, the targets are highlighted by adding additional special target tokens [tar] at the be- tweets with [tar] tokens [tar]whatsapp[tar] çöktü de biraz rahatladım bildirimlerden kurtuldum ([tar]whatsapp[tar] is crashed so i'm little relieved, got rid of notifications) [tar]coca cola[tar] daha iyi lezzet olarak ([tar]coca cola[tar]'s taste is better) ## results all proposed bert variants and baselines are evaluated for targeted sentiment analysis over our introduced dataset. macro averaged f1-score is used as the evaluation metric in these experiments. the results are presented in ## discussion our results suggest that target oriented models can significantly improve the performance for targeted sentiment analysis. bert architectures that perform successfully in the relation extraction field are shown to be successful for the targeted sentiment analysis task. target markers make bert models understand target related context better compared to the [cls] token. all three models with target markers outperform the baselines and t-bert. hence, adding target markers is an effective approach for improving tsa performance. t-bert marked -ts and t-bert marked -mp are shown to perform slightly better than the other target oriented models. the common aspect of these models, apart from the target tokens, is that they both focus on the outputs of the target-related tokens rather than the [cls] tokens. therefore, it can be concluded that target outputs improves the performance for the tsa task. we only considered one target in each sentence and annotated according to that target. other targets in the sentence, if any, are ignored. multiple targets with conflicting targeted sentiment in the same sentence can be a problem to consider. there are cases where a sentence has more than one target, and each target has a different targeted sentiment. for example, in a comparison, the sentiment toward one target may actually depend on the sentiment of another target in the same sentence. in this work, the scope is limited to only one target in each sentence. target markers are also used only for this one target in the sentence and other possible targets are ignored. the lack of proper treatment of such cases in this work may affect the performance of all models. sentence and targeted sentiment are identical for 79% of the dataset. thus, if a traditional sa model, which is designed to predict the overall sentence sentiment, is used for the tsa task, its success for this task would be overestimated. the results demonstrate that targeted sentiment analysis models perform significantly better than traditional sentiment analysis models on the tsa task. however, the performance of the tsa models increases when they are tested on the entire test dataset, rather than on a subset containing only tweets with different sentence and targeted sentiment labels. this highlights that they may still be biased in favor of sentence sentiment to some extent. ## ethical considerations and limitations the dataset contains public tweets in turkish that are provided by the official twitter api for research. only tweet id's and labels of the tweets are shared publicly to follow twitter's terms and conditions. the annotators have no affiliation with any of the companies that are used as targets in the dataset, so there is no potential bias due to conflict of interest. the models developed in this work are not yet satisfactory to use their results without human monitoring. it is recommended to manually check the predictions of these models before using them. ## conclusion and future work we presented a manually annotated turkish twitter dataset specifically created for targeted sentiment analysis and is also suitable for the traditional sentiment analysis task. this allowed us to develop and evaluate novel models for targeted sentiment analysis in a low-resource language such as turkish. we adapted and investigated bert-based models with different architectures for targeted sentiment analysis. experiments show significant improvement on baseline performance. as future work, we plan to expand our dataset so that it contains more sentences with different sentence and targeted sentiment. moreover, novel methods for sentences with multiple targets will be investigated.