A Corpus for Dimensional Sentiment Classification on Y ou T ube Streaming Service (2021)

Abstract: The streaming service platform such as YouTube provides a discussion function for audiences worldwide to share comments. YouTubers who upload videos to the YouTube platform want to track the performance of these uploaded videos. However, the present analysis functions of YouTube only provide a few performance indicators such as average view duration, browsing history, variance in audience’s demographics, etc., and lack of sentiment analysis on the audience’s comments. Therefore, the paper proposes multi-dimensional sentiment indicators such as YouTuber preference, Video preferences, and Excitement level to capture comprehensive sentiment on audience comments for videos and YouTubers. To evaluate the performance of different classifiers, we experiment with deep learning-based, machine learning-based, and BERT-based classifiers to automatically detect three sentiment indicators of an audience’s comments. Experimental results indicate that the BERT-based classifier is a better classification model than other classifiers according to F1-score, and the sentiment indicator of Excitement level is quite an improvement. Therefore, the multiple sentiment detection tasks on the video streaming service platform can be solved by the proposed multi-dimensional sentiment indicators accompanied with BERT classifier to gain the best result.

## introduction due to the rapid rise of new media and the popularization of mobile phone networks, audiences' viewing habits have shifted from tv to online social media platforms. now people can watch videos on different platforms such as facebook, dailymotion, and youtube anytime and anywhere. youtube has 16 million active users in taiwan monthly, and nearly 93% of users have visited youtube. in addition, youtube has become ubiquitous and played an increasingly important role in modern life and entertainment. also, youtube provides a discussion function for audiences to express their opinion by clicking like or dislike bottom or leaving comments. therefore, comprehensive sentiment analysis for comments of the audience on youtube is necessary. it is verified that public views, comments, and attitudes towards many events can be analyzed through social media @xcite . public reviews on amazon were used to evaluate users' opinions and determine the audience's preference by classifying opinions into negative, positive, and neutral @xcite . another research investigated the popularity of videos by indicators such as the number of likes, dislikes, and views @xcite . social media, especially youtube. is considered the largest video sharing site, and the platform has developed into a leading marketing tool. @xcite inspired by the above analysis tasks and the rapid growth status of youtube, we propose the multidimensional sentiment indicators to analyze comments on youtube, which aim to help youtubers check their videos' performance uploaded on the youtube platform. in general, sentiment analysis focuses on determining the positive, negative, or neutral emotions in many pieces of research @xcite . @xcite extend the emotional detection, which includes highly positive, optimistic, neutral, negative, and highly damaging, the emotional variance may have a different dimension, such as excitement which expresses the audience's fluctuating emotion. so, we also propose detecting the audience's excitement level on youtube because excitement more precisely determines how much the audience likes youtubers or videos. to obtain comprehensive sentiment indicators, we design three indicators: youtuber preference, video preference, and excitement level to analyze a multi-dimensional aspect of the audience's comments. in the experiment, these three sentiment indicators also represent three detection tasks that aim to detect the audience's motivations behind a myriad of comments. various models deal with text-based sentiment classification tasks. machine learning-based models are used to address the text classification task @xcite . other deep learning models have been used for sentiment analysis and obtained acceptable performances @xcite . recently, it has refreshed the best performance of using pre-trained language models as soon as it appears. elmo @xcite and @xcite have been effective because pre-trained models have learned by detecting other tasks from a larger corpus which capture more linguistic structure. this paper's objectives are: (1) to create a corpus for multi-dimensional sentiment indicators, which include youtuber preference, video preferences, and excitement level; (2) train an automatic sentiment detection model, including machine learning-based, deep learning-based, and bertbased models. overall, the contributions of this paper are: (1) we establish a benchmark dataset of dimensional sentiment classification for analyzing comments on youtube. (2) successfully using different models to deal with sentiment classification issues. ## related work more and more researchers undertook experiments on youtube as the data source. the purpose is to obtain an understanding of the community commenting behavior. @xcite showed that although most audiences present their opinions as comments, some abuse this mechanism by posting links to external web pages or posting disruptive, false, or offensive comments to fool and provoke other users. based on the result of the above research, this paper's dataset eliminates nonrelative comments such as links that guide people to external web pages and advertisements that have no relation to video content. @xcite work on youtube video comments, likes, and dislikes to show that it genuinely influences users' perceptions of like or dislike towards videos when reviewing valuable comments. moving to some purposes of text classification used nowadays, @xcite did sentiment analysis by establishing an unsupervised classifier to judge the positivity or negativity of product reviews (cars, banks, and tourist destinations) and movie reviews. another paper presented an approach based on a clustering of comment content, leading to appropriate video categories @xcite . machine learning approaches are then introduced to automatically classify comments according to their usefulness @xcite ). as the above papers show, comments can achieve various objectives by using different technical methods. there are currently two approaches to address sentiment analysis: (1) lexicon-based techniques (2) algorithm-based techniques. lexicon-based techniques rely on predefined words and rules to guide the sentence towards the tendency of emotion. algorithm-based techniques can be divided into two groups: machine learning-based models s and deep learning-based models. zhang and zheng (2016) discussed machine learning methods for sentiment analysis. @xcite employed deep-learning approaches with word embedding and tf-idf to solve sentiment analysis problems. as a result, the best behavior when using the word embedding method against tf-idf of all models has been proved. another research used pre-trained word embedding as an important component for it downstream models. (t. miyato, a. m. dai, et al., 2017) thus, we identify that word embedding is in conjunction with deep learning-based models in our experimental and pre-trained word embedding offer significant improvement over embedding learned from scratch. moreover, due to the effectiveness of pretrained language models, adding one additional output layer can fine-tuned models and accelerating the accuracy of classification problems. @xcite al. (2019) fine-tuned with bidirectional encoder representations from the transformers (bert) model and achieved stateof-the-art results using comments. liat ein-dor et al. (2020) using bert based models for binary classification tasks to deal with sentiment analysis tasks, these papers all share some commons. firstly, comments in nowadays social media, especially youtube, are of great value and even thoroughly necessary. secondly, despite different methods that are conducted, comments do reflect users' opinions on social media. the above works are similar to this paper, all using comments as data sources but a different way to solve the task; we reference the above methods and determine to use all the methods, including deep learning-based, machine learning-based, and bert-based classifiers. however, each method has been experimented with separately so as to carry out a comparative study. also, the difference is that we focus our experiment on multi-class text classification. ## methodology figure 1 shows the proposed method for sentiment analysis and classification processes as follows: firstly, we collected the audience's comments from youtube platform and subsequently labeled these comments to provide meaningful and informative labels such as three sentiment indicators for model training. data preprocessing works are conducted to clear text. next, all comment's texts need to be converted into vectors to serve as the model's input. and then, machine learning and deep learning models propose to train detection models for our proposed sentiment indicators. finally, by the experiment stage, we evaluate the performance of each classifier in three detection tasks and discuss a comparative study. ## conclusion this paper focuses on sentiment analysis using the core of bert pre-trained language models and accompanied by one deep learning-based model and three machine learning-based models. after conducting experiments, the method of deep learning and bert perform better than the machine learning method. we also show that bert can deal with sentiment polarity by determining the audience's likes or dislikes towards youtubers. finally, bert is perfectly addressing the multi-classification problem. before utilizing these classifiers, introducing related labeling jobs as a prerequisite is vital to getting a reliable dataset. through these methods, we genuinely fill the gaps between human semantic comprehension. analyzing the public's perception of youtubers and the influence of their videos is a challenging task for researchers so far. proposing different sentiment indicators and utilizing different classifiers has been done in this paper, but there still is a long way to overcome some problems. in this paper, we have emphasized the following problems in order to make our results improve. (1) informal language styles such as sparse emojis used by the audiences may impede models from capturing linguistic structure. (2) the semantic comprehension gap among annotators needs to be reduced to improve annotation consistency. in the future, we may explore other techniques for optimizing multiple-dimensional sentiment analysis tasks, such as training youtubers' names as embedding before utilizing different models. in this way, perhaps models can precisely filter out non-relative comments towards youtubers. in addition, others indicators, such as whether the comments contain an ironic statement or whether the comments are erotic, can be added for analyzing other aspects of the audience's comments. the latter proposed indicator may serve as a guard for children users, and the former indicator may prevent youtubers from getting into conflict with their fans.