Transformer-based cynical expression detection in a corpus of S panish Y ou T ube reviews (2023)

Abstract: Consumers of services and products exhibit a wide range of behaviors on social networks when they are dissatisfied. In this paper, we consider three types of cynical expressions negative feelings, specific reasons, and attitude of being right and annotate a corpus of 3189 comments in Spanish on car analysis channels from YouTube. We evaluate both token classification and text classification settings for this problem, and compare performance of different pre-trained models including BETO, SpanBERTa, Multilingual Bert, and RoBERTuito. The results show that models achieve performance above 0.8 F1 for all types of cynical expressions in the text classification setting, but achieve lower performance (around 0.6-0.7 F1) for the harder token classification setting.

## introduction consumers of services and products actively engage through social networks when they are dissatisfied, exhibiting a wide range of behaviors. @xcite . encinas presents a classification of dysfunctional consumer behaviors: mild behaviors such as rudeness, complaints, skepticism, or tantrums; moderate behaviors such as manifestations of cynicism, attempts at manipulation, or inappropriate comments and foul language; and intense consumer behaviors such as fraud, theft, verbal aggression, or revenge. we focus on cynical expressions of consumers, specifically in comments written in videos on the youtube platform. cynicism is a negative attitude with a broad or specific focus and comprises cognitive, affective, and behavioral components @xcite . consumer cynicism can generate feelings of betrayal and deception, leading to anger and the desire to stop purchasing products or services from the source that generates their anger @xcite . within expressions of cynicism, we focus on the following specific expressions: negative feelings where consumers reflect negatively on a product, usually in a subjective way that is influenced by their personal experiences. specific reasons where consumers identify the specific aspects or components of a product to which their negative feelings are directed, for instance, fuel efficiency or seating comfort. attitude of being right where consumers express their rejection of the product and in contrast assert their own correctness. such expressions come in many forms, written both by users who have directly experienced the products on which they are commenting, and by users who have yet to consume or use the product being discussed. ## related work the analysis of feelings is a broad field of research. some behaviors in social media, such as offensive language, sarcasm, irony, and aggressiveness, correspond to the negative sentiment side. cynical expressions are related to the negative aspect and spanish example ## dataset the corpus was generated from youtube comments downloaded from new car analysis channels 1 . the comments were filtered, taking into account two requirements: comments must contain at least ten words, and comments must have a minimum of 5 likes. the goals of these constraints were to ensure sufficient text to judge the presence or absence of cynical expressions, and to focus on comments deemed to be relevant to the discussion. the result was a total of 3189 comments 2 . two annotators were given a set of annotator guidelines containing examples of each type of cynical expression. one of the annotators was a master's student in computer science, and the second was a university teacher in computer science. the annotation guidelines had three sections: an introduction to the topic of consumer cynicism, examples of each type of cynical expression, and 1 @autodinamico, @autossergiooliveira, @autocosmosmx, @gonzalo_conducir, @alonsomaldonado0 examples of what the annotation should look like using different colors to mark the text. the annotators were also given a description of the context of the research and a video tutorial foot_0 on how to use the annotation tool. figure 1 shows a screenshot of the annotation tool. a group of 50 pre-training comments was used to familiarize the annotators with the annotation task. to calculate agreement between annotators, we counted two annotations as matching if the text segment of one annotator was contained within the segment the other annotator. a coverage of 90% of the matching was established. if it was lower, the text was considered a disagreement, and the document was not considered for the machine learning models. we discarded the cynical expression suspicions for having a low level of agreement, and then had the annotators annotate the remaining comments. ## methodology we considered two cynicism detection tasks: token classification we frame the cynicism detection task using the standard inside-outsidebeginning format for token-by-token classification. for evaluation, a 10-fold cross-validation method was performed. for each cynical expression, the following bert models were run: spanberta, mbert, and beto. the parameters with the best performance were: 160 epochs, 3 × 10-5 of the learning rate, and a batch size of 16. the number of epochs during the fine-tuning was 20, 80, 160, and 200. the batch was computed with 16 and 32 sizes. text classification we assigned a label to each youtube comment as positive for a class if any part of the comment was annotated for that class, and as negative if none of the comment was annotated for that class. for evaluation, we used the model (mbert foot_1 ). the training (75%), validation (12.5%), and test(12.5%) collections were constructed. the parameters with the best performance were: 10 epochs and a batch size of 16. however, the number of epochs during the finetuning was 10 and 20. the earlystopping was also included. we also applied the pysentimiento/robertuito model directly, without fine-tuning. we considered several different pre-trained models to be fine-tuned and evaluated on our dataset: beto the beto foot_2 model @xcite was trained following the bert paradigm @xcite , but only on spanish documents. it is similar in size to bert-basedmultilingual-cased. spanberta the spanberta model foot_3 was trained following the roberta paradigm @xcite , but trained on 18 gb of oscar's spanish corpus. it is similar in size to bert-base. we also consider a model trained specifically for hate speech detection, which is related to negative feelings and thus has potential to be usable without fine-tuning on our cynicism corpus. ## results because of the minimal differences between the models for the token classification task, we ran only the mbert model for text classification task. we can see from ## explaining cynicism classifications to give some insights into the behavior of our trained models, we apply lime @xcite to the mbert text classification models. in the following figures, green words contribute positively to the model prediction, and red contribute negatively to the model prediction. figure 2 shows an example of specific reason classification. words like "suspension" and "independent" that relate to a car specification contribute positively, as does 'mazdetos", a spanish term for owners of mazda cars, while words like "previous" and "better" contribute negatively. figure 3 shows an example of attitude of being right classification. the phrase "there's (0.237) more (0.484) value(0.017) for(0.123)" that indicates value assessment contributes positively, while words like "prices(-0.300)" and "money(-0.410)" that are characteristic of the cars have a negative impacts on the model. figure 4 shows an example of negative feeling classification. words that are strongly related to negative sentiment, such as "crap(0.165)", contribute positively to the model, but terms like "people(0.373)" and "money(0.204)" also contribute positively. place of origin of car manufacture, "brazil", and the word "brands" also negatively impact the model. ## discussions the results achieved in the experiment show that it is possible to detect the three cynical expressions with reasonable reliability. some of the results are discussed below.