Applying Transformer Architectures to Detect Cynical Comments in S panish Social Media (2025)

Abstract: Detecting cynical comments in online communication poses a significant challenge in human-computer interaction, especially given the massive proliferation of discussions on platforms like YouTube. These comments often include offensive or disruptive patterns, such as sarcasm, negative feelings, specific reasons, and an attitude of being right. To address this problem, we present a web platform for the Spanish language that has been developed and leverages natural language processing and machine learning techniques. The platform detects comments and provides valuable information to users by focusing on analyzing comments. The core models are based on pre-trained architectures, including BETO, SpanBERTa, Multilingual BERT, RoBERTuito, and BERT, enabling robust detection of cynical comments. Our platform was trained and tested with Spanish comments from car analysis channels on YouTube. The results show that models achieve performance above 0.8 F1 for all types of cynical comments in the text classification task but achieve lower performance (around 0.6-0.7 F1) for the more arduous token classification task.

## introduction the exponential growth of social networks has created an environment where cynical comments, such as sarcasm, negative sentiments, and dogmatic attitudes, can significantly impact discussions and public perception. in this work, we have focused on negative comments that could generate dysfunctional behaviors among social media users. cynical behavior is a negative attitude with a broad or specific focus and comprises cognitive, affective, and behavioral components. cynicism refers to customers' disbelief of companies or the market due to customers' perception of dishonesty and integrity on the seller's part @xcite . also, cynicism can generate feelings of betrayal and deception, leading to anger and the desire to stop purchasing products or services from the source that generates their anger @xcite . in this work, we have focused our efforts on the following elements: sarcasm, negative feelings, specific reasons, and attitude toward being right. such expressions come in many forms, written by users who have directly experienced the products they are commenting on and by users who have yet to consume or use the product being discussed. the automotive industry is relevant to emerging economies @xcite , consumer decision-making, and the strong influence of online opinions on brand perceptions, which impacts the sales of automotive brands. by focusing on this specific domain, we seek to identify linguistic and expressive patterns characteristic of cynicism in digital communication. furthermore, this analysis has broader implications, as the methods developed can be applied to other datasets involving product reviews, services, or online content, allowing for a better understanding of the impact of negative emotions on public opinion. the contributions of our research are as follows: figure 1 shows examples of the elements analyzed in our platform. each comment is shown in the language of study, spanish, with its english translation. ## related work cynical comments are related to negative aspect and are specific elements that characterize the dark side of consumers of products or services. the closest related work are tasks on irony and sarcasm. although both irony and cynicism are close because of the negativity of the content, cynicism can be understood as an extreme form of irony, in which criticism is not only insinuated but used to challenge morality and social conventions openly @xcite . for irony detection, almazrua et al. (2022) created an annota d corpus of tweets with 8089 positive texts in the arabic language. the fleiss's kappa agreement value was 0.54, a moderate level. this work uses machine learning and deep learning models and reports a 0.68 accuracy with the svm algorithm. one of the challenges in this work was detecting implicit phrases as part of the irony. @xcite annotate a corpus of 5566 tweets for the dutch language, with 2783 labeled as irony. this work reported for a binary classification task a 78.98% for implicit irony and 78.88% for explicit and implicit sentiment. the svm model performed better than the bert model. irony has also been approached with cnns and embeddings (fasttext, word2vec) @xcite . this study analyzed monolingual and multilingual architectures in three languages, with the monolingual configuration performing better. a second approach, rcnn-roberta, consisting of a pre-trained roberta transformer followed by bidirectional long-term memory (bilstm), achieved 0.80 f1 on the semeval-2018 dataset and 0.78 f1 on the reddit politics dataset @xcite . in a binary classification task performed on spanish variants for irony detection @xcite , different representation approaches, such as word embeddings (word2vec, fasttext) and n-grams, were presented. our research used contextual transformer representations @xcite . sarcasm detection has received recent nlp research, particularly within sentiment analysis, as sarcasm often leads to misinterpretations of the intended sentiment. early models relied on traditional machine learning techniques, such as support vector machines (svm), which utilized handcrafted features like word frequency and sentiment polarity to detect sarcasm @xcite . however, these methods needed help to capture sarcasm's subtleties and contextdependent nature. recent advancements have led to the adoption of deep learning models, including long-short-term memory (lstm) networks and bidirectional encoder representations from transformers (bert), which have improved performance. these models can better understand the context in which sarcasm occurs, such as hyperbole, tone, or contrast between expectations and reality @xcite . for instance, models like cascade use context-driven approaches to capture sarcasm more accurately by analyzing dialogues on platforms like reddit @xcite . further developments have seen the rise of multimodal approaches that incorporate both text and audio and visual data, which enhance detection accuracy by providing additional cues like facial expressions or intonation. ensemble learning techniques, combining multiple models, have also improved performance in sarcasm detection by leveraging the strengths of different algorithms @xcite . despite these advances, challenges remain, especially when identifying sarcasm in short texts (e.g., tweets) or highly nuanced expressions @xcite . future research will likely focus on improving model robustness in such environments and integrating more sophisticated contextual understanding @xcite . the use of ai for detecting cynicism also intersects with ethical concerns. algorithms designed to filter harmful content sometimes over-censor, inadvertently suppressing freedom of speech by removing comments that are not genuinely harmful but might be misinterpreted by the model @xcite ; this delicate balance between moderating harmful content and preserving free expression is a continuing challenge for ai developers. recent work explores sentiment prediction in online communities, where ai models attempt to predict the likelihood of cynical comments based on previous patterns of behaviors @xcite . while promising, these predictive models are still in the early stages and require more refinement to effectively capture the nuances of negative emotional expression. artificial intelligence has come a long way in detecting explicitly harmful content in social networks, however, it is still difficult to accurately identify cynical negative sentiments. ## dataset our corpus was constructed in several stages. first, spanish-language youtube channels were selected, primarily from latin america and focusing on new car reviews, and their video comments were downloaded. these comments were then filtered to include only those with at least ten words and five likes, ensuring sufficient text for cynicism analysis and focusing on relevant discussion. this initial filtering resulted in 3705 comments. two human annotators independently tagged the filtered comments, freely identifying text segments containing any elements analyzed in this study. to prepare them for this task, we developed a comprehensive visual guide, including: an introduction to consumer cynicism and cynical comments: examples of different types of cynical comments; visual examples demonstrating the annotation process, using color coding to mark the text. the annotators, a computer science master's student, and a computer science professor, also received a description of the research context and an explanatory video. to ensure consistent annotation, a calibration stage was conducted using 50 comments from the initial pool (which were subsequently excluded from the final corpus). inter-annotator agreement was measured by checking if one annotator's marked text segment was contained within the other's. a 90% overlap was considered a match. comments with less than 90% overlap were deemed disagreements and were excluded from the final labeled corpus, which consisted of 2041 comments. finally, comments tagged as "suspicions" were also excluded from the experiments due to their scarcity. ## methodology we consider two tasks for detecting cynical comments. for token classification, we use the standard inside-out-inside format for token-by-token classification. for text classification, we assigned a label to each youtube comment as positive for a class if any part of the comment was annotated for that class and as negative if no part of the comment was annotated for that class. we explored several pre-trained models as potential candidates for fine-tuning and subsequent evaluation on our dataset: beto foot_0 (cañete et al., 2020) was trained following the bert paradigm @xcite , but only on spanish documents. it is similar in size to bert-based-multilingual-cased. spanberta foot_1 was trained following the roberta paradigm @xcite , but trained on 18 gb of oscar's spanish corpus. it is similar in size to bert-base. we further investigate a model that was specifically trained for hate speech detection. this model, which is designed to identify expressions of negativity and hostility, could potentially be directly applied to our cynicism corpus without requiring additional fine-tuning: robertuito 4 is based on the roberta model architecture and the beto tokenizer @xcite . it was trained on 622m tweets in spanish from 432k users for hate speech detection, sentiment and emotion analysis, and irony detection. for token classification evaluation, a 10-fold cross-validation method was performed. for each cynical comment, the following bert models were run: spanberta, mbert, and beto. the parameters with the best performance were: 160 epochs, 3 × 10-5 of the learning rate, and a batch size of 16. the number of epochs during the finetuning was 20, 80, 160, and 200. the batch was computed with 16 and 32 sizes. for text classification evaluation, training (75%), validation (12.5%), and test (12.5%) collections were constructed. for each cynical comment, the following models were run: mbert (fine-tuned on our annotated data) and pysentimiento/robertuito (not fine-tuned on our data). we fine-tuned only mbert because, as will be seen in the results section, there were minimal differences between mbert and the other pre-trained models. the mbert parameters with the best performance were: 10 epochs and a batch size of 16. however, the number of epochs during the fine-tuning was 10 and 20. earlystopping was also included. after the experimentation, the best-performing models were deployed to the huggingface model hub, and we proceeded with the implementation of a web platform. the objective was to create an online platform where the user only places the link to the youtube video, and the analysis is performed automatically. the framework is illustrated in figure 2 . the extraction and data processing models are executed every time a new youtube link needs to be analyzed. the youtube comments are extracted with python using the "youtubecommentdownloader" api. the comments are then subjected to a cleaning, tokenization, and preprocessing process using python. the tensorflow models are used in the web platform through the hugging-face api, which allows models to make predictions using the resources of that platform. ## results tables 3 and 4 show overall results for the token classification task (using a macro-average over the b/i/o labels) and the text classification task, respectively. as with the detailed token classifica- ## codisco platform interface we evaluated several bert-based architectures, of which three have been trained on spanish corpora (spanberta, beto and robertuito) and one was trained on multiple languages (mbert). our prior research suggested that models tuned for the spanish language would obtain the best results @xcite . however, on the current dataset, mbert, spanberta, and beto all performed similarly. for implementing the platform we thus arbitrarily selected beto. we have named our platform codisco foot_3 , after its acronym in spanish (spanish: comportamientos disfuncionales de los consumidores). the apis generated by the huggingface platform are the following: as previously defined in section 4, we wanted to make the interface as easy to use as possible. so, we decided to develop a single screen where the input and output processing are performed when the user enters the internet address of a youtube video. ## discussion the results obtained in the experiment show that it is possible to detect the four types of cynical comments in spanish with reasonable reliability. however, we found some points for reflection. regarding the two tasks analyzed, we found that the performance was higher for the easier text classification task and lower for the more difficult token classification task. however, token classification is closer to the goal of this work, which is to detect exactly which part of the comment represents the cynical comment. it may be helpful to investigate two-stage approaches, in which text classification is first used to identify the general region of cynical comments, and token classification is then used to delineate specific sentences. for comments labeled as negative feelings, the beginnings of utterances (b) were the easiest to identify, probably because they often begin with terms used to describe dissatisfaction. for comments labeled as specific reasons and attitudes of being right, the middle of utterances (i) were the easiest to identify, probably because these types of cynicism include car-specific terms that might be easier to identify. future work could investigate whether joint learning of these models could help better establish the boundaries of the different types of cynical comments. experiments with robertuito highlight that simply using a trained model for hate speech detection will not provide a solution for detecting cynical comments, even in the related category of negative sentiment: an adjusted robertuito achieves only 0.671 f1, whereas an adjusted mbert achieves 0.925 f1. nevertheless, these results indicate some overlap between the two tasks, and the detection of cynical comments could benefit from the hate speech detection models, for example, by using the predictions of the hate speech model as features in the cynical comment detection model. ## conclusions the analysis of cynical comments is crucial, as the sentiments and opinions of vocal customers can significantly influence decisions. even cynical comments may induce undesirable behavior in other people. we annotated a corpus with four types of cynical comments: negative feelings, specific reasons, an attitude of being right, and sarcasm. we trained models on this corpus for text and token classification tasks. our results demonstrate the feasibility of training models to detect cynical comments accurately in this domain. we envision our work as a foundational step toward technologies that can quantify the level of cynicism in youtube videos. such analyses could empower companies to position their products strategically based on consumer perceptions. our implementation with pre-trained models in spanish represents a substantial advancement in comment moderation on platforms like youtube. however, areas for improvement include expanding the corpus to encompass more dialectal variations and enhancing the model's robustness in ambiguous contexts. we plan to fine-tune the model with a complementary corpus for future work. the platform has the potential to be adapted for other languages and applications beyond comment moderation, such as sentiment analysis or fake news detection.