Sentiment Analysis on Video Transcripts: Comparing the Value of Textual and Multimodal Annotations (2025)

Abstract: This study explores the differences between textual and multimodal sentiment annotations on videos and their impact on transcript-based sentiment modelling. Using the UniC and CH-SIMS datasets which are annotated at both the unimodal and multimodal level, we conducted a statistical analysis and sentiment modelling experiments. Results reveal significant differences between the two annotation types, with textual annotations yielding better performance in sentiment modelling and demonstrating superior generalization ability. These findings highlight the challenges of cross-modality generalization and provide insights for advancing sentiment analysis.

## introduction with the rise of the internet and online platforms, especially the proliferation of social media, usergenerated content (ugc) has become widely accessible to the public. ugc appears in various forms and modalities, ranging from online textual movie reviews on platforms like imdb foot_0 and rotten tomatoes foot_1 , to video blogs (vlogs) in video-sharing platforms such as youtube foot_2 and tiktok 4 . ugc holds significant value for companies, marketers and politicians @xcite , as it contains sentiment-rich information that can be leveraged to monitor public opinion and support the decision-making process @xcite . for instance, sentiment analysis based on tweets has been utilized to model user satisfaction in mobile payments @xcite and predict election outcomes @xcite . sentiment analysis on ugc predominantly focuses on text, partly because textual sentiment modelling is more developed and computationally effi-cient compared to other modalities, such as audio and video. in contrast, systems capable of automatically understanding the content and sentiment of videos are still in their infancy @xcite . consequently, sentiment analysis of non-textual ugc is often converted to text-based analysis through subtitles or transcripts. for instance, @xcite investigated the use of video transcripts to capture contextual and emotional information in videos. a critical question arises when annotating transcripts: should information from non-textual modalities be considered during annotation? in real-life scenarios, sentiment annotations typically reflect the emotional status across modalities. as a result, some studies incorporate multimodal information into the final annotation @xcite @xcite . however, another common approach is to perform annotation solely based on textual information, excluding other modalities to avoid interference @xcite @xcite . this approach is practical since the input for sentiment modelling is usually text, and annotating textual data is less complex compared to multimodal data. both approaches to sentiment annotation have their merits and are often intertwined. in some cases, researchers do not differentiate between them, applying multimodal annotations to textual sentiment modelling under the assumption that sentiment labels across modalities are consistent. however, this assumption does not always hold true. for instance, the text "i love this weather" might be labelled as positive, but when the speaker's tone is sarcastic and they wear a frown, the sentiment might be perceived as negative. previous studies have shown that emotion labels in multimodal setups do not always align with those derived from textual modalities alone @xcite @xcite . accurate annotations are crucial for building effective models. however, in the field of sentiment analysis on ugc transcripts, few studies have compared sentiment annotations derived solely from textual information with those incorporating multimodal information, or examined the impact of these differences on sentiment modelling. to address this gap, this paper seeks to answer the following research questions: recently, sentiment analysis has evolved beyond textual analysis to incorporate other modalities, such as audio and video, giving rise to multimodal sentiment analysis @xcite . this shift reflects the growing prevalence of opinion-sharing in video formats on platforms like youtube and tiktok @xcite , where diverse modalities provide richer contextual information for understanding sentiments. an essential aspect of multimodal sentiment analysis is the fusion of different modalities @xcite . fusion strategies are broadly categorized into two types: early fusion and late fusion. early fusion, also known as feature-level fusion, integrates features from each modality at the input level, whereas late fusion, or decision-level fusion, combines the outputs of unimodal sentiment analyses to generate the final prediction. recently, advanced fusion approaches, such as tensor fusion networks @xcite and dynamic fusion methods @xcite , have been proposed to enhance performance. while incorporating non-textual information generally improves the performance of multimodal sentiment analysis, there remains a heavy reliance on textual modalities. this phenomenon, termed text-predominance, is evident in studies showing a significant drop in classification accuracy -from approximately 80% to 54% -when textual information is excluded from multimodal models trained on multimodal data @xcite . in contrast, removing audio or visual information results in only a marginal accuracy decline, such as a reduction from 87% to 85% @xcite , a trend corroborated by @xcite . it seems that we can still rely on textual information despite the availability of other modalities, especially when considering the imbalance of the cost and the improvement when introducing nontextual modalities. however, when we decide to take into consideration only the textual modality of the opinioned videos, which set of annotations should be used, the textual one or the multimodal one, as multimodal labels do not always reflect sentimental states in texts @xcite . in the following, we are going to investigate the differences and influence of the two sets of sentiment annotations. ## datasets the definition of ugc varies across disciplines. in the context of social media, ugc is defined as any kind of text, data or action performed by online digital systems users, published and disseminated by the same user through independent channels, that incur an expressive or communicative effect either on an individual manner or combined with other contributions from the same or other sources @xcite . based on this definition, we selected two datasets for our study: the ugc dataset unic @xcite , and the non-ugc dataset ch-sims @xcite . unic is an english audio-visual emotion dataset with independent annotations for each modality (i.e., text, audio, and silent video) as well as overall emotion states of the videos. this ugc dataset comprises nearly 1,000 video clips collected from youtube, focusing on the topic of reviews. ch-sims is a chinese multimodal sentiment analysis dataset featuring over 2,000 curated video segments with both multimodal and independent unimodal annotations. the videos in ch-sims are sourced from movies, tv series, and variety shows, implying that the professional actors in ch-sims tend to express emotions more explicitly in all modalities than the non-professionals in unic. this difference may also influence the sentiment annotations across modalities. for both datasets, sentiment labels were originally designed as negative, weakly negative, neutral, weakly positive and positive. in this paper, we grouped weakly negative and weakly positive into negative and positive, respectively, for further experiments and analysis. ## conclusion this study investigated the differences between sentiment annotations on video transcripts derived from textual and multimodal setups, as well as their impact on transcript-based sentiment modelling. the statistical analysis revealed a significant difference between the two types of sentiment annotations with absolute similarities less than 70% and kappa scores less than 0.55, highlighting the influence of multimodal information on sentiment labelling in video data. the modelling experiments further demonstrated that text-based annotations outperformed multimodal annotations when evaluated against both textual and multimodal labels. also, a significant cross-modality performance gap was observed. for instance, the macro f1 score dropped from 74.62 to 53.03 when the evaluation labels shifted from text-based to multimodality for unic, underscoring the challenges of generalizing sentiment models across different modalities. for future research, we will investigate the in-corporation of additional modalities (e.g., audio and facial expressions) and advanced models (e.g., multimodal fusion models), enabling a more comprehensive and nuanced analysis.