During real-life interactions , people are naturally gesturing and modulating their voice to emphasize specific points or to express their emotions . With the recent growth of social websites such as YouTube , Facebook , and Amazon , video reviews are emerging as a new source of multimodal and natural opinions that has been left almost untapped by automatic opinion analysis techniques . This paper presents a method for multimodal sentiment classification , which can identify the sentiment expressed in utterance-level visual datastreams . Using a new multimodal dataset consisting of sentiment annotated utterances extracted from video reviews , we show that multimodal sentiment analysis can be effectively performed , and that the joint use of visual , acoustic , and linguistic modalities can lead to error rate reductions of up to 10.5 % as compared to the best performing individual modality . Video reviews represent a growing source of consumer information that gained increasing interest from companies , researchers , and consumers . Popular web platforms such as YouTube , Amazon , Facebook , and ExpoTV have reported a significant increase in the number of consumer reviews in video format over the past five years . Compared to traditional text reviews , video reviews provide a more natural experience as they allow the viewer to better sense the reviewer 's emotions , beliefs , and intentions through richer channels such as intonations , facial expressions , and body language . Much of the work to date on opinion analysis has focused on textual data , and a number of resources have been created including lexicons ( Wiebe and Riloff , 2005 ; Esuli and Sebastiani , 2006 ) or large annotated datasets ( Maas et al . , 2011 ) . Given the accelerated growth of other media on the Web and elsewhere , which includes massive collections of videos ( e.g. , YouTube , Vimeo , VideoLectures ) , images ( e.g. , Flickr , Picasa ) , audio clips ( e.g. , podcasts ) , the ability to address the identification of opinions in the presence of diverse modalities is becoming increasingly important . This has motivated researchers to start exploring multimodal clues for the detection of sentiment and emotions in video content ( Morency et al . , 2011 ; Wagner et al . , 2011 ) . In this paper , we explore the addition of speech and visual modalities to text analysis in order to identify the sentiment expressed in video reviews . Given the non homogeneous nature of full-video reviews , which typically include a mixture of positive , negative , and neutral statements , we decided to perform our experiments and analyses at the utterance level . This is in line with earlier work on text-based sentiment analysis , where it has been observed that full-document reviews often contain both positive and negative comments , which led to a number of methods addressing opinion analysis at sentence level . Our results show that relying on the joint use of linguistic , acoustic , and visual modalities allows us to better sense the sentiment being expressed as compared to the use of only one modality at a time . Another important aspect of this paper is the introduction of a new multimodal opinion database annotated at the utterance level which is , to our knowledge , the first of its kind . In our work , this dataset enabled a wide range of multimodal sentiment analysis experiments , addressing the relative importance of modalities and individual features . The following section presents related work in text-based sentiment analysis and audio-visual emotion recognition . Section 3 describes our new multimodal datasets with utterance-level sentiment annotations . Section 4 presents our multimodal sen-timent analysis approach , including details about our linguistic , acoustic , and visual features . Our experiments and results on multimodal sentiment classification are presented in Section 5 , with a detailed discussion and analysis in Section 6 . In this paper , we presented a multimodal approach for utterance-level sentiment classification . We introduced a new multimodal dataset consisting AU6 AU12 AU45 AUs 1,1 + 4 of sentiment annotated utterances extracted from video reviews , where each utterance is associated with a video , acoustic , and linguistic datastream . Our experiments show that sentiment annotation of utterance-level visual datastreams can be effectively performed , and that the use of multiple modalities can lead to error rate reductions of up to 10.5 % as compared to the use of one modality at a time . In future work , we plan to explore alternative multimodal fusion methods , such as decision-level and meta-level fusion , to improve the integration of the visual , acoustic , and linguistic modalities .

Challenge: The growth of media with a large collection of videos, images, and audio clips calls abilities to identify opinions in such diverse modalities.
Approach: They propose a method for multimodal sentiment classification which identifies expressions in utterance-level visual datastreams and a dataset of sentiment annotated utterances from video reviews.
Outcome: They show that visual, acoustic, and linguistic modalities can reduce error rates by up to 10.5% compared to the best single model model.