A Multi-modal Approach to Fine-grained Opinion Mining on Video Reviews (2020)

Abstract: Despite the recent advances in opinion mining for written reviews, few works have tackled the problem on other sources of reviews. In light of this issue, we propose a multi-modal approach for mining fine-grained opinions from video reviews that is able to determine the aspects of the item under review that are being discussed and the sentiment orientation towards them. Our approach works at the sentence level without the need for time annotations and uses features derived from the audio, video and language transcriptions of its contents. We evaluate our approach on two datasets and show that leveraging the video and audio modalities consistently provides increased performance over text-only baselines, providing evidence these extra modalities are key in better understanding video reviews.

## introduction sentiment analysis (sa) is an important task in natural language processing, aiming at identifying and extracting opinions, emotions, and subjectivity. as a result, sentiment can be automatically collected, analyzed and summarized. because of this, sa has received much attention not only in academia but also in industry, helping provide feedback based on customers' opinions about products or services. the underlying assumption in sa is that the entire input has an overall polarity, however, this is usually not the case. for example, laptop reviews generally not only express the overall sentiment about a specific model (e.g., "this is a great laptop"), but also relate to its specific aspects, such as the hardware, software or price. subsequently, a review may convey opposing sentiments (e.g., "its performance is ideal, i wish i could say the same about the price") or objective information (e.g., "this one still has the cd slot") for different aspects of an entity. aspect-based sentiment analysis (absa) or fine-grained opinion mining aims to extract opinion targets or aspects of entities being reviewed in a text, and to determine the sentiment reviewers express for each. absa allows us to evaluate aggregated sentiments for each aspect of a given product or service and gain a more granular understanding of their quality. this is of especial interest for companies as it enables them to refine specifications for a given product or service, and leading to an improved overall customer satisfaction. fine-grained opinion mining is also important for a variety of nlp tasks, including opinion-oriented question answering and opinion summarization. in practical terms, the absa task can be divided into two sub-steps, namely aspect extraction (ae) and (aspect level) sentiment classification (sc), which can be tackled in a pipeline fashion, or simultaneously (aesc). these tasks can be regarded as a token-level sequence labeling problem, and are generally tackled using supervised learning. the 2014 and 2015 semeval workshops, co-located with coling 2014 and naacl 2015 respectively, included shared tasks on absa @xcite and also followed this approach, which has also served as a way to encourage developments alongside this line of research @xcite @xcite . the flexibility provided by the deep learning setting has helped multi-modal approaches to bloom. examples of this include tasks such as machine translation @xcite , word sense disambiguation @xcite , visual question answering @xcite , language grounding (beinborn et al.; @xcite , and sentiment analysis @xcite . specifically in this last example, the task focuses on generalizing text-based sentiment analysis to opinionated videos, where three communicative modalities are present: language (spoken words), visual (gestures), and acoustic @xcite . although reviews often come under the form of a written commentary, people are increasingly turning to video platforms such as youtube looking for product reviews to help them shop. in this context, @xcite explored a new direction, arguing that video reviews are the natural evolution of written product reviews and introduced a dataset of annotated video product review transcripts. similarly, @xcite recently presented an improved version of the pom movie review dataset @xcite , with annotated fine-grained opinions. although the videos in these kinds of datasets represent a rich multi-modal source of opinions, the features of the language in them may fundamentally differ from written reviews given that information is conveyed through multiple channels (one for speech, one for gestures, one for facial expressions, one for vocal inflections, etc.) in these, different information channels complement each other to maximize the coherence and clarity of their message. this means that although the content of each channel may be comprehended in isolation, in theory we need to process the information in all the channels simultaneously to fully comprehend the message @xcite . in this context, information extracted from nonverbal language in videos, such as gestures and facial expressions, as well as from audio in the manner of voice inflections or pauses, and from scenes, object or images in the video, become critical for performing well. in light of this, our paper introduces a multi-modal approach for fine-grained opinion mining. we conduct extensive experiments on two datasets built upon transcriptions of video reviews, youtubean @xcite and a fine-grain annotated version of the persuasive opinion multimedia (pom) dataset @xcite , adapting them to our setting by associating timestamps to each annotated sentence using the video subtitles. our results demonstrate the effectiveness of our proposed approach and show that by leveraging the additional modalities we can consistently obtain better performance. ## related work our work is related to aspect extraction using deep learning, a task that is often tackled as a sequence labeling problem. in particular, our work is related to @xcite , who pioneered in the field by using multi-layered rnns. later, @xcite successfully adapted the architectures by @xcite which were originally developed for slot-filling in the context of natural language understanding. literature offers related work on the usage of rnns for open domain targeted sentiment @xcite , where @xcite experimented with neural crf models using various rnn architectures on a dataset of informal language from twitter. regarding target-based sentiment analysis, the literature contains several ad-hoc models that account for the sentence structure and the position of the aspect on it @xcite . these approaches mainly use attention-augmented rnns for solving the task. however, they require the location of the aspect to be known in advance and therefore are only useful in pipeline models, while instead we model aspect extraction and sentiment classification as a joint task or using multitasking. aesc has also often been tackled as a sequence labeling problem, mainly using conditional random fields (crfs) @xcite . to model the problem in this fashion, collapsed or sentiment-bearing iob labels @xcite are used. pipeline models (i.e. task-independent model ensembles) have also been extensively studied by the same authors. @xcite performed aesc by modeling the linking relation between aspects and the sentiment-bearing phrases. when it comes to the video review domain, there is related work on youtube mining, mainly focused on exploiting user comments. for example, @xcite exploited crowdsourced textual data from timesynced commented videos, proposing a temporal topic model based on lda. @xcite introduced a similar approach for nico nico, using time-indexed social annotations to search for desirable scenes inside videos. on the other hand, @xcite proposed a systematic approach to mine user comments that relies on tree kernel models. additionally, @xcite performed sentiment analysis on youtube comments related to popular topics using machine learning techniques, showing that the trends in users' sentiments is well correlated to the corresponding realworld events. @xcite presented an analysis of dependencies between comments and comment ratings, proving that community feedback in combination with term features in comments can be used for automatically determining the community acceptance of comments. we also find some papers that have successfully attempted to use closed caption mining for video activity recognition @xcite and scene segmentation @xcite . similar work has been done using closed captions to classify movies by genre @xcite and summarize video programs @xcite . regarding multi-modal approaches for sentiment analysis, we see that previous work has focused mainly on sentiment classification, or the related task of emotion detection @xcite , where the cmu mosi dataset @xcite appears as the main resource. in this setting, the main problem is how to model and capture cross-modality interactions to predict the sentiment correctly. in this regard @xcite proposed a tensor fusion layer that can better capture cross-modality interactions between text, audio and video inputs, while @xcite modeled inter-dependencies across difference utterances of a single video, obtaining further improvements. @xcite are, to the best of our knowledge, the first to tackle scalable multi-modal sentiment classification using both visual and acoustic modalities. more recently @xcite proposed an rnnbased multi-modal approach that relies on attention to learn the contributing features among multi-utterance representations. on the other hand @xcite introduced multi-modal sequence-to-sequence models which perform specially well in bi-modal settings. finally, @xcite proposed a multi-modal, multi-task approach in which the inputs from a video (text, acoustic and visual frames), are exploited for simultaneously predicting the sentiment and expressed emotions of an utterance. our work is related to all of these approaches, but it is different in that we apply multi-modal techniques not only for sentiment classification, but also for aspect extraction. finally, @xcite and @xcite contributed multi-modal datasets obtained from product and movie reviews respectively, specifically for the task of fine-grained opinion mining. furthermore, @xcite recently used the latter to propose a hierarchical multi-modal model for opinion mining. compared to them, our approach follows a more traditional setting for fine-grained opinion mining, while also offering a more general framework for the problem. @xcite utilize a single encoder that receives as input the concatenation of the features for each modality, for each token. this requires explicit alignment between the features of the different modalities at the token level. in contrast, since each modality is encoded separately in our approach, we only require the feature alignment to be at the sentence level. ## task description opinion mining can be performed at several levels of granularity, the most common ones being the sentence level, and the more fine-grained aspect level. finegrained opinion mining can be further subdivided in two tasks: aspect extraction and aspect-level sentiment classification. the former deals with finding the aspects being referred to, and the latter with associating them with a sentiment. previous work usually casts this task as a sequencelabeling problem, where models have to predict whether a token is a part of an aspect and infer its sentiment polarity @xcite @xcite . depending on the dataset annotations, aspect categories are in some cases specified as well. formally, given a sentence @xmath0, . . . , x n ], we want to automatically annotate each token x i with its aspect membership and polarity. in the simpler case where we only want to perform aspect extraction, a common annotation scheme is to tag each token with a label y @xmath1, o, b}. in this scheme, commonly known as iob, o labels indicate that a token is not a member of an aspect, b labels indicate that a token is at the beginning of an aspect, and i labels indicate that the token is inside an aspect. similarly, performing token-level sentiment classification only is equivalent to tagging each token with a label y @xmath2, +, -}, and φ denotes no sentiment, + denotes a positive polarity and a negative one. it is also possible to define a collapsed annotation scheme, where aspect membership and sentiment polarity are encoded in a single tag. we define the label set for this setting as l @xmath3, b+, b-, i+, i-}. labels can be further augmented with type information. for example @xcite used different tags for opinion targets (e.g. b-targ), and opinion expressions (e.g., b-expr), however, we do not rely on this information. ## proposed approach we propose a multi-modal approach for aspect extraction and sentiment classification that leverages video, audio and textual features. this approach assumes we have a video review v containing opinions, its extracted audio stream a, and a transcription of the audio into a sequence of sentences s. further, each sentence @xmath4. these segments do not necessarily cover the whole video i.e. ∪ v s ⊂ v since the reviews may include parts that have no speech and therefore no sentences are associated to those. our end goal is to produce a sequence of labels @xmath5, . . . , y n ] for each sentence @xmath6, . . . , x n ] while exploiting the information contained in v s and a s . figure 1 presents a high-level overview of our approach. we rely on an encoder-decoder paradigm to create separate representations for each modality @xcite . the text encoding module generates a representation for each token in the input text, while the video and audio encoding layers produce utterancelevel representations from each modality. we propose combining these representations with an approach inspired by early-fusion @xcite , which allows for the word-level representations to interact with audio and visual features. finally, a sequence labeling module is in charge of taking the final token-level representations and producing a token-level label. in the following sub-sections we describe each component of our model. ## experimental setup we evaluate our proposal in several experimental settings based on previous work. previous work has also shown that most sentences present a single aspect, and therefore a single sentiment @xcite @xcite , which motivates the introduction of the csl and jsl settings. for these cases we filtered out sentences that do not fit this description. ## results to evaluate the effectiveness of our proposals, we perform several ablation studies on the simple setting for the youtubean dataset. using variations of our baseline with pre-trained glove embeddings (gv), conditional random field (crf), audio and video modalities (a+v). experiments are also performed using 5-fold cross-validation, and comparisons are always tested for significance using paired two-sided t-tests. as we also compare our results to the performance reported by @xcite , who experimented on the simple and csl settings. their models also use pre-trained word embedding -although different from on a more broad perspective, we think the performance differences across datasets are related to the nature of each dataset. meanwhile youtubean contains reviews about actual physical products, which are often shown in the videos at the same time the reviewer is speaking, the pom dataset contains movie reviews where the speakers directly face the camera during most of the video, without utilizing any additional support material. as a result, the video reviews in the youtubean dataset mainly focus on capturing images of the products under discussion, with relatively fewer scenes showing the reviewer. this means that there may be few visual cues in the manner of facial expressions or other specific actions that the models could exploit in order to perform better at the sentiment classification task, but more cues useful for aspect extraction. this situation is reverted in the pom dataset, which could explain why our models tend to perform better for sentiment classification, but offering smaller gains for the ae task. we also think performance differences across datasets are to some extent explained by the nature of the annotations on each case. the annotation guidelines utilized to elaborate each dataset are actually quite different, with the annotations in the youtubean dataset closely following those of the well-known semeval datasets, which are target-centric and the pom standards substantially diverging from this. concretely, @xcite propose a two-level annotation method, where "the smallest span of words that contains all the words necessary for the recognition of an opinion" are to be annotated. as a result, aspects annotated in the pom dataset often include pronouns which are more difficult to identify as aspects, often requiring co-reference resolution. with regards to aspect polarity, while it can be extracted directly from the youtubean annotations, in the case of pom we needed some pre-processing as target and sentiment are annotated using independent text spans. qualitative results of the pom and youtubean dataset in a multitask cal can be seen in figure 3 and 4 respectively, results suggest that the method learn to use the information from additional modalities and enhance the sentiment and aspect prediction. finally, as we observe that our models tend to obtain bigger gains on the ae tasks rather than on sc, we think this behavior can be partially attributed to the inductive bias of our model, which makes it specially suitable for sequence segmentation tasks. ## conclusions in this paper we have presented a multi-modal approach for fine-grained opinion mining, introducing a modular architecture that utilizes features derived from the audio, video frames and language transcription of video reviews to perform aspect extraction and sentiment classification at the sentence level. to test our proposals we have taken two datasets built upon video review transcriptions containing fine-grained opinions, and introduced a technique that leverages the video subtitles to associate timestamps to each annotated sentence. our results offer empirical evidence showing that the additional modalities contain useful information that can be exploited by our models to offer increased performance for both aspect extraction and sentiment classification, consistently outperforming text-only baselines. for future work, we are interested in exploring other ways to capture cross-modal interactions, exploit the temporal relationship between the representations of different modalities, and test alternative ways to better deal with our multi-task settings.