A Case Study of Deep Learning-Based Multi-Modal Methods for Labeling the Presence of Questionable Content in Movie Trailers (2021)

Abstract: In this work, we explore different approaches to combine modalities for the problem of automated age-suitability rating of movie trailers. First, we introduce a new dataset containing videos of movie trailers in English downloaded from IMDB and YouTube, along with their corresponding age-suitability rating labels. Secondly, we propose a multi-modal deep learning pipeline addressing the movie trailer age suitability rating problem. This is the first attempt to combine video, audio, and speech information for this problem, and our experimental results show that multi-modal approaches significantly outperform the best mono and bimodal models in this task.

## introduction movie trailers can be found in abundance throughout the web using services such as video streaming platforms. however, not all types of content in trailers are suitable for every audience. specifically, movie trailers may contain explicit, aggressive, or violent content that may be harmful to the psyche of young viewers. previous research has documented that some of the negative effects of mass media in young viewers include aggression and anxiety @xcite , as well as increasing the risk of sexual onset and alcohol and drug consumption, unwanted pregnancies, and sexually transmitted diseases @xcite . the advertising administration of the motion picture association of america ("mpaa") established guidelines for manually rating the agesuitability of movie trailers (motion picture association). the rating of movie trailers is independent of the rating of the movie itself, as a trailer includes only a short overview of the entire movie. due to the time consuming nature, as well as the challenges to scale the mpaa rating process, automating the task is of practical value. moreover, automating the task poses interesting challenges to multi-modal classification systems, as the source of the objectionable content can come from any, or the combination of, these sources: language (use of bad words or discussion of adult themes), images (graphic violent scenes, nudity, drug or alcohol use), and audio (loud noises and music score denoting suspenseful content). a successful rating approach should integrate evidence provided by the multiple modalities when making the predictions. in this paper, we study the performance of different multi-modal deep learning methods, to automatically predict the mpaa age-suitability rating of movie trailers using cues from the video, audio, and text modalities. our goal is to show the feasibility of automating the rating task, and in particular, the relevance of multimodal solutions. we explore the use of late fusion, feature concatenation fusion and gated multi-modal unit (gmu) fusion @xcite . since the proposed pipeline does not use any type of metadata for trailers or movies, it can be easily extended to be applied to any type of online video content. the main contributions of this work are: (i) we introduce a new task in multi-modal classification; rating videos based on the mpaa rating metric for movie trailers; (ii) we introduce the multi-modal movie trailer rating (mm-trailer) dataset that contains movie trailers and their corresponding mpaa tags, audio files, subtitles of the trailers, and the metadata of the target movie; and (iii) we demonstrate empirically that combining the different modalities yields significant improvements over the strongest monomodal model. our results show that both, the gmu and late fusion approaches yield promising results. ## related work this work is related to four different areas, namely: (i) text classification, (ii) video classification, (iii) audio classification, and (iv) movie classififcation datasets. text classification: in @xcite , the authors proposed an rnn-based architecture for detecting violence in movies on a segment level as well as the full movie level, by using the movie's script. in @xcite , the authors proposed an rnn-based architecture with an attention mechanism that jointly models the genre and the emotions in movie script to predict the mpaa rating of a full movie. the main difference between our work and aforementioned papers is that they only use scripts to predict the movie ratings (violence rating and mpaa ratings), while we employ various modalities (audio, video, and text) to predict if a trailer (not the entire movie) is appropriate or not for children. it should be noted that the rating schema is different for trailers compared to movies (details in section 3), and movies are not freely available on the internet. ## dataset to the best of our knowledge, there is no previous trailer dataset with age-suitability rating. thus, we assembled the multi-modal movie trailer dataset (mm-trailer) 1 by collecting the rated trailers from the imdb website and youtube. typically trailers are advertising movies soon to be released and shown in theaters before a movie starts. rating in trailers is shown by a colored band (red, yellow, green) and a message that appears at the beginning of the trailer. the rating of the trailers adheres to the rating of the movie being shown in the theater. for instance, if the movie playing at the theater is rated as nc-17 (no one under 17 is recommended to watch this movie), the green band trailer that is advertised before this movie may not be appropriate for children even if the color is green. the green band trailers red band trailers total trailers 1,040 403 1, 443 we also extracted separate audio files and trailer subtitles. subtitles include narrator and actor speech. some of the youtube trailers include the video subtitle. for these cases, we pre-process the subtitles by removing timestamps to keep only words. for trailers that do not include a subtitle file, we use a python speech recognition tool @xcite to automatically generate the subtitle from the audio. our dataset includes 11g of audio streams. for each trailer the audio file is a combination of background music and vocals together, so the duration of audio is the same as the duration of the trailer. the number of total words in all trailer scripts is equal to 1,478,139 (on average, there are 576 words per trailer). note that 20,783 words of the vocabulary set are unique words. ## methodology our goal is to predict the age-suitability rating for movie trailers following the guidelines of the advertising administration of mpaa for trailer rating. the problem is formulated as a binary classification task where trailers are labeled as either appropriate for all audiences (green-band trailers) or restricted audiences (red-band trailers). to achieve this goal, the multi-modal movie trailer rating (mmtr) system is proposed. within this system, the trailers are modeled as a fusion of three modalities: subtitles, audio, and video of the trailers. we train recurrent neural networks (rnns) for subtitles and audio, and a combination of convolutional neural networks (cnns) with lstm for video, as separate streams in order to extract a representation for each respective modality. then, we combine all stream representations using a fusion module to take advantage of the cues coming from different modalities. figure 1 shows the overall design for the system architecture. our approach is based on independently identifying the best individual modality model and then combining information from all three monomodal models (subtitle, audio, and video) through one of the following three fusion methods: (i) gated multi-modal unit (gmu) @xcite , (ii) feature concatenation fusion, or (iii) late fusion. all modules of the system are described in the following sections. ## experiments the goal of this section is to demonstrate that a multi-modal approach is an effective way to solve the task. we, therefore, compare the prediction performance of single modality models against all multimodal variations of the system. as mentioned in the dataset section, the mm-trailer dataset is imbalanced. thus, to obtain reliable results, 5 fold cross-validation was selected as an evaluation method. in each fold, we select 10% of the train sent as the validation set to obtain the best model. it should be noted that the dataset was split using the stratified approach, so as to ensure that each set has the same proportion of examples from each class. the metric used to evaluate the performance is the weighted f1 score, averaged over all 5 folds for each experiment. ## results we also report the result for different combinations of two modalities using all fusion methods to show the effect of engaging all modalities (t-bad + a-mfcc, t-bad + v-cnn/lstm , a-mfcc +v-cnn/lstm , t-bad + a-mfcc + v-cnn/lstm). based on the results, the combination of two modalities works better than every single modality, yet not as good as the combination of all modalities. when comparing the different fusion approaches, we can see that gmu fusion outperforms the concatenation fusion systems. we speculate that the gains from gmu come from the ability of the gated unit to dynamically adapt the contribution of each modality to the intermediate representation. statistical significance testing using t-test, demonstrated a significant difference between gmu and feature concatenation fusion (p-@xmath1.05). however, the test does not confirm a significant difference model test-wf single modality baselines most frequent baseline 60.37 text baseline -traditional machine learning 75.02 text baseline -bert+ attention (t-ba) 81.99 text baseline -bert+ attention+ nrc 81.67 text baseline -deepmoji+fc 68.23 video baseline 75.33 audio baseline 72.62 single modality models audio-mfcc (a-mfcc) 73.86 text-bert+ attention+ deepmoji (t-bad) 82.67* video-cnn/lstm (v-cnn/lstm) 79.41 late (fusion using two modalities) t-bad + a-mfcc 82.41 t-bad + v-cnn/lstm 84.12 a-mfcc + v-cnn/lstm 79.68 concatenation (fusion using two modalities) t-bad + a-mfcc 82.17 t-bad + v-cnn/lstm 82.80 a-mfcc + v-cnn/lstm 78.70 gmu (fusion using two modalities) t-bad + a-mfcc 83.37 t-bad + v-cnn/lstm 83.34 a-mfcc + v-cnn/lstm 80.35 fusion using all modalities (mmtr) late (t-bad the results for t-bad and t-ba indicate that deepmoji is a relevant feature for the rating task, and it helps the model to better discriminate redband trailers from green-band ones. however, the result of deepmoji+fc shows that the deepmoji model is not sufficient to solve the task. to obtain a better understanding of fusion results, we also provide other evaluation metrics using the mmtr system variant with gmu fusion in it is thus reasonable to assume that this class is more heterogeneous than the green band class. we plan to explore the possibility of a fine-grained classification of objectionable content as the next steps in this work. ## discussion to analyze the weaknesses and strengths of the mmtr system, we first investigate the incorrectly predicted cases using the most effective version of the system (gmu fusion) on each fold of the data. by averaging results over all folds, about 35% of incorrectly predicted cases with the mmtr system are also incorrectly predicted by each and every modality independently, fusion is thus unlikely to help in this case. we found that in about 50% of the instances where two modalities predict the wrong rating, the mmtr system justifiably trusts the single modality that is correct. in about 93% of the cases where only one modality is wrong, the mmtr gmu fusion variant predicts the correct label, relying on the other two modalities. we also investigated the genre of incorrectly predicted trailers in one of the data folds. the interesting point is that, for incorrectly classified redband trailers, 55% are categorized as "thriller" or "horror" movies and 30% as "comedy" (based on imdb metadata). we do not incorporate metadata into our model to make the model suitable for any kind of online content. this observation shows that the genre of the movie can be a potential feature for the model if we have metadata available. ## conclusion in this paper, we present a deep learning system named mmtr for automating the task of movie trailer age-suitability rating. mmtr fuses information from the video, audio, and text modalities. we also introduce a new data set to support research in this area. this dataset contains movie trailer videos along with their rating and metadata. the results of comparing our model with strong baselines demonstrated that the task is not easy, and a complicated multi-modal systems (gmu and late fusion) can achieve performance gains compared to other baselines. beyond the practical use of a binary classification system, we are interested to move to the more challenging task of detecting the type of objectionable content and introducing explainability elements within the mmtr system.