From None to Severe: P redicting Severity in Movie Scripts (2021)

Abstract: In this paper, we introduce the task of predicting severity of age-restricted aspects of movie content based solely on the dialogue script. We first investigate categorizing the ordinal severity of movies on 5 aspects: Sex, Violence, Profanity, Substance consumption, and Frightening scenes. The problem is handled using a siamese network-based multitask framework which concurrently improves the interpretability of the predictions. The experimental results show that our method outperforms the previous state-of-the-art model and provides useful information to interpret model predictions. The proposed dataset and source code are publicly available at our GitHub repository.

## introduction estimating the severity level of objectionable content for movies can provide convenience for users to judge whether a movie is suitable for watching. for example, parents may want to make sure there are no violent scenes in a movie when they plan to watch it with their kids, because exposure to violent scenes may increase youth aggressive behavior and decrease their empathy @xcite . however, existing rating systems (e.g., mpaa) only provide simple age restrictions and do not include the suitability level on a specific aspect of the content. furthermore, a system that can automatically track the severity level of objectionable content helps the creative professionals evaluate the age suitability of their work. they may get assisted by this function and adjust the product creation or marketing strategy based on the corresponding target audiences. this system can be easily applied to any dialogue-intensive compositions like novel and screenplay writing. content evaluation and intervention by the writers can happen at any stage of production to assess age-restricted contents. in this work, we propose to solve the problem of predicting the severity of age-restricted content solely using the dialogue script data. text is much more lightweight than visual data (such as images and videos), so the processing procedure can be more efficient and scalable considering the increasing fidelity of multimedia content. we initiate our exploration on movies from five aspects of contents: there are a small number of previous works that studied modeling age-restricted content. @xcite initiated the research of predicting mpaa ratings of the movies leveraging movie script and metadata. @xcite focused on violence detection using movie scripts while @xcite expanded the scope to violence, substance abuse, and sex. both works intended to predict the severity of age-restricted content into three manually defined levels: low, mid, and high. in this work, we introduce two more aspects of interest: frightening and profanity. instead of manually downgrading severity levels into three categories, we explore with a more challenging setting: rating on 4 originally defined fine-grained severity levels from collective rating by customers: none, mild, moderate, and severe. the major contributions of our research can be summarized as follows: (1) this work is the first attempt to solve the age-restricted content severity predicting problem from 5 aspects. we studied multiple baselines and presented a competitive method to inspire future exploration. (2) to our best knowledge, the dataset we developed is the first publicly available dataset for this task. the size is roughly five times larger than the restricted datasets from the previous works. (3) we proposed an effective multitask ranking-classification framework to solve this problem. our method dealt with long movie scripts successfully and achieved state-of-the-art results in all five aspects of age-restricted contents with rich interpretability. ## methodology we investigate predicting the severity of 5 objectionable aspects of movies. this problem is formulated as a multi-class classification task. the average length of the dialogue scripts is around 10,000 words, which drastically exceeds the limit of current popular transformer-based models. to leverage the strong semantic representation capability of the transformers, we propose to represent each utterance as the basic unit, and further encode the context with the recurrent modules. finally, we use a fully connected layer on top of the encoded representations to produce the classification predictions. for this model, we first leverage sentencetransformers @xcite to encode each dialogue utterance. then, a bi-directional lstm encoder is deployed to model the sequential interrelations of the utterance flow. we finally apply a max-pooling operation on all time steps of the hidden states of the recurrent module to get the document representation for classification following the practice in @xcite . we also study another strong word-level deep learning model, textrcnn @xcite , to probe the significance of lexical signals. ## dataset the dataset used in this work was developed based on the script data used in @xcite . we collected the up-to-date user ratings for age-restricted content from imdb.com for more than 15,000 movies. the age-restricted aspects are adopted from the parents guide section of each movies, and there are five aspects: sex & nudity, violence & gore, profanity, alcohol, drugs & smoking, and frightening & intense scenes. each of the aspects has four severity levels for the users to rate on the corresponding movies from low to high, which are none, mild, moderate, and severe. in this work, we pick the ratings on the website as the category label for each aspect. after collecting the user ratings, we filter out movies with less than 5 votes to make sure the collected severity level is robust to use. at last, we have roughly 4,400 to 6,600 movies for each aspect. the movie scripts have a median/average length of around 10,000 words. the vocabulary size of each aspect is roughly 330,000 to 450,000. detailed dataset descriptions are attached in the appendix. comparing to the previous works with restricted data access @xcite , our dataset is roughly five times larger, and the data source is free to access. we made the updated dataset publicly available with the same data partitions in this work for reproducibility purposes. ## experimental results we evaluate the effectiveness of different methods on macro f1 score because the data is imbalanced. the dataset is first split and stratified using an 80/10/10 ratio for training, development, and test for each age-restricted aspect. the experimental results on the test set are reported in baseline models include average glove embedding @xcite with svm/logistic regression, textcnn @xcite , and bert @xcite . words or a frightening shot with screaming, however, not all such scenes come with particular dialogues. for sex & nudity and alcohol, drugs & smoking, it is even more difficult to infer from plain text without any visual evidence. we also experiment with models from related previous works by @xcite . the former leverages lstm with attention and nrc emotion to model the textual signal to predict mpaa ratings for the movie with the script; the latter uses rnn on dialogue encoded by a pretrained moviebert model and sentiment embeddings to predict severity of different aspects simultaneously. for fair and reliable comparison, we conduct the benchmarking within the same setting of using the dialogue script as the only available information in the same 10-fold cross-validation configuration from @xcite . ## discussion and analysis we investigate several popular movies with at least 200,000 imdb ratings from the test set of each aspect. we collect the top 5 movies with the largest number of severity ratings from each severity level as comparators, then do the pairwise ranking between each movie and all its comparators for test. comparison results are shown in successful examples: for the movie deadpool (2016), the model gives a correct prediction on the profanity aspect with a convincing signal from pairwise ranking. this movie is determined to have a higher severity level in profanity than any candidate comparators from none/mild/moderate level. for severe-level comparators, the test instance is lower than one, equal to three, and higher than one of the comparators. the movie pride & prejudice (2005) comes with a correct prediction on violence. four comparators from none show they have equal severity while only one denotes higher. three mild comparators give equal results while two indicate the test movie is lower than them. for moderate and severe comparators, all of them evince the test movie has lower severity. we can therefore reason that @xcite has a severity level of none in terms of violence content. there is a similar case as presented in the greatest showman @xcite . it tends to appear higher than none but roughly mild level among the comparators. then the model gives it a correct prediction to mild. therefore, by investigating the pairwise ranking results, the interpretation of this prediction is more vivid to make sense to users concerning the relative severity level. unsuccessful examples: for the movie django unchained (2012) on nudity and the movie a clockwork orange (1971) on alcohol, drugs & smoking, the ranking gives a very clear pattern: higher than none, equal to mild, and lower than moderate. however, the actual severity level is moderate instead of mild. we conservatively argue aspects such as nudity and alcohol, drugs & smoking can hardly be inferred from dialogue text. it tends to be unsurprising if there are adult scenes or drinking scenes existing in a movie without any verbal indication. ## conclusion & future work in this work, we applied a deep learning-based method to predict severity of age-restricted content based on movie script data. the experimental results show the proposed multi-task rankingclassification model outperforms the previous stateof-the-art method and can give rich interpretability by demonstrating severity using example comparator movies. our work provides a reasonable groundbreaking exploration in this research topic for the community. for future work, we propose to investigate other modalities to capture relevant patterns and fine-grained aspects like violence types.