Joint Estimation and Analysis of Risk Behavior Ratings in Movie Scripts (2020)

Abstract: Exposure to violent, sexual, or substance-abuse content in media increases the willingness of children and adolescents to imitate similar behaviors. Computational methods that identify portrayals of risk behaviors from audio-visual cues are limited in their applicability to films in post-production, where modifications might be prohibitively expensive. To address this limitation, we propose a model that estimates content ratings based on the language use in movie scripts, making our solution available at the earlier stages of creative production. Our model significantly improves the state-of-the-art by adapting novel techniques to learn better movie representations from the semantic and sentiment aspects of a character’s language use, and by leveraging the co-occurrence of risk behaviors, following a multi-task approach. Additionally, we show how this approach can be useful to learn novel insights on the joint portrayal of these behaviors, and on the subtleties that filmmakers may otherwise not pick up on.

## introduction in one of the longest running movie franchises in history, fictional british secret service agent james bond is more often than not portrayed as an extremely charming gentleman, a cold-blooded killer, a smoker, and a severe alcoholic @xcite . this is not a unique character trait, as other critically acclaimed films-such as, the exorcist @xcite , pulp fiction @xcite , and a clockwork orange @xcite follow narratives where the main characters engage in a similar collection of risk behaviors. the portrayals of these risk behaviors typically include acts of violence, sexual and substance-abusive behaviors in scenes of fighting, bloodshed, gunplay; intercourse and nudity; and alcohol, smoking and drug use, respectively. while these tend to attract audiences @xcite and facilitate a movie's global market reach @xcite , they have long sparked concerns about the potential side effects of repeated exposure. particularly, in the case of at-risk populations, such as children and adolescents, where this exposure has been linked to increased risk for engaging in violence @xcite , smoke and alcohol consumption @xcite , and earlier sexual initiation @xcite . although various automated tools have been designed to recognize risk behaviors portrayals (e.g., @xcite ), many rely on cinematic principles from film theory such as illumination, rapid shot transitions or musical score selection @xcite . this limits their practical impact to an almost-final edition of the content, specifically where visual and sound effects have been added in, making it too late or expensive to implement any modifications. hence, there is an opportunity on being able to identify these depictions from an earlier stage of content creation as to offer additional useful insights for film-makers and movie producers during the complex creative process. to this end, our work leverages on two key insights: first, that while all of these works focus on a specific behavior, risk behaviors frequently cooccur with one another both in real-life @xcite and in entertainment media @xcite @xcite . second, that the language use in movie scripts can characterize portrayals of risk behaviors at the earliest form of content creation-even before production begins. for example, by identifying when mr. bond orders his usual alcoholic drink, pulp fiction's main characters plotting to kill someone, or the evil incarnated in the exorcist cursing in a sexually explicit manner. the present work, to the best of our knowledge, is the first to model the co-occurrence of risk behaviors from linguistic cues found in movie scripts. our proposed model is a multi-task approach that predicts a movie script's violent, sexual and substance-abusive content from vectorial representations of the character's utterances. we hypothesize that this multi-task approach will help improve violent content classification, as well as in providing insights on their relation to other dimensions of risk behaviors depicted in film media. specifically, the contributions of this work are: we use this model to obtain better representations for the semantics of a character's language 3. a novel large-scale analysis on the joint portrayals, and their relation to other ratings, of violence, sex, and substance abuse in film. ## related work to understand the prevalence of risk behaviors in film and tv, social scientists have often relied on relatively small human annotated data sets (typically under a 100). this includes a study of portrayals of violence in 74 to 77 films from the last decade @xcite , as well as portrayals of teenage sex in 90 of the top-grossing films @xcite . among other findings, these studies provide evidence that mpaa 2 ratings (the primary rating system used for films in the u.s.) are overly sensitive to sexual content, and less effective at identifying other types of risk behaviors @xcite . however, most of these works are limited to the study of a particular behavior, even though risk behaviors frequently cooccur with one another in media @xcite @xcite ). the task of identifying risk behaviors from language is perhaps closely related to that of recognizing abusive language (al; @xcite . al is an umbrella term that includes offensive language, including sexist and racist language, and hate-speech. al computational models are usually designed using popular document classification techniques @xcite , based on features such as n-grams @xcite ; affective language @xcite and distributed semantic representations @xcite . recent efforts (e.g., @xcite explore a supervised finetuning approach that start from pre-trained models of highly-contextualized word representations from transformers @xcite . most similar to our work are efforts in predicting a single movie-level rating from language either in movie scripts @xcite or in transcripts @xcite . these works explore the use of recurrent neural networks (rnn) over sequences of vector representations, each composed by the concatenation of lexical, semantic and sentiment features, to learn a movie representation from which the target rating is predicted. there are two notable differences between these and our proposed model. first, our model incorporates additional information in the form of other prediction targets (i.e., multi-task paradigm) and multiple attention layers @xcite . the former is motivated by the previously mentioned notion that characters tend to engage in joint portrayals of risk behaviors @xcite ; the latter allows the model to jointly attend to information from different representation sub-spaces. second, these previous works explore an early-fusion method where linguistic features are concatenated and fed to a self-attention mechanism on top of the rnn layer. this assumes that in an effort to construct a meaningful interpretation of the features, the attention layer will be powerful enough to disentangle different aspects of language, such as semantic and sentiment. instead, we use a late-fusion approach where we separate semantics from sentiment, and direct them through different pathways in our model-all the way up to independent attention layers. thus, our attention layers have the relatively easier task of identifying what is of importance for a particular view of language in a particular task. this allows our model to attend to what is being said (semantic) and, independently, how it is being said (sentiment). we expect this to be more informative about the content of each utterance, leading to a better representation construction. ## method our model learns to map sequences of character utterances' representations to overall movie-level ratings. each representation is composed by two parts: one representing its semantics, and one for its sentiment. these representations are obtained from models trained on larger out-of-domain corpora but have been validated on related tasks in domains similar to those we study in this work (e.g., classification of movie review sentiment @xcite ). our decision to start from character utterance representation (as opposed to word representations) comes from the limited number of labeled expert curated content ratings in our dataset (see section 4). ## data we collected a large number of movie scripts from three publicly available sources. the first source was related works who shared their movie scripts datasets @xcite ; the second source was online collections of produced scripts foot_1 , and the final source was online communities where non-produced scripts are shared 5 . in total we collected 12, 706 scripts, some of which correspond to produced films or tv episodes. to improve the quality of this dataset, we clean it by extracting text, limiting to files with more than 1, 000 lines, and replacing nonascii characters. in case of any error, we remove the file from the collection. this procedure resulted in 6, 057 movie scripts spanning 23 genres with an average of 1450.6 utterances per movie (σ = 456.11, @xmath0.0). we use this collection to fine-tune moviebert. to evaluate the performance of our model, and directly compare it to previous work, we manually align a subset of 989 movie scripts from our dataset to the content ratings found in @xcite . these ratings come from common sense media (csm), a non-profit organization that promotes safe technology and media for children foot_3 . csm experts rate movies from 0 (lowest) to 5 (highest) with each rating manually checked by the executive editor to ensure consistency across raters. a manual inspection of the dataset revealed that the movies with the least scores across all risk behaviors correspond to the romantic genre, whereas the movies with the most risky content were in the horror genre. additionally, we investigate if csm expert raters capture the co-occurrence of risk behavior portrayals. figure 2 shows that, on average, when one risk-behavior rating increases so does the others. this was corroborated by significant positive spearman's correlations between violence and sexual content (@xmath1.161, @xmath2.001); violence and substance-abuse (@xmath3.129, @xmath4.001), and sexual content and substance-abuse (@xmath5.467, @xmath6.001). ## experimental setup in this section we discuss the model implementation, parameter selection, baseline models and sensitivity analysis setup. ## co-occurrence analysis in this section, we focus on some of the insights that our proposed model may provide film-makers and producers during the creative process. in particular, our analyses centers on three insights: first, on understanding how joint portrayals of risk behaviors appear on screen; second, in identifying temporal patterns that arise from these joint portrayals, and finally, in showcasing the relation between risk behaviors and mpaa ratings. for this analysis, we re-trained the best performing model over the complete movie script dataset (@xmath7. on the relation between joint portrayals of risk behaviors. we find a strong association between predictions of substance-abuse and sexual content: the odds for a movie script to be rated high on sexual content are twice as high when it has a high rating in substance-abuse compared to when it has a low rating (95% confidence interval [ci] 2.01 to 34.05). moreover, we find that the odds of rating high on all three risk behaviors simultaneously are inversely proportional to the predicted violence rating (95% ci, high:0.11 to 0.82 and med:0.12 to 0.88). hence, this suggests that film-makers compensate low levels of violence with joint portrayals of sexual and substance-abuse behaviors. on the temporal patterns of the joint portrayals. if there is a temporal relation between the portrayals, when the model picks up a cue for a particular behavior at time t (i.e., a spike in the attention signal), we expect to see a corresponding spike in the attention signal of another task some time after t. to compute this relation, for each movie script we obtained the maximum correla-tion and its corresponding time lag (∆ ∈ [-m, m]) by using sample cross correlation function (ccf) between the attention weights of each task. ccf is a measure of similarity between two time series as a function of the displacement of one relative to the other. as an example, fig. 4 shows the co-evolution of attentions weights and the lags corresponding to their maximum correlation for two movies. on average, attention to the sexual sentiment content precedes attention to violence semantics by ∆ = 15.50 utterances (95% ci, 10.88 to 17.4), with an average correlation coefficient of @xmath8.192 ± 0.02. this lag increases for movies with higher content ratings on both violence and sex ( ∆ = 21.46, @xmath9.202), whereas movies with low sex and violent content have almost no temporal difference, and a significantly lower correlation coefficient ( ∆ = 0.75, @xmath10.172, perm.test @xmath11, @xmath12.034). these results suggests, as @xcite points out, that characters engage in sexual and violent behaviors in a small time span from one another. on the relation between risk behaviors and mpaa ratings. finally, we measure the relation between the predicted risk behaviors and the movie's mpaa rating. we find that as sexual content increases, the association between violent (or substance-abuse) content and mpaa rating decreases. specifically, movies with high sexual rating are more likely to be rated as r foot_5 , irrespective of their violent or substance-abuse content (odds ratio @xmath13.172 (95% ci: 7.86 to 19.46)). in contrast, the mpaa rating of a movie with low sexual content is strongly associated with both their violent content rating (χ 2 (6) = 18.595, @xmath14.004) and their substance-abuse content rating (χ 2 (3) = 17.99, @xmath15.001). these results point out the overly sensitiveness of mpaa raters towards sexual content and corroborate previous findings from small manuallyannotated samples of films @xcite . ## conclusion we designed a multi-task model to capture the cooccurrence of depictions of violent content as well as sexual and substance abuse risk behaviors in film through the language data available in scripts. our proposed model achieves significant improvements over previous state-of-the-art models for violent content rating prediction. while complementing audio-visual methods, our language-based models can be used to identify subtleties in the way risk behavior content is portrayed, before production begins, offering a valuable tool for content creators and decision makers in entertainment media.