Supporting Comedy Writers: Predicting Audience’s Response from Sketch Comedy and Crosstalk Scripts (2020)

Abstract: Sketch comedy and crosstalk are two popular types of comedy. They can relieve people’s stress and thus benefit their mental health, especially when performances and scripts are high-quality. However, writing a script is time-consuming and its quality is difficult to achieve. In order to minimise the time and effort needed for producing an excellent script, we explore ways of predicting the audience’s response from the comedy scripts. For this task, we present a corpus of annotated scripts from popular television entertainment programmes in recent years. Annotations include a) text classification labels, indicating which actor’s lines made the studio audience laugh; b) information extraction labels, i.e. the text spans that made the audience laughed immediately after the performers said them. The corpus will also be useful for dialogue systems and discourse analysis, since our annotations are based on entire scripts. In addition, we evaluate different baseline algorithms. Experimental results demonstrate that BERT models can achieve the best predictions among all the baseline methods. Furthermore, we conduct an error analysis and investigate predictions across scripts with different styles.

## introduction comedy plays a major role in people's lives in that it relieves stress and anxiety @xcite . there are two popular types of comedy: sketch comedy and crosstalk. a sketch comedy usually presents a short story and is performed by multiple comedians in various short scenes; while in a crosstalk performance, which is similar to a talk show, there are usually two performers telling humorous stories behind a desk. although these two types of comedy are different, both of them are performed based on scripts. a script breaks down a story into pieces along with the details that describe which performer should take what action or say which lines at a specific point @xcite . therefore, the quality of the script is critical and it directly influences whether the audience enjoys the performance. however, it is difficult for script writers to ensure a high-quality comedy script and be productive. firstly, writers have to assess if audiences will react as expected, in particular laughing at specific points. it is necessary to rehearse multiple times to continuously improve the script, which is time-consuming and can be costly. secondly, to develop laughter triggers, writers need to identify the potential points from the script where there are possibilities for performers to use funny body moves, tone or tell amusing stories to make the audience laugh. thirdly, the more times a script is publicly performed, the less laughter it can bring, since the audience have become too familiar with it. thus, it is essential for comedy writers to explore new laughter triggers constantly. since natural language processing (nlp) has been widely and successfully applied to a number of fields @xcite , we investigate how nlp methods can support comedy writers to produce high-quality scripts more efficiently. this paper specifies this challenge as a new task, i.e. the prediction of the audience's response to sketch comedy and crosstalk scripts. to address this challenge, we explore the use of two different nlp methodologies: 1) text classification: we predict whether or bing jia: okay. (he duly closes his eyes) legend of laughter (season 1), 2017 contributions firstly, we introduce a chinese corpus of annotated comedy scripts collected from popular tv entertainment programmes. our annotations include both text classification and information extraction labels. tables 1 and 2 present annotation examples. the corpus can be used to build an intelligent system to benefit the script writing for comedy writers. it may also be useful for dialogue system research and discourse analysis. secondly, we evaluate a number of nlp methods and the results demonstrate that bert models @xcite are able to achieve the best prediction performance among all methods. we also further conduct an error analysis which may be useful for further improving the performance. lastly, we experimentally show that our corpus can also be used to predict laughter triggers for scripts which have very different styles compared to training data. our work is closely related to humour detection, which has been widely studied for many years in natural language processing. mihalcea and strapparava (2006); yang et al. (2015); chen 2 the lines are from the dialogue of a comedy performance. each line consists of an actor's name and the sentences this actor speaks in performance. and soo (2018); blinov et al. (2019) investigated if a text fragment is a one-liner. 3 zhang and liu (2014); ortega-bueno et al. (2018); chiruzzo et al. (2019) explored the humour classification task on tweets. castro et al. (2018) collected humour values and funniness scores of spanish tweets by using crowdsourcing. chiruzzo et al. (2019) proposed a regression task that predicts the humour score for a tweet. li et al. (2020) collected chinese internet slang expressions and combined them with a humor detecting method to analyse the sentiment of weibo 4 posts. it should be noted that the examples in all of the corpora used or constructed in the above-mentioned studies are independent of each other. since our corpus is based on entire scripts, the annotated lines and text spans might also benefit the researchers who are interested in modelling long-context-aware algorithms to understand humour. apart from the studies on short text fragments, bertero (2019) and hasan et al. (2019) created corpora from television (tv) sitcoms such as the big bang theory 5 and ted talks 6 respectively. their goal is to predict whether or not a sequence of texts will trigger immediate laughter. yang et al. (2015); zhang et al. (2019) extracted the key words such as sing, sign language and pretty handy from jokes, which are similar to our information extraction annotations. 44 3 corpus 3.1 data collection source selection in order to ensure the highquality of scripts, we carefully selected thirty performances (the total duration is approximately 473 minutes), including both sketch comedies and crosstalks, of which the leading roles are famous chinese comedians. these performances were played on well-known chinese tv entertainment programmes such as chinese new year gala and ace vs ace 7 . since there were many people in the audience present for the recording of these performances, the annotators can judge whether the audience laughed based on the performance videos. please refer to the appendix for the full list of performances which gives details of their titles, leading comedians and sources. lastly, we manually typed up actors' lines for each performance and completed thirty scripts. although there may be differences between our scripts and the real scripts used by comedians in terms of format or content, we assume that our scripts contain the key information about the real scripts, i.e., the actors' lines. therefore the corpus can be useful for the development of intelligence-assistant comedy script writing systems. ## baselines and results discussion in order to understand how well the machine learning methods work on our corpus, we evaluate the performances of a number of models on 5-fold cross-validation random splits of the scripts in our corpus and report the average results in this section. 10 all the bert models were pre-trained by using a mixture of large chinese corpora. 11 please 9 the house is actually small. since there is almost no furniture in the house, the comedian said it looked big. 10 model implementations were adapted from https://github.com/649453932/ chinese-text-classification-pytorch , https: //github.com/luopeixiang/named_entity_ @xcite . 11 more details are listed in https://github.com/ dbiir/uer-py/wiki/modelzoo . baselines tables 5 and 6 respectively present the results of text classification and information extraction. bert-base has the best f1-scores among all the methods. we also note that the classification recall of rcnn @xcite is much higher than other methods. therefore, we suggest using this model if users prefer a classifier with a high recall. in addition, we observe that the scores are not high, especially for the information extraction task. the reason may be if the audience laughter highly depends on the conversation contexts which were not considered by baselines. therefore, taking a longer conversation context of a line into consideration is a worthy research direction. prediction errors tables 7 and 8 present examples incorrectly predicted by the bert-base. the first 3 examples describe how the model failed to predict the laughter triggers, while the last 3 examples show false positive predictions. incorporating the context information of lines may further reduce these errors. ## conclusion and future work we study the prediction of laughter triggers from comedy scripts by using text classification and information extraction methods. firstly, we introduce a corpus including high-quality and annotated sketch comedy and crosstalk scripts. secondly, we evaluate a number of baselines and find that bert models achieve the best performance. we note that the information extraction performance was very low, indicating that this task is particularly challenging. we also conduct an error analysis of incorrect predictions. the errors suggest the incorporation of rich context information may further improve the performance. therefore, it is worth investigating a model which can take such infor-model p r f acc. cnn @xcite 42.75 65.69 51.63 65.04 rcnn @xcite 42.32 71.26 52.32 62.79 bilstm @xcite 41.35 60.76 48.81 63.71 + attention @xcite 43.20 53.76 47.57 66.55 fasttext @xcite 40.86 67.43 50.70 62.62 dpcnn @xcite 41.26 66.82 50.42 62.44 transformer @xcite 41.57 70.15 51.92 63.06 bert-tiny @xcite 43.01 56.76 48.69 66.23 @xcite 44.95 55.59 49.09 67.38 bert-base @xcite 47.28 58.13 51.72 69.39 we hope this study will benefit script writing by inspiring the community to develop intelligent systems for comedy writers and other artists in the field. the corpus might also be useful for researchers who are working on related or similar tasks, such as discourse analysis and humorous response generation for dialogue systems. pengfei liu, xipeng qiu, and xuanjing huang. 2016. recurrent neural network for text classification with multi-task learning. in proceedings of the twenty-fifth international joint conference on artificial intelligence, ijcai'16, page 2873-2879. aaai press. rada mihalcea and carlo strapparava. 2006. learning to laugh (automatically): computational models for humor recognition. computational intelligence, 22(2):126-142. hiroki nakayama, takahiro kubo, junya kamura, yasufumi taniguchi, and xu liang. 2018. doccano: text annotation tool for human. software available from https://github.com/doccano/doccano . an thanh nguyen, byron wallace, junyi jessy li, ani nenkova, and matthew lease. 2017. aggregating and predicting sequence labels from crowd annotations. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pages 299-309, vancouver, canada. association for computational linguistics. reynier ortega-bueno, carlos e muniz-cuza, josé e medina pagola, and paolo rosso. 2018. uo upv: deep linguistic humor detection in spanish social media. in proceedings of the third workshop on evaluation of human language technologies for iberian languages (ibereval 2018) co-located with 34th conference of the spanish society for natural language processing (sepln 2018), pages 203-213, seville, spain. lawrence rabiner and b juang. 1986. an introduction to hidden markov models. ieee assp magazine, 3(1):4-16. jj randolph. 2005. free-marginal multirater kappa: an alternative to fleiss' fixed-marginal multirater kappa. joensuu university learning and instruction symposium. delip rao and brian mcmahan. 2019. natural language processing with pytorch: build intelligent language applications using deep learning. " o'reilly media, inc.". serdar sarıtas ¸, hasan genc ¸, s ¸erafettin okutan, ramazan i̇nci, ahmet özdemir, and gülnaz kizilkaya. 2019. the effect of comedy films on postoperative pain and anxiety in surgical oncology patients. complementary medicine research, 26(4):231-239. iulia turc, ming-wei chang, kenton lee, and kristina toutanova. 2019. well-read students learn better: the impact of student initialization on knowledge distillation. arxiv preprint arxiv:1908.08962. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, ł ukasz kaiser, and illia polosukhin. 2017. attention is all you need. in i. guyon, u. v. luxburg, s. bengio, h. wallach, r. fergus, s. vishwanathan, and r. garnett, editors, advances in neural information processing systems 30, pages 5998-6008. curran associates, inc. sandy m williams, pauline k arnold, and jennifer n mills. 2005. coping with stress: a survey of murdoch university veterinary students. journal of veterinary medical education, 32(2):201-212. diyi yang, alon lavie, chris dyer, and eduard hovy. 2015. humor recognition and humor anchor extraction. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 2367-2376, lisbon, portugal. association for computational linguistics. dongyu zhang, heting zhang, xikai liu, hongfei lin, and feng xia. 2019. telling the whole story: a manually annotated chinese dataset for the analysis of humor in jokes. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 6402-6407, hong kong, china. association for computational linguistics. renxian zhang and naishi liu. 2014. recognizing humor on twitter. in proceedings of the 23rd acm international conference on conference on information and knowledge management, cikm '14, page 889-898, new york, ny, usa. association for computing machinery. zhe zhao, hui chen, jinbin zhang, xin zhao, tao liu, wei lu, xi chen, haotang deng, qi ju, and xiaoyong du. 2019. uer: an open-source toolkit for pretraining models. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp): system demonstrations, pages 241-246, hong kong, china. association for computational linguistics. peng zhou, wei shi, jun tian, zhenyu qi, bingchen li, hongwei hao, and bo xu. 2016. attention-based bidirectional long short-term memory networks for relation classification. in proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: short papers), pages 207-212, berlin, germany. association for computational linguistics.