Can Language Models Make Fun? A Case Study in C hinese Comical Crosstalk (2023)

Abstract: Language is the principal tool for human communication, in which humor is one of the most attractive parts. Producing natural language like humans using computers, a.k.a, Natural Language Generation (NLG), has been widely used for dialogue systems, chatbots, machine translation, as well as computer-aid creation e.g., idea generations, scriptwriting. However, the humor aspect of natural language is relatively under-investigated, especially in the age of pre-trained language models. In this work, we aim to preliminarily test *whether NLG can generate humor as humans do*. We build a largest dataset consisting of numerous **C**hinese **C**omical **C**rosstalk scripts (called **C**3 in short), which is for a popular Chinese performing art called ‘Xiangsheng’ or ‘相声’ since 1800s.We benchmark various generation approaches including training-from-scratch Seq2seq, fine-tuned middle-scale PLMs, and large-scale PLMs (with and without fine-tuning). Moreover, we also conduct a human assessment, showing that 1) *large-scale pretraining largely improves crosstalk generation quality*; and 2) *even the scripts generated from the best PLM is far from what we expect*. We conclude humor generation could be largely improved using large-scaled PLMs, but it is still in its infancy. The data and benchmarking code are publicly available in [ https://github.com/anonNo2/crosstalk-generation ]( https://github.com/anonNo2/crosstalk-generation ).

## introduction artificial intelligence (ai) has been widely used in natural language processing (nlp), computer vision, speech, robots, and further applied biology, etc. in nlp, pre-trained language models (plms) e.g., bert @xcite and gpt @xcite , have notably improved many natural language tasks including text classification, question answering, and nlg. although its technical contribution to the human community has been widely explored, the social or cultural effect is somehow under-investigated. to explore the side social or cultural effect of plms, in this paper, we leverage the generation ability of pre-trained language models to save endangered cultural heritage, i.e., chinese comical crosstalk. we believe the diversity of generations from pre-trained language models could enrich the chinese comical crosstalk, especially leveraging modern topics. from a broader view, we aim to test to which degree ai could make fun in the context of plms (especially large-scale gpt). humor has been rooted in the chinese language, originating from the book 'records of the grand historian' written by a chinese historian qian sima 2000 years ago foot_0 which includes a chapter titled 'biography of humor' 《滑稽列传》. since then, humor is an inseparable ingredient of the chinese language. as the first step, this work aims to explore traditional performing art in chinese comedy crosstalk, which has a very long history originating from the north of china since roughly 1800, see tab. 1 for an example crosstalk script, . it began as a form of street performance, incorporating joketelling, comedic banter, imitations, or borrowing from other performance arts, such as peking opera, all with the express purpose of making audiences laugh. the characteristics of crosstalk scripts are 1) multiple-turn; 2) humor-oriented; 3) with a novel language style; 4) culturally-grounded; and 5) lowresourced, see more details in sec. 2. humor generation is a challenging task since, for instance, we may not know exactly what makes a joke funny. solving this problem with purely neural network models requires deep semantic understanding @xcite . this even becomes more challenging if cultural and other contextual cues are considered as in chinese comical crosstalk. from a practical point of view, the data preparation usually goes earlier than the development of algorithms and models. since new models cannot be well-evaluated before (especially large-scale) data is ready 3 . as the first step, we collect many crosstalk scripts from the internet. the dataset is publicly available with an open-resource license (apache 2.0). we also conduct several basic generation approaches including train-from-scratch seq2seq generation @xcite , fine-tuned middlescale plms, and large-scale plms (with and without fine-tuning). furthermore, the current research community also explored the potential to use largescale plms for creation. for example, @xcite claims that gpt-3 can generate synthetic news articles that human evaluators have difficulty distinguishing from human-generated articles. we do not expect that gpt has a 'sense of humor'. alternatively, we test to which degree gpt-3 is creative in crosstalk generation. the contributions of this paper are as follows: 1) firstly, culturally, we release the largest-ever publicly-available chinese crosstalk dataset, contributing to both the nlp research community and the traditional chinese culture community. this is the first work to generate an entire dialogue crosstalk script (instead of utterance pair) thanks to pre-trained language models. this would inspire 3 one can see a concrete example in computer vision that imagenet dataset @xcite largely promotes the development of image classification models @xcite , and concrete examples in nlp are glue @xcite and squad @xcite benchmarks that benefit natural language understanding @xcite . ## human assessment setting we randomly select 50 scripts in the test set. we take the first ten utterances as the input for all models. these models will generate the next ten utterances, utterance by utterance or character by character. for each script, we show participants with 20 utterances (including the raw 10 utterances and the generated 10 utterances), see the web ui in app. g. participants are required to 1) rate fivepoint scores for the general quality and humor degree of each generated script ('5' for the best and '1' for the worst); and 2) rate binary scores for coherence and an ethically-risky flag of each generated example ('1' for true and '0' for false). we ask no-paid volunteers to participate to rate these generated results from 10 models. 15 participants have completed all ratings and other incomplete ratings from other 11 participants are discarded. the score is calculated as an average score among all dialogues and all participants for each model. results human assessment is shown in tab. 7. raw scripts achieve the best general quality, probably evidencing that humans are much better than sota models in terms of being creative and humorous. among these models, gpt-3 and its fine-tuned version (gpt3-davinci-finetuned) outperform others in terms of general quality. interestingly, fine-tuned gpt-3 outperforms zero-shot gpt-3 although the former has poorer performance in automatic evaluation (see tab. 5 ). similar to the automatic evaluations in tab. 5, zero-shot large-scale chinese plms (the third group) underperform these fine-tuned middlescaled plms (like unilm, t5, and gpt). seq2seq performs the worst; this may be due to seq2seq does not utilize pre-training. interestingly, cpmlarge produces much more insulting words than others; the reason needs to be further investigated. ## discussions as seen from the automatic evaluation in tab. 5 and human assessment in tab. 7 , the adoption of large-scale pre-trained language models could largely improve the quality of crosstalk generation, compared to these models without large-scaled pretraining. we show some generated examples from large-scale pre-trained language models with and without fine-tuning. although large-scale pre-trained language models largely improve crosstalk generation. based on the human assessment, we could preliminarily conclude that the best generation approach (finetuned gpt-3) achieves fairly good crosstalk (2.27 vs. 3.52 for general quality), while it is far away from what we expect. the reasons are twofold. first, the evaluation criterion for humor generation is problematic. observing the inconsistency between tab. 5 and tab. 7, a better performance evaluated using bleu and rouge does not lead to a better performance in human assessment, this probably suggests that bleu or related metrics for generation is not inappropriate for humor generation since humor itself is diverse and subjective that does not have textual ground truth. one could see the correlations between human and automatic evaluation in app e which is relatively high but somehow overestimated. moreover, human assessment is expensive and cannot give realtime feedback during model training. secondly, current methods did not consider prime ingredients of humor. core ingredients of humor include incongruity, surprise, cultural empathy, and interpersonal effect, without which simply training on data is a soft way to memorize the training data and it can't generate real humor. ## conclusion and future work in this paper, we collect a dataset for chinese crosstalk. based on the dataset, we evaluate several existing generation models including lstm seq2seq, gpt, unilm, cpm, pangu-α, zhouwenwang, and gpt-3 for crosstalk generation. this is a preliminary step for humor generation, indicating that large-scale pretraining largely improves crosstalk generation quality while there still exists a big gap between the generated scripts and human-created scripts. note that there are some concerns about bias/stereotypes for crosstalk, e.g., educational background discrimination and gender bias. in future work, we are interested in collecting crosstalk audios to promote the end2end crosstalk generation with an adapted humorous accent. xingxing zhang and mirella lapata. chinese poetry generation with recurrent neural networks. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pp. 670-680, 2014. zhengyan zhang, xu han, hao zhou, pei ke, yuxian gu, deming ye, yujia qin, yusheng su, haozhe ji, jian guan, et al. cpm: a large-scale generative chinese pre-trained language model. ai open, 2:93-99, 2021.