Creative Natural Language Generation (2023)

Abstract: Large language models such as GPT-3, GPT4, Claude etc., have advanced the state of the art in several natural language generation tasks such as text summarization and machine translation. However when it comes to open-ended tasks with a focus on creativity such as generating stories, poetry, or various forms of figurative language, these state-of-the-art language models are often found to be inadequate. This tutorial aims to bring awareness of the important and emerging research area of open-domain creative generation, with a focus on language generation while also touching on multi-modal generation (e.g., image captioning, visual metaphors). It targets natural language processing (NLP) and artificial intelligence (AI) researchers as well as creative writing practitioners who are interested in building systems that are capable of emulating as well as augmenting human creativity. In particular, we will review recent studies on creative language generation both at the sentence level as well as longer forms of text. We will provide the audiences with a holistic view of 1) the importance and challenges of building creative language generation systems; 2) how we incorporate content planning, domain knowledge and creativity specific heuristics for different forms of creative language generation such as story, poetry, humor, metaphors etc 3) how can we build better evaluation methods for creative text generation? In particular, how could the recent advancement of AI shape the future workforce for creativity? We will conclude the tutorial by outlining future research directions in this area.

## introduction large language models such as gpt-3 @xcite , bart, @xcite etc., have advanced the state of the art in several natural nanguage generation tasks such as text summarization @xcite and machine translation @xcite . however when it comes to open-ended tasks with a focus on creativity such as generating stories @xcite , poetry @xcite , or various forms of figurative language @xcite , these state-of-the-art language models are often found to be inadequate. the principal reason for this is that, in addition to composing grammatical and fluent sentences to articulate the intended content, these tasks usually also require extensive world and commonsense knowledge, as well as discourse modeling, to make sure the outputs maintain long-term coherence while remaining creative. it should also be noted that current approaches to text generation for specialized tasks require lots of training data for supervision. however, most existing corpora for creative forms of text are limited in size. even if such a corpus existed for creative tasks, learning the distribution of existing data and sampling from it will unlikely lead to truly novel, creative output. creative composition requires deviating from the norm, whereas standard generation approaches seek to mimic the norm. this tutorial aims to bring awareness of the important and emerging research area of open-domain creative generation, with a focus on language generation while also touching on multi-modal generation (e.g., image captioning, visual metaphors, and visual story generation). it targets natural language processing (nlp) and artificial intelligence (ai) researchers as well as creative writing practitioners who are interested in building systems that are capable of emulating as well as augmenting human creativity. in particular, we will review recent studies on creative language generation both at the sentence level as well as longer forms of text. we will provide the audiences with a holistic view of 1) the importance and challenges of building creative language generation systems; 2) methods for different forms of creative language generation such as story @xcite , poetry @xcite , humor @xcite , metaphors @xcite @xcite , sarcasm @xcite , and hyperbole @xcite 3) how can models for creativity infer user intention and preferences, allow for fine-grained control, and take (natural language) feedback? in particular, how could the recent advancement of ai shape the future workforce for creativity? we will conclude the tutorial by outlining future research directions in this area. ## tutorial outline in this tutorial, we will review the history of creative language generation both in shorter and longer forms. then, we will move to the recent advances in creative language generation that employ transformer-based language models as well as external world knowledge from existing resources. we will also touch upon how much creativity can we elicit from larger models like gpt3 @xcite ) and where they are still lacking. finally, we will discuss the real-world implications of creative language generation and how humans can interact or collaborate with these models to satisfy their specific needs. in particular, we will present recent community efforts in the following topics: ## history due to the lack of vast research on creative language generation and its importance in training and testing generative models, it is necessary to have a cutting-edge tutorial on an emerging and timely topic. we are unaware of any tutorials on the exact same topic in the past 4 years' acl/emnlp/eacl/naacl conferences, with the only exception of the acl 2020 tutorial @xcite on stylized text generation: approaches and applications. the tutorial was mainly about style transfer. while there are some overlaps between style transfer and creative language generation, we believe our tutorial will benefit the audiences in terms of learning the vast landscape of creative language generation in the age of pre-trained language models. finally, our tutorial will also touch upon human-ai collaboration for creativity as well as creativity for vision and language tasks which has not been touched upon in prior tutorials. ## prerequisite knowledge our target audience is general nlp conference attendances; therefore, no specific knowledge is assumed of the audience except basic machine learning and nlp background: ## tutorial content this tutorial presents a systematic overview of the history and the frontier of creative language generation. we will also introduce methods for sentence level and longer forms of creative language generation, and careful consideration in designing the evaluation of model outputs as well as how llm's can aid in providing assistance during the process of creative writing.we will then do deep dives. the detailed contents are outlined below. ## tutorial coverage and suggested reading list while the tutorial will include our own work @xcite @xcite @xcite @xcite @xcite @xcite @xcite , we anticipate that roughly 40% of the tutorial content will be pulled from work by other researchers in nlp and machine learning communities include but not limited to @xcite @xcite @xcite @xcite @xcite . a more comprehensive list of related papers will be provided before the tutorial. ## tutorial instructors our instructors consist of experts who have conducted research in different aspects related to the tutorial topic. nanyun (violet) peng nanyun (violet) peng is an assistant professor in the department of computer science at the university of california los angeles. she received her ph.d. in computer science from johns hopkins university. her research focuses on the generalizability of nlp technologies, with applications to creative language generation, low-resource information extraction, and zero-shot cross-lingual transfer. her works have won the outstanding paper award at naacl 2022, the best paper award at aaai 2022 deep learning on graphs workshop, and have been featured an ijcai 2022 early career spotlight. she has given a tutorial at naacl 2018 on information extraction. tuhin chakrabarty tuhin chakrabarty is a ph.d. candidate in computer science at columbia university and a part of the natural language processing group, where he is advised by smaranda muresan. his research is supported by the columbia center of artificial intelligence & technology (cait) and amazon science ph.d. fellow-ship. he was also a fellow at the new york times r&d team working on natural language generation. his overarching research question centers around how we can use large language models for creativity. he has published several papers in various nlp conferences and journals including acl, naacl, tacl and emnlp. he he he he is an assistant professor of computer science and the center for data science at new york university. she is affiliated with the cilvr lab, the machine learning for language group, and the alignment research group. her research focuses on building intelligent systems that can communicate with humans effectively and enable individuals to achieve their goals. today's systems are often opaque, brittle, and difficult to control, which limits their usefulness in humancentered applications. to make them our trustworthy collaborators, her research aims to (i) understand the computational foundation of generalization in novel scenarios, and (ii) build interactive systems that align with users' goals. she has given a tutorial at emnlp 2021 on robustness and adversarial examples in nlp. vishakh padmakumar vishakh padmakumar is a ph.d. student in data science at new york university advised by he he. his research is broadly in the field of natural language processing and human-ai collaboration with a focus on collaborative text generation for creative writing tasks and other interactive settings. prior to this, he was a graduate research associate at the nyu center for social media and politics working on political stance classification and multimodel content sharing in online disinformation campaigns. he has published papers at several nlp and machine learning venues including acl, emnlp, and icml and was the chair of the acl 2023 student research workshop.