The Amazing World of Neural Language Generation (2020)

Abstract: Neural Language Generation (NLG) – using neural network models to generate coherent text – is among the most promising methods for automated text creation. Recent years have seen a paradigm shift in neural text generation, caused by the advances in deep contextual language modeling (e.g., LSTMs, GPT, GPT2) and transfer learning (e.g., ELMo, BERT). While these tools have dramatically improved the state of NLG, particularly for low resources tasks, state-of-the-art NLG models still face many challenges: a lack of diversity in generated text, commonsense violations in depicted situations, difficulties in making use of factual information, and difficulties in designing reliable evaluation metrics. In this tutorial, we will present an overview of the current state-of-the-art in neural network architectures, and how they shaped recent research directions in text generation. We will discuss how and why these models succeed/fail at generating coherent text, and provide insights on several applications.

## introduction natural language generation (nlg) forms the basis of many natural language processing (nlp) tasks such as document summarization, machine translation, image captioning, conversational dialogue, and creative writing, making it an essential component in human-machine communication tasks. with recent progress in training deep neural networks, there has been a paradigm shift from template based approaches to neural methods as the predominant building blocks for text generation systems. specifically, the rich representation learning capabilities of neural networks have allowed nlg models to be trained directly from large amounts of training data, significantly reducing the need for manual feature engineering. many benefits have emerged from this new research direction. first, the prototypical framework for training neural networks in an end-to-end fashion has allowed for a diverse array of contextual information to be incorporable into text generation systems @xcite @xcite , allowing for a richer range of stylistic variability in generated text. simultaneously, the combination of deep neural networks, large-scale text data and cheap computational power has accelerated new developments in neural network language models. however, nlg models still raise many challenges which are the focus of a growing body of work. examples of such limitations are the lack of diversity in generated texts, difficulty in controlling the discourse coherence of the generated text, the lack of commonsense in generated outputs, an uncertain reliance on provided factual information, and more general open questions on architecture design and optimization settings. in this tutorial, we will start with an introduction to neural language generation, presenting neural language models and encoder-decoder models. we will then discuss the capabilities and limitations of recent text generation models, the suitable architectures for text generation in various specific applications, and then provide insights into why and how these generation models can be adapted for a particular task @xcite @xcite . the discussion on evaluation metrics will start from ngram matching up to the recent progress on text generation evaluation metrics. in the end, this tutorial will be concluded by presenting and discussing major current research directions in the field of neural language generation. all materials (including slides, code, and demos) will be publicly available online on the day of the tutorial. we do not assume any particular prior knowl-edge in text generation or language modeling. familiarity with standard neural network modules (lstm/cnn/transformer) is a plus but not required. the intended length of the tutorial is 3 hours, including a coffee break. ## diversity and inclusion diversity. the background of the instructors of this tutorial is evenly distributed among academia and industry. the instructors consist of a group of researchers ranging from an assistant professor at university of virginia (yangfeng ji), a senior ph.d. student at university of washington with years of industry research experience (antoine bosselut) and two senior research scientists in industry (thomas wolf and asli celikyilmaz), who both have years of industry research experience. the tutorial instructors are also from different countries and continents (the netherlands and usa). ## additional details audience size. based on the increasing interest in natural language generation (larger growth rate in submissions compared to other areas of nlp foot_0 ), we anticipate that between 150 and 200 attendees will be interested in this tutorial. special requirements. the tutorial will require internet access for participants to be able to access the slides and, optionally, to access hands-on coding notebooks. preferred venues. our preferred venues are emnlp 2020, acl 2020, and coling 2020. open access. we agree to allow the publication of our slides and a video recording of our tutorial in the acl anthology. all our materials will additionally be posted on our tutorial website.