INSET : Sentence Infilling with IN ter- SE ntential Transformer (2020)

Abstract: Missing sentence generation (or sentence in-filling) fosters a wide range of applications in natural language generation, such as document auto-completion and meeting note expansion. This task asks the model to generate intermediate missing sentences that can syntactically and semantically bridge the surrounding context. Solving the sentence infilling task requires techniques in natural language processing ranging from understanding to discourse-level planning to generation. In this paper, we propose a framework to decouple the challenge and address these three aspects respectively, leveraging the power of existing large-scale pre-trained models such as BERT and GPT-2. We empirically demonstrate the effectiveness of our model in learning a sentence representation for generation and further generating a missing sentence that fits the context.

## introduction generating a span of missing tokens in a text chunk, known as "text infilling," has attracted many attentions recently @xcite @xcite @xcite . here we study the related but somewhat different task of "sentence infilling." specifically, as illustrated in figure 1 , intermediate sentences (or chunks of text) are removed from long-form text (e.g., paragraphs, documents), and the task is to generate the missing pieces that can smoothly blend into and fit the context both syntactically and semantically. the generation can be either based only on context, or based on both context and side information such as constraint keywords. compared with text infilling, sentence infilling requires the model to handle inter-sentential correlation and to reason about missing semantic information. developing models for sentence infilling can potentially * these authors contributed equally to this work. she was extremely happy with our hotel and we had a complimentary buffet. beautiful beachside boutique hotel with great views and modern decoration. my favorite part about this hotel is definitely the restaurant, uva. i recently visited uva to attend a friend's birthday party. ... facilitate many text generation applications. possible scenarios include, but are not limited to: document auto-completion by detecting and suggesting missing bridging sentences in the surrounding context; collaborative document writing by modifying and unifying different writing styles from multiple authors; meeting note expansion by extending a set of keywords (lexical constraints) to a full sentence, leveraging the surrounding context. there are many challenges associated with this long-form sentence infilling task, which is typically a one-to-many problem in that the possible outputs can be diverse. as the generated sentence should connect separate text pieces in a syntactically and semantically smooth and coherent manner, the task requires a wide range of understanding, planning, and generation techniques. largescale pre-trained language models such as bert @xcite and gpt-2 @xcite have dramatically enhanced the understanding and generation modules. however, how to in-tegrate them in a holistic manner, and to analyze and establish the long-range dependence structure by high-level semantic planning is still challenging and yet to explore, as semantic appropriateness is usually subtler than syntactic appropriateness, which can be well characterized by autoregressive language models. several works have been done in this direction. mass @xcite obtains sentence representations by predicting a span of missing tokens. it can be used to generate missing text, but the missing span length needs to be pre-specified. other related works @xcite also require knowledge of the span length as an input to their models, and thus are different from our work. text infilling @xcite sequentially generates tokens for the missing part of a sentence until an end-of-blank token is generated. its generation can be of arbitrary length. by design, all these previous approaches operate at the token level, and thus arguably focus more on lexical appropriateness than the global semantics. in this paper, we propose inter-sentential transformer (inset), a novel approach to sentence infilling. our model first produces sentence-level semantic features that capsulate the missing highlevel information. then, grounded on the predicted semantic features, the model generates the syntactic and lexical features to embody the predicted sentence. specifically, understanding, planning, and generation are handled by three modules in a synergistic manner: ## conclusions and outlook we study the task of sentence infilling, which is analogous to the masked language modeling task for (pre-)training bert, but now it is at the sen-example from the tripadvisor dataset example from the tripadvisor dataset example from the recipe dataset preceding context it was such a pleasure to see somthing new every night. it was not very crowded so we were able to get great seats at either the pool or the beach. the vip sevice was great for dinner reservations and pillow service. the walls are very thin. since this is a family vacation type of hotel, people are up at the pool/bbq area/hallways during all hours of the night. do not stay here if you need a quite night of sleep. after another 15 minutes or so the mixture should thicken up. the mixture will continue to thicken as it cools. sterilize your jars and lids and while still hot fill with the jam leaving about a 1/2 inch headspace. place lids onto the jars and boil in a water bath with jars covered by 3 inches of water for 10 minutes. also, the elevator situation is weird. remove from the heat and stir in your amaretto.