A- TIP : Attribute-aware Text Infilling via Pre-trained Language Model (2022)

Abstract: Text infilling aims to restore incomplete texts by filling in blanks, which has attracted more attention recently because of its wide application in ancient text restoration and text rewriting. However, attribute- aware text infilling is yet to be explored, and existing methods seldom focus on the infilling length of each blank or the number/location of blanks. In this paper, we propose an Attribute-aware Text Infilling method via a Pre-trained language model (A-TIP), which contains a text infilling component and a plug- and-play discriminator. Specifically, we first design a unified text infilling component with modified attention mechanisms and intra- and inter-blank positional encoding to better perceive the number of blanks and the infilling length for each blank. Then, we propose a plug-and-play discriminator to guide generation towards the direction of improving attribute relevance without decreasing text fluency. Finally, automatic and human evaluations on three open-source datasets indicate that A-TIP achieves state-of- the-art performance compared with all baselines.

## introduction originating from cloze tests @xcite ), text infilling aims to fill in missing blanks in a sentence or paragraph by making use of the preceding and subsequent texts. for example, given two infilling tasks e1 and e2 in fig. 1 , text infilling models are supposed to provide fine-grained control over the location of any number of blanks and infill a variable number of missing tokens for each blank. text infilling has been gaining increasing attention in a number of prevailing research fields, including ancient text restoration @xcite ), text editing and rewriting @xcite , and conversation generation @xcite . however, current text infilling methods are based only on bidirectional semantic constraints @xcite , and other abundant attribute-based constraints, e.g., sentiment and topics, remain to be studied. in reality, infilling attribute-aware content can better satisfy human needs and introduce more diversity. for instance, as shown in fig. 1 , a-tip can fill in blanks under the guidance of an attribute to satisfy sentiment or expert knowledge infilling, while current text infilling models mainly focus on fluency, which leads to meaningless and monotonous infilling contents @xcite . designing a simple but efficient attribute-aware text infilling model is a challenging task. first, to achieve attribute awareness, simply modifying a text infilling model architecture or finetuning with attribute-specific data will destroy the model's ability to infill blanks or require a significant cost for re-training @xcite . second, if the model infills blanks towards the direction of improving text attributes, avoiding ill-formedness between infilling content and its bidirectional context becomes a challenge. for instance, "the movie interesting and perfect us" with _ as blanks. finally, current methods lack fine-grained control over automatic determination of the number/location of blanks or the infilling length for each blank. for example, markov assumption-based models @xcite hardly adapt to variable infilling lengths, while masked language model (mlm)based methods @xcite are incapable of generating more than one word per blank, and generative lm-based methods @xcite cannot guarantee the output will match the number of missing blanks in the input. to circumvent the above dilemma, in this paper, we propose an attribute-aware text infilling method based on a pre-trained lm (a-tip), in which a plug-and-play discriminator provides finegrained control over bidirectional well-formed fluency and attribute relevance. 1 specifically, 1) we first propose a general text filling framework that fine-tunes a standard lm with many artificiallymasked examples in an auto-regressive manner. moreover, to ensure that the number of infilling contents equals the number of blanks, we design a new attention mechanism, where unmasked tokens can attend to each other but masked tokens can attend only to the preceding context (fig. 2 (a) ). we also adopt two-level positional encoding to combine inter-and intra-blank positional information to automatically learn the length of blanks. 2) to achieve attribute-aware generation without modifying lm's architecture or re-training, we propose a plug-and-play discriminator that shifts the output distribution of the text infilling model towards the semantic space of given guide attributes. we also design two additional strategies to ensure the infilling content is well-formed with its bidirectional context without decreasing attribute relevance. the main contributions are summarized as follows: ## related work in this section, we briefly review the most relevant studies to our work on pre-trained lms, text infilling, and constrained text generation. ## preliminaries to clarify our method, we first introduce some essential background knowledge and then define the task of attribute-aware text infilling. language models reveal the degree of how much a sentence (a sequence of words) is likely to be a realistic sequence of a human language. formally, let w be the vocabulary set and w 1:@xmath0, . . . , w n } is a sentence with n words, where w @xmath1. an lm measures the joint probability by decomposing the sequence one by one: task definition: attribute-aware text infilling is to take incomplete text w, containing one or more missing blanks, and return completed text w under the constraints of c. as in fig. 1 , several attributes are listed in c. specifically, let [mask] be a placeholder for a contiguous sequence of one or more missing tokens. then, w is a sequence of tokens in which a number of them are @xcite . to map w to w, constrained with attribute c, an infilling strategy must specify both how many and which tokens to generate for each @xcite . note that there may be many logical w for a given w. hence, we are interested in learning a distribution p(w| w, c). specifically, in accordance with bayes' theorem, we formulate the probability of predicting the token w i for its corresponding [mask] as: ## methodology the overall framework of a-tip is shown in fig. 2 . a-tip contains two components: a text infilling model and a plug-and-play attribute-aware controller. ## experimentation as shown in ## conclusion in this paper, we presented a simple strategy for text infilling a-tip that leverages an lm by proposing new attention mechanisms and two-level positional encoding to effectively improve the quality of generation in limited data settings. furthermore, our plug-and-play discriminator can guide the generation towards the direction of improving text attribute relevance. in future work, we plan to incorporate the plug-and-play discriminator into more systems that assist humans in the writing process, where we hope that our work encourages more investigation of text infilling.