Rigid Formats Controlled Text Generation (2020)

Abstract: Neural text generation has made tremendous progress in various tasks. One common characteristic of most of the tasks is that the texts are not restricted to some rigid formats when generating. However, we may confront some special text paradigms such as Lyrics (assume the music score is given), Sonnet, SongCi (classical Chinese poetry of the Song dynasty), etc. The typical characteristics of these texts are in three folds: (1) They must comply fully with the rigid predefined formats. (2) They must obey some rhyming schemes. (3) Although they are restricted to some formats, the sentence integrity must be guaranteed. To the best of our knowledge, text generation based on the predefined rigid formats has not been well investigated. Therefore, we propose a simple and elegant framework named SongNet to tackle this problem. The backbone of the framework is a Transformer-based auto-regressive language model. Sets of symbols are tailor-designed to improve the modeling performance especially on format, rhyme, and sentence integrity. We improve the attention mechanism to impel the model to capture some future information on the format. A pre-training and fine-tuning framework is designed to further improve the generation quality. Extensive experiments conducted on two collected corpora demonstrate that our proposed framework generates significantly better results in terms of both automatic metrics and the human evaluation.

## introduction recent years have seen the tremendous progress in the area of natural language generation especially benefiting by the neural network models such as recurrent neural networks (rnn) or convolutional neural networks (cnn) based sequence-tosequence (seq2seq) frameworks (bahdanau et al., 1 code: http://github.com/lipiji/songnet let me not to the marriage of true minds admit impediments, love is not love which alters when it alteration finds or bends with the remover to remove. ## conclusion we propose to tackle a challenging task called rigid formats controlled text generation. a pre-training and fine-tuning framework songnet is designed to address the problem. sets of symbols are tailordesigned to improve the modeling performance for format, rhyme, and sentence integrity. extensive experiments conducted on two collected corpora demonstrate that our framework generates significantly better results in terms of both automatic metrics and human evaluations given arbitrary cold start formats.