Lingxi: A Diversity-aware C hinese Modern Poetry Generation System (2023)

Abstract: Chinese modern poetry generation has been a challenging task. One issue is the Chinese word segmentation (CWS) which is critical to comprehend the Chinese language but was not always considered in common tokenization methods. Another is the decoding (sampling) method which may induce repetition and boredom and severely lower the diversity of the generated poetry. To address these issues, we present Lingxi, a diversity-aware Chinese modern poetry generation system. For the CWS issue, we propose a novel framework that incorporates CWS in the tokenization process. The proposed method can achieve a high vocabulary coverage rate with a reasonable vocabulary size. For the decoding method and the diversity issue, we propose a novel sampling algorithm that flattens the high likelihood part of the predicted distribution of the language model to emphasize the comparatively low-likelihood words and increase the diversity of generated poetry. Empirical results show that even when the top 60% of cumulative probability mass of the predicted distribution is flattened, our method achieves comparable or even better performance than baseline sampling methods. Our system is available at http://lingxi.website .

## introduction chinese modern poetry generation has been a challenging natural language processing (nlp) task. one issue is the chinese word segmentation (cws) problem. unlike english words that are naturally delimited by white spaces, chinese words do not have explicit word delimiters. since different cws strategies may completely change the semantics of chinese words, cws is always crucial for humans to comprehend the chinese language and is regarded as a critical chinese nlp task. despite its *corresponding author. another challenge is the high diversity requirement for poetry generation. the task is unique compared with common neural generation tasks, which try to predict the correct answers from the training corpus and focus on the fluency of the generated texts. concretely, a piece of easily understood and highly fluent poetry with high-likelihood words might not always be considered poetic, while semantically ambiguous poetry with low-likelihood words might be diversified and creative. we consider the diversity issue from the view of the decoding (sampling) algorithm of the language model. the widely acknowledged golden methods include the nucleus sampling @xcite , top-k sampling @xcite , and temperature sampling, which have been proved to be able to control the quality and diversity of generated texts. however, in our system, these golden methods might occasionally generate boring and repetitive samples with low diversity, severely hurting the quality of the generated poetry. thus it inspires us to explore novel decoding algorithms to address the diversity issue for poetry generation. to address these challenges, we present lingxi, a diversity-aware chinese modern poetry generation system with the following features. for the cws issue, we propose a novel framework that incorporates cws in the tokenization process. the proposed method not only leverages the human knowledge from the cws model but also com-bines the advantage of frequency-based tokenization methods, which can achieve a high coverage rate of the vocabulary while maintaining a reasonable vocabulary size. for the high-diversity issue, we propose a novel decoding (sampling) method, namely the nucleus sampling with flattened head (ns-fh) algorithm. it flattens the high-likelihood part of the predicted distribution to suppress the high-likelihood words and emphasize the less likely words to improve the diversity of the generated poetry. surprisingly, we find that even if the top 60% of cumulative probability mass of the vocabulary's distribution is flattened, the model achieves comparable or even better performance than baseline methods. following the pre-training/fine-tuning paradigm, we collect about 3,500 published books of chinese novels as the pre-training corpus, aiming at chinese literary language modeling. then we collect about 220,000 passages of chinese modern poetry and lyrics as the fine-tuning corpus. to build the vocabulary, most researchers directly apply frequency-based tokenization methods such as byte pair encoding (bpe, @xcite or unigram language model (unigram lm, @xcite without considering the chinese word segmentation (cws) issue, which is a critical chinese nlp task and can be modeled by supervised learning @xcite @xcite . despite these advances, it is still difficult to directly deploy cws into the tokenization process due to the "long-tail" issue. figure 1 illustrates the cumulative coverage rate of the vocabulary (sorted by the word frequency) of our corpus produced by the cws software thulac @xcite . it generates a large vocabulary (containing about 3.6 million words) with a very long low-frequency "tail", which grows slowly to the coverage rate of the vocabulary and takes a nonnegligible portion of coverage. in our case, the top 20,000 words take about 90% coverage on the corpus, and the remaining 3.6 million minus 20,000 words take the remaining 10% coverage, which satisfies zipf's law @xcite . truncating the "tail" would result in a low coverage rate and too many "unknown" tokens. ## the diversity-aware sampling we train an auto-regressive transformer language model on the two collected corpus for generation (see the appendices for the model details). as is widely acknowledged, the decoding module plays a crucial role in neural text generation. stochastic sampling methods such as nucleus sampling @xcite , top-k sampling @xcite and temperature sam-pling can generate texts with higher diversity than traditional decoding methods such as beam search. recent advances such as mirostat @xcite and typical sampling @xcite use adaptive filtering on top of these methods. however, in our model, we observe that traditional algorithms occasionally generate boring and repetitive poetry with low diversity, thus severely hurting the quality of generated poetry. the high-likelihood part (top q) of the distribution is flattened. the proposed algorithm has a close relationship with nucleus sampling. their truncation mechanisms on the predicted distribution are identically based on the threshold {p, q} of the cumulative probability density function. and our method features in using an additional threshold q to determine another smaller portion of the predicted distribution and flattening their probability density function. and since q is also a threshold for the cumulative probability density function, the flattening manipulation will not affect low-entropy distributions that only contain an "unquestionably correct" word which takes most of the probability mass, thus not hurting the fluency of the generated poetry. we later show in the empirical results that the proposed method can closely follow the behavior of nucleus sampling when using a small value of q while exhibiting higher diversity when setting a large value of q, achieving diversity-aware sampling. ## conclusion we present lingxi, a diversity-aware chinese modern poetry generation system. to address the cws issue, we propose a novel framework that combines cws with the frequency-based method, which can create a vocabulary with a suitable size and high coverage. to increase the diversity of generated poetry, we propose nucleus sampling with flattened head (ns-fh) algorithm which achieves controllable diversity with on-par or better performance compared to traditional sampling methods. the proposed sampling algorithm provides a new approach to increase the diversity of neural text generation via the decoding module, which might be beneficial for artistic generation cases that have high requirements for the diversity or novelty. parameters value number of transformer layers 24 number of transformer attention heads 16 embedding size 1,024 vocabulary size 17,589 maximum context length 128 number of network parameters 330 million pre-training epochs 20 fine-tuning epochs 10 batch size per gpu 32 number of training gpus 8 nvidia r geforce r rtx 2080 ti pre-training learning rate 2 × 10 -4 fine-tuning learning rate {1 × 10 -5 , 2 × 10 -5 } learning rate decay linear decay warm-up steps 1% of total steps optimizer adam optimizer (kingma and ba, 2014) weight decay 0.01 ppl on validation set after pre-training 17.58 best fine-tuning epoch epoch 4, learning rate being 2 × 10 -5 best ppl on fine-tuning validation set 16.75 full sampling hyperparameter space for the trade-off curves from figure 10 to figure 14 .