Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT 2 (2021)

Abstract: The semantics of a text is manifested not only by what is read but also by what is not read. In this article, we will study how those implicit “not read” information such as end-of-paragraph () and end-of-sequence () affect the quality of text generation. Specifically, we find that the pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. Experimental results on English story generation show that can lead to higher BLEU scores and lower perplexity. We also conduct experiments on a self-collected Chinese essay dataset with Chinese-GPT2, a character level LM without and during pre-training. Experimental results show that the Chinese GPT2 can generate better essay endings with .

## introduction large-pretrained neural models such as gpt @xcite and bert @xcite have achieved the state-of-the-art on many nlp tasks. among these models, openai's gpt2 @xcite , for example, has shown to be capable of generating long fluent text in many areas, such as stories @xcite , recipes (h. @xcite , patent claims @xcite , and news @xcite . however, the semantics of a text goes beyond what's written to what's not written: when to break paragraphs and when to end. we wish to experiment on this issue: how much do eop and eos markers affect our ability to generate texts with gpt2. to study the strength of gpt2 as a language generator, @xcite conduct experiments in the context of story generation with the writing-prompts @xcite dataset. they find that the generated results of gpt2 have higher-quality content (using more rare words, concrete words, and named entities) by comparing the top 150 generated words. however, the average story length of the dataset is 12 paragraphs, 368 words. in such lengthy human writings, the overall layout and text endings are also important, but whether the gpt2 can generate them properly is unclear, and how to generate better endings has not been investigated. in this work, we find the generated endings are not only affected by eos, but also eop. eop can also help improve the topic relevance of the generated text. we first conduct essay completion experiments with chinese gpt2 @xcite , which is a character-level lm without eos or eop during pre-training. experimental results show that fine-tuning with eop can achieve higher ending quality score and topic relevance score in human evaluation. we further conduct story generation experiments on dataset writingprompts with english gpt2-117, which holds the line break "\n" (nl) in the vocabulary. thus, the nl can be treated as the end-of-paragraph during fine-tuning @xcite . experimental results show that learning to end the paragraph can benefit the word/token perplexity, blue score, eos perplexity, and human evaluated ending quality score. our contributions are as follows: we show that not only the well-known eos but also the eop, is part of the semantics of a text, and training with this information improves the text generation itself. the paragraph information not only can help improve the effectiveness of the generation model but also help to generate the end of the text. we also investigate different approaches to incorporating paragraph information into the lm generator. our findings indicate that sep/eop and eos should be introduced to gpt2 types of models during pretraining, to generate better text in length. our target task is to conduct auto-regressive language modeling over writingprompts and the chineseessay dataset. the basic assumption of auto-regressive generation is that the probability of a word sequence equals the product of conditional word probability: p (w 1:t |w 0 ) = t @xmath0:t-1 , w 0 ) where w 0 is the given context, and in this work, w 0 can be a story prompt or the beginning of an essay. the generated sequence length t is usually determined by the time t generating the eos (end-of-sequence) token: p (w t |w 1:t -1 , w 0 ) = p (eos|w 1:t-1 , w 0 ) in this work, the model computing the conditional probabilities is self-attention transformer @xcite . we train our model with the crossentropy loss between the predicted conditional probabilities and the ground-truth next token. when the target of generation consists of multiple paragraphs, there are several approaches to indicating the paragraph ending. the most common and obvious approach is to separate paragraphs with line break nl: w 1:@xmath1, nl, ..., p n-1 , nl, p n , eos where p @xmath2:e i } is the words sequence of paragraph i, from the beginning word w b i to the ending word w e i . however, not every paragraph ends with nl, and during the pre-training, not every nl represents the paragraph separator (sep) . a better option is to append a new specific token eop to indicate the end of the paragraph: w 1:@xmath3, 1 , ..., p , n-1 , p , n , eos where p , @xmath4:e i , eop}. then, each paragraph can end with the eop and the transformer-based language model can learn this feature with every paragraph in the training data, without distinguishing when to generate eop and when not to. it is well known that greedy decoding and beam search usually lead to repetitive and degenerate outputs @xcite . sampling-based decoding methods have shown a strong ability in generating diversity, fluency and repetitiveness of the generation with pre-trained language models, such as top-k and top-p sampling. in this work, we choose the top-p sampling decoding algorithm and set the p equals to 0.95. 1 . ## results the results of different settings of utilizing paragraph information (paratype) are shown in we further compared the relations between eos and different eop/sep, which is shown in figure 1 . the horizontal axis represents the relative paragraph index, 0 means the beginning paragraph and 100 means the last paragraph of the story. the vertical axis represents the ranking position of the eos probability among all tokens in the vocabulary predicted by the last token of each paragraph. as eos should only be predicted by the last token of the last paragraph, the ranking at 100 should be higher and the other position should be lower. ac-cording to figure 1(a ), all models rank eos higher as the paragraph index increasing. eop works better than sep as the eop models rank eos higher at the 100th position and lower on the other positions, which can be seen from figure 1(b) . human evaluations. human evaluation results are shown in figure 2 . each cell represents the percentage of the examples that the row system wins the column system on. cells will be filled in blue if the row system outperforms the column system over 10%. it should be noted that cannot distinguish examples are skipped when counting winners for these figures. from figure 2 , we can first find that learning to end paragraphs leads to better ending quality: 63% and 66% results are rated better when comparing sep/eop with none systems, while only 37% and 34% results are rated better for none system. we also find that eop's text endings are slightly better than sep. this is consistent with eos ppl and eos% results in none eop none 45 eop 55 (a) topic relevance none eop none 51 eop 49 (b) fluency none eop none 43 eop 57 (c) ending quality none eop none 47 eop 53 (d) overall preference figure 4: average percentage of systems in row outperform system in column. none eop none 0.67 eop 0.67 (a) topic relevance none eop none 0.69 eop 0.69 (b) fluency none eop none 0.72 eop 0.72 (c) ending quality none eop none 0.58 eop 0.58 (d) overall preference fluency, the overall preference of sep compared with eop is 50%, which means these two systems are similar for human rates. besides, we also find that sep and eop achieve better topic relevance and overall preference. for fluency, there is no significant difference among different systems. we also report fleiss' kappa κ in figure 3 , to access the reliabilities of raters' agreement. κ < 0 means poor agreement, and κ ∼ (0.6, 0.8) means substantial agreement. from this figure, we can find that most of them fall into the substantial agreement group. the overall preference falls into moderate agreement, because this metric is more subjective than the others. human evaluation results for writingprompts are shown in figure 4 and figure 5 . assessors still prefer model fine-tuned with eop rather than without eop/sep. case study. we further conduct case study and detailed in appendix b. the most important observation is that, without eop, the beginning of the generation is more relevant to the end of the input prompt, but the more it generates, the poor quality is. while the generator with eop can generate multiple paragraphs related to the input with a reasonable ending but each paragraph is more independent than human writings. ## conclusion in this paper, we have demonstrated that eop and eos information helps generating better text. chinese gpt2 and english gpt2 are two existing models pre-trained without and with eop respectively, which provides a perfect platform for our proposed experiments. on the chineseessay dataset, the text generation when fine-tuned with eop and eos information is significantly improved. on the other hand for the english task, although (english) gpt-2 was trained with nl which serves as eop to some degree, learning to end paragraphs can still benefit the story generation in terms of automatic metrics and human evaluation results.