Toward Better Storylines with Sentence-Level Language Models (2020)

Abstract: We propose a sentence-level language model which selects the next sentence in a story from a finite set of fluent alternatives. Since it does not need to model fluency, the sentence-level language model can focus on longer range dependencies, which are crucial for multi-sentence coherence. Rather than dealing with individual words, our method treats the story so far as a list of pre-trained sentence embeddings and predicts an embedding for the next sentence, which is more efficient than predicting word embeddings. Notably this allows us to consider a large number of candidates for the next sentence during training. We demonstrate the effectiveness of our approach with state-of-the-art accuracy on the unsupervised Story Cloze task and with promising results on larger-scale next sentence prediction tasks.

## introduction computer generation of stories and other kinds of creative writing is a challenging endeavor. it entangles two difficult tasks: the generation of fluent natural language and the generation of a coherent storyline. in the recent year, neural language models have made tremendous progress with respect to fluency @xcite @xcite , but coherency is still a major challenge @xcite . the generation of coherent stories has recently been addressed with additional conditioning: @xcite suggest conditioning on a story prompt, @xcite propose collaboration between a generative model and a human writer, and @xcite suggest attending to a commonsense graph relevant to the story plot. conditioning based on a generated story plan @xcite @xcite , a se- * university of pennsylvania, †google quence of images @xcite or character roles @xcite have also been considered. our work is orthogonal to these efforts. rather than considering additional conditioning, we propose a model which takes as input several sentences of context and selects the best next sentence within a large set of fluent candidate sentences. we leverage pre-trained bert embeddings @xcite to build this sentence-level language model. given the embeddings of the previous sentences of the story, our model learns to predict a likely embedding of the next sentence. this task isolates the modeling of long-range dependencies from the prediction of individual words, which has several advantages. first, since our model only needs to determine how well each candidate sentence would fit as a coherent continuation to the story, it does not spend capacity and time to learn fluency. second, our model does not manipulate individual words but full sentences, which allows us to consider tens of thousands of candidate sentences at a time. this contrasts with prior work @xcite where the need to learn token-level representations limited the number of candidate next sentences that could be considered to a few hundred. third, we can rely on compact model architectures that train quickly because we take advantage of strong semantic representation from a pre-trained bidirectional language model, bert, as our sentence embeddings. of course, these benefits also imply that our sentence representation is limited to the information extracted by the pre-trained model. nevertheless, we show that our model achieves state-of-the-art accuracy among unsupervised approaches on the story cloze task: predicting which of two sentences coherently ends a short story. our work also opens up the possibility of ranking thousands of candidate sentences from a large literature repository. on the roc stories dataset, we observe that training with a large number of candidates is key for selecting the most coherent ending among a large set of candidates at test time. we also show preliminary results on the efficacy of our method for ranking candidate next sentence on the toronto book corpus @xcite , a much larger book dataset. we envision that our methods for scoring many candidate next sentences by their coherence with the context might be useful to downstream generation tasks where it is possible to generate many fluent continuations of a text, but it remains an unsolved problem how to refine and choose the best of them. to encourage this exploration, we release our code and models foot_0 . ## proposed method we propose a sentence-level language model: our model estimates p (s t+1 |s 1:t ), the probability distribution for sentence s t+1 given the t previous sentences, s 1 , . . . s t . since it is intractable to marginalize over all possible candidate next sentences, we consider a finite but large set of n valid, fluent sentences. without loss of generality, we can consider s t+1 ∈ {1, . . . , n } as an integer index into that set of possible next sentences. this strategy resembles negative sampling in word2vec @xcite . our model represents sentences with precomputed vector embeddings. specifically, sentences are represented by the mean of the 768dimensional contextual word embeddings of the second-to-last layer of bert @xcite . this representation has shown to encode more transferable features compare to other layers @xcite . alternative sentence representations were considered, including embeddings from the universal sentence encoder @xcite and a weighted mean of the bert embeddings using inverse document frequency weighting @xcite . none of these alternatives improved our results however. motivated by simplicity, we consider a classical multi-layer perceptron (mlp) f θ which takes as input the context sentence embeddings concatenated into a single vector. at the output layer, we perform a softmax operation. if we represent candidate sentences {1, . . . , n } by the embeddings {e i } n @xmath0, our model estimates the probability that i is the next sentence by the softmax log p (s t+1 = i|s 1:t ) = e i h -log z(h) where @xmath1:t ) is the output of the mlp given context s 1:t , and z(h) = n @xmath2. at train time, the candidate set {1, . . . , n } consists of the correct next sentence along with n -1 distractor sentences. the distractors can either be static (the same set used throughout training) or dynamic (picked at random from a larger set for each train batch). in this case, the "vocabulary" of next values to choose from changes with each train step, similar to negative sampling @xcite . at test time, novel sentences can be embedded with bert and scored by our model. like a classical language model, we optimize for the likelihood of the true next sentence's embedding. however, when training we found that the sentences from the context (s 1 , . . . , s t ) often ended up being given very high scores by our model. inspired by work in sentence reordering @xcite , we incorporated an auxiliary loss, which we refer to as csloss, that only includes the context sentences s 1:t in the distractor set. lastly, we consider a residual variant of the mlp (referred to as resmlp) with skip connection between layers, as described in @xcite . the residual model trains faster and sometimes achieves higher accuracy than the non-residual model. though we experimented with recurrent @xcite and self-attention @xcite ) models, we did not observe improvements, perhaps because the input to our model is already the high-dimensional output of a large mask language model. we leave deeper architecture exploration, which will be especially critical as context length is extended, to future work. ## experimental setup we first describe our experiments on the roc stories dataset of short 5-sentence stories before showing our setup on the larger toronto book corpus. ## results we evaluate on the story cloze task, a binary classification task, as well as on the task of ranking a large set of possible next sentences. ## conclusions this work introduces a sentence-level language model which takes a sequence of sentences as context and predicts a distribution over a finite set of candidate next sentences. it takes advantage of pretrained bert embeddings to avoid having to learn token-level fluency, allowing the model to focus solely on the coherence of the sentence sequences. our results on the story cloze task highlight the advantage of this strategy over word-level language models. at train time, our model considers much larger amounts of text per update than typical tokenlevel language models. we show that this strategy context: my family got up one morning while on vacation. we loaded our boat onto a trailer and drove to the beach. after loading up from the dock, we took off on our boat. after only a few minutes on the sea, dolphins began to swim by us. gt: (22.89) we played with them for a while and then returned to the dock. rank: 9 top scored: (25.06) we were definitely lucky to see them and it made the trip more fun! (24.31) they loved everything about that trip and vowed to do it again! (23.76) we were sad to come home but excited to plan our next vacation. (23.72) it was one of our best vacations ever! context: ellen wanted to be smart. she started reading the dictionary. she learned two hundred new words the first day. ellen felt smart and educated. gt: (30.23) she couldn't wait to use the new words. rank: 1 top scored: (30.23) she couldn't wait to use the new words. (29.78) she felt like a new woman when she was done! (29.01) she decided to go back to speaking like her normal self! (28.95) she felt like a new girl! context: it was a very cold night. becky was shivering from the cold air. she needed to cover up before she caught a cold. she wrapped up in her favorite blanket. gt: (18.717398) becky finally got warm. rank: 3,028 top scores: (39.09) laura ended up shivering, wrapped in a blanket for hours. (36.71) after being cold all day, the warmth felt so good. (33.77) sam was able to bundle up and stay cozy all winter. (33.38) the breeze felt good on her wet shirt. context: benjamin enjoyed going fishing with his grandfather as a kid. they would pick a new location to go to every summer. benjamin liked seeing who would catch the biggest fish. even after his grandfather passed he continued the tradition. gt: (26.65) he now takes his own grandchildren to create memories for themselves. rank: 2,281 top ranked: (34.71) greg grew to love golfing and is now his favorite thing to do. (33.82) it was a tradition tim continues with his own family. (33.63) alex learned to be grateful of his family's unique tradition. (33.40) tom was sad that he would have to let his son down. as future work, we plan to further evaluate the impact of different sequential architectures, longer contexts, alternative sentence embeddings, and cleverer selection of distractors. inspired by deliberation networks and automatic post editing methods @xcite , we ultimately want to apply our model to two-step generation, first selecting a sentence from a large set before refining it to fit the context.