Semantic Frame Forecast (2021)

Abstract: This paper introduces Semantic Frame Forecast, a task that predicts the semantic frames that will occur in the next 10, 100, or even 1,000 sentences in a running story. Prior work focused on predicting the immediate future of a story, such as one to a few sentences ahead. However, when novelists write long stories, generating a few sentences is not enough to help them gain high-level insight to develop the follow-up story. In this paper, we formulate a long story as a sequence of “story blocks,” where each block contains a fixed number of sentences (e.g., 10, 100, or 200). This formulation allows us to predict the follow-up story arc beyond the scope of a few sentences. We represent a story block using the term frequencies (TF) of semantic frames in it, normalized by each frame’s inverse document frequency (IDF). We conduct semantic frame forecast experiments on 4,794 books from the Bookcorpus and 7,962 scientific abstracts from CODA-19, with block sizes ranging from 5 to 1,000 sentences. The results show that automated models can forecast the follow-up story blocks better than the random, prior, and replay baselines, indicating the feasibility of the task. We also learn that the models using the frame representation as features outperform all the existing approaches when the block size is over 150 sentences. The human evaluation also shows that the proposed frame representation, when visualized as word clouds, is comprehensible, representative, and specific to humans.

## introduction writing a good novel is hard. creative writers can get stuck in the middle of their drafts and struggle to develop follow-up scenes. writing support systems, such as heteroglossia @xcite , generate paragraphs or ideas to help writers figure out the next part of the ongoing story. however, figure 1 : the semantic frame forecast is a task that predicts the semantic frames that will occur in the next part of a story based on the texts written so far. generating a coherent story across multiple sentences is challenging, even with cutting-edge pretrained models @xcite . to generate coherent stories, researchers often first generate a highlevel representation of the story plots and then use it as a guide to generate a full story. for example, @xcite propose an event representation that uses an svo tuple to generate story plots; plan-and-write @xcite uses the rake algorithm @xcite to extract the keyword in each sentence to form a storyline and treat it as an intermediate representation; @xcite use predicate-argument pairs annotated by semantic role labelers to model the structure of stories; and @xcite take words with a certain part-of-speech tag as anchors and show that using anchors as the intermediate representation can improve the story quality. however, these projects all focused on short stories: the event representation is developed on a wikipedia movie plot summary dataset @xcite , where a summary has an average of 14.52 sentences; plan-andwrite uses the rocstories dataset @xcite , where each story has only 5 sentences; fan et al. test their algorithm on the writing-prompts dataset @xcite , where stories have 734 words (around 42 sentences) on average; and zhange et al.'s anchor representation is developed on the vist dataset @xcite , where a story has 5 sentences. all the existing intermediate representations are generated on a sentence basis, meaning that the length of the representations increases along with the story length. that is, when applying these representations to novels that usually have more than 50,000 words (as defined by the national novel writing month (wik, 2020)), it is not likely that such representations can still work. we thus introduce a new frame representation that compiles semantic frames into a fixed-length tf-idf vector and a semantic frame forecast task that aims to predict the next frame representation using the information in the current story block (see figure 1 ). two different datasets are built to examine the effectiveness of the proposed frame representation: one from bookcorpus @xcite , a fiction dataset; and one from coda-19 @xcite , a scientific abstract dataset. we establish several baselines and test them on different story block sizes, up to 1,000 sentences. the result shows that the proposed frame representation successfully captures the story plot information and helps the semantic frame forecast task, especially for story blocks with more than 150 sentences. to enable humans to perceive and comprehend frame representations, we further propose a process that visualizes a vector-based frame representation as word clouds. human evaluations show that word clouds represent a story block with reasonable specificity, and our proposed model produces word clouds that are more representative than that of bert. ## semantic frame forecast as shown in figure 1 , we formulate a long story as a sequence of fixed-length story blocks. each story block (figure 2 (1)) has a set of semantic frames (figure 2 (2)) @xcite . we convert a story block into the frame representation (figure 2 (3)), a tf-idf vector over semantic frames, by computing the term frequency in that story block and the inverse document frequency over all the story blocks in the corpus. framenet @xcite , 1998) defined a total of 1,221 different semantic frames, so the generated tf-idf has 1,221 dimensions. the semantic frame forecast is then defined as a task to predict the frame representation of the n+1-th story block using the foregoing content, namely the n-th story block. evaluation metric. we use cosine similarity between the predicted vector and the gold-standard vector (complied from the human-written story block) for evaluation. many other metrics, such as mean-squared error (mse), also exist to measure the distance between two vectors. ## data we build the dataset from the existing bookcorpus dataset @xcite and coda-19 dataset @xcite . this section describes how we preprocess the data, remove undesired content, and build the final dataset. bookcorpus dataset. we obtain a total of 15, 605 raw books and their corresponding meta data. to get high-quality fictional content, we remove books using the following heuristic rules: (i) short books whose size is less than 10kb; (ii) books that contain html code; (iii) books that are in the epub format (an e-book file format); (iv) books that are not in english; (v) books that are in the "non-fiction" genre; (vi) books that are in the "anthologies" genre; (vii) books that are in the "graphic novels & comics" genre. since most books contain book information, author information, and some nonfictional content at the beginning and end of the book, we use regular expressions to match the term "chapter" to locate the chapter title. only the contents between the first chapter title and the last chapter title are kept. the last chapter is also removed as there are no certain boundaries to identify the story ending. books whose chapter titles are unlocatable are also removed. after removing all the unqualified books, a total of 4, 794 books were used in our dataset. we transliterate all non-ascii characters into ascii characters using unidecode ( https://pypi.org/project/unidecode/ ) to fulfill the requirement of open-sesame @xcite . open-sesame is then used to parse the semantic frames for each sentence. the books are split into training/validation/test sets following a 70/10/20 split, resulting in 3, 357, 479, and 958 books, respectively. to measure the effect of frame representation for different context lengths, we vary the story block length, using 5, 10, 20, 50, 100, 150, 200, 300, 500, and 1, 000 sentences. when creating instances, we first split a book into story blocks with the specified length and extract all the consecutive two story blocks as instances when context window size (see figure 1 ) is set to 1. the idf of the semantic frame is then computed over the story blocks using all the training sets. combining with the tf value in each story block, we convert story blocks into frame representations. we use scikit-learn's implementation @xcite of tf-idf but with a slight modification on idf: scikit-learn uses idf (t) = log( n df (t)+1 ) to compute a smoothing idf, but we use idf (t) = log( n df (t) ). the detailed statistic information is shown in coda-19 dataset. we envision a broader definition of "creativity" in writing and attempt to apply story arc prediction technologies to the do we remove sentences that are annotated as "other," an aspect for sentences that are not directly related to the content (e.g., terminology definitions or copyright notices.) abstracts that contain unicode characters are also removed. a total of 7, 962 abstracts are used in our dataset. we then use open-sesame to parse the semantic frames for each sentence. we adopt coda-19's original split, where the training set, validation set, and testing set have 6, 509, 737, and 716 abstracts, respectively. three different lengths of story block are used: 1, 3, and 5. we then create instances and compute tf-idf as described above. ## models we implement two naive baselines, an information retrieval baseline, two machine learning baselines, two deep learning baselines, an existing model and a text generation baseline. replay model. for each instance, the replay model takes the frame representation in the n-th story block as the prediction, i.e., the same frames will occur again. prior model. the prior model computes the mean of the frame representation over the training set and uses it as the prediction for all the testing instances. information retrieval with frame representation. for each instance, the information retrieval model searches for the most similar story block in the training set and takes the frame representation from its next story block as the prediction. in this setting, we adopt the cosine similarity on frame representations to measure the story similarity. for block size 5 in the bookcorpus dataset, there are around 3.7 million instances in the training set, which is infeasible to finish. ## experimental results and analysis predicting forthcoming semantic frames is remarkably challenging yet possible. machinelearning models outperform the two naive baselines for different story lengths. in the bookcorpus dataset, bert performs the best for story blocks under 100 sentences, while lgbm performs the best for story blocks over 150 sentences. in the coda-19 dataset, scibert performs the best for block sizes of 1 and 3, while dae performs the best for a block size of 5. while the task is very challenging, these results shed light on the semantic frame forecast task. however, the improvement replay baseline shows the relation of consecutive story blocks. the replay baseline assumes that the events that happen now will likely happen again shortly. the results in event-rep works better in short stories. in the bookcorpus dataset, event representation works better than the frame representation in small block sizes (5, 10, and 20). however, starting from a block size of 50, the model cannot perform as well as the other models. we thus conclude that event representation works better in short stories. the main reason is that event representations are generated on a sentence-by-sentence basis and will create overwhelming information on long stories. the existing intermediate representations (see section 1) are mostly generated from sentences and will likely have the same issue as the event representation. compared to the existing works, the proposed frame representation encodes a story block, no matter how long it is, into a fixed-length vector and therefore performs better on longer stories. bert performs very well in short stories. the results of bert and scibert in the good performance does not merely come from the number of instances. deep learning methods often require more instances for training. to show that the result in gpt-2 is not effective. gpt-2 is not effective in predicting the story flow even though it can generate reasonable sentences. even the naive replay ## human evaluation we further evaluate the proposed method with humans. we first visualize the vector of semantic frames into word clouds so that humans can perceive and comprehend it. we then use online crowd workers to test the (i) representativeness and (ii) the specificity of the produced word clouds. ## conclusion this paper proposes a semantic frame forecast task that aims to forecast the semantic frames in the next 10, 100, or even 1,000 sentences of a story. a long story is formulated as a sequence of story blocks that contain a fixed number of sentences. we further introduce a frame representation that can encode a story block into a fixed-length tf-idf vector over semantic frames. experiments on both the bookcorpus dataset and coda-19 dataset show that the proposed frame representation helps semantic frame forecast in large story blocks. by visualizing the frame representation as word clouds, we also show that it is comprehensible, representative, and specific to humans. in the future, we will introduce the frame representation into story generation models to ensure coherence when generating long stories. we will also explore the possibility of supporting writers to develop the next part of their stories by generating semantic frames as clues using semantic frame forecast.