Paper: End-to-end Story Plot Generator (ICLR-2024)

Reviewer comment (Evaluation):
My major concern is the lack of human evaluation. The authors state multiple times that “story plot is relatively short and thus easy for humans to evaluate” yet perform no human evaluation. Relying solely on a model like GPT-4 for such a complex task is, in my opinion, a major limitation to the soundness of the claims. I hence invite the authors to pair the main evaluations (ie Table 5 and 6) with human evaluation. I will then happily advocate for acceptance.

Author response:
Thanks for your suggestion. We added human evaluation results, and the details can be found in the global response.

Reviewer comment (Experiments):
Another minor limitation is that RLHF results in 5 different models, which are each compared against a single SFT model. Could the authors consider combining the different rewards into a single RLHF model and then ask humans to compare it against SFT according to the metrics Q1-6?

Author response:
Training a unified model that can produce better story plots in multiple aspects simultaneously would definitely be an interesting future direction, which is also studied in a concurrent work [1]. Note that according to our human evaluation results, the model RLPlot_Q1 achieves better performance than E2EPlot consistently in all the aspects, which shows that even if we only use a reward model for a specific aspect, it might generalize well to other aspects since usually different aspects might be correlated.

Reviewer comment (Novelty):
The Related Work section is comprehensive. To make the paper’s scope broader and linked to work in the computer vision and multimodal communities, I would recommend adding a brief paragraph on work in story generation for image-to-text [1], text-to-image [2] and text-to-video [3] tasks.

Author response:
Thanks for your suggestion. We have added a brief paragraph on these three works in the revision.

Reviewer comment (Writing):
In page 2, when you say “a completion model which accepts a suffix” I was not entirely sure what you meant. It then became clear later throughout the paper. I recommend adding a brief explanation that you mean a model that can do text infilling given a text prefix and a text suffix, and cite [4].

Author response:
Thanks for your suggestion. We have fixed it in the revision.

Reviewer comment (Presentation):
Line 3 of Sec 3 should reference Table 7, not Table 1.

Author response:
Thanks for catching this! We have corrected it in the revision.

Reviewer comment (Writing):
First line of Sec 3.2 might make it explicit that the RLHF results in 5 models.

Author response:
Thanks for your suggestion. We explicitly mentioned that RLHF results in 5 models in the revision.

Reviewer comment (Experiments):
Since the author has spent significant effort to collect human ratings on 7000 story pairs, why not do another 300 pairs? This would create a much more solid evaluation.

Author response:
Thanks for your suggestion. We added human evaluation results during the rebuttal to make the evaluation more solid. Please see the global response for details.

Reviewer comment (Writing):
In Section 2.1.1, the authors do not define what is meant by "supports suffix". Although the meaning can be inferred from context, it creates temporary confusion.

Author response:
We have fixed it in the revision to avoid confusion. Thanks for your comment.

Reviewer comment (Theory):
The authors make several claims about how humans supposedly do things, such as that humans write stories in a specific way (Page 2, Paragraph 2) and that humans write from coarse to fine (Challenge 1). However, these claims are unsubstantiated.

Author response:
The planning-based method is common in many generation tasks (e.g., [3],[4]).  The coarse-to-fine generation has been widely used in previous work (such as event-to-sentence generation in [2], which is mentioned by reviewer zk9y).  Also, it is supported by the comment of reviewer G5uN that “The task of story plot generation is interesting and allows to disentangle two major phases of content creation often seen in humans: planning and coarse-to-fine generation”.

Reviewer comment (Novelty):
What novel or surprising scientific insights or findings are reported by this paper?

Author response:
We discussed this in detail in the global response (see the **Novelty** section in the global response).

Reviewer comment (Evaluation):
What evidence can support the claim that humans write stories by first planning an outline?

Author response:
**[Justification of planning-based/coarse-to-fine methods]** For the planning-based method or coarse-to-fine method, we agree that not everyone will adopt such a strategy. However, this is a common strategy that many authors may use, including an amateur novel writer in our team. More importantly, these methods are proven to be effective in the generative method in the story generation literature (e.g., [3][4], also referenced as “event-to-sentence” in [2]) , which inspires us to use similar approaches to generate stories automatically.

Reviewer comment (Evaluation):
Although the paper describes generated units as 'plots' it stands in-between substantial previous work of Plan-based plot generation [Riedel and Young, 2010] where plot elements were narrative functions or operators and text-based story generation [Wang and Wan, 2019] (in the paper's references), originating in story completion experiments, up to systems to which the approach is compared that generate plot + narrative text [Yang et al., 2023 (2022a in the paper's references)]. It is thus unclear whether what is presented in the paper, in particular in Figure 7 are plots or storyboards, and this is not just a terminological issue, as it affects the ability to apply structural evaluation methods to plots as well as creating an unusual setting for users to 'evaluate' the plot as opposed to evaluation methods based on end-story quality or story understanding (e.g. QUEST graphs [Graesser et al., 1992] used in [Christian and Young, 2004]).

Author response:
We agree that the terminology ‘plot’ does not have a rigorous definition, and there are different choices of the format of the plot. The most suitable definition of the plot in our paper should be ‘some certain structure that captures most of the essential information of the story, which is often created in the planning stage’. 

In our work, we choose the structure of the plot (premise, settings, characters, outlines)  mainly due to the following reason:

- The underlying principle is “coarse-to-fine”, i.e., to progressively provide more and more detailed information. This coarse-to-fine representation makes end2end autoregressive generation easier.
- While short, this representation still contains a good amount of information about the story, which is helpful for multiple perspective evaluations (represented by multiple questions). 
- This representation is used by the SOTA story generation work DOC [1] for the planning stage. It can be used to generate high-quality full-length stories. This demonstrates that such a structure can lead to full story generation.
- The plot structure we chose in this paper is text-based, which could greatly benefit from the powerful text-based LLMs that take the plot as input and produce the whole story (usually with multiple steps such as the second stage of DOC [1]).

Reviewer comment (Novelty):
It would thus be necessary to much better justify the approach compared to end-story text generation (not just completion), or non-DL, non-LLM based plot generation (e.g. Plan-based). In particular, considering that LLM text generation could be used in conjunction to other plot/backbone generation methods, or that the DOC method is in reality generating both plot and (textual) narrative.

Author response:
**[Defintion of plots and difference from previous literature]** As we mentioned in the previous response, the (premise, setting, characters, outlines) structure of our plot is proven (by the SOTA story generation work DOC [1])  to be efficient in carrying the essential information of the whole story and generating the whole story. We agree that our plot has a different format from those of some of the previous work, but it is hard or even impossible and maybe unnecessary to propose an efficient format of plot that can unify all the previous formats of plots. We also want to emphasize that our work is an LLM-based story generation method, which makes it natural to choose a text-based plot that might be different from previous work that is not LLM-based.  The LLM-based nature of our work makes our story plot different from some previous non-LLM-based works, and this nature makes it possible to generate a story plot at a much faster speed, and our generator model can be easily improved from feedback.

Reviewer comment (Novelty):
Regarding reward models, there should probably be a discussion of previous approaches in text-based narrative generation, for instance [Ammanabrolu et al., 2019] and [Castricato et al., 2022].

Author response:
Compared to [Ammanabrolu et al., 2019],  where the reward is based on how close the generated event is to a pre-trained goal, our reward model training is more similar to CARP [Castricato et al., 2022], a contrastively-trained preference model as a reward signal in story generation. Note that our reward model is trained on story plots (using cross-entropy loss on the preference label) instead of the whole story, which makes learning and human labeling easier without losing important signals. We added these contents in Section 2.3 in the revision.

Reviewer comment (Evaluation):
Evaluation techniques are somehow underspecified considering previous work in evaluating narrative generation. The expression of comparative preferences by GPT4 is moderately replicable and appears rather qualitative and not sufficiently related to structural properties of the plot and rigorous definitions of the above properties.

Author response:
**[Evaluation techniques]** Comparison-based evaluation technique on a specific aspect such as interestingness or the overall quality is also used and proven to be efficient by previous works such as DOC [1]. We emphasize that these aspects are good evaluation metrics on story plots because:
1. These high-level aspects remain roughly invariant on the whole story and the story plot;
2. The plots are text-based and relatively short, which can either be read by humans or parsed by LLMs easily.

We are happy to address any further questions.