Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation (2023)

Abstract: Goal-oriented Script Generation is a new task of generating a list of steps that can fulfill the given goal. In this paper, we propose to extend the task from the perspective of cognitive theory. Instead of a simple flat structure, the steps are typically organized hierarchically — Human often decompose a complex task into subgoals, where each subgoal can be further decomposed into steps. To establish the benchmark, we contribute a new dataset, propose several baseline methods, and set up evaluation metrics. Both automatic and human evaluation verify the high-quality of dataset, as well as the effectiveness of incorporating subgoals into hierarchical script generation. Furthermore, We also design and evaluate the model to discover subgoal, and find that it is a bit more difficult to decompose the goals than summarizing from segmented steps.

## introduction scripts are forms of knowledge representations for ordered events directed by particular goals @xcite . as shown in figure 1 , to obtain a ph.d degree (i.e., goal), one shall follow specific events step-by-step, including do the research, write the paper, etc. such procedure knowledge not only provides a process of problem solving, but also benefits many real-world applications, such as narrative understanding @xcite , task bots @xcite , and diagnostic prediction @xcite . therefore, the task of script generation is proposed to automatically generate events given a goal @xcite . existing works typically assume that events are sequentially arranged in a script, while we argue that this assumption leads to linear generation that is far from enough for comprehensively acquiring the representation about how events are organized towards a task goal. when humans compose a figure 1 : an illustration of the hierarchical decomposition of the goal "how to obtain a ph.d. degree". we use blue for the steps, and yellow for the subgoals. conventional task focuses on steps only, while we highlight the breaking in the middle (red arrows), which refers to the switch between higher-level subgoals. in this paper, we propose to investigate subgoals as an intermediate between goals and steps. given a goal, instead of generating steps in a linear manner, our task seeks to generate scripts at two levels. the first level consists of subgoals, and the second level is detailed steps, where each subgoal contains several steps. such a new setting not only accords to people's cognitive process @xcite but also investigates the model's abilities to understand knowledge from two aspects: problem decomposition and summarization. we propose three research questions to assist the investigation: 1) how to identify the subgoals? 2) how to effectively introduce subgoals to models to improve script generation? 3) does the generated hierarchy align with human intuition? to answer the first question, we construct a new dataset, namely "instructables" foot_0 , about d.i.y projects involving goals, subgoals, and steps. besides, we extend the existing wikihow dataset @xcite with the subgoals. to automatically obtain subgoals, we deploy a segmentation method that separates steps in training data. for each segment, we further leverage a promptbased fine-tuning @xcite ) method to learn subgoal generation. we have designed quantitative metrics for evaluation, which verifies the reasonability of the two-level hierarchy of script. for the second question, we build the benchmark with multiple baselines. the basic idea is to incorporate the goal, subgoals, and steps into one prompt template, and use special tokens to preserve structure information. then, we finetune pre-trained language models (plms) to generate in a top-down or interleaving manner. this allows the model to take a break to conclude on each subgoal before generating the succeeding steps. we have conducted extensive experiments. the results show that by including subgoals, the model generates scripts with better soundness, diversity, and making better sense to achieve the goal. in fact, given gold standard segmentation and subgoals, the improvement is more substantial, indicating space for improvement in our predicted subgoals. to address the third question, we conduct human evaluation to assess the quality of both steps and subgoals, as well as the above two types of model abilities. we observe that the language model shows a weaker ability to decompose the goals than to summarize the steps. we have also analyzed the errors in detail and found that the models some-times generate repetitive subgoals and low-quality steps. the models still have much room for improvement in generating high-quality hierarchical scripts if the goal is too complicated. to summarize, our work has three contributions: 1) we construct the dataset "instructables" and extend the wikihow dataset to investigate the subgoals as intermediate between goals and steps. 2) we build a benchmark with several baselines towards hierarchical script generation. 3) we conduct extensive evaluation to investigate the effects of subgoals qualitatively and quantitatively. controlled nlg script generation is a form of controlled text generation task @xcite since the generated scripts are attributed to the given goal. to increase the controllability of text generation, research efforts investigate the ways of constrained decoding. neurologic decod-ing @xcite improves controlled generation upon semantic constraints by enforcing predicate logic formula at the decoding stage. neurologic a*esque decoding @xcite further incorporates a lookahead heuristic to estimate future constraint satisfaction. controlled text generation tasks can take other forms like generating descriptions conditioned on subparts of a table @xcite . another classic application of controlled text generation is storytelling, whereby stories are generated based on a prompt or a storyline. @xcite generate hierarchical stories conditioned on a prompt that was generated first. @xcite enhance the coherence among different levels of a hierarchical story with a verbattention mechanism. unlike tasks like storytelling, script generation is not open-ended since it is goaloriented. ## task and dataset in this section, we first formulate the new task setting and then introduce the new dataset that we constructed, namely instructables. ## experiments to evaluate the proposed framework for hierarchical script generation, we conduct extensive experiments on the presented wikihow and instructables datasets. we compare our proposed framework with prior strong baseline method and discuss the results ( § 5.2- § 5.3). we have also investigated the best segmentation method as a secondary experiment. ( § 5.4).we conduct ablation studies ( § 5.5). ## case study and error analysis we present two example scripts generated using our method with the wikihow dataset. through these two examples, we analyze the common errors and the typical mistakes encountered. we put two more example scripts from instructables dataset in appendix e since they are rather long. these instructables scripts manifest not only the common errors in this section but also a third typical mistake. repetitive subgoals according to our observation, the pervasive problem is that the generated subgoals are repeats of one another or the goal. this error appears in examples in figure 6 . one cause of this error is the inaccurate segmentation in the training dataset, which raises the difficulty of the subgoal prediction. frequently, the subgoal labeled to a segment is no other but this segment's main goal. the reason is that the segment is a fraction of the script used as input when training the subgoal predictor. moreover, the frequent occurrence of repetitive subgoals in the training dataset may seem like a pattern for the generation model, generating more scripts with repetitive subgoals. a revised loss function that penalizes repetition among goals and subgoals is a possible solution. irrelevant or low-quality steps another mistake with the generated scripts resides in the quality of the steps. for instance, in figure 6 , the model possibly mistakes "black powder" for "black pepper" and generates steps related to cooking. this mistake could originate from the lack of weaponrelated knowledge in the training dataset. figure 7 shows the script of "go green," with two subgoals. despite the reasonable subgoals generated, the steps under "reduce carbon footprint" are irrelevant. the correct interpretation of "go green" is about environmental-friendly measures, while the steps discuss "green and healthy lifestyle." since both wikihow and instructables use pictures to supplement text descriptions, a multi-modal approach may reduce ambiguity in the goal interpretation. ## conclusion this work studies a new task of hierarchical script generation. to facilitate the research on this task, we contribute a new dataset instructables to supplement the existing wikihow resource, gathering over 100,000 hierarchical scripts. we further propose a method to build the benchmark, which learns to segment steps before script generation and concludes the subgoal each segment represents. then, we finetune a t5-base model using prompts that combine subgoals and steps with special tokens. experiment results from both automatic and human evaluation show that scripts generated by our method are in better quality than the baseline. meanwhile, the gap towards the performance upper-bound still indicates much room for improvement. in the future, we are interested in exploring the idea of hierarchical generation in dealing with paragraphs in document generation.