Multimedia Generative Script Learning for Task Planning (2023)

Abstract: Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotypical activities. An important aspect of this process is the ability to capture historical states visually, which provides detailed information that is not covered by text and will guide subsequent steps. Therefore, we propose a new task, Multimedia Generative Script Learning, to generate subsequent steps by tracking historical states in both text and vision modalities, as well as presenting the first benchmark containing 5,652 tasks and 79,089 multimedia steps. This task is challenging in three aspects: the multimedia challenge of capturing the visual states in images, the induction challenge of performing unseen tasks, and the diversity challenge of covering different information in individual steps. We propose to encode visual state changes through a selective multimedia encoder to address the multimedia challenge, transfer knowledge from previously observed tasks using a retrieval-augmented decoder to overcome the induction challenge, and further present distinct information at each step by optimizing a diversity-oriented contrastive learning objective. We define metrics to evaluate both generation and inductive quality. Experiment results demonstrate that our approach significantly outperforms strong baselines.

## introduction robots rely on understanding the present realworld state and predicting the subsequent steps to better assist humans in daily stereotypical tasks such as meal preparation and gardening @xcite . as an example, robohow @xcite uses articles from wikihow foot_1 to assist robots in everyday tasks in human working and living environments. however, the problem is that not all daily tasks are well documented. thus, generating a sequence of steps that lead to a given goal (i.e., goal-oriented generative script learning) @xcite @xcite @xcite has a fundamental importance in allowing robots to perform unseen tasks by understanding the patterns in previously observed similar tasks. despite this, previous goal-oriented generative make a bubbling brook bracelet goal: ## problem formulation we propose a new multimedia generative script learning task: given an activity goal g, an optional subgoal m that specifies the concrete needs, and the previous multimedia step history ## dataset collection using articles from gardening and crafts categories as case studies, we create a new dataset based on the english wikihow dump (2021/05). there are typically three levels of hierarchy in a wikihow article: goals which describe the overall task, subgoals which represent the intermediate process to accomplish a goal, and steps which are the specific actions to complete a subgoal. for each wikihow article, we collect step-image pairs as well as their goals and methods foot_2 . we split the whole dataset based on the task categories. therefore, the validation and test sets contain tasks not 4 method ## evaluation metrics generation quality evaluation following common practice in text generation, we first evaluate our model with bleu @xcite , rouge @xcite , and meteor @xcite scores to examine the content overlap between generated steps and ground truth. inductive quality evaluation in order to determine whether the inferred subsequent steps are factually correct, we further evaluate the models with bartscore @xcite and the semantic similarity score @xcite . the semantic similarity score uses a cross-encoder pretrained on stsbenchmark @xcite to calculate the semantic similarity between two sentences. in addition to evaluating whether the generated step matches the next step, we also check whether the generated step matches any subsequent step. this enables the model to earn credit if it generates a step that appears in the future. we propose a multimodal-retrieval based metric: for each generated step, we use it as a query to search all corresponding step-image pairs under the same subgoal/goal from the testing set. we then compute hit@1 for results that fall into ground-truth future step-image pairs. similar to section 4.3, we use sbert @xcite to rank the most similar steps under the same subgoal to get text@1 (t@1). to compute image@1 (i@1), we use clip @xcite to rank the most similar images under the same subgoal. if the top-1 retrieval results appear in the subsequent steps, we consider it a hit. the retrieval-based metric captures normalized semantic similarity concerning all related steps under certain subgoals. the clipbased retrieval metric also enables the evaluation of the cross-modality semantic similarity. additional details of the evaluation setup are in the appendix c. ## related work previous script learning tasks fall into two forms: selective and generative. the selective script learning tasks focus on modeling the script interactions given a list of candidates, including multichoice goal step inference/ordering @xcite , script retrieval @xcite , action anticipation @xcite , procedure segmentation @xcite @xcite , multi-choice visual goal-step inference @xcite , multimedia procedure planning @xcite , multimedia step ordering @xcite , instructional video retrieval @xcite , and step classification @xcite . despite promising results, their performance heavily relies on the given candidates, making them difficult to generalize for unseen activities. the second category is text-based generative script learning @xcite @xcite @xcite @xcite . however, this is the first work to provide a multimedia goal-oriented generative script learning along with a new multimodalretrieval based metric. different from @xcite , which uses a video to generate the next step, our new task uses step image-text pairs as input. unlike previous multimedia script learning frameworks with a multimedia encoder to capture visual and textual information, we use a captioner to convert images into captions summarizing the important objects in images. the gosc dataset @xcite ) contains the steps of daily stereotypical tasks, but most of the steps (52.6%) in this dataset are unordered, making it infeasible to evaluate the next-step prediction. consequently, we adapted the best model mt5 @xcite in gosc to our settings, i.e., the monolingual version t5, and used it as an equivalent baseline to show the comparison with the state-of-the-art model. to handle irrelevant sentences in the input, instead of using a token-level gating mechanism that only depends on the token itself @xcite , we introduce a sentence (step/caption) level gating mechanism whose gates depend on global context and weighted sentence representations. our work is also related to retrieval-augmented text generation models @xcite @xcite . however, instead of retrieving knowledge from an external corpus, we use steps from similar tasks in training data to guide the generation process. moreover, we introduce a new contrastive learning loss to increase diversity. previous contrastive learning-based text generation methods usually use negative samples constructed by sequence manipulation @xcite or perturbation @xcite . inspired by @xcite which uses self-negatives for knowledge graph completion and that the generation output tends to repeat the input, we extend self-negatives for sequence-to-sequence contrastive learning. we also retrieve similar steps from the training set as additional hard negatives. ## conclusion we propose a novel multimedia generative script learning task with the first benchmark featuring step and descriptive image pairs to generate subsequent steps given historical states in both text and vision modalities. moreover, we build a new script learning framework consisting of a selective multimedia encoder, a retrieval-augmented decoder, and a diversity-oriented contrastive learning objective to generate the next steps. furthermore, we define a new multimodal-retrieval based metric which can be used for multimedia script learning tasks. automatic and human evaluation results demonstrate consistent performance improvements.