From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Looking forward to your feedback 
comment: Dear reviewers,

Thanks for your constructive and insightful comments. We are glad that all reviewers agree that the paper is novel in the task, dataset, and model design of generative script learning. In discussion stage 1, we have revised and improved the paper according to the comments. We addressed concerns by providing detailed responses and adding more discussions to the main paper and appendix. We would appreciate it if you would let us know whether our responses have addressed your concerns.

As part of discussion stage 2, we look forward to discussing further with you any other questions and suggestions you might have. Again, we are grateful for your time and efforts in reviewing our paper!

Best,

Authors

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Response to Reviewer 8Xq1 part2
comment: **Q5** We understand the reviewer‘s concern is about the capacity/inducting metrics. Therefore, we propose new multimodal-retrieval-based metrics apart from regular automatic metrics in section 3.2, which use prediction as a query to check whether a step appears in the future based on illustrative images and text descriptions under the same subgoal. Evaluating the predictions in multimedia can achieve better inductive ability than text-based metrics.

**Q6** We agree that there is inconsistency for different metrics. However, those metrics focused on different aspects of quality. However, our proposed model(BART+CP+ME+RD+CL) generally achieves the best scores according to the human evaluation, which is more comprehensive. Because this is a pilot study for a new task, we leave proposing comprehensive automatic metrics to future work.


Finally, thank you for pointing out the minor errors. We fixed the abstract and caption for Table 1 and added additional model details to Appendix G.

**References**
1. Deitke, M., etc. (2022). Retrospectives on the Embodied AI Workshop. arXiv preprint arXiv:2210.06849.
2. Wang, Zhenhailong, et al. "Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners." NeurIPS (2022).

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Response to Reviewer 8Xq1 part1
comment: Thank you for your helpful comments. We are glad that you found our paper containing detailed experiment results for a new task. We have revised the paper and its supplementary. 

**Q1** 
1. We understand the reviewer’s concern on multimedia vs vision-only vs text-only. However, we respectfully disagree. We call it "multimedia" script learning because it is the first task to take both textual and visual historical steps as input.  We propose this new task because the illustrative image and the corresponding text are complementary (the CLIPScore between the image and corresponding text is not high). Future steps should not be only based on previous steps, but also on visual scenes depicting the current status. In this regard, we argue that as a novel task to consider both modalities and make a joint decision to predict the next step, it is "multimedia" script learning (rather than vision-only or text-only script learning). 
2. About the "multimedia challenge", the definition on Page 2 as shown in Sec 2 Problem Formulation) defines the input as “previous multimedia step history”, which is perfectly aligned with "multimedia challenge". If the confusion is about the first challenge of “Visual-State Trackable”, we listed challenges here to compare with traditional text-only script learning, rather than a definition (as shown in the beginning of this paragraph “a good script has three hallmarks”). We would like to echo that oOur paper is the first to extend traditional text-only script learning to multimedia input. As a result, we believe the key challenge is the trackability of visual scenes, and we highlight how vision can contribute to this traditional text-only task. We argue that this major challenge of how to effectively add vision information into the model is aligned with the task definition.
3. About the "multimedia encoder", we use a modular encoder (leveraging BLIP) since the performance is much better than end-to-end multimedia encoder. As shown in Table 3, we included a baseline called “CLIP-BART” (detailed in Section 4.1). It is an end-to-end multimedia encoder by directly using CLIP embeddings, and we finetune them on the domain datasets for a fair comparison.  Results in Table 3 prove that  CLIP-BART underperforms the BART model, showing that CLIP features only hurt the generation performance rather than help. We suspect that the current multimodal encoder is not focusing on the information needed for task planning but more on the detailed visual features. Similar discoveries have been proven in [1,2] that modular encoders achieve better performance than end-to-end encoders. 

**Q2** In our experiments, we have employed automatic evaluation metrics, including Self-BLEU and unique n-gram percentage, to evaluate the diversity of model outputs. The results are presented in Table 6 and Table 7. The results show that the diversity-oriented contrastive loss help the model generate more diverse steps. We further conduct the human evaluation to verify the diversity of the model outputs. The results are shown in Table 8. Those results further verify the effectiveness of diversity-oriented contrastive loss in improving the diversity of predictions.

**Q3** We agree with the reviewer’s concern about the effect of historically relevant steps. We use the same semantic similarity score mentioned in Section 3.2 to check the correlations between the quality of retrieved historical relevant steps and the quality of model predictions. The results are shown in Appendix H. Figure 5 shows that the quality of model predictions is positively related to the quality of retrieved historical relevant steps. 

**Q4** We see that the reviewer raises an interesting question about the multimedia baseline. Our clip-BART baseline is multimedia and directly takes in the image CLIP embeddings and text as input. However, as shown in Table 3, directly applying visual features with text input hurts the performance, suggesting that captions usually carry more straightforward information than images and the current multimodal encoders still cannot perfectly embed text and images into the same semantic space.

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Response to Reviewer oNPU part2
comment: 
**D**

* (i) As shown in Table 1, there are 1857 tasks for gardening and 2654 tasks for crafts. We use the tasks defined by WikiHow. So those tasks are all explicit. 
* (ii) The goal is not to train a large language model from scratch but to leverage the ability of large-scale pretraining models such as BART and add our data on top of it to make it capable of planning tasks. We presented an effective way to use WikiHow task planning data and proved its effectiveness. We agree that data might not be sufficient for training a large language model from scratch, and we accommodate the noise problem by adding a gate mechanism (as detailed in Sec 2.5) to focus on the important information rather than all information. 
* (iii) As detailed in Sec 4.2 and Sec 4.3, we deal with all possible steps in both automatic evaluation and human evaluation. In automatic evaluation, the multimedia-retrieval based metrics check all possible future steps. The multimedia-retrieval based metrics use the predicted step as a query to check whether it appears in the future, where we use both illustrative images and text descriptions under the same subgoal as the ground truth. These metrics are acting as classification-based metrics. In human evaluation, we check the future step correctness, which measures whether the generated step matches any of the future steps.
* (iv) We do not use the ending step to evaluate the inductive ability since script learning is originally designed to help robots to execute a task step by step. However, we agree that inductive ability should be evaluated and we use two metrics to check the inductive quality between the prediction and the ground truth in Section 3.3. The basic idea is to ignore the surface feature of the predicted text but to evaluate whether the predictions are semantically aligned with the ground truth. As a result, we assume BARTScore and Semantic similarity can capture such inductive quality to some extent. However, we agree with the reviewer that such metrics are not strong enough. We hope the current benchmark can lay a foundation for multimedia script learning, and as a future work, we further plan to introduce structured entity information and entity state tracking to capture the salient information in a step and rely on them to check whether the prediction step is aligned with the ground truth.   

Details: Pg 2, first paragraph, last sentence: 
Given a step, we first retrieve the step that is closely related, and then refer to its future step. In detail, step 2 got filtered out because of two reasons: (1) it is not similar to any of the retrieved historical relevant steps; (2) this step is far away from the target step. From previous training instances, such steps are usually down-weighted because early steps usually have less relevance to the target step. We have made it clear in the revision. 

In addition, thank you for pointing out the typos, and we fixed them in the revision.

**References**
1. Deitke, M., , ... & Wu, J. (2022). Retrospectives on the Embodied AI Workshop. arXiv preprint arXiv:2210.06849.
2. Ayan Sengupta, Amit Kumar, Sourabh Kumar Bhattacharjee, and Suman Roy. Gated Transformer for Robust De-noised Sequence-to-Sequence Modelling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3645–3657, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.309. 
3. Qing Lyu, Li Zhang, and Chris Callison-Burch. Goal-oriented script construction. In Proceedings of the 14th International Conference on Natural Language Generation, pp. 184–200, Aberdeen,Scotland, UK, August 2021. Association for Computational Linguistics.
4. Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. Machine Learning Repository, arXiv:2201.07207, 2022
5. Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Burch. Visual goal-step inference using wikiHow. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2167–2179, 
6. Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischedel, and Nanyun Peng. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4525–4542
7. Zhao, He, et al. "P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Response to Reviewer oNPU part1
comment: Thank you for your detailed and constructive comments. We are glad that you find our task novel. We have revised the paper and its supplementary accordingly. 

**A**
*“CLIP embedding instead of an information-losing text form”*

As shown in Table 3, we included a baseline called “CLIP-BART” (detailed in Section 4.1). It directly uses CLIP embeddings as the input rather than a text form, and we finetune them on the domain datasets for a fair comparison.  Results in Table 3 prove that  CLIP-BART underperforms the BART model, showing that CLIP features only hurt the generation performance rather than help. We suspect that the current multimodal encoder is not focusing on the information needed for task planning but more on the detailed visual features. Similar discoveries have been shown in [1] that modular settings achieve better performance than end-to-end settings. As a result, although mapping images into captions will result in some information loss, image captions share the same semantic space as input step texts. Moreover, the captions may help remove less relevant information from images. 

**B**

We adopt explicit masking since the task planning data is not as sufficient as the general text.  We have collected all related data regarding crafts and gardening domains, but each domain only contains 20-30k training samples, where the Transformer model may be unable to learn the correct masking behavior. Our results also show that explicit masks are helping.  It has also been proven in [2] that explicit maskings can help a Transformer model omit noise and focus more on the part of the input text that is of interest.We agree that the masking gate can be a task-specific embedding because the learnable masking embedding serves as a denoising vector.  

**C** We understand the reviewer’s concern about interchangeable steps. 

* (i) Following the traditional definition of script learning [3,4,5,6], we collect explicit steps listed in WikiHow, where the task procedure is formulated as an ordered sequence of steps. The generated script can be used as a sequence of commands for robots to execute tasks. As shown in Table 1, there are 2654 in the crafts domain, with six steps on average in each task, and 1857 tasks in the gardening domain, with three steps on average in each task. 
* (ii) 32% of subgoals have parallel steps by random checking 50 subgoals. We believe that the more strictly ordered tasks will present more challenges to script generation since they require a high level of accuracy, and there are no alternatives available.  In the domain of crafts and gardening, steps are normally sequential, and interchangeable steps are rare compared to the cooking domain.  For example, in Appendix I Figure 4, the system needs to first “identify shrubs” and then “rule out diseases”. 
* (iii) Yes, this is why multimodal can be helpful in planning a task more effectively. The visual history scenes can be informative for guiding the model to focus on salient objects. Figure 1 shows an example that is generated automatically by our model. Here, the order of steps cannot be interchanged.  The image of Step 1 shows the necessary items, which, thereby, will be focused on by the model in the following steps.  For example, a bead is shown in this image, and the model successfully predicts to leverage of the bead in our model's output, while BART is unable to do so. 
*(iv) Immediate next step refers to the step right after the previous given steps. We have made it clear in the revision (Appendix G). Thank you for pointing it out.
* (v) We introduce diversity control since the following steps should not overlap with the previous steps, so that the prediction can be informative. That is to say, the steps within a script should be diverse. Regardless of whether the steps can be interchanged, the script we generated should always be diverse steps, in order to present different information at each step. As a result, it is controlling diversity between steps, and it does not affect or relate to interchangeable steps. I understand your concern of interchangeable steps, which will affect the diversity between scripts (rather than steps). To accommodate interchangeable steps, we can ask the models to generate multiple alternative sequences, which is also adopted by [7].

From: ICLR.cc/2023/Conference/Paper3466/Authors
title: Response to Reviewer F6rc
comment: Thank you very much for your constructive comments. We are glad that you find our paper to be clearly written and have created a new exciting task. First, we add additional sample illustrative images into the new dataset in supplement material. We also fixed the misspelled words in section 2.2. We notice that the reviewer raises an interesting suggestion about the ablation of the evaluation of longer task sequences. We include the ablation studies for different history lengths in Appendix D and the quality of retrieved historical relevant steps in Appendix H. Finally, we refine the discussion sections with additional hypotheses stated before examples.

From: ICLR.cc/2023/Conference/Paper3466/Reviewer_F6rc
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The novel contributions of this paper include: a novel transformer contrastive learning based architecture with a new contrastive loss measure during training for improving goal-oriented next steps in task sequence generation, a new multi-modal retrieval based metric for evaluation for next task text generation using both images and narrative, and a new dataset for evaluation of multi-modal script learning with images and narratives of goals, subtasks and actions intrinsic to those subtasks.
strength_and_weaknesses: Strengths:

1. The works contributions are interesting and novel in the domain applied
2. The work is clearly written, results are well presented in comparison with modern SOTA techniques in tabular form and experiments are reproducible
3. Transitioning within sections is smooth and easy to follow

Weaknesses:

1. Lacking visuals of the actual new dataset tasks.
2. Mispelled word in 2.2
3. Could use ablation of evaluation of longer task sequences for goals
4. Discussions section is missing actual discussion as to why the authors hypothesis the results occurred with any suggestions
clarity,_quality,_novelty_and_reproducibility: This paper is well written, is clear and also shows novelty. Reproducibility of experiments also seems sufficiently good. I would recommend going further with ablations to evaluate longer sequences of tasks for given goals as some of the models used for comparison (e.g. GPT-2) do that.
summary_of_the_review: This paper is well written and the contributions appear to be novel and beneficial to the community. Technically it was interesting how the authors pulled together aspects of transformers with the contrastive learning loss measure and how they employed training with negative examples to improve performance.

I think the authors need to correct the mispelling on 2.2, add more in the discussions section and add an ablations section that compares generation of longer task sequences for goals.

This is a pass for me.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3466/Reviewer_oNPU
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper tries to predict the next step from a sequence of steps. Especially this work does not just rely on the previous steps in text form but also on the context of its matched image. Although the problem itself is interesting, it is unclear to the reader why their model choice is better,   what happens if some of the methods are not included,  or      how well this method works in more complicated tasks. 
strength_and_weaknesses: ### Strength
First of all, this paper is generally well-written and easy to follow. Also, well-prepared supplementary helps the reader to understand how the data is collected and what are the outputs from the models. Lastly, the symbols and equations in this paper are well-aligned and easy to understand with the figures.

### Weakness
Although it is easy to understand, I am unsure whether the method they took is well-aligned with the description in the text. Also, it is hard to estimate how this method would work when it learns from more extensive data with complex steps. Please check all details in the next section.
clarity,_quality,_novelty_and_reproducibility: (note: to prevent confusion, I would say gardening/crafts as Task/Goal and each step in the Task as a Subgoal when performing a Task. Please replace it with the preferred term)

### Major
#### Method
(A) In Image encoding of Section 2.3, the author introduces an image captioner and takes the caption as an additional input to the transformer. As the author is "concerned with the overall semantics of the salient objects," I think there is no reason not to take the most salient features from an object detector, for instance. Theoretically, when a model transforms data from one modality to a different modality, it should include more noise by default unless it is fully transferrable between them. In this context, I failed to find a reason why Image Encoding should have to be provided with more noise. Considering the use of the CLIP feature in the last paragraph of Page 6, I think this part is at least replaceable with a CLIP embedding instead of an information-losing text form. Without seeing a performance with such a setting, I believe it is hard to justify the reason for replacing context with a caption with more noise. 


(B) In Sections 2.3 and 2.4, the author describes EXPLICIT maskings ('ME' and 'RD' in the experiment table, if my understanding is correct). Specifically, the author introduces a sigma-weighted sum-based gate function for both, but I wonder why such explicit parts are needed. If the transformer behaves correctly, then the transformer should be able to learn such behavior (so, I guess the performance of removing one or both could be similar or better). Weight-visualization-based comparison with the performances of each may be required to sell the importance of such formulation. By the way, if my understanding is correct, the equations in these sections look like a task-specific embedding injection based on the gating function instead of simply masking because the \textbf{emb} looks like a learnable embedding.


#### Method + Problem setting
(C) Generally speaking, when there is a Task like making a coffee, there is a set of subgoals that can be performed in any order, like 'adding sugar or 'adding cream.' At the same time, there are some other types of steps that should have to be performed after such parallel steps, like 'stirring the coffee with a stick.'  Considering such characteristics in mind, (i) it is hard to see the explicit steps of each Task in this dataset, (ii) how many parallel steps this dataset has, and (iii) how well this "multimodal" approach can handle such cases. For instance, if there is no 'cream,' but only 'sugar' is visible in the previous set of images, then we can guess that the model may have a higher chance of predicting 'stirring' after 'adding sugar.' However, I failed to find any clue for all three (sub)questions. I also missed (iv) how the author defines the 'immediate next step' in Section 2.4 and (v) how the model should behave when multiple possible next steps exist, especially in the diversity-oriented contrastive loss (Section 2.6) context.


#### Evaluation settings
(D) Along with the problems in (C), it is hard to guess (i) how many explicit tasks exist and (ii) whether the data is enough to train large language models (with or without external noise like what I described in A). (iii) Especially for such a small number of average Subgoal (step) per sequence (around 3 for Gardening and 6 for Crafts, based on Table 1), thoroughly relying on language metrics could not be enough to measure the performance since each step may include a similar noun or verb. Instead, I think it would be better to show all possible steps in the dataset and present a classification-based performance as a side. Apart from the dataset side, (iv) I also think that the performance of predicting the "immediate next step" should be measured as inductive quality, aligned with (C), instead of "any subsequent step" as in the second paragraph of page 6. If we focus on the latter, we might value more for a model that can predict the very last step, although we want to measure the performance of predicting the "next step," not the "ending."



### Minor
#### Details
- (a). Pg 2, first paragraph, last sentence: It is tough to catch how the model knows which one is not closely related to the future steps without seeing any of the methods. Generally, this paragraph misses a lot of details, so I recommend adding a pointer, at least.

#### Typos
- (b) Pg 3, first paragraph, last sentence:  I think you can omit G, M from 'where G, M, S_{i} ...'.
- (c) Pg 3, second paragraph, first sentence: subgal -> subgoal
summary_of_the_review: Learning to predict the next step from a set of instructions is an important yet exciting problem to solve. However, I cannot see (1) why the model choices are the best choice and (2) how this approach would behave when we provide a more complex task like multiple subgoals can be performed in any order.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper3466/Reviewer_8Xq1
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a new script learning task to generate the next instructional step in natural language with both text histories and visual clues. The authors describe the multimedia, induction, and diversity challenges in this task. A model built upon BART with a selective multimedia encoder, a retrieval-augmented decoder, and a diversity-oriented contrastive learning objective is proposed to handle the challenges. A benchmark dataset is collected from WikiHow, and the proposed method is evaluated on the new benchmark.
strength_and_weaknesses: ###### Pros
- The new task setting is a reasonable extension of the existing script learning problem. Incorporating step-wise images makes the task more practical and potentially useful.
- Human evaluation results are provided, which complements the experimental results with objective evaluation metrics and makes the discussions more detailed.
- The ablation study is also quite detailed, and the extra discussion section is informative.
- The code is provided. The benchmark is also a plus.

###### Cons
- Though the task is described as "multimedia" script learning, the proposed method focuses mainly on NLP. For example:
  - The name of "multimedia challenge" does not correctly follow the definition described on Page 2. The challenge is more about learning from visual data rather than multimedia data.
  - Similarly, the name "multimedia encoder" is also not proper. After the very first image captioning step (which directly uses BLIP), most parts of the encoder are purely on the text domain.
- Diversity challenge and its solution is described well and is one crucial part of this paper. However, its connection to the performance of script learning is not well discussed nor evaluated, which impairs the contribution of proposing the topic of diversity.
- Compared with selective methods that are given script candidates, the proposed generative one is more flexible and practical. However, it seems to rely highly on (the size, quality, and relevancy of) the provided historical steps. The effects of the historically relevant steps should be discussed and analyzed.
- Most compared baselines do not take visual input at all, and the other few only use the images via generated captions. This may be an incomplete comparison of the proposed "Multimedia" Generative Script Learning. For example, T5 and Retrieval BART obtain comparable performance even without visual inputs and are expected to improve by simply using visual inputs.
- As mentioned in Sec. G.3, the evaluation metrics are more about language generation/retrieval but may not reflect the capacity of learning/inducting knowledge. Even the semantic similarity metrics may not be the best option, but this may leave a common question of script learning or even broader.
- Given too many metrics, the result numbers can not help draw a clear and consistent conclusion on the ablation studies. Besides, the reasons for the performance degradation need more discussion.
clarity,_quality,_novelty_and_reproducibility: - The writing is clear.
- Code is provided.

###### Minor Questions

- In Abstract, Line 3, the "however" is incongruous with the previous sentence.
- In Table 1, what is the definition of #Step and its differences from "Step len"? "Step len" seems to be #Task/#Pair.
- Fig. 2 is too general to illustrate the necessary details of the method (that are described in the text). Including more details in Fig. 2 or having another figure is suggested.
summary_of_the_review: Overall, the most impressive aspect of this paper is providing a new and potentially useful setting for script learning. The proposed method has some merits but is mostly composited with existing modules. The comparisons and evaluations could be improved.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold