Paper: PlaSma: Procedural Knowledge Models for Language-based Planning and Re-Planning (ICLR-2024)

Reviewer comment (Novelty):
Novelty: The main contribution of this work -- to train a small LM to imitate an LLM by using the LLM as a teacher to train the small LM, is not novel. It has already been demonstrated in [A] (and has not been cited here) for a variety of tasks including complex reasoning. If considered in the context of [A], the novelty here is limited to its extension to planning. The use of beam search-based planning is also very similar to [B], however, given its recency, I have discounted it in my evaluation.

Author response:
Thanks for pointing out Orca that we overlooked. Indeed there is a recent line of work on building general-purpose small models which warrant a discussion and reference. While our work shares similar spirit with Orca, we find the key distinction in (1) our goal is to develop a specialized small model for procedural/counterfactual planning and replanning with potential application to embodied domain, and (2) Orca is focused on learning from GPT-4 explanations (Chain of Thought) to improve models capabilities. Nonetheless, building specialized models on top of them can be explored in future works as we only worked on models that were accessible at the time of submission.

We will add this discussion in a new extended related work in the Appendix.

Thanks for bringing [B] to our attention. We believe such contemporaneous works indicate the timeliness of our contributions as they reflect the needs and interests of the research community.

Reviewer comment (Experiments):
Missing Key Experiments: While the authors motivate the use of knowledge distillation and compare their distilled models with that of the teacher model, the comparisons with the original (undistilled) model seem to be missing. Without this, it is hard to gauge the performance enhancement from distillation.

Author response:
Indeed we have tried to get an undistilled T5 model to work on the planning task. However, trying several prompt formats (such as “How to <goal>?”, Provide steps to achieve the <goal>”, etc.), we couldn’t get any meaningful outputs from this baseline to run human evaluation on or apply guided decoding. Most outputs were just repeating the given inputs or empty strings. However, we successfully ran a few-shot variant of undistilled T5 (with some post-processing to format the outputs). Below is the human evaluation result for this baseline on the same 250 randomly sampled test examples (Curie and PlaSma scores are compiled from Table 1):

|                         | Coverage | Order | Overall |
|-------------------------|----------|-------|---------|
| Fewshot T5 (undistilled) | 2.63     | 3.05  | 2.63    |
| Fewshot Curie (teacher) | 3.75     | 4.27  | 3.75    |
| PlaSma (best)           | 4.53     | 4.77  | 4.58    |

Results of Few-shot undistilled T5 is significantly behind those of fewshot Curie (teacher) and all our model variants.

Reviewer comment (Experiments):
The dataset ProScript is not well explained, making it hard to qualify the complexity of the instructions.

Author response:
Proscript is a manually-curated dataset of ~6k scripts at varying levels of granularity, for a wide range of everyday activities (e.g., `bake a cake`, `audition for a musical`, `go sailing`, etc.). Given the diversity and granularity of goals in ProScript, we believe it provides a good venue to study procedural knowledge in language models available through distillation.

Table 2  includes both evaluation on the ProScript as well as our collected CoPlan dataset.  The goal is to investigate the utility of CoPlan that is obtained through “generation from an LLM” in the presence of manually-curated ProScript.

Reviewer comment (Evaluation):
The results in Table 2 are hard to understand because we don’t know about the relative complexity of the task and the diversity of tasks.

Author response:
1. **Lexical diversity**: We use generally accepted measures [1] to analyze the diversity of datasets. We compute 1/2/3-gram entropy and the mean segmented token type ratio (MSTTR). To establish a comparison, we compute these values for three other datasets: XSUM (extreme summarization of news articles), DialogSum (real-life scenario dialogue summarization), and TinyStories. These have been specifically picked as they are stylistically different from our goal and script setup, and they often contain longer more natural sentences. We observe that even though the goals and steps in our dataset are shorter, the overall lexical diversity of Proscript and Coplan are comparable with other datasets. XSUM displays a higher MSTTR score, but this is likely attributed to the characteristics of news and more formal text. We also note that the machine-generated CoPlan exhibits slightly higher lexical diversity than the human-written proScript.
2. **Perplexity of an LM**: We also report the perplexity of an off-the-shelf language model which measures the degree of uncertainty (surprise) of an LM when it generates the next tokens. The higher the perplexity, the more surprised the LM is. As we see from the last column, with the exception of XSUM, the remaining datasets exhibit comparable perplexity scores. The higher perplexity in the case of XSUM is, again, attributed to its association with the news domain, a distinct characteristic compared to the other datasets, which predominantly encompass everyday scenarios. Also, intuitively LMs generally have lower perplexity scores on machine-generated data (as seen for CoPlan vs proScript).


| Dataset     |   H1  |   H2  |   H3  | MSTTR | Perplexity (GPT2-med) |
|-------------|:-----:|:-----:|:-----:|:-----:|:--------------------:|
| XSUM        | 10.45 | 15.49 | 17.15 |  0.80 |         18.55        |
| DialogSum   |  9.04 | 14.38 | 16.73 |  0.66 |         13.20        |
| TinyStories |  8.65 | 14.25 | 17.80 |  0.61 |         7.53         |
| ProScript   |  8.86 | 13.67 | 15.61 |  0.59 |         12.93        |
| CoPlan      |  9.27 | 15.00 | 18.05 |  0.64 |         10.05        |


[1] The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (Gehrmann et al., GEM 2021)

Reviewer comment (Novelty):
The tone of the phrase “we introduce the task of counterfactual planning” should be reduced. A quick search in Google Scholar for “plan revision” shows existing related work, such as:  
- Ow, P. S., Smith, S. F., & Thiriez, A. Reactive plan revision. AAAI 1998.  
- Williams, K., & Burdick, J. Multi-robot boundary coverage with plan revision. In Proceedings 2006 IEEE International Conference on Robotics and Automation.

Author response:
While plan revision has been explored in classical AI (with restricted action vocabularies) and Robotics, here we are investigating counterfactual and constrained (re)planning capabilities of contemporary language-based agents. As LMs are gaining more attention in robotics and physically embodied application, it becomes imperative to enhance their capabilities by integrating counterfactual planning and contained planning mechanisms.

We thank the reviewer for bringing this to our attention, and we will revise this phrase accordingly in our updated version.

Reviewer comment (Theory):
Abuse of some terms — the “step verifier” is not actually verifying, but adding a bias. For instance, the paper mentions that in the case of embodied agents, verification can be taken over by a safety module. In that scenario, with reasonable α that follows the LLM when the so-called verification is not providing anything relevant, the aggregation in Eq (3) cannot prevent catastrophic errors that are considered attractive by the LLM. Perhaps a better name would be “quality bias” or something explicitly indicating bias.

Author response:
We agree with the reviewer. While during training the verifier acts as a binary classifier to identify the validity of a next-step, we are using this as a bias/regularizer term during decoding the plan (Eq. 3). Thank you for pointing this out. We will update our paper to reflect this.

Reviewer comment (Writing):
Abuse of some terms — the same issue applies to the notion of “symbolic knowledge distillation”. It is problematic that in AI we associate natural language with symbolic, as the term "symbolic" is overloaded with a large body of work in AI ranging from logic to graphical models. It should be made clearer by using a term like “instruction distillation”.

Author response:
We totally agree with the reviewer that “symbolic” became an overloaded word in AI. However, we use this term in reference  to the previous work (West et al. 2020, “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”).

Reviewer comment (Evaluation):
Although human evaluations were conducted, the executability conditions for the plans in the domains used in these experiments seem to be loose. It would be beneficial to evaluate the models in domains which have hard executability conditions (like the domains used in International Planning Competitions), where the correctness can be objectively determined, to more accurately gauge the language planning abilities of the proposed method.

Author response:
- We indeed conducted an extrinsic evaluation in a domain with hard executability conditions, i.e., **VirtualHome (Section 3.3)**. This also helps to investigate the application of PlaSma and its generalization to the embodied agent planning domain. Given our strong results in the VirtualHome environment, we expect that PlaSma can serve as a strong foundation model and be easily transferred to other embodied domains with minimal domain adaptation.
- On the general domain (i.e., on CoPlan); however, we follow the common practice of Likert scale human evaluation [1-3] across multiple dimensions. However, we made sure these dimensions are aligned with plan ability in an embodied domain. For example, **temporal ordering** is related to **executability** (e.g., `grasping` should be done before `picking up`), and **coverage/completeness** is related to success/correctness. 

We hope this clarifies the reviewer concern regarding our evaluation.

[1] Recent Advances in Neural Text Generation: A Task-Agnostic Survey. (Tang et al., 2023)

[2] Human evaluation of automatically generated text: Current trends and best practice guidelines (Van Der Lee et al., CSL 2021)

[3] The use of rating and Likert scales in Natural Language Generation human evaluation tasks: A review and some recommendations (Amidei et al. iNLG 2019)

Reviewer comment (Experiments):
A comparison with GPT-4, in addition to GPT-3, could provide additional insights into the effectiveness of the method.

Author response:
Our data was collected using GPT-3 Curie (smaller and less powerful than GPT-4) as the teacher due to cost constraints. We do expect to get better results for our distillation if using GPT-4 as our teacher. Nonetheless, as suggested by the reviewer, we ran a human evaluation comparing GPT-4, GPT-3.5, our best PlaSma model and its teacher on 50 instances (total of 200):

|                         | Coverage | Order | Overall |
|-------------------------|----------|-------|---------|
| Fewshot Curie (teacher) | 3.53     | 4.37  | 3.58    |
| PlaSma (best)           | 4.31     | 4.68  | 4.23    |
| GPT-3.5 (Davinci-003)   | 4.78     | 4.84  | 4.81    |
| GPT-4                   | 4.78     | 5.00     | 4.81    |

As we observe, the trend remains the same, with GPT-4 surpassing its predecessor (GPT-3.5) only in the `ordering` dimension. We will add this comparison in the **Appenidx F** of our updated pdf.

Reviewer comment (Evaluation):
The potential for increased bias due to the distillation from larger language models is mentioned in the limitations section but remains a concern.

Author response:
We acknowledge potential biases, and  have discussed  them in the limitations section, warranting the need for further investigation in  future research.

Reviewer comment (Novelty):
The LLM-to-Planning-Model teacher-student paradigm for planning is not well motivated. Cost, performance (from specialization), controllable procedure, better-integration with downstream tasks (e.g. decoding/execution) and so on? It is more about better understanding of the key capabilities of existing techniques and combining them to solve the critical problems. For example, if it is more about specializing common knowledge embedded in LLMs to do planning, then smaller LLM might not be the right solution — the same proposed paradigm can be combined with LLM of the same size or even larger LLMs for superior planning capabilities. What are the real problems and the corresponding means could be better sorted out?

Author response:
Finetuning LLMs requires updating models’ parameters which is not only costly but often inaccessible for the broader community. To clarify, our goal is not to make superior large models, but to distill procedural knowledge of LLMs into smaller models.
Reducing the scale and cost of strong models via teacher-distillation is the key in developing open-sourced LMs that are accessible to all, facilitating fine-tuning and seamless adaptation to various domains and custom use cases. Given the large-scale training dataset used for PlaSma, we hope it can serve as a foundation model that can be quickly adapted to specific domains with minimal additional annotation (like we demonstrate by adapting Plasma to VirtualHome).
Moreover, we could not augment most of the LLMs with our decoding-time algorithm due to limited access to the model's log probabilities.

Thanks for the comment. We will update our pdf to include this.

Reviewer comment (Novelty):
The true contribution and its relevancy might be hidden in the paper title and the current way of writing. The proposal is composed of three parts: (1) planning data generation from LLMs with human-in-loop curation, (2) teacher-student distillation training, (3) language model decoding with step verifier. There is less text regarding teacher-student distillation. This might indicate that the teacher-distillation importance is over-estimated. With the planning data generation and verifier-guided decoding generation, there might be other ways to enhance planning abilities, e.g. finetuning the original LLMs to specialize in planning domains. If the distillation step is an important component in the ingredients, please detail it and discuss more.

Author response:
Finetuning LLMs requires updating models’ parameters which is not only costly but often inaccessible for the broader community. To clarify, our goal is not to make superior large models, but to distill procedural knowledge of LLMs into smaller models.
Reducing the scale and cost of strong models via teacher-distillation is the key in developing open-sourced LMs that are accessible to all, facilitating fine-tuning and seamless adaptation to various domains and custom use cases. Given the large-scale training dataset used for PlaSma, we hope it can serve as a foundation model that can be quickly adapted to specific domains with minimal additional annotation (like we demonstrate by adapting Plasma to VirtualHome).
Moreover, we could not augment most of the LLMs with our decoding-time algorithm due to limited access to the model's log probabilities.

Thanks for the comment. We will update our pdf to include this.

Reviewer comment (Writing):
There are good ideas within the paper, but the writing could be improved to make these good ideas clear and stand out. For example, how to train the step-verifier from human-written plans along with more formal analysis of the impact of the step-verifier.

Author response:
We included implementation details of the verifier in **Appendix B.3** due to space limit. We will move these materials to the main text as we recognize the potential inconvenience: 

- For training, we reuse 3k human-written plans from the existing ProScript dataset [2] to automatically create 47K positive and negative pairs of (plan-so-far, next-step). 
- Our perturbation strategies to generate pseudo-negative examples include: **(1) Reordered steps**: Conflicting logical order results from inaccurate causal or temporal dependencies in a plan. Thus, we apply both near and distant reordering by randomly reordering two consecutive and two distant steps. **(2) Repetitive steps**: Degeneration i.e., generating repetitive text is commonly observed in language models (also observed in the planning domain [1]). Similarly, we include both near and distant repetition by repeating the immediate previous step and distant previous step as a pseudo-negative next-step. **(3) Missing steps**: Another common mistake made by language models is missing necessary steps, leading to (semantically) incoherent plans. To simulate this behavior, we randomly select a non-immediate step as a pseudo-negative next-step. Below we show symbolic examples of perturbed pairs:

Goal: g

Oracle Plan (steps): s1 s2 s3 s4 s5


We have the following perturbations: 

Reordering: (g + s1 + s3, s2) → 0

Repetitive: (g + s1 + s2 + s3,  s2)  → 0

Missing: (g + s1 + s2, s4) → 0


Note that our perturbation strategies follow general rules applicable to any new domain with some task-specific adaptation.

- Using the constructed positive and negative pairs of (plan-so-far, next-step), the verifier is trained using the standard binary Cross Entropy loss to identify the validity of the candidate next-step. It achieved an F1 score of 78% on a held-out test set.

### Re formal analysis of impact of the step-verifier:
In Table 1, we include several ablations to show the contribution of multitasking, **step-verifier**, and scale. For example, the comparison between PlaSma and PlaSma+ across different model sizes shows non-negligible performance boosts as a result of the step verifier in our proposed tree-based decoding.

Is the reviewer seeking a specific formal analysis? Thanks.

Reviewer comment (Theory):
Please define the loss functions formally with teacher-student distillation and verifier-training.

Author response:
All models are trained with the standard Cross Entropy (CE) loss function with an early stop based on the validation loss. We will update our manuscript soon to include the loss used for distillation and verifier training.

Reviewer comment (Theory):
For the step verifier, "we design perturbations … ordering, semantic completeness, topicality and fluency", please provide detailed analysis of these data-side steps regarding their intuition and formal properties if possible. How does a single verifier score reflect all these criteria? Any special design to achieve them with a simple RoBERTa-based classifier?

Author response:
We included implementation details of the verifier in **Appendix B.3** due to space limit. We will move these materials to the main text as we recognize the potential inconvenience: 

- For training, we reuse 3k human-written plans from the existing ProScript dataset [2] to automatically create 47K positive and negative pairs of (plan-so-far, next-step). 
- Our perturbation strategies to generate pseudo-negative examples include: **(1) Reordered steps**: Conflicting logical order results from inaccurate causal or temporal dependencies in a plan. Thus, we apply both near and distant reordering by randomly reordering two consecutive and two distant steps. **(2) Repetitive steps**: Degeneration i.e., generating repetitive text is commonly observed in language models (also observed in the planning domain [1]). Similarly, we include both near and distant repetition by repeating the immediate previous step and distant previous step as a pseudo-negative next-step. **(3) Missing steps**: Another common mistake made by language models is missing necessary steps, leading to (semantically) incoherent plans. To simulate this behavior, we randomly select a non-immediate step as a pseudo-negative next-step. Below we show symbolic examples of perturbed pairs:

Goal: g

Oracle Plan (steps): s1 s2 s3 s4 s5


We have the following perturbations: 

Reordering: (g + s1 + s3, s2) → 0

Repetitive: (g + s1 + s2 + s3,  s2)  → 0

Missing: (g + s1 + s2, s4) → 0


Note that our perturbation strategies follow general rules applicable to any new domain with some task-specific adaptation.

- Using the constructed positive and negative pairs of (plan-so-far, next-step), the verifier is trained using the standard binary Cross Entropy loss to identify the validity of the candidate next-step. It achieved an F1 score of 78% on a held-out test set.

Reviewer comment (Evaluation):
Regarding the evaluation metrics, please provide more details of the AMT human steps. Are coverage, order, and overall quality complete to evaluate a plan? Any comparison or correlation on the human evaluation metrics and the BLEU numbers and the Embodied Environment's metrics? If not well-correlated, any proposal on automatically evaluating plans? Also, how to relate and align human evaluation, BLEU-style sequence matching metrics, embodied environment testable metrics, and real-world execution measurable metrics?

Author response:
### Q4.1 Are human evaluation dimensions enough?
Given the open-ended nature of CoPlan goals having varying levels of actionability (not all of them are grounded), we follow the common practice of Likert human evaluation across multiple dimensions [1-3]. Notably, we make sure we evaluate plans on dimensions that are related to actual plan ability in an embodied domain. For example, temporal ordering is related to executability (e.g., grasping should be done before picking up), and coverage/completeness is related to success/correctness. Nonetheless, we believe that extrinsic evaluation is crucial; for this reason, we use **VirtualHome** to investigate the application of PlaSma and its generalization to the embodied agent with hard executability **(Section 3.3)**. Given our strong VH results, we expect that PlaSma can serve as a strong foundation model and be easily transferred to other embodied domains with minimal domain adaptation.

### Q4.2. Correlation between human and BLEU scores: 
We indeed compute the correlation between BLEU metric and human scores. We find that BLEU has very weak correlations to human scores of coverage, ordering an overall quality, with a **Pearson correlation of 7.7%, 5.9%, and 5.6%**. This verifies the fact that n-gram-based metrics may not provide an informative measure of performance (as also suggested by previous works [3,4])

### Q4.3: Proposal for better automatic evaluation:
We appreciate the interesting questions raised by the reviewer regarding human and automatic evaluation alignment. We also acknowledge the need for better automatic evaluation of plans, ideally those that take into account the causal, temporal, and completeness of steps towards achieving a goal. However, these are out of the scope of the current project and we hope future works can explore this direction.

[1] Huang et al. “Language models as zeroshot planners: Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agent.” ICML 2022.

[2] Sakaguchi et al. “proScript: Partially ordered scripts generation”. In Findings of the Association for Computational Linguistics: EMNLP 2021.

[3] Novikova et al. “Why We Need New Evaluation Metrics for NLG” EMNLP 2017

[4] Celikyilmaz et al. “Evaluation of Text Generation: A Survey“