A ct P lan-1 K : Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities (2024)

Abstract: Large language models(LLMs) have been adopted to process textual task description and accomplish procedural planning in embodied AI tasks because of their powerful reasoning ability. However, there is still lack of study on how vision language models(VLMs) behave when multi-modal task inputs are considered. Counterfactual planning that evaluates the model’s reasoning ability over alternative task situations are also under exploited. In order to evaluate the planning ability of both multi-modal and counterfactual aspects, we propose ActPlan-1K. ActPlan-1K is a multi-modal planning benchmark constructed based on ChatGPT and household activity simulator iGibson2. The benchmark consists of 153 activities and 1,187 instances. Each instance describing one activity has a natural language task description and multiple environment images from the simulator. The gold plan of each instance is action sequences over the objects in provided scenes. Both the correctness and commonsense satisfaction are evaluated on typical VLMs. It turns out that current VLMs are still struggling at generating human-level procedural plans for both normal activities and counterfactual activities. We further provide automatic evaluation metrics by finetuning over BLEURT model to facilitate future research on our benchmark.

## introduction recent researches adopt large language model (llm) or multi-modal large language model (mllm) as agents to accomplish embodied ai tasks @xcite @xcite @xcite @xcite . the llms generate procedural plan according to textual descriptions or embodied observations. the procedural plan consists of action sequences that needs to achieve in order to finish the task. generate admissible procedural plans for assembling gift baskets. there are four baskets on the floor, cookies, cheeses, and bows on the tables. the goal is to place one of each item inside on of the baskets as shown in the images. ## actplan-1k benchmark actplan-1k contains multi-modal planning instances both of normal activities and counterfactual activities, with same instance form in section 3.1. the instance collection process of them is also same with bddl definition and illustrated in section 3.2. details of constructing counterfactual activities are introduced in section 3.3. the benchmark statistics is presented in section 3.4. ## evaluation metrics evaluation metrics over vlm generated procedural plan p * and gold plan p have two types: human evaluation and automatic evaluation. ## conclusion we present actplan-1k, a multi-modal counterfactual household planning dataset, to evaluate both the procedural planning and counterfactual planning ability of vision language models. our dataset constructed based on chatgpt and household simulators consists of 1.2k instances covering normal and counterfactual activities. evaluation on typical vlms shows that actplan-1k is still challenging for the models.