PARADISE : Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset (2024)

Abstract: Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q&A format on practical procedural text sourced from wikiHow. It involves tip and warning inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://anonymous.4open.science/r/paradise-53BD/README.md.

## introduction recent breakthroughs in emergent (or lack of) abilities of large language models (llms) have given rise to empirical studies that employ language models as planners @xcite figure 1 : a procedural tutorial on "removing ink stains from fabric". here, one can damage the fabric if they ignore the warning "always blot, never rub, when dealing with ink stains". 2023; song et al., 2023) (i.e., agentic models) and analysis studies that investigate their planning capabilities @xcite @xcite . majority of these studies use toy simulation environments like al-fred @xcite , blocksworld, and virtualhome @xcite which have little lexical and domain variance and limited number of actions (e.g., verbs). additionally, the planning task is mostly formulated as a generation problem which can only be evaluated on the closed problem domain; and models with decoder-based architectures. hence evaluating open-domain planning abilities for a wide range of models still remain as a challenge. planning requires a combination of a wide range of complex reasoning abilities. one line of research focuses on distinct set of reasoning abilities (e.g., commonsense @xcite , arithmetic @xcite , logical @xcite , temporal @xcite etc...) of language models on more realistic, open-domain text. however, such open-domain text mostly does not contain a goal or a plan, hence lack the complex linguistic phenomenon (e.g., implicit relations, complex temporal, and coreference links, large event-cause chains etc...) that is common in procedural text. on the other hand, existing studies that utilize plans as a test bed for reasoning are mostly limited to extracting direct and explicit relations @xcite , i.e., more simplistic reasoning compared to implicit reasoning. here, we formulate the implicit planning skills as abductive reasoning skills over the procedural plans with missing information. we hypothesize that a model with planning abilities would be able to infer the warnings and tips about the plan without seeing the explicit instructions (see fig 1 ). although abductive reasoning has been studied in the past, they either employ additional source of information @xcite or focus on consecutive elements @xcite @xcite , both of which diminish the notion of implicity. in this work, we present paradise, an extensive, expert-curated dataset for warning and tip inference tasks covering a wide range of domain derived from wikihow foot_0 . unlike previous works, our tasks focus on the implicit relationship between goals and warnings/tips, bypassing intermediate steps @xcite . this requires a model to possess implicit knowledge of intermediate steps (i.e., the plan) solely based on the provided goal in the absence of explicit instructions. furthermore, we use a question answering formulation, which allows for easier evaluation with standard metrics, and testing of broader-range of model architectures including encoder-based ones. we establish robust baselines by fine-tuning pretrained language models like de-berta @xcite and zero-shot prompting with a varied set of large language models such as mistral-7b @xcite and @xcite . our extensive experiments address a broad range of research questions, delving into the relationship between memorization and performance ( §4.1), the differences in failures between plms and llms ( §4.2), and the knowledge transfer capacity of the proposed tasks to unseen tasks ( §4.5). we observe that fine-tuning small models tailored to specific tasks proves more effective than zero-shot prompting across all llms, including gpt-4. however, it's noteworthy that all models, despite these efforts, still lag behind human performance. our exhaustive analysis also provide interesting insights such as large models getting less affected from dropping matched keywords; bert-family struggling more with physical goals, while gpt-4 struggling with abstract, digital and social objectives; and proposed tasks providing beneficial prior knowledge to unseen procedural tasks. we release all the resources at https://github.com/gglab-ku/paradise . ## paradise initially, we augment the wikihow corpus @xcite ) by integrating it with a recent compilation foot_1 of 21k tutorials. the extended corpus maintains the json format, except for the segregation of warnings and tips, an example of which can be seen in appendix a. each wikihow tutorial comprises procedural steps to achieve its objective, with some tutorials featuring step-specific or general warnings and tips. we automatically generate downstream task data incorporating these warnings and tips, as elaborated in sec. 2.2. the process also involves expert human annotation, detailed in sec. 2.3. ## experimental setup to evaluate language models in our tasks, we establish two setups: 1) finetuning setup for pretrained encoder lms such as bert @xcite and 2) zero-shot setup for large language models such as gpt-4 (openai, 2023b). ## experiments and results we experiment with the plms and llms on par-adise using the setup explained in sec 3. we also calculate the random and majority baselines, and evaluate the human performance by averaging the accuracy of two human annotators 7 on a random set of 100 examples. our main results are given in fine-tuning among the fine-tuned models, de-berta performs the best in both tasks; yet, it still falls behind human performance. considering de-berta's performance in previous abductive reasoning datasets, such as cosmosqa @xcite , swag @xcite , and hellaswag @xcite , such results indicate that its success is not task-specific and its performance is transferable to warning and tip inference. distil-bert, bert, and roberta cannot perform on par with deberta and fall well behind human performance, although they perform considerably well on the proposed tasks. all models perform better in tip inference compared to warning inference. zero-shot although their performances are worse than fine-tuned plms, proprietary llms perform considerably better than open-source llms. gpt-4, which also tops many benchmarks (openai, 2023b), is the best-performing llm, while palm-2 is a close runner-up. one surprising finding is that the performances of open-source llms are not correlated with their size, as mistral 7b outperforms vicuna 33b and llama-2 70b in both tasks. similar to plms, llms also perform better in tip inference compared to warning inference, with llama-2 70b being an exception. to gain further insights on behaviours of the models we ask the following research questions: ## related work commonsense reasoning is a broad domain that branches into a wide range of subdomains such as linguistic reasoning @xcite @xcite , abductive reasoning @xcite , reasoning about the physical world @xcite @xcite , temporal reasoning @xcite @xcite . although there exists some abductive reasoning tasks that involve finding the most likely explanation for a set of incomplete observations (bhargava and ng, 2022), they are outside of the domain of procedural language understanding. for example, cosmosqa @xcite present commonsense abductive reading comprehension and art @xcite propose abductive natural language inference with narrative contexts. thus, they depend on additional paragraphs provided by the question, which enrich the level of information provided to the model. furthermore, abductive reasoning resources obtained from procedural texts are either artificially made difficult with targeted models (such as swag @xcite and hellaswag @xcite ) or covers another form of procedural texts (e.g., natural phenomenons of wiqa @xcite ). moreover, they focus on the continuous elements (i.e. consecutive steps or events), which we believe diminishes the degree of implicity. furthermore our main resource wikihow has been extensively used for a wide range of tasks thanks to its rich body of well-structured procedural documents, including but not limited to summarization @xcite , intent detection @xcite , reasoning @xcite , linking actions @xcite , and next event prediction @xcite @xcite . ## conclusion there has been a growing interest in procedural data, tasks, and reasoning. however, the spotlight has been on explicit and direct relations when studying reasoning within procedural documents. to address the the lack of resources to study implicit relations and reasoning, we introduce paradise and strong baseline models evaluated and analyzed with extensive experiments. paradise contains +104k warnings and tips in total and serves as a reliable testbed for the evaluation of abductive and implicit commonsense reasoning skills of language models. moreover, it brings improvement to zero-and-few-shot performances in out-of-domain procedural tasks. our experiments reveal that plms do not possess inherent reasoning skills; yet, most of them outperform llms when fine-tuned. however, best-performing models from both groups fall behind human performance, indicating room for improvement. we release all the resources publicly to further research.