P izza C ommon S ense: A Dataset for Commonsense Reasoning about Intermediate Steps in Cooking Recipes (2024)

Abstract: Understanding procedural texts, such as cooking recipes, is essential for enabling machines to follow instructions and reason about tasks, a key aspect of intelligent reasoning. In cooking, these instructions can be interpreted as a series of modifications to a food preparation.For a model to effectively reason about cooking recipes, it must accurately discern and understand the inputs and outputs of intermediate steps within the recipe.We present a new corpus of cooking recipes enriched with descriptions of intermediate steps that describe the input and output for each step. PizzaCommonsense serves as a benchmark for the reasoning capabilities of LLMs because it demands rigorous explicit input-output descriptions to demonstrate the acquisition of implicit commonsense knowledge, which is unlikely to beeasily memorized. GPT-4 achieves only 26% human-evaluated preference for generations, leaving room for future improvements.

## introduction procedural text is a type of writing that provides instructions on how to perform a task using resources to achieve a final goal. common real-world examples include scientific articles, diy instruction books, or cooking recipes @xcite . in the latter case, the procedural text instructs an agent on how to prepare a dish. to understand and follow a recipe, one must be able to reason about the steps involved and the effects of each step on the ingredients. this requires common sense knowledge about cooking, such as knowing how different cooking techniques and food properties affect the final product. humans can easily imagine the effects of each step in a recipe as they read it, even if they have never prepared the dish before. they can also use commonsense to reason about the recipe, the purpose of the action in the instruction, as well as the input and output of the cooking step, determining what comestibles are necessary for performing a specific step, predicting and understanding the effects of performing a cooking action, giving explanations about the undertaken actions, identifying alternative orderings, and adapting to the cooking conditions. with the advent of increasingly capable artificial tools, such as large language models (llms) comes the need to investigate their commonsense reasoning abilities in following procedural text such as cooking recipes. inspired by the recent line of work of prompting llms @xcite @xcite to generate a reasoning chain along with the answer, with the goal of mimicking the human reasoning process, we argue the need to evaluate the correctness of the intermediate reasoning steps. for this reason, we propose pizzacommonsense, a dataset for commonsense reasoning about inter-mediate and implicit steps for cooking recipes. a visual representation of the purpose of the data set is given in figure 1 . this dataset contains pizza recipes that are parsed into atomic cooking steps such that each step contains only one cooking action. the recipes are organized in a tabular format with four different columns, the cooking instruction, the input preparation, the relevant cooking action and the output preparation. the task is at the interface between question answering (qa) and natural language inference. given the set of instructions of a recipe, models are required to predict a description for the input and output preparations for each cooking step. this is particularly challenging because models are required to reason and predict accurate descriptions of the intermediate steps. the intermediate steps of a cooking recipe are the steps that are performed after the initial preparation and before the final plating and presentation of the dish. these steps are typically where the main cooking and transformation of the ingredients take place. concretely, the task involves individuating the (i) explicit input comestibles (e.g. "sauté the onion in the skillet") and (ii) output comestibles (e.g. "put the tomato sauce in a bowl"). natural language makes often use of omissions and anaphoras which a good model should be able to resolve by identifying (iii) implicit input comestibles (e.g. "season the sauce to taste" with the implicit input being "seasoning" while "sauce" is explicit) and implicit output comestibles (e.g. "mix the flour, water, salt and yeast" where the implicit output is "dough"). we propose baseline methods to solve the task: t5 @xcite with fine tuning; flan-t5 @xcite with prompting and fine tuning, gpt-3.5 with demonstrations, fine-tuned gpt-3.5 and gpt-4 with chain-of-thought (cot) prompts. the contributions of this work are threefold: (1) we propose a new task for procedural text comprehension, namely predicting the input and output of a given action while giving self-contained descriptions of the transformed resources for each step of the procedural text; (2) we construct an annotated dataset to facilitate the studies of commonsense reasoning for procedural texts with the dataset being made publicly available; and (3) we benchmark the performance of state-of-the-art generative language models on our dataset and demonstrate the difficulty of the task. ## motivation commonsense reasoning is central to human intelligence. it is essential for humans to operate in the real-world. in ai, we lack a full understanding of commonsense reasoning or the means to simulate it. by creating datasets that reflect domains suitable for investigating commonsense reasoning, we can explore modeling techniques. cooking, for instance, is a domain where commonsense reasoning is vital. consider developing a robot to cook; it needs to deeply understand recipe instructions, which involves interpreting ambiguous language, handling missing information, and resolving references common in procedural texts. to teach a robot to cook effectively, it must understand recipes deeply, including ingredients, final products, and intermediate states. recognizing these intermediates and their properties is crucial for appropriate actions-for instance, suppose a robot is using a pizza recipe and it reads the instruction to mix 20oz of flour with 10oz of water, it needs to use commonsense reasoning that the result is dough (often this is not explicitly stated), and then inferring that it is dough, the robot can decide that it can move this intermediate comestible around the kitchen by hand. later in the recipe, suppose the robot reads that it needs to put 10oz of tomatoes into the blender. in this case, the robot needs to use commonsense reasoning to determine that the intermediate comestible will be tomato puree, and that it cannot move it in its hands, but rather requires a receptacle. achieving human-level cooking requires an agent to understand nuances of each step and anticipate outcomes, translating ambiguous recipe language into precise instructions. our dataset uniquely provides annotated recipe instructions, labeling intermediate input and output comestibles. by focusing on the implicit transformations between these elements, we aim to bridge a critical gap in research and develop models that can truly understand and execute complex cooking tasks. this can then enhance the capabilities of autonomous agents in various roles, including robotic cooking and virtual assistants. ## pizzacommonsense why pizza recipes? the choice of focusing on pizza recipes stemmed from their inherent compositional nature within a controlled setting. in fact, pizza preparations typically involves a lim-ited set of steps: creating the base, spreading a sauce, adding toppings, and baking/chilling/setting. this breakdown makes pizza an ideal candidate for studying the compositional aspects of recipe instructions. each stage (base, sauce, toppings) acts as a distinct "building block", allowing for easier segmentation and analysis of the construction and reasoning process. recipes often provide high-level instructions that necessitate deconstruction into precise, lowlevel actions. this process is significantly influenced by the characteristics of intermediates. while our dataset focuses on pizza recipes, we believe that methods developed for commonsense reasoning with intermediates can be generalized to other cooking recipes, and then to other situations involving actions and intermediates. ## task we extend procedural text comprehension by additionally identifying each instruction step's intermediate input and output. this involves predicting both the input and output for each instruction, including implicit and explicit ingredients, action outcomes, and each step's resulting condition. we posit that this capability will generate faithful recipe variations and enhance reasoning about the recipe's structure and components. problem formalization we have a tabular representation of a recipe r with @xmath0, input, action, output), and m rows, one for each instruction of the recipe. additionally, (i) the output cell of the row t -1 can be the content of the input cell of the row t and (ii) the input and output cells of a row t can be the same, if the cooking action is not transformative. the goal is to predict the content of the columns input and output, given the content of instruction and action. ## experimental setup parameters we use t5-base and flan t5-base models with a batch size of 16 and a learning rate of 1e -5 , using adam optimizer (kingma and welling, 2013). we fine-tune for 30 epochs and save the checkpoint with the best performance on the validation set. both t5 based models are pretrained in english. both t5 and flan t5 employ around an hour on 4 gtx gpus. all values are the averaged across 3 runs. for the llm-based methods, we use gpt-3.5-turbo-1106 and gpt-4-turbo models with temperature set at 0. the cost of finetuning the gpt-3.5 was 15 usd. ## results we analyze the performance of the baseline models in predicting the correct input and output for the intermediate steps of a cooking recipe. to perform well, the models should predict (i) only comestibles in the input/output pairs; (ii) all the comestibles necessary to perform the cooking action at the given time step, implicitly and explicitly stated in the cooking instruction. in particular, it should not fail to include the output of the previous step if the next cooking instruction is a transformation of the comestible; (iii) the comestibles implicitly mentioned could also be inferred by the cooking action, i.e. the input associated with salt the sauce should be (salt; sauce) and finally (iv) specific descriptions of the output comestibles, i.e. although not strictly wrong, describing a sauce as a mixture of tomato paste, oil, water and seasonings is undesirable. t5 and flan-t5 the performance analysis of t5-based methods, summarized in gpt models gpt-3.5 in 1-shot learning achieves a 22.3% an ema score on our test set, while t5 and flan t5 score 12.9% and 13.8%, respectively. the input rouge l score is 36.9. a closer inspection reveals that while inputs and outputs are semantically correct, the models often fail to follow the set rules, such as wrongly including tools or locations, or misinterpreting instructions like "dissolve yeast in water". however, the fine-tuned gpt-3.5 corrects some of these issues, achieving a 32.6% exact match score and the highest bertscore at 90.6. gpt-4+cot processing has a 26.7% ema score, with improved input and output rouge l scores of 51.4 and 52.3, respectively, but does not outperform the fine-tuned gpt-3.5. its bertscore of 88.9 indicates well-contextualized outputs but falls short of the benchmark set by gpt-3.5+ft. human evaluation note: although a human ema of 48.6% may appear low, it's important to note that ema is a stringent metric sensitive to minor phrasing variations from a reference, which can substantially impact scores. this sensitivity also accounts for the low output rouge l score. despite some subjectivity in the phrasing of inputs and outputs, the primary goal-assessing the ability to follow instructions requiring commonsense cooking reasoning-remains objective. llms often struggle with understanding intermediate steps and adjusting ingredient lists according to cooking instructions, highlighting deficiencies in their commonsense reasoning abilities. ## analysis & discussion the evaluation of baseline performance highlighted areas needing improvement. we analyze the results to identify the model's strengths and weaknesses, gaining insights into its limitations. we categorize the common errors into three classes: (1) missing or incorrect predictions, (2) inclusion of noncomestibles, and (3) non-descriptive predictions. ## conclusion in this work, we introduce pizzacommonsense, a dataset for evaluating models' understanding of cooking instructions, focusing on implicit ingredient transformations. we set baselines and demonstrate the challenging nature of the task through evaluations of llms. our experiments underscore the significant limitations of llms when applied to this dataset, highlighting their inability to effectively handle complex reasoning that necessitates commonsense knowledge. these findings emphasize the need for advancements in model architecture and training methodologies to address these challenges. overcoming these hurdles will enable the development of improved models capable of better understanding commonsense reasoning with procedural texts, with potential applications including autonomous agents using procedural information in scientific articles, industrial processes, diy, as well as cooking.