Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario (2022)

Abstract: People can acquire knowledge in an unsupervised manner by reading, and compose the knowledge to make novel combinations. In this paper, we investigate whether pretrained language models can perform compositional generalization in a realistic setting: recipe generation. We design the counterfactual recipe generation task, which asks models to modify a base recipe according to the change of an ingredient. This task requires compositional generalization at two levels: the surface level of incorporating the new ingredient into the base recipe, and the deeper level of adjusting actions related to the changing ingredient. We collect a large-scale recipe dataset in Chinese for models to learn culinary knowledge, and a subset of action-level fine-grained annotations for evaluation.We finetune pretrained language models on the recipe corpus, and use unsupervised counterfactual generation methods to generate modified recipes.Results show that existing models have difficulties in modifying the ingredients while preserving the original text style, and often miss actions that need to be adjusted. Although pretrained language models can generate fluent recipe texts, they fail to truly learn and use the culinary knowledge in a compositional way. Code and data are available at https://github.com/xxxiaol/counterfactual-recipe-generation.

## introduction reading is an effective way to gain knowledge. when people read, mental processes like structured information extraction and rule discovery go on in our brains @xcite . in the case of cooking, we read recipes of various dishes, gain knowledge of ingredients and flavors, and compose them to cook other dishes. 1 figure 1 : an example of the recipe learning process of human for the dish red-braised crucian carp. this process involves knowledge acquisition and composition. as shown in figure 1 , when people read recipes, they distill the knowledge of flavors and ingredients, like soy sauce is usually used to get red-braised flavor and people often make diagonal cuts on fish to better marinate. people can then cook new dishes like red-braised crucian carp by composing existing knowledge about how to form the red-braised flavor and how to cook fish. we expect models to acquire culinary knowledge unsupervisedly, and be able to use the knowledge skillfully, e.g., composing new dishes. current recipe processing tasks do not examine this ability explicitly. recipe understanding tasks usually evaluate the models in a specific aspect under supervision, like identifying ingredient states or red-braised pork 1. wash the pork. 2. blanch the pork in a pot. 3. … what if we replace the ingredient pork to crucian carp? red-braised crucian carp level 1: ingredient red-braised crucian carp 1. wash the crucian carp. 2. … red-braised crucian carp 1. wash the crucian carp. 2. … level 2: ingredientrelated actions red-braised crucian carp 1. … 2. blanch the pork in a pot. 2. make diagonal cuts on the fish. red-braised crucian carp 1. … 2. blanch the pork in a pot. 2. make diagonal cuts on the fish. relationship between actions. recipe generation tasks evaluate whether models can generate fluent recipes, but do not investigate whether the generation ability relies on simple word correlation or culinary knowledge composition. current models exhibit strong compositional generalization ability in the understanding of synthetic texts @xcite @xcite , but few of them conduct experiments on realistic data, which is more challenging in two ways: (1) there are far many synonymous expressions in real-world texts, like various mentions of ingredients and actions in recipes @xcite , but synthetic data often use a single expression for one meaning @xcite ; (2) the knowledge in synthetic data is often expressed accurately and clearly, but knowledge in data provided by real users is varying. for example, when forming the red-braised flavor, most people use soy sauce while few use iced sugar instead. moreover, previous compositional generalization tasks mainly focus on semantic parsing and language grounding, while we aim to examine models in the form of natural language generation. we propose the counterfactual recipe generation task, to examine models' compositional generalization ability in understanding and generating recipe texts. this task requires models to answer the question: given a recipe of a dish, how will the recipe change if we replace or add one ingredient? as shown in figure 2 , models are asked to modify the base recipe of red-braised pork to form a new recipe of red-braised crucian carp while preserving the original cooking and writing style. we develop baseline methods to solve the task, as existing compositional generalization models cannot fit in the counterfactual generation task form. we finetune pretrained language models (plms) on our collected chinese recipe corpus to learn culinary knowledge, and use prevalent unsupervised counterfactual generation frameworks to generate counterfactual recipes given {base dish, base recipe, target dish}, where the target dish differs from the base dish in a main ingredient. instead of annotating gold target recipes, we evaluate models in two levels of compositional competencies, which is less labor-consuming. the surface level (l1) is the fusion of the changing ingredient and the base recipe: the new ingredient should be added to the recipe, and the replaced ingredient should be removed. for instance, the original step wash the pork needs to be changed to wash the crucian carp. we evaluate the ingredient coverage ratio of the added and replaced ingredient, and the degree of recipe modification. results show that existing plms can hardly cover the new ingredient and delete the replaced ingredient without making unnecessary modifications to the base recipe. the deeper level (l2) is the fusion of actions related to the changing ingredient and the base recipe: actions to process the new ingredient should be inserted, and actions only related to the replaced ingredient should be deleted. this is a much harder problem involving decompositions and compositions of actions. in the case of figure 2 , a model should know blanch is related to pork and is not suitable for a crucian carp, in order to delete the action blanch the pork. also, the action make diagonal cuts is widely used in fish dishes, and should be added to the recipe. our action-level evaluations show that most models fail to either remove all the irrelevant actions, or insert all the necessary ones, indicating that current plms have not fully learned the patterns of ingredient processing. our main contributions are as follows: 1) we propose the counterfactual recipe generation task to test models' compositional generalization ability in a realistic scenario. 2) we collect a large-scale chinese recipe dataset and build a counterfactual recipe generation testbed with fine-grained actionlevel annotations, which can also increase the research diversity for procedural text understanding. 3) we examine models' compositional generalization ability from two levels. our experiments show current plms are unable to modify the ingredient and preserve the original text style simultaneously, and will miss actions related to the changing ingredient that need to be adjusted. further analysis reveals that current models are still far from human experts in the deep understanding of procedural texts, like tracking entities and learning implicit patterns. ## task definition we formulate our task in the form of counterfactual generation @xcite : ## baseline methods we use two competitive unsupervised counterfactual generation models @xcite as our baselines. we do not consider existing compositional generalization models as it is hard to adapt them to our task setup. in order to make those models learn culinary knowledge, we finetune gpt-2 on the recipe corpus and use it as the backbone of the baselines. ## evaluation 6.1 l1: surface-level evaluation we first evaluate the generated texts from two aspects, to answer the l1 question: can models incorporate the changing ingredient into the base recipe without making excessive changes? 6.1.1 evaluation metrics coverage of ingredients. we check the coverage ratio of newly added ingredients in the generated recipes. for instances where an old ingredient is replaced, we also check whether the replaced ingredient still exists in the generated recipes: ## conclusion we propose counterfactual recipe generation, a task to evaluate models' compositional generalization ability in realistic settings. we explore two levels of compositional competencies: modifying the changing ingredient, and modifying the ingredientrelated actions in the base recipe. experimental results show that current models have difficulties in composing the new ingredient and actions into the base recipe while preserving the cooking and writing styles. by proposing this task, we reveal the large gap between current pretrained language models and humans in the deep understanding of procedural text, and want to elicit attention to pretrained language models' abilities in unsupervisedly learning and generalizing in real-life scenarios.