Aligning Actions Across Recipe Graphs (2021)

Abstract: Recipe texts are an idiosyncratic form of instructional language that pose unique challenges for automatic understanding. One challenge is that a cooking step in one recipe can be explained in another recipe in different words, at a different level of abstraction, or not at all. Previous work has annotated correspondences between recipe instructions at the sentence level, often glossing over important correspondences between cooking steps across recipes. We present a novel and fully-parsed English recipe corpus, ARA (Aligned Recipe Actions), which annotates correspondences between individual actions across similar recipes with the goal of capturing information implicit for accurate recipe understanding. We represent this information in the form of recipe graphs, and we train a neural model for predicting correspondences on ARA. We find that substantial gains in accuracy can be obtained by taking fine-grained structural information about the recipes into account.

## introduction cooking recipes are a type of instructional text that many people interact with in their everyday lives. a recipe explains step by step how to cook a certain dish, describing the actions a chef needs to perform as well as the ingredients and intermediate products of the cooking process. however, recipes for the same dish often differ in which cooking actions they describe explicitly, how they describe them, and in which order. 1 for instance, in the three recipes in fig. 1 , the overall process of assembling a batter and making waffles is explained with different levels of detail: recipe (a) explains the process with three distinct cooking actions (in bold); recipe (b) with eight distinct actions; and recipe (c) with nineteen actions. the set of recipes in fig. 1 provides more general information about how to make waffles than each individual recipe can. to our knowledge, only one previous work focuses on alignment of instructions across recipes to facilitate recipe interpretation on a dish level: @xcite (henceforth referred to as l'20) present the microsoft research multimodal aligned recipe corpus to align english recipe text instructions to each other and to video sequences. l'20 focus on alignment of instructions as sentences. yet, as sentences can be quite long and contain multiple actions, defining instructions at the sentence level often glosses over the relationship between individual actions and excludes the complex event structure that makes recipe interpretation challenging and compelling. for example, in l'20, "ladle the batter into the waffle iron and cook [...]" in recipe (b) is aligned to "spoon the batter into preheated waffle iron in batches and cook [..]" in recipe (c). if aligned at the sentence level only, the action correspondence of (b)'s recipe-initial "preheat" and (c)'s "preheated" much later in the recipe is not (and cannot be) accounted for. the relationship between the individual actions ((b) "ladle," (c) "spoon") as well as between both instances of "cook" is additionally obscured by coarse-grained sentence alignment. aligning recipes at the action level (i.e. the bold items in fig. 1 ) instead of the sentence level is practically useful to aggregate detailed information about how to cook a dish in different ways. this alignment would additionally offer greater insight into how recipe actions implicitly contain information about ingredients, tools, cooking processes, and other semantic information necessary for accurate recipe interpretation. in this paper, we make two contributions towards the goal of complete and accurate alignment of actions across similar recipes. first, we collect a novel corpus that aligns cooking actions across recipes, aligned recipe actions (ara), (section 4), by crowdsourcing alignment annotations on top of the l'20 corpus for a select number of dishes. in order to determine the annotation possibilities, we develop a neural recipe parser to identify descriptions of actions and substances and arrange them in a recipe graph (section 3). second, we present an alignment model which identifies correspondences between actions across recipe graphs based on our parsed corpus (section 5). error analysis of both human and machine performance illustrates that, though complex, the task of aligning recipe actions is achievable with our methodology and can inform future work on aligning sets of instructions. our corpus and code are publicly available. 5 2 background and related work recipe text. the recipes in fig. 1 illustrate some of the idiosyncracies of recipe text: all texts are written entirely in imperative mood ((a) "cook"; (b) "preheat"; (c) "sift"); definite noun phrases frequently drop their determiners ((a) "eggs"; (c) "milk," "egg whites"); many arguments are elided or left implicit ((b) "beat ∅"; (c) "pour ∅ over"); bare adjectives are used to describe desired end states ((b) "until crisp and golden"); and many anaphoric expressions refer to entities which were not explicitly introduced before ((b) "the liquid"; (c) "the flour mixture"). accurate interpretation of single recipe instructions requires familiarity with situated food ingredients, knowledge of verb semantics to identify how each cooking step relates to the others, and general commonsense about the cooking environment and instructional syntax. recipe graphs and corpora. representing recipes as graphs is the dominant choice @xcite @xcite @xcite @xcite . of relevance to this paper is the recipe corpus of @xcite (y'20), which consists of 300 english recipes annotated with graphs as in fig. 2 . we train a recipe parser on this corpus (section 3) and use the trained parser to identify actions in the l'20 corpus. as noted earlier, @xcite (l'20) created the microsoft research multimodal aligned recipe corpus of roughly 50k text recipes and 77k recipe videos across 4000 dishes in english. the recipes and video transcriptions were segmented into sentences (but not parsed), and 200 recipe pairs were manually annotated for correspondences between recipe sentences. l'20 cluster dishes based on exact match of recipe title; they perform sentence alignments on pairs of recipes using the unsupervised algorithm of @xcite . we use l'20's dataset as a basis for our alignment task in sections 4 and 5. recipe parsing. parsing recipes into graphs is usually comprised of two steps: (i) tagging mentions of substances and cooking steps, and (ii) linking these mentions with input and output edges. recent work on english recipes has achieved f-scores above 90 for identifying mentions @xcite and f-scores above 80 for adding the edges @xcite @xcite . most of this work uses supervised learning based on hand-annotated recipe datasets. using unsupervised methods, @xcite train a generative neural model on a large corpus of unannotated recipe texts and achieve an f-score of 80 on predicting edges given gold information about the nodes; the output graphs are less detailed than ours. @xcite achieves an f-score of 75 on the same task and presents a subtask of creating action graphs similar to ours in section 3.4. event alignment. our work shares an interest in modelling procedural knowledge with the detection and alignment of script events. @xcite identified event types from text according to their predicate-argument structures and behavior in event chains via countbased statistics. we capture similar information in a crowdsourcing task reminiscent of @xcite to automatically align actions without all surface text @xcite . ## parsing recipes into graphs the main contribution of this paper is a corpus of action alignments between action graphs of cooking recipes. basing our corpus on unannotated recipe texts from l'20, we are dependent on an accurate tagger and parser for pre-processing. the tagger identifies the alignable actions in a recipe, and the parser structures recipes into graph representations. for both tasks, we train neural models on the data of @xcite (y'20); we set a new state of the art on this dataset. finally, we distill the y'20-style recipe graphs into more focused action graphs (section 3.4) which the alignment model (section 5) takes as input. ## a corpus of recipe action alignments we collect a corpus of crowdsourced annotations of action alignments between recipes for a subset of l'20's corpus. our aligned recipe action (ara) corpus makes two important contributions: (i) manual alignments, as opposed to the majority of l'20's automatically aligned corpus; (ii) alignments are at the action level, as opposed to the more coarse-grained sentence level. ## automatic alignment of actions we develop two automatic alignment models that mimic our crowdsourcing task; these models align action nodes between our parsed action graphs (section 3.4). we evaluate both models on the crowdsourced alignment dataset (section 4): one using features solely from the actions to be aligned, and one incorporating parent and child action nodes into the alignment decision. ## conclusion & future work in this paper, we have made two contributions: (i) ara, a novel corpus of human-annotated alignments between corresponding cooking actions, and (ii) a first neural model to align actions across recipe graphs. we find that incorporating structural information about recipes improves the accuracy of the neural model, highlighting the usefulness of recipe graphs and of recipe parsers. compared to previous work, our corpus and model represent alignments at the level of individual actions and not of entire sentences. in refining the sentence-level alignments of l'20 to the cooking action level, we find that judging when and how specific actions "correspond" is a complex task that requires intricate knowledge of the cooking domain. alternatively, the complexity of recipe interpretation can be framed as a matter of recognizing nuances in how meaning is construed in recipe text given the genre and its preferred syntactic constructions @xcite . looking ahead, our work lays a foundation for research which automatically aggregates multiple recipe graphs for the same dish, identifying common and distinct parts of the different recipes. this opens up a variety of applications in the cooking domain, including dialogue systems which can explain a recipe at different levels of abstraction. label meaning explanation f food eatable; also intermediate products t tool knife, container, etc. d duration duration of cooking q quantity quantity of food ac action by chef verb representing a chef's action ac2 discontinuous ac (english only) second, non-contiguous part of a single action by chef af action by food verb representing action of a food at action by tool (english only) verb representing a tool's action sf food state food's initial or intermediate state st tool state tool's initial or intermediate state