From Sentence to Action: Splitting AMR Graphs for Recipe Instructions (2023)

Abstract: Accurately interpreting the relationships between actions in a recipe text is essential to successful recipe completion. We explore using Abstract Meaning Representation (AMR) to represent recipe instructions, abstracting away from syntax and sentence structure that may order recipe actions in arbitrary ways. We present an algorithm to split sentence-level AMRs into action-level AMRs for individual cooking steps. Our approach provides an automatic way to derive fine-grained AMR representations of actions in cooking recipes and can be a useful tool for downstream, instructional tasks.

## introduction procedural texts are special kinds of text that serve the purpose of guiding humans through the steps required to accomplish a specific task. recipe texts are an idiosyncratic kind of procedural texts whose successful execution depends on accurately interpreting which actions need to be carried out in which order and which ingredients and tools are involved in each step. for example, the instruction "turn the dough out onto a surface dusted with flour and knead briefly until smooth." presents three actions: (i) dusting a surface; (ii) placing the dough on that surface; and (iii) kneading the dough until smooth. in recipe texts, actions often depend on other actions but there is often flexibility with respect to the overall order in which actions are instructed as some actions can take place in parallel or can be carried out at different stages of the cooking process. for example, dusting the surface could be instructed before preparing the dough. recipe texts frequently combine several actions in one sentence and often there are no uniform methods for putting specific actions into the same instruction; different versions of the same recipe may even differ in how equivalent or parallel actions are distributed across sentences. tasks such as adapting a recipe to a specific situation or presenting a recipe interactively to a user require the generation of a coherent recipe text that presents actions in a potentially different order and combination. to flexibly generate new versions of a recipe that present actions in an adapted order it is necessary to correctly decompose the recipe into the individual actions. previous work on recipe texts proposed identifying cooking actions and objects in recipes and representing the dependency relations between them in domain-specific graph representations with nodes for actions, ingredients and tools @xcite . these representations are attractive for fine-grained analysis of recipe texts but lack the expressivity to represent details such as adverbs or relations such as conditions and alternatives. yet, this kind of information is important for correctly reconstructing the original meaning when generating instructions. we explore generating recipe instructions at the action level from abstract meaning representation (amr) graphs. amr is able to represent finegrained and rich semantic relations, and its focus on predicate-argument structure makes it attractive for representing cooking instructions. yet, amr is a sentence-level representation that represents individual sentences in individual graphs. this paper addresses the challenge of splitting sentence-level amr graphs into the individual action-level amrs. we present a splitting algorithm that considers the semantic relationships between actions in recipe instructions ( §3) 1 . we evaluate our approach in a direct manual evaluation of the action-level representations as well as in an automatic and human evaluation of recipes generated from the created representation ( §4). findings show that our algorithm accurately identifies actionlevel subgraphs in amr recipe representations, suggesting its utility for amr representations of procedural texts and for generating action-level instructions. ## related work at first glance, recipe texts may look quite simple. yet recipe texts are semantically quite complex: subjects of actions are implicit, often as a result of being written in imperative mood; anaphoric expressions often refer to intermediate products that are outputs of actions and only partially corefer to input ingredients; and zero anaphora objects are frequent. to model recipe structure, most work on recipe text involves the identification and tagging of cooking actions, ingredients, intermediate substances and tools which get then used to create a structured representation @xcite @xcite . the most common representation approaches are graph and tree structures, which represent the flow and dependencies of actions and involved entities @xcite @xcite @xcite . the abstract meaning representation (amr) @xcite framework represents the meaning of sentences with a focus on predicateargument relationships, a key component of recipe structure. figure 1a shows the amr for the instruction "turn the dough out onto a surface dusted with flour and knead briefly until smooth." on the left. as shown, amrs are rooted, directed graphs in which nodes correspond to concepts and edges to the semantic relations between concepts. the framework makes use of a rich set of node and edge labels including frames from propbank @xcite , and within-sentence coreference is represented by re-entrancy. previous work on adapting amr to the nonsentence level proposes approaches to create a multi-sentence amr representation ( @xcite or dialogue amr graphs @xcite but essentially keeps the sentence-level amrs as part of the extended representations. amr has also been used in the tasks of summarization @xcite and text style transfer @xcite . ## discussion in this section we discuss the performance of our splitting algorithm and the results of the generation experiments in more detail. splitting algorithm. as described in §4.1, the splitting approach can successfully separate the s-amrs of almost all instructions in the ara1 and ara2 recipes into a-amrs. the iterative 13 statistical significance testing was performed using the software r (r core team, 2021) and the lme4 package @xcite . we used linear mixed effect models with condition as fixed effect, and by-subject and by-item intercepts and slopes as random effects, @xmath0.05. many of the a-amrs that are incorrectly split are based on a wrong s-amr. we found that often the same tokens or specific types of tokens lead to parsing mistakes and that these tokens are mostly specific to the recipe domain (e.g. "grease", "parmesan", "knead"). these observations are in line with the findings from @xcite that the main challenge for out-ofdomain amr parsing is the correct prediction of concepts. additionally, when ignoring splitting mistakes resulting from parsing mistakes, almost all incorrect a-amrs contain one of the following concepts: mean-01, have-degree-91 and have-quant-91. these concepts are used to represent complex relations and they introduce path patterns into the amr that are quite different and not covered by our algorithm. finally, during the manual evaluation of the a-amrs we encountered a number graphs for which it is not straightforward to decide whether the specific splitting is adequate because of the specific semantic characteristics and especially temporal interactions of the actions. for example, "bring a pot of salted water to a boil." gets split into "salt water." and "bring a pot of water to a boil.". however, none of the two potential orderings of the instructions is entirely adequate. on the one hand, the salt should be added before boiling the water. on the other hand, the water cannot be salted before it is filled into a pot but the "filling" action is only implicitly included in the original instruction. generated texts. in the automatic evaluation, fine-tuning on our recipe amr dataset resulted in a considerable improvement compared to generating the instructions with the pretrained amrlib model. we found that the amrlib model struggles to produce the recipe specific writing style. for example, the amrlib model generates "stir for a commingling" where the gart5 models generate "stir to combine". in contrast to the automatic evaluation, we found that in the human evaluation the recipes generated with context were not judged significantly better. the performance of the amr parser also affected the quality of the generated texts as wrong concepts in the amr lead to inadequate or nonsensical instructions. for example, representing "spoon" in the last instruction of the original version with save-01 resulted in a wrong instruction generated by gart5-0. general discussion. our findings suggest that amr representations are promising for representing and generating recipe instructions at the action level. the focus on predicate-argument structure makes them attractive for the representations of instructions as they center around actions and objects required to carry them out. additionally, amr graphs provide rich and fine-grained information about the semantic relations, the dependencies and also within-sentence coreference which makes it possible to identify the individual action events and to split even s-amrs for long and nested instructions into their a-amr components. furthermore, our approach produces again rich representations of the action events from which instructions for the individual action events can be generated. heuristically splitting the textual instructions instead of the amr representations would require a combination of different tools to predict all the relevant information such as dependencies and semantic roles. additionally, splitting the instructions at the text level using our dependency baseline more often resulted in ungrammatical sentences as reflected by the significantly higher grammar and fluency ratings for the texts generated from the a-amrs compared to the baseline. ## conclusion & future work we have presented an approach to split sentencelevel amr representations for cooking recipe instructions into more fine-grained amr representations of the individual action events. our rulebased algorithm provides an automatic way to identify which cooking actions in a recipe instruction constitute separate action events to be carried out and to systematically breaking up the sentence-level amrs into representations of the action events that provide more concise instructions. the predicate-argument oriented structure of amr facilitates this process, and our approach achieves high performance on accurately breaking up the s-amrs to more concise representations that can be used to generate instructions. one bottleneck of our approach is the performance of the amr parser in the domain of cooking recipes. future work might investigate adapting amr parsers to out-of-domain recipe vocabulary and processes. regardless, representations of action events can support analysis and comparisons of actions in different cooking recipes as well as instruction generation in tasks that require more flexibility with respect to the exact order in which actions are instructed. as the presented approach makes use of the domain-independent structure of amrs, we expect that it can generalize to other procedural texts, as well. on patterns of the pos tags of successive tokens to decide about the selection of tokens that do not have an aligned node in any of the a-amrs such as prepositions and determiners. participle actions such as "shredded" get stemmed to ensure that the extraction heuristic creates grammatical imperative instructions. additionally, the tokens selected for a gold instruction get partially reordered such that the action predicate is at the correct position in the new sentence if possible. training details. for training we build on the training scripts from the amrlib library foot_9 and adapt the required input format to our training set-up. all models are trained using the adam optimizer and a linear learning rate scheduler with warm-up with 1e -4 as the initial learning rate. the dropout rate is set to 0.1 for all reported experiments. the training and validation batch size is set to 24. we train all models using early stopping based on the train loss with a patience of 15 and a threshold of 0.00005 and select the final model based on the best bleu score on the validation set. following previous work on using transformer lms for amr-to-text generation, we use a graph linearization based on the penman format of the amrs @xcite . we create the input to the model by concatenating the context sentence and the amr in penman format including the node-to-token alignments and introduce a special token to separate the context and the graph (see generation. we set the token limitation for each generated sequence to 1024 and let the model output the best sequence using a beam size of 1. automatic evaluation. we compute bleu, rouge-1 (r-1), rouge-2 (r-2), rouge-l, meteor (m) and bleurt (blrt) scores. for the computation of all automatic metrics we use the hugging-face evaluate metric package foot_10 which provides wrappers around the original metric implementations or the implementations from the sacrebleu tool for comparable evaluation scores. we leave all parameters at their default values. for bleu, we compute case-insensitive bleu-4 at the corpus level. for bleurt, we use the pre-trained bleurt-large-512 checkpoint and average over all predicted sentence-level scores to obtain a final bleurt score.