Understanding the Cooking Process with E nglish Recipe Text (2023)

Abstract: Translating procedural text, like recipes, into a graphical representation can be important for visualizing the text, and can offer a machine-readable formalism for use in software. There are proposals for translating recipes into a flow graph representation, where each node represents an ingredient, action, location, or equipment, and each arc between the nodes denotes the steps of the recipe. However, these proposals have had performance problems with both named entity recognition and relationship extraction. To address these problems, we propose a novel framework comprising two modules to construct a flow graph from the input recipe. The first module identifies the named entities in the input recipe text using BERT, Bi-LSTM and CRF, and the second module uses BERT to predict the relationships between the entities. We evaluate our framework on the English recipe flow graph corpus. Our framework can predict the edge label and achieve the overall F1 score of 92.2, while the baseline F1 score is 43.3 without the edge label predicted.

## introduction recipes, one of everyday life's most common procedural texts, explain how to cook a dish using clear step-by-step instructions. with many global challenges directly related to food, nutrition and healthcare @xcite , analysing recipe texts is now attracting attention in the field of natural language processing(nlp) and artificial intelligence as a cross-disciplinary research focus @xcite . having demonstrated that flowcharts are reasonable representations of procedural texts like recipes due to their ability to clearly and concisely present workflows or processes @xcite , much research has focused on how they can be obtained from text @xcite @xcite . each flowchart node represents a single word or a sequence of words in the input text, and reflecting basic concepts such as tools, actions and materials, and the edges state the relationship between two nodes. an example of a flowchart with its original recipe text is shown in figure 1 , and we refer the reader to see more examples in appendix d. generally, the frameworks for this task includes two parts, named entity recognition (ner) and relationship extraction (re). however, the previous frameworks for english recipes can only detect which nodes are interconnected without identifying the label of the edges in the flowchart, and the overall f1 score for the whole framework is only 43.3 @xcite . although a subsequent proposal by @xcite improves the overall f1 score to 72.3, it still does not solve the edge label recognition problem, and there is room for performance improvement. thus, this research aims to address these issues by proposing a better framework for understanding english recipe texts that can produce a complete flow graph based on the input recipe. in our framework, we use bert @xcite , bi-lstm @xcite and crf @xcite to recognise the named entities as the nodes of the output flow graph and then extract their relationship by applying bert to construct the edges. unlike the previous work, which employed a dependency parser or used manually selected features to produce a spanning arborescence to generate a flowchart, we treat the edge prediction task as a classification problem that can directly predict the relationship of any pair of nodes by taking advantage of the large pre-trained model. we evaluated our framework on the english recipe flow graph corpus @xcite , and our framework surpasses the existing methods' performance in this task when the edges' label is predicted. this implies that our framework has potential to provide guidance to cooking robots, detect allergic food sources in recipes, convert recipes into low-calorie versions, etc. ## related work there has been a growing interest in the analysis of recipes by scholars in nlp and deep learning, for example, studying of recipes from a linguistic perspective @xcite , detecting allergens in recipes for users to improve recipe recommendation systems @xcite , finding patterns of spice and food combinations in recipes @xcite , exploring food and recipes to help meet the challenges of sustainable and healthy eating @xcite and making a robot that can cook @xcite . meanwhile, much research has focused on how to represent the semantics of the input text. momouchi (1980) proposes using flowcharts to represent recipes, which was the first proposal for procedural text understanding. representing recipes as graphs allows for analysis, such as measuring the similarity between recipes @xcite ., which indicates that graph structure can usefully reflect the semantic information in a recipe. @xcite proposes a machine learning approach for processing japanese recipe texts. the limitation of their study is that it only focuses on entity relations within the same sentence, which means that their work is insufficient to convert recipes into flowcharts. the manually annotated results of @xcite show that directed acyclic graphs are a suitable representation for natural language understanding of the procedural cooking text. @xcite proposes a framework composed of ner and maximum spanning tree estimation for procedural text understanding. although the results of their framework tests on the japanese recipe corpus @xcite are not very encouraging, the design of the whole framework is inspiring for the follow-up work. @xcite annotates a corpus of english recipes containing 100 recipes using the method of @xcite . they aim to analyse the structural differences between japanese recipes and english recipes. they compare the annotated named entities in the japanese and english corpora, identify lexical, semantic and underlying structural differences between the japanese and english recipes, and analyse their reasons. one of their most important results is the demonstration that it is feasible to annotate english recipe texts using the @xcite al. (2014) guide, as most previous work has focused on japanese recipe texts. building on the work of yamakata et al. (2017), yamakata et al. (2020) add 200 recipe texts in english annotated with named entities and flowchart representations to the original corpus of 100 recipes to produce a new english flow graph corpus (300r). the label of the named entities is shown in ## the whole framework to convert the input recipe text into flowchart output, we need two steps to process it, namely recognising named entities and identifying relationships between named entities. therefore, in our complete framework, we have two main modules to deal with those two tasks. the following subsections will introduce why we choose such models and how we use them in detail. tag meaning f food t tool d duration q quantity ac actions by the chef ac2 discontinuous action af action by foods at action by tool sf food state st tool state ## discussion while our framework addresses the remaining issues of low accuracy in identifying named entities and the inability to predict edge labels, we still have some problems. the flowchart we get when we enter "cook the rest of the pancakes, one at a time but remember to melt a small knob of butter before adding the batter" in the framework is shown in figure 7 , and the correct flowchart is shown in figure 6 . we can observe that our framework identifies the rest of the flowchart correctly, except for the relationship between the named entity "cook" and the named entity "melt". in the sentence, "melt" is the action on the butter and "cook" is the action on the whole pancake. so the butter should be melted before the pancakes can be cooked, while our framework thinks the pancakes should be cooked before the butter is melted. however, surprisingly, if we change the input to "remember to melt a small piece of butter before adding the batter to cook the pancakes." our framework can correctly predict all the relationships. we can see those two inputs express the same meaning, but we obtained a different result. because of the flexibility of the language, the same recipe may be written in several ways. in this example, organising the input sentence with a clear order and purpose is an improvement for the learning process of the framework. hence, it illustrates getting the framework to learn the expression of the same sentence in different writing styles is also a key point in natural language understanding. using data augmentation to change the same sentence into diverse representations with the same named entity information inside would give our framework a more robust text understanding capability. besides, we can see that one of the weaknesses of our framework is that we do not have a specific strategy for choosing the root node. we should select "cook" as the root node in this case. although our framework's accuracy is reasonable, it could be further improved if there was a suitable way to determine the root node. @xcite suggests choosing the ac labelled named entity that was last entered into the framework as the root node. however, in this example, we can observe that due to the language's flexibility, the last ac labelled named entity could be "adding" or "cook". so this approach does not always work. this study can be further investigated in future work. in addition, as our framework focuses on learning the relationships between individual nodes, not much attention is paid to the completeness of the flowchart. this is the main drawback of our framework. although a few incomplete graphs were observed during the test, it is still the prior issue we need to address in the future. ## conclusion this paper proposes a framework for english recipe understanding by constructing flowcharts. this includes identifying the edge's label, indicating that we can efficiently output a complete flow graph. the overall framework performance achieves a satisfactory result and exceeds all previous work on the r-300 corpus, so it has the potential to be deployed in intelligent software kitchen assistants, cooking robots and diet management, etc. despite the good results of our framework, it still has some problems and some improvements that could be made in future work. firstly, our framework performs better when predicting labels with more significant amounts of data. therefore, adding more labelled recipes in the future would benefit the training. the second is that the frame-work can perform better by selecting a root node and forcing all nodes to be linked together. choosing a root node is a difficult task, and a suitable root node can facilitate the learning process of the framework. however, if the correct root node is not chosen, this can have a significant negative impact. in the future, we could also try to use data augmentation to allow the framework to learn different sentences with the same semantic meaning, giving it a more vital generalisation ability. also, we would like to investigate if the framework could understand recipe text better by focusing on only one type of recipe, such as dessert or main course. moreover, exploring other more powerful pre-trained models and combining some linguistic features would be a good direction. yoko yamakata, john carroll, and shinsuke mori. 2017. a comparison of cooking recipe named entities between japanese and english. in proceedings of the 9th workshop on multimedia for cooking and eating activities in conjunction with the 2017 international joint conference on artificial intelligence, cea2017, page 7-12, new york, ny, usa. association for computing machinery. yoko yamakata, shinsuke mori, and john carroll. 2020. english recipe flow graph corpus. in proceedings of the 12th language resources and evaluation conference, pages 5187-5194, marseille, france. european language resources association.