Recipe Instruction Semantics Corpus ( RIS e C ): R esolving Semantic Structure and Zero Anaphora in Recipes (2020)

Abstract: We propose a newly annotated dataset for information extraction on recipes. Unlike previous approaches to machine comprehension of procedural texts, we avoid a priori pre-defining domain-specific predicates to recognize (e.g., the primitive instructionsin MILK) and focus on basic understanding of the expressed semantics rather than directly reduce them to a simplified state representation (e.g., ProPara). We thus frame the semantic comprehension of procedural text such as recipes, as fairly generic NLP subtasks, covering (i) entity recognition (ingredients, tools and actions), (ii) relation extraction (what ingredients and tools are involved in the actions), and (iii) zero anaphora resolution (link actions to implicit arguments, e.g., results from previous recipe steps). Further, our Recipe Instruction Semantic Corpus (RISeC) dataset includes textual descriptions for the zero anaphora, to facilitate language generation thereof. Besides the dataset itself, we contribute a pipeline neural architecture that addresses entity and relation extractionas well an identification of zero anaphora. These basic building blocks can facilitate more advanced downstream applications (e.g., question answering, conversational agents).

## introduction recently, several efforts have aimed at understanding recipe instructions (see section 2). we consider such recipes as prototypical for procedural texts, for which processing is complex due to the need to (i) understand the ordering of steps (not unlike, e.g., event ordering in news), (ii) solve frequent ellipsis (i.e., zero anaphora) and coreference resolution, and (iii) track the state changes they involve (e.g., ingredients processed/combined to new entities). especially the latter distinguishes procedural text processing from more traditional information extraction (e.g., from news). most existing works on recipes focus on recognizing pre-defined predicates, typically in the form of a limited set of instruction types (e.g., to convert the recipe to robot instructions) with predefined argument slots to fill. further, they often rely on an available starting list of ingredients (which may not be available in other procedural text). hence, current approaches towards recipe understanding make assumptions that are rather domain specific. in contrast, we aim for a more basic and generic structured representation of the procedural text, limiting domain-specific knowledge and building on more general semantic concepts. in particular, we build on semantic concepts as defined in propbank @xcite , which are not domain-specific. note that our proposed form of structured representations not necessarily allows directly solving informational queries that require explicit reasoning and/or state tracking (e.g., "where are the tomatoes after step 5?"). we however pose that properly detecting the various entities (e.g., ingredients and their derivations) and the actions that are executed on them (as described by verbs), with the appropriate coreference and zero anaphora resolution, would enable constructing a graph that facilitates such tracking. thus, while our proposed representation based on the idea of joint entity and relation extraction @xcite , provides useful input for it, such explicit state tracking and representation (e.g., as in propara, @xcite is left out of scope here. in summary, this paper reports on our work-inprogress and makes two main contributions. first, we present our newly annotated recipe instruction semantic corpus (risec) dataset (section 3), following the frame-semantic representation of propbank @xcite . since propbank is domain-agnostic, the approach should be largely generalizable 1 to other procedural text settings. second, we introduce a baseline framework (section 4) to solve (i) entity recognition (ingredients, tools and actions), (ii) relation extraction (ingredients and tools linked to the actions), (iii) zero anaphora identification. experimental evaluation thereof on risec is provided (section 5). ## related work from the perspective of structured representation, @xcite define the minimal instruction language for the kitchen (milk), which is based on first-order logic to describe the evolution of ingredients throughout a recipe, and use it for annotation in the curd dataset. building on this effort, jermsurawong and habash (2015) extend curd toward ingredientinstruction dependency tree parsing in simmr: they present an ingredient-instruction dependency tree representation of the recipe, but do not model instruction semantics. this contrasts with @xcite , who propose a pipeline framework for information extraction on japanese recipes from the the recipe flow graph (r-fg) dataset @xcite . maeta et al. use word segmentation, named entity recognition and syntactic analysis to extract predicate-argument structures and build a recipe flow graph that is conceptually similar to a simmr tree. their work is conceptually closest to ours, in that they propose a chain of nlp subtasks (but we do not need word boundary identification in our english corpus). yet, we build on a more elaborate and generic semantic relation scheme, propbank @xcite . further, methodologically we adopt neural network models as opposed to their logistic regression for ner and a maximum spanning tree (mst) parser for the relations (i.e., graph arcs). tracking state changes is another key to understanding recipe language. @xcite predict the dynamics of action and entity attributes in recipes by employing a recurrent memory network. their work includes sentence generation, but does not address the zero anaphora problem (see further) directly. besides recipes, other works focus on different procedural tasks. the propara 2 project aims at 1 while some of our entity types are specific to the cooking domain (e.g., "food", "temperature"), the relations that link action verbs to them are not (cf. propbank). in a casserole, combine soup mix, artichoke hearts, cheese and crab meat. bake [the crab mixture] for 30 minutes; then serve [the baked crab] immediately. comprehending scientific processes and tracking the status of entities in them: @xcite focus on tracking entity locations (as well as their creation/destruction) using a specific matrix state representation (with a row per step, a column per entity). the proposed models however do not incorporate entity recognition and are specifically filling the chosen state representation. in our work, we rather stick to a more "basic" understanding, which is broader in scope than location tracking. in terms of datasets beyond the recipe domain, the work of @xcite is noteworthy: it focuses on material synthesis and annotates domain-specific entities (materials, operations, conditions, etc.) and relations. the latter in our case are rather domain-agnostic (using propbank). ## the risec dataset the following paragraphs describe our dataset and the annotations underlying the presented extraction task foot_0 . ## model we focus on two tasks: (1) joint entity recognition, relation extraction and zero anaphora identification, and (2) zero anaphora description generation. next we present our models for each. ## experiments and results we split our risec dataset into 50% training, 20% development and 30% test sets, using the same splits as simmr @xcite . we tune hyperparameters on the development set. reported performance metrics are obtained on the test set. in general, our span-based model shows good performance in the extraction task, as shown in individual entity and relation type performance is reported in tables 2 3 . as expected, ## conclusion and future work this paper introduced risec, a dataset for extracting structural information and resolving zero anaphora from unstructured recipes. the corpus consists of 260 recipes from simmr and provides semantic graph annotations of (i) recipe-related entities, (ii) generic verb relations (from propbank) connecting these entities, (iii) zero anaphora verbs having implicit arguments, and (iv) textual descriptions of those implicit arguments. we reported on our work-in-progress with two baseline models using our corpus: (i) a neural span-based model extracting entities, zero anaphora verbs and relations, and (ii) a sequence-to-sequence attention model generating noun phrases for zero anaphora verbs. we plan to continue working in this direction, making the dataset larger and more finegrained, and especially, to investigate how it can be leveraged for human-machine interaction experiments.