What does it take to bake a cake? The R ecipe R ef corpus and anaphora resolution in procedural text (2022)

Abstract: Procedural text contains rich anaphoric phenomena, yet has not received much attention in NLP. To fill this gap, we investigate the textual properties of two types of procedural text, recipes and chemical patents, and generalize an anaphora annotation framework developed for the chemical domain for modeling anaphoric phenomena in recipes. We apply this framework to annotate the RecipeRef corpus with both bridging and coreference relations. Through comparison to chemical patents, we show the complexity of anaphora resolution in recipes. We demonstrate empirically that transfer learning from the chemical domain improves resolution of anaphora in recipes, suggesting transferability of general procedural knowledge.

## introduction anaphora resolution is a core component in information extraction tasks @xcite and critical for various downstream natural language processing tasks, such as named entity recognition @xcite and machine translation @xcite . it consists of two primary anaphoric types, coreference @xcite and bridging @xcite . most anaphora corpora @xcite @xcite , however, only focus on either coreference or bridging. to fill the gap in anaphora resolution, it is becoming increasingly important to have both types annotated. current research on anaphora resolution is mostly based on declarative text @xcite @xcite , such as news or dialogue. procedural text, such as chemical patents or instruction manuals, has received limited attention despite being critical for human knowledge @xcite . in turn, correct resolution of entities is the cornerstone of procedural text comprehensionresolution of anaphora in these texts is required to determine what action applies to which entity. we focus in this work on the procedural text type of recipes. as shown in fig. 1 , recipes have rich and complex anaphora phenomena. here, the expression the biscuits appears several times in text; while each occurrence relates to the same biscuits concept, their state and semantic meaning vary. our aim in this paper is to address anaphora resolution in procedural text, especially for recipes, identifying anaphoric references and determining the relationships among the entities. we first investigate the textual properties of procedural texts, i.e. chemical patents and recipes. we then adapt an existing anaphora annotation schema developed for chemical patents @xcite to recipes, and define four types of anaphora relationships, encompassing coreference and bridging. we further create a dataset based on this schema and achieve high inter-annotator agreement with two annotators experienced with the domain. we additionally explore the feasibility of applying transfer learning from the chemical domain to model recipe anaphora resolution. the dataset and related code are publicly available. 1 our contributions in this paper include: (1) adaptation of the anaphora annotation framework from chemical patents for modeling anaphoric phenomena in recipes; (2) creation of a publicly accessible recipe anaphora resolution dataset based on the annotation framework @xcite ; (3) investigation of the textual properties of chemical patents and recipes; and (4) demonstration of the benefit of utilizing procedural knowledge from the chemical domain to enhance recipe anaphora resolution via transfer learning. ## related work anaphora relation subsumes two referring types: existing anaphora corpora mostly focus on declarative text, across a range of domains @xcite @xcite . there have been attempts to annotate procedural text corpora for anaphora, but most focus exclusively on coreference @xcite . @xcite developed the conll 2012 corpus for generic coreference resolution. it consists of declarative texts including news and magazine articles, across three languages -english, chinese, and arabic. this corpus adopted the ontonotes 5.0 @xcite annotation scheme, modeling coreference in terms of two subtypes: (1) identity, where the anaphoric references and referents are identical; and (2) appositive, where a noun phrase is modified by an intermediately-adjacent noun phrase. it models coreference as a clustering task, ignoring the direction of relations. following largely the same annotation framework, the wikicoref corpus (ghaddar and langlais, 2016) targeted wikipedia texts. the inscript corpus @xcite consists of 1,000 stories from 10 different scenarios corresponding to a "script", i.e. a standardised sequence of events. the corpus includes coreference annota-tions for noun phrases. bionlp-st 2011 @xcite is a generelated coreference corpus based on abstracts from biomedical publications. it consists of four types of coreference: relat (relative pronouns or relative adjectives, e.g. that), pron (pronouns, e.g. it), dnp (definite nps or demonstrative nps, e.g. nps that begin with the) and appos (coreferences in apposition). as it only focuses on gene-related annotation, coreference is limited. craft-st 2019 @xcite annotates 97 full biomedical articles for coreference resolution, based on a slightly-modified version of the ontonotes 5.0 annotation scheme. compared to the bionlp 2011 corpus, it contains a wider range of relation types, and is not limited to only abstracts. scierc @xcite ) contains 500 abstracts from scientific articles, and coreference annotation. due to the complexities of defining bridging @xcite , different corpora have adopted different definitions of bridging. according to @xcite , bridging can be divided into: (1) referential, where the anaphoric references rely on the referent to be interpretable (e.g. a new town hallthe door, the old oak treeleaves, etc.); and (2) lexical, encompassing lexical-semantic relations, such as meronymy or hyponymy (e.g. europe and spain are in a wholepart relation). the arrau corpus @xcite consists of three types of declarative text: news, dialogue and narrative text. the bridging annotations are mostly lexical, with a much smaller number of referential references. the isnotes corpus @xcite is based on 50 wall street journal (wsj) texts from the ontonotes corpus, and contains both coreference and referential bridging. similar to isnotes, bashi @xcite is based on another 50 wsj texts from ontonotes with referential bridging. with the same annotation scheme as bashi, scicorp @xcite focuses on scientific text and referential bridging. a small number of domain-specific anaphora corpora have been developed for procedural text. the chemu-ref corpus @xcite contains 1,500 chemical patent excerpts describing chemical reactions. based on generic and chemical knowledge, the corpus contains five types of anaphora relationships, i.e. coreference, transfers, reaction-associated, work-up, and contained. @xcite developed the sofc-exp corpus based on 45 material sciences articles, for the purposes of information extraction. the corpus is primarily targeted at named entity recognition and relation extraction, with coreference as a secondary annotation task, based on coindexation between a common noun or pronoun and a more specific mention earlier in the text. also in the context of material sciences, @xcite annotated 230 synthesis procedures for coreference, largely based on text in parentheses and coreferent abbreviations. recent work in recipe comprehension includes visual instructions @xcite and linguistic texts @xcite @xcite across japanese @xcite and english @xcite . most research analyzes the text of recipes as a workflow graph based on actions @xcite @xcite , where the vertices represent name entities (e.g. action, food, etc.) and edges represent relational structure (e.g. action complement, food complement, etc.). although interactions among ingredients can be derived via action nodes, this approach doesn't sufficiently capture anaphora phenomena, i.e. coreference and bridging. the risec corpus @xcite identifies candidate expressions for zero anaphora verbs in english recipes. however, they do not capture generic anaphoric phenomena. in terms of modeling, most research has handled coreference and bridging separately due to limited data availability (and a lack of annotated datasets containing both coreference and bridging). for coreference resolution, span ranking models @xcite have become the benchmark method, supplanting mention ranking models @xcite @xcite . various span ranking variants have been proposed @xcite @xcite , and achieved strong results. with the increasing number of coreference corpora, transfer learning @xcite involving pretraining on a source domain and fine-tuning on a target domain has shown great potential at improving coreference resolution. bridging methods can be categorised into: (1) rule-based methods @xcite @xcite ; and (2) machine learning methods @xcite @xcite . hou (2020) modeled bridging resolution as a question answering task, and fine-tuned the question answering model from generic question answering corpora. by utilizing transfer learning, they achieved a stronger performance on the bridging task. @xcite proposed a joint training framework for bridging and coreference resolution based on an end-to-end coreference model @xcite . similar to coreference, they modeled bridging as a clustering task. through joint training, they achieved substantial improvements for bridging, but the impact on coreference was less clear. @xcite adopted the same end-to-end framework for joint training, modeling bridging as a mention pair classification task, and achieved improvements on both subtasks. ## annotation scheme in this section, we describe our adapted annotation scheme for recipe anaphora annotation. the complete annotation guideline is available at @xcite . ## task definition following the approach of @xcite , anaphora resolution is modeled as a two-step task: (1) mention detection; and (2) anaphora relation detection. as anaphora relation types in recipes are defined hierarchically, we can derive a simplified version of the recipe anaphora resolution task by removing state changes. that is, coreference and transformed can be merged when we remove consideration of state changes, and ingredient(without-state-change)-associated and ingredient(with-state-change)-associated can similarly be merged. as such, we evaluate recipe anaphora resolution both with state change (4-way), and without state change (2-way). as our corpus includes one-to-many anaphoric relations for bridging, standard coreference evaluation metrics @xcite @xcite , which assume a given mention only occurs in a unique cluster, are not suitable for this task. although coreferences involving one-to-one relations in this task could be evaluated with these metrics, to maintain a unified evaluation for bridging and coreference, we utilize precision, recall and f1 as our core metrics. 3 specifically, we follow the evaluation of the chemu-ref corpus, scoring coreference from two perspectives: (1) surface coreference, where a coreferent anaphor links to its closest antecedent; and (2) atom coreference, where a coreferent anaphor links to a correct antecedent @xcite . for manual annotation, we use the brat rapid annotation tool. 4 in an attempt to achieve high quality, we went through 8 rounds of annotation training and refinement of the anaphora annotation with two annotators experienced with the recipe domain. in each round of training, the annotators independently annotated 10 recipes (different for each round of annotation) and met afterwards to compare annotation results. further refinements of the annotation guidelines were made based on the discussion. after training, we reached a high inter-annotator agreement (iaa) of krippendorff's α = 0.85, mention-level @xmath0.88, and relation-level @xmath1.67. as a point of comparison, the respective values after the first round of annotator training were 0.45, 0.51 and 0.29, respectively. we use 80 double-annotated recipes with harmonized annotations as our corpus. the statistics of this corpus in comparison with the chemu-ref corpus @xcite are shown in ## methodology to investigate the benefit of transfer learning from the chemical domain, we follow the configuration of @xcite , modeling bridging as a classification task and adopting the benchmark end-toend neural coreference model of @xcite for joint training of the two anaphora resolution types. for each span x i , the model learns: (1) a mention score s m i for mention detection: and (2) a distribution p (•) over possible antecedent spans y (i) for coreference resolution: a span representation s i is the concatenation of output token representations (x * i ) from a bidirectional lstm (bilstm) @xcite , the syntactic head representation (h i ) obtained from an attention mechanism @xcite , and a feature vector of the mention (ϕ(i)): for mention loss, we use cross-entropy loss: and gold m is the set of gold mentions that are involved in anaphora relations. for coreference resolution, we compute the loss as follows, where gold c (i) is the gold coreferent antecedents that span i refers to: for bridging resolution, the loss is obtained by multiclass cross-entropy: for coreference l b for bridging l c + l b for joint training 6 experiments in this section, we present experimental results both with and without state change for recipe anaphora resolution. we use a similar configuration to @xcite . specifically, we use the concatenation of 300-dimensional glove embeddings @xcite , 1024-dimensional elmo word representations @xcite , and 8dimensional character embeddings that are learned from a character cnn with windows of 3, 4, and 5 characters as the pretrained token embeddings. each feed-forward neural network consists of two hidden layers with 150 dimensions and rectified linear units @xcite . the gold mentions are separated in coreference and bridging. for joint training, the gold mentions are combined. we use 10-fold cross-validation to evaluate our model on recipe anaphora resolution. since end-toend model performance varies due to random initialization @xcite , we randomly shuffle the dataset 5 times and run cross-validation 3 times for each shuffle. averaged results are reported. overall joint_train 62.1 ± 0.7 37.0 ± 0.5 46.0 ± 0.5 37.4 ± 0.7 21.8 ± 0.5 27.1 ± 0. 5 we also experimented with joint coreference resolution and change-of-state classification, and observed similar trends in the results, at reduced performance levels due to the difficulty in additionally predicting state changes (as shown in appendix a). ## conclusion in this paper, we have extended earlier work on anaphora resolution over chemical patents to the domain of recipes. we adapted the annotation schema and guidelines for chemical patents, and created a labeled anaphora resolution corpus for recipes. we further defined two tasks for modeling anaphora phenomena in recipes, with and without consider-ation of state change. our experiments show the benefit of joint training, and also transfer learning from the chemical domain.