The Coreference under Transformation Labeling Dataset: Entity Tracking in Procedural Texts Using Event Models (2023)

Abstract: We demonstrate that coreference resolution in procedural texts is significantly improved when performing transformation-based entity linking prior to coreference relation identification. When events in the text introduce changes to the state of participating entities, it is often impossible to accurately link entities in anaphoric and coreference relations without an understanding of the transformations those entities undergo. We show how adding event semantics helps to better model entity coreference. We argue that all transformation predicates, not just creation verbs, introduce a new entity into the discourse, as a kind of generalized Result Role, which is typically not textually mentioned. This allows us to model procedural texts as process graphs and to compute the coreference type for any two entities in the recipe. We present our annotation methodology and the corpus generated as well as describe experiments on coreference resolution of entity mentions under a process-oriented model of events.

## introduction entity coreference resolution is a critical component for understanding most natural language text @xcite . however, when events in the text introduce changes to the state of participating entities, it is often impossible to accurately link entities in anaphoric and coreference relations without an understanding of the transformations those entities undergo. for example, events can bring about changes in entities that are not reflected in actual text mentions: that is, while it is anaphorically bound to the garlic, it is not strictly coreferential, as the garlic has undergone a transformation @xcite . * these authors contributed equally to this work. events can also introduce new entities into the discourse or narrative, through the use of creation predicates @xcite . this is pervasive in procedural text, where the goal is to describe a sequence of transformations to apply to multiple objects to build up a goal object. this can be seen, for example, in (2a), where the entities are transformed into a hidden result argument, which then licenses the definite np the mixture in (2b). in addition, procedural text witnesses both argument drop, as in (2d), where the direct object is elided, as well as metonymies, where a container refers to its content, as with bowl in (2d). in this paper, we demonstrate how a processoriented event model (poem), based on dynamic event structure proposed in @xcite , motivated by and generalized from gl-verbnet @xcite , can significantly help classify entity coreference in procedural texts. we argue that all transformation predicates, not just creation verbs, output a new entity into the discourse, as a kind of generalized result role, which is typically not textually mentioned @xcite . this allows us to model procedural texts as input/output (i/o) process graph structures, as shown in fig. 1 . each edge in the graph represents poem, an event reduced to an i/o process. the "output" nodes of events are generalization of the result role from the verbnet frames, as well as placeholders of syntactic drops and shadow arguments. the poem graph, thus, is one way to serialize the abstraction of complex semantics including event-argument structures, subevent structures, temporal ordering and coreference chains, which we can unfold to re-construct other semantic structures. for example, from the graph, one can compute the type of (conventional) coreference or what we call a "coreference under transformation" relation for any two entities in the recipe. to this end, we present cutl 1 , a novel annotation methodology and dataset that integrates both the tracking of entity transformations and coreference chains of entities into a single framework. our pilot annotation contains 100 double-annotated cooking recipes, showing high agreement on relation f1 scores. based on our process-oriented semantic model of events, we introduce a distinction between two relations: (i) coreference under identity (cui), where two entities have identical state information; and (ii) coreference under transformation (cut), where some change has occurred distinguishing two entities. we, then, use various methods from tu et al. 2023 to paraphrase transformed entities (generalized result role) that do not appear as textual mentions, while being aware of transformations entities have undergone. we use the annotated data to train models to predict various coreference relations between entities and show the value of transformation-aware entity representation in developing a coreference resolution system that works with entities in procedural text. our experiment also shows an interaction between our semantic model and llms to generate reliable and natural paraphrases. the contributions outlined in this paper include: (1) studying the anaphoric and coreference behavior inherent in procedural texts, focusing on cooking recipes; (2) operationalization of the poem, where steps in a procedure are annotated with explicit i/o entity nodes, regardless of whether they are mentioned in the text; (3) creation of an annotation guideline and gui environment based on the event model, identifying events and their semantic class, all ingredient entities, and set of coreference relations between entities, typed according to the kind of transformation; and (4) the creation of a dataset, cutl, containing these entity coreference links and the events involved. ## related work understanding procedural narratives involves many core competencies in language comprehension @xcite . not only is it crucial to per-1 annotation data, scheme, tool and experiment code is available at https://github.com/brandeis-llc/dp-cutl form anaphora resolution @xcite , but equally important is to perform state tracking on the entities as they undergo transformations described in the text @xcite . the task of anaphora resolution covers a range of coreference relations @xcite , as well as non-identity anaphoric relations, known as bridging phenomena @xcite . most work on anaphora resolution has focused on declarative narratives or dialogue datasets @xcite . interestingly, while there are several datasets of procedural texts that have been annotated and studied, these have been mostly in the context of entity state tracking and qa tasks @xcite @xcite , rather than coreference resolution; two notable exceptions include @xcite and @xcite . examples of how entity state tracking datasets contribute to reasoning and inferencing tasks can be seen with @xcite , who presents the neural process networks to model the state changes in procedural texts. the actions and entities are both predefined sets. they use soft attention to select actions and entities from the predefined sets to generate a state embedding for each entity at every step in the recipe. @xcite is an example of how entity state tracking datasets contribute to reasoning and inferencing tasks. the paper extends the propara dataset @xcite which contains texts describing processes. workers were given a prompt (e.g., "what happens during photosynthesis?") and then asked to author a series of sentences describing the sequence of events in the procedure. the goal is to predict the state (location, created, destroyed) change of all the participants. also working with propara, @xcite approach the task of tracking state change by first parsing every sentence in propara with the verbnet parser @xcite , and then leveraging the lexical information from verbnet and propbank to predict the state change. the interaction of anaphora resolution with state tracking makes it challenging to classify the relationships that result between entities mentioned in the text, in order to judge whether they are coreferential or somehow related, but not the same. to this end, the above distinction between coreference and bridging (non-identity anaphora) becomes relevant @xcite . this is how @xcite approach the problem of np reference in procedural text. they first adopt the distinction made in @xcite between two types of bridging (referential, where the np requires an antecedent to be fully understood; and lexical, which may involve any number of lexical semantic relations between two nps. their dataset (reciperef) of coreference relations includes both coreference and bridging relations. for the latter, they distinguish three types, depending on the state of the entities being associated: (a) no change; (b) physical change; and (c) chemical change. another work focusing on anaphora in recipes is @xcite , which introduces risec, a dataset for extracting structural information and resolving zero anaphora from unstructured recipes. our work is in the same spirit, as they utilize a general lexical resource, propbank, rather than a limited inventory of pre-defined predicates as in tasse and smith 2008 . the corpus provides semantic graph annotations of (i) recipe-related entities, (ii) generic verb relations (from propbank) connecting these entities, (iii) zero anaphora verbs having implicit arguments, and (iv) textual descriptions of those implicit arguments. the corpus however, does not contain state changes between entities. yamakata et al. 2020 introduce a corpus of annotated english recipes. the annotation is a flow graph (i.e., dag with a single root) including entities and relationships between these entities. the direction of edges also indicates dependencies between actions. the label of edges explicitly specify the state change of entities. while their graph representation is similar to ours in many respects, they do not encode coreference or bridging relations. there are newly emerging datasets focusing on both anaphora and bridging, many of them released as part of the most recent shared task on anaphora and bridging relation detection @xcite . unfortunately, procedural datasets were not included in this task. ## cutl dataset and annotation scheme procedural texts, such as recipes, are interesting to cl researchers for several reasons. one of those is that they are step-driven narratives requiring minimal temporal ordering recognition. as a result, semantic interpretation can focus on the changes that are taking place in the course of a sequence of events in the narrative, while assuming that the events are temporally ordered in a narrative progression. the goal of our cutl annotation is to create a dataset of cooking recipe texts annotated with the following information: the relations we adopt reflect the view laid out in @xcite , which distinguishes nearidentity from (true) identity when drawing coreference relations between referring expressions. thus we identify those relations derived from i/o as near-identities and other non-i/o coreference relations as true identities. ## coreference resolution with cut we implemented a coreference resolution system using cutl dataset. this section will describe the system design and the performance of the system. experiment setup under the poem and cut relations, coreference "chains" now can include phantom entity mentions (with res. prefix). these phantom mentions serve two purposes; 1) make all event outputs explicit, 2) fill syntactic drop arguments in the following event. however, these mentions textually do not exist in the text, thus cannot be easily modelled by any language modelbased system, that are based on vector embedding of the surface text. to address this problem, we adopted dense paraphrasing (dp), a text enrichment technique @xcite to first recover all drop argument (as empty slots) and "paraphrase" drop argument nodes and res. nodes, to create natural language representation of the cutl annotated data. concretely, we apply both prefixp and subgraph-gpt methods from @xcite to all the drop and phantom entities from the recipe to generate paraphrases. prefixp is a heuristic method that paraphrase the entities by prepending the prefixes to reflect changes due to actions. subgraph-gpt use the gpt-3 model @xcite to paraphrase the linearized subgraph that is rooted from the drop argument node or the res. node. figure 3 shows an example from the different paraphrasing methods. once the text is replaced with paraphrase with recovery of of drop arguments and insertion of generalized result nodes, we can use the new text in a coreference resolution task. 4 for the coreference resolution task, we adopt the neural coreference model and the configuration from @xcite and formulate the problem into a joint training of an antecedent assignment model @xcite and a classification model. the system first detects all possible mentions of coreference. then, for cui resolution, the coreference resolution model would learn to assign a set of antecedents to each mention. and for cut resolution, the bridging model will do a multi-class classification on each pair of mentions detected. given the hierarchical nature of our cutl types, we design the coreference resolution in two fashions: (i) coarse: we only consider if there is transformation due to event, so there will only be cui and cut, and we treat all sub-relations under cui as coreference. (ii) fine-grained: we consider each relation type as an individual class and only treat identity as coreference. machine learning model details we train a neural coreference resolution model with a configuration similar to @xcite . specifically, we use 300-dimensional glove embeddings @xcite with window @xmath0. and we train elmo embeddings @xcite on both cutl and reciperef corpora. we also trained a cnn model with windows of 3, 4, and 5 to learn the character embeddings, and concatenate all three embeddings as the token representation. for each experiment, we do a 5-fold cross validation and train the model for 20 epochs on 4 nvidia titan xp gpus. ## conclusion in this paper, we presented a new dataset, cutl, annotated using a novel integration of integrating event semantics and coreference linking annotation. we applied a process-oriented event model and argument structure as coreference relations between event input(s) and output(s). we showed that using cutl is a very efficient way of analyzing and annotating entity transformation and coreference chains in procedural text, by conducting pilot annotations on cooking recipe text. the cutl dataset and annotation material are available under open-source licenses. additionally, we conducted multi-stage experiments to build the baselines for coreference identifier and classifier that focus on utilizing our human annotations. the results from the coreference resolution systems show that the subgraph representation of our annotation is a good resource for llms such as gpt-3 to generate reliable paraphrases in natural language, which can further improve the multi-class coreference resolution task.