Chop and Change: Anaphora Resolution in Instructional Cooking Videos (2022)

Abstract: Linguistic ambiguities arising from changes in entities in action flows are a key challenge in instructional cooking videos. In particular, temporally evolving entities present rich and to date understudied challenges for anaphora resolution. For example “oil” mixed with “salt” is later referred to as a “mixture”. In this paper we propose novel annotation guidelines to annotate recipes for the anaphora resolution task, reflecting change in entities. Moreover, we present experimental results for end-to-end multimodal anaphora resolution with the new annotation scheme and propose the use of temporal features for performance improvement.

## introduction anaphora resolution is the task of identifying the antecedent of an anaphor, i.e., find a language expression that a given entity refers to. for example, in the sentence take a potato and wash it, the pronoun it is an anaphor that refers to the antecedent a potato. this is a challenging nlp task which has been attracting much attention @xcite @xcite . different types of anaphoric relations have been identified and described in the scientific literature, e.g., identity @xcite , near-identity @xcite , and bridging @xcite . recipes provide a rich source for referring expressions @xcite of transformed entities, and offer a challenge for anaphora resolution tasks. @xcite use written recipes with anaphora annotations to trace the temporal change of entities. while the ingredients undergo physical or chemical change in the action flow, they can be still referred to in the same way. for example, an egg before and after it is boiled can be referred to with the same noun egg. compared to text recipes, instructional cooking videos raise additional challenges for anaphora resolution owing to chop the bread mix the cubes with mixture ... their intrinsic multimodality @xcite . @xcite point to various "channels of information" in the transmission of each modality. a "shared reference of entities" is introduced when two modalities refer to the same description @xcite . as presented in cooking instructions of videos when two modalities refer to the same entity, the use of a referring expression is affected by both modalities. for example, the cubes is used in figure 1a to denote the bread pieces in the text modality because the instruction chop the bread shaped them into cubes in the video modality. the choice of referring expressions might also differ with respect to the changes of the entities. in figure 1b the same nominal phrase refers to a different 1 object (the whole salmon piece; and then one of the halves) whereas in figure 1c a coreferential pronoun is used although the object has changed. figure 1c is in fact the most well-behaved in terms of keeping the language expressions consistent across instructions and with the entities being referred to. figure 1d shows the use of null arguments: the second instruction cook in the oven does not explicitly mention what to cook, whereas the image of the instruction displays it. the main contributions of this paper are as follows: (i) we propose an anaphora annotation scheme for instructional cooking videos that allows us to address linguistic ambiguities in anaphora resolution. in particular, we define different types of anaphoric relations to keep track of spatio-temporal changes of entities. we also provide a clear definition of "identity of reference" and specify categories that make an essential change resulting in a different entity. (ii) we annotate the youcookii dataset @xcite according to our scheme and make it publicly available. foot_0 (iii) null anaphors, e.g., mix in the bowl, are included in the annotation thanks to cooking videos that offer the precise visual observation of null anaphors to annotators. (iv) we provide a baseline multimodal anaphora resolution model for this dataset. in particular, we adapt an end-to-end @xcite coreference model for the anaphora resolution task. (v) we offer a novel method to improve anaphora resolution models for instructional language by leveraging temporal features capturing temporal order of instructions instead of using the token distance as @xcite and @xcite . ## related work reference resolution the reference resolution task addresses the linguistic ambiguities in state changes of entity mentions by linking the entities to their corresponding instructions @xcite @xcite , e.g., the mashed potato and the fork refer to the instruction mash the potatoes with a fork. we depart from this type of approaches, as they rely on unsound ontological assumptions (actions/events and entities are different objects) and they introduce unnecessary semantic ambiguities (by linking different entity mentions to the same instruction). anaphoric relations: identity, near-identity, association. anaphoras mainly come in two forms: coreference and bridging. coreference is defined as language expressions referring to the same entity @xcite , whereas bridging is an anaphoric phenomenon based on a non-identical associated antecedent via lexical-semantic, framebased, or encyclopedic relations @xcite . a coreferring anaphor and its antecedent in a text refer to the same entity (identity relation), e.g., a black mercedes and the car, while in bridging, an anaphor and its antecedent refer to different entities (non-identity relation), e.g., the car and the engine in the utterance i saw [a black mercedes] parked outside the restaurant. [the car] belonged to bill. [the engine] was still running. @xcite . as @xcite point out, bridging studies so far employ various methods to describe bridging dissimilar to the coreference definition. nevertheless, both the concept of sameness in the coreference definition and the bridging associations neglect the changes referents may undergo. therefore, the concept of near-identity was introduced by @xcite as a middle ground between coreference and bridging. it addresses spatio-temporal changes of entities, e.g., the entity postville in the text: on homecoming night [postville] feels like hometown, . . . it's become a miniature ellis island . . . for those who prefer [the old postville], mayor john hyman has a simple . . . . this sample exemplifies the referential ambiguity, arising from two language expressions referring to "almost" the same entity, i.e., postville and the old postville @xcite . @xcite and @xcite claim that the introduction of the additional near-identity category in between coreference and bridging introduces more uncertainty. nevertheless, we consider the near-identity relationship suitable because spatiotemporal changes are essential in recipes and the information they convey describes the visual content. coreference and bridging annotations. coreference is a well studied and clearly defined concept with some noticeable exceptions. in recent years several annotated corpora with different coreference guidelines have been released. ontonotes v5.0 @xcite exclusively focus on coreference using a schema similar to @xcite and wikicoref (ghad-dar and langlais, 2016) with two different relations: one is identity, a symmetrical and transitive relation, and the other appositive for adjacent noun phrases. the extraction of the mentions and the use of prepositions in mentions are crucial questions for corerefence annotation @xcite . there are many extant hypotheses explaining how bridging relations function with different annotation schemes for bridging @xcite . the arrau corpus @xcite consists of general language annotated with bridging relations of noun phrases (such as set membership, subset, possession and unrestricted.) @xcite present isnotes derived from ontonotes with unrestricted bridging relations in addition to ontonotes coreferences. the bashi corpus @xcite is based on ontonotes content and the bridging relations in the bashi corpus restrict the bridging anaphors to be truly anaphoric, i.e., not interpretable without an antecedent. all aforementioned annotation studies focus solely on the anaphoric relation between two discourse entities and neglect the change of entities over time. instructional language raises a novel question in anaphora resolution: the definition of anaphoric relations based on the change of language with entities that undergo change. therefore, reciperef @xcite considers the state changes for preparing the annotation guideline for recipe text based on the chemu-ref @xcite anaphora annotation on chemistry patent documents. reciperef annotation was applied to the recipedb data @xcite ) that was aggregated from recipe websites and each recipe was divided into two parts, the ingredients section, and the cooking instructions. the cooking instructions of recipedb contains only textual instructions without any visual content. the state changes are addressed in reciperef as a subtype of bridging relation, even though bridging is clearly defined as an associative relation in the literature @xcite @xcite . besides, null anaphors are not included in the annotation of reciperef, despite their frequent use in recipes. ## corpus we use the youcookii dataset @xcite that includes manually provided descriptions (i.e., instructions) of actions in the cooking videos. the dataset contains 2,000 unconstrained instructional videos from 89 cooking recipes. the videos provide a visual input of the corresponding objects to observe the changes clearly. to obtain a variety of ingredients and their state changes, we choose at least three random samples for each the 89 cooking recipes for the training set and one sample for the test set. there is no intersection between training and test recipe samples. in total, we have 264 training documents and 89 test documents as shown in recipe a recipe is text containing a list of cooking instructions with a list of ingredients, see figure 2. here, we use the youcookii annotation, all instructions for each video are manually annotated with temporal boundaries and described by imperative english sentences. since the video inputs show the entities and actions clearly, the use of referring expressions and null entities is very common contrary to textual recipes. instruction. each video recipe contains 3 to 15 instructions. each instruction is a temporallyaligned imperative sentence that is described according to the corresponding action on the video by human annotators. the instructions are not uttered by the instructor of the video but annotated by the human annotator from a third-person viewpoint while watching the video. each instruction defines an action, i.e., a predicate, applied to a set of objects, i.e., entities. video segments provide the visual status of the spatio-temporal changes for the mentioned entities for each instruction. unlike other common types of texts, cooking instructions focus on processes and entities undergoing change during the process. so, the corresponding videos in the youcookii dataset enable us to comprehend the use of referring expressions of entities for each change. ## annotation categories and guidelines in this section, we explain our strategy of mention selection and the use of our annotation schema on the youcookii data. ## method in this section, we present our end-to-end multimodal anaphora resolution model. figure 3 shows our joint neural model similar to @xcite and @xcite , adapted from @xcite . we extend the model with novel temporal features, see section 5.3. ## conclusion and future work we introduce a novel anaphora annotation scheme including the state changes of entities and nearidentical relations. this fresh approach relies on video inputs for visual observation for anaphora annotation. likewise, we provide baseline anaphora resolution results with novel temporal features on the annotated data. in future work, the mention detection model will be designed to perform with null entities and singleton mentions to improve the performance of the end-to-end model. additionally, different visual feature extraction methods for single frames, e.g., clip @xcite or for videos, e.g., s3d @xcite will be investigated to find the best way of learning from cooking videos for anaphora resolution.