Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization (2023)

Abstract: In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, “oil” mixed with “salt” and “pepper” could later be referred to as a “mixture”. Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression “mixture”, and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as Visual-linguistic Ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of visual-linguistic ambiguity , consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization with one joint model in multitask learning, as compared to a strong single-task baseline.

## introduction deep neural networks have achieved enormous success in various language and computer vision tasks, such as multimodal understanding using video-text and image-text data @xcite @xcite @xcite . however, many current systems require a large number of accurate annotations, including image-level labels, location-level labels (bounding boxes and key points), and pixellevel labels. a specific type of video data with naturally occurring semi-aligned texts are narrated instructional videos. such videos are available in large quantities (e.g. on youtube) and often chosen for learning add red wine to the pan add the beef into a hot pan boil pierogi in boiled water fry them with onion and butter figure 1: examples of visual and linguistic ambiguities. figure 1 ) represents the visual ambiguity related to which specific pan (in figure 1a ) is referenced with the phrase the pan because many pans occur on the stove. figure 2 ) shows the linguistic ambiguity with the use of the pronoun them (in figure 2b ). in this work, we focus on modeling cases where both ambiguities are intertwined, a phenomenon we refer to as visual-linguistic ambiguity. ... ... ... 1 provides a motivating example: to find which pan in (1b) is denoted by the textual span the pan, we need to find its antecedent in the preceding text. the visual object localization of the textual antecedent the hot pan in (1a) then includes supplementary visual information about the correct pan: the referent is the hot pan with beef in it. in figure 1 , to find the correct location of the object referred to as them in (2b), we first need to find the textual antecedent to understand what them refers to. when identifying pierogi as antecedent, we can use the pierogi for the visual object localization of them. an even more complex case is null pronouns (figure 2 ): to do object localization and anaphora resolution, the first requirement is the detection of the null pronoun, which then needs to be resolved and located in the image. a located visual object of a null pronoun then assists in finding the textual antecedent, and a resolved null pronoun helps to apply object localization. our guiding idea in this work is that anaphora resolution, the task of connecting linguistic expressions such as the anaphor (i.e., the repeated reference) and its antecedent (i.e., the previous mention in the document). @xcite @xcite , and object localization, the task of identifying the location of one or more objects in an image and drawing bounding boxes around their visual space @xcite @xcite , can jointly help to resolve visual-linguistic ambiguities. to test this we propose a multitask learning neural model for jointly resolving visual-linguistic ambiguity. our contributions are two-fold: first, we present a new dataset foot_0 (section 4), find2find, for the joint evaluation of anaphora resolution and object localization based on an extension of chop&change @xcite . our new data set contains 500 recipes with annotated anaphora and associated object localization. together with the new data set, we propose the new task of multimodal resolution of visual-linguistic ambiguities. the task provides a unique opportunity for models to fuse text and vision information for solving anaphora resolution and object localization at the same time. second, we present a new multitask learning system 1 for modeling the two tasks of anaphora resolution and object localization jointly, using a fusion of visual and textual data. our experiments show that information from each of the tasks mutually benefits performance on the other task. our idea is based on the fact that in both tasks, the goal is to extract mentions from a given text: what connects object localization and anaphora resolution is that in visual object localization, a corresponding language expression needs to be found, and in anaphora resolution, accurate spans that resolve the anaphoric relations between the anaphor and the antecedents need to be identified. ## conclusion and future work in this work, we study the problem of visuallinguistic ambiguity in multimodal data and propose the novel task of joint anaphora resolution and object localization. we create the find2find dataset for object localization and anaphora resolution with cooking recipes to improve the performance of both tasks by exploiting their commonalities. we implement a model for the joint learning of anaphora resolution and object localization, fusing visual and textual information, and show empirically that a multitask learning paradigm mutually improves both tasks. especially, the results of zero anaphora resolution indicate that object localization helps to avoid linguistic ambiguity of null pronouns. our joint multitask learning approach does not apply the temporal features of evolving visual entities even though the temporal features of textual recipes @xcite are included due to instruction order. in future work, we claim the temporal encoding of visual objects can improve the results of joint anaphora resolution and object localization. luowei zhou, nathan louis, and jason j. corso. 2018a. weakly-supervised video object grounding from text by loss weighting and object interaction. in bmvc. luowei zhou, chenliang xu, and jason j. corso. 2018b. towards automatic learning of procedures from web instructional videos. in aaai. dimitri zhukov, jean-baptiste alayrac, ramazan gokberk cinbis, david fouhey, ivan laptev, and josef sivic. 2019. cross-task weakly supervised learning from instructional videos. in proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 3537-3545. add the onions boil potatoes and sweet potatoes for 5-7 minutes heat some oil in a pan give a further chop on the onions mix [∅] shape the mixture into small patty fry patties in a pan with some oil 1. 2. 3. 4. shred the potatoes and sweet potatoes to a bowl add garlic salt , black pepper and chilli powder 5. 6. 7. 8. 9. 10. cook the onions in it