MMAR : Multilingual and Multimodal Anaphora Resolution in Instructional Videos (2024)

Abstract: Multilingual anaphora resolution identifies referring expressions and implicit arguments in texts and links to antecedents that cover several languages. In the most challenging setting, cross-lingual anaphora resolution, training data, and test data are in different languages. As knowledge needs to be transferred across languages, this task is challenging, both in the multilingual and cross-lingual setting. We hypothesize that one way to alleviate some of the difficulty of the task is to include multimodal information in the form of images (i.e. frames extracted from instructional videos). Such visual inputs are by nature language agnostic, therefore cross- and multilingual anaphora resolution should benefit from visual information. In this paper, we provide the first multilingual and multimodal dataset annotated with anaphoric relations and present experimental results for end-to-end multimodal and multilingual anaphora resolution. Given gold mentions, multimodal features improve anaphora resolution results by ~10 % for unseen languages.

## introduction a procedural text is a sequence of instructions describing how to create or change an object in a certain way. among the many genres of texts, procedural texts are the ones related to real-world applications such as robotics with human-robot interaction @xcite , video understanding of how-to videos @xcite @xcite , etc. procedural text understanding requires a system to track how a given entity undergoes change. however, using language for describing entity changes may raise linguistic ambiguities, which are a key challenge, especially in instructional videos. in particular, temporally evolving entities present rich and, to date, understudied challenges. cooking recipes provide a paradigmatic source of potentially ambiguous referring expressions for ingredients that test train roll the dough place on a hot pan gebe in eine heiße pfanne rolle den teig aus ... ... ... ... ... ... ... are undergoing physical or chemical changes such as chopping, boiling, mixing, etc. @xcite . existing approaches include huang et al. @xcite , who suggest reference resolution, or @xcite and @xcite , who propose anaphora resolution for tracing the temporal change of entities in recipes. anaphora resolution is the task of identifying the antecedent of an anaphor, where the antecedent is a language expression in the previous context that a given anaphor refers to @xcite @xcite . anaphoric language expressions can occur in many forms, such as sentences, nominal phrases including pronouns, and challenging zero anaphora (a.k.a, implicit arguments or null pronouns) which are not verbalized. an example of such a case is given in figure 1 , where the zero anaphora ϕ in the instruction place ϕ on a hot pan refers to the antecedent the dough in the previous context of the recipe. while anaphora resolution is classically seen as a uni-modal textbased task, there have been some attempts at adding modalities. @xcite present the first attempt to formulate a multimodal anaphora reso-multilingual multimodal parallel anaphorcity zero-pronoun paws @xcite 2018 ) ✓ ✗ ✓ c ✓ corefud 1.0 (nedoluzhko et al., 2022) ✓ ✗ ✓ c ✓ parcorfull2.0 (lapshinova-koltunski et al., 2022) ✓ ✗ ✓ c ✓ crac (žabokrtský et al., 2022) ✓ ✗ ✗ c ✗ cin (goel et al., 2022) ✗ ✓ ✗ c ✗ find-2-find (oguz et al., 2023a) ✗ ✓ ✗ b,n,c ✓ vwp (hong et al., 2023) ✗ ✓ ✗ c ✗ mmar (ours) ✓ ✓ ✓ b,n,c ✓ multilingual coreference resolution has been researched extensively: for example, @xcite show the importance of languageindependent hand-crafted features, and @xcite demonstrate the effectiveness of deep learning methods for multilingual coreference resolution. however, to the best of our knowledge, we have made the first attempt to investigate multimodal representations for multilingual anaphora resolution in procedural texts. we show that visual inputs provide important language-agnostic features and improve multilingual resolution. our contributions are as follows: (1) we provide novel multimodal parallel data 1 in english, turkish, and german, which are annotated with anaphora resolution, including zero anaphora; (2) we provide a novel model architecture for multilingual and multimodal anaphora resolution with mention detection that outperforms a strong baseline @xcite ; (3) we show that multimodal representations outperform language-only multilingual representations on unseen data for anaphora resolution. to our knowledge, our dataset is the first and only multimodal parallel anaphora resolution dataset for multiple languages. ## related work our work is at the intersection of two research areas: anaphora resolution and procedural text un-1 https://github.com/oguzcennet/mmar derstanding. in this section, we summarize related work and emphasize the most important common points and differences between our data and already existing data sets, as well as our approach and previous methods. ## data language data to create our data set, we started with available english multimodal anaphora resolution data and translated these data into more languages. in this study, we build on the chop&change anaphora resolution dataset and an-notation schema of @xcite for two reasons. first, it includes near-identity anaphoric relations (unlike other approaches), bridging, and coreference for noun phrases, pronouns, and null pronouns (a.k.a, implicit arguments or zero anaphora). second, cooking videos of the recipes are provided with annotations for videoinstruction alignment. in total, in our work here we provide 400 multimodal training recipes and 100 multimodal test recipes in english; for data statistics, see appendix a. we then manually translate each recipe into turkish and german by native language speakers based on the original english recipe and video inputs, refer to a. as much as possible, we keep the sentence structure of the instructions while translating manually. for example, we do not drop the direct or prepositional objects or expand any zero anaphora in the source in the translations. additionally, we use a pronoun in the translation whenever a pronoun is used for an entity. however, we also prepare and analyze pro-drop data in turkish a. we keep the linguistic feature distribution as in the original english documents and manually apply the anaphora annotation to the other languages based on the annotation of english recipes. an example of a german translation and annotation of an english recipe is provided in figure 2 . in total, 1,200 training and 300 test recipes, a third each in english, german, and turkish, are included in the multilingual and multimodal mmar language dataset, with recipes in each language making up equal parts of the dataset. for detailed information regarding the translation and annotation of turkish and german recipes, refer to appendix a. visual data for each video, procedural steps in a recipe are annotated with temporal boundaries and described by imperative instructions in the text. in other words, each recipe instruction is aligned to a part of the corresponding cooking video. for example, the instruction 'thinly slice the beef' in figure 2 is annotated with the corresponding starting and ending times of the segment in the video. following @xcite , each textual instruction is temporally aligned to a segment of the corresponding cooking video. because of the mapping to a whole instruction video segment, a gold mention has no precise alignment to a specific frame. thus, zhou et al. (2018 to select the best frame and the best region for the instruction in a preprocessing step (i.e., clip is not trained with the rest of the model). to investigate the contribution of multimodal features, we train and evaluate our model with and without visual features of each gold and candidate mentions. we present the outcomes of two pre-trained video encoders used for extracting multimodal features in appendix a.0.2. ## experiments we conduct various cross-and multi-lingual experiments to investigate the impact of visual features on anaphora resolution. the main body of the paper focuses solely on english monolingual experiments incorporating multi-lingual and frame features, while the appendix b details experiments involving turkish and german, along with video encoders. ## results and discussion in overview. we present the anaphora resolution results in the cooking domain with multimodal and multi-lingual features for both candidate and gold mentions in english, german, and turkish, in two settings: training on mono-lingual data only, i.e., a cross-lingual setting, and training on all three languages simultaneously, i.e., a multi-lingual setting. overall, the results with candidate mentions in ## conclusion and future work in this study, we presented a novel multilingual and multimodal dataset for anaphora resolution. additionally, we empirically validated the benefit of multimodal features on anaphora resolution in a multilingual setting. we hope that our work will help to show that using additional languageagnostic modalities, such as the vision modality, can help to bridge gaps between languages, especially if training data are sparse or unavailable. our results also indicate that mention detection is a bottleneck for anaphora resolution for unseen languages that conceals the positive impact of visual features. thus, span-based language models like spanbert @xcite need to be developed for multi-lingual settings.