Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension (2022)

Abstract: Procedural Multimodal Documents (PMDs) organize textual instructions and corresponding images step by step. Comprehending PMDs and inducing their representations for the downstream reasoning tasks is designated as Procedural MultiModal Machine Comprehension (M3C). In this study, we approach Procedural M3C at a fine-grained level (compared with existing explorations at a document or sentence level), that is, entity. With delicate consideration, we model entity both in its temporal and cross-modal relation and propose a novel Temporal-Modal Entity Graph (TMEG). Specifically, graph structure is formulated to capture textual and visual entities and trace their temporal-modal evolution. In addition, a graph aggregation module is introduced to conduct graph encoding and reasoning. Comprehensive experiments across three Procedural M3C tasks are conducted on a traditional dataset RecipeQA and our new dataset CraftQA, which can better evaluate the generalization of TMEG.

## introduction multimodal machine comprehension (m 3 c) is a generalization of machine reading comprehension by introducing multimodality. due to its differences from visual question answering (vqa) @xcite in the form of understanding multimodal contexts and conducting multimodal questions and answers, there has been a lot of attention in recent years devoted to this field. in this paper, we investigate a task that has been typical of m 3 c recently, named procedural m 3 c, a task of reading comprehension of procedural multimodal documents (pmds). as shown in figure 1 , a recipe that contains successive multimodal instructions is a typical pmd. reading a recipe seems trivial for humans but is still complex for a machine reading comprehension system before it can comprehend both textual and visual contents and capture their relations. current procedural m 3 c studies @xcite comprehend pmds by encoding text and images at each procedure step. these efforts, however, only scratch the surface and lack deep insight into the elementary unit of pmds, that is, entity. from now on, we use entity to refer uniformly to entity in text and object in image. for instance, the recipe in figure 1 involves multiple entities, i.e., strawberry and sugar cookie dough, etc. in this work, we target at approaching the procedural m 3 c task at a fine-grained entity level. we observe that a pmd essentially assembles an evolution process of entities and the relations between them. specifically, the relation between entities can be summarized in the following two categories: based on the above observations, we believe that simultaneously modeling entities and the temporal and multimodal relations is a key challenge in understanding pmds. recent efforts @xcite are devoted to encoding temporal relations of entities, while it neglects the multimodal relation. heterogeneous graphs have become the preferred technology for representing, sharing, and fusing information to modern ai tasks, e.g., relation extraction @xcite and recommendation @xcite . inspired by this, we construct a heterogeneous graph, with entities as nodes and relations as edges. therefore, the research goal of the procedural m 3 c task will shift from understanding unstructured pmds to learning structured graph representations. in this work, we propose a novel temporal modal entity graph model, namely tmeg. our model approaches procedural m 3 c at a finegrained entity level by constructing and learning a graph with entities and temporal and multimodal relations. specifically, tmeg consists of the following components: 1) node construction, which extracts the token-level features in text and objectlevel features in images as the initial entity embeddings; 2) graph construction, which constructs the temporal, intra-modal, and cross-modal relations separately to form a unified graph; and 3) graph aggregation, which utilizes the graphbased multi-head attention mechanism to perform fusion operations on graphs to model the evolution of entities and relations. finally, the graph representation is fed into a graph-based reasoning module to evaluate the model's understanding ability. in addition, in order to further advance the research of procedural m 3 c, we release craftqa, a multimodal semantically enriched dataset that con-tains about 27k craft product-making tutorials and 46k question-answer pairs for evaluation. we evaluate three representative subtasks, i.e., visual cloze, visual coherence, and visual ordering, on craftqa and a public dataset recipeqa @xcite . the quantitative and qualitative results show the superiority of tmeg compared to the stateof-the-art methods on all three tasks. the main contributions of this paper can be summarized as follows: the experimental results on craftqa and recipeqa show tmeg outperforms several state-of-the-art methods. 2 related work procedural text comprehension. procedural text comprehension requires an accurate prediction for the state change and location information of each entity over successive steps. several datasets have been proposed to evaluate procedural text comprehension, e.g., recipe (bosselut et al., 2018), propara (mishra et al., 2019), and openpi (tandon et al., 2020). to model entity evolution, kg-mrc (das et al., 2019) constructs knowledge graphs via reading comprehension and uses them for entity location prediction. dynapro (amini et al., 2020) introduces a pre-trained language model to dynamically obtain the contextual embedding of procedural text and learn the attributes and transformations of entities. prograph (zhong et al., 2020) enables state prediction from context by constructing a heterogeneous graph with various knowledge inputs. koala (zhang et al., 2021b) utilizes the external commonsense knowledge injection and data enhancement to reason the states and locations of entities. tslm (faghihi and kordjamshidi, 2021) formulate comprehension task as a question answering problem and adapt pre-trained transformer-based language models on other qa benchmarks. real (huang et al., 2021) builds … temporal modal graph aggregation node encoding graph-based multi-head attention add & norm feed forward network add & norm l × graph-based reasoning … … position segment mlp mlp text input step 1: ingredients: 16 oz strawberry, 9 oz blueberries, … step 2: make or buy your sugar cookie dough. … text 1 image 1 image 2 image n a graph-based fusion text 2 text n t … strawberry blueberry orange cls cheese sugar sugar…dough strawberry tcls … … edge encoding inter-modal edge (node-based) inter-modal edge intra-modal edge textual node visual node temporal edge temporal edge (node-based) inter-modal edge (edge-based) temporal edge (edge-based) intra-modal edge a general framework to systematically model the entity, action, location by using a graph neural network. inspired by these previous works, we propose a temporal-modal entity graph model, which is designed with temporal encoding and modal encoding to model multiple types of entities. multimodal graph. in recent multimodal research, graph structure has been utilized to model the semantic interaction between modalities. @xcite propose a graph-based multimodal fusion encoder for neural machine translation, which converted sentence and image in a unified multimodal graph. @xcite convert image regions and the region grounded captions into graph structure and introduced graph memory networks for visual question answering. @xcite propose a multimodal graph fusion approach for named entity recognition, which conducted graph encoding via multimodal semantic interaction. @xcite focus on multimodal sentiment analysis and emotion recognition, which unified video, audio, and text modalities into an attention graph and learned the interaction through graph fusion, dynamic pruning, and the read-out technique. in contrast to the above methods, we formulate our multimodal graph on temporal entities and successfully deploy it in procedural m 3 c. ## proposed method in this section, we introduce: (1) problem definition of procedural m 3 c in section 3.1; (2) the homogeneous graph of each textual instruction (im-age) and our tmeg in section 3.2 and section 3.3, respectively; and (3) graph aggregation module to conduct graph encoding and reasoning in section 3.4. figure 2 gives a high-level overview of tmeg. ## conclusion in this paper, we propose a novel temporal-modal entity graph (tmeg) to approach procedural m foot_1 c. based on tmeg, we introduce graph-based fusion module and reasoning module, which are used to aggregate node features and solve downstream reasoning tasks. what's more, we introduce another procedural m 3 c dataset called craftqa to assist in evaluating the generalization performance of tmeg in different domains and domain transfer. extensive experiments on the recipeqa and craftqa validate the superiority of tmeg. a promising future direction is to introduce temporal-modal entity graphs into the video understanding task @xcite , which also calls for an enhancement of the temporal and the cross-modal reasoning capability.