MM - GATBT : Enriching Multimodal Representation Using Graph Attention Network (2022)

Abstract: While there have been advances in Natural Language Processing (NLP), their success is mainly gained by applying a self-attention mechanism into single or multi-modalities. While this approach has brought significant improvements in multiple downstream tasks, it fails to capture the interaction between different entities. Therefore, we propose MM-GATBT, a multimodal graph representation learning model that captures not only the relational semantics within one modality but also the interactions between different modalities. Specifically, the proposed method constructs image-based node embedding which contains relational semantics of entities. Our empirical results show that MM-GATBT achieves state-of-the-art results among all published papers on the MM-IMDb dataset.

## introduction despite the huge success of learning algorithms for applications involving unimodal data such as text, less is known for applications involving multimodal data, i.e. scenarios where each data entity has data attributes from multiple modes, such as text and image. while the previous works show that models with multimodal representation outperforms unimodal representation in downstream tasks such as classification, vqa, and disambiguation, the benefit of multimodal representation mostly comes from only one mode (such as text), while the other mode only contribute a marginal improvement. that is, the performance difference between text-only representation and multimodal representation is smaller than that of the image-only representation and multimodal representation @xcite @xcite @xcite . we suspect that improper usage of imagemodality presents a limitation in creating multimodal representation. existing multimodal models figure 1 : given movie poster and text information, the problem is to predict the multilabel genres of movies. our method narrows down this problem into a node classification task by constructing a multimodal entity graph where each node represents a movie entity and edge represents a shared feature between the movie entities. we demonstrate our claim by considering a multilabel classification task using the mm-imdb dataset @xcite as in figure 1 . in the mm-imdb dataset, each movie entity is provided with image and text, and our goal is to predict the movie's genre. using this data, we construct a graph where each node represents a movie, and is given the movie image as an attribute. furthermore, we connect two nodes if the corresponding movies share features, i.e. if they have the same producer, director, etc. by capturing dependency and interaction between the entities generated from graph attention network (gat) @xcite , we expect to gain latent information that cannot be extracted from the image encoder solely. the contributions of this work are as follows: (1) we propose a novel multimodal graph attention network (mm-gatbt) which enables interaction between data modalities. (2) to our best knowledge, this is the first attempt to construct imagebased entity graph to enrich image representation by capturing relational semantics between the entities. (3) mm-gatbt achieves state-of-the-art results on the multilabel classification task among all published papers on mm-imdb dataset. graph neural network graph neural network (gnn) is powered by neural message passing and generates node embeddings. a graph @xmath0, e) is defined as a tuple such that v is a set of vertices and e ⊆ v × v is a set of edges. we also employ the node feature matrix @xmath1. vanilla gnn @xcite averages neighbor messages for each layer using the mean aggregation function. formally, it is defined by the following eq. ( 1 ) where l is the layer index, h l i is hidden representation of node i at layer l, and u l is a learnable parameter. here, deg i and n i denote the degree and the neighbor set of node i, respectively, and σ(.) is a nonlinear activation function. graph convolution network (gcn) @xcite improves vanilla gnn by employing symmetric normalization @xcite . this model runs a spectral-based convolution operation. because the spectral method assumes fixed graph, it often leads to poor generalization ability @xcite . therefore, spatial-based models such as graphsage @xcite are often considered to enable inductive generalization. in eq. ( 2 ), [h l-1 i ; h l-1 j ] denotes a concatenated representation between the node's previous hidden state h l-1 i and an aggregated representation of local neighbor nodes h l-1 j where @xmath2. attention mechanism attention mechanism @xcite computes a probability distribution α = (α t1 , α t2 , ...α ts ) over the encoder's hidden states h (s) that depends on the decoder's current hidden state h (t) . @xcite computes global attention by similarly, gat performs attention on the neighbor nodes. ## experiment system configuration during the training phase, we used a single nvidia rtx 3090 with a batch size of 12. we implemented our model using pytorch @xcite and dgl @xcite on top of mmbt code available on the public repository. foot_0 for every encoder, we used pre-trained models to reduce the computational cost and maximize their performance. in the case of the text encoder, we used the bert uncased base model available from hugginface @xcite . for the image encoder, we used pre-trained efficientnet b4 @xcite . for gnn, we chose gat @xcite available from dgl. we pre-trained gat before employing to mm-gatbt. we used five features to construct our graph, as was explained in section 3.3 and eq. ( 4 ) therein. the average degree of the resulting graph is 59 and it has 554 isolated nodes. experiment setup we used multimodal imdb (mm-imdb) dataset from @xcite . this dataset consists of 23351 movie entities. each movie in the dataset has a title, description, movie poster, producer, and related genres. note that each movie can have multiple genres, making this task a multi-label classification task. empirical results from previous works imply that text modality carries more significant importance than image modality @xcite . the dataset is provided in a splitted format where the number of training set and testing set are 15552 and 7799 respectively. ## result we validated our model using the following metrics: micro f1, macro f1, weighted f1, and samples f1. the results are rounded to 3 decimal places. we report our results as well as the state of the art in secondly, mm-gatbt reflects the connectivity structure of the constructed graph. as visualized in figure 3 , the constructed graph consists of both connected and isolated nodes. therefore, it is crucial for the architecture to address the graph's density and sparsity. indeed, the text encoder on the top of figure 2 generates the word embedding neglecting the graph structure, which justifies its high performance on isolated nodes. in contrast, the gat on the bottom of figure 2 takes into account the connectivity of nodes. this well justifies why mm-gatbt also performs well on nonisolated nodes. by fusing these two embeddings, mm-gatbt leverages both connected and isolated nodes effectively. note that neither bert nor image-based node embedding could achieve the accuracy of 0.685 before they were fused. ## conclusion we proposed mm-gatbt, a novel graph-based multimodal architecture, to address the multilabel classification task on the mm-imdb dataset. mm-gatbt leverages image-based node embedding and attention mechanism during the early fusion phase. the results show that the proposed model successfully captures the latent information generated from the interaction between the entities and achieves state-of-the-art results among all published works on the mm-imdb dataset.