Multimodal Weighted Fusion of Transformers for Movie Genre Classification (2021)

Abstract: The Multimodal Transformer showed to be a competitive model for multimodal tasks involving textual, visual and audio signals. However, as more modalities are involved, its late fusion by concatenation starts to have a negative impact on the model’s performance. Besides, interpreting model’s predictions becomes difficult, as one would have to look at the different attention activation matrices. In order to overcome these shortcomings, we propose to perform late fusion by adding a GMU module, which effectively allows the model to weight modalities at instance level, improving its performance while providing a better interpretabilty mechanism. In the experiments, we compare our proposed model (MulT-GMU) against the original implementation (MulT-Concat) and a SOTA model tested in a movie genre classification dataset. Our approach, MulT-GMU, outperforms both, MulT-Concat and previous SOTA model.

## introduction information on the internet has grown exponentially. much of this information is multimodal (e.g. images, text, videos, etc.). for example, in platforms like youtube and facebook, multiple modalities can be extracted like video frames, audio and captions on different languages. in this context, it becomes increasingly important to design new methods that are able to analyze and understand automatically these type of multimodal content. one popular scenario is the movie streaming service (e.g. netflix, prime video, etc.), where there is also an increasing interest in performing automatic movie understanding. in this paper we take as a case study the task of movie genre prediction. our proposal exploits movie trailer frames and audio, plot, poster and a variety of metadata information, via deep learning techniques that have enough flexibility to fuse and learn to weight from all these modalities in a simultaneous way. the success of the transformer architecture @xcite and its variants in nlp, has also inspired researchers to propose and extend these architectures in multimodal settings. some examples include vilbert @xcite , @xcite , @xcite , uniter @xcite , mmbt @xcite and lxmert @xcite . however, the vast majority of these multimodal architectures were designed and tested only on bimodal data, more specifically, on text and visual information. besides, models that only allow for early fusion, have the disadvantage that they rely solely on this mechanism that hinders the interpretability. while in models that output a feature per modality, an additional late fusion mechanism can be implemented to further fuse modalities and learn a richer representation, which is the case for the mult model. nonetheless, late fusion in this model was originally performed by means of concatenation, diminishing its fusion capacity. contributions of this work are twofold: we first adapt the mult model @xcite to support additional number of modalities. then, we consider a mechanism that learns to fuse all the modalities dynamically before making the prediction over each particular instance. this is a crucial step, given that for movies belonging to different genres, the relevant modalities could be quite different. for example, in animation movies, visual information might be more relevant given the visual style, while for drama movies, sound may be more helpful because of loud noises and screams. in order learn to fuse the final representation of each modality we propose to adapt the gmu module @xcite . these units are highly interpretable gates which decide how each modal-ity influences the layer output activation units, and therefore, decide how relevant each modality is in order to make the prediction. this is a crucial step, given that for this task, not all modalities are going to be equally relevant for each observation, as has been shown is previous work like @xcite and @xcite . our evaluation shows that our mult-gmu model, which uses weighted fusion by gmu, can outperform sota results in the movie genre classification task by 4%-10% on all metrics (µap, map and sap). we explore for the first time the use of the mult in the movie genre prediction task. we demonstrate that the original mult model, which uses late fusion by concatenation (mult-concat) can achieve sota results task for this . then, we show that further improvements can be achieved by our proposed model with the gmu module (mult-gmu). the contributions can be summarized as follows: ## approach in sections 2.1 and 2.2, we briefly describe the mult architecture and then explain how to adapt the gmu units at the top of the model to perform a more robust late fusion of modalities. ## qualitative analysis to understand how the gmu units are weighting the relevance of each modality according to each instance (movie) i, we inspected the gates z i of the gmu module for all the observations in the test set. to achieve this, we selected the observations that contained each of the genres and averaged the gate activations per modality. we show results for 5 different movie genres in figure 2 , where each row already takes into account the average of all test movies of the corresponding genre. in general, text and visual modalities were the most relevant according to the gmu module. we can see relatively low activations for the audio modality compared with the other ones. this is expected as it has been shown that audio modality is not as useful as the other ones, for this task ( mangolin et al. (2020), cascante-bonilla et al. (2019) ). there is also a relationship between audio and video signals. in genres where video is the strongest, audio is the weakest. taking the audio modality as an example, where horror and drama had the highest gmu activations overall, we could think that this was the case given that this kind of movies usually have loud noises like screams in the trailers, so this could be a good indicator that the movie is likely to belong to one of these two genres. there are other interesting scenarios, for example the text modality had the highest activation for genres like comedy and drama. in the case of the video modality, comedy and family genres had the highest activation. we only show the activations for 5 movie genres. ## conclusion we proposed an adapted version of the multimodal transformer, mult-gmu, by performing weighted late fusion with a gmu module. this approach achieved sota results in the multimodal movie genre classification task. moreover, we improved the interpretability of the mult model by performing a qualitative analysis, visualizing the activations of the gmu module, which allowed us to have a better understanding about relevant modalities for the model, depending on the genre of the movie. to the best of our knowledge, this is the first time multimodal transformer-based architectures are tested in the task of movie genre classification.