Dynamic Regularization in UDA for Transformers in Multimodal Classification (2023)

Abstract: Multimodal machine learning is a cutting-edge field that explores ways to incorporate information from multiple sources into models. As more multimodal data becomes available, this field has become increasingly relevant. This work focuses on two key challenges in multimodal machine learning. The first is finding efficient ways to combine information from different data types. The second is that often, one modality (e.g., text) is stronger and more relevant, making it difficult to identify meaningful patterns in the weaker modality (e.g., image). Our approach focuses on more effectively exploiting the weaker modality while dynamically regularizing the loss function. First, we introduce a new two-stream model called Multimodal BERT-ViT, which features a novel intra-CLS token fusion. Second, we utilize a dynamic adjustment that maintains a balance between specialization and generalization during the training to avoid overfitting, which we devised. We add this dynamic adjustment to the Unsupervised Data Augmentation (UDA) framework. We evaluate the effectiveness of these proposals on the task of multi-label movie genre classification using the Moviescope and MM-IMDb datasets. The evaluation revealed that our proposal offers substantial benefits, while simultaneously enabling us to harness the weaker modality without compromising the information provided by the stronger.

## introduction multimodal machine learning focuses on methods for modeling information from more than one modality, such as text, image, audio, and video. it is crucial for models to simultaneously analyze and organize diverse modalities. recent advancements in multimodal applications have primarily been attributed to the availability of new large-scale multimodal datasets, increased computational capacity, and enhanced represen-tations of individual modalities. this development encompasses a broad scope, for example, multimodal classification @xcite , gesture recognition @xcite , and audio-video clustering @xcite . multimodal machine learning is a great challenge due to the multiple factors involved. in this work, we focus on multi-label movie genre classification utilizing two modalities: text and image. we focus on addressing four main difficulties: (i) models trained on different modalities learn and generalize at different speeds @xcite , and the optimization process benefits one modality more than the other. the consequence is that during the training phase, the model gives more weight to the most relevant (stronger) modality and disregards the less informative (weaker) one. (ii) there is a lack of effective fusion since prediction by fusion modalities depends on the correlation between modalities and their representation. (iii) complex deep neural architectures require a large computing capacity, and most multimodal models use complex modules for fusion that increment the computational cost. (iv) multimodal models are prone to overfitting since they have more parameters @xcite . to tackle these issues, our proposal focuses on model regularization and better use of weak modality information (e.g., image). specifically, for (i) and (iii), we design a two-stream model called multimodal bert-vit (mmbv). our proposed mmbv consists of two transformers @xcite , one for text and one for image, interconnected by the cls token. thereby, mmbv combines the modalities using self-attention without a complex fusion module, unlike most of the previous classification approaches that are based on large complex neural modules that increase computational cost and are prone to overfitting @xcite . mmbv is possible due to the recent creation of the vision transformer (vit) @xcite , a transformer pre-trained in a large image dataset that incorporates the cls token for classification similar to text transformers. for (ii) and (iv), we design a ratio for the specialization and generalization of a model at each epoch to identify the onset of overfitting and correct it. we add the ratio to the consistency training framework unsupervised data augmentation (uda) @xcite . the intuitive idea is to obtain a score that automatically increases or decreases the contribution of the consistency loss in uda during training. for multimodal classification, we can augment only one modality or the two modalities jointly. to reduce the breach between the learning speeds of different modalities, we experiment with augmenting only the strong modality. we evaluate these ideas in the challenging scenario of multi-label classification of movie genre using two datasets @xcite . our results show that combining mmbv with dynamic uda improves the performance of both image and text modalities by balancing between the weak and strong modalities. we also find that data augmentation in just one modality helps the other one. the data augmentation retards the training, which may reduce the difference between the learning rates of different modalities. overall, the dynamic uda improves the base model performance across all models tested. ## cls fusion model: mmbv we introduce the multimodal bert-vit (mmbv) model 1 , a novel two-stream model for text-image classification that combines two transformers through the cls token. see figure 1 for an illustration of the architecture. we use as image encoder the novel pre-trained transformer for image vit @xcite , and for the text encoder, we use the transformer bert @xcite . in this way, both modalities have an encoder with a similar architecture, where the encoders are pre-trained in large datasets for each domain. since both models have a token (cls) that resumes the information of the input sequence, we do the fusion by interconnecting this token from one modality to the other. ## proposed dynamic unsupervised data augmentation bert vit [clstxt] t oken1 . . . t okenr position embeddings ecls txt e1 txt . . . er txt hcls txt h1 txt . . . hr txt hcls i-t adding hcls img hcls txt h1 txt . . . hr txt hcls img text pooled out uda @xcite to supervised multimodal classification. uda has not yet, to the best of our knowledge, been used for this task. the purpose of uda is to force model consistency with realistic noise by applying data augmentation techniques. we propose a dynamic regularization of the loss function that is useful when there is not a large unsupervised data set and the computational resources are limited. ## dataset and experimental settings datasets: we used two multimodal datasets, moviescope @xcite and multimodal imdb (mm-imdb) @xcite . the task is the multi-label classification of a movie's genre based on its plot and image poster. the performance metrics are macro and micro f1 for moviescope and macro and micro average precision for mm-imdb. since both datasets have unbalanced labels, the most important metric is the micro. mm-imdb is the largest dataset since its training partition is almost five times larger than the one for moviescope. moviescope has 13 movie genre categories, and mm-imdb has 26 categories. these categories include genres such as action, animation, biography, comedy, crime, drama, family, fantasy, horror, and others. ## results this section presents the results obtained by the proposed model mmbv and the proposed framework dynamic uda for the task of multilabel movie genre classification. we compare mmbv performance against its unimodal parts, vit @xcite and bert @xcite . also, we compare mmbv against multimodal bitransformer (mmbt) @xcite , which reportedly holds the best-performing result in mm-imdb dataset without a multimodal pertaining. for the second proposal, since the dynamic uda framework does not depend on the model, we experiment with three unimodal and two multimodal models for the same task. ## conclusions we developed two strategies to tackle key issues in multimodal learning for the text-image classification task. first, to address the lack of effective fusion and excessive computational complexity, we designed a text-image model mmbv for classification. mmbv has shown an effective fusion due to a similar use of the modalities and the cls fusion. the second proposal, dynamic uda, focuses on reducing the gap between the learning rates of the different modalities and having an effective regularization that automatically identifies the overfitting and corrects it. we successfully modified the uda framework by extending it to a multimodal supervised domain and overcoming the sensitivity of uda to the choice of regulation parameter. future work includes extending the model to other image-text tasks and more than two modalities and applying the dynamic uda to other general classification tasks. since dynamic uda can be used to extract more information from the weaker modality in analogous scenarios, our work could have an impact on virtually any multimodal or multichannel application.