Multi Task Learning based Framework for Multimodal Classification (2021)

Abstract: Large-scale multi-modal classification aim to distinguish between different multi-modal data, and it has drawn dramatically attentions since last decade. In this paper, we propose a multi-task learning-based framework for the multimodal classification task, which consists of two branches: multi-modal autoencoder branch and attention-based multi-modal modeling branch. Multi-modal autoencoder can receive multi-modal features and obtain the interactive information which called multi-modal encoder feature, and use this feature to reconstitute all the input data. Besides, multi-modal encoder feature can be used to enrich the raw dataset, and improve the performance of downstream tasks (such as classification task). As for attention-based multimodal modeling branch, we first employ attention mechanism to make the model focused on important features, then we use the multi-modal encoder feature to enrich the input information, achieve a better performance. We conduct extensive experiments on different dataset, the results demonstrate the effectiveness of proposed framework.

## introduction with the easy-access of mobile devices, the world has witnessed the explosion of multimedia data, which contains various modalities, such as image, audio and text. generally speaking, different modality can provide complementary information. however, many previous attempts focus on one single modality, as the multimodal data is more complex. the applications of multimodal data analysis seem to evident in several fields, such as, emotion recognition, medical diagnosis. recently, the development of multimodal machine learning approaches has witnessed growing interest @xcite . on the other hand, deep learning has witnessed dramatically progress in various fields: ranges from computer vision, natural language processing and speech recognition @xcite . due to the great success of deep learning in single modality, great interests have been given for the multimodal deep learning framework @xcite . despite of sustainable efforts have been made, multimodal deep learning is still far from been fully solved, using deep learning. moreover, traditional approach train the classifiers on different modal and weighted average to generate the predictions, which is time-consuming and cannot model the interaction between different modal. in this short paper, a general multimodal data classification task is proposed, leveraging multi task-based deep learning. the framework consists of two branches: multi-modal autoencoder branch and attention-based multi-modal modeling branch. the framework takes the interaction between different modals into consideration. to demonstrate the efficacy and robustness of proposed method, we conduct extensive experiments on different dataset and the results support our claims. ## dataset and evaluation in this paper, we use the adoption prediction dataset from kaggle foot_0 to do our research, which is a real world and challenging dataset. the dataset is composed of three different modal features: tabular features (the basic information about each pet), textual features (the pet description written by english) and visual features (the photo of each pet), and it aims to predict how quickly a pet is adopted. there are 14993 instances in this dataset, and the label is the categorical speed of adoption, there are five different classes from 0 to 4, in details, 0 means this pet was adopted on the same day as it was listed, 1 means this pet was adopted between 1 and 7 days after being listed. figure 1 shows some example instances. besides, in this classification task, due to the number of classes is balanced, we use accuracy to evaluate different models' performance. ## proposed approach in this paper, our proposed approach has two parts: multi-modal autoencoder branch and attentionbased multi-modal modeling branch. ## conclusion in this paper, we proposed the a novel framework for multimodal data classification. the framework consists of multi-modal autoencoder module and attention-based multi-modal modeling module. we evaluate the model on the large-scale multimodal datasets. our framework shows an advantage on accuracy with compared to other approaches. in the future, we will try to extract more features, such as the semantic information of images, thus the similarity or dissimilarity between different modality can be calculated. moreover, our framework could be adapted to other types of multimodal machine learning task, for instance, the detection task. on the other hand, we will conduct more experiments on larger dataset.