Curriculum Learning Meets Weakly Supervised Multimodal Correlation Learning (2022)

Abstract: In the field of multimodal sentiment analysis (MSA), a few studies have leveraged the inherent modality correlation information stored in samples for self-supervised learning. However, they feed the training pairs in a random order without consideration of difficulty. Without human annotation, the generated training pairs of self-supervised learning often contain noise. If noisy or hard pairs are used for training at the easy stage, the model might be stuck in bad local optimum. In this paper, we inject curriculum learning into weakly supervised multimodal correlation learning. The weakly supervised correlation learning leverages the label information to generate scores for negative pairs to learn a more discriminative embedding space, where negative pairs are defined as two unimodal embeddings from different samples. To assist the correlation learning, we feed the training pairs to the model according to difficulty by the proposed curriculum learning, which consists of elaborately designed scoring and feeding functions. The scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, where the pairs with large losses are defined as hard pairs. Notably, the hardest pairs are discarded in our algorithm, which are assumed as noisy pairs. Moreover, the feeding function takes the difference of correlation losses as feedback to determine the feeding actions (‘stay’, ‘step back’, or ‘step forward’). The proposed method reaches state-of-the-art performance on MSA.

## introduction with the rapid development of social media, multimodal data have been widely-used to perform many downstream tasks, including multimodal sentiment analysis @xcite , multi-omics integrative analysis @xcite , human action recognition @xcite , etc. multimodal sentiment analysis, aiming to mine humans' sentiments and opinions from language, audio, and visual sequences, has attracted significant attention. manual annotation of large-scale multimodal datasets is challenging and time-consuming @xcite . many models suffer from overfitting due to limited training samples and the large amount of parameters introduced by complex fusion algorithms @xcite or large pre-trained transformers @xcite , which degrades the generalizability. recently, to leverage the correlation information between modalities inherently stored in each sample, a few contrastive learning frameworks are proposed @xcite @xcite , which generate abundant training pairs of unimodal embeddings across samples to train a robust model. these methods learn the similarity between modalities, such that unimodal embeddings from the same sample are pulled closer while those from different samples or classes are pushed away. however, they feed the generated pairs to the model in a random order without consideration of their difficulty, which might cause inaccurate learning. if hard pairs are used for training at early stage where the model is not well trained, the optimization may be stuck in a bad local optimum @xcite . more severely, without human supervision, the training data of unsupervised or weakly supervised tasks often contain noise. for instance, two modalities from the same sample might convey contradictory semantic meanings. on the other hand, two unimodal pairs can share semantic similarity even if they come from different samples. the noisy pairs are often the hardest pairs that cannot be well predicted by the model, which should be identified to prevent the feature extraction capability from being damaged. in contrast, the education of humans is highly organized, which is based on a curriculum that introduces different concepts at different times according to difficulty, exploiting previously learned concepts to ease the learning of new abstractions. inspired by the mode of human education, we focus on selecting the generated pairs for training dynamically (see fig. 1 ), and follow the idea of curriculum learning which enables training data to be fed into the model in a certain order @xcite @xcite . by choosing which data to present and in which order to present them, we can guide the learning and find a better local minimum for the model. specifically, we first construct a weakly supervised modality correlation learning task to capture the shared information between modalities, where a positive pair and a negative pair consists of unimodal representations from the same sample and from different samples respectively. the task is said to be weakly supervised as it incorporates label information to score the pairs to learn a more discriminative embedding space for downstream tasks (while the score is not always correct). then, we inject curriculum learning into correlation learning which consists of difficulty scoring and pair feeding functions. the scoring function determines the hardness of training pairs based on pre-trained and current correlation predictors, and sorts the training pairs based on hardness. the feeding function determines the pace by which the training pairs are fed to the model. different from traditional methods @xcite , our feeding function has three candidate actions ('stay', 'step back', or 'step forward'), which enables a more refined and accurate feeding strategy. it takes the difference of current and previous correlation losses as feedback to determine the feeding actions. the proposed curriculum learning enables the model to be learned gradually from easy pairs to hard pairs which effectively prevents the model from falling into bad local optimum. to address the issue of noisy pairs, we dynamically identify and discard the hardest pairs during training to minimize the negative effect of noisy data. in brief, the contributions are listed as follows: ## related work with the availability of multimodal data, msa draws increasing attention for its capability to interpret human language and mine sentiments and opinions. many works focus on designing fusion strategies to obtain informative multimodal embedding @xcite . two simple and explicit fusion methods are early fusion (i.e., featurelevel fusion) @xcite @xcite and late fusion (i.e., decision-level fusion) @xcite @xcite . recently, modern fusion methods are proposed to explore intra-/intermodal interactions. specifically, tensor-based methods @xcite draw increasing attention because they can learn joint embedding with high expressive power. graphbased fusion methods @xcite @xcite learn interactions on the graph across time series and modalities. to highlight important information across modalities, many algorithms apply cross-modal attention mechanisms @xcite @xcite . to improve the interpretablility of multimodal fusion, quantuminspired fusion @xcite and capsule-based fusion @xcite are proposed. moreover, a few methods use kldivergence, canonical correlation analysis and so on to regularize the learning of unimodal distributions and better match different modalities @xcite @xcite . with the success of bert @xcite , there has been a trend to fine-tune pretrained transformers using multimodal data @xcite @xcite . however, these methods do not fully leverage the modality correlation information inherently stored in each sample, and may easily suffer from overfitting and have poor generalizability especially when the dataset is small @xcite . more recently, self-supervised learning on multimodal data has attracted significant research attention, where the self-supervised learning tasks typically include modality matching (especially image-text pair) and masked reconstruction @xcite @xcite . another popular strategy of selfsupervised learning is to apply contrastive learning to learn representations by pushing unimodal representations describing the same sample closer, and pushing those of different samples apart @xcite @xcite . for example, hycon @xcite explores intra-modal and intermodal dynamics across samples, which defines two unimodal representations from the same class as positive pairs. however, these methods do not consider feeding the training pairs into the model based on difficulty, which might cause the model to be stuck in the bad local optimum. in contrast, we perform weakly supervised modality correlation learning to learn a more discriminative embedding space for downstream task, and leverage curriculum learning to enable a more smooth and accurate learning. moreover, we also seek to identify noisy modality pairs for minimizing the negative effect of noisy data. curriculum learning has been applied in various tasks and shows promising results @xcite @xcite @xcite . to the best of our knowledge, we are the first to introduce curriculum learning into weakly supervised/unsupervised multimodal learning algorithms. ## experiments in this section, we conduct extensive experiments to evaluate the effectiveness of the proposed wscl-cl. for the lack of space, more details about the introduction of baselines and the feature extraction details are placed in the appendix. ## conclusion we introduce the curriculum learning and inject it into a weakly supervised modality correlation learning task. via curriculum learning, we can feed the model with training pairs based on their difficulties, such that the model does not fall into a bad local optimum. the main novelty of this paper is the design of the curriculum learning, in which the difficulty scoring and pair feeding functions are proposed. the scoring function computes the difficulty of pairs using pre-trained and current correlation predictors, and the feeding function takes the difference in correlation losses as feedback to determine the feeding action. the proposed method achieves state-of-the-art performance on multimodal sentiment analysis.