M 2 DF : Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysis (2023)

Abstract: Multimodal Aspect-based Sentiment Analysis (MABSA) is a fine-grained Sentiment Analysis task, which has attracted growing research interests recently. Existing work mainly utilizes image information to improve the performance of MABSA task. However, most of the studies overestimate the importance of images since there are many noise images unrelated to the text in the dataset, which will have a negative impact on model learning. Although some work attempts to filter low-quality noise images by setting thresholds, relying on thresholds will inevitably filter out a lot of useful image information. Therefore, in this work, we focus on whether the negative impact of noisy images can be reduced without modifying the data. To achieve this goal, we borrow the idea of Curriculum Learning and propose a Multi-grained Multi-curriculum Denoising Framework (M2DF), which can achieve denoising by adjusting the order of training data. Extensive experimental results show that our framework consistently outperforms state-of-the-art work on three sub-tasks of MABSA.

## introduction multimodal aspect-based sentiment analysis (mabsa) is a fine-grained sentiment analysis task @xcite , which has received extensive research attention in the past few years. mabsa has derived a number of subtasks, which all revolve around predicting several sentiment elements, i.e., aspect term(a), sentiment polarity(p), or their combinations. for example, given a pair of sentence and image, multimodal aspect term extraction (mate) aims to extract all the aspect terms mentioned in a sentence @xcite , multimodal aspect-oriented sentiment classification (masc) aims to identify the s: a 1 out and about in [nyc] p 2 :neutral a 2 . ( may, 2 nd ) i: sub-tasks input output multimodal aspect term s + i a 1 , a 2 extraction (mate) multimodal aspect-oriented s + i + a 1 p 1 sentiment classification (masc) s + i + a 2 p 2 joint multimodal aspect-sentiment s + i (a 1 , p 1 ), analysis (jmasa) (a 2 , p 2 ) the assumption behind these sub-tasks is that the image information can help the text content identify the sentiment elements @xcite , so how to make use of image information lies at the heart of the mabsa task. to this end, a lot of efforts have been devoted, roughly divided into three categories: (1) transforming the image into a global feature vector and utilizing attention mechanisms to sift through visual content pertinent to the text @xcite ; (2) partitioning the image into multiple vi-sual regions averagely, and subsequently establishing connections between text sequences and visual regions @xcite @xcite ; (3) retaining exclusively the regions containing visual objects in the image and enabling dynamic interactions with the text sequences @xcite @xcite @xcite . despite their promising results, most of the studies overestimate the importance of images since there are many noisy images unrelated to the text in the dataset, which will have a negative impact on model learning. in the mabsa task, @xcite , @xcite , @xcite and @xcite developed the cross-modal relation detection module that can filter low-quality noisy images by setting thresholds. however, relying on thresholds will inevitably filter out a lot of useful image information. thus, in this work, we focus on another aspect: can we reduce the negative impact of noisy images without filtering the data? to answer the above question, we borrow the idea of curriculum learning (cl) @xcite , because cl can achieve denoising by adjusting the order of training data @xcite . in other words, cl no longer presents training data in a completely random order during training, but encourages training more time on clean data and less time on noisy data, which reveals the denoising efficacy of cl on noisy data. the key to applying cl is how to define clean/noisy examples and how to determine a proper denoising scheme. by analyzing the characteristics of the task, we tailor-design a multi-grained multicurriculum denoising framework (m2df). m2df first evaluates the degree of noisy images contained in each training instance from multiple granularities, and then the denoising curriculum gradually exposes the data containing noisy images to the model starting from clean data during training. in this way, m2df theoretically assigns a larger learning weight to clean data @xcite , thereby achieving the effect of noise reduction. our contributions are summarized as follows: ## related work in this section, we review the existing studies on multimodal aspect-based sentiment analysis (mabsa) and curriculum learning (cl). ## conclusion in this paper, we propose a novel multi-grained multi-curriculum denoising framework (m2df) for the mabsa task. specifically, we first define a coarse-grained noise metric and a fine-grained noise metric to measure the degree of noisy images contained in each training instance. then, in order to reduce the negative impact of noisy images, we design a single denoising curriculum and a multi-ple denoising curriculum. results from numerous experiments indicate that our denoising framework achieves better performance than other state-of-theart methods. further analysis also validates the superiority of our denoising framework.