Dynamic Routing Transformer Network for Multimodal Sarcasm Detection (2023)

Abstract: Multimodal sarcasm detection is an important research topic in natural language processing and multimedia computing, and benefits a wide range of applications in multiple domains. Most existing studies regard the incongruity between image and text as the indicative clue in identifying multimodal sarcasm. To capture cross-modal incongruity, previous methods rely on fixed architectures in network design, which restricts the model from dynamically adjusting to diverse image-text pairs. Inspired by routing-based dynamic network, we model the dynamic mechanism in multimodal sarcasm detection and propose the Dynamic Routing Transformer Network (DynRT-Net). Our method utilizes dynamic paths to activate different routing transformer modules with hierarchical co-attention adapting to cross-modal incongruity. Experimental results on a public dataset demonstrate the effectiveness of our method compared to the state-of-the-art methods. Our codes are available at https://github.com/TIAN-viola/DynRT .

## introduction sarcasm is a widely used figurative language to give the ironic expression in our daily life, which typically means the opposite of what it really wants to express @xcite . as an important step to analyze people's opinions and sentiments in communication, sarcasm detection benefits a wide range of applications such as natural language dialogue @xcite , public opinion mining @xcite and social media analysis @xcite . with the rapid growth of multimodal user-generated content, multimodal sarcasm detection has gained increasing research attention in recent years @xcite @xcite @xcite @xcite *corresponding author 2022; @xcite , and has become an important research topic in natural language processing and multimedia computing. the sarcastic clues of multimodal contents are mainly relevant to the incongruity across image and text @xcite @xcite @xcite @xcite . existing studies model this characteristic of incongruity between image and text with various approaches, including decomposition and relation network @xcite , attention mechanisms @xcite , graph-based methods @xcite , and optimal transport method @xcite . in addition, external knowledge is also introduced to boost the performance of multimodal sarcasm detection @xcite . as it is shown in multimodal samples in figure 1 , there are diverse kinds of sarcastic image-text pairs. in some cases, the image and text express the incongruous meaning with local segments, where visual regions or objects are contrastive to the meaning of words or phrases in the text, as those in figure 1 (a) and (b). in other cases, the feelings implied in the image and text respectively are totally opposite, as those in figure 1 (c ) and (d). to detect these sarcastic image-text pairs, current approaches mainly focus on modeling the cross-modal incongruity. however, these methods rely on static networks to capture the characteristic of incongruity, which use fixed architectures on different kinds of inputs, thus lacking the flexibility to adapt to diverse image-text pairs. to tackle this problem, the dynamic aspect of incongruity between image and text should be considered. one possible solution is to model dynamic mechanism with a routing-based dynamic network, where a series of modules can capture the incongruity between image and text dynamically via selecting one or more most suitable modules according to different image-text pairs. existing routing-based method in multimodal dynamic networks @xcite performs routing only on single-modality data, which is insufficient to model the dynamic image-text incongruity in cross-modal sarcasm detection. therefore, we extend the existing routing scheme to multimodal setting with dynamic network design, aiming to better model the dynamic mechanism for multimodal sarcasm detection. in this paper, we propose a novel dynamic routing transformer network, namely dynrt-net, whose router helps model route on dynamic routing transformer modules with hierarchical co-attention adapting to cross-modal incongruity prevalent in diverse image-text pairs. the main contributions of our work are as follows: ## method figure 2 shows the overall architecture of our proposed dynamic routing transformer network dynrt-net, which is composed of three components: encoding, dynamic routing transformer, and classification. we first encode the text and a paired image into multimodal features respectively via two pre-trained models. then, we feed them into the dynamic routing transformer to route on hierarchical co-attention dynamically and learn crossmodal incongruity, resulting in the routed features with cross-modal information. finally, we feed the routed features and image features into the classifier for multimodal sarcasm classification. ## conclusion to model the cross-modal incongruity that is adjustable to diverse image-text pairs, we propose the dynamic routing transformer network dynrt-net to activate different modules with hierarchical coattention for multimodal sarcasm detection. this dynamic mechanism in network design can help capture the sarcastic clues in accordance with different image-text inputs. experimental results on a public dataset demonstrate the effectiveness of our proposed method. our future work shall explore diverse types of co-attention between image and text to further improve the adaptability of our method.