Building a Bridge: A Method for Image-Text Sarcasm Detection Without Pretraining on Image-Text Data (2020)

Abstract: Sarcasm detection in social media with text and image is becoming more challenging. Previous works of image-text sarcasm detection were mainly to fuse the summaries of text and image: different sub-models read the text and image respectively to get the summaries, and fuses the summaries. Recently, some multi-modal models based on the architecture of BERT are proposed such as ViLBERT. However, they can only be pretrained on the image-text data. In this paper, we propose an image-text model for sarcasm detection using the pretrained BERT and ResNet without any further pretraining. BERT and ResNet have been pretrained on much larger text or image data than image-text data. We connect the vector spaces of BERT and ResNet to utilize more data. We use the pretrained Multi-Head Attention of BERT to model the text and image. Besides, we propose a 2D-Intra-Attention to extract the relationships between words and images. In experiments, our model outperforms the state-of-the-art model.

## introduction it is becoming popular today for people using text with images to express their emotions and feelings in social media. this makes sarcasm detection more challenging. sometimes, only when the text and image are read together can one know whether it is sarcasm. for example in figure 1 , which are from the multimodal twitter dataset @xcite , the images contain the necessary information to determine whether it is a sarcasm. the previous works about the image-text sarcasm detection @xcite and also the imagetext sentiment analysis @xcite @xcite have about two steps: (1) summarizing the image and text; (2) fusing the summaries of the image and text. although some works try to explore the early fusion, it is still limited. some details of text and image would be dropped when summarizing. recently, some multi-modal models based on the architecture of bert are proposed such as vil-bert @xcite , lxmert @xcite , @xcite , and b2t2 @xcite . however, these models are pretrained only on image-text data. in contrast, bert can be pretrained on much lager text data than image-text data. resnet can also make use of more image data. in linear algebra, matrix multiplication can be understood as a kind of vector space transformation. in this paper, we provide a new perspective, the vector space transformation perspective, on this task. we propose a model to connect the text and image, and design a bridge layer to build the connection. the low-level and high-level image features are passed into bert @xcite as the embedding of bert. we use the pretrained bert and pretrained resnet directly without any further pretraining for this task. any bert-like models that are based on transformer @xcite can still be used in our model in the future. any visual models can be used in our model in the future as well. besides, our model does not require huge computing resources and time for pretraining. based on the idea that sarcasm relies on the semantic relationships and contrasts between words, @xcite uses a softmax and a max function to extract the relationships and contrasts. however, the max function drops some information. in this paper, we propose a method called 2d-intra-attention with a 2d-softmax to handle the 2d relationships. assuming n is the number of inputs, with the 2d-softmax, n 2 relationships are considered every time. in contrast, with the max function @xcite , only n relationships are considered. besides, we also add the image features into the 2d-intra-attention, so the relationships between words and images are considered. in experiments, our model outperforms the stateof-the-art model. our main contributions are summarized as follows: ## approach figure 2 shows the architecture of our model. our model contains two parts: image-text fusion and 2d-intra-attention. figure 2 : architecture of our model, where "v" denotes the text and "g" denotes the image. ## conclusion in this paper, we propose an image-text model for image-text sarcasm detection. we propose a novel way to integrate image and text information. our model outperforms the state-of-the-art model. comparing with multi-modal models, our model utilizes more text and image data instead of only the image-text data. our model can adopt different pretrained language models and visual models directly without any further pretraining.