Understanding Social Media Cross-Modality Discourse in Linguistic Space (2022)

Abstract: The multimedia communications with texts and images are popular on social media. However, limited studies concern how images are structured with texts to form coherent meanings in human cognition. To fill in the gap, we present a novel concept of cross-modality discourse, reflecting how human readers couple image and text understandings. Text descriptions are first derived from images (named as subtitles) in the multimedia contexts. Five labels – entity-level insertion, projection and concretization and scene-level restatement and extension — are further employed to shape the structure of subtitles and texts and present their joint meanings. As a pilot study, we also build the very first dataset containing over 16K multimedia tweets with manually annotated discourse labels. The experimental results show that trendy multimedia encoders based on multi-head attention (with captions) are unable to well understand cross-modality discourse and additionally modeling texts at the output layer helps yield the-state-of-the-art results.

## introduction the growing popularity of multimedia is revolutionizing the communications on social media. the conventional text-only form has been expanded to cross modalities involving texts and images in information exchange. for multimedia messages, the language understanding acquires more than making sense of both visual and textual semantics; it also matters to figure out what glues them together to exhibit the coherent senses in human's mind. nevertheless, most progress made in social media language understanding relies on texts to learn the message-level semantics @xcite , largely ignoring the rich meanings conveyed in images @xcite . other recent multimodal studies focus on model designs to combine visual and textual signals (park et al., 2019; li et al., 2020; * corresponding author @xcite , ignoring the insights from how humans understand the implicit structure underlying a multimedia post. in light of these concerns, we consider images as an integral part of social media language and propose a novel concept of cross-modality discourse, which defines how human readers structure the coherent meanings from image and text modalities. our work is inspired by @xcite examining the information overlap between images and texts, whereas we take a step further to characterize how multimedia messages make sense to humans, which is beyond a simple yes-or-no prediction to whether new thing is observed. to the best of our knowledge, we are the first to extend discourse -a pure linguistic concept -to define the linguistic roles played by images and their pragmatic relations with texts to shape the coherent meanings. in general, cross-modality discourse is defined by the operations adopted in human perception to couple image and text semantics. readers may first extract the information from the images acquired to complete the cross-modality understanding, either in form of the local objects (entities) or global scenes @xcite . then, the extracted entities or scenes are represented in texts, named as the images' subtitles, which can further contribute to structure the entity-level or scene-level discourse with the matching texts in the multimedia contexts. concretely, for entity-level discourse, it is detailed into insertion, projection, and concretization, according to whether the entity is omitted, described, or mapped; similarly, scene-level restatement and extension are employed to reflect whether the story in one modality recurs or continues in the other. to illustrate the definitions above, figure 1 shows five multimedia twitter posts. as can be seen from (a), readers may concentrate on the object "strawberry" and insert its name into the texts omitting the entity. as for (b), the "coffee" object tweet text freshly picked off my allotment (strawberry) today, well chuffed one step closer to summer (track towards beach) happy sunday! my best friend and i have coffee in the sunshine this dog has to hold hands on the car cartel leader whose arrest sparked killings is sentenced to prison in dallas court discourse labels entity-level: insertion (strawberry) scene-level: extension (track towards beach) entity-level: concretization (coffee) scene-level: restatement (dog hold hands in car) entity-level: projection (court) (a) (b) (c) (d) (e) tweet image vempalaet al. (2019) labels image adds meaning text not represented image not adds meaning text represented image not adds meaning text not represented image not adds meaning text represented image adds meaning text not represented should be extracted from the image to concretize the word "coffee" in the text. in (c), the word "court" in text is linked with the "gavel" object. the image in (d) helps restate the texts scene (a dog holds hands in the car). in (e), the global scene works as an extension to texts and completes the story: "we are one step closer to summer following the track towards beach.". on the contrary, the image-text relations in vempala and preotiuc-pietro (2019) are limited to whether images add new meanings to texts, which is nonetheless insufficient to reflect how language is understood in multimedia contexts. as a pilot study of cross-modality discourse, we also present the very first dataset to explore the task. it is collected from twitter and contains 16k highquality multimedia posts with manual annotations on their discourse labels. 1 we believe our task and the associated dataset, being the first of its kind, will be potentially beneficial to help machines gain the ability to understand social media language with multimodal elements. to that end, we present a framework to learn the discourse structure across texts and images. inspired by the recent advances in multimodal learning @xcite , we employ the multi-head attention mechanism @xcite to explore the visual-textual representations reflecting cross-modality interactions. besides, to characterize subtitles for discourse learning, image captions generated from model trained on coco captioning dataset @xcite are leveraged as additional features. for empirical studies on cross-modality discourse, we conduct comprehensive experiments on our dataset. the comparison results on classification show the challenges for machines to infer discourse structure and it is beyond the capability of advanced multimodal encoders to well handle our task. nevertheless, exploring correlations of texts, captions, and visual-textual interactions helps exhibit the state-of-the-art performance in both the intra-class and overall evaluation. we further examine the effects of varying modalities and text length and find that text signals are crucial for discourse inference while joint effects of texts, images, and captions present the best results. at last, the qualitative analysis demonstrates how the multi-head attention in our model interprets discourse structure. ## related work our paper crosses the lines of multimedia learning and discourse analysis in natural language processing. here comes more details. multimedia learning. our paper is in the line with cross-media research that attempts to fuse textual and visual features. there are various deep learning methods proposed to leverage crossmodal features, either based on advanced neural architectures like co-attentions @xcite and multi-head attentions @xcite , or pre-trained visual-lingual representations @xcite @xcite . their effectiveness are demonstrated in both conventional visionlanguage tasks, such as image captioning @xcite @xcite and visual question answering (vqa) @xcite @xcite , and social media applications, such as sarcasm detection @xcite , event tracking @xcite , keyphrase prediction @xcite . it is seen that most progress to date made in this line focus on advancing methodology designs for general purposes @xcite or specific applications @xcite to better capture the matched semantics across varying modalities. however, their effectiveness over social media data would be inevitably compromised resulted from the intricate image-text interactions @xcite . we thus borrow the insights from human perception to interpret image-text relations from the linguistic viewpoints and propose the task to learn discourse structure in multimedia contexts. it is a fundamental research exhibiting the potential to help the models gather cross-modality understanding capability and might benefit various downstream applications. we are also related with previous categorization tasks on social media to understand image-text relations, such as information overlap @xcite , point-of-interest types (villegas and aletras, 2021), author purposes @xcite , object possessions @xcite , and so forth. besides, interestingly, the "discourse" concept is also employed to examine the image-text relations in cooking recipes @xcite . compared with these studies concatenating visual and textual embeddings in a "common" space, we craft text-formed subtitles to convey visual stories and explore how they shape the coherent meanings with the post texts in linguistic space. it will consequently allow deep semantic learning to capture the implicit structure holding image and text modalities, while the existing models might be incapable to gather senses of language understanding via simple feature concatenation. discourse analysis. this work is related to prior studies on text-level discourse structures. the popular tasks in the styles of either rst (rhetorical structure theory) @xcite or pdtb (penn discourse tree bank) @xcite explore the rhetorical relations of discourse units (e.g., phrases or sentences) that cohesively connect them form a sense of coherence. these studies have demonstrated their helpfulness in diverse stream of nlp applications @xcite , such as sentiment analysis @xcite ), text categorization @xcite , and microblog sum-marization @xcite . nevertheless, limited work examines a social media image as a discourse unit of the pragmatic structure in multimedia contexts, which is a gap to be filled in this work. ## study design in this section, we first define the task to predict cross-modality discourse in §3.1. then, we introduce how we construct the dataset in §3.2, followed by the data analysis in §3.3 and the potential applications in §3.4. ## the discourse learning framework in this section, we describe our framework that couples the signals from images and texts to predict their discourse labels. as shown in fig. 4 , the model architecture leverages representations learned from texts, images, and image captions (to reflect subtitles), which will be introduced in §4.1. then, we will discuss how we combine multi-modality representations §4.2. at last, §4.3 presents how we predict the discourse labels and design the training processes. ## experimental setup model settings. the length of tweet texts (l) and captions (n ) are both capped at 20 by truncation. the batch size is set to 100, the learning rate to 5 × 10 -5 . the head number of all multi-head attention layers are set to 6. for image encoding, image feature map size m is set to 14. for text and comment encoding, the representations are extracted from the bottom 6-layers of the bertweet model, which are further fine-tuned in training. in the setup, we randomly split 80%, 10% and 10% for training, validation, and test. for evaluation, we report f1 scores in the prediction of each label and the weighted f1 to measure the overall results. baselines and comparisons. we first consider two text-level discourse parsers proposed in @xcite and rutherford and xue (2016), where we extend their text encoders into multimodal encoders to fit the image-text pairs. then, we compare with a popular multimodal classifier @xcite that employs a dual attention network to fuse the visual and textual features. besides, we evaluate varying sets of feature combinations in our model test + image, text + caption, and text + image + caption (the full set). recall that our framework employs multihead attention to integrate features learned from different modalities. in experiments, we also test the performance of other modality fusion alternatives based on simple feature concatenation (concatfuse), the conventional attention mechanism (attention), the co-attention mechanism (co-attention). ## experimental discussions this section first presents the main comparison results ( §6.1). then, we discuss model sensitivity to varying modalities and text length in §6.2. finally, §6.3 presents a case study to provide more insights. ## conclusion we have presented a novel task to learn crossmodality discourse that advances models to gain social media language understanding capability in multimedia contexts. to handle the intricate imagetext interactions, the visual semantics are first converted into text-formed subtitles and then compared with post texts to explore deep syntactic relations in linguistic space. for empirical studies, we further contribute the first dataset presenting 16k humanannotated tweets with discourse labels for imagetext pairs. the main comparison results on our dataset have shown the effectiveness of multi-head attentions in exploring interactions among text, image, and caption modalities. further discussions demonstrate our potential to produce meaningful representations indicating implicit image-text structure. these discourse features, conveying essential linguistic clues consistent with human senses, may largely benefit the future advances of automatic cross-modality understanding on social media.