Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings (2020)

Abstract: Social media produces large amounts of contents every day. To help users quickly capture what they need, keyphrase prediction is receiving a growing attention. Nevertheless, most prior efforts focus on text modeling, largely ignoring the rich features embedded in the matching images. In this work, we explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. To better align social media style texts and images, we propose: (1) a novel Multi-Modality MultiHead Attention (M3H-Att) to capture the intricate cross-media interactions; (2) image wordings, in forms of optical characters and image attributes, to bridge the two modalities. Moreover, we design a unified framework to leverage the outputs of keyphrase classification and generation and couple their advantages. Extensive experiments on a large-scale dataset newly collected from Twitter show that our model significantly outperforms the previous state of the art based on traditional attention mechanisms. Further analyses show that our multi-head attention is able to attend information from various aspects and boost classification or generation in diverse scenarios.

## introduction the prominent use of social media platforms (such as twitter) exposes individuals with an abundance of fresh information in a wide variety of forms such as texts, images, videos, etc. meanwhile, the explosive growth of multimedia data has far outpaced individuals' capability to understand them, presenting a concrete challenge to digest the massive amount of data, distill the salient contents therein, and provide users with a quick access to the information they need when navigating noisy online data. to that end, extensive efforts have been made to social media keyphrase prediction foot_1 -aiming to produce a sequence of words that reflect a post's key concern. nevertheless, previous work mostly focuses on the use of textual signals @xcite , which sometimes provide limited features as social media language is essentially informal and fragmented. to enrich the contexts, here we resort to exploiting the matching images, which are widely used in social media posts to deliver auxiliary information from authors (e.g., opinions, feelings, topics, etc.), primarily due to the flourish of mobile internet. to illustrate our motivation, figure 1 shows the texts and images of two twitter posts (tweets). the left is tagged with a keyphrase "cat", which can be clearly signaled with its image while the paired text is an anthropomorphic description and hardly unveils its real semantics. for the right, the image depicts a basketball game scene with optical characters "2019 nba finals", directly indicating its keyphrase, which is difficult to identify from the texts. in both examples, images play a more vital role in reflecting the key information. these points motivate our cross-media keyphrase prediction study that examines how the salient contents can be indicated by the coupled effects of post texts and their matching images. previous work @xcite employs co-attention networks @xcite to encode multimedia posts, where a single attention function is concurrently performed to infer either visual or textual distributions. we argue that they might be suboptimal to model intricate text-image associations, as a recent finding @xcite points out there can be four diverse semantic relations held by images and texts on twitter. to allow for better modeling, in this work, we take advantage of the recent advance of multi-head attention @xcite capable of learning from different representation subspaces and extend it to capture diverse cross-media interactions, named as multi-modality multi-head attention (m 3 h-att). moreover, to well align the images' semantics to texts', we adopt image wordings and define two forms for that -explicit optical characters (such as "nba finals" in post (b)) detected from the optical character reader (ocr) and implicit image attributes @xcite , high-level text labels predicted to summarize the image's semantic concepts (such as a "cat" label for post (a)). furthermore, unlike prior work employing either classification @xcite or generation models @xcite , we propose a unified framework to couple the advantages of keyphrase classification and generation. specifically, in addition to the joint training of both modules, we further extend the copy mechanism @xcite to explicitly aggregate classification outputs together with tokens from the source input. empirical results show that integrating classification outputs not only keeps classification's superiority to predict common keyphrases (figure 5(c )) while enables keyphrase creation beyond a predefined candidate list, but also largely benefits the keyphrase generation with better absent keyphrase prediction (figure 5(b) ). for experiments, we collect a large-scale tweet dataset with texts and images, which is presented as part of our work. the empirical results show that our model significantly outperforms the state-ofthe-art (sota) methods using traditional attention. for example, we obtain 47.06% f1@1 compared with 43.17% by @xcite (keyphrase generation from texts only) and 42.12% by @xcite (multi-modal keyphrase classification). we then examine how we perform to handle absent and present keyphrases, and varying keyphrase frequency and post length. the results indicate the consistent performance boost brought by our m 3 h-att design and unified framework in diverse scenarios ( §5.3). we further quantify the effects of different settings of multi-head attention and image wordings to see when and how they work the best ( §5.4). lastly, we provide qualitative analysis to interpret why our model results in superior multimedia understanding ( §5.5). ## related work social media keyphrase prediction. traditional keyphrase prediction studies focus on using two-step pipeline methods: candidates are first extracted with handcrafted features (e.g. part-ofspeech tags @xcite ) and then ranked by unsupervised @xcite or supervised algorithms @xcite . these methods undergo labor-intensive feature engineering and hence lead to the growing popularity of adopting data-driven neural networks. specifically for social media keyphrase prediction, most efforts are based on sequence tagging style extraction @xcite or classification from a predefined candidate list @xcite , which are however unable to produce keyphrases absent in the post or the fixed list. inspired by the recent success of keyphrase generation for scientific articles @xcite , @xcite employ sequence-to-sequence (seq2seq) models to allow unseen keyphrases to be flexibly created for social media posts. unlike them, we propose a novel unified framework to combine the benefits of keyphrase classification and generation. similar to this, @xcite also exploits the power of classification for keyphrase generation but in a separated retrieval manner, where we elegantly integrate them with a tailored copy mechanism and allow for the end-to-end joint training. while most of prior work focuses on the modeling of texts, we additionally exploit their matching images and study the coupled effects for indicating keyphrases. cross-media research. we are also related to cross-media research, where texts and images are jointly exploited for a variety of applications, such as personalized image captioning @xcite , event extraction @xcite , sarcasm detection @xcite , and text-image relation classification @xcite . some of them have pointed out the usefulness of ocr texts @xcite and image attributes @xcite to endow images with higher-level semantics beyond visual features, where we are the first to study how ocr texts and image attributes work together to indicate keyphrases. closest to our work, @xcite study multimedia hashtag classification and employ co-attention @xcite to model the text-image associations, while we extend the multi-head attention @xcite to better capture diverse styles of cross-modal interactions in social media. while multi-head attention has been widely exploited in many vision-language (vl) tasks, such as image captioning @xcite , visual question answering @xcite , and visual dialog @xcite , its potential benefit to model flexible cross-media posts has been previously ignored. due to the informal style in social media, cross-media keyphrase prediction brings unique difficulties mainly in two aspects: first, its textimage relationship is rather complicated @xcite while in conventional vl tasks the two modalities have most semantics shared; second, social media images usually exhibit a more diverse distribution and a much higher probability of containing ocr tokens ( §4), thereby posing a hurdle for effectively processing. ## our unified cross-media keyphrase prediction framework given a collection c with |c| text-image post pairs {(x n , i n )} |c| @xmath0, we aim to predict a keyphrase set @xmath1. following @xcite , we copy the source input pair multiple times to allow each paired to have one keyphrase. we represent each input as a triplet (x, i, y), where x and y are formulated as word sequences @xmath2, ..., x lx and @xmath3, ..., y ly (l x and l y denote the number of words). we show the overview of our proposed crossmedia keyphrase prediction model in figure 2 . we first encode a text-image tweet into three modalities: text, attribute, and vision ( §3.1), and propose a multi-modality multi-head attention (m 3 h-att) to capture their intricate interactions ( §3.2). then, we feed the learned multi-modality representations for either keyphrase classification or generation, followed with a tailored aggregator to combine their outputs ( §3.3). lastly, the entire framework can be jointly trained via multi-task learning ( §3.4). ## conclusion this paper studies cross-media keyphrase prediction on social media and presents a unified framework to couple the advantages of generation and classification models for this task. moreover, we propose a novel multi-modality multi-head attention to capture the dense interactions between texts and images, where image wordings explicit in optical characters and implicit in image attributes are further exploited to bridge their semantic gap. experimental results on a large-scale newly-collected twitter corpus show that our model significantly outperforms sota either generation or classification models with traditional attention. post (a): i thought older hanzo died after d'vorah killed him? @nether-realm #mortalkombat11 post (b): congrats producer of the year, non-classical winner -williams #grammys post (c): last year's highest rated animated movie spider man into the spider-verse is now streaming on netflix! #spiderman post (d): we need to make sure the ratings are high #saveshadowhunters (mortal kombat story all full movie) (williams at grammy awards) (spider man into the spider-verse) (will someone save shadow hunters) gen-copy: quote cls-co-att: destiny 2 our: mortal kombat 11 gen-copy: live under par cls-co-att: a star is born our: grammys gen-copy: spider verse cls-co-att: marvel our: spider man gen-copy: teacher goals cls-co-att: brexit our: save shadowhunters post (a): good night, everyone. i hope that you have had a delightful day and a restful weekend. #hoorayfordogs post (b): head up, chest out! a handsome purple finch poses for a shot. #birds #wildlife #photography post (c): i was watching all the bees honeybee collecting pollen on the flowers bouquet #savethe-bees #catsoftwitter post (d): for 1970, plymouth intended to make its gtx model a street powerhouse. #musclecar #classiccar (dog white yellow brown plate) (branch bird red top small) (cat white pink grey flowers) (car roof park old meter) gen-copy: friday feeling cls-co-att: hooray for dogs our: hooray for dogs gen-copy: gap ol cls-co-att: birding our: birds; wildlife gen-copy: photography cls-co-att: springwatch our: cats of twitter gen-copy: plymouth cls-co-att: mopar our: classic car figure 11: more qualitative examples showing the effectiveness of encoding image attributes. our model that considers image attributes correctly predicts the keyphrases for all these cases (in bold). blue tokens are the top five predicted attributes, which reveal the main image contents and thus help to indicate keyphrases.