Point-of-Interest Type Prediction using Text and Images (2021)

Abstract: Point-of-interest (POI) type prediction is the task of inferring the type of a place from where a social media post was shared. Inferring a POI’s type is useful for studies in computational social science including sociolinguistics, geosemiotics, and cultural geography, and has applications in geosocial networking technologies such as recommendation and visualization systems. Prior efforts in POI type prediction focus solely on text, without taking visual information into account. However in reality, the variety of modalities, as well as their semiotic relationships with one another, shape communication and interactions in social media. This paper presents a study on POI type prediction using multimodal information from text and images available at posting time. For that purpose, we enrich a currently available data set for POI type prediction with the images that accompany the text messages. Our proposed method extracts relevant information from each modality to effectively capture interactions between text and image achieving a macro F1 of 47.21 across 8 categories significantly outperforming the state-of-the-art method for POI type prediction based on text-only methods. Finally, we provide a detailed analysis to shed light on cross-modal interactions and the limitations of our best performing model.

## introduction a place is typically described as a physical space infused with human meaning and experiences that facilitate communication @xcite . the multimodal content of social media posts (e.g. text, images, emojis) generated by users from specific places such as restaurants, shops, and parks, contribute to shaping a place's identity, by offering information about feelings elicited by participating 1 code and data are available here: https://github .com/danaesavi/poi-type-prediction imagine all the people sharing all the world ∼ next stop: nyc in an activity or living an experience in that place @xcite . fig. 1 shows examples of twitter posts consisting of image-text pairs, shared from two different places or point-of-interests (pois). users share content that is relevant to their experience in the location. for example, the text imagine all the people sharing all the world which is accompanied by a photograph of the imagine mosaic in central park; and the text next stop: nyc along with a picture of descriptive items that people carry at an airport such as luggage, a camera and a takeaway coffee cup. developing computational methods to infer the type of a poi from social media posts @xcite is useful for complementing studies in computational social science including sociolinguistics, geosemiotics, and cultural geography @xcite @xcite , and has applications in geosocial networking technologies such as recommendation and visualization systems @xcite @xcite . previous work in natural language processing (nlp) has investigated the language that people use in social media from different locations, by inferring the type of a poi of a given social media post using only text and posting time, ignoring the visual context @xcite . however, communication and interactions in social media are naturally shaped by the variety of available modalities and their semiotic relationships (i.e. how meaning is created and communicated) with one another @xcite @xcite . in this paper, we propose poi type prediction using multimodal content available at posting time by taking into account textual and visual information. our contributions are as follows: ## results the results of poi type prediction are presented in models that simply concatenate text and image vectors have close performance to bert (44.0 for concat-bert+xception) or lower (41.56 for concat-bert+efficientnet). this suggests that assigning equal importance to text and image information can deteriorate performance. it also shows that modeling cross-modal interactions is necessary to boost performance of poi type classification models. surprisingly, we observe that the pre-trained multimodal lxmert fails to improve over bert (40.17 f1) while its performance is lower than simpler concatenative fusion models. we speculate that this is because lxmert is pretrained on data where both, text and image modalities share common semantic relationships which is the case in standard vision-language tasks including image captioning and visual question answering @xcite . on the other hand, text-image relationships in social media data for inferring the type of location from which a message was sent are more diverse, highlighting the particular challenges for modeling text and images together @xcite . our proposed mm-gated-xatt model achieves 47.21 f1 which significantly (t-test, @xmath0.05) improves over bert, the best performing model in sánchez @xcite and consistently outperforms all other image-only and multimodal approaches. this confirms our main hypothesis that modeling text with image jointly to learn the interactions between modalities benefit performance in poi type prediction. we also observe that using only the gating mechanism (mm-gate) outperforms (44.64 f1) all other models except for mm-gated-xatt. this highlights the importance of controlling the information flow for the two modalities. using cross-attention on its own (mm-xatt), on the other hand, fails to improve over other multimodal approaches, implying that learning crossmodal interactions is not sufficient on its own. this supports our hypothesis that language and visual information in posts sent from specific locations may be or may not be related, and that managing the flow of information from each modality improves the classifier's performance. finally, we investigate using less noisy textimage pairs in alignment with related computational social science studies involving text and images @xcite @xcite . we train and test lxmert, mm-gate, mm-xatt, and mm-gated-xatt on tweets that are originally accompanied by an image (see section 3), excluding all text-only tweets. the results are shown in ## analysis 7.1 modality contribution to determine the influence of each modality in mm-gated-xatt when assigning a particular label to a tweet, we compute the average percentage of activations for the textual and visual modalities for each poi category on the test set. the outcome of this analysis is depicted in fig. 3 . as anticipated, the textual modality has a greater influence on the model prediction, which is consistent with our findings in section 6. the category where the visual modality has greater impact on the predicted label is professional & other places (43.20%) followed by shop & service (43.11%). to examine how the visual information impacts the poi type prediction task, fig. 4 shows examples of posts where the contribution of the image is large while the text-only model (bert) misclassified the poi category. we observe that the text content of post (a) misled bert towards food, probably due to the term 'powder'. on the other hand, mm-gated-xatt can filter irrelevant information from the text, and prioritize relevant content from the image in order to assign the correct poi category for post (a) (great outdoors). likewise, post (b) was correctly classified by mm-gated-xatt as shop & service and misclassified by bert as arts & entertainment. for this post 40% of the contribution corresponds to the image and 60% to text. this shows how image information can help to address the ambiguity in short texts @xcite , improving poi type prediction. ## conclusion and future work this paper presents the first study on multimodal poi type classification using text and images from social media posts motivated by studies in geosemiotics, visual semiotics and cultural geography. we enrich a publicly available data set with images and we propose a multimodal model that uses: (1) a gate mechanism to control the information flow from each modality; (2) a cross-attention mechanism to align and capture the interactions between modalities. our model achieves state-of-the-art performance for poi type prediction significantly outperforming the previous text-only model and competitive pretrained multimodal models. in future work, we plan to perform more granular prediction of poi types and user information to provide additional context to the models. our models could also be used for modeling other tasks where text and images naturally occur in social media such as analyzing political ads (sánchez villegas et al., 2021), parody @xcite and complaints @xcite @xcite .