Are People Located in the Places They Mention in Their Tweets? A Multimodal Approach (2022)

Abstract: This paper introduces the problem of determining whether people are located in the places they mention in their tweets. In particular, we investigate the role of text and images to solve this challenging problem. We present a new corpus of tweets that contain both text and images. Our analyses show that this problem is multimodal at its core: human judgments depend on whether annotators have access to the text, the image, or both. Experimental results show that a neural architecture that combines both modalities yields better results. We also conduct an error analysis to provide insights into why and when each modality is beneficial.

## introduction twitter is a social network in which users post short messages known as tweets. while statistics vary depending on the source and publication time, official reports state that 187 million users logged in daily in the third quarter of 2020 @xcite , and 500 million tweets were published worldwide on a daily basis in 2014-the last year the number was made public @xcite . according to a recent report @xcite , 24% of all americans use twitter (45% between 18 and 24 years of age), and 46% of them use it at least once a day (26% more than once). tweets contain not only text (including hashtags, links, emojis, etc.), but also multimedia content such as images and videos. indeed, 42% of tweets have images @xcite , and marketing research reveals that having an image improves user engagement: 18% more click throughs, 89% more likes and 150% more retweets @xcite . when it comes to noisy user-generated content and spatial information, most previous work falls under two main topics: (a) named entity recognition @xcite and disambiguation @xcite , and (b) geolocation @xcite . the former identifies, among others, location named entities and links them to a knowledge base without specifying who is there. the latter determines one location per user-even if it is not explicitly mentioned. for example, place of residence can be inferred, at least to a certain degree, from the locations of other users and language usage patterns. in this paper, we tackle a complimentary problem: to determine whether people are located in the places they mention in their tweets. extracting this kind of spatial information is challenging. first, people often mention places in their tweets even though they are not located there. second, one must often rely on nuances in both the text and images to make a decision. consider the tweets in figure 1 . the author of the tweet on the left was in phoenix when the tweet was published. note that the text alone could arguably be enough to conclude so, but the image provides additional evidence: the background is compatible with the phoenix area (desert landscape, mountains, etc.), and the person in the picture is (most likely) enjoying the weather there during a short trip for memorial day. the author of the tweet on the right, on the other hand, was not in atlanta when the tweet was published. in this example, the image together with the text provides evidence that the author is working rather than enjoying memorial day in atlanta with coworkers. while the work presented here could be considered fundamental research, it opens the door to several applications. for example, emergency management systems could issue customized alerts to individuals who were, are, or are about to be located near a natural disaster. similarly, eyewitness verification could benefit as the locations of people and the events they claim to witness must be compatible (within some temporal bounds). the main contributions of this paper are: 1 (a) a corpus of 6,540 tweets with annotations indicating whether the author was in the places mentioned in the tweets; (b) analysis demonstrating that this is a multimodal problem: the ground truth changes depending on whether annotators have access to the text, the image, or both; (c) experimental results showing that taking into account both modalities is beneficial; and (d) qualitative analysis providing insights into (d.1) when are the text and image beneficial, and (d.2) the remaining sources of errors. ## a corpus of tweets and spatial information about the authors our main goal is to understand what kind of spatial information one can infer between authors of tweets and the places they mention in their tweets. to our knowledge, we are the first to tackle this problem, so we create a new corpus. this allows us to explore whether human judgments change depending on whether annotators have access to the text, image or both (section 4) as well as conduct experiments to automate the task (section 5). ## corpus analysis the 6,540 tweets in our corpus mention 96 unique cities. the most frequent cities are miami (17% of tweets) and chicago (6%); other cities account for at most 5% of tweets each. the tweets mention all the events we target (section 3). the most common event is spring break (37% of tweets) followed by memorial day (27%). other events account for between 5% and 10% of tweets except election day, which accounts for 3% of tweets. ## experiments and results armed with our corpus (section 3), we conduct experiments to automatically determine whether authors of tweets are located in the cities they mention in their tweets. we reduce the problem to a classification task. the input to the model is a (tweet, city) pair, and the output is a label indicating the text component is bert @xcite and the image component is vgg16 @xcite . we use the pretrained models released by huggingface @xcite and pytorch @xcite . we train the neural network for up to 100 epochs using the adam optimizer (kingma and ba, 2014), categorical cross entropy as the loss function, and batch size 8. we stop the training process before 100 epochs if there is no improvement in the validation set for 10 epochs. we implement the neural network with pytorch @xcite . ## qualitative analysis to better understand why and when the text and image are most beneficial, we perform a qualitative analysis of the errors made by each model. more specifically, we answer the following questions: when does the image complement the text? we start the qualitative analysis providing insights into when is the image beneficial to solve the task. the most frequent error that benefits from taking into account the image (38%) occurs when the image (apparently) does not have a connection with the location at hand. instead, it (visually) depicts some event that (a) occurred in the location at hand and (b) is mentioned in the text. consider the example on the left ( the second most common error fixed by the full network (31%) occurs when the tweet is an advertisement and the text component alone wrongly predicts yes (e.g., middle tweet in the third most common error that benefits from taking into account the image (14%) occurs when the image depicts a typical scene of the location at hand. for example, in the right tweet in ## conclusions we have introduced the task of determining whether people are located in the places mentioned in their tweets. going beyond named entity recognition and disambiguation, this problem is about figuring out whether the authors of tweets are located in the places mentioned their tweets. our new corpus (6,540 tweets) shows that people often mention cities in their tweets even though they are not located there (48.9% of city mentions)-or at least there is not enough evidence in the tweet for crowdworkers to conclude so. importantly, we have shown that human judgments change substantially depending on whether crowdworkers have access to the text, the image, or both. these changes in human judgments indicate that when it comes to understanding spatial information about the authors of tweets, the text and images complement each other. to our knowledge, our corpus (krippendorf's α = 0.61) is the first to tackle this challenging problem. experimental results show that the task can be automated although our neural network obtains modest results. in particular, coupling independent representations of the text and image (2 representations) with a joint representation of the text and image yields the best results. these empirical results mirror the observation that human judgments change depending on which modalities crowdworkers have access to. we have also presented a qualitative analysis providing insights into how the image and text complement each other. in summary, they are usually beneficial if they provide additional details about the location at hand or an event that occurred in the location at hand.