Building Location Embeddings from Physical Trajectories and Textual Representations (2020)

Abstract: Word embedding methods have become the de-facto way to represent words, having been successfully applied to a wide array of natural language processing tasks. In this paper, we explore the hypothesis that embedding methods can also be effectively used to represent spatial locations. Using a new dataset consisting of the location trajectories of 729 students over a seven month period and text data related to those locations, we implement several strategies to create location embeddings, which we then use to create embeddings of the sequences of locations a student has visited. To identify the surface level properties captured in the representations, we propose a number of probing tasks such as the presence of a specific location in a sequence or the type of activities that take place at a location. We then leverage the representations we generated and employ them in more complex downstream tasks ranging from predicting a student’s area of study to a student’s depression level, showing the effectiveness of these location embeddings.

## introduction due to the rising adoption of smartphones over the past decade, the number of services with full or partial information about people's spatial mobility has skyrocketed. inspired by the natural language processing (nlp) literature, we investigate various properties of location embeddings. we explore whether valuable information is encoded in individual location embeddings, as well as embeddings that encompass a sequence of locations. we begin by exploring whether they are able to represent aspects such as location presence or location functionality. ultimately, we test the hypothesis that if enough underlying information is encoded, embedding models should aid in predicting user-centered descriptors, such as area of study, academic status, or mental health. location data can be used by university administrators for applications that improve student life. from the frequency and the type of locations accessed in one's daily routine, we may be able to identify someone who is depressed or someone who is overworked. importantly, opt-in frameworks can be established to supplement existing counseling and advising offices, allowing for early intervention in the case of mental health and academic concerns. with proper privacy safeguards in place, such models could readily be applied on most university campuses, as wifi connection data (from which we infer location) is likely already available. in addition, universities could use this data in an aggregate form to better understand student life and well-being, and find ways to promote healthy and engaging behaviors on campus. such aggregate location information can also be used by architectural firms or municipalities to help with the selection of buildings' locations, architecture, and design; with road and pedestrian traffic optimization; or for emergency response. we also know that such data is already available to large technology companies that track their users, and it is important to spread awareness about the personal information that can be gleaned. research like ours helps inform users about privacy concerns, and may open up a path to stricter legislation regarding the use of such data in the future. while we envision numerous positive applications of these methods, there are clear privacy drawbacks that the public should be aware of in the current technological environment. our work focuses on building an understanding of what information is encoded in location embeddings. in addition to creating embeddings using location trajectories, we propose an alternative method that synthesizes text from online sources to build representations that we hypothesize will better encode certain properties of locations. we show that using dense location embeddings that incorporate both movement patterns and text data improves our ability to model downstream tasks. we see that although we are not able to recover as much surface level information from embeddings of location sequences as we are from a simpler representation, the additional semantic information that is encoded allows us to better predict some user attributes. in our work, we find inspiration in the research by @xcite , who propose a formalized evaluation technique for sentence embeddings using a suite of ten classification tasks focusing on: (1) surface information (e.g., length, word content), (2) syntactic information (e.g., bigram shift, tree depth), and (3) semantic information (e.g., tense). the deep learning methods gave the best results overall, but the bag-of-vectors approach was a solid baseline for the word content task, where it outper-formed the deep learning models. applications of embeddings for location data. @xcite were among the first to use the skip-gram model on location data. they use locations visited before and after a target location as context to create location embeddings. these are then used in a personalized location recommendation system. @xcite similarly create embeddings of check-in data, but use the cbow model. their application task is reversed, predicting future visitors for a location instead of predicting locations that a user will visit. chang et al. (2018) also predict next check-ins for users using a model based on skip-gram. their work is uniquely related to ours in that they also build prediction of the text content of check-ins into the objective function. @xcite trained a skip-gram model to build location embeddings, and use them to understand the flow between urban locations. @xcite explore location embeddings from the perspective of geoinformatics, paving the way for our probing tasks. the work of @xcite is most similar to our own. they use gps data from cell phones as input to create embeddings and use data from a university setting. our work differs in that we use the skip-gram model and incorporate text-based embeddings. we also propose probing tasks to better understand the embeddings that we create, and predict additional user attributes from our new dataset that go beyond demographic information. ## representing locations we use location trajectories and text data to create vector representations of locations and, subsequently, embeddings of sequences of locations that are visited by a single person. after pre-processing using the method described in section 3.1, the location input data consists of a series of sorted, nonoverlapping locations for a number of users with start and end times. we discuss multiple methods to create vector representations based on this data. ## probing location representations while some of the methods we use (i.e., skip-gram) have been used in the past to represent locations for certain tasks, there has been less work studying them intrinsically. we propose surface level tasks to probe the properties encoded in location embeddings, which are important to gain a deeper understanding of the type of information they capture. we split surface level tasks into two categories: those that focus on individual locations and those that focus on location sequences. in addition to these surface level tasks, we propose a set of downstream prediction tasks to validate the utility of such embeddings. ## experimental setup we perform 10-fold cross validation on 729 instances, where each instance represents a student. preliminary classification experiments were conducted on a small subset using svm with linear and rbf kernels, random forests, decision trees, and naïve bayes, yet linear svm had the most robust performance. accordingly, our experiments consist of classification tasks using linear svm. as many of the classes are unbalanced, we more heavily weight updates for the minority class(es) by modifying the loss function to use a weight that is inversely proportional to the class's prevalence. to predict a student attribute, we create one vector for each month of data collection pertaining to each student, using the process described in section 4.4. our training framework is illustrated in figure 4 . we start by feeding the sequence vectors through a svm classifier, which predicts monthlevel labels. these are then concatenated to form a student instance and are passed to a meta-classifier that decides the final class label for that student. we use the meta-classification approach to allow the first classifier more data to learn from; without this approach, the number of input samples is relatively small (729). the process for surface level sequence tasks is similar, but no meta-classifier is used, as the gold standard labels have a month-level granularity. ## results and discussion figure 5 and tables 3 and 5 show the results obtained for the probing tasks. in addition to the loc2vec trajectory and text-based models, we run our experiments with two combination models, using the methods discussed in section 4.3. we employ the reddit variation for these combination models due to its strong performance on downstream tasks; we incorporate one model using concatenation and a model using retrofitting. we refer to these models as "loc2v-reddit," and "reddit-retrofit," respectively. we compare our classification performance against a random baseline. in order to introduce a stronger supervised baseline for our methods, we employ simpler location representations, in the form of one-hot vectors, which are passed as input in our supervised evaluation framework (figure 4 ). we take the mean of those one-hot vectors to create month sequence vectors as we do for the embeddings. ## conclusions in this paper, we addressed the task of building and probing location embeddings. we investigated several strategies to construct them, as well as a suite of probing tasks to understand the type of information encoded within. first, we showed that while all embedding methods encode both physical distance and functionality, methods using trajectories yield better spatial representations and methods using text data better encode location functionality. we showed that, like in the case of sentence embeddings from natural language, sequence embeddings of location data are able to encode surface level information (location-presence, and functionalitypresence), as well as information that can be effectively used in downstream tasks. overall, we found that an embedding model that accounts for both location trajectories and text related to locations (loc2v-reddit) gives the best performance over a diverse range of downstream tasks, from prediction of depression or sleep to prediction of academic area of study. importantly, we also found that embeddings of locations tend to underperform more traditional one-hot encodings on surface-level tasks, yet they generally outperform these representations on downstream tasks. this suggests that while such embeddings do not explicitly record distinct locations that people visit (thus being more privacy preserving and counteracting negative actions like stalking), they may be more effective for downstream applications that can yield positive outcomes, such as population-level mental health tracking or opt-in tracking for individuals who are in therapy. our code is publicly available at http://lit. eecs.umich.edu/downloads.html .