H e G e L : A Novel Dataset for Geo-Location from H ebrew Text (2023)

Abstract: The task of textual geolocation — retrieving the coordinates of a place based on a free-form language description — calls for not only grounding but also natural language understanding and geospatial reasoning. Even though there are quite a few datasets in English used for geolocation, they are currently based on open-source data (Wikipedia and Twitter), where the location of the described place is mostly implicit, such that the location retrieval resolution is limited. Furthermore, there are no datasets available for addressing the problem of textual geolocation in morphologically rich and resource-poor languages, such as Hebrew. In this paper, we present the Hebrew Geo-Location (HeGeL) corpus, designed to collect literal place descriptions and analyze lingual geospatial reasoning. We crowdsourced 5,649 literal Hebrew place descriptions of various place types in three cities in Israel. Qualitative and empirical analysis show that the data exhibits abundant use of geospatial reasoning and requires a novel environmental representation.

## introduction and background textual geolocation identification, a crucial component of geographic information retrieval (gir), is the task of resolving the location, i.e., coordinates of a place, based on the reference to it in a text. it requires a combination of language and environmental knowledge. on top of the usual non-spatial linguistic challenges in natural language understanding (nlu), such as named entity recognition (ner), anaphora resolution, bridging anaphora, etc., the textual geolocation task presents geospatial challenges that require multimodal processing and grounding @xcite @xcite @xcite . proper names, such as 'rabin square', also known as named entities in natural language procesing (nlp), and as rigid designators in formal semantics @xcite , can be easily grounded based on a gazetteer or a simple map. however, geolocating linguistic terms that involve spatial expressions without the explicit mention of a proper name still present an open challenge. this interpretation challenge includes the understanding and resolution of (at least): (i) definite descriptions, such as 'the school' (ii) geospatial terms, such as cardinal directions; 'east of'; and (iii) geospatial numerical reasoning; 'two buildings away from the pharmacy'. to address these and other challenges, we need to both ground entity mentions to their corresponding physical entities in the environment, and to reason about geospatial relations expressed between entities -these two processes being closely intertwined. to do so, we need a corpus for the geolocation task that maps rich geospatial place descriptions to their corresponding location coordinates. however, current corpora for geolocation are based on naturally-occurring open-source resources, such as wikipedia articles @xcite @xcite @xcite , which are not spatially oriented, i.e., the description of locations is implicit or absent in the corresponding text. subsequently, the accuracy of retrieval is fairly low (around 100 km). furthermore, all geolocation datasets previously studied in nlp are in english, with a dearth of corpora for low-resource languages, in particular, for morphologically rich languages, such as hebrew. to understand the geolocation challenges and build models that do various spatial reasoning tasks, english cannot be our sole focus @xcite . hebrew, a semitic morphologically rich language is notoriously difficult to parse @xcite . moreover, resources that are place description: the place is located near the rothschild complex -at the end of rothschild street, as you go towards the sea, take a right for about three streets and then you will see the tower high above you. available for hebrew nlp research focus on traditional tasks, such as part-of-speech (pos) tagging, syntactic parsing, etc; and lack corpora for understanding and reasoning in real-world situations. in this work we present hegel, a novel dataset for hebrew geo-location, the first ever hebrew nlu benchmark involving both grounding and geospatial reasoning. to create hegel, we crowdsourced 5,649 geospatially-oriented hebrew place descriptions of various place types from three cities in israel. we designed our task based on a realistic scenario of human place description, relying on people's memory of the world, rather than, e.g., using a map @xcite . crucially, relying on environmental cognition results in various levels of geospatial knowledge @xcite that are manifested in the descriptions and the geospatial reasoning that is required to resolve their location @xcite . to avoid the much simpler task of grounding proper named entities, we explicitly restricted the use of proper names in the description of the place and adjacent landmarks. unlike the text-based navigation task @xcite @xcite @xcite , which requires representing an agent's current perspective, reflecting its route knowledge, we show that the hegel task requires a full-environment representation, thus, capturing complex geospatial relations among multiple physical entities. through a thorough linguistic and empirical analysis, we demonstrate the characteristics and challenges associated with hebrew place descriptions, showing that hegel serves both as a challenging nlu benchmark and as a corpus for geospatial cognition research. ## the hegel task and dataset this work addresses the task of geolocating places on a map based on natural language (nl) geospatial descriptions that are given in a colloquial language and based on participants' memory of the environment (i.e., cognitive map). the input to the hegel task is as follows: (i) an nl place description of the whereabouts of the place, and (ii) a map with rich details of the environment (e.g., physical entities names, geospatial relations, and attributes). the output is a pair of coordinates (x,y) specifying the physical location of the place described in the text. figure 1 shows an example of a place description from hegel translated from hebrew. to simplify the crowdsourcing task and encourage participants' engagement, we frame the data crowdsourcing process as the well-known game, the treasure hunt task @xcite , in which the instructor-participant is required to describe in writing the location of the treasure, a known place in the city, to a different followerparticipant who then needs to locate it on a map. thus, the online assignment is divided into two tasks: the instructor's writing of place descriptions and the follower's validation. to avoid preconceived notions as to the 'correct' way to describe a place, we first presented the participants with the task of writing a place description, and once completed, the validation task was given. 2 we hereby provide the details of the two ui tasks: ## data statistics and analysis the resulting hegel dataset contains 5,649 validated descriptions paired with their coordinates on a map. the locations are divided among three cities: 2,142 in tel aviv, 1,442 in haifa, and 2,065 in jerusalem. 1,833 participants completed the writing task, inserting in total 10,946 place descriptions, and 2,050 participants completed 12,655 validation tasks. the dataset is balanced, with about 33 descriptions per place. figure 2 shows a venn diagram representing the relation of the three sets of city-based vocabularies (formed from unique lemmas produced by @xcite lemmatization tool). the intersection of the three cities contains only 15.07% of the entire vocabulary (the union of the three cities' vocabularies). the shared language is not focused on city-specific terms, such as 'knesset'. instead, it includes rich spatial terms, such as 'between', modified prepositions such as 'next to', and nondefinite entities, such as 'street'. from the venn diagram we also conclude that almost half of the lemmas of the three vocabularies, corresponding to the three cities, contain city-specific lemmas: 48.6%, 40.65%, and 49.3% for tel aviv, haifa, and jerusalem, respectively. as such, hegel enables a city-split setup, training on one city and testing on a different unseen city, where city-reserved named entities present an out-of-vocabulary (oov) challenge for models trained on another city. we further perform a quantitative analysis of word tokens and lemmas that appear in hegel, depicted in in ## experiments we create a zero-shot (zs) city-based split, such that we train on one city and test on another. the train, development, and test sets correspond to the descriptions collected in tel aviv, haifa, and jerusalem, respectively. we evaluate different baseline models for the geolocation task on the hegel dataset. we use three evaluation metrics based on retrieval error: mean, median, and task completion (tc) accuracy -the percentage of place descriptions located within the 300 meters threshold. we provide three baselines for the hegel task. we first assess a brute-force ner approach; i.e., we test whether recognizing named entities in the text and retrieving their corresponding coordinates is sufficient for solving the hegel task of geolocation. to this end, we used google maps api and produced two baseline models: (i) google maps api query -we queried the api with the full raw text descriptions as input, with no prepossessing; and (ii) oracle ner -we queried all 1-5 n-grams against google maps api and retrieved the closest geolocation to the goal. in our second approach, we employ a dualencoder model. one encoder encodes the text using a hebrew monolingual pre-trained encoder, aleph-bert @xcite , which produces a 768dimension vector representation of the text. the other encoder processes the environment, which is represented as a graph based on osm data. each point of interest in the graph is connected to an s2cell 4 , which contains its geometry and is based on s2-geometry. these s2cells are encoded using a random-walk algorithm to produce a 64dimensional vector for each cell. these vectors are then passed through a linear layer to produce 768-dimensional vectors. we calculate the cosine similarity score between the text and environment vectors and use it to align the respective representations via maximization of the cosine similarity score with a cross-entropy loss over the scores. 4 s2cells are based on s2-geometry ( https://s2geometry.io/ ), a hierarchical discretization of the earth's surface @xcite . performing an anova test, we found a significantly (@xmath1.05) different distribution between place type descriptions and the retrieval error of the oracle ner. the mean retrieval error of the path and node place types were the lowest in both human verification and oracle ner. this suggests that both of these place types are easier for humans to geolocate. the results in ## conclusion the contribution of this paper is threefold. first, we present the first geolocation benchmark with hebrew place descriptions. second, to the best of our knowledge, this is the only crowdsourced geolocation dataset, thus, eliciting explicit geospatial descriptions, allowing for better retrieval resolution. finally, our analysis shows that the dataset presents complex spatial reasoning challenges which require novel environmental model representation.