Geo- BERT Pre-training Model for Query Rewriting in POI Search (2021)

Abstract: Query Rewriting (QR) is proposed to solve the problem of the word mismatch between queries and documents in Web search. Existing approaches usually model QR with an end-to-end sequence-to-sequence (seq2seq) model. The state-of-the-art Transformer-based models can effectively learn textual semantics from user session logs, but they often ignore users’ geographic location information that is crucial for the Point-of-Interest (POI) search of map services. In this paper, we proposed a pre-training model, called Geo-BERT, to integrate semantics and geographic information in the pre-trained representations of POIs. Firstly, we simulate POI distribution in the real world as a graph, in which nodes represent POIs and multiple geographic granularities. Then we use graph representation learning methods to get geographic representations. Finally, we train a BERT-like pre-training model with text and POIs’ graph embeddings to get an integrated representation of both geographic and semantic information, and apply it in the QR of POI search. The proposed model achieves excellent accuracy on a wide range of real-world datasets of map services.

## introduction point-of-interest (poi) search plays an important role in map services, such as google maps, gaode maps, didi, etc. query rewriting (qr) is critical for poi search @xcite to solve the semantic gap between queries and pois, created by users' mistype. currently, lots of methods have been tried to solve the qr problem @xcite @xcite @xcite . recently, the transformer-based seq2seq models @xcite significantly improve the feature representation ability and rewriting performance. while the transformer-based rewriting method shows its effectiveness in qr, it could be further improved in the following aspects when applyed in poi search: (1) the input of poi search is different from the general search scenario, as it may contain rich geographic information such as the user's current location. for example, when people located in city a search "the olive" (a poi in city b ), yet they actually want to find "ten olive"(a poi in city a ). however, it is extremely hard to rewrite "the olive" without the position information. (2) sometimes the location information is useless, while user's intention city is mainly obtained through query. effectively capturing the geographic information corresponding to the query becomes particularly crucial to qr tasks in poi search. to solve the above challenges, we propose a pretraining model called geo-bert that combines geographic feature graph with textual semantics in the qr task. first, we introduce a geographic feature graph to map multiple geographic granularity information to a unified graph representation space. specifically, we connect the neighboring pois to each other based on the longitude and latitude, meanwhile we connect the different administrative district granularity together with the above poi. after that, we propose a pre-training model that integrates text and pois' graph embeddings, and fuse geographic features into the text semantic space by predicting masked geographic information. finally, we fuse the pre-training model of geographic text into a transformer-based seq2seq model. our contributions can be summarized as follows. ## background usually, incorporating external knowledge could enhance the performance of nlp tasks @xcite @xcite . graphbased representation is able to express structured external knowledge effectively, @xcite and leverages node feature information to infer unseen data by aggregating subsampled local neighborhoods. @xcite incorporate breadth-first search and depth-frst search in neighborhood sampling to learn node embeddings. @xcite use subgraph sampling to reduce time and memory cost when using graph convolutuoin neural networks to learn larger graphs. recently, pre-training models such as bert (jacob devlin and toutanova, 2019) have shown their power in both understanding and generative tasks @xcite . @xcite raise a bert-like model to incorporate informative entities in knowledge graphs. considering that pois' geographic neighborhood relationship can be also expressed as graphs, we follow @xcite to incorporate geographic information in transformer-based query rewrite models. ## methodology in this section, we present the overall framework(see figure 1 ) of the proposed model. ## conclusion in this paper, we proposed a pre-training model called geo-bert, and applied it to the qr task in poi search. specially, we adopt a multiple geographic granularity graph and combine texual semantics with geographic infomation of pois. the proposed pre-trained model adopts sepcial masked strategy to learn meaningful geographic features. experimental results show that our model outperforms many strong baselines on a wide range of real-world datasets of map services.