Paper: G2PTL: A Pre-trained Model for Delivery Address and its Applications in Logistics System (ICLR-2024)

Reviewer comment (Reproducibility):
At inference time, do I need to have graph as input besides the text input for this model? From the model design, it seems that the graphomer is used in the model, thus I would expect edge and node input for the model? If the task is just address normalization without graph input, can we still use this model?

Author response:
**Questions 1: At inference time, do I need to have graph as input besides the text input for this model? From the model design, it seems that the graphomer is used in the model, thus I would expect edge and node input for the model? If the task is just address normalization without graph input, can we still use this model?**

**Response：**

During the model inference phase, addresses need to be constructed as a graph. For tasks such as address normalization without graph input: Just build a single-node subgraph based on the address (This is a Self-Loop subgraph, edge_type_code = 1000 (Have a self-loop, no delivery route, no AOI co-locate, and no Alias)), and then enter this subgraph into the model.

Because when sampling subgraphs during the training phase, a weighted adjacency matrix is used to describe the relationship between nodes, and the weight value is edge_type_code. In particular, we use self-loop edge to describe the relationship between the node and itself. Its edge_type_code is 1000, which is used to distinguish it from the other 3 types. Therefore, when the downstream task does not have a graph input, you only need to build a single-node graph and add a self-loop edge to this node, which can be used for fine-tuning and inference of a single address. (This part of the code is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL. All experiments in the paper are tested in this way.)

Reviewer comment (Experiments):
Do we need to have a correct geographical map before using the proposed model in real life? What if the map is incorrect? How robustness of this model when the graphical information in inference time is noisy?

Author response:
**Questions 2: Do we need to have a correct geographical map before using the proposed model in real life? What if the map is incorrect? How robustness of this model when the graphical information in inference time is noisy?**

**Response：**

1. Whether a geographical map is required depends on your application scene.

    ○ If your application scene is a graph-level task (such as graph-based ETA, path prediction, etc.), we recommend that you input correct geographical map information to the model, which will help the model make more accurate judgments. If you do not have a geographical map, you can construct S2 co-locate edges through the S2 method to replace the AOI co-locate edges (this edge can also represent part of the geographical co-location information, but the method of dividing the earth is different), but the lack of other edges will reduce the performance of the model.

    ○ If your application scene is a single-address level task (such as address normalization, geocoding, etc.), at this time, you only need to construct the address as a single-node graph(The method is in the response to question 1).
2. When creating large-scale heterogeneous graphs, noise is already present in it ：

    ○ AOI co-locate：Limited by the performance of the geocoding service, the data source of AOI co-location edge may be wrong

    ○ Delivery Route: the courier’s delivery route may be wrong (positioning drift, etc.)

    ○ Alias: As shown in Fig1, each node will have multiple address descriptions, and the user will bring in input noise when writing the address. 

We believe that these noises will slowly disappear as the amount of training data increases, because these noises are unbiased. At the same time, these noises will enhance the adaptability and robustness of the model. This is also reflected in our actual industrial deployment applications. We have deployed G2PTL into our online logistics system, which processes tens of millions of addresses every day (with the development of logistics business, there are many a new address that the model has never seen before). After the deployment of G2PTL, the accuracy of the address entity recognition online service has been improved by about 9% (84%->93%, the previous online model was BERT), which also reflects the good anti-noise performance of the model.

Reviewer comment (Experiments):
How large the graph do we need to prepare before using this model for address normalization? Do we need to have a global map or local map? If only local maps are used, is it possible that this method cannot learn the address similarity beyond a local region?

Author response:
**Questions 3: How large the graph do we need to prepare before using this model for address normalization? Do we need to have a global map or local map? If only local maps are used, is it possible that this method cannot learn the address similarity beyond a local region?**

**Response：**

If you want to use this model for address normalization, you do not need to prepare a map, but you need to use the addresses to build a single-node graph as input (The method is in the response to question 1).

In the inference phase, the model only needs a local map; in the training phase, a global map is needed.

If you use a local map, our method can only learn information such as the spatial relationship of addresses inside the local map, but cannot learn the spatial relationship between addresses outside the local map. For addresses outside the local map, the model can only calculate the similarity of text-level addresses because the model has not learned the spatial information of the corresponding address.

Reviewer comment (Experiments):
Given that a fixed heterogeneous graph has already been predefined, how can the proposed method be scaled to new addresses?

Author response:
1. If you are only using the model for inference on new addresses, you only need to build a single-node subgraph based on the new address(This is a Self-Loop subgraph, edge_type_code = 1000 (Have a self-loop, no delivery route, no AOI co-locate, and no Alias)), and then enter this subgraph into the model. Because when sampling subgraphs during the training phase, a weighted adjacency matrix is used to describe the relationship between nodes, and the weight value is edge_type_code. In particular, we use self-loop edge to describe the relationship between the node and itself. Its edge_type_code is 1000, which is used to distinguish it from the other 3 types. Therefore, when a downstream task takes a new address as input, you only need to build a single-node graph and add a self-loop edge to this node, which can be used for fine-tuning and inference of a single address. (This part of the code is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL. All experiments in the paper are tested in this way.)  

2. If you want to add new addresses to the model for fine-tuning. There are 2 ways to extend the model to new addresses:

    ○ If there are a large number of addresses, a new heterogeneous graph can be constructed and trained according to the pre-training method we proposed. (If you cannot obtain edges with domain knowledge such as AOI co-locate and Delivery Route, you can use edges similar to S2 co-locate (refer to ERNIE-GeoL，KDD‘22))

    ○ If there are only a small number of addresses (<=10k), we recommend fine-tuning the model using single-node subgraphs constructed from new addresses.

Reviewer comment (Reproducibility):
How do you balance the significance of various tasks within the loss function? Based on the findings from the ablation experiments, it appears that HTC plays a more crucial role in pre-training.

Author response:
We have observed that the convergence speeds of different losses are different during the training process. Among them, HTC-Loss converges the fastest because this task is simpler than MLM and Geocoding tasks (HTC only needs to focus on administrative hierarchy relationships among address entities. It only needs to learn all geographical administrative relationships, and the solution space is smaller than the other two tasks). We did not do any special work to adjust the balance between the three loss functions. We just maintained the loss weight setting of 1:1:1. Under this setting, we trained for about 1 week. 

Due to training time and cost reasons, we terminated the training. At this time, HTC task has converged, MLM task is close to convergence, and the GC task has not fully converged. Our training loss log is also in the open source link:https://huggingface.co/Cainiao-AI/G2PTL.

We believe that different tasks focus on different knowledge.

In the experiment in Table 1, both the AEP and AET tasks are for a single address, and they should pay more attention to the the relationship of sub-words in the address. Therefore, from the experimental results, the importance of the HTC task will be higher, which is consistent with the goal of HTC task design.

For tasks related to geographical knowledge (absolute position positioning of addresses and relative positions between addresses), the importance of GC will be higher (the ablation experiments of the GC task in Table 1 and the ablation experiments of table 5 in the Appendix both prove this).

The MLM task helps the model handle more input noise (incomplete, partially duplicated, misspelled, etc.) and ensures the model's ability to understand address semantics. The cases in Figures 6 and 7 also prove the effectiveness of the MLM task.

Reviewer comment (Presentation):
The right side of Figure 1 depicts an edge between node 1 and node 2 labeled "001", suggesting "no delivery route, no AOI co-location, and has Alias." However, the left side appears to contradict this by showing a delivery route between node 1 and node 2.

Author response:
Sorry, we didn't describe the edge encoding clearly enough. In the binary encoding result of the edge, position 0 is the last bit in the result(rightmost) and position 2 is the first bit(leftmost). Therefore, the edge labeled "001" between node 1 and node 2 in  Figure 1 indicates " no AOI co-locate, no alias, only delivery route". 

| code position | position 2 | position 1 | position 0 |  
| :--: | :--: | :--: | :--: |
| edge type description | alias | AOI co-locate | delivery route |  
| edge type code | 0 (None) | 0 (None) | 1 (Have) |