Paper: Routing with Rich Text Queries via Next-Vertex Prediction Models (ICLR-2024)

Reviewer comment (Experiments):
There is a lack of comparable baselines to the proposed method. The authors claim that the unguided search electrical flows with omnipotent referee is a strong baseline, but several other methods are mentioned in the introduction and related work seem to be viable candidates. If these methods are inadequate as the authors claim, then results should be shown against a reasonable cross section of such methods showing where these methods fail and the proposed method succeeds.

Author response:
The main contribution of our paper is demonstrating that a next-vertex prediction approach yields surprisingly good results for natural language constrained routing. To the best of our knowledge, there are only two alternative approaches: the use of cost modifiers (learning edge costs so one can run an algorithm like Dijkstra’s) and the use of unguided search algorithms (e.g., using electrical flows). For the former, it is immediate that it is impossible for cost modifiers to handle point-of-interest-like queries, which would require route-level costs that cannot be decomposed additively into edge costs. For the latter, we do include an electrical flows baseline, which clearly demonstrates that the routing problems in our experiment are non-trivial and difficult. We are open to implementing other baseline alternatives to our next-vertex prediction approach, but are not aware of any others.

We also want to note that there are ways that one could have implemented our next-vertex prediction approach differently. One could have used a different network to embed the receptive field, e.g. with a graph neural network rather than a transformer. However, we believe that we already have found a reasonable and effective implementation that demonstrates the efficacy of a next-vertex prediction approach; further architectural tunings are interesting but beyond the scope of this paper.

Reviewer comment (Evaluation):
The results do not elucidate why the proposed method works. The authors present a large volume of great results, but none of which that explain how the model is working. It seems like there is a lack of information for the model to adequately plan a route using just local information in the network.

Author response:
We agree it may be initially surprising that only local information is needed for constrained routing. Two factors play a role.

1) *Real-world road networks (which is what our experiments use) are easier to route on than worst-case graphs.* This is because travel time between locations usually approximates geographic distance up to small constant factors. Our model performs poorly on graphs that deviate from this, e.g. where the fastest route to a destination 1km away is to route through a midpoint 50km away.

2) *In beam search, each candidate route allows our model to explore a different local region of the graph.* Our models may therefore be limited to using local information but can be strategic about where to collect this local information. In fact, we can think of beam search as performing some efficient graph search. Even with a 1-hop receptive field, a sufficiently large beam width lets one observe the entire graph before choosing a route. Our model not benefiting from a larger receptive field (Figure 11) is also therefore because the most effective way to give our model more global information is to increase its beam width (Figure 4).

Reviewer comment (Experiments):
The complexity of the natural language queries does not seem to be a large component of the solution, but does seem to be a large component of how complex the problem could be in practice. It seems like the method is just converting natural language to a one-hot encoding a points of interest and matching those with the points of interest in the receptive field of the encoded vertices in the input. Are the natural language queries in the experimental results accomplishing more than just one-hot matching between the natural language query and the points of interest in the receptive field?

Author response:
Our method does not use structured query inputs that can be directly mapped to one-hot encodings. Rather, it directly takes as input unstructured natural language queries from users. Our dataset of natural language user queries are complex and diverse (see Table 24), and involve “and”/”or” combinations that require explicit reasoning.
We can be confident that our models perform some form of non-trivial natural language processing since we use a test set that exclusively includes previously unseen user queries.

Reviewer comment (Evaluation):
Are there particular types of instances where the model performs quite well and other instances where the model performs poorly?

Author response:
We agree it may be initially surprising that only local information is needed for constrained routing. Two factors play a role.

1) *Real-world road networks (which is what our experiments use) are easier to route on than worst-case graphs.* This is because travel time between locations usually approximates geographic distance up to small constant factors. Our model performs poorly on graphs that deviate from this, e.g. where the fastest route to a destination 1km away is to route through a midpoint 50km away.

2) *In beam search, each candidate route allows our model to explore a different local region of the graph.* Our models may therefore be limited to using local information but can be strategic about where to collect this local information. In fact, we can think of beam search as performing some efficient graph search. Even with a 1-hop receptive field, a sufficiently large beam width lets one observe the entire graph before choosing a route. Our model not benefiting from a larger receptive field (Figure 11) is also therefore because the most effective way to give our model more global information is to increase its beam width (Figure 4).

Reviewer comment (Experiments):
Based on Figure 11, it appears that the method does not benefit from a larger than a 2 (or 3)-hop receptive field. Is this a fact of the types of routes being considered at training and test time? Could the method benefit from a larger receptive field if the

Author response:
We agree it may be initially surprising that only local information is needed for constrained routing. Two factors play a role.

1) *Real-world road networks (which is what our experiments use) are easier to route on than worst-case graphs.* This is because travel time between locations usually approximates geographic distance up to small constant factors. Our model performs poorly on graphs that deviate from this, e.g. where the fastest route to a destination 1km away is to route through a midpoint 50km away.

2) *In beam search, each candidate route allows our model to explore a different local region of the graph.* Our models may therefore be limited to using local information but can be strategic about where to collect this local information. In fact, we can think of beam search as performing some efficient graph search. Even with a 1-hop receptive field, a sufficiently large beam width lets one observe the entire graph before choosing a route. Our model not benefiting from a larger receptive field (Figure 11) is also therefore because the most effective way to give our model more global information is to increase its beam width (Figure 4).

Reviewer comment (Evaluation):
What information is the model using to "plan" its route? Is it using the coordinates of the destination relative to the source, and just choosing next vertices which greedily move in that direction (analogously for points of interest specified in the constraints)? It seems like there should not always be enough information in the input to accurately predict the next vertex on the path? Are there any examples of failure cases of this method?

Author response:
Our model is able to observe the coordinates of the destination, source, and every vertex within a k-hop neighborhood of one’s candidate route.

Reviewer comment (Reproducibility):
The paper could benefit from more detailed implementation details to facilitate the reproducibility of the study. Providing code or more information on the specific choices of model architecture, training procedures, and evaluation metrics would make it easier for other researchers to replicate and build upon the proposed method.

Author response:
Experiment details for reproducibility are already provided in Section C. This includes dataset details and dataset statistics, details on the choice of model architecture (C.1-C.3) and model hyperparameters (C.4). Hyperparameter tuning details are also already provided in Section B.7. We will be releasing the full source code for these experiments in the de-anonymized version of this paper post-acceptance.

Reviewer comment (Experiments):
The authors could include more ablation studies to better understand the impact of different components of the proposed method; detailed analysis could provide a more comprehensive understanding of the factors that contribute to the method's success.

Author response:
We have conducted extensive ablation studies. As noted on the bottom of Page 9, we include in the Appendix additional experiments that study the relationship between model performance and (1) how much of the road network the models observe (Figure 11), (2) the scale of the models’ network architectures (Figure 12, 13), (3) the difficulty of a dataset’s route customization queries (Figure 14), and (4) one’s choice of inference algorithm (Figure 23). We also perform ablation studies of the road embedding network (Figure 17) and secondary scoring model (Figure 22). We also visualize and analyze the attention matrices of our networks (Figures 15, 16) and the road embeddings that our networks learn (Figure 18).

Reviewer comment (Evaluation):
How does the proposed method compare to other state-of-the-art approaches in terms of computational efficiency and scalability? Providing a detailed analysis of the method's performance on large-scale graphs and complex queries would help to establish its advantages over existing techniques.

Author response:
We already conduct experiments where we vary the scale of our model (Section B.2) and the density of points-of-interest in the road network (Section B.3). Our experiments also already study how our performance varies for simpler vs more complicated queries; for instance, compare the left and right-hand columns of Table 1.

Reviewer comment (Evaluation):
Are there any potential limitations or challenges in applying the proposed method to real-world scenarios with more complex constraints and larger datasets? Discussing these limitations and potential solutions would provide a more comprehensive understanding of the method's applicability and future research directions.

Author response:
Experiments are conducted on the US road network and on a dataset of natural language text queries with almost 1 billion unique sentences.

Reviewer comment (Theory):
Relying only on local graph context could limit long-range reasoning. The paper relies on representing each vertex and edge using features derived only from their local neighborhood in the graph. This local context allows the model to scale to massive graphs. However, it means the model may struggle with some types of long-range reasoning during routing. One example is to identify long shortcuts in the graph between distant vertices. With only local context, long shortcuts may not be recognized. Another example is reasoning about global properties of the graph such as determining a route which should avoid an entire region of the graph. In summary, the reliance on local context enables scaling to large graphs but inherently restricts the model's ability to reason about global graph structure and long-range dependencies. This could become a limitation for certain complex routing queries that require broader reasoning.

Author response:
We found that local graph context is sufficient for real world routing problems. Our experiment in Section B.1 compares the performance of models allowed differing amounts of graph context, and finds that one does not gain any statistically significant amount of long-range reasoning capability by using a more than three-hop neighborhood as context.

There are two main reasons local graph context is sufficient.
1) *Real-world road networks (which is what our experiments use) are easier to route on than worst-case graphs.* This is because travel time between locations usually approximates geographic distance up to small constant factors.
2) *In beam search, each candidate route allows our model to explore a different local region of the graph.* Our models may therefore be limited to using local information but can be strategic about where to collect this local information. In fact, we can think of beam search as performing some efficient graph search. Even with a 1-hop receptive field, a sufficiently large beam width lets one observe the entire graph before choosing a route. We also note that we train a secondary model to choose from the final set of candidate routes; this secondary model has the full context of each of the routes when scoring/ranking a solution.

Reviewer comment (Experiments):
Lack comparisons to other learning-based routing methods. The paper does not provide direct comparisons to other machine learning approaches for routing, such as graph neural networks for routing, reinforcement learning for routing, attention-based models for vehicle routing. Without comparisons, it is hard to assess if the performance is truly state-of-the-art among learning-based methods.

Author response:
The main contribution of our paper is demonstrating that a next-vertex prediction approach yields surprisingly good results for natural language constrained routing. To the best of our knowledge, there are only two alternative approaches: the use of cost modifiers (learning edge costs so one can run an algorithm like Dijkstra’s) and the use of unguided search algorithms (e.g., using electrical flows). For the former, it is immediate that it is impossible for cost modifiers to handle point-of-interest-like queries, which would require route-level costs that cannot be decomposed additively into edge costs. For the latter, we do include an electrical flows baseline, which clearly demonstrates that the routing problems in our experiment are non-trivial and difficult. We are open to implementing other baseline alternatives to our next-vertex prediction approach, but are not aware of any others.

We also want to note that there are ways that one could have implemented our next-vertex prediction approach differently. One could have used a different network to embed the receptive field, e.g. with a graph neural network rather than a transformer. However, we believe that we already have found a reasonable and effective implementation that demonstrates the efficacy of a next-vertex prediction approach; further architectural tunings are interesting but beyond the scope of this paper. Similarly, we agree that one could have used reinforcement learning to tune our model, in the same way that people finetune large language models with reinforcement learning; however, we see this slightly orthogonal question to be beyond the scope of this work.