Paper: Massively Scalable Inverse Reinforcement Learning in Google Maps (ICLR-2024)

Reviewer comment (Reproducibility):
Generalizability and reproducibility: given the repeated nods to engineering and deployment constraints, how generalizable and reproducible are the results? How many organizations face global scale routing optimization? While the achieved improvements are definitely impressive in terms of e.g. customer satisfaction, the contribution to the scientific community is not clear-cut, especially given that code and data (I assume) will not be released. Reproducibility and code / data availability are not even mentioned in passing.

Author response:
* **Re: how generalizable and reproducible are the results?** In Figure 6 we studied one aspect of generalizability, namely the ability of reward models to generalize across geographic regions. The p-value analysis in Table 1 and Appendix D.3 provides strong statistical guarantees on our claims. Not shown in this paper, we also studied the generalizability of our results _across time_, and only noted a minor drop in performance. We'd be happy to discuss other dimensions for testing the generalizability of the results. [For reproducibility concerns, see below discussion on data availability]

* **Re:  How many organizations face global scale routing optimization?** We're aware of a few organizations, plus a larger number of open-source projects for global routing based on OpenStreetMaps [1]. Perhaps more importantly, we want to call out Reviewer 5CwT's comment: 
> [Note to ACs and other reviewers]: Although the proposed method is framed for discrete MDP-based route optimization, note that there are several ways to generalize this framework to other interesting problem settings quite trivially, (see e.g. [A]) - as such, the findings here are actually quite broadly applicable.

    We chose this setting primarily because it's the largest scale application we could find to test our methods on.

* **Re: data and code availability** Due to strict user privacy requirements, we believe it's unlikely a routing dataset of the scope used in this paper could be publicly released in the foreseeable future. We strongly support open-sourcing experiments when legally and ethically permissible. 

**On wider applicability and generalizability** Will do!

**On data and code availability** We'll address these points in the paper, and release the code if deemed feasible.

Reviewer comment (Writing):
The wording "largest published benchmark of IRL algorithms [...]" (abstract, p.2, p.9) is misleading. I assume that the authors do not intend to publish the actual benchmark (e.g., data and evaluation metrics), but solely the results of this evaluation. This should be revised.

Author response:
* **Re: "largest published benchmark"** Perhaps "largest published empirical study" is more precise? This could also be replaced with "empirical analysis" or "experiment".

Reviewer comment (Writing):
Typos: "rouute" (Footnote 1)

Author response:
* **Re: Typo in footnote 1** Fixed! Thanks.

Reviewer comment (Experiments):
The experimental results are not from real-time execution of the proposed method and utilizes static features of the road network for route optimization. Incorporating dynamic features, for example varying traffic flow throughout a day, planned or unplanned diversions and road closures etc. would increase the difficulty of obtaining a scalable DP approach.

Author response:
* **Re: Real-time features** Due to the format of our training examples (described shortly), using dynamic instead of static features would actually have no impact on training time or dataset disk space. Every example in our dataset consists of $(\tau, \mathcal{M}\_i)$ where $\tau$ is the demonstration trajectory and $\mathcal{M}\_i$ is the corresponding region MDP (with static features). Although this format uses significant disk space due to storing an MDP with every demonstration trajectory, it simplifies training since examples are fully self-contained and may be serially read from disk. Disk space is also relatively cheap compared to compute costs.

    To switch from static to dynamic features, one could simply replace the static features in $\mathcal{M}\_i$ with a snapshot of the dynamic features at the time of the demonstration $\tau$. There would be zero change to the training pipeline – the lack of dynamic features is purely a limitation of the upstream logging infrastructure that creates $\mathcal{M}\_i$ and not a limitation of our machine learning setup.

Reviewer comment (Experiments):
The paper does not provide much details about the road graph. Would the authors be able to provide any intuition about the relation between the coarseness of the road network graph and the choice of H?

Author response:
* **Re: Road graph coarseness and choice of $H$** Broadly speaking, a coarser road graph will require a smaller choice of $H$. For example, if we split every road segment (node) into two equal shorter road segments, then one could double $H$ to get exactly equivalent training results. Note that with the 'merge' compression strategy, there is a single node between every feasible turn, meaning that the road graph is as coarse as possible without removing roads from the graph. As discussed in [6] and Section 6, we could prune additional nodes from the graph to make it even more coarse, although this creates significant engineering complexity.

    We would be happy to discuss any additional details about the road graph that you're interested in.

Reviewer comment (Experiments):
Fig 4 and 12 highlight an interesting outcome of the sparse reward modeling approach in correcting data quality errors. Is this a consistent observation across different geographical regions? Or is there any noticeable difference in the road graph network when this method of reward modeling demonstrates a particular benefit over others?

Author response:
* **Re: Fig 4 and 12** At least anecdotally, we observed this behavior across all geographies that we manually inspected. We suspect it occurs more frequently in regions with a higher rate of data quality issues, although statistically verifying this claim is challenging without knowing the true issue rate (an unobserved quantity).

Reviewer comment (Novelty):
The literature review is compact and the theory background provides a rapid but very nice summary of classical IRL results (in particular the unifying view of stochastic vs. deterministic policy trade-offs is helpful). One relevant piece of prior work that isn't mentioned however is the improved MaxEnt approach(es) by Snoswell et al. (e.g. [B, C]) - which address theoretical and empirical limitations with Ziebart's MaxEnt model, and are specifically applied to the problem of route optimization (albeit at city scale, not a global scale).

Author response:
* **Re: Snoswell et al.** Good catch, thanks. We've added these to our related work section.

Reviewer comment (Writing):
Unclear notation - Paragraph 'Parallelism strategies' under Sec 4 defines the global reward based on the sharded MDP as $r(s,a) = r_{\theta_i}(s,a),(s,a) \in \mathcal{M}_i$. This notation isn't clear to me - is there a typo here? Is this mean to be a product over the sharded individual reward functions?

Author response:
* **Re: Sec 4 parallelism strategies notation** [context] The global $\mathcal{M}$ is partitioned into $m$ disjoint MDPs $\mathcal{M}\_1, \dotsc, \mathcal{M}\_m$, where each state $s$ belongs to exactly one $\mathcal{M}\_i$. Each MDP $\mathcal{M}\_i$ has a corresponding learned reward $r\_{\theta\_i}$.

   We meant for the notation in this section to express the fact that for state-action pair $(s, a)$ corresponding to $\mathcal{M}\_i$, the reward $r(s, a)$ is solely determined by region-specific expert $r\_{\theta\_i}$. In other words, the reward of each $(s, a)$ is determined by a single $r\_{\theta\_i}$. Perhaps a more clear notation here would be $r(s, a) = r\_{\theta\_i}(s, a)$ **where** $(s, a) \in \mathcal{M}\_i$? We're open to alternative suggestions.

Reviewer comment (Writing):
Typo in footnote 1 - 'rouute'

Author response:
* **Re: Typo in footnote 1** Fixed! Thanks.