Paper: Active Retrosynthetic Planning Aware of Route Quality (ICLR-2024)

Reviewer comment (Evaluation):
My main concern is that it is not very clear to me what the quality metric would actually be in a real-life scenario and how it would be implemented. First, the framework proposed by the authors seems to only make sense if querying the quality metric is _expensive but not too expensive_: if the quality oracle is not much more expensive that the single-step model itself (e.g. if it's a yield prediction model), then there is no need for the adaptive querying; if the metric is very expensive (e.g. asking a human chemist or running the reaction in the lab) then it wouldn't make sense to allow the algorithm to query it autonomously, even if it only does so for e.g. 20% of the cases (as that would still be prohibitively expensive and the human/lab would become a bottleneck). What would a realistic quality metric be? The authors in the paper use a trained model, which perhaps falls into the category of "not expensive enough to bother with adaptive querying", unless the model is *much* larger than the single-step model itself. Or perhaps the quality metric could result from e.g. running expensive quantum chemistry simulation, but this is a bit forward-looking as I don't think any existing frameworks for such calculations are robust enough to plug them in directly. Overall, at the very least I would expect more discussion on how the quality metric should be realized in practice.

Author response:
**Real-life scenarios**

> We wholeheartedly agree with the reviewer that the quality metric required by our framework in a real-life scenario should be **expensive but not prohibitively so**. This consideration constitutes the primary motivation behind our active planning framework, aiming to query a minimum number of reaction quality annotations while still planing high-quality routes.
> 
> While our current implementation involves querying the surrogate model, our inspiration is drawn directly from real-life retrosynthesis planning scenarios, such as in online softwares like SYNTHIA, where **chemists are pivotal end users**. In this context, **integrating chemists as valuable resources into the AI planning process** will be invaluable for planning routes that are not only feasible but also of practical high-quality. We envision the successful deployment of our framework in this scenario for several reasons.
> - Online annotation by chemists introduces minimal time delays and manageable labor costs, making it an **ideal candidate for a route quality metric** that is expensive but not prohibitively so. 
> - Our framework is intentionally designed to be **compatible with various types of annotations, including a coarse-grained quality rating from 0 to 10**. We believe such a rating is sufficient for the planner to make satisfactory decisions. Additionally, this rating can also be seamlessly integrated into our current framework by replacing the bucket index to which a quality value belongs (see details in Section 3.2 in the revision) with this discrete rating.
>  - Chemists contribute **valuable insights beyond mere reaction yields**, such as knowledge about preferred reactions in real-world synthesis contexts, which can include factors like toxicity, material costs and work-up difficulty (post-process, like purification or separation).

> I guess this makes sense; my thinking was that if the chemist is queried for e.g. 20% of the reactions then it would still be a bottleneck, but perhaps if it's a few % then that would be fine. 

We concur with the reviewer's assessment. Indeed, our proposed ARP framework has demonstrated its effectiveness in reducing the query rate (as low as **below $10$%**) when the query cost is high, all the while maintaining an almost consistent normalized route quality. Further details and evidence can be found in Table 3 and 4.

> A related question would be how to train the model to make use of the oracle without actually calling the oracle, as one may want to avoid introducing chemists during the training process to save their time (also, an untrained model may intially try to query the oracle much more often than is optimal). Perhaps it would be fine to train with a surrogate model of chemist preference and then use an actual chemist during inference.  \

We value the insightful suggestion provided by the reviewer, which is consistent with our current protocol.
- Currently, training of our surrogate model involves two steps: (1) pre-training on the USPTO-MIT dataset, and (2) finetuning on an in-house expert dataset of routes featuring high-yield reactions. It is important to note that we introduce **step (2) deliberately to ensure that high predictive probabilities from our surrogate model align with high yields**. Thus, this surrogate model inherently reflects a preference towards high yields.
- In the future, we plan to **extend the training of the surrogate model by incorporating chemist preference** gathered from the literature and in-house experiments. Additionally, during inference, we aim to adhere to our motivation by deploying the model through interactive queries with chemists.
-------------------------------------
We are grateful that the reviewer has acknowledged the resolution of other concerns and expressed satisfaction with our revision. We appreciate the reviewer's insightful feedback on the real-application scenario of our framework, and will provide further details on it in the Introduction of the final version.

Reviewer comment (Writing):
In many places there is a missing space before parenthesis (see the "Multi-step planning" part of the "Related work" section for some examples).

Author response:
**Nitpicks** 

> We appreciate your advice to help us improve this paper and revise the mentioned nitpicks in the revision.

Reviewer comment (Writing):
"with is initialized" -> "which".

Author response:
**Nitpicks** 

> We appreciate your advice to help us improve this paper and revise the mentioned nitpicks in the revision.

Reviewer comment (Writing):
"Moreover, We define" -> lowercase "we".

Author response:
**Nitpicks** 

> We appreciate your advice to help us improve this paper and revise the mentioned nitpicks in the revision.

Reviewer comment (Writing):
"There are three steps for each roll" -> "rollout"?

Author response:
**Nitpicks** 

> We appreciate your advice to help us improve this paper and revise the mentioned nitpicks in the revision.

Reviewer comment (Writing):
First sentence in the "Reaction cost annotation" paragraph is a bit hard to parse.

Author response:
**Nitpicks** 

> We appreciate your advice to help us improve this paper and revise the mentioned nitpicks in the revision.

Reviewer comment (Evaluation):
Why doesn't the success rate change monotonically with the query rate, unlike the normalized route quality?

Author response:
We address **Q1** and **Q2** together because of their high correlation. We have added a discussion into Section 4.2 and a case illustration in Appendix D in the revision. 
- (1) **The success rate and route quality are two distinct and potentially conflicting objectives**, as illustrated in the case study of Fig 8 in Appendix D. In the figure, the root molecule $M_0$ has three candidate reactions, and $R_0$ is identified as a high quality reaction. However, the child molecule $M_1$ of $R_0$ is an unexpandable dead node with no further reaction candidates. 
    - If the planner selects based only on observable next-state molecular structures of $M_1$, $M_2$ and $M_3$, **being agnostic of the reaction quality values of $R_0$, $R_1$ and $R_2$,** it might select $R_2$ for its most synthesizable next-state molecule $M_3$. 
    - Alternatively, **given observable reaction quality values $R_0$, $R_1$ and $R_2$**, the planner could be misled into selecting $R_0$ due to its highest route quality. 
- (2) The "actor" in Table 3/4 is a policy network $\pi_\phi$ that decides whether to query the quality of reactions. By taking the reaction quality as rewards (see Eq.(1) for details), **the actor is trained to prioritize querying those reactions that contribute to high-quality routes**. 
- Considering (1) the disparity between high-quality and high success rates and (2) the actor optimized towards high-quality only, it is reasonable that
   - actor+critic does not show consistent and significant improvement on success rates compared to random+critic;
   - the success rate does not change monotonically with the query rate.

Reviewer comment (Novelty):
Important related work [1] that targets on multi-step planning should be discussed.

Author response:
We thank the reviewer for pointing out this recent  and important related work [1], which we have duly cited and discussed in the revision. 
 - [1] introduces a novel multi-step planning approach via in-context learning, departing from conventional search algorithms. However, its primary objective is still evaluating success rates of planned routes.
 - Our major contributions lie in an active framework capable of synsthesizing routes with not only high success rates but also high quality. Thus, we have proved the **compatibility and orthogonality of ours with other multi-step planning strategies** like Retro*; it is also readily extendable to state-of-the-art planning methods like [1].

Reviewer comment (Experiments):
In reaction cost annotation, the authors adopt a surrogate model to provide reaction cost annotation, and hope “the model prioritizes the identification of high-yield reactions over high-frequency reactions”. Can the authors empirically verify this claim?

Author response:
- Our active planning framework utilizes the **reaction quality** annotated by our surrogate model as **rewards**. Consequently, **as long as reaction qualities predicted by the surrogate model show high correlations with yields**, the model learns a policy that prioritizes high-yield reactions over high-frequency ones. 
 - We proceed to validate that reaction qualities by our surrogate model indeed demonstrates a high correlation with yields.
    - The training of our surrogate model involves two steps: (1) pre-training on the USPTO-MIT dataset, and (2) finetuning on an in-house expert dataset of routes featuring high-yield reactions. It is important to note that we **introduce step (2) precisely to ensure that high predictive probabilities from our surrogate model align with high yields**. 
    - To evaluate our surrogate model, we resort to a route-with-yield test set. Following the method described in [2], we extract synthesis routes with yields from the USPTO-milligram-scale reaction yield dataset [3]. For evaluation purposes, we randomly select 200 routes, encompassing approximately 1000 reactions. 
    - We thereby calculate the Pearson correlation coefficient(PCC) between the reaction quality predicted by our surrogate model and the literature yield. In Fig 7 of the revised manuscript, (a) illustrates the 0.059 PCC of the pre-trained model while (b) shows the **0.611 PCC of the finetuned model**, providing strong evidence that **the surrogate model accurately predicts yields**.

Reviewer comment (Evaluation):
Are there real cases when high-yield reactions and high-frequency reactions are not overlapping? Intuitively, high-frequency reactions are usually high-yield ones, otherwise they cannot be frequently collected in a dataset, right?

Author response:
Frequently collected reactions in a single-step dataset are **not necessarily** high-yield, which we substantiate based on an analysis from [3] that explores yields reported in the open-source USPTO dataset. 
 
 - The **USPTO dataset with reaction yields** in sub-gram scale [3] contains a large number of reactions and a broad range of superclasses, and a reaction distribution closely resembling that of the USPTO single-step dataset, such as USPTO-MIT. 
 -  The actual reaction yield distribution of the above dataset, originally presented in [3], is depicted in Fig 5c. Notably, a significant proportion of reactions within the dataset exhibits relatively low yields, affirming that the USPTO single-step dataset is **not inherently biased to high-yield reactions**. 
 - **High-frequency $\neq$ high-yield** Fig 5a (originally presented in [3]) shows various superclasses of reactions, where each color corresponds to a superclass and the coverage area of each color roughly represents the frequency of that superclass of reactions in the dataset. Combining Fig 5a and 5b, we conclude that high-frequency superclasses do not show a significant correlation with high yields. For example, the superclasses annotated in purple and cyan demonstrate low yields, with only the green reaction superclass corresponding to high yields in Fig 5b.

Reviewer comment (Experiments):
In Selection, is there any exploration or possibility to generate multiple routes?

Author response:
We thank the reviewer's great comment, and have accordingly updated Section 3.3 to detail the definition of **$Q^*$ which involves exploration**.
 
 Concretely, the function $Q^*$ is defined as 
 $$
 Q^*(s_t,a^r_t)=Q(s_t, a^r_t)+\beta\frac{\sqrt{N(s_{t-1},a^r_{t-1})}}{1+N(s_t,a^r_t)}
 $$
 which follows the UCT(Upper Confidence Bound applied to Trees) function to balance between exploitation of the route with the maximum $Q$ and exploration of those less frequently visited routes.

Reviewer comment (Reproducibility):
How is the masked value M set?

Author response:
- We set the mask as a $d^M$-dimensional trainable embedding. 
 - Meantime, it is crucial to emphasize the **consistency in the representation of $\mathcal{O}(a^r,a^q)$**:
   - When $a^q=1$, we derive $\mathcal{O}(a^r,a^q)$ from (1) discretizing continuous reaction quality values into $N^M$ discrete buckets, (2) learning $N^M$ trainable embeddings in $d^M$ dimensions for all buckets within our critic, and (3) determining the bucket index that the queried quality value $u$ belongs to and thereby the associated bucket embedding. 
   - when $a^q=0$, as mentioned above, $\mathcal{O}(a^r,a^q)=\mathbb{M}$ as a $d^M$ dimensional embedding. 
 - In our implementation, we consider $d^M=512$ and $N^M=18$ buckets which are defined in Fig 6 in the revision. These buckets are obtained via (1) collecting ~28M reactions during planning by GRASP and Retro$^*$, (2) computing their reaction qualities by our surrogate model, and (3) defining the bucket boundaries to ensure that each bin covers a similar number of reactions.

Reviewer comment (Evaluation):
During inference, does the method require the external surrogate model to obtain the reaction cost? How accurate/reliable is this surrogate model?

Author response:
- Our major contributions revolve around exactly an **active** planning framework that trains a querying actor/critic during training and later applies it **during inference** to plan a high-quality route with **minimal reaction quality annotation from an external surrogate model or a chemist expert**. 
 - Still, ours is compatible with existing multi-step planning methods such as Retro* as backbone planners. In scenarios where an external surrogate model or a chemist is unavailable, ours reduces to the backbone planner.  
 - In **the response to Q1**, we have empirically validated the effectiveness of our surrogate model. Besides, our **case study in Appendix E** showcases that ours achieves **much higher route quality ratings by chemists**. This outcome provides strong evidence that the surrogate model faithfully reflects yields.

Reviewer comment (Writing):
The term “reaction cost” is misleading and inconsistent: “cost” usually refers to some undesired property that should be as low as possible, however, this paper defines cost as yield (i.e., higher “cost” indicates better quality). This is misleading and counterintuitive. Meanwhile, “observing in Fig, 2 that a molecule with a low preceding reaction cost should be prioritized to expand first”, is clearly inconsistent with Fig. 2 (i.e., low-cost=low-quality route should not be prioritized).

Author response:
We apologize for any confusion arising from the term "reaction cost". For a more straightforward and intuitive understanding, we have revised all instances of the term "reaction cost" into "reaction quality" as well as Fig 2 in the updated version.

Reviewer comment (Writing):
The description of Expansion in the inference stage is not clear. Q* sometimes refers to a value, or a function (i.e., in Eq. (5)), which is confusing. What is Q*(a) and Q*(s) exactly?

Author response:
For a better understanding, we have replaced $Q^*(s_t)$ and $Q^*(a^r_t)$ with a unified form $Q^*(s_t, a^r_t)$ in Eq 5 and Eq 6. $Q^*$ is a function that quantifies the synthesizability of a state-action pair as outlined below. Inside the equation, $N(s, a^r)$ denotes the visit count number. $Q(s_t, a^r_t)$ is a value initialized by $Q_\theta(s_{t+1},\mathcal{O}(a^r_t,\pi_\phi(a^r_t)))$ and updated afterwards.
$$
Q^*(s_t,a^r_t)=Q(s_t, a^r_t)+\beta\frac{\sqrt{N(s_{t-1},a^r_{t-1})}}{1+N(s_t,a^r_t)}
$$
During a selection step, We recursively perform a reaction action selection $a^r_t=argmax_{a^{r}\in ch(s_t|T)}Q^*(s_t,a^{r})$ and obtain the next state $s_{t+1}$ by tranition function $\mathcal{P}$ until reaching a leaf molecule node. In a update step, we perform a $Q(s_t,a^r_t)$ value backward traversal as follow where $Q'$ is the $Q$ values of newly added state-action pairs.
$$
Q'(s_t,a^r_t) = Q(s_t,a^r_t) + \frac{1}{N(s_t,a_t)}(\gamma Q'-Q(s_t,a^r_t))
$$

Reviewer comment (Presentation):
somewhat disconnected from previous evaluation

Author response:
> - To begin, we seek to draw a comparison between the metrics employed in our framework and those utilized in prior studies.
>   - Our metrics: (c1) success rate, (o2) query rate, and (o3) normalized route quality. 
>   - Metrics in prior studies: (c1) success rate, (p2) average route length, and (p3) number of single-step model calls. 
> - We have already substantiated the superior performance of our framework in terms of (c1) success rate compared to previous planning methods.
> - Concerning (p2) average route length, its application in previous works aimed to quantify a specific aspect of route quality. However, it is worth noting that this metric is **independent of real-world route quality**, as illustrated in Figure 1 of the main text. This motivates our adoption of (o3) normalized route quality for a more faithful evaluation.
> - we extend our assessment to (p3) the number of single-step model calls. The performances of various baselines and our framework are outlined below. The GRASP approach achieves the fewest iterations among all baselines. Our approach, GRASP with ARP, surpasses GRASP by 1.2 in the benchmark dataset and by 3.9 in the expert dataset. In conclusion, our framework **enhances the planning efficiency** of the implemented planners."
> 
> 
> | Model | Retro*-0 | Retro* | Retro*-0+ | Retro*+ | EG-MCTS-0 | EG-MCTS | GRASP | Retro* with ARP | GRASP with ARP |
> | ---- | ---- | ---- | ---- | ---- | ----- | ---- | ---- | ----- | ----- |
> | Benchmark test set | 69.2 | 53.5 | 52.6 | 48.7 | 45.9 | 43.1 | 41.3 | 46.0 | 40.1 |
> | Expert test set | 36.7 | 33.2 | 28.9 | 25.7 | 33.4 | 28.9 | 22.6 | 25.6 | 18.7 |

Reviewer comment (Reproducibility):
the model to assign quality is not described in enough detail

Author response:
> - We had elucidated the training methodology for acquiring the surrogate model that assigns quality in the **paragraph of Reaction quality annotation in Section 4.1**. 
>   - "*Initially, the method in Guo et al. (2020) is employed to pre-train a model utilizing the USPTO-MIT dataset, followed by the fine-tuning of the model in reactions derived from the high-quality, expert-annotated dataset*"
>   - We introduced this fine-tuning step on the expert dataset consisting of high yield reactions precisely to ensure that high predictive probabilities from our surrogate model **align with high yields**.
> - We had included additional information on the implementation details in Appendix C and presented a thorough evaluation of the surrogate model itself in Appendix D.

Reviewer comment (Writing):
it is not convincingly defined what reaction quality is

Author response:
> - We had defined the labor-intensive reaction quality in the Introduction section.
>   - **Line 4 Section 1**: "*The ideal reaction qualities that meet real-world chemical practicability, e.g., **yield of a reaction**, can be either annotated experimentally in a laboratory or by experienced chemists.*"
> - We had also detailed how to implement the reaction qualities so far for our framework in the **paragraph of Reaction quality annotation in Section 4.1**:
>    - "*Unfortunately, there is no established large-scale available reaction data set with respective reaction qualities or promising reaction performance, e.g. yield, prediction model(Jiang et al. (2022)), and both lab verification and expert annotations are expensive and time-consuming. We adopt a surrogate model to provide reaction quality annotations. ... During deployment, it is practical to replace the surrogate model with a chemist to provide online annotations, e.g. a coarse-grained quality rating from 0 to 10.*"

Reviewer comment (Novelty):
prior work is not correctly attributed

Author response:
> As clarified in the aforementioned ``**Clarification on Our Major Contributions**'', our primary contributions stand distinct from the prior works [1-4]. Furthermore, our proposed framework further is compatible with and enhances the real-world practicality of there existing planners.
>   - The planning methods in [1]-[4] only accommodate reaction cost metrics that are **efficient to have yet inaccurate to reflect real-world reaction quality such as yields**.
>     - [1] utilzes a fixed heuristic based on starting material prices and reaction labor costs, unrelated to yields. 
>     - [2][3] consider the cost for each reaction as a uniform 1 and optimize towards shortest routes. However, a short route might have a lower yield than a long route. 
>     - [4] relies on a forward prediction likelihood biased to the high-frequency reactions and the SCScore. SCScore, which gauges molecule synthesizability (a.k.a. feasibility) but overlooks the real-world reaction performance quality.
>  - The methodologies above **require the annotation of costs for every reaction along a route**. In the pursuit of incorporating reliable reaction qualities derived from chemists or lab experimentation, which entail significant costs, these methods tend to be economically impractical for real-world deployment.
> - Therefore, our essential contribution lies in introducing  a framework designed to  **(1) actively decide when to query human evaluation (surrogate model in our current implementation)** and **(2) select a minimum number of reactions that are most informative to observe their quality annotations**. Our framework not only reduces query costs but also renders eminently practical in real-world scenarios.
> - By specifically addressing the challenge of annotation costs, a facet often overlooked by previous works, our framework **not only accommodates but synergistically integrates with existing models**. This compatibility facilitates the seamless implementation of their models within our framework, thereby extending their active query capabilities.

Reviewer comment (Novelty):
The claim of providing, for the first time, an insight into the disappointing practicality of existing retrosynthetic planners that regard single-step probabilities as reaction qualities is questionable, as different and flexible ways to incorporate step and route cost beyond single-step probabilities were already included in many prior works (e.g., Segler 2018, Coley 2019, Schwaller 2020), which also used reaction feasibility models as a proxy for quality. These works should be referenced accordingly.

Author response:
> In response to the reviewer's summary characterizing our work as ``an RL based search algorithm that incorporates some definition of route quality for retrosynthesis is proposed'', we express concerns about potential misinterpretations of the major contributions of our work. To address this, we elaborate as follows.
> - Our contribution lies in not first incorporation of route quality related metrics into retrosynthesis. 
> - Our contribution lies in being the first in acknowledging the cost of annotating reaction quality, and thereby enabling the practical incorporation of costly-yet-accurate route quality evaluation into retrosynthesis through an **active and interactive** manner. Our motivations and contributions can be encapsulated in the following points:
>   - (1) There has been **a lack of efficient-to-evaluate and meantime qualified route quality metrics**; for example,
>     - (a) Predicted yield by a prediction model [efficient but inaccurate]: constructing accurate and robust yield prediction models has historically been challenging, explaining existing models being confined to specific reaction types.
>     - (b) Route length [efficient but unfaithful]: The illustration in Fig. 1 of the main text demonstrates that a short route might yield less than a longer one.
>     - (c ) Single-step probabilities [efficient but inaccurate]: As noted in our Introduction, single-step probabilities primarily reflect feasibility rather than reaction quality.
>     - (d) SCScore [efficient but inaccurate]: While SCScore quantifies synthesizability (feasibility), it neglects actual reaction performance quality.
>     - (e) Chemist rating [more accurate but less inefficient]: Chemists offer an overall evaluation of reaction yield, but this process is less efficient than a prediction model.
>     - (f) Lab experiment verification [accurate but inefficient]: While reflective of groundtruth reaction quality, including yields, this approach is exceedingly time-consuming.
>   - (2) The preceding works, including [1]-[4] the reviewer mentioned, operate **under the assumption that annotating route quality is labor-free and trivial**. Consequently, they accommodate metrics (a)-(d), as incorporation of highly accurate evaluation (e)-(f) for every reaction along the planned route is deemed prohibitively costly.
>   - (3) Thus, we are motivated to propose this **"active"** retrosynthetic planning framework, strategically **minimizing the reactions where we query metrics such as (e)-(f) while sufficiently improve the route quality**.
>      - Example: In the case of a molecule requiring 25 synthesis steps, our framework endeavors to pinpoint only 2 key reactions (10% querying rate) for the evaluation of either (e) or (f).
>      - This framework **maintains compatibility with existing retrosynthesis planning methods**, a validation substantiated in our experiments on Retro* and GRASP. Take Liu et al. 2023 [3] as an example. If training the PDVN with ARP online, the synthesizability value Network $R$ and the cost value network $Q$ in PDVN are incorporated into the estimator $\mathcal{E(s_t)}$ in the ARP framework by $\mathcal{E(s_t)}=R(s_{t-1},a^r_{t-1})*Q(s_{t-1},a^r_{t-1})+(1-R(s_{t-1},a^r_{t-1}))*c_{dead}$.
>   - (4) In our current implementation, we adopt the surrogate model as a surrogate for (e)-(f), which however does not compromise the effectiveness of our active planner.

> [1] Segler, M., Preuss, M. & Waller, M. Planning chemical syntheses with deep neural networks and symbolic AI. Nature 555, 604–610 (2018).
>
> [2] John S. Schreck, Connor W. Coley, and Kyle J. M. Bishop Schreck. Learning Retrosynthetic Planning through Simulated Experience. ACS Central Science 2019
>
> [3] Guoqing Liu, Di Xue, Shufang Xie, Yingce Xia, Austin Tripp, Krzysztof Maziarz, Marwin Segler, Tao Qin, Zongzhang Zhang, and Tie-Yan Liu. 2023. Retrosynthetic planning with dual value networks. In Proceedings of the 40th International Conference on Machine Learning (ICML'23)
>
> [4] Schwaller P, Petraglia R, Zullo V, Nair VH, Haeuselmann RA, Pisoni R, et al. Predicting Retrosynthetic Pathways Using a Combined Linguistic Model and Hyper-Graph Exploration Strategy. ChemRxiv. Cambridge: Cambridge Open Engage; 2019.

Reviewer comment (Novelty):
The paper could receive a higher score if prior work is referenced properly, but in its current form this referencing is insufficient.

Author response:
> As clarified in the aforementioned ``**Clarification on Our Major Contributions**'', our primary contributions stand distinct from the prior works [1-4]. Furthermore, our proposed framework further is compatible with and enhances the real-world practicality of there existing planners.
>   - The planning methods in [1]-[4] only accommodate reaction cost metrics that are **efficient to have yet inaccurate to reflect real-world reaction quality such as yields**.
>     - [1] utilzes a fixed heuristic based on starting material prices and reaction labor costs, unrelated to yields. 
>     - [2][3] consider the cost for each reaction as a uniform 1 and optimize towards shortest routes. However, a short route might have a lower yield than a long route. 
>     - [4] relies on a forward prediction likelihood biased to the high-frequency reactions and the SCScore. SCScore, which gauges molecule synthesizability (a.k.a. feasibility) but overlooks the real-world reaction performance quality.
>  - The methodologies above **require the annotation of costs for every reaction along a route**. In the pursuit of incorporating reliable reaction qualities derived from chemists or lab experimentation, which entail significant costs, these methods tend to be economically impractical for real-world deployment.
> - Therefore, our essential contribution lies in introducing  a framework designed to  **(1) actively decide when to query human evaluation (surrogate model in our current implementation)** and **(2) select a minimum number of reactions that are most informative to observe their quality annotations**. Our framework not only reduces query costs but also renders eminently practical in real-world scenarios.
> - By specifically addressing the challenge of annotation costs, a facet often overlooked by previous works, our framework **not only accommodates but synergistically integrates with existing models**. This compatibility facilitates the seamless implementation of their models within our framework, thereby extending their active query capabilities.

Reviewer comment (Presentation):
I would suggest to order the related work section, in particular the Multi-step planning section, by the order in which the works appeared, because subsequent works influenced each other.

Author response:
> We arrange the order of the related work section by the search algorithm previous studies used. We highlight the search algorithm categories, e.g. $A^*$, $MCTS$，$RL$, in the revision version updated.