Paper: Curriculum reinforcement learning for quantum architecture search under hardware errors (ICLR-2024)

Reviewer comment (Experiments):
(Major) The absence of the comparison of some key methods in terms of theory and experiments. This paper emphasizes the design for quantum hardware errors, but there is no mention or comparison of some methods of QNAS that are also designed for noise, such as QuantumNAS (a noise-adaptive quantum circuit search) [1]. In addition, as far as I know, QCAS (Du et al., 2022) is tested on depolarizing errors, and QuantumDARTS (Wu et al., 2023) is evaluated readout errors. However, these methods under noise are not compared in the experiments.

Author response:
> Absence of the comparison of some key methods in terms of theory and experiments. Such as QuantumNAS (a noise-adaptive quantum circuit search) [1]. In addition, as far as I know, QCAS (Du et al., 2022) is tested on depolarizing errors, and quantumDARTS (Wu et al., 2023) is evaluated readout errors. However, these methods under noise are not compared in the experiments.

It's possible that due to the text structure, it was missed, but we did indeed consider these other works (or similar), and reported on it in the paper. 
For instance, we did compare our CRLQAS algorithm to QCAS (Du et al., 2022) under hardware constraints (both noise and qubit-connectivity) in Table 2 of the revised manuscript (Table 4 in the old one).
We did not highlight it enough to the reader in the first version which is rectified now.
We do not compare to QuantumNAS (Wang et al., 2022) as it is quite similar to QCAS algorithm (both are one-shot super-circuit methods).
We refer the referee to our *Response 3 to reviewer WndL* for a more detailed reasoning.

We respectfully disagree with the assessment of the reviewer that quantumDARTS (Wu et al., 2023) is evaluated with readout errors for quantum chemistry tasks. 
It only considers readout errors for the MNIST classification task which is not the focus of this work.
Consequently, we only compared to quantumDARTS in the noiseless case and numerically demonstrate that our CRLQAS performs better.
We believe that our comparison to existing methods (like QCAS, qubit-ADAPT VQE, quantumDARTS) is fair, and covers most of the existing approaches for quantum chemistry problems in both noisy and noiseless cases.

Reviewer comment (Presentation):
(Minor) The representations of some tables are not clear enough. For example, if the best result is bolded, then ‘7.21*10^-8’ and ‘2.9*10^-4’ should be bolded in Table 1. In Table 4, CRLQAS (RH) has two rows of results but they're not marked what do they mean.

Author response:
> The representations of some tables are not clear enough.

We appreciate the feedback on the inconsistencies within the Tables.
We have amended such mistakes, and hope that they improve the readability of the manuscript.
We note that the Table 4 from the older version is now Table 2 in the new version of the manuscript.

Reviewer comment (Novelty):
The proposed method is based on reinforcement learning, but only one related method is mentioned in this paper, and there are several QNSA works based on reinforcement learning that are not mentioned, such as [2,3,4]. Furthermore, compared with these methods, what is the innovation of CRLQAS, and whether it can reflect the advantage of the experiment?

Author response:
> The proposed method is based on reinforcement learning, but only one related method is mentioned in this paper, and there are several QNSA works based on reinforcement learning that are not mentioned, such as [2,3,4]. Furthermore, compared with these methods, what is the innovation of CRLQAS, and whether it can reflect the advantage of the experiment?


We would like to thank the reviewer for suggesting the papers. 
Regrettably, we were unaware of [3,4], as they were published within a month of the ICLR paper submission deadline. 
We respectfully disagree with the reviewer that [4] is a RL based QAS algorithm--it actually proposes a differentiable QAS approach, not geared towards RL for QAS but rather quantum reinforcement learning (not VQE).
Perhaps the reviewer wanted us to compare to a differentiable QAS method.
Indeed, we do compare to quantumDARTS (Wu et al., 2023), a state-of-the-art differentiable QAS algorithm and in fact often beat them for the molecules considered in this work across a spectrum of metrics.  

Additionally, references [2, 3] concentrate on addressing the state preparation problem specifically for $2$- and $3$-qubit GHZ states ([2] considers hardware noise) whereas our focus lies in addressing the fundamentally distinct challenge of ground state energy estimation for quantum chemistry tasks. Please refer to our *Response 2 to reviewer K92h* as well. We will add the relevant references in Related Works section in the updated manuscript.

Reviewer comment (Evaluation):
In Table 4, why is CRLQAS(wo-RH) better than CRLQAS? What is the meaning of random halting (RH)?

Author response:
> In Table 4, why is CRLQAS(wo-RH) better than CRLQAS? What is the meaning of random halting (RH)?

Random Halting is one of the new techniques that we proposed and described in sec. 3.3 of the manuscript. 
By default, in CRLQAS, we utilize all three novel features namely illegal actions, tensor-based encoding and random halting.
We denote wo-RH and RH as experiment configurations where the random halting feature is turned off and on respectively during CRLQAS training.
As the *reviewer wt33* rightly pointed out the role of illegal actions might be pivotal within the CRLQAS algorithm.
Consequently, we conducted an ablation study to assess the effectiveness of the new features introduced, both in noiseless and noisy scenarios. Detailed results are available in *Table 1 of Response 3 to reviewer wt33*. 
Our analysis reveals that incorporating illegal actions without random halting prompts the agent to achieve a positive signal (a successful episode) early on, albeit resulting in larger circuits.
Conversely, introducing random halting encourages the agent to discover shorter circuits, there is a trade-off as the agent receives the positive signal at a later stage.
We further clarify this in Table 2 in the revised manuscript (which was Table 4 in the old version of the manuscript).

Reviewer comment (Novelty):
What are the advantages and disadvantages of the proposed method and the Predictor-based QAS methods [5,6]?

Author response:
> What are the advantages and disadvantages of the proposed method and the Predictor-based QAS methods [5,6]?

Predictor-based QAS methods such as in [5,6] aim to predict circuit performance using a neural network. However, effectively training this neural network demands a comprehensive training set comprising diverse circuits and their actual performances. To guarantee accurate evaluations, gathering an ample amount of training samples becomes crucial, especially when navigating the combinatorially large search space of quantum circuits. Consequently, the data collection process still entails significant computational expenses.

Furthermore, both [5,6] tackle the QAS problem in the noiseless scenario, specifically addressing Ising problems. 
They meticulously select a gate set, including YY-even, YY-odd, ZZ-odd, ZZ-even (illustrated in fig. 4 of reference [5] and fig. 10 of reference [6]), drawing inspiration from the gate set utilized in the quantum alternating optimization algorithm (QAOA). 
The comparison to our method is hence difficult as we allow for a much harder problem, with a set of gates that is native to quantum hardware, and not to the problem.

Reviewer comment (Reproducibility):
The paper does not specify the function used to fit the spin orbital of the molecule. If the 'sto-3g' function is used, the claim of achieving chemical accuracy might be an overstatement. While it's not impossible, further clarification is necessary.

Author response:
> The paper does not specify the function used to fit the spin-orbital of the molecule. What is the function you used to fit the spin orbital of the molecule?

We are not 100\% certain we know what the reviewer means by the following sentence:
`the function used to fit the spin-orbital'.
In our experiments, we construct the molecular electronic structure Hamiltonians by standard methods using *OpenFermion* and *Psi4* packages.
In short, from the geometrical description of the molecule, and using the Born-Oppenheimer approximation, we fix a finite basis set, in our case STO-3G, to discretize the system.
Based on the subsequent remarks of the reviewer , it is likely they refer to the basis set used to discretize the system.
We have added a more elaborated description of the molecules, and the basis set used to compute their Hamiltonians at the beginning of Section 4 Experiments in the revised manuscript.

> If the 'STO-3G' function is used, the claim of achieving chemical accuracy might be an overstatement. While it's not impossible, further clarification is necessary.

While we think we understand the referee's point, we respectfully disagree that talking about chemical accuracy is not appropriate, but do agree that a clarification helps, and one is added in the revision.
In computational chemistry, chemical accuracy (CA) takes an exact value,
\begin{equation}
    \text{CA} = 1 \frac{\text{kcal}}{\text{mol}} = 0.0016 \text{Hartree},
\end{equation}
and it defines the precision such that a computational chemistry method is capable of making realistic predictions.
Our system thus achieves chemical accuracy with respect to the Hamiltonian that defines our cost function, and this is how the term is used in literature on quantum algorithms for chemistry.
Of course, the Hamiltonian we use is not the exact molecular second-quantized Hamiltonian, as it has been derived through numerous approximations, including the use of the STO-3G basis to describe a molecular system.
However, this concern is about computational chemistry methods, which is far beyond the scope of this manuscript.

Reviewer comment (Experiments):
I'm concerned about the computational cost and efficiency of the reinforcement learning approach, especially when adding the noise injection method. Even with GPU acceleration, reinforcement learning is inherently expensive. Given that the noise model is derived from "current" calibration data and the gate-level noise model is only an approximation, there are concerns about its accuracy. For instance, although noise might seem stable over a short time frame, over a more extended period (e.g., a week), one can observe significant shifts. If the framework has a high computational cost, it may yield unreliable results if the calibration data changes rapidly during training.

Author response:
>  I'm concerned about the computational cost and efficiency of the reinforcement learning approach, especially when adding the noise injection method. Even with GPU acceleration, reinforcement learning is inherently expensive. 

We agree with the reviewer that the computational cost of the method is high. We have to make a distinction between the computational cost of the reinforcement learning algorithm for the VQE problem, and the computational cost of the noise simulation backend. For the former, the computational cost of RL does not stem from the training of the underlying neural network, but from the search space of all the possible quantum circuits getting combinatorially large with an increase in the system size. Due to this both the episode lengths (number of actions in the episode) and the total number of episodes need to be increased to efficiently explore the search space.
In such a large space, any algorithm that is not based on human intuition [1] or domain knowledge (e.g., physics models)  has to pay this cost to explore the subspaces that can not be explored by the algorithms based on those [2, 3]. Yet the exploration of these subspaces, and their interpretation, could allow new physics-based models. For RLQAS we can claim our method is the best one so far for the task of from-scratch-circuit generation (i.e., starting from an empty circuit and adding gates one by one). 

For the latter, the computational cost of the noisy backend stems both due to the training-based (i.e., optimizing after each action) approach making too many queries to the backend and due to growing cost in qubit numbers and noise models. To tackle the first, we significantly decreased the function evaluations by introducing two variants of Adam-SPSA. 
To tackle the second, we developed the fastest noisy quantum circuit simulator (for RL-based QAS methods) to mitigate the computational cost of the noisy backend. We discuss the details of the computational cost of the noisy backend in Appendix G (of the revised manuscript) and answer the question below.

> What is the computational cost for your framework? Say, based on the GPU acceleration, on what specific classical hardware, how long does it take? Please specify in detail.

As explained in the appendix G (of the revised manuscript), the computational cost stems from the numerous queries to the noisy simulation backend that implements subsequent dense matrix-vector multiplications between $4^n$ dimensional Pauli-Transfer matrices and density matrices in Pauli-Liouville representation. Here $n$ denotes the number of qubits. The training times scale linearly with the total number of function evaluations (i.e., queries to the simulator) as $n_{train} = n_{feval} t_{feval}$. Here $n_{feval}$ denotes the total number of function evaluations, and $t_{feval}$ denotes the average value of a function evaluation. The total number of function evaluations is given by $n_e \times n_a \times n_f$. Here $n_e$ denotes the number of episodes, $n_a$ denotes the number of actions per episode, and $n_f$ denotes the number of function evaluations per action (utilized in optimization).  Each evaluation takes $t_{feval}^{(i)} \approx t_{feval}$ amount of time at each action $i$, which can roughly be taken as a constant around all actions for a given qubit number. We used a NVIDIA-A100 GPU for these simulations. In that hardware, $t_{feval}$ is roughly $450\mu s$ for four-qubit circuits, $430\mu s$ for three-qubit circuits, and (counter-intuitively) $480\mu s$ for two-qubit circuits. We apply $n_a = 40$ actions (when random halting is not applied) for each number of qubits, while $n_f = 500$ for two \& three-qubit circuits and $n_f = 1675$ (or $3300$ for some experiments) for $4$-qubit circuits when Adam-SPSA is used at each action (finite-sample noise is on). Also, we use a rather lightweight neural network, so we spend $20$ secs at each episode on neural network training for each number of qubits. Combining each of these constituents roughly fits our empirical tests on RL training time that yields an average $30$ seconds per episode for two & three qubit problem, and $50$ seconds per episode for four qubit problem on NVIDIA-A100 GPU. We used Weights \& Biases (WandB) platform to streamline our experiments. The existing statistics of WandB indicate that we roughly used $100$ days of GPU compute time for our experiments. These experiments involved using multiple random seeds (typically 3 and 5 for noisy and noiseless experiment, respectively) for each RL agent, and each training corresponded to a different noise model and/or qubit number.


[1] Melnikov et al., "Active learning machine learns to create new quantum experiments." PNAS 115, no. 6 (2018)

[2] Peruzzo et al., "A variational eigenvalue solver on a photonic quantum processor." Nature communications 5, no. 1 (2014)

[3] Wecker et al., "Progress towards practical quantum variational algorithms." Physical Review A 92, no. 4 (2015)

Reviewer comment (Writing):
Some parts of the paper could be clearer. For example, Table 4 should specify which molecule task is being discussed. There are several sections in the paper with similar clarity issues, making it harder to understand.

Author response:
> Some parts of the paper could be clearer. For example, Table 4 should specify which molecule task is being discussed. There are several sections in the paper with similar clarity issues, making it harder to understand.

We agree with the reviewer regards to Table 4 needing more clarity. We fixed the clarity issues, and added comparisons in the revised text. Additionally, we note that the Table 4 in the older version of the manuscript is now Table 2 in the new version of the manuscript.

Reviewer comment (Experiments):
There's a notable absence of results on real quantum hardware. Testing only on a noisy simulator might not be entirely convincing, especially when using reinforcement learning, which is assumed to achieve the upper bound of performance. Given that you utilize a noise model from the current noisy environment and consider only one seed, real hardware testing is crucial.

Author response:
> Given that you utilize a noise model from the current noisy environment and consider only one seed, real hardware testing is crucial.

Going back to the potential use of free access, a typical VQE function evaluation (query) on IBM Lagos device with $20,000$ shots takes roughly $0.3$ seconds and training even a single circuit needs more than a thousand evaluations.
Having said this, we can only barely manage to train just one episode of our method with the free access.
On the other hand, one could potentially look for an experimental group interested on running such methods.
However, in the vast majority of cases, experimental labs have dedicated hardware and software to run experiments which is really difficult to integrate for RL.
Nonetheless, the goal of running noisy classical simulations with realistic noise models is to bring evidence to experimental groups that RL can be used in real hardware.
We hope that our work motivates simultaneously theorists and experimentalists to run RL methods on real hardware for quantum architecture search.


> Have you tried to implement your final result on real hardware? - I believe it should be easy for you since you already get your optimal arch and parameters from noisy simulation. You can just run this circuit on real quantum hardware and obtain the result.

The first part of the question has been addressed in the *Response 4 to reviewer WndL*.
We have not attempted to run any of the experiments (training CRLQAS agent) in real hardware due to time and access limitations, even for the minimal examples of running the optimal circuits for all the molecular systems considered in this study.
While this experiment is interesting in its own right, it will not make the results more or less interesting: either the results are close, and our noise model is accurate, or the results are significantly different and we must adjust our noise model.
The really interesting experiment is to run the full CRLQAS pipeline such that one can study the behaviour of the RL agent, assess the actual computational cost, and study the differences with respect to noiseless and noisy classical simulations.
Unfortunately, to perform such an experiment we need access to a dedicated machine with no time limits.
Moreover, public access to IBM Quantum devices is limited to 10 minutes per month, as of November 2023, with exceedingly high waiting times in the queues. A typical VQE function evaluation (query) on these devices with $20,000$ shots takes roughly $0.3$ seconds and training even a single circuit needs more than a thousand evaluations.

However, we do perform a small comparative experiment on the real hardware and the noisy simulators (exactly modelled as in our work) for $H_2$ molecule with a circuit produced by a trained CRLQAS agent.
We will update the manuscript in the coming days with the results of this experiment.
In future, we hope to engage with experimental labs interested in attempting such a challenging computation.

Reviewer comment (Evaluation):
As with most approaches in the field of quantum computing, the evaluation is rather weak and does not allow drawing too strong a conclusion (which, luckily, the authors did not draw).

Author response:
The reviewer is of course correct that it is difficult to draw strong conclusions form the volume and scale of simulations and especially real QC runs which are broadly possible today, but as the rest of the community, we put in maximal efforts to push the boundary.
In our case, nearly a year's worth of work was invested in transitioning our simulation methods. We experimented with various off-the-shelf SDKs, such as Tensorflow Quantum, Qulacs, Qibo, and TorchQuantum, before settling on the relatively faster JAX framework.

As a result of this transition, we've developed an in-house JAX package specifically tailored to reinforcement based QAS.
This transition enabled us to perform slightly bigger simulations with more types of noise.

Reviewer comment (Presentation):
The presentation of the sequence of experiments is rather convoluted and is not helped by putting the plots very late in the document. More clear-cut scenario descriptions would be helpful.

Author response:
We are sorry to hear that the ordering as presented affected the readability for the reviewer.
Our approach followed the convention of our research group, focusing on presenting important contributions upfront, followed by results and detailed descriptions in the appendix. 
Due to page limitations, we moved some descriptions to the appendix.
To further enhance readability, we tried to improve on those and to ensure clarity, we have added more details in the appendix wherever applicable. 
If the reviewer has further suggestions for improving readability, we are more than happy to try to implement them in our work.

Reviewer comment (Evaluation):
The discussion lacks an explanation of how adaptations largely motivated by the accommodation of noise models manage to outperform the state-of-the-art in the noiseless case.

Author response:
We agree that additional ablation studies would strengthen the paper.
To this end, we ran simulations to conduct a detailed study to find out which of the newly proposed features stands out.
These simulations implement a controlled study of each feature by training a different CRLQAS agent for a given problem with and without each feature.
Our analysis reveals that incorporating illegal actions without random halting prompts the agent to achieve a positive signal (a successful episode) early on, albeit resulting in larger circuits.
Conversely, introducing random halting encourages the agent to discover shorter circuits, but there is a trade-off as the agent receives the positive signal at a later stage.
We provide a table (see *Table 1*) of these studies. 
We will update the manuscript and visualize the data from the *Table1* more tactfully in the coming days.

Reviewer comment (Writing):
The writing still has several problems including:  
- p1, second paragraph, last sentence: no verb  
- p2, fig1, caption: "(i)" without "(ii)"  
- p3: "Methods like this" should read "Methods like these"  
- p4, eq2: parameters/variables are not sufficiently explained in the text, at least not in a concise manner  
- p4, last paragraph before 3.1, first sentence: It is unclear how the "environment" is deterministic/stochastic. Refer to transition function, reward function etc.  
- p5, section3.3, throughout: Typo "T_{s^e}" instead of "T _s^e"?  
- p4, section3.4: do not put a comma after "where"  
- p6, section3.6: Use "it is", not "it's"  
- p6, section3.6: "the Appendix" randomly appears in a sentence

Author response:
We are thankful to the reviewer for pointing out the problems and typos in the text. In the revised version of the manuscript, we incorporated the changes and typing mistakes.

By "environment" is deterministic/stochastic we meant that the reward function can be stochastic because it depends on the cost function evaluation. 
The nature of the CRLQAS reward function depends on the cost function evaluation. 
The cost function evaluation can be deterministic or stochastic based on both the classical optimizer and quantum noise. 
Stochastic optimizers, like SPSA, tend to converge toward different parameters, resulting in different function values across multiple trials, even when initialized identically. 
Moreover, the quantum noise may also lead to different function values even with the same parameters.
We have clarified this aspect in the text by including this description in the revised manuscript.

Reviewer comment (Reproducibility):
One area for improvement in the paper is the absence of the promised source code, which limits the ability of others to replicate and build upon the work. It is essential for the authors to provide the source code to enhance the transparency and applicability of their research. Therefore, it is recommended that the authors include a link to the open-sourced code or an appendix containing the code in any subsequent revision of the paper. This addition would be a valuable resource for the community and would greatly facilitate further research and validation of the proposed methods.

Author response:
We thank the reviewer for bringing this issue to our notice. Currently, we are in the process of refining and polishing the codebase.
An anonymous version of the code will be made available before November 22, 2023. Thank you for your understanding and patience.

Reviewer comment (Writing):
The terminology used in the paper, specifically the term "architecture," may lead to ambiguity within the broader scientific community. In the realm of computer architecture, "architecture" typically refers to the low-level hardware design, as discussed in sources like the one available on IEEE Xplore. Since this work is centered around the optimization of parameterized quantum circuits, and not the hardware itself, clarity could be improved by selecting a term that more accurately describes the subject of optimization. A term like "circuit design" or "circuit configuration" may better capture the essence of the research without the potential for misinterpretation.

Author response:
We understand the reviewers concern about the use of 'architecture' to describe a quantum circuit, as this concept exists with broad meanings depending on the field.
Unfortunately, the term 'quantum architecture search' has been used in the quantum computing community since its introduction in 2020 by (Du et al. npj Quantum Inf 8, 62, 2022) and (Zhang et al. Quantum Science and Technology 7 045023, 2022).
The concept is an extension to that of neural architecture search (NAS) in classical machine learning.
We agree with the reviewer that there exist alternative ways of describing the process of building a quantum circuit, but it would require a community effort to get rid of the 'quantum architecture search' concept.
Furthermore, the 'architecture' is integral to the title of our paper, and per conference guidelines, we are unable to modify it.
We apologize for any confusion this concept might cause, and hope that the reviewer understands our position.

Reviewer comment (Novelty):
Originality: This paper extends the ideas from previous work (Ostaszewski et al., 2021). Some techniques such as feedback driven curriculum learning and Adam-SPSA seem to be the direct adaptation of existing methods.

Author response:
We agree with the reviewer that the paper treats (Ostaszewski et al., 2021) as a baseline to improve upon. On the other hand, we respectfully disagree with the reviewer in regard to the comment that this work is a mere adaptation of existing methods. 
In particular, we numerically demonstrate that the newly developed tensor-based binary encoding, the illegal actions, and random halting all improve the previous work *quantitatively* by orders of magnitude in metrics such as the first successful episode number, the total number of successful episodes, circuit depth, etc. Similarly, a version of Adam-SPSA we developed in this paper helps the RL agent tackle finite-sampling noise in a query way.  Moreover, we improved the existing works *qualitatively* by making such training, and their simulation, possible under a realistic noise model.
Without the curation, creation and adaptation of such elements \& methods training a CRLQAS agent would simply be not possible under realistic noise.
Our goal was to demonstrate that further improvements can be taken to combine reinforcement learning methods with near-term quantum hardware. And, such studies are contingent on the techniques developed in this paper.

Reviewer comment (Experiments):
Related to the previous question, in the noisy simulation section, the paper conducts no comparison with other RL-based QAS methods. Therefore it is unclear what is the improvement of this algorithm over previous methods in noisy environments. Moreover, for molecules such as LiH-4, the experiments only consider shot and 1-qubit depolarizing noise, which is far from real noise models. This undermines the persuasiveness of this work.

Author response:
We respectfully disagree with the reviewer as we provided results in that matter, but we agree that we should have highlighted them more. 
We would like to point to the reviewer that in section 4.2 we studied two more noise models for LiH-4, even though they did not achieve chemical accuracy. 
Namely, a model where 1-qubit depolarizing noise of strength $0.001$ is combined with 2-qubit depolarizing noise of strength $0.005$, and the median noise profile of IBM Mumbai device as described in the paper. 
We conducted further experiments since the submission of the manuscript. 
We will elaborate more about these results in the revised version in the coming days. 
For now, please refer to *Table 1 of Response 3 to reviewer wt33*.

Reviewer comment (Novelty):
Important related work not discussed. Despite little consideration of noise in previous reinforcement learning methods for QAS, there have been many attempts in traditional architecture field. The paper may need more comparison with other QAS algorithms under hardware erors, especially those targeting noise-aware search, for example Wang et al. QuantumNAS 2022.

Author response:
We kindly disagree with the reviewer. 
It may be the case it was missed due to the text structure, but we in fact did consider these other works (or similar), and reported on it in the paper. 
We have referred to QAS algorithms employing search strategies such as sampling-based, evolutionary algorithms (EA) and RL in sec. 2 of our manuscript.
Furthermore, our study includes a comparison with QCAS (Du et al., 2022), a one-shot super-circuit method, under both hardware noise and qubit-connectivity constraints for the $4$-qubit $H_2$ molecule.

Our understanding of the paper suggests that QuantumNAS (Wang et al., 2022) falls within the family of one-shot super-circuit methods.
This technique involves sharing gate parameters among sub-circuits after one shot training to accelerate circuit evaluation. 
However, optimizing the super-circuit poses a challenge, and furthermore, there is a weak correlation in performances between sub-circuits with inherited parameters and those with optimal parameters from individual training.

We investigated QuantumNAS manuscript in an attempt to facilitate either a fully-fledged quantitative or qualitative comparison.
Unfortunately, the work lacks essential information for such an analysis. Specifically, details about the molecular geometries and quantum circuits are absent, preventing us from conducting a quantitative experiment for comparison. 
Consequently, this would require us to run their algorithm from scratch for the molecules considered in this study.
Hence, due to the fact that we already compared to an algorithm (QCAS) which is akin to QuantumNAS and the reasons mentioned above, we believe that our comparison to existing methods (like QCAS, qubit-ADAPT VQE, quantumDARTS) is fair, and covers most of the existing approaches for quantum chemistry problems.

Nevertheless, a qualitative comparison is only feasible in the case of 2-qubit $H_2$ molecule. 
Examining fig. 17 in ref. [2], we note that the best result achieved in those experiments exhibits an energy error of $0.1$ Hartree. In contrast, our $H_2$ simulations, conducted over three independent seeds (for median of IBM Mumbai device), yield a mean noisy energy error of $1.116 \times 10^{-3} \pm 3.41 \times 10^{-4}$—a performance three orders of magnitude superior to their outcomes. Furthermore, we outperform their results for IBM Mumbai at max and 10max noise levels, with mean noisy energy errors of $0.907 \pm 0.008$ and $0.0912 \pm 0.004$, respectively. 
Regrettably, there is no available information regarding the number of gates, depth, or parameters achieved by the method presented in ref. [2].

Reviewer comment (Writing):
Some descriptions are unclear to me. In Section 3.3, what does "$n_s$ is the number of successes" mean? Details in Section 3.4 could also be better stated. Moreover, from a reader's point of view, the organization of Section 3 is scattered and lacks emphasis on major contributions.

Author response:
We agree with the reviewer regarding the inconsistencies within the text. We conducted a thorough proofreading and introduced several revisions in the updated manuscript. 
For instance, as highlighted by the reviewer, we restructured the text. Instead of the 'number of successes' of the negative binomial distribution we refer to 'number of failed Bernoulli trials' of this distribution in the text. 
Additionally, we've refined the method descriptions in sec. 3, focusing solely on the presentation of new methods. The detailed explanation of the modified Feedback-driven curriculum, initially included in sec. 3, has been relocated to Appendix B.1.

Reviewer comment (Experiments):
Does the model need retraining for different noise models to deliver competitive results? I am curious about the efficiency for adaptation to new noise models. Also I believe experiments on real quantum machines comparing the proposed method with other algorithms would make the work more solid.

Author response:
The reviewer raises a fascinating point that we are also deeply interested in; i.e., the matters of transfer \& active learning in the context of different noise models are indeed relevant questions both for practical concerns (e.g., leveraging training time by pre-training one and using it on the other), and for the fundamental understanding of the RL methods. 
But, this is not the core of our question in this work.
Nonetheless, to lay the foundations for such a study, we trained different CRLQAS agents in different noisy environments for different Hamiltonians of the same system size. Also, we conducted two naive experiments where the neural network weights \& the training memory buffer of a CRLQAS agent were transferred to different problems without any alterations. In the first one, we transferred agents trained in noiseless environments for the chemistry problem of different bond dimensions of the same molecule. Here we observed an agent trained within the scope of a higher bond-dimension problem (more difficult) to perform well in an easier problem, but not the other way around. In the second one, we transferred agents trained in the noiseless environment of the same molecule to a noisy environment. Here we also found that the agent was able to react to the new environment by synthesizing circuits that achieve chemical accuracy. On the other hand, we found that after a few thousand episodes the agents were not able to perform their task as adequately anymore due to catastrophic forgetting. We deemed a more systematic, elaborate future study is needed for this task. We believe such a study is *contingent* on the results \& techniques developed in our study.

Reviewer comment (Experiments):
Although the authors claim to tackle the general employment of VQAs, numerical experiments on applications are actually limited to VQE for quantum chemistry tasks. The authors should avoid overstatement, or apply the method to general VQA settings and tasks.

Author response:
We agree with the reviewer that within the scope of this paper, we are only dealing with a particular variant of VQAs, namely the VQEs in the quantum chemistry context. We will make the corrections in the revised manuscript. On the other hand, we kindly disagree with the statement that our method is limited to VQEs. No component of our method imposes such a restriction or is limited to quantum chemistry tasks.

Reviewer comment (Evaluation):
There is almost no discussion of limitations. What is the weakness of this method and what are the potential future directions?

Author response:
Thank you for pointing this out. We have added a new section in the Appendix A of the revised manuscript on the topics of computational demands, evolution of RL methods, validation on real hardware, scalability challenges and transfer learning exploration.