Original: F REE ENERGY - BASED REINFORCEMENT LEARNING USING A QUANTUM PROCESSOR
Revised: R EINFORCEMENT LEARNING VIA REPLICA STACKING OF QUANTUM MEASUREMENTS FOR THE TRAINING OF QUANTUM B OLTZMANN MACHINES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks.
Revised: Recent theoretical and experimental results suggest the possibility of using current and near-future quantum hardware in challenging sampling tasks.

Original: In this paper, we introduce free energy-based reinforcement learning (FERL) as an application of quantum hardware.
Revised: In this paper, we introduce free-energy-based reinforcement learning (FERL) as an application of quantum hardware.

Original: We propose a method for processing a quantum annealer’s measured qubit spin conﬁgurations in approximating the free energy of a quantum
Revised: We propose a method for processing a quantum annealer’s measured qubit spin conﬁgurations in approximating the free energy of a quantum

Original: Boltzmann machine (QBM).
Revised: Boltzmann machine (QBM).

Original: We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer.
Revised: We then apply this method to perform reinforcement learning on the grid-world problem using the D-Wave 2000Q quantum annealer.

Original: The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks
Revised: The experimental results show that our technique is a promising method for harnessing the power of quantum sampling in reinforcement learning tasks

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Reinforcement learning  Sutton & Barto (1998); Bertsekas & Tsitsiklis (1996) has been successfully applied in ﬁelds such as engineering Derhami et al.
Revised: Reinforcement learning (RL) Sutton & Barto (1998); Bertsekas & Tsitsiklis (1996) has been successfully applied in ﬁelds such as engineering Derhami et al.

Original: ; Syaﬁie et al.
Revised: ; Syaﬁie et al.

Original: (2007), sociology Erev & Roth (1998); Shteingart & Loewenstein (2014), and economics Matsui et al.
Revised: (2007), sociology Erev & Roth (1998); Shteingart & Loewenstein (2014), and economics Matsui et al.

Original: Sui et al.
Revised: Sui et al.

Original: The training samples in reinforcement learning are provided by the interaction of an agent with an ambient environment.
Revised: The training samples in reinforcement learning are provided by the interaction of an agent with an ambient environment.

Original: For example, in a motion planning problem in uncharted territory, it is desirable for the agent to learn  in the fastest way possible to correctly navigate, making the fewest blind decisions.
Revised: For example, in a motion planning problem in uncharted territory, it is desirable for the agent to learn to correctly navigate in the fastest way possible, making the fewest blind decisions.

Original: That is, neither exploration nor exploitation can be pursued exclusively without either facing a penalty or failing at the task.
Revised: That is, neither exploration nor exploitation can be pursued exclusively without either facing a penalty or failing at the task.

Original: Our goal is, therefore, not only to design an algorithm that eventually converges to an optimal policy, but for the algorithm to be able to generate suboptimal policies early in the learning process.
Revised: Our goal is, therefore, not only to design an algorithm that eventually converges to an optimal policy, but for the algorithm to be able to generate suboptimal policies early in the learning process.

Original: Free energy-based reinforcement learning (FERL) using a restricted Boltzmann machine (RBM), as suggested by Sallans & Hinton (2004), relies on approximating a utility function for the agent, called the Q -function, using the free energy of an RBM.
Revised: Free-energy-based reinforcement learning (FERL) using a restricted Boltzmann machine (RBM), as suggested by Sallans & Hinton (2004), relies on approximating a utility function for the agent, called the Q-function , using the free energy of an RBM.

Original: RBMs have the advantage that their free energy can be efﬁciently calculated using closed formulae.
Revised: RBMs have the advantage that their free energy can be efﬁciently calculated using closed formulae.

Original: RBMs can represent any joint distribution over binary variables Martens et al.
Revised: RBMs can represent any joint distribution over binary variables Martens et al.

Original: ; Hornik et al.
Revised: ; Hornik et al.

Original: Le Roux &
Revised: Le Roux &

Original: Bengio (2008); however, this property of universality may require exponentially large RBMs
Revised: Bengio (2008); however, this property of universality may require exponentially large RBMs

Original: Martens et al.
Revised: Martens et al.

Original: (2013); Le Roux & Bengio (2008).
Revised: (2013); Le Roux & Bengio (2008).

Original:  Crawford et al. 
Revised: General Boltzmann machines (GBM) are proposed in an effort to devise universal Q -function approximators with polynomially large Boltzmann networks Crawford et al. Traditionally, Monte Carlo simulation is used to perform the computationally expensive tasks of approximating the free energy of GBMs under a Boltzmann distribution. One way to speed up the approximation process is to represent a GBM by an equivalent physical system and try to ﬁnd its Boltzmann distribution.

Revised: An example of such a physical system is a quantum annealer consisting of a network of pair-wise interacting quantum bits (qubits).

Revised: Although quantum annealers have already been used in many areas of computational science, including combinatorial optimization and machine learning, their application in RL has not been explored.

Original: (2016) generalize this method by proposing the useof a quantum or quantum-inspired algorithm for efﬁciently approximating the free energy of a general Boltzmann machine (GBM) (in general, using GBMs involves the NP-hard problem of sampling from a  Boltzmann distribution).
Revised: In order to use quantum annealing for RL, we ﬁrst represent the Q -function as the free energy of a physical system, that is, that of a quantum annealer. We then slowly evolve the state of the physical system from a well-known initial state toward a state with a Boltzmann-like probability distribution. Repeating the annealing process sufﬁciently long can provide us with samples from the Boltzmann distribution so that we can empirically approximate the free energy of the physical system under this distribution. Finally, approximating the free energy of the system would give us an estimate of the

Original: Using numerical simulations, Crawford et al.
Revised: Q -function.

Original: show that FERL using a deep Boltzmann machine (DBM) can provide a drastic improvement in the early stages of learning.
Revised: Up until the past few years, studies were limited to the classical Boltzmann machines.

Original: A quantum annealer consisting of a network of quantum bits (qubits) can provide samples that approximate a Boltzmann distribution of a system of pairwise interacting qubits called the transverse-ﬁeld Ising model (TFIM).
Revised: Recently,

Original: The free energy of the Hamiltonian of a TFIM contains not only information about the qubits in the measurement basis, but also about their spin in a transverse direction.
Revised: Crawford et al. (2018) generalized the classical method toward a quantum or quantum-inspired algorithm for approximating the free energy of GBMs.

Original: Using numerical simulation,
Revised: Using simulated quantum annealing (SQA)

Original: Crawford et al.
Revised: Crawford et al.

Original: show that this richer, many-body system can provide  the same (in fact, slightly better) learning efﬁciency.
Revised: (2018) showed that FERL using a deep Boltzmann machine (DBM) can provide a drastic improvement in the early stages of learning, yet performing the same procedure on an actual quantum device remained a difﬁcult task. This is because sampling from a quantum system representing a quantum Boltzmann machine is harder than the classical case, since at the end of each anneal the quantum system is in a superposition. Any attempt to measure the ﬁnal state of the quantum system is doomed to fail since the superposition would collapse into a classical state that does not carry the entirety of information about the ﬁnal state. In this work, we have two main contributions.

Revised: We ﬁrst employ a quantum annealer as a physical device to approximate the free energy of a classical Boltzmann machine.

Revised: Second, we generalize the notion of classical Boltzmann machines to quantum Boltzmann machines within the ﬁeld of RL and utilize a quantum annealer to approximate the free energy of a quantum system.

Original: In this paper, we report the results of  using the D-Wave 2000Q quantum processor to experimentally verify the conjecture of Crawford et al. (2016): the applicability of a quantum annealer in reinforcement learning.
Revised: In order to deal with the issue of superposition mentioned above, we propose a novel stacking procedure in that we attempt to reconstruct the full state of superposition from the partial information that we get from sampling after each anneal. Finally we report proof-of-concept results using the D-Wave 2000Q quantum processor to provide experimental evidence for the applicability of a quantum annealer in reinforcement learning as predicted by Crawford et al.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We refer the reader to Sutton & Barto (1998) and Yuksel (2016) for an exposition on Markov decision processes (MDP), controlled Markov chains, and the various broad aspects of reinforcement learning.
Revised: We refer the reader to Sutton & Barto (1998) and Yuksel (2016) for an exposition on Markov decision processes (MDP), controlled Markov chains, and the various broad aspects of reinforcement learning.

Original: A Q-function is deﬁned by mapping a tuple p π, s, a q of a given stationary policy π , a current state s , and an immediate action a of a controlled Markov chain to the expected value of the instantaneous and future discounted rewards of the Markov chain that begins with taking action a at initial state s and continuing according to π :
Revised: A Q-function is deﬁned by mapping a tuple p π, s, a q of a given stationary policy π , a current state s , and an immediate action a of a controlled Markov chain to the expected value of the instantaneous and future discounted rewards of the Markov chain that begins with taking action a at initial state s and continuing according to π :

Original: [Equation]
Revised: [Equation]

Original: Here, r p s, a q is a random variable, perceived by the agent from the environment, representing the immediate reward of taking action a from state s , and Π is the Markov chain resulting from restricting the controlled Markov chain to the policy π .
Revised: Here, r p s, a q is a random variable, perceived by the agent from the environment, representing the immediate reward of taking action a from state s , and Π is the Markov chain resulting from restricting the controlled Markov chain to the policy π .

Original: The ﬁxed real number γ P p 0 , 1 q is the discount factor of the MDP.
Revised: The ﬁxed real number γ P p 0 , 1 q is the discount factor of the MDP.

Original: From Q ˚ p s, a q “ max π Q p π, s, a q , the optimal policy for the MDP can be retrieved via
Revised: From Q ˚ p s, a q “ max π Q p π, s, a q , the optimal policy for the MDP can be retrieved via

Original: [Equation]
Revised: [Equation]

Original: This reduces the MDP task to  computing Q ˚ p s, a q .
Revised: This reduces the MDP task to that of computing Q ˚ p s, a q .

Original: Through the Bellman optimality equation Bellman (1956), we get
Revised: Through the Bellman optimality equation Bellman (1956), we get

Original: [Equation]
Revised: [Equation]

Original: so Q ˚ is the ﬁxed point of the following operator deﬁned on L 8 p S ˆ A q :
Revised: so Q ˚ is the ﬁxed point of the following operator deﬁned on L 8 p S ˆ A q :

Original: [Equation]
Revised: [Equation]

Original: In this paper, we focus on the TD(0) Q -learning method, with the Q -function parametrized by neural networks in order to ﬁnd π ˚ p s q and Q ˚ p s, a q , which is based on minimizing the distance between
Revised: In this paper, we focus on the TD(0) Q -learning method, with the Q -function parametrized by neural networks in order to ﬁnd π ˚ p s q and Q ˚ p s, a q , which is based on minimizing the distance between

Original: T p Q q and Q .
Revised: T p Q q and Q .

Original: 2. C LAMPED B OLTZMANN MACHINES
Revised: 2. C LAMPED B OLTZMANN MACHINES

Original: A clamped Boltzmann machine is a GBM in which all visible nodes v are prescribed ﬁxed assignments and removed from the underlying graph.
Revised: A clamped Boltzmann machine is a GBM in which all visible nodes v are prescribed ﬁxed assignments and removed from the underlying graph.

Original: Therefore, the energy of the clamped Boltzmann machine may be written as
Revised: Therefore, the energy of the clamped Boltzmann machine may be written as

Original: [Equation]
Revised: [Equation]

Original: where V and H are the sets of visible and hidden nodes, respectively, and by a slight abuse of notation, a letter v stands both for a graph node v P V and for the assignment v P t 0 , 1 u .
Revised: where V and H are the sets of visible and hidden nodes, respectively, and by a slight abuse of notation, the letter v stands both for a graph node v P V and for the assignment v P t 0 , 1 u .

Original: The interactions between the variables represented by their respective nodes are speciﬁed by real-valued weighted edges of the underlying undirected graph represented by w vh , and w hh 1 denotes the weights between visible and hidden, or hidden and hidden, nodes of the Boltzmann machine, respectively.
Revised: The interactions between the variables represented by their respective nodes are speciﬁed by real-valued weighted edges of the underlying undirected graph represented by w vh , and w hh 1 denotes the weights between visible and hidden, or hidden and hidden, nodes of the Boltzmann machine, respectively.

Original: A clamped quantum Boltzmann machine (QBM) has the same underlying graph as a clamped GBM, but instead of a binary random variable, qubits are associated to each node of the network.
Revised: A clamped quantum Boltzmann machine (QBM) has the same underlying graph as a clamped GBM, but instead of a binary random variable, qubits are associated to each node of the network.

Original: The energy function is substituted by the quantum Hamiltonian of an induced TFIM, which is mathematically a Hermitian matrix
Revised: The energy function is substituted by the quantum Hamiltonian of an induced transverse ﬁeld Ising model (TFIM), which is mathematically a Hermitian matrix

Original: [Equation]
Revised: [Equation]

Original: where σ zh represent the Pauli z -matrices and σ xh represent the Pauli x -matrices.
Revised: where σ zh represent the Pauli z -matrices and σ xh represent the Pauli x -matrices.

Original: Thus, a clamped
Revised: Thus, a clamped

Original: QBM with Γ “ 0 is equivalent to a clamped classical Boltzmann machine.
Revised: QBM with Γ “ 0 is equivalent to a clamped classical Boltzmann machine.

Original: This is because, in this case, H v is a diagonal matrix in the σ z -basis, the spectrum of which is identical to the range of the classical Hamiltonian (3).
Revised: This is because, in this case, H v is a diagonal matrix in the σ z -basis, the spectrum of which is identical to the range of the classical Hamiltonian (3).

Original: We note that (4) is a particular instance of a TFIM:
Revised: We note that (4) is a particular instance of a TFIM.

Revised: 2. F REE - ENERGY - BASED REINFORCEMENT LEARNING

Revised: Let us begin with the classical Boltzmann machine case.

Revised: Following Sallans & Hinton (2004), for an assignment of visible variables v , F p v q denotes the equilibrium free energy , and is given via

Original: [Equation]
Revised: [Equation]

Original: The remainder of this section is formulated for the clamped QBMs, acknowledging that it can easily be specialized for clamped classical Boltzmann machines. 2. F REE ENERGY - BASED REINFORCEMENT LEARNING
Revised: where β “ 1 k B T is a ﬁxed thermodynamic beta. In Sallans & Hinton (2004), it was proposed to use the negative free energy of a GBM to approximate the Q -function through the relationship

Original: Let β “ 1 k B T be a ﬁxed thermodynamic beta.
Revised: [Equation] for each admissible state–action pair p s, a q P S ˆ A . Here, s and a are binary vectors encoding the state s and action a on the state nodes and action nodes, respectively, of a GBM. In RL, the visible nodes of a GBM are partitioned into two subsets of state nodes S and action nodes A .

Original: For an assignment of visible variables v , F p v q denotes the equilibrium free energy , and is given via
Revised: Here, w represents the vector of weights of a GBM as in (3). Each entry w of w can now be trained using the TD(0) update rule: [Equation]

Revised: and

Revised: [Equation]

Revised: where x h y and x hh 1 y are the expected values of the variables and the products of the variables, respectively, in the binary encoding of the hidden nodes with respect to the Boltzmann distribution of the classical Hamiltonian (3).

Revised: To develop a FERL method using QBMs, let β “ 1 k B T be a ﬁxed thermodynamic beta as in the classical case.

Revised: As before, for an assignment of visible variables v , F p v q denotes the equilibrium free energy, and is given via

Original: [Equation]
Revised: [Equation]

Original: Here, Z v “ tr p e ´ β
Revised: Here, Z v “ tr p e ´ β

Original: H v q is the partition function of the clamped QBM and ρ v is the density matrix ρ v “ 1 Z v e ´ β H v .
Revised: H v q is the partition function of the clamped QBM and ρ v is the density matrix ρ v “ 1 Z v e ´ β H v .

Original: The term ´ tr p ρ v ln ρ v q is the entropy of the system.
Revised: The term ´ tr p ρ v ln ρ v q is the entropy of the system.

Revised: Note that (8) is a generalization of (5).

Original: The notation x¨
Revised: The notation x¨

Original: ¨ ¨
Revised: ¨ ¨

Original: y is used for the expected value of any observable with respect to the Gibbs measure (i.e., the Boltzmann distribution), in particular, 
Revised: y is used for the expected value of any observable with respect to the Gibbs measure (i.e., the Boltzmann distribution), in particular, [Equation] This is also a generalization of the weighted sum ř

Original: [Equation]
Revised: [Equation]

Revised: P p

Revised: [Equation]

Revised: |

Revised: [Equation]

Original:  Inspired by the ideas of Sallans & Hinton (2004) and Amin et al.
Revised: q [Equation] v p h q in (5). Inspired by the ideas of Sallans & Hinton (2004) and Amin et al.

Original: (2016), we use the negative free energy of a QBM to approximate the Q -function through the relationship [Equation] [Equation]
Revised: (2016), we use the negative free energy of a QBM to approximate the Q -function exactly as in the classical case:

Original: [Equation]
Revised: [Equation]

Original: for each admissible state–action pair p s, a q P S ˆ A .
Revised: for each admissible state–action pair p s, a q P S ˆ A .

Original: Here, s and a are binary vectors encoding the state s and action a on the state nodes and action nodes, respectively, of a QBM.
Revised: As before, s and a are binary vectors encoding the state s and action a on the state nodes and action nodes, respectively, of a Boltzmann machine.

Original: In reinforcement learning, the visible nodes of a GBM are partitioned into two subsets of state nodes S and action nodes A .
Revised: In RL, the visible nodes of a Boltzmann machine are partitioned into two subsets of state nodes S and action nodes A .

Original: Here, w represents the vector of weights of a QBM as in (4).
Revised: Here, w represents the vector of weights of a QBM as in (4).

Original: Each entry w of w can now be trained using the TD(0) update rule:
Revised: Each entry w of w can now be trained using the TD(0) update rule:

Original: [Equation]
Revised: [Equation]

Original: As shown in Crawford et al.
Revised: As shown in Crawford et al.

Original: (2016), from (6) we obtain
Revised: (2018), from (8) we obtain

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: This concludes the development of the FERL method using QBMs.
Revised: This concludes the development of the FERL method using QBMs.

Original: We refer the reader to Algorithmin Crawford et al.
Revised: We refer the reader to Algorithmin Crawford et al.

Original: (2016) for more details.
Revised: (2018) for more details.

Original: What remains to be done is to approximate values of the free energy F p s, a q and also the expected values of the observables x σ zh y and x σ zh σ zh 1 y .
Revised: What remains to be done is to approximate values of the free energy F p s, a q and also the expected values of the observables x σ zh y and x σ zh σ zh 1 y .

Original: In this paper, we demonstrate how quantum annealing can be used to address this challenge.
Revised: In this paper, we demonstrate how quantum annealing can be used to address this challenge.

Original: 2. A DIABATIC EVOLUTION OF OPEN QUANTUM SYSTEMS
Revised: 2. A DIABATIC EVOLUTION OF OPEN QUANTUM SYSTEMS

Original: The evolution of a quantum system under a slowly changing, time-dependent Hamiltonian is characterized by Born & Fock (1928).
Revised: The evolution of a quantum system under a slowly changing, time-dependent Hamiltonian is characterized by Born & Fock (1928).

Original: The quantum adiabatic theorem (QAT) in Born & Fock (1928) states that the system remains in its instantaneous steady state, provided there is a gap between the eigen-energy of the steady state and the rest of the Hamiltonian’s spectrum at every point in time.
Revised: The quantum adiabatic theorem (QAT) in Born & Fock (1928) states that a system remains in its instantaneous steady state, provided there is a gap between the eigen-energy of the steady state and the rest of the Hamiltonian’s spectrum at every point in time.

Original: QAT motivated Farhi et al.
Revised: QAT motivated Farhi et al.

Original: (2000) to introduce a paradigm of quantum computing known as quantum adiabatic computation which is closely related to the quantum analogue of simulated annealing, namely quantum annealing (QA), introduced by Kadowaki & Nishimori (1998).
Revised: (2000) to introduce a paradigm of quantum computing known as quantum adiabatic computation which is closely related to the quantum analogue of simulated annealing, namely quantum annealing (QA), introduced by Kadowaki & Nishimori (1998).

Original: The history of QA and QAT inspired manufacturing efforts towards physical realizations of adiabatic evolution via quantum hardware Johnson et al.
Revised: The history of QA and QAT inspired manufacturing efforts towards physical realizations of adiabatic evolution via quantum hardware Johnson et al.

Original: In reality, the manufactured chips are operated at a non-zero temperature and are not isolated from their environment.
Revised: In reality, the manufactured chips are operated at a non-zero temperature and are not isolated from their environment.

Original: Therefore, the existing adiabatic theory does not cover the behaviour of these machines.
Revised: Therefore, the existing adiabatic theory does not cover the behaviour of these machines.

Original: A contemporary investigation in quantum adiabatic theory was therefore initiated to study adiabaticity in open quantum systems
Revised: A contemporary investigation in quantum adiabatic theory was therefore initiated to study adiabaticity in open quantum systems

Original: Sarandy & Lidar (2005); Venuti et al.
Revised: Sarandy & Lidar (2005); Venuti et al.

Original: ; Albash et al.
Revised: ; Albash et al.

Original: ; Avron et al.
Revised: ; Avron et al.

Original: ; Bachmann et al.
Revised: ; Bachmann et al.

Original: These sources prove adiabatic theorems for open quantum systems under various assumptions, in particular when the quantum system is coupled to a thermal bath satisfying the
Revised: These sources prove adiabatic theorems for open quantum systems under various assumptions, in particular when the quantum system is coupled to a thermal bath satisfying the

Original: Kubo–Martin–Schwinger condition, implying that the instantaneous steady state is the instantaneous
Revised: Kubo–Martin–Schwinger condition, implying that the instantaneous steady state is the instantaneous

Original: Gibbs state.
Revised: Gibbs state.

Original: This work in progress shows promising opportunities to use quantum annealers to sample from the Gibbs state of a TFIM.
Revised: This work in progress shows promising opportunities to use quantum annealers to sample from the Gibbs state of a TFIM.

Original: In practice, due to additional complications (e.g., level crossings and gap closure, described in the references above), the samples gathered from the quantum annealer are far from the Gibbs state of the ﬁnal Hamiltonian.
Revised: In practice, due to additional complications (e.g., level crossings and gap closure, described in the references above), the samples gathered from the quantum annealer are far from the Gibbs state of the ﬁnal Hamiltonian.

Original: In fact, Amin (2015) suggests that the distribution of the samples would instead correspond to an instantaneous Hamiltonian at an intermediate point in time, called the freeze-out point .
Revised: In fact, Amin (2015) suggests that the distribution of the samples would instead correspond to an instantaneous Hamiltonian at an intermediate point in time, called the freeze-out point .

Original: Unfortunately, this point and, consequently, the strength Γ of the transverse ﬁeld at this point, is not known a priori, and also depends on the TFIM undergoing evolution.
Revised: Unfortunately, this point and, consequently, the strength Γ of the transverse ﬁeld at this point, is not known a priori, and also depends on the TFIM undergoing evolution.

Original: Our goal is simply to associate a single (average) virual Γ to all TFIMs constructed through FERL.
Revised: Our goal is simply to associate a single (average) virual Γ to all TFIMs constructed through FERL.

Original: Another unknown parameter is the inverse temperature β , at which the Gibbs state, the partition function, and the free energy are attained.
Revised: Another unknown parameter is the inverse temperature β , at which the Gibbs state, the partition function, and the free energy are attained.

Original: In a similar fashion, we wish to associate a single virtual β to all TFIMs encountered.
Revised: In a similar fashion, we wish to associate a single virtual β to all TFIMs encountered.

Original: The quantum annealer used in our experiments is the D-Wave 2000Q, which consists of a chip of superconducting qubits connected to each other according to a sparse adjacency graph called the
Revised: The quantum annealer used in our experiments is the D-Wave 2000Q, which consists of a chip of superconducting qubits connected to each other according to a sparse adjacency graph called the

Original: Chimera graph .
Revised: Chimera graph .

Original: The Chimera graph structure looks signiﬁcantly different from the frequently used models in machine learning, e.g., RBMs and DBMs, which consist of consecutive fully connected bipartite graphs.
Revised: The Chimera graph structure looks signiﬁcantly different from the frequently used models in machine learning, for example, RBMs and DBMs, which consist of consecutive fully connected bipartite graphs.

Original: Fig 1 shows two adjacent blocks of the Chimera graph which consist of 16 qubits, which, in this paper, serve as the clamped QBM used in FERL.
Revised: Fig. 1 shows two adjacent blocks of the Chimera graph which consist of 16 qubits, which, in this paper, serve as the clamped QBM used in FERL.

Original: Another complication when using a quantum annealer as a QBM is that the spin conﬁgurations of the qubits can only be measured along a ﬁxed axis (here the z -basis of the Bloch sphere).
Revised: Another complication when using a quantum annealer as a QBM is that the spin conﬁgurations of the qubits can only be measured along a ﬁxed axis (here the z -basis of the Bloch sphere).

Original: Once σ z is measured, all of the quantum information related to the projection of the spin along the transverse ﬁeld (i.e., the spin σ x ) collapses and cannot be retrieved.
Revised: Once σ z is measured, all of the quantum information related to the projection of the spin along the transverse ﬁeld (i.e., the spin σ x ) collapses and cannot be retrieved.

Original: Therefore, even with a choice of virtual Γ , virtual β , and all of the measured conﬁgurations, the energy of (5) is still unknown.
Revised: Therefore, even with a choice of virtual Γ , virtual β , and all of the measured conﬁgurations, the energy of the TFIM is still unknown.

Original: We propose a method for overcoming this challenge based on the Suzuki–Trotter expansion of the TFIM, which we call replica stacking , the details of which are explained in § 3.4.
Revised: We propose a method for overcoming this challenge based on the Suzuki–Trotter expansion of the TFIM, which we call replica stacking , the details of which are explained in § 3.4.

Original: In § 4, we perform a grid search over values of the virtual parameters β and Γ .
Revised: In § 4, we perform a grid search over values of the virtual parameters β and Γ .

Original: The accepted virtual parameters are the ones that result in the most-effective learning for FERL in the early stages of training.
Revised: The accepted virtual parameters are the ones that result in the most-effective learning for FERL in the early stages of training.

Original:  F REE ENERGY OF QUANTUM B OLTZMANN MACHINES 3. S UZUKI –T ROTTER REPRESENTATION
Revised:  F REE ENERGY OF QUANTUM B OLTZMANN MACHINES 3. S UZUKI –T ROTTER REPRESENTATION

Original: By the Suzuki–Trotter decomposition Suzuki (1976), the partition function of the TFIM deﬁned by the Hamiltonian (4) can be approximated using the partition function of a classical Hamiltonian denoted by H eﬀ v and called an effective Hamiltonian, which corresponds to a classical Ising model of one dimension higher.
Revised: By the Suzuki–Trotter decomposition Suzuki (1976), the partition function of the TFIM deﬁned by the Hamiltonian (4) can be approximated using the partition function of a classical Hamiltonian denoted by H eﬀ v and called an effective Hamiltonian, which corresponds to a classical Ising model of one dimension higher.

Original: More precisely,
Revised: More precisely,

Original: [Equation]
Revised: [Equation]

Original: where r is the number of replicas, w ` “ 12 β log coth ´
Revised: where r is the number of replicas, w ` “ 12 β log coth ´

Original: Γ βr ¯ , and h k represent spins of the classical system of one dimension higher.
Revised: Γ βr ¯ , and h k represent spins of the classical system of one dimension higher.

Original: Note that each hidden node’s Pauli z -matrices σ zh are represented by r classical spins, denoted by h k , with a slight abuse of notation.
Revised: Note that each hidden node’s Pauli z -matrices σ zh are represented by r classical spins, denoted by h k , with a slight abuse of notation.

Original: In other words, the original Ising model with a non-zero transverse ﬁeld represented through non-commuting operators can be mapped to a classical Ising model of one dimension higher.
Revised: In other words, the original Ising model with a non-zero transverse ﬁeld represented through non-commuting operators can be mapped to a classical Ising model of one dimension higher.

Original: Fig.
Revised: Fig.

Original: 2 shows the underlying graph of a TFIM on a two-dimensional lattice and a corresponding ten-replica effective Hamiltonian in three dimensions.
Revised: 2 shows the underlying graph of a TFIM on a two-dimensional lattice and a corresponding 10-replica effective Hamiltonian in three dimensions.

Revised: The intuition behind the Suzuki–Trotter decomposition is that the superposition of the spins in a quantum system is represented classically by replicas in the z -basis.

Revised: In other words, the measurement of the quantum system in the z -basis is interpreted as choosing one replica at random.

Revised: Note that the probabilities of measuring ` 1 or ´ 1 for each individual spin are preserved.

Revised: This way, each hidden node in the quantum Boltzmann machine carries more information than a classical one; in fact, a classical representation of this system requires r classical binary units via the Suzuki–Trotter decomposition.

Revised: Consequently, the connections between the hidden nodes become more complicated in the quantum case as well and can carry more information on the correlations between the hidden nodes.

Original:  3. A PPROXIMATION OF FREE ENERGY USING G IBBS SAMPLING
Revised: Note that the coupling strengths between the replicas are not arbitrary, but come from the mathematical decomposition following the Suzuki–Trotter formula. As a result, the quantum Boltzmann machine can be viewed as an undirected graphical model but in one dimension higher than the classical Boltzmann machine. 3. A PPROXIMATION OF FREE ENERGY USING G IBBS SAMPLING

Revised: In the case of classical GBMs without further restrictions on the graph structure, x h y , x hh 1 y , and

Revised: Q p s, a q « ´ F p s , a ; w q are not tractable.

Revised: Consequently, to perform the weight update in (6) one requires samples from the Boltzmann distribution corresponding to energy function (3) to estimate x h y , x hh 1 y , and F p s , a ; w q empirically.

Original: To approximate the right-hand side of (7) and (8), we sample from the Boltzmann distribution of the  effective Hamiltonian using (Suzuki, 1976, Theorem 6).
Revised: To approximate the right-hand side of (9) and (10), we sample from the Boltzmann distribution of the energy function represented by the effective Hamiltonian using (Suzuki, 1976, Theorem 6).

Original: We ﬁnd the expected values of the observables x σ zh y and x σ zh σ zh 1 y by averaging the corresponding classical spin values.
Revised: We ﬁnd the expected values of the observables x σ zh y and x σ zh σ zh 1 y by averaging the corresponding classical spin values.

Original: To approximate the  Q -function, we use (Suzuki, 1976, Theorem 4) to substitute (6) by
Revised: To approximate the free energy of a QBM and consequently a Q -function, we use (Suzuki, 1976, Theorem 4) to substitute (8) by

Original: [Equation]
Revised: [Equation]

Original: where H eﬀ v is the effective Hamiltonian and c eﬀ ranges over all spin conﬁgurations of the effective classical Ising model of one dimension higher, deﬁned by H eﬀ v .
Revised: where H eﬀ v is the effective Hamiltonian and c eﬀ ranges over all spin conﬁgurations of the effective classical Ising model of one dimension higher, deﬁned by H eﬀ v .

Original: The above argument also holds in the absence of the transverse ﬁeld, that is, for the classical

Original: Boltzmann machine.

Original: In this case, the TD(0) update rule is given by

Original: [Equation] and [Equation] where x h y and x hh 1 y are the expected values of the variables and the products of the  variables, respectively,in the binary encoding of the hidden nodes with respect to theBoltzmann distribution of the classical Hamiltonian (3).
Revised: Here, x h y and x hh 1 y are the expected values of the variables and the products of the binary variables, respectively, with respect to the

Original: The values of the Q -functions in (11) and (12) can also be approximated empirically, since, in a classical Boltzmann machine,

Original: F p v q “ ÿ h

Original: P p h | v q E v p h q ` 1 β ÿ h

Original: P p h | v q log P p h | v q (13) “ ´ ÿ s P S h P H w sh s x h y ´ ÿ a P A h P H w ah a x h y ´
Revised: Boltzmann distribution of the classical effective Hamiltonian (11).

Original: ÿ t h,h 1 uĎ H u hh 1 x hh 1 y `β ÿ h

Original: P p h | s , a q log P p h | s , a q .

Original: 3. S IMULATED QUANTUM ANNEALING
Revised: 3. S IMULATED QUANTUM ANNEALING

Original: One way to sample spin values from the Boltzmann distribution of the effective Hamiltonian is to use the simulated quantum annealing algorithm (SQA) (see (Anthony Brabazon, 2015, p. 422) for an introduction).
Revised: One way to sample spin values from the Boltzmann distribution of the effective Hamiltonian is to use the simulated quantum annealing algorithm (SQA) (see (Brabazon et al., 2015, p. 422) for an introduction).

Original: SQA is one of the many ﬂavours of quantum Monte Carlo methods, and is based on the Suzuki–Trotter expansion described above.
Revised: SQA is one of the many ﬂavours of quantum Monte Carlo methods, and is based on the Suzuki–Trotter expansion described above.

Original: This algorithm simulates the quantum annealing phenomena of a TFIM by slowly reducing the strength of the transverse ﬁeld at ﬁnite temperature to the desired target value.
Revised: This algorithm simulates the quantum annealing phenomena of a TFIM by slowly reducing the strength of the transverse ﬁeld at ﬁnite temperature to the desired target value.

Original: In our implementation, we have used a single spin-ﬂip variant of SQA with a linear transverse-ﬁeld schedule as in Martoˇn´ak et al.
Revised: In our implementation, we have used a single spin-ﬂip variant of SQA with a linear transverse-ﬁeld schedule as in Martoˇn´ak et al.

Original: and Heim et al.
Revised: and Heim et al.

Original: Experimental studies have shown similarities in the behaviour of SQA and that of quantum annealing Isakov et al.
Revised: Experimental studies have shown similarities in the behaviour of SQA and that of quantum annealing Isakov et al.

Original: ; Albash et al.
Revised: ; Albash et al.

Original: (2014) and its physical realization  Brady & van Dam (2016);
Revised: (2014) and its physical realization by D-Wave Systems Brady & van Dam (2016);

Original: Shin et al.
Revised: Shin et al.

Original: (2014) by D-Wave Systems.

Original: The classical counterpart of SQA is conventional simulated annealing (SA), which is based on thermal annealing.
Revised: The classical counterpart of SQA is conventional simulated annealing (SA), which is based on thermal annealing.

Original: This algorithm can be used to sample from Boltzmann distributions that correspond to an
Revised: This algorithm can be used to sample from Boltzmann distributions that correspond to an

Original: Ising spin model in the absence of a transverse ﬁeld (i.e., Γ “ 0 in (5)).
Revised: Ising spin model in the absence of a transverse ﬁeld.

Original: Unlike SA, it is possible to use SQA not only to approximate the Boltzmann distribution of a classical Boltzmann machine, but also that of a quantum Hamiltonian in the presence of a transverse ﬁeld.
Revised: Unlike SA, it is possible to use SQA not only to approximate the Boltzmann distribution of a classical Boltzmann machine, but also that of a quantum Hamiltonian in the presence of a transverse ﬁeld.

Original: This can be done by reducing the strength of the transverse ﬁeld to the desired value deﬁned by the model, rather than to zero.
Revised: This can be done by reducing the strength of the transverse ﬁeld to the desired value deﬁned by the model, rather than to zero.

Original: It has been proven by Morita & Nishimori (2006)
Revised: It has been proven by Morita & Nishimori (2006)

Original: that the spin system deﬁned by SQA converges to the Boltzmann distribution of the effective classical Hamiltonian of one dimension higher that corresponds to the quantum Hamiltonian
Revised: that the spin system deﬁned by SQA converges to the Boltzmann distribution of the effective classical Hamiltonian of one dimension higher that corresponds to the quantum Hamiltonian

Original: Therefore, it is straightforward to use SQA to approximate the free energy in (10) as well as the observables x σ zh y and x σ zh σ zh 1 y .
Revised: Therefore, it is straightforward to use SQA to approximate the free energy in (12) as well as the observables x σ zh y and x σ zh σ zh 1 y .

Revised: However, any Boltzmann distribution sampling method based on Markov chain Monte Carlo (MCMC) has the major drawback of being extremely slow and computationally involved.

Revised: Actually, it is an NP-hard problem to sample from the Boltzmann distribution.

Revised: Another option is to use variational approximation Salakhutdinov & Hinton (2009), which suffers from lack of accuracy and works in practice only in limited cases.

Revised: As explained above, quantum annealers have the potential to provide samples from Boltzmann distributions (in the z -basis) corresponding to TFIM in a more efﬁcient way.

Revised: In what follows, we explain how to use quantum annealing to approximate the free energy corresponding to an effective Hamiltonian which in turn can be used to approximate the free energy of a QBM.

Original: 3. R EPLICA STACKING
Revised: 3. R EPLICA STACKING

Original: As explained in § 2.3, a quantum annealer provides measurements of σ z spins for each qubit in the
Revised: As explained in § 2.3, a quantum annealer provides measurements of σ z spins for each qubit in the

Original: TFIM.
Revised: TFIM.

Original: The observables x σ zh y and x σ zh σ zh 1 y can therefore be approximated by averaging over the spin conﬁgurations measured by the quantum annealer.
Revised: The observables x σ z h y and x σ z h σ z h 1 y can therefore be approximated by averaging over the spin conﬁgurations measured by the quantum annealer.

Original: Moreover, by (Suzuki, 1976, Theorem 6) and translation invariance, each replica of the effective classical model is an approximation of the spin measurements of the TFIM in the measurement bases σ z .
Revised: Moreover, by (Suzuki, 1976, Theorem 6) and translation invariance, each replica of the effective classical model is an approximation of the spin measurements of the TFIM in the measurement bases σ z .

Original: Therefore, a σ z -conﬁguration sampled by a quantum annealer that operates at a given virtual inverse temperature β  and anneals up to a virtual transverse-ﬁeld strength Γ , may be viewed as an instance of a classical spin conﬁguration from a replica of the classical effective Hamiltonian of one dimension higher.
Revised: Therefore, a σ z -conﬁguration sampled by a quantum annealer that operates at a given virtual inverse temperature β , and anneals up to a virtual transverse-ﬁeld strength Γ , may be viewed as an instance of a classical spin conﬁguration from a replica of the classical effective Hamiltonian of one dimension higher.

Original: This suggests the following method to approximate the free energy from (10) for a TFIM.
Revised: This suggests the following method to approximate the free energy from (12) for a TFIM.

Original: We gather a pool C of conﬁgurations sampled by the quantum annealer for the TFIM considered, allowing repetitions.
Revised: We gather a pool C of conﬁgurations sampled by the quantum annealer for the TFIM considered, allowing repetitions.

Original: Let r be the number of replicas.
Revised: Let r be the number of replicas.

Original: We write c eﬀ “ p c 1 , . . .
Revised: We write c eﬀ “ p c 1 , . . .

Original: , c r q to indicate an effective conﬁguration c eﬀ with the classical conﬁgurations c 1 to c r as its replicas.
Revised: , c r q to indicate an effective conﬁguration c eﬀ with the classical conﬁgurations c 1 to c r as its replicas.

Original: We write c eﬀ to denote the underlying set t c 1 , . . .
Revised: We write c eﬀ to denote the underlying set t c 1 , . . .

Original: , c r u of replicas of c eﬀ (forgetting their ordering).
Revised: , c r u of replicas of c eﬀ (without considering their ordering).

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: The argument in the previous paragraph can now be employed to allow the assumption
Revised: The argument in the previous paragraph can now be employed to allow the assumption

Original: [Equation]
Revised: [Equation]

Original: R

Original: W P ö W Ò Ð Ð Ò Ð ÐÒ ÐÒ Ð Ð Ò Ò ÐÒ ÐÒ In other words, the probability mass function of the effective conﬁgurations is supported in the subset of those conﬁgurations synthesized from the elements of C as candidate replicas .
Revised: In other words, the probability mass function of the effective conﬁgurations is supported in the subset of those conﬁgurations synthesized from the elements of C as candidate replicas .

Original: The conditional probability P r c eﬀ “ t c 1 , . .
Revised: The conditional probability P r c eﬀ “ t c 1 , . .

Original: , c r u| c eﬀ Ď C s can be sampled from by drawing r elements c 1 , . . .
Revised: , c r u| c eﬀ Ď C s can be sampled from by drawing r elements c 1 , . . .

Original: , c r from C .
Revised: , c r from C .

Original: We then sample from P“
Revised: We then sample from P

Original: [Equation]
Revised: [Equation]

Original: ‰ , according to the following distribution over c eﬀ :
Revised: according to the following distribution over c eﬀ :

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation] [Figure] [Equation] We consider π p c eﬀ q our target distribution and construct the following Markov Chain Monte Carlo (MCMC) method for which the limiting distribution is π p c eﬀ q .
Revised: We consider π p c eﬀ q our target distribution and construct the following MCMC method for which the limiting distribution is π p c eﬀ q .

Original: We ﬁrst attach the r classical spin conﬁgurations to the SQA’s effective conﬁguration structure uniformly at random.
Revised: We ﬁrst attach the r classical spin conﬁgurations to the SQA’s effective conﬁguration structure uniformly at random.

Original: We then transition to a different arrangement with  Metropolis acceptance probability.
Revised: We then transition to a different arrangement with a Metropolis acceptance probability.

Original: For example, we may choose two classical conﬁgurations at random and exchange them with probability
Revised: For example, we may choose two classical conﬁgurations at random and exchange them with probability

Original: [Equation]
Revised: [Equation]

Original: where E p c eﬀ q “ w ` ř 
Revised: where E p c eﬀ q “ w ` ř h

Original: [Equation]
Revised: [Equation]

Original: ¯ .
Revised: ¯ .

Original: It is well-known that such a stochastic process satisﬁes the detailed balance condition.
Revised: Such a stochastic process is known to satisfy the detailed balance condition.

Original: Consequently, the MCMC  allows us to sample from the effective spin conﬁgurations.
Revised: Consequently, the MCMC method allows us to sample from the effective spin conﬁgurations.

Original: This procedure of sampling and then performing  MCMC  creates a pool of effective spin conﬁgurations, which are then employed in equation (10) in order to approximate the free energy of the TFIM empirically.
Revised: This procedure of sampling and then performing the MCMC method creates a pool of effective spin conﬁgurations, which are then employed in equation (12) in order to approximate the free energy of the TFIM empirically.

Original: However, we consider a relatively small number of hidden nodes in our experiments, so the number of different σ z -conﬁgurations sampled by the quantum annealer is limited.
Revised: However, we consider a relatively small number of hidden nodes in our experiments, so the number of different σ z -conﬁgurations sampled by the quantum annealer is limited.

Original: As a consequence, there is no practical need to perform the MCMC  deﬁned above.
Revised: As a consequence, there is no practical need to perform the MCMC method deﬁned above.

Original: Instead, we attach classical spin conﬁgurations from the pool to the SQA effective conﬁguration structure at random.
Revised: Instead, we attach classical spin conﬁgurations from the pool to the SQA effective conﬁguration structure at random.

Original: In other words, in r iterations, a spin conﬁguration is sampled from the pool of classical spin conﬁgurations described above and inserted as the next replica of the effective classical Hamiltonian consisting of r replicas. 
Revised: In other words, in r iterations, a spin conﬁguration is sampled from the pool of classical spin conﬁgurations described above and inserted as the next replica of the effective classical Hamiltonian consisting of r replicas. It is worthwhile to reiterate that this replica stacking technique yields an undirected graphical model.

Original: T HE EXPERIMENTS
Revised: Speciﬁcally, the structure described in Fig. 2 (right) is an undirected graphical model in the space of hidden nodes, where the node statistics are obtained from the Boltzmann distribution. One difference between this model and a classical Boltzmann machine is that each hidden node activation is governed by a series of r replicas in one dimension higher, and the undirected, replica-to-replica connections calculated therein. Moreover, the energy function of this extended model differs from the energy function of the classical Boltzmann machine (compare (11) and (3)).

Original:  We benchmark our various FERL methods on a 3 ˆ 5 grid-world problem Sutton (1990) with an agent capable of taking the actions up, down, left, or right, or standing still, on a grid-world withone deterministic reward, one wall, and one penalty, as shown in Fig.
Revised: The free energy of the extended graphical model serves as the function approximator to the Q -function.  T HE EXPERIMENTS We benchmark our various FERL methods on a 3 ˆ 5 grid-world problem Sutton (1990) with an agent capable of taking the actions up, down, left, or right, or standing still, on a grid-world with

Revised: Algorithm initialize weights of QBM for all training samples p s 1 , a 1 q do s 2 Ð a 1 p s 1 q , a 2 Ð argmax a Q p s 2 , a q calculate x σ zh i y , x σ zh i σ zh 1 i y ,

Revised: x H eﬀ s i , a i y , and P p c eff | s i , a i q using Algorithm 2, for ( i “ 1 , 2 ) calculate F p s i , a i q using (12) for ( i “ 1 , 2 )

Revised: Q p s i , a i q Ð ´ F p s i , a i q for p i “ 1 , 2 q update QBM weights using (9) and (10) π p s 1 q Ð argmax a Q p s 1 , a q end for return π

Revised: Algorithm 2 Replica stacking initialize the structure of the effective Hamiltonian in one dimension higher for i “ 1 , 2 , ..., m do for j “ 1 , 2 , ..., r do obtain spin conﬁguration sample in z -basis from QA attach this spin conﬁguration to j -th replica of the i -th effective conﬁguration structure end for perform the MCMC technique described in § 3.4 with transition probabilities (13) to obtain the i -th instance of effective spin conﬁgurations end for obtain x H eﬀ s i , a i y from the average energy of the m effective spin conﬁgurations obtain x h y and x hh 1 y by averaging over all h and h 1 replicas in each spin conﬁguration gather statistics from P p c eﬀ | s i , a i q using the m effective spin conﬁgurations return x

Revised: h y , x hh 1 y , x H eﬀ s i , a i y , and P p c eﬀ | s i , a i q one deterministic reward, one wall, and one penalty, as shown in Fig.

Original: 3 (left).
Revised: 3 (top).

Original: The task is to ﬁnd an optimal policy, as shown in Fig.
Revised: The task is to ﬁnd an optimal policy, as shown in Fig.

Original: 3 (right), for the agent at each state in the grid-world.
Revised: 3 (bottom), for the agent at each state in the grid-world.

Original: All of the
Revised: All of the

Original: Boltzmann machines used in our algorithms consist of 16 hidden nodes.
Revised: Boltzmann machines used in our algorithms consist of 16 hidden nodes.

Original: The discount factor, as explained in § 2, is set to 0 .
Revised: The discount factor, as explained in § 2, is set to 0 .

Original: The agent attains the reward R “ 200 in the top-left corner, the neutral value of moving to any empty cell is 100 , and the agent is penalized by not receiving any reward if it moves to the penalty cell with value P “ 0 .
Revised: The agent attains the reward R “ 200 in the top-left corner, the neutral value of moving to any empty cell is 100 , and the agent is penalized by not receiving any reward if it moves to the penalty cell with value P “ 0 .

Original: For T r independent runs of every FERL method, T s training samples are used.
Revised: For T r independent runs of every FERL method, T s training samples are used.

Original: The ﬁdelity measure at the i -th training sample is deﬁned by
Revised: The ﬁdelity measure at the i -th training sample is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where π ˚ denotes the best known policy and A p s, i, l q denotes the action assigned at the l -th run and i -th training sample to the state s .
Revised: where π ˚ denotes the best known policy and A p s, i, l q denotes the action assigned at the l -th run and i -th training sample to the state s .

Original: In our experiments, each algorithm is run 100 times. 
Revised: In our experiments, each algorithm is run 100 times. R W P ö

Revised: W Ò Ð Ð Ò Ð ÐÒ Ð Ò Ð Ð Ò Ò ÐÒ Ð Ò

Original: Fig.
Revised: Fig.

Original: 4 demonstrates the performance of a fully connected deep Q -network Mnih et al.
Revised: 4 demonstrates the performance of a fully connected deep Q -network Mnih et al.

Original: (2015) consisting of an input layer of 14 state nodes, two layers of eight hidden nodes each, and an output layer of ﬁve nodes representing the values of the Q -function for different actions, given a conﬁguration of state nodes.
Revised: (2015) consisting of an input layer of 14 state nodes, two layers of eight hidden nodes each, and an output layer of ﬁve nodes representing the values of the Q -function for different actions, given a conﬁguration of state nodes.

Original: We use the same number of hidden nodes in the fully connected deep Q -network as in the other networks described in this paper.
Revised: We use the same number of hidden nodes in the fully connected deep Q -network as in the other networks described in this paper.

Original: 4. G RID SEARCH FOR VIRTUAL PARAMETERS ON THE D-W AVE 2000Q
Revised: 4. G RID SEARCH FOR VIRTUAL PARAMETERS ON THE D-W AVE 2000Q

Original: We treat the network of superconducting qubits represented in Fig.
Revised: We treat the network of superconducting qubits represented in Fig.

Original: 1 as a clamped QBM with two hidden layers, represented using blue and red colours.
Revised: 1 as a clamped QBM with two hidden layers, represented using blue and red colours.

Original: The state nodes are considered fully connected to the blue qubits and the action nodes are fully connected to the red qubits.
Revised: The state nodes are considered fully connected to the blue qubits and the action nodes are fully connected to the red qubits.

Original: For a choice of virtual parameters Γ ‰ 0 and β , which appear in (9) and (10), and for each query to the D-Wave 2000Q chip, we construct 150 effective classical conﬁgurations of one dimension higher, out of a pool of 3750 reads, according to the replica stacking method introduced in § 3.4.
Revised: For a choice of virtual parameters Γ ‰ 0 and β , which appear in (11) and (12), and for each query to the D-Wave 2000Q chip, we construct 150 effective classical conﬁgurations of one dimension higher, out of a pool of 3750 reads, according to the replica stacking method introduced in § 3.4.

Original: Theconﬁgurations are, in turn, employed to approximate the free energy of the quantum Hamiltonian.
Revised: Theconﬁgurations are, in turn, employed to approximate the free energy of the quantum Hamiltonian.

Original: We conduct 10 independent runs of FERL in this fashion and ﬁnd the average ﬁdelity over theruns and over the T s “ 300 training samples.
Revised: We conduct 10 independent runs of FERL in this fashion, and ﬁnd the average ﬁdelity over theruns and over the T s “ 300 training samples.

Original: Fig.
Revised: Fig.

Original: 5 shows a heatmap of the average ﬁdelity of each choice of virtual parameters β and Γ .
Revised: 5 shows a heatmap of the average ﬁdelity of each choice of virtual parameters β and Γ .

Original: In the Γ “ 0 row, each query to the D-Wave 2000Q is considered to be sampling from a classical GBM with Fig.
Revised: In the Γ “ 0 row, each query to the D-Wave 2000Q is considered to be sampling from a classical GBM with Fig.

Original: 1 as the underlying graph.
Revised: 1 as the underlying graph.

Original: 4. FERL FOR THE GRID - WORLD PROBLEM
Revised: 4. FERL FOR THE GRID - WORLD PROBLEM

Original: Fig.
Revised: Fig.

Original: 6 shows the growth of the average ﬁdelity of the best known policies generated by different
Revised: 6 shows the growth of the average ﬁdelity of the best known policies generated by different

Original: FERL methods.
Revised: FERL methods.

Original: For each method, the ﬁdelity curve is an average over 100 independent runs, each with T s “ 500 training samples.
Revised: For each method, the ﬁdelity curve is an average over 100 independent runs, each with T s “ 500 training samples.

Original: In this ﬁgure, the “D-Wave Γ “ 0 . 5 , β “ 2 . 0 ” curve corresponds to the D-Wave 2000Q replica stacking-based method with the choice of the best virtual parameters Γ “ 0 . 5 and β “ 2 . 0 , as shown in the heatmap in Fig.
Revised: In this ﬁgure, the “D-Wave Γ “ 0 . 5 , β “ 2 . 0 ” curve corresponds to the D-Wave 2000Q replica stacking-based method with the choice of the best virtual parameters Γ “ 0 . 5 and β “ 2 . 0 , as shown in the heatmap in Fig.

Original: The training is based on formulae (7), (8), and (10).
Revised: The training is based on formulae (9), (10), and (12).

Original: The “SQA Bipartite Γ “ 0 . 5 , β “ 2 . 0 ” and “SQA Chimera Γ “ 0 . 5 , β “ 2 . 

Revised: The “SQA Bipartite Γ “ 0 . 5 , β “ 2 . 0 ” and “SQA Chimera Γ “ 0 . 5 , β “ 2 .

Original: curves are based on the same formulae with the underlying graphs being a bipartite (DBM) and a Chimera graph, respectively, with the same choice of virtual parameters, but the effective Hamiltonian conﬁgurations generated using SQA as explained in § 3.3.
Revised: curves are based on the same formulae with the underlying graphs being a bipartite (DBM) and a Chimera graph, respectively, with the same choice of virtual parameters, but the effective Hamiltonian conﬁgurations generated using SQA as explained in § 3.3.

Original: [Figure] [Figure]
Revised: The “SA Bipartite β “ 2 . 0 ” and “SA Chimera β “ 2 . 0 ” curves are generated by using SA to train a classical DBM and a classical GBM on the Chimera graph, respectively, using formulae (6), (7), and (5). SA is run with a linear inverse temperature schedule, where β “ 2 . 0 indicates the ﬁnal value. The “D-Wave Classical β “ 2 . 0 ” curve is generated using the same method, but with samples

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The “SA Bipartite β “ 2 . 0 ” and “SA Chimera β “ 2 . 0 ” curves are generated by using SA to train a classical DBM and a classical GBM on the Chimera graph, respectively, using formulae (11), (12), and (13). SA is run with a linear inverse temperature schedule, where β “ 2 . 0 indicates the ﬁnal value. The “D-Wave Classical β “ 2 . 0 ” curve is generated using the same method, but with samples obtained using the D-Wave 2000Q.
Revised: obtained using the D-Wave 2000Q.

Original: The “RBM” curve is generated using the method in Sallans &
Revised: The “RBM” curve is generated using the method in Sallans &

Original: Hinton (2004).
Revised: Hinton (2004).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We solve the grid-world problem using various Q -learning methods with the Q -function parametrized by different neural networks.
Revised: We solve the grid-world problem using various Q -learning methods with the Q -function parametrized by different neural networks.

Original: For comparison, we demonstrate the performance of a fully connected deep Q -network method that can be considered state of the art.
Revised: For comparison, we demonstrate the performance of a fully connected deep Q -network method that can be considered state of the art.

Original: This method efﬁciently processes every training sample, but, as shown in Fig.
Revised: This method efﬁciently processes every training sample, but, as shown in Fig.

Original: 4, requires  very large number of training samples to converge to the optimal policy.
Revised: 4, requires a very large number of training samples to converge to the optimal policy.

Original: Another conventional method is free energy-based reinforcement learning using an RBM.
Revised: Another conventional method is free-energy-based RL using an RBM.

Original: This method is also very successful at  the scale of the reinforcement learning task considered in our experiment.
Revised: This method is also very successful at learning the optimal policy at the scale of the RL task considered in our experiment.

Original: Although this method is not outperforming other FERL methods that take advantage of a highly efﬁcient sampling oracle, the processing of each training sample is efﬁcient, as it is based on closed formulae.
Revised: Although this method does not outperform other FERL methods that take advantage of a highly efﬁcient sampling oracle, the processing of each training sample is efﬁcient, as it is based on closed formulae.

Original: In fact, for the size of problem considered, the RBM-based
Revised: In fact, for the size of problem considered, the RBM-based

Original: FERL outperforms the fully connected deep Q -network method.
Revised: FERL outperforms the fully connected deep Q -network method.

Original: The comparison of results in Fig.
Revised: The comparison of results in Fig.

Original: 6 suggests that replica stacking is a successful method for estimating effective classical conﬁgurations obtained from a quantum annealer, given that the spins can only be measured in measurement bases.
Revised: 6 suggests that replica stacking is a successful method for estimating effective classical conﬁgurations obtained from a quantum annealer, given that the spins can only be measured in measurement bases.

Original: For practical use in reinforcement learning, this method provides a means of treating the quantum annealer as a QBM.
Revised: For practical use in RL, this method provides a means of treating the quantum annealer as a QBM.

Original: FERL using the quantum annealer, in conjunction with the replica stacking technique, provides signiﬁcant improvement over FERL using classical Boltzmann machines.
Revised: FERL using the quantum annealer, in conjunction with the replica stacking technique, provides signiﬁcant improvement over FERL using classical Boltzmann machines.

Original: The curve representing SQA-based FERL using a Boltzmann machine on the Chimera graph is almost coincident with the one obtained using the D-Wave 2000Q, whereas the 
Revised: The curve representing SQA-based FERL using a Boltzmann machine on the Chimera graph is almost coincident with the one obtained using the D-Wave 2000Q, whereas the SQA-based FERL using a

Original: SQA-based FERL using a DBM slightly outperforms it.
Revised: DBM slightly outperforms it.

Original: This suggests that quantum annealing chips with greater connectivity and more control over annealing time can further improve the performance of the replica stacking method applied to reinforcement learning tasks.
Revised: This suggests that quantum annealing chips with greater connectivity and more control over annealing time can further improve the performance of the replica stacking method applied to RL tasks.

Original: This is further supported by comparing the performance of SA-based FERL using a DBM versus SA-based FERL using the Chimera graph.
Revised: This is further supported by comparing the performance of SA-based FERL using a DBM versus SA-based FERL using the Chimera graph.

Original: This result shows that DBM is a better choice of neural network compared to the Chimera graph, due to its additional connections.
Revised: This result shows that DBM is, due to its additional connections, a better choice of neural network compared to the Chimera graph.

Original: For practical reasons, we aim to associate an identical choice of virtual parameters β and Γ to all of the TFIMs constructed using FERL.
Revised: For practical reasons, we aim to associate an identical choice of virtual parameters β and Γ to all of the TFIMs constructed using FERL.

Original: Benedetti et al.
Revised: Benedetti et al.

Original: (2016) and Raymond et al.
Revised: (2016) and Raymond et al.

Original: (2016) provide methods for estimating the effective inverse temperature β for other applications.
Revised: (2016) provide methods for estimating the effective inverse temperature β for other applications.

Original: However, in both studies, the samples obtained from the quantum annealer are matched to the Boltzmann distribution of a classical Ising model.
Revised: However, in both studies, the samples obtained from the quantum annealer are matched to the Boltzmann distribution of a classical Ising model.

Original: In fact, the transverse-ﬁeld strength is a second virtual parameter that we consider.
Revised: In fact, the transverse-ﬁeld strength is a second virtual parameter that we consider.

Original: The optimal choice Γ “ 0 .
Revised: The optimal choice Γ “ 0 .

Original: 5 corresponds to 2 { 3 of the annealing time, in agreement with the work of Amin (2015), who also consider TFIM with 16 qubits.
Revised: 5 corresponds to 2 { 3 of the annealing time, in agreement with the work of Amin (2015), who also considers TFIM with 16 qubits.

Original: The agreement of FERL using quantum annealer reads treated as classical Boltzmann samples with that of FERL using SA and classical Boltzmann machines suggests that, at least for this task and this size of Boltzmann machine, the measurements provided by the D-Wave 2000Q can be considered good approximations of Boltzmann distribution samples of classical Ising models. 
Revised: The agreement of FERL using quantum annealer reads treated as classical Boltzmann samples with that of FERL using SA and classical Boltzmann machines suggests that, at least for this task and this size of Boltzmann machine, the measurements provided by the D-Wave 2000Q can be considered good approximations of Boltzmann distribution samples of classical Ising models. The extended undirected graphical model developed in this paper using the replica stacking method is not limited to Q -function approximation in RL tasks. Potentially, this method can be applied to tasks where Boltzmann machines can be used. This method provides a mechanism for approximating the activations and partition functions of quantum Boltzmann machines that have a signiﬁcant transverse ﬁeld.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper we describe a free energy-based reinforcement learning algorithm using  existing quantum hardware, namely the D-Wave 2000Q.
Revised: In this paper, we describe a free-energy-based reinforcement learning algorithm using an existing quantum annealer, namely the D-Wave 2000Q.

Original: Our method relies on the Suzuki–Trotter decomposition and the use of the measured conﬁgurations by the D-Wave 2000Q as replicas of an effective classical 
Revised: Our method relies on the Suzuki–Trotter decomposition and the use of the measured conﬁgurations by the D-Wave 2000Q as replicas of an effective classical Ising model of one dimension higher.

Original: Ising model of one dimension higher.
Revised: The results presented here are ﬁrst-step proofs of concept of a proposed quantum algorithm with a promising path towards outperforming reinforcement learning algorithms devised for digital hardware.

Original: Future research can employ these principles to solve larger-scale reinforcement learning tasks in the emerging ﬁeld of quantum machine learning
Revised: Given appropriate advances in quantum annealing hardware, future research can employ the proposed principles to solve larger-scale reinforcement learning tasks in the emerging ﬁeld of quantum machine learning. A CKNOWLEDGEMENTS

Revised: The authors would like to thank Marko Bucyk for editing this manuscript.