Original: C LASSICAL AND Q UANTUM A LGORITHMS FOR O RTHOGONAL N EURAL N ETWORKS
Revised: C LASSICAL AND Q UANTUM A LGORITHMS FOR O RTHOGONAL N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Orthogonal neural networks have recently been introduced as a new type of neural networks imposing orthogonality on the weight matrices.
Revised: Orthogonal neural networks have recently been introduced as a new type of neural networks imposing orthogonality on the weight matrices.

Original: They could achieve higher accuracy and avoid evanescent or explosive gradients for deep architectures.
Revised: They could achieve higher accuracy and avoid evanescent or explosive gradients for deep architectures.

Original: Several classical gradient descent methods have been proposed to preserve orthogonality while updating the weight matrices, but these techniques suffer from long running times and/or provide only approximate orthogonality.
Revised: Several classical gradient descent methods have been proposed to preserve orthogonality while updating the weight matrices, but these techniques suffer from long running times and/or provide only approximate orthogonality.

Original: In this paper, we introduce a new type of neural network layer called Pyramidal Circuit, which implements an orthogonal matrix multiplication.
Revised: In this paper, we introduce a new type of neural network layer called Pyramidal Circuit, which implements an orthogonal matrix multiplication.

Original: It allows for gradient descent with perfect orthogonality with the same asymptotic running time as a standard fully connected layer.
Revised: It allows for gradient descent with perfect orthogonality with the same asymptotic running time as a standard fully connected layer.

Original: This algorithm is inspired by quantum computing and can therefore be applied on a classical computer as well as on a near term quantum computer.
Revised: This algorithm is inspired by quantum computing and can therefore be applied on a classical computer as well as on a near term quantum computer.

Original: It could become the building block for quantum neural networks and faster orthogonal neural networks.
Revised: It could become the building block for quantum neural networks and faster orthogonal neural networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the evolution of neural network structures, adding constraints to the weight matrices has often been an effective path.
Revised: In the evolution of neural network structures, adding constraints to the weight matrices has often been an effective path.

Original: Recently, orthogonal neural networks (OrthoNNs) have been proposed Jia et al.
Revised: Recently, orthogonal neural networks (OrthoNNs) have been proposed Jia et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Nosarzewski (2018); Bansal et al.
Revised: Nosarzewski (2018); Bansal et al.

Original: as a new type of neural networks for which, at each layer, the weight matrix should remain orthogonal.
Revised: as a new type of neural networks for which, at each layer, the weight matrix should remain orthogonal.

Original: This property is useful to reach higher accuracy performance and avoid vanishing or exploding gradient for deep architectures.
Revised: This property is useful to reach higher accuracy performance and avoid vanishing or exploding gradient for deep architectures.

Original: Several classical gradient descent methods have been proposed to preserve the orthogonality while updating the weight matrices.
Revised: Several classical gradient descent methods have been proposed to preserve the orthogonality while updating the weight matrices.

Original: However, these techniques suffer from longer running time and sometimes only approximate the orthogonality.
Revised: However, these techniques suffer from longer running time and sometimes only approximate the orthogonality.

Original: In particular, the main method for achieving orthogonality during training is to ﬁrst perform the usual gradient descent to update the weight matrix (which is now not going to be orthogonal) and then perform Singular Value Decomposition to orthogonalize or almost orthogonalize the weight matrix.
Revised: In particular, the main method for achieving orthogonality during training is to ﬁrst perform the usual gradient descent to update the weight matrix (which is now not going to be orthogonal) and then perform Singular Value Decomposition to orthogonalize or almost orthogonalize the weight matrix.

Original: We can see then why achieving orthogonality hinders a fast training, since at every step an SVD computation needs to be performed (See
Revised: We can see then why achieving orthogonality hinders a fast training, since at every step an SVD computation needs to be performed (See

Original: Section A.1).
Revised: Section A.1).

Original: In the emergent ﬁeld of quantum machine learning, several proposals have been made to implement neural networks.
Revised: In the emergent ﬁeld of quantum machine learning, several proposals have been made to implement neural networks.

Original: Some algorithms rely on long term and perfect quantum computers Kerenidis et al.
Revised: Some algorithms rely on long term and perfect quantum computers Kerenidis et al.

Original: (2020); Allcock et al.
Revised: (2020); Allcock et al.

Original: (2018), while others try to harness the existing quantum devices using variational circuits
Revised: (2018), while others try to harness the existing quantum devices using variational circuits

Original: Cong et al.
Revised: Cong et al.

Original: ; Farhi & Neven (2018).
Revised: ; Farhi & Neven (2018).

Original: As in classical neural networks, they use gradient descent methods to update the quantum parameters of the circuits.
Revised: As in classical neural networks, they use gradient descent methods to update the quantum parameters of the circuits.

Original: Such quantum neural networks have been trained for very small sizes, however there is still a need to understand how such architectures will scale and whether they will provide efﬁcient and accurate training.
Revised: Such quantum neural networks have been trained for very small sizes, however there is still a need to understand how such architectures will scale and whether they will provide efﬁcient and accurate training.

Original: In this work, we present a new training method for neural networks that preserves perfect orthogonality while having the same running time as usual gradient descent methods without the orthogonality condition, thus achieving the best of both worlds, most efﬁcient training and perfect orthogonality.
Revised: In this work, we present a new training method for neural networks that preserves perfect orthogonality while having the same running time as usual gradient descent methods without the orthogonality condition, thus achieving the best of both worlds, most efﬁcient training and perfect orthogonality.

Original: The main idea comes from the quantum world, where we know that any quantum circuit corresponds to an operation described by a unitary matrix, which if we only use gates with real amplitudes is an orthogonal matrix.
Revised: The main idea comes from the quantum world, where we know that any quantum circuit corresponds to an operation described by a unitary matrix, which if we only use gates with real amplitudes is an orthogonal matrix.

Original: In particular, we propose a novel special-architecture quantum circuit, for which there is an efﬁcient way to map the elements of the orthogonal weight matrix to the parameters of the gates of the quantum circuit and vice versa.
Revised: In particular, we propose a novel special-architecture quantum circuit, for which there is an efﬁcient way to map the elements of the orthogonal weight matrix to the parameters of the gates of the quantum circuit and vice versa.

Original: In other words, while performing a gradient descent on the elements of the weight matrix individually does not preserve orthogonality, performing a gradient descent on the parameters of the quantum circuit preserves orthogonality (since any quantum circuit with real parameters corresponds to an orthogonal matrix) and is equivalent to updating the weight matrix.
Revised: In other words, while performing a gradient descent on the elements of the weight matrix individually does not preserve orthogonality, performing a gradient descent on the parameters of the quantum circuit preserves orthogonality (since any quantum circuit with real parameters corresponds to an orthogonal matrix) and is equivalent to updating the weight matrix.

Original: We also prove that performing gradient descent on the parameters of the quantum circuit can be done efﬁciently classically (with constant update cost per parameter) thus concluding that there exists a quantum-inspired, but fully classical way of efﬁciently training perfectly orthogonal neural networks.
Revised: We also prove that performing gradient descent on the parameters of the quantum circuit can be done efﬁciently classically (with constant update cost per parameter) thus concluding that there exists a quantum-inspired, but fully classical way of efﬁciently training perfectly orthogonal neural networks.

Original: Moreover, the special-architecture quantum circuit we deﬁned has many properties that make it a good candidate for NISQ implementation: it uses only one type of quantum gates, requires simple connectivity between the qubits, has depth linear in the input and output node sizes, and beneﬁts from powerful error mitigation techniques that make it resilient to noise.
Revised: Moreover, the special-architecture quantum circuit we deﬁned has many properties that make it a good candidate for NISQ implementation: it uses only one type of quantum gates, requires simple connectivity between the qubits, has depth linear in the input and output node sizes, and beneﬁts from powerful error mitigation techniques that make it resilient to noise.

Original: This allows us to also propose an inference method running the quantum circuit on data which might offer a faster running time, given the shallow depth of the quantum circuit.
Revised: This allows us to also propose an inference method running the quantum circuit on data which might offer a faster running time, given the shallow depth of the quantum circuit.

Original: Our main contributions are summarized in Table 1, where we have considered the time to perform a feedforward pass, or one gradient descent step.
Revised: Our main contributions are summarized in Table 1, where we have considered the time to perform a feedforward pass, or one gradient descent step.

Original: A single neural network layer is considered, with input and output of size n . 
Revised: A single neural network layer is considered, with input and output of size n . Algorithm

Original: [Table]
Revised: [Table]

Original: This
Revised: Update

Original: [Table] 2 n/δ 2 = O ( n/δ 2 )
Revised: Quantum Pyramidal Circuit (This work) 2 n/δ 2 = O ( n/δ 2 )

Original: O ( n 2 /δ 2 )
Revised: O ( n 2 /δ 2 )

Original: [Table] 2 n ( n − 1) =
Revised: Classical Pyramidal Circuit (This work) 2 n ( n − 1) =

Original: O ( n 2 ) O ( n 2 )
Revised: O ( n 2 ) O ( n 2 )

Original: Classical Approximated OrthoNN (SVB)Jia et al. n [-
-] =
Revised: Classical Approximated OrthoNN (SVB) n [-
-]2 =

Original: O ( n 2 )
Revised: O ( n 2 )

Original: O ( n 3 )
Revised: O ( n 3 )

Original: Classical Strict OrthoNN (Stiefel Manifold)Jia et al. n [-
-] = O ( n 2 )
Revised: Classical Strict OrthoNN (Stiefel Manifold) n [-
-]2 = O ( n 2 )

Original: O ( n 3 )
Revised: O ( n 3 )

Original: [Table] n 2 = O ( n 2 )
Revised: Standard Neural Network (non orthogonal) n 2 = O ( n 2 )

Original: O ( n 2 )
Revised: O ( n 2 )

Original:  A PARAMETRIZED QUANTUM CIRCUIT FOR ORTHOGONAL NEURAL NETWORKS
Revised:  A PARAMETRIZED QUANTUM CIRCUIT FOR ORTHOGONAL NEURAL NETWORKS

Original: In this section, we deﬁne a special-architecture parametrized quantum circuit that will be useful for performing training and inference on orthogonal neural networks.
Revised: In this section, we deﬁne a special-architecture parametrized quantum circuit that will be useful for performing training and inference on orthogonal neural networks.

Original: As we said, the training will be completely classical in the end, but the intuition of the new method comes from this quantum circuit, while the inference can happen both classically or by applying this quantum circuit.
Revised: As we said, the training will be completely classical in the end, but the intuition of the new method comes from this quantum circuit, while the inference can happen both classically or by applying this quantum circuit.

Original: A basic introduction to quantum computing concepts necessary for this work is given in the Appendix (Section A.3).
Revised: A basic introduction to quantum computing concepts necessary for this work is given in the Appendix (Section A.3).

Original: 2. T HE RBS G ATE
Revised: 2. T HE RBS G ATE

Original: The quantum circuit proposed in this work (see Fig.1), which implements a fully connected neural network layer with an orthogonal weight matrix, uses only one type of quantum gate, the Reconﬁgurable Beam Splitter ( RBS ) gate.
Revised: The quantum circuit proposed in this work (see Fig.1), which implements a fully connected neural network layer with an orthogonal weight matrix, uses only one type of quantum gate, the Reconﬁgurable Beam Splitter ( RBS ) gate.

Original: This two-qubit gate is parametrizable with one angle θ ∈ [0 , 2 π ] .
Revised: This two-qubit gate is parametrizable with one angle θ ∈ [0 , 2 π ] .

Original: Its matrix representation is given as:
Revised: Its matrix representation is given as:

Original: [Equation]
Revised: [Equation]

Original: We can think of this gate as a rotation in the two-dimensional subspace spanned by the basis {| 01 (cid:105) , | 10 (cid:105)} , while it acts as the identity in the remaining subspace {| 00 (cid:105) , | 11 (cid:105)} .
Revised: We can think of this gate as a rotation in the two-dimensional subspace spanned by the basis {| 01 (cid:105) , | 10 (cid:105)} , while it acts as the identity in the remaining subspace {| 00 (cid:105) , | 11 (cid:105)} .

Original: Or equivalently, starting with two qubits, one in the | 0 (cid:105) state and the other one in the state | 1 (cid:105) , the qubits can be swapped or not in superposition.
Revised: Or equivalently, starting with two qubits, one in the | 0 (cid:105) state and the other one in the state | 1 (cid:105) , the qubits can be swapped or not in superposition.

Original: The qubit | 1 (cid:105) stays on its wire with amplitude cos θ or switches with the other qubit with amplitude + sin θ if the new wire is below ( | 10 (cid:105) (cid:55)→ 
Revised: The qubit | 1 (cid:105) stays on its wire with amplitude cos θ or switches with the other qubit with amplitude + sin θ if the new wire is below ( | 10 (cid:105) (cid:55)→ |

Original: [Figure]
Revised: [Figure]

Original: We now propose a quantum circuit that implements an orthogonal layer of a neural network.
Revised: We now propose a quantum circuit that implements an orthogonal layer of a neural network.

Original: The circuit is a pyramidal structure of RBS gates, each with an independent angle, as represented in
Revised: The circuit is a pyramidal structure of RBS gates, each with an independent angle, as represented in

Original: Fig.2a.
Revised: Fig.2a.

Original: In Section 2.3 and 3, more details are provided concerning respectively the input loading, and the equivalence with a neural network’s orthogonal layer.
Revised: In Section 2.3 and 3, more details are provided concerning respectively the input loading, and the equivalence with a neural network’s orthogonal layer.

Original: [Figure]
Revised: [Figure]

Original: To mimic a given classical layer with a quantum circuit, the number of output qubits should be the size of the classical layer’s output.
Revised: To mimic a given classical layer with a quantum circuit, the number of output qubits should be the size of the classical layer’s output.

Original: We refer to the square case when the input and output sizes are equal, and to the rectangular case otherwise (Fig.3a).
Revised: We refer to the square case when the input and output sizes are equal, and to the rectangular case otherwise (Fig.3a).

Original: The important property to note is that the number of parameters of the quantum pyramidal circuit corresponding to a neural network layer of size n × d is (2 n − 1 − d ) ∗ d/ 2 exactly the same as the number of degrees of freedom of an orthogonal matrix of dimension n × d .
Revised: The important property to note is that the number of parameters of the quantum pyramidal circuit corresponding to a neural network layer of size n × d is (2 n − 1 − d ) ∗ d/ 2 exactly the same as the number of degrees of freedom of an orthogonal matrix of dimension n × d .

Original: [Figure]
Revised: [Figure]

Original: For simplicity, we pursue our analysis using only the square case but everything can be easily extended to the rectangular case.
Revised: For simplicity, we pursue our analysis using only the square case but everything can be easily extended to the rectangular case.

Original: As we said, the full pyramidal structure of the quantum circuit described above imposes the number of free parameters to be N = n ( n − 1) / 2 , the exact number of free parameters to specify a n × n orthogonal matrix.
Revised: As we said, the full pyramidal structure of the quantum circuit described above imposes the number of free parameters to be N = n ( n − 1) / 2 , the exact number of free parameters to specify a n × n orthogonal matrix.

Original: In Section 3 we will show how the parameters of the gates of this pyramidal circuit can be easily related to the elements of the orthogonal matrix of size n × n that describes it.
Revised: In Section 3 we will show how the parameters of the gates of this pyramidal circuit can be easily related to the elements of the orthogonal matrix of size n × n that describes it.

Original: We note that alternative architectures can be imagined as long as the number of gate parameters is equal to the parameters of the orthogonal weight matrix and a simple mapping between them and the elements of the weight matrix can be found.
Revised: We note that alternative architectures can be imagined as long as the number of gate parameters is equal to the parameters of the orthogonal weight matrix and a simple mapping between them and the elements of the weight matrix can be found.

Original: Note ﬁnally that this circuit has linear depth and is convenient for near term quantum hardware platforms with restricted connectivity.
Revised: Note ﬁnally that this circuit has linear depth and is convenient for near term quantum hardware platforms with restricted connectivity.

Original: Indeed, the distribution of the RBS gates requires only nearest neighbor connectivity between qubits.
Revised: Indeed, the distribution of the RBS gates requires only nearest neighbor connectivity between qubits.

Original: 2. L OADING THE D ATA
Revised: 2. L OADING THE D ATA

Original: Before applying the quantum pyramidal circuit, we will need to upload the classical data into the quantum circuit.
Revised: Before applying the quantum pyramidal circuit, we will need to upload the classical data into the quantum circuit.

Original: We will use one qubit per feature of the data.
Revised: We will use one qubit per feature of the data.

Original: For this, we use a unary amplitude encoding of the input data.
Revised: For this, we use a unary amplitude encoding of the input data.

Original: Let’s consider an input sample x = ( x 0 , · · · , x n − 1 ) ∈ R n , such that (cid:107) x (cid:107) 2 = 1 .
Revised: Let’s consider an input sample x = ( x 0 , · · · , x n − 1 ) ∈ R n , such that (cid:107) x (cid:107) 2 = 1 .

Original: We will encode it in a superposition of unary states:
Revised: We will encode it in a superposition of unary states:

Original: [Equation]
Revised: [Equation]

Original: We can also rewrite the previous state as | x (cid:105) =
Revised: We can also rewrite the previous state as | x (cid:105) =

Original: (cid:80) n − 1 i =

Revised: (cid:80) n − 1 i =

Original: x i | e i (cid:105) , where | e i (cid:105) represents the i th unary state with a | 1 (cid:105) in the i th position | 0 · · · 010 · · · 0 (cid:105) .
Revised: x i | e i (cid:105) , where | e i (cid:105) represents the i th unary state with a | 1 (cid:105) in the i th position | 0 · · · 010 · · · 0 (cid:105) .

Original: Recent work Kerenidis (U.S. Patent Application
Revised: Recent work Kerenidis (U.S. Patent Application

Original: No. 16/986,553 and 16/987,235, 2020) proposed a logarithmic depth data loader circuit for loading such states.
Revised: No. 16/986,553 and 16/987,235, 2020) proposed a logarithmic depth data loader circuit for loading such states.

Original: Here we will use a much simpler circuit.
Revised: Here we will use a much simpler circuit.

Original: It is a linear depth cascade of n -1 RBS gates which, due to the particular structure of our quantum pyramidal circuit, only adds 2 extra steps to our circuit.
Revised: It is a linear depth cascade of n − 1 RBS gates which, due to the particular structure of our quantum pyramidal circuit, only adds 2 extra steps to our circuit.

Original: The circuit starts in the all | 0 (cid:105) state and ﬂips the ﬁrst qubit using an X gate, in order to obtain the unary state | 10 · · · 0 (cid:105) as shown in Fig.4.
Revised: The circuit starts in the all | 0 (cid:105) state and ﬂips the ﬁrst qubit using an X gate, in order to obtain the unary state | 10 · · · 0 (cid:105) as shown in Fig.4.

Original: Then a cascade of RBS gates allow to create the state | x (cid:105) using a set of n − 1 angles α 0 , · · · , α n − 2 .
Revised: Then a cascade of RBS gates allow to create the state | x (cid:105) using a set of n − 1 angles α 0 , · · · , α n − 2 .

Original: Using Eq.(1), we will choose the angles such that, after the ﬁrst RBS gate of the loader, the qubits would be in the state x 0 | 100 · · ·(cid:105) + sin( α 0 ) | 010 · · ·(cid:105) and after the second one in the state x 0 | 100 · · ·(cid:105) + x 1 | 010 · · ·(cid:105) + sin( α 0 ) sin( α 1 ) | 001 · · ·(cid:105) and so on, until obtaining | x (cid:105) as in Eq.(2).
Revised: Using Eq.(1), we will choose the angles such that, after the ﬁrst RBS gate of the loader, the qubits would be in the state x 0 | 100 · · ·(cid:105) + sin( α 0 ) | 010 · · ·(cid:105) and after the second one in the state x 0 | 100 · · ·(cid:105) + x 1 | 010 · · ·(cid:105) + sin( α 0 ) sin( α 1 ) | 001 · · ·(cid:105) and so on, until obtaining | x (cid:105) as in Eq.(2).

Original: To this end, we simply perform a classical preprocessing to compute recursively the n -1 loading angles, in time O ( n ) .
Revised: To this end, we simply perform a classical preprocessing to compute recursively the n − 1 loading angles, in time O ( n ) .

Original: We choose α 0 = arccos( x 0 ) , α 1 = arccos( x 1 sin − 1 ( α 0 )) , α 2 = arccos( x 2 sin − 1 ( α 0 ) sin − 1 ( α 1 )) and so on.
Revised: We choose α 0 = arccos( x 0 ) , α 1 = arccos( x 1 sin − 1 ( α 0 )) , α 2 = arccos( x 2 sin − 1 ( α 0 ) sin − 1 ( α 1 )) and so on.

Original: The ability of loading data in such a way relies on the assumption that each input vector is normalized, i.e. (cid:107) x (cid:107) 2 = 1 .
Revised: The ability of loading data in such a way relies on the assumption that each input vector is normalized, i.e. (cid:107) x (cid:107) 2 = 1 .

Original: This normalization constraint could seem arbitrary and impact the ability to learn from the data.
Revised: This normalization constraint could seem arbitrary and impact the ability to learn from the data.

Original: In fact, in the case of an orthogonal neural network, this normalization shouldn’t degrade the training because orthogonal weight matrices are in fact orthonormal and thus norm-preserving.
Revised: In fact, in the case of an orthogonal neural network, this normalization shouldn’t degrade the training because orthogonal weight matrices are in fact orthonormal and thus norm-preserving.

Original: Hence, changing the norm of the input vector, by diving each component by (cid:107) x (cid:107) 2 , in both classical and quantum settings is not a problem.
Revised: Hence, changing the norm of the input vector, by diving each component by (cid:107) x (cid:107) 2 , in both classical and quantum settings is not a problem.

Original: The normalization would impose that each input has the same norm, or the same ”luminosity” in the context of images, which can be helpful or harmful depending on the case.
Revised: The normalization would impose that each input has the same norm, or the same ”luminosity” in the context of images, which can be helpful or harmful depending on the case.

Original:  O RTHO NN S F EEDFORWARD P ASS
Revised:  O RTHO NN S F EEDFORWARD P ASS

Original: In this section, we will detail the effect of the quantum pyramidal circuit on an input encoded in a unary basis, as in Eq.(2).
Revised: In this section, we will detail the effect of the quantum pyramidal circuit on an input encoded in a unary basis, as in Eq.(2).

Original: We will also see in the end how to simulate this quantum circuit classically with a small overhead and thus be able to provide a fully classical scheme.
Revised: We will also see in the end how to simulate this quantum circuit classically with a small overhead and thus be able to provide a fully classical scheme.

Original: Let’s ﬁrst consider one pure unary input, where only the qubit j is in state | 1 (cid:105) (e.g. | 00000010 (cid:105) ).
Revised: Let’s ﬁrst consider one pure unary input, where only the qubit j is in state | 1 (cid:105) (e.g. | 00000010 (cid:105) ).

Original: This unary input will be transformed into a superposition of unary states, each with an amplitude.
Revised: This unary input will be transformed into a superposition of unary states, each with an amplitude.

Original: If we consider again only one of these possible unary outputs, where only the qubit i is in state | 1 (cid:105) , its amplitude can be interpreted as a conditional amplitude to transfer the | 1 (cid:105) from qubit j to qubit i .
Revised: If we consider again only one of these possible unary outputs, where only the qubit i is in state | 1 (cid:105) , its amplitude can be interpreted as a conditional amplitude to transfer the | 1 (cid:105) from qubit j to qubit i .

Original: Intuitively, this value is the sum of the quantum amplitudes associated to each possible path that connects the qubit j to qubit i , as shown in Fig.5.
Revised: Intuitively, this value is the sum of the quantum amplitudes associated to each possible path that connects the qubit j to qubit i , as shown in Fig.5.

Original: Using this image of connectivity between input and output qubits, we can construct a matrix W ∈ R n × n , where each element W ij is the overall conditional amplitude to transfer the | 1 (cid:105) from qubit j to qubit i .
Revised: Using this image of connectivity between input and output qubits, we can construct a matrix W ∈ R n × n , where each element W ij is the overall conditional amplitude to transfer the | 1 (cid:105) from qubit j to qubit i .

Original: Fig.5 shows an example where exactly three paths can be taken to map the input qubit j = 6 (the 7 th unary state) to the qubit i = 5 (the 6 th unary state).
Revised: Fig.5 shows an example where exactly three paths can be taken to map the input qubit j = 6 (the 7 th unary state) to the qubit i = 5 (the 6 th unary state).

Original: Each path comes with a certain amplitude.
Revised: Each path comes with a certain amplitude.

Original: For instance, one of the paths (the red one in Fig.5) moves up at the ﬁrst gate, and then stays put in the next three gates, with a resulting amplitude of − sin( θ 16 ) cos( θ 17 ) cos( θ 23 ) cos( θ 24 ) .
Revised: For instance, one of the paths (the red one in Fig.5) moves up at the ﬁrst gate, and then stays put in the next three gates, with a resulting amplitude of − sin( θ 16 ) cos( θ 17 ) cos( θ 23 ) cos( θ 24 ) .

Original: The sum of the amplitudes of all possible paths give us the element W 56 of the matrix W (where, for simplicity, s ( θ ) and c ( θ ) respectively stand for sin( θ ) and cos( θ ) ):
Revised: The sum of the amplitudes of all possible paths give us the element W 56 of the matrix W (where, for simplicity, s ( θ ) and c ( θ ) respectively stand for sin( θ ) and cos( θ ) ):

Original: [Equation]
Revised: [Equation]

Original: In fact, the n × n matrix W can be seen as the unitary matrix of our quantum circuit if we solely consider the unary basis, which is speciﬁed by the parameters of the quantum gates.
Revised: In fact, the n × n matrix W can be seen as the unitary matrix of our quantum circuit if we solely consider the unary basis, which is speciﬁed by the parameters of the quantum gates.

Original: A unitary is a complex unitary matrix, but in our case, with only real operations, the matrix is simply orthogonal.
Revised: A unitary is a complex unitary matrix, but in our case, with only real operations, the matrix is simply orthogonal.

Original: This proves the correspondence between any matrix W and the pyramidal quantum circuit.
Revised: This proves the correspondence between any matrix W and the pyramidal quantum circuit.

Original: The full unitary U W in the Hilbert Space of our n -qubit quantum circuit is a 2 n × 2 n matrix with the n × n matrix W embedded in it as a submatrix on the unary basis.
Revised: The full unitary U W in the Hilbert Space of our n -qubit quantum circuit is a 2 n × 2 n matrix with the n × n matrix W embedded in it as a submatrix on the unary basis.

Original: This is achieved by loading the data as unary states and by using only RBS gates that keep the number of 0s and 1s constant.
Revised: This is achieved by loading the data as unary states and by using only RBS gates that keep the number of 0s and 1s constant.

Original: For instance, as shown in Fig.6, a 3-qubit pyramidal circuit is described as a unique 3 × 3 matrix, that can be easily veriﬁed to be orthogonal.
Revised: For instance, as shown in Fig.6, a 3-qubit pyramidal circuit is described as a unique 3 × 3 matrix, that can be easily veriﬁed to be orthogonal.

Original: In Fig.5, we considered the case of single unary for both the input and output.
Revised: In Fig.5, we considered the case of single unary for both the input and output.

Original: But with actual data, as seen in Section 2.3, input and output states are in fact a superposition of unary states.
Revised: But with actual data, as seen in Section 2.3, input and output states are in fact a superposition of unary states.

Original: Thanks to the linearity of quantum mechanics in absence of measurements, the previous descriptions remain valid and can be applied on a linear combination of unary states.
Revised: Thanks to the linearity of quantum mechanics in absence of measurements, the previous descriptions remain valid and can be applied on a linear combination of unary states.

Original: Let’s consider an input vector x ∈ R n encoded as a quantum state | x (cid:105)
Revised: Let’s consider an input vector x ∈ R n encoded as a quantum state | x (cid:105)

Original: =
Revised: =

Original: (cid:80) n − 1 i =

Revised: (cid:80) n − 1 i =

Original: x i | e i (cid:105) where | e i (cid:105) represents the i th unary state (see Section 2.3).
Revised: x i | e i (cid:105) where | e i (cid:105) represents the i th unary state (see Section 2.3).

Original: By deﬁnition of W , each unary | e i (cid:105) will undergo a proper evolution | e i (cid:105) (cid:55)→ (cid:80) n − 1 j =0 W ij | e j (cid:105) .
Revised: By deﬁnition of W , each unary | e i (cid:105) will undergo a proper evolution | e i (cid:105) (cid:55)→ (cid:80) n − 1 j =0 W ij | e j (cid:105) .

Original: This yields, by linearity, to the following mapping
Revised: This yields, by linearity, to the following mapping

Original: [Equation]
Revised: [Equation]

Original: As explained above, our quantum circuit is equivalently described by the sparse unitary U W ∈ R 2 n × 2 n or on the unary basis by the matrix W ∈ R n × n .
Revised: As explained above, our quantum circuit is equivalently described by the sparse unitary U W ∈ R 2 n × 2 n or on the unary basis by the matrix W ∈ R n × n .

Original: This can be summarized with
Revised: This can be summarized with

Original: [Equation]
Revised: [Equation]

Original: We see from Eq.(4) and Eq.(5) that the output is in fact | y (cid:105) , the unary encoding of the vector y =
Revised: We see from Eq.(4) and Eq.(5) that the output is in fact | y (cid:105) , the unary encoding of the vector y =

Original: Wx , which is the output of a matrix multiplication between the n × n orthogonal matrix W and the input x ∈ R n .
Revised: Wx , which is the output of a matrix multiplication between the n × n orthogonal matrix W and the input x ∈ R n .

Original: As expected, each element of y is given by y k
Revised: As expected, each element of y is given by y k

Original: = (cid:80) n − 1 i =

Revised: = (cid:80) n − 1 i =

Original: W ik x i .
Revised: W ik x i .

Original: See Fig.7 for a diagram representation of this mapping.
Revised: See Fig.7 for a diagram representation of this mapping.

Original: Therefore, for any given neural network’s orthogonal layer, there is a quantum pyramidal circuit that reproduces it.
Revised: Therefore, for any given neural network’s orthogonal layer, there is a quantum pyramidal circuit that reproduces it.

Original: On the other hand, any quantum pyramidal circuit is implementing an orthogonal layer of some sort.
Revised: On the other hand, any quantum pyramidal circuit is implementing an orthogonal layer of some sort.

Original: Additional details concerning multi-layers branching, the tomography at the end of each layer, and the way to apply the non linearities are given in the Appendix, Section A.4.
Revised: Additional details concerning multi-layers branching, the tomography at the end of each layer, and the way to apply the non linearities are given in the Appendix, Section A.4.

Original: As a side note, we can ask if a circuit with only log( n ) qubits could also implement an orthogonal matrix multiplication of size n × n .
Revised: As a side note, we can ask if a circuit with only log( n ) qubits could also implement an orthogonal matrix multiplication of size n × n .

Original: Indeed, it would be a unitary matrix in R n × n , but since the circuit should also have n ( n − 1) / 2 free parameters to tune, this would come at a cost of large depth, potentially unsuitable for NISQ devices.
Revised: Indeed, it would be a unitary matrix in R n × n , but since the circuit should also have n ( n − 1) / 2 free parameters to tune, this would come at a cost of large depth, potentially unsuitable for NISQ devices.

Original: Classical implementation While we presented the quantum pyramidal circuit as the inspiration of the new methods for orthogonal neural networks, it is not hard to see that these quantum circuits can be simulated classical with a small overhead, thus yielding classical methods for orthogonal neural networks.
Revised: Classical implementation While we presented the quantum pyramidal circuit as the inspiration of the new methods for orthogonal neural networks, it is not hard to see that these quantum circuits can be simulated classical with a small overhead, thus yielding classical methods for orthogonal neural networks.

Original: This classical algorithm is simply the simulation of the quantum pyramidal circuit, where each RBS gate is replaced by a planar rotation between its two inputs.
Revised: This classical algorithm is simply the simulation of the quantum pyramidal circuit, where each RBS gate is replaced by a planar rotation between its two inputs.

Original: As shown in Fig.8, we propose a similar classical pyramidal circuit, where each layer is constituted of n ( n − 1)planar rotations, for a total of 4 × n ( n − 1)= O ( n 2 ) basic operations.
Revised: As shown in Fig.8, we propose a similar classical pyramidal circuit, where each layer is constituted of n ( n − 1)planar rotations, for a total of 4 × n ( n − 1)= O ( n 2 ) basic operations.

Original: Therefore our single layer feedforward pass has the same complexity O ( n 2 ) as the usual matrix multiplication.
Revised: Therefore our single layer feedforward pass has the same complexity O ( n 2 ) as the usual matrix multiplication.

Original: One may still have an advantage in performing the quantum circuit for inference, since the quantum circuit has depth O ( n ) , instead of the O ( n 2 ) classical complexity of the matrix-vector multiplication.
Revised: One may still have an advantage in performing the quantum circuit for inference, since the quantum circuit has depth O ( n ) , instead of the O ( n 2 ) classical complexity of the matrix-vector multiplication.

Original: Nevertheless, as we will see below, the main advantage of our method is that we can now train orthogonal weight matrices classically in time O ( n 2 ) , instead of the previously best-known O ( n 3 ) .
Revised: Nevertheless, as we will see below, the main advantage of our method is that we can now train orthogonal weight matrices classically in time O ( n 2 ) , instead of the previously best-known O ( n 3 ) .

Original:  O RTHO NN TRAINING : A NGLE ’ S G RADIENT C ALCULATION AND O RTHOGONAL M ATRIX U PDATE
Revised:  O RTHO NN TRAINING : A NGLE ’ S G RADIENT C ALCULATION AND O RTHOGONAL M ATRIX U PDATE

Original: Basic introduction and notations to the backpropagation in fully connected neural networks are given in the Appendix, Section A.2.
Revised: Basic introduction and notations to the backpropagation in fully connected neural networks are given in the Appendix, Section A.2.

Original: Looking through the prism of our pyramidal quantum circuit, the parameters to update are no longer the individual elements of the weight matrices directly, but the angles of the RBS gates that give rise to these matrices.
Revised: Looking through the prism of our pyramidal quantum circuit, the parameters to update are no longer the individual elements of the weight matrices directly, but the angles of the RBS gates that give rise to these matrices.

Original: Thus, we need to design an adaptation of the backpropagation method to our setting based on the angles.
Revised: Thus, we need to design an adaptation of the backpropagation method to our setting based on the angles.

Original: We will start by introducing some notation for a single layer (cid:96) , which will not be explicit in the notation for simplicity.
Revised: We will start by introducing some notation for a single layer (cid:96) , which will not be explicit in the notation for simplicity.

Original: We assume we have as many output bits as input bits, but this can easily be extended to the rectangular case.
Revised: We assume we have as many output bits as input bits, but this can easily be extended to the rectangular case.

Original: We ﬁrst introduce the notion of timesteps inside each layer, which correspond to the computational steps in the pyramidal structure of the circuit (see Fig.9).
Revised: We ﬁrst introduce the notion of timesteps inside each layer, which correspond to the computational steps in the pyramidal structure of the circuit (see Fig.9).

Original: It is easy to show that for n inputs, there will be 2 n − 3 such timesteps , each one indexed by an integer λ ∈ [0 , · · · , λ max ] .
Revised: It is easy to show that for n inputs, there will be 2 n − 3 such timesteps , each one indexed by an integer λ ∈ [0 , · · · , λ max ] .

Original: Applying a timestep consists in applying the matrix w λ , made of all the RBS gates aligned vertically at this timestep ( w λ is the unitary in the unary basis, see Section 3 for details).
Revised: Applying a timestep consists in applying the matrix w λ , made of all the RBS gates aligned vertically at this timestep ( w λ is the unitary in the unary basis, see Section 3 for details).

Original: Each time a timestep is applied, the resulting state is a vector in the unary basis named inner layer and noted ζ λ .
Revised: Each time a timestep is applied, the resulting state is a vector in the unary basis named inner layer and noted ζ λ .

Original: This evolution can be written as ζ λ +1 = w λ · ζ λ .
Revised: This evolution can be written as ζ λ +1 = w λ · ζ λ .

Original: We use this notation similar to the real layer (cid:96) , with the weight matrix W (cid:96) and the resulting vector z (cid:96) (see Section A.2).
Revised: We use this notation similar to the real layer (cid:96) , with the weight matrix W (cid:96) and the resulting vector z (cid:96) (see Section A.2).

Original: In fact we have the correspondences ζ 0 = a (cid:96) − 1 for the ﬁrst inner layer , which is the input of the actual layer, and z (cid:96) =
Revised: In fact we have the correspondences ζ 0 = a (cid:96) − 1 for the ﬁrst inner layer , which is the input of the actual layer, and z (cid:96) =

Original: w λ max · ζ λ max for the last one.
Revised: w λ max · ζ λ max for the last one.

Original: We also have W (cid:96) =
Revised: We also have W (cid:96) =

Original: w λ max · · · w 1 w 0 .
Revised: w λ max · · · w 1 w 0 .

Original: We use the same kind of notation for the backpropagation errors.
Revised: We use the same kind of notation for the backpropagation errors.

Original: At each timestep λ we deﬁne an inner error δ λ
Revised: At each timestep λ we deﬁne an inner error δ λ

Original: =
Revised: =

Original: ∂
Revised: ∂

Original: C ∂ζ λ .
Revised: C ∂ζ λ .

Original: This deﬁnition is similar to the layer error ∆ (cid:96) =
Revised: This deﬁnition is similar to the layer error ∆ (cid:96) =

Original: ∂ C ∂z (cid:96) .
Revised: ∂ C ∂z (cid:96) .

Original: In fact we will use the same backpropagation formulas, without non linearities, to retrieve each inner error vector δ λ
Revised: In fact we will use the same backpropagation formulas, without non linearities, to retrieve each inner error vector δ λ

Original: = ( w λ ) T · δ λ
Revised: = ( w λ ) T · δ λ

Original: +1 .
Revised: +1 .

Original: In particular, for the last timestep, the ﬁrst to be calculated, we have δ λ max = ( w λmax ) T · ∆ (cid:96) .
Revised: In particular, for the last timestep, the ﬁrst to be calculated, we have δ λ max = ( w λmax ) T · ∆ (cid:96) .

Original: Finally, we can retrieve the error at the previous layer (cid:96) − 1 using the correspondence ∆ (cid:96) − 1 = δ 0 (cid:12) σ (cid:48)
Revised: Finally, we can retrieve the error at the previous layer (cid:96) − 1 using the correspondence ∆ (cid:96) − 1 = δ 0 (cid:12) σ (cid:48)

Original: ( z (cid:96) ) .
Revised: ( z (cid:96) ) .

Original: The reason for this breakdown into timesteps is the ability to efﬁciently obtain the gradient with respect to each angle.
Revised: The reason for this breakdown into timesteps is the ability to efﬁciently obtain the gradient with respect to each angle.

Original: Let’s consider one gate with angle θ i , acting at the timestep λ on qubits i and i + 1 .
Revised: Let’s consider one gate with angle θ i , acting at the timestep λ on qubits i and i + 1 .

Original: We will decompose the gradient ∂ C ∂θ
Revised: We will decompose the gradient ∂ C ∂θ

Original: i using each component, indexed by the integer k , of the inner layer and inner error vectors
Revised: i using each component, indexed by the integer k , of the inner layer and inner error vectors

Original: [Equation]
Revised: [Equation]

Original: Since timestep λ is only composed of separated RBS gates, the matrix w λ consists of diagonally arranged 2x2 block submatrices given in Eq.(1).
Revised: Since timestep λ is only composed of separated RBS gates, the matrix w λ consists of diagonally arranged 2x2 block submatrices given in Eq.(1).

Original: Only one of these submatrices depends on the angle θ considered here, at the position i and i + 1 in the matrix.
Revised: Only one of these submatrices depends on the angle θ considered here, at the position i and i + 1 in the matrix.

Original: The above gradient can be rewritten as:
Revised: The above gradient can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: Therefore we have shown a way to compute each angle gradient: During the feedforward pass, one must apply sequentially each of the 2 n − 

Revised: Therefore we have shown a way to compute each angle gradient: During the feedforward pass, one must apply sequentially each of the 2 n −

Original: = O ( n ) timesteps, and store the resulting vectors, the inner layers ζ λ .
Revised: = O ( n ) timesteps, and store the resulting vectors, the inner layers ζ λ .

Original: During the backpropagation, one obtains the inner errors δ λ by applying the timesteps in reverse.
Revised: During the backpropagation, one obtains the inner errors δ λ by applying the timesteps in reverse.

Original: One can ﬁnally use a gradient descent on each angle θ i , while preserving the orthogonality of the overall equivalent weight matrix θ (cid:96)i ← θ (cid:96)i − λ ∂ C ∂θ (cid:96)i .
Revised: One can ﬁnally use a gradient descent on each angle θ i , while preserving the orthogonality of the overall equivalent weight matrix θ (cid:96)i ← θ (cid:96)i − λ ∂ C ∂θ (cid:96)i .

Original: An interesting aspect of this gradient descent is the fact that the optimization is performed in the angle landscape, and not on the equivalent weight landscape.
Revised: An interesting aspect of this gradient descent is the fact that the optimization is performed in the angle landscape, and not on the equivalent weight landscape.

Original: These landscapes can potentially be different and hence our optimization can produce different models.
Revised: These landscapes can potentially be different and hence our optimization can produce different models.

Original: We leave open the question of ﬁnding a theoretical argument to compare the properties of both landscapes.
Revised: We leave open the question of ﬁnding a theoretical argument to compare the properties of both landscapes.

Original: As one can see from the above description, this is in fact a classical algorithm to obtain the angle’s gradients, which allows us to train our OrthoNN efﬁciently classically while preserving the strict orthogonality.
Revised: As one can see from the above description, this is in fact a classical algorithm to obtain the angle’s gradients, which allows us to train our OrthoNN efﬁciently classically while preserving the strict orthogonality.

Original: To obtain the angle’s gradient, one needs to store the 2 n -3 inner layers ζ λ during the feedforward pass.
Revised: To obtain the angle’s gradient, one needs to store the 2 n − 3 inner layers ζ λ during the feedforward pass.

Original: Next, given the error at the following layer, we perform a backward loop on each timestep (see Fig.8).
Revised: Next, given the error at the following layer, we perform a backward loop on each timestep (see Fig.8).

Original: At each timestep , we obtain the gradient for each angle parameter, by simply applying Eq.(4).
Revised: At each timestep , we obtain the gradient for each angle parameter, by simply applying Eq.(4).

Original: This requires O (1) operations for each angle.
Revised: This requires O (1) operations for each angle.

Original: Since there are at most n/ 2 angles per timesteps , estimating gradients has a complexity of O ( n 2 ) .
Revised: Since there are at most n/angles per timesteps , estimating gradients has a complexity of O ( n 2 ) .

Original: After each timestep , the next inner error δ λ − 1 is computed as well, using at most 4 n/ 2 operations.
Revised: After each timestep , the next inner error δ λ − 1 is computed as well, using at most 4 n/ 2 operations.

Original: In the end, our classical algorithm allows us to compute the gradients of the n ( n − 1) / 2 angles in O ( n 2 ) , in order to perform a gradient descent respecting the strict orthogonality of the weight matrix.
Revised: In the end, our classical algorithm allows us to compute the gradients of the n ( n − 1) / 2 angles in O ( n 2 ) , in order to perform a gradient descent respecting the strict orthogonality of the weight matrix.

Original: This is considerably faster than previous methods based on Singular Value Decomposition methods and provides a training method that is asymptotically as fast as for normal neural networks, while providing the extra property of orthogonality.
Revised: This is considerably faster than previous methods based on Singular Value Decomposition methods and provides a training method that is asymptotically as fast as for normal neural networks, while providing the extra property of orthogonality.

Original:  N UMERICAL E XPERIMENTS
Revised:  N UMERICAL E XPERIMENTS

Original: We performed basic numerical experiments to verify the abilities of our pyramidal circuit, on the standard MNIST dataset LeCun & Cortes (2010).
Revised: We performed basic numerical experiments to verify the abilities of our pyramidal circuit, on the standard MNIST dataset LeCun & Cortes (2010).

Original: Note that current quantum hardware and software are not yet suited for bigger experiments.
Revised: Note that current quantum hardware and software are not yet suited for bigger experiments.

Original: We ﬁrst compared the training of our Classical OrthoNN to the SVB algorithm from Jia et al.
Revised: We ﬁrst compared the training of our Classical OrthoNN to the SVB algorithm from Jia et al.

Original: (see Section A.1).Results as reported in Fig.10, and more in the Appendix, Section A.4.3.
Revised: (see Section A.1).Results as reported in Fig.10, and more in the Appendix, Section A.4.3.

Original: These small scale tests conﬁrmed that the pyramidal circuits and the corresponding gradient descent on the angles were efﬁcient for learning a classiﬁcation task.
Revised: These small scale tests conﬁrmed that the pyramidal circuits and the corresponding gradient descent on the angles were efﬁcient for learning a classiﬁcation task.

Original: Then, we implemented the quantum circuit on a real quantum computer provided by IBM.
Revised: Then, we implemented the quantum circuit on a real quantum computer provided by IBM.

Original: We used a 16 and 5 qubits device to perform respectively a [8,2] and a [4,2] orthogonal layer.
Revised: We used a 16 and 5 qubits device to perform respectively a [8,2] and a [4,2] orthogonal layer.

Original: A pyramidal
Revised: A pyramidal

Original: OrthoNN was trained classically, and the resulting angles were transferred to test the quantum circuit on a classiﬁcation task on classes 6 and 9 of the MNIST dataset, over 500 samples.
Revised: OrthoNN was trained classically, and the resulting angles were transferred to test the quantum circuit on a classiﬁcation task on classes 6 and 9 of the MNIST dataset, over 500 samples.

Original: We compared the real experiment with a simulated one, and the classical pyramidal circuit as well.
Revised: We compared the real experiment with a simulated one, and the classical pyramidal circuit as well.

Original: Results are reported in Table 2.
Revised: Results are reported in Table 2.

Original: [Table]
Revised: [Table]

Original: .4.32 and ibmq guadalupe v1.2.17 are respectively 5 and 16 qubits quantum computers.
Revised: .4.32 and ibmq guadalupe v1.2.17 are respectively 5 and 16 qubits quantum computers.

Original:  C ONCLUSION AND O UTLOOK
Revised:  C ONCLUSION AND O UTLOOK

Original: In this work, we have proposed for the ﬁrst time training methods for orthogonal neural networks (OrthoNNs) that run in quadratic time, a signiﬁcant improvement from previous methods based on
Revised: In this work, we have proposed for the ﬁrst time training methods for orthogonal neural networks (OrthoNNs) that run in quadratic time, a signiﬁcant improvement from previous methods based on

Original: Singular Value Decomposition.
Revised: Singular Value Decomposition.

Original: The main idea of our method is to replace usual weights and orthogonal matrices with an equivalent pyramidal circuit made of two-dimensional rotations.
Revised: The main idea of our method is to replace usual weights and orthogonal matrices with an equivalent pyramidal circuit made of two-dimensional rotations.

Original: Each rotation is parametrizable by an angle, and the gradient descent takes place in the angle’s optimization landscape.
Revised: Each rotation is parametrizable by an angle, and the gradient descent takes place in the angle’s optimization landscape.

Original: This unique type of gradient backpropagation ensures perfect orthogonality of the weight matrices while improving the running time compared to previous works.
Revised: This unique type of gradient backpropagation ensures perfect orthogonality of the weight matrices while improving the running time compared to previous works.

Original: Moreover, we propose both classical and quantum methods for inference, where the forward pass on a near term quantum computer would provide a provable advantage in the running time.
Revised: Moreover, we propose both classical and quantum methods for inference, where the forward pass on a near term quantum computer would provide a provable advantage in the running time.

Original: This work expands the ﬁeld of quantum deep learning by introducing new tools, concepts, and equivalences with classical deep learning theory.
Revised: This work expands the ﬁeld of quantum deep learning by introducing new tools, concepts, and equivalences with classical deep learning theory.

Original: We have highlighted open questions regarding the construction of such pyramidal circuits for neural networks and their potential new advantages in terms of execution time, accuracy, and learning properties.
Revised: We have highlighted open questions regarding the construction of such pyramidal circuits for neural networks and their potential new advantages in terms of execution time, accuracy, and learning properties.

Revised:  R EPRODUCIBILITY S TATEMENT

Revised: In this paper, we have introduced new methods for implementing and training orthogonal neural networks.

Revised: We have explained in detail each part of both algorithms’ implementation (see Appendix for more details).

Revised: The dataset used here for basic simulation is the well-known MNIST dataset, easily accessible to everyone.

Revised: It is in our belief that anyone with classical software skills can re-implement the classical algorithm (see Fig.9), in Python for instance.

Original: A A PPENDIX A. R ELATED W ORK ON C LASSICAL O RTHO NN S
Revised: Similarly, on any quantum software platform, one can implement the quantum gates given in Fig.11 and perform the whole algorithm using Fig.12, Fig. and mathematical details through the paper. This would allow simulating the quantum circuit using accessible platforms such as IBM’s qiskit. However, in order to perform the actual experiment on a quantum computer currently accessible, as we have done on IBM 5 and 16 qubits devices one must have access to their service (or any other provider).

Revised: Some quantum computers are available for free, but others require speciﬁc agreements.

Revised:  E THICS S TATEMENT

Revised: Our work on quantum and classical algorithms for orthogonal neural networks is mainly a theoretical work aiming at improving the complexity of neural networks training on current computers and near term quantum computers.

Revised: Therefore, any ethical concerns regarding classical neural networks may also concern our work.

Revised: A A PPENDIX A. R ELATED W ORK ON C LASSICAL O RTHO NN S

Original: The idea behind Orthogonal Neural Networks (OrthoNNs) is to add a constraint to the weight matrices corresponding to the layers of a neural network.
Revised: The idea behind Orthogonal Neural Networks (OrthoNNs) is to add a constraint to the weight matrices corresponding to the layers of a neural network.

Original: Imposing orthogonality to these matrices has theoretical and practical beneﬁts in the generalization error Jia et al.
Revised: Imposing orthogonality to these matrices has theoretical and practical beneﬁts in the generalization error Jia et al.

Original: Orthogonality ensures a low weights redundancy and preserves the magnitude of the weight matrix’s eigenvalues to avoid vanishing gradients.
Revised: Orthogonality ensures a low weights redundancy and preserves the magnitude of the weight matrix’s eigenvalues to avoid vanishing gradients.

Original: In terms of complexity, for a single layer, the feedforward pass of an OrthoNN is simply a matrix multiplication, hence has a running time of O ( n 2 ) if n × n is the size of the orthogonal matrix.
Revised: In terms of complexity, for a single layer, the feedforward pass of an OrthoNN is simply a matrix multiplication, hence has a running time of O ( n 2 ) if n × n is the size of the orthogonal matrix.

Original: It is also interesting to note that OrthoNNs have been generalized to Convolutional
Revised: It is also interesting to note that OrthoNNs have been generalized to Convolutional

Original: Neural Networks Wang et al.
Revised: Neural Networks Wang et al.

Original: The main difﬁculty of OrthoNNs is to preserve the orthogonality of the matrices while updating them during gradient descent.
Revised: The main difﬁculty of OrthoNNs is to preserve the orthogonality of the matrices while updating them during gradient descent.

Original: Several algorithms have been proposed to this end Wang et al.
Revised: Several algorithms have been proposed to this end Wang et al.

Original: Bansal et al.
Revised: Bansal et al.

Original: (2018); Lezcano-Casado & Martınez-Rubio (2019),
Revised: (2018); Lezcano-Casado & Martınez-Rubio (2019),

Original: but they all point that pure orthogonality is computationally hard to conserve.
Revised: but they all point that pure orthogonality is computationally hard to conserve.

Original: Therefore, previous works allow for approximations: strict orthogonality is no longer required, and the matrices are often pushed toward orthogonality using regularization techniques during weights update.
Revised: Therefore, previous works allow for approximations: strict orthogonality is no longer required, and the matrices are often pushed toward orthogonality using regularization techniques during weights update.

Original: We present two algorithms from Jia et al.
Revised: We present two algorithms from Jia et al.

Original: for updating orthogonal matrices.
Revised: for updating orthogonal matrices.

Original: The ﬁrst algorithm is an approximated one, called Singular Value Bounding (SVB).
Revised: The ﬁrst algorithm is an approximated one, called Singular Value Bounding (SVB).

Original: It starts by applying the usual gradient descent update on the matrix, therefore making it not orthogonal anymore.
Revised: It starts by applying the usual gradient descent update on the matrix, therefore making it not orthogonal anymore.

Original: Then, the singular values of the new matrix are extracted using Singular Value Decomposition (SVD), their values are manually pushed to be close to 1, and the matrix is recomposed hence enforcing orthogonality.
Revised: Then, the singular values of the new matrix are extracted using Singular Value Decomposition (SVD), their values are manually pushed to be close to 1, and the matrix is recomposed hence enforcing orthogonality.

Original: This method shows less advantage on practical experiments Jia et al.
Revised: This method shows less advantage on practical experiments Jia et al.

Original: It has a complexity of O ( n 3 ) due to the SVD, which in practice is better than the next algorithm.
Revised: It has a complexity of O ( n 3 ) due to the SVD, which in practice is better than the next algorithm.

Original: Note that this running time is still longer than O ( n 2 ) , the running time to perform standard gradient descent.
Revised: Note that this running time is still longer than O ( n 2 ) , the running time to perform standard gradient descent.

Original: The second algorithm can be considered perfect since it ensures strict orthogonality by performing the gradient descent in the manifold of orthogonal matrices, called the Stiefel Manifold.
Revised: The second algorithm can be considered perfect since it ensures strict orthogonality by performing the gradient descent in the manifold of orthogonal matrices, called the Stiefel Manifold.

Original: In practice
Revised: In practice

Original: Jia et al.
Revised: Jia et al.

Original: (2019) this method showed a substantially advantageous classiﬁcation results on standard datasets.
Revised: (2019) this method showed a substantially advantageous classiﬁcation results on standard datasets.

Original: This algorithm requires O ( n 3 ) operations, but is very prohibitive in practice.
Revised: This algorithm requires O ( n 3 ) operations, but is very prohibitive in practice.

Original: We give an informal step-by-step detail of this algorithm (see Jia et al.
Revised: We give an informal step-by-step detail of this algorithm (see Jia et al.

Original: (2019), Appendix G for details):
Revised: (2019), Appendix G for details):

Original: Compute the gradient G of the weight matrix W . 2.
Revised: Compute the gradient G of the weight matrix W . 2.

Original: Project the gradient matrix G in the tangent space, (The space tangent to the manifold at this point W ): multiply G by some other matrices based on W :
Revised: Project the gradient matrix G in the tangent space, (The space tangent to the manifold at this point W ): multiply G by some other matrices based on W :

Original: [Equation]
Revised: [Equation]

Original: This requires several matrix-matrix multiplications.
Revised: This requires several matrix-matrix multiplications.

Original: In the case of square n × n matrices, each has complexity O ( n 3 ) .
Revised: In the case of square n × n matrices, each has complexity O ( n 3 ) .

Original: the result of this projection is called the manifold gradient
Revised: the result of this projection is called the manifold gradient

Original: Ω . 3.
Revised: Ω . 3.

Original: update W (cid:48)
Revised: update W (cid:48)

Original: =
Revised: =

Original: W − η Ω , where η is the chosen learning rate.
Revised: W − η Ω , where η is the chosen learning rate.

Original: Perform a retraction from the tangent space to the manifold.
Revised: Perform a retraction from the tangent space to the manifold.

Original: To do so we multiply W (cid:48) by Q factor of the QR decomposition , obtained using Gram Schmidt orthonormalization, which has complexity O (2 n 3 ) .
Revised: To do so we multiply W (cid:48) by Q factor of the QR decomposition , obtained using Gram Schmidt orthonormalization, which has complexity O (2 n 3 ) .

Original: A. C LASSICAL B ACKPROPAGATION A LGORITHM
Revised: A. C LASSICAL B ACKPROPAGATION A LGORITHM

Original: The backpropagation in a fully connected neural network is a well know and efﬁcient procedure to update the weight matrix at each layer Hecht-Nielsen (1992);
Revised: The backpropagation in a fully connected neural network is a well know and efﬁcient procedure to update the weight matrix at each layer Hecht-Nielsen (1992);

Original: Rojas (1996).
Revised: Rojas (1996).

Original: At layer (cid:96) , we note its weight matrices W (cid:96) and biases b (cid:96) .
Revised: At layer (cid:96) , we note its weight matrices W (cid:96) and biases b (cid:96) .

Original: Each layer is followed by a non linear function σ , and can therefore be written as
Revised: Each layer is followed by a non linear function σ , and can therefore be written as

Original: [Equation]
Revised: [Equation]

Original: After the last layer, one can deﬁne a cost function C that compares the output to the ground truth.
Revised: After the last layer, one can deﬁne a cost function C that compares the output to the ground truth.

Original: The goal is to calculate the gradient of C with respect to each weight and bias, namely ∂ C ∂W (cid:96) and
Revised: The goal is to calculate the gradient of C with respect to each weight and bias, namely ∂ C ∂W (cid:96) and

Original: ∂ C ∂b (cid:96) .
Revised: ∂ C ∂b (cid:96) .

Original: In the backpropagation, we start by calculating these gradients for the last layer, then propagate back to the ﬁrst layer.
Revised: In the backpropagation, we start by calculating these gradients for the last layer, then propagate back to the ﬁrst layer.

Original: We will require to obtain the error vector at layer (cid:96) deﬁned by ∆ (cid:96) =
Revised: We will require to obtain the error vector at layer (cid:96) deﬁned by ∆ (cid:96) =

Original: ∂ C ∂z (cid:96) .
Revised: ∂ C ∂z (cid:96) .

Original: One can show the backward recursive relation ∆ (cid:96) =
Revised: One can show the backward recursive relation ∆ (cid:96) =

Original: ( W (cid:96) +1 ) T · ∆ (cid:96) +1 (cid:12) σ (cid:48)
Revised: ( W (cid:96) +1 ) T · ∆ (cid:96) +1 (cid:12) σ (cid:48)

Original: ( z (cid:96) ) , where (cid:12) symbolizes the Hadamard product, or entry-wise multiplication.
Revised: ( z (cid:96) ) , where (cid:12) symbolizes the Hadamard product, or entry-wise multiplication.

Original: Note that the previous computation requires simply to apply the layer ( ie apply matrix multiplication) in reverse.
Revised: Note that the previous computation requires simply to apply the layer ( ie apply matrix multiplication) in reverse.

Original: We can then show that each element of the weight gradient matrix at layer (cid:96) is given by ∂ C
Revised: We can then show that each element of the weight gradient matrix at layer (cid:96) is given by ∂ C

Original: [Equation]
Revised: [Equation]

Original: (cid:96)jk
Revised: (cid:96)jk

Original: [Equation]
Revised: [Equation]

Original: Similarly, the gradient with respect to the biases is easily deﬁned as ∂ C ∂b (cid:96)j = ∆ (cid:96)j .
Revised: Similarly, the gradient with respect to the biases is easily deﬁned as ∂ C ∂b (cid:96)j = ∆ (cid:96)j .

Original: Once these gradients are computed, we update the parameters using the gradient descent rule, with learning rate λ :
Revised: Once these gradients are computed, we update the parameters using the gradient descent rule, with learning rate λ :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. P RELIMINARIES IN Q UANTUM C OMPUTING
Revised: A. P RELIMINARIES IN Q UANTUM C OMPUTING

Original: We present a succinct broad-audience quantum information background necessary for this work.
Revised: We present a succinct broad-audience quantum information background necessary for this work.

Original: See Nielsen & Chuang (2002) for a detailed course.
Revised: See Nielsen & Chuang (2002) for a detailed course.

Original: Qubits: In classical computing, a bit can be either 0 or 1.
Revised: Qubits: In classical computing, a bit can be either 0 or 1.

Original: With a quantum information perspective, a quantum bit or qubit can be is state | 0 (cid:105) , | 1 (cid:105) .
Revised: With a quantum information perspective, a quantum bit or qubit can be is state | 0 (cid:105) , | 1 (cid:105) .

Original: We use the braket notation |·(cid:105) to specify the quantum nature of the bit.
Revised: We use the braket notation |·(cid:105) to specify the quantum nature of the bit.

Original: The qubits can be in superposition of both states α | 0 (cid:105) + β | 1 (cid:105) where α, β ∈ C such that | α | 2 + | β | 2 = 1 .
Revised: The qubits can be in superposition of both states α | 0 (cid:105) + β | 1 (cid:105) where α, β ∈ C such that | α | 2 + | β | 2 = 1 .

Original: The coefﬁcients α and β are called amplitudes .
Revised: The coefﬁcients α and β are called amplitudes .

Original: The probabilities of observing either 0 or 1 when measuring the qubit are linked to the amplitudes:
Revised: The probabilities of observing either 0 or 1 when measuring the qubit are linked to the amplitudes:

Original: [Equation]
Revised: [Equation]

Original: As quantum physics teaches us, any superposition is possible before the measurement, which gives special abilities in terms of computation.
Revised: As quantum physics teaches us, any superposition is possible before the measurement, which gives special abilities in terms of computation.

Original: With a n qubits, 2 n possible binary combinations (e.g.
Revised: With a n qubits, 2 n possible binary combinations (e.g.

Original: | 01 · · · 1001 (cid:105) ) can exist simultaneously, each with its own amplitude.
Revised: | 01 · · · 1001 (cid:105) ) can exist simultaneously, each with its own amplitude.

Original: A n qubits system can be represented as a normalized vector in a 2 n dimensional Hilbert space.
Revised: A n qubits system can be represented as a normalized vector in a 2 n dimensional Hilbert space.

Original: A multiple qubit system is called a quantum register .
Revised: A multiple qubit system is called a quantum register .

Original: If | p (cid:105) and | q (cid:105) are two quantum states or quantum registers, the whole system can be represented as a tensor product | p (cid:105) ⊗ | q (cid:105) , also written as | p (cid:105) | q (cid:105) or | p, q (cid:105) .
Revised: If | p (cid:105) and | q (cid:105) are two quantum states or quantum registers, the whole system can be represented as a tensor product | p (cid:105) ⊗ | q (cid:105) , also written as | p (cid:105) | q (cid:105) or | p, q (cid:105) .

Original: Quantum Computation:
Revised: Quantum Computation:

Original: As logical gates in classical circuits, qubits or quantum registers are processed using quantum gates.
Revised: As logical gates in classical circuits, qubits or quantum registers are processed using quantum gates.

Original: A quantum gate is a unitary mapping in the Hilbert space, preserving the unit norm of the quantum state vector.
Revised: A quantum gate is a unitary mapping in the Hilbert space, preserving the unit norm of the quantum state vector.

Original: Therefore, a quantum gate acting on n qubits is a matrix U ∈ C 2 n such that UU † =
Revised: Therefore, a quantum gate acting on n qubits is a matrix U ∈ C 2 n such that UU † =

Original: U † U
Revised: U † U

Original: =
Revised: =

Original: I , with U † being the adjoint, or conjugate transpose, of
Revised: I , with U † being the adjoint, or conjugate transpose, of

Original: U .
Revised: U .

Original: Common single qubit gates include the Hadamard gate 1 √(cid:18) 1 −(cid:19) that maps | 0 (cid:105) (cid:55)→ 1 √ 2 ( | 0 (cid:105)
Revised: Common single qubit gates include the Hadamard gate 1 √(cid:18) 1 −(cid:19) that maps | 0 (cid:105) (cid:55)→ 1 √ 2 ( | 0 (cid:105)

Original: + | 1 (cid:105) ) and | 1 (cid:105) (cid:55)→ 1 √ 2 ( | 0 (cid:105) − | 1 (cid:105) ) , creating the quantum superposition, the NOT gate (cid:18) 01(cid:19) that permutes | 0 (cid:105) and | 1 (cid:105) , or R y rotation gate parametrized by an angle θ , given by
Revised: + | 1 (cid:105) ) and | 1 (cid:105) (cid:55)→ 1 √ 2 ( | 0 (cid:105) − | 1 (cid:105) ) , creating the quantum superposition, the NOT gate (cid:18) 01(cid:19) that permutes | 0 (cid:105) and | 1 (cid:105) , or R y rotation gate parametrized by an angle θ , given by

Original: [Equation]
Revised: [Equation]

Original: Common two-qubits gates includes the CNOT gate   1 0 00 1 00 0 00 

Revised: Common two-qubits gates includes the CNOT gate   1 0 00 1 00 0 00

Original: 1  which is a NOT gate applied on the second qubit only if the ﬁrst one is in state | 1 (cid:105) , or similarly the CZ gate   1 0 0 1 0 0 0 0 0 −  .
Revised: 1  which is a NOT gate applied on the second qubit only if the ﬁrst one is in state | 1 (cid:105) , or similarly the CZ gate   1 0 0 1 0 0 0 0 0 −  .

Original: In this work, we use the RBS gate given in Eq.(1).
Revised: In this work, we use the RBS gate given in Eq.(1).

Original: This gate can be implemented rather easily, either as a native gate, known as FSIM Foxen et al.
Revised: This gate can be implemented rather easily, either as a native gate, known as FSIM Foxen et al.

Original: (2020), or using four Hadamard gates, two R y rotation gates, and two two-qubits CZ gates:
Revised: (2020), or using four Hadamard gates, two R y rotation gates, and two two-qubits CZ gates:

Original: The main advantage of quantum gates is their ability to be applied to a superposition of inputs.
Revised: The main advantage of quantum gates is their ability to be applied to a superposition of inputs.

Original: Indeed, given a gate U such that U | x (cid:105) (cid:55)→ | f ( x ) (cid:105) , we can apply it to all possible combinations of x at once U ( 1 C (cid:80) x
Revised: Indeed, given a gate U such that U | x (cid:105) (cid:55)→ | f ( x ) (cid:105) , we can apply it to all possible combinations of x at once U ( 1 C (cid:80) x

Original: | x (cid:105) )
Revised: | x (cid:105) )

Original: (cid:55)→ 1 C (cid:80) x | f
Revised: (cid:55)→ 1 C (cid:80) x | f

Original: ( x ) (cid:105) .
Revised: ( x ) (cid:105) .

Original: A. A DDITIONAL D ETAILS ON THE Q UANTUM P YRAMIDAL C IRCUIT A.4. T OMOGRAPHY AND E RROR M ITIGATION
Revised: A. A DDITIONAL D ETAILS ON THE Q UANTUM P YRAMIDAL C IRCUIT A.4. T OMOGRAPHY AND E RROR M ITIGATION

Original: As shown in Fig.7, when using the quantum circuit, the output is a quantum state | y (cid:105) =
Revised: As shown in Fig.7, when using the quantum circuit, the output is a quantum state | y (cid:105) =

Original: | Wx (cid:105) .
Revised: | Wx (cid:105) .

Original: As often in quantum machine learning Aaronson (2015), it is important to go all to way and consider the cost of retrieving classical outputs, using a procedure called tomography.
Revised: As often in quantum machine learning Aaronson (2015), it is important to go all to way and consider the cost of retrieving classical outputs, using a procedure called tomography.

Original: In our case this is even more crucial since, between each layer, the quantum output will be converted into a classical one in order to apply a non linear function, and then reloaded for the next layer.
Revised: In our case this is even more crucial since, between each layer, the quantum output will be converted into a classical one in order to apply a non linear function, and then reloaded for the next layer.

Original: Error Mitigation Before detailing the tomography procedure, it is interesting to notice that with our restriction to unary states, a strong beneﬁt appears for error mitigation purposes.
Revised: Error Mitigation Before detailing the tomography procedure, it is interesting to notice that with our restriction to unary states, a strong beneﬁt appears for error mitigation purposes.

Original: Indeed, as we expect to obtain only quantum superposition of unary states at every layer, we can post process our measurements and discard the ones that present non unary states ( i.e. states with more than one qubit in state | 1 (cid:105) , or the ground state).
Revised: Indeed, as we expect to obtain only quantum superposition of unary states at every layer, we can post process our measurements and discard the ones that present non unary states ( i.e. states with more than one qubit in state | 1 (cid:105) , or the ground state).

Original: The most expected error is a bit ﬂip between | 1 (cid:105) and | 0 (cid:105) .
Revised: The most expected error is a bit ﬂip between | 1 (cid:105) and | 0 (cid:105) .

Original: The case where two bit ﬂips happened, which would pass through our error mitigation, is even less probable.
Revised: The case where two bit ﬂips happened, which would pass through our error mitigation, is even less probable.

Original: Tomography Retrieving the amplitudes of a quantum state comes at cost of multiple measurements.
Revised: Tomography Retrieving the amplitudes of a quantum state comes at cost of multiple measurements.

Original: which requires running the circuit multiples times, hence adding a multiplicative overhead in the running time.
Revised: which requires running the circuit multiples times, hence adding a multiplicative overhead in the running time.

Original: A ﬁnite number of samples is also a source of approximation in the ﬁnal result.
Revised: A ﬁnite number of samples is also a source of approximation in the ﬁnal result.

Original: In this work, we will allow for (cid:96) ∞ errors Kerenidis et al.
Revised: In this work, we will allow for (cid:96) ∞ errors Kerenidis et al.

Original: The (cid:96) ∞ tomography on a quantum state | y (cid:105) with unary encoding on n qubits requires O (log( n ) /δ 2 ) measurements, where δ > 0 is the error threshold allowed.
Revised: The (cid:96) ∞ tomography on a quantum state | y (cid:105) with unary encoding on n qubits requires O (log( n ) /δ 2 ) measurements, where δ > 0 is the error threshold allowed.

Original: For each j ∈ [ n ] , | y j | will be obtained with an absolute error δ , and if | y j | < δ , it will most probably not be measured, hence set to 0.
Revised: For each j ∈ [ n ] , | y j | will be obtained with an absolute error δ , and if | y j | < δ , it will most probably not be measured, hence set to 0.

Original: In practice, one would perform as many measurements as is convenient during the experiment, and deduce the equivalent precision δ from the number of measurements made.
Revised: In practice, one would perform as many measurements as is convenient during the experiment, and deduce the equivalent precision δ from the number of measurements made.

Original: The next logical part is to obtain the sign of each component of the vector.
Revised: The next logical part is to obtain the sign of each component of the vector.

Original: Indeed, we only measure probabilities that are the square module of the quantum amplitudes (see Section A.3).
Revised: Indeed, we only measure probabilities that are the square module of the quantum amplitudes (see Section A.3).

Original: In the case of neural networks, it is important to obtain the sign of the layer’s components in order to apply certain type of non linearities.
Revised: In the case of neural networks, it is important to obtain the sign of the layer’s components in order to apply certain type of non linearities.

Original: For instance, the ReLu activation function is often used to set all negative components to 0.
Revised: For instance, the ReLu activation function is often used to set all negative components to 0.

Original: In Fig.12, we propose a speciﬁc enhancement to our circuit to obtain the signs of the vector’s components at low cost.
Revised: In Fig.12, we propose a speciﬁc enhancement to our circuit to obtain the signs of the vector’s components at low cost.

Original: The sign retrieval procedure consists of three parts.
Revised: The sign retrieval procedure consists of three parts.

Original: a) The circuit is ﬁrst applied as described above, allowing to retrieve each squared amplitude y 2 j with precision δ > 0 using the (cid:96) ∞ tomography.
Revised: a) The circuit is ﬁrst applied as described above, allowing to retrieve each squared amplitude y 2 j with precision δ > 0 using the (cid:96) ∞ tomography.

Original: The probability of measuring the unary state | e 1 (cid:105) ( i.e. | 100 ... (cid:105) ), is p ( e 1 ) =
Revised: The probability of measuring the unary state | e 1 (cid:105) ( i.e. | 100 ... (cid:105) ), is p ( e 1 ) =

Original: y 2 1 .
Revised: y 2 1 .

Original: b) We apply the same steps a second time on a modiﬁed circuit.
Revised: b) We apply the same steps a second time on a modiﬁed circuit.

Original: It has additional RBS gates with angle π/ 4 at the end,
Revised: It has additional RBS gates with angle π/ 4 at the end,

Original: which will mix the amplitudes pair by pair.
Revised: which will mix the amplitudes pair by pair.

Original: The probabilities to measure | e 1 (cid:105) and | e 2 (cid:105) are now given by p ( e 1 ) =
Revised: The probabilities to measure | e 1 (cid:105) and | e 2 (cid:105) are now given by p ( e 1 ) =

Original: ( y 1 + y 2 ) 2 and p ( e 2 ) =
Revised: ( y 1 + y 2 ) 2 and p ( e 2 ) =

Original: ( y 1 − y 2 ) 2 .
Revised: ( y 1 − y 2 ) 2 .

Original: Therefore if p ( e 1 ) > p ( e 2 ) , we have sign ( y 1 ) (cid:54) =
Revised: Therefore if p ( e 1 ) > p ( e 2 ) , we have sign ( y 1 ) (cid:54) =

Original: sign ( y 2 ) , and if p ( e 1 ) < p ( e 2 ) , we have sign ( y 1 )
Revised: sign ( y 2 ) , and if p ( e 1 ) < p ( e 2 ) , we have sign ( y 1 )

Original: =
Revised: =

Original: sign ( y 2 ) .
Revised: sign ( y 2 ) .

Original: The same holds for the pairs ( y 3 , y 4 ), and so on.
Revised: The same holds for the pairs ( y 3 , y 4 ), and so on.

Original: c) We ﬁnally perform the same where the RBS are shifted by one position below.
Revised: c) We ﬁnally perform the same where the RBS are shifted by one position below.

Original: Then we compare the signs of the pairs ( y 2 , y 3 ), ( y 4 , y 5 ) and so on.
Revised: Then we compare the signs of the pairs ( y 2 , y 3 ), ( y 4 , y 5 ) and so on.

Original: At the end, we are able to recover each value y j with its sign, assuming that y 1 > 0 for instance.
Revised: At the end, we are able to recover each value y j with its sign, assuming that y 1 > 0 for instance.

Original: This procedure has the beneﬁt of not adding depth to the original circuit, but requires 3 times more runs.
Revised: This procedure has the beneﬁt of not adding depth to the original circuit, but requires 3 times more runs.

Original: The overall cost of the tomography procedure with sign retrieval is given by (cid:101) O ( n/δ 2 ) .
Revised: The overall cost of the tomography procedure with sign retrieval is given by (cid:101) O ( n/δ 2 ) .

Original: In Fig.13 we propose another method to obtain the values of the amplitudes and their signs.
Revised: In Fig.13 we propose another method to obtain the values of the amplitudes and their signs.

Original: Compared to the above procedure, it relies on one circuit only, but requires an extra qubit and a depth of 3 n + O (1) instead of 2 n + O (1) .
Revised: Compared to the above procedure, it relies on one circuit only, but requires an extra qubit and a depth of 3 n + O (1) instead of 2 n + O (1) .

Original: This circuit initializes the qubits in ( | 0 (cid:105) + | 1 (cid:105) ) | 0 (cid:105) , where the last | 0 (cid:105) correspond to the n qubits that will be processed by the pyramidal circuit and the loaders.
Revised: This circuit initializes the qubits in ( | 0 (cid:105) + | 1 (cid:105) ) | 0 (cid:105) , where the last | 0 (cid:105) correspond to the n qubits that will be processed by the pyramidal circuit and the loaders.

Original: Next, applying the data loader for the normalized input vector x
Revised: Next, applying the data loader for the normalized input vector x

Original: (see Section 2.3) and the pyramidal circuit will, according to Eq.(4), map the state to
Revised: (see Section 2.3) and the pyramidal circuit will, according to Eq.(4), map the state to

Original: [Equation]
Revised: [Equation]

Original: Then, we use an additional data loader for the uniform norm-1 vector ( 1 √ n , · · · , 1 √ n ) .
Revised: Then, we use an additional data loader for the uniform norm-1 vector ( 1 √ n , · · · , 1 √ n ) .

Original: Note that this loader is simply built in reverse order to ﬁt the pyramid and limit the augmentation of the depth.
Revised: Note that this loader is simply built in reverse order to ﬁt the pyramid and limit the augmentation of the depth.

Original: We also apply the adjoint of this loader after a controlled operation on the ﬁrst extra qubit.
Revised: We also apply the adjoint of this loader after a controlled operation on the ﬁrst extra qubit.

Original: Recall that if a circuit U is followed by U † , it is equivalent to the identity.
Revised: Recall that if a circuit U is followed by U † , it is equivalent to the identity.

Original: Therefore, this will load the uniform state only when the ﬁrst qubit is in state | 1 (cid:105) :
Revised: Therefore, this will load the uniform state only when the ﬁrst qubit is in state | 1 (cid:105) :

Original: [Equation]
Revised: [Equation]

Original: Finally, a Hadamard gate will mix both parts of the amplitudes on the extra qubit to give us the desired state:
Revised: Finally, a Hadamard gate will mix both parts of the amplitudes on the extra qubit to give us the desired state:

Original: [Equation]
Revised: [Equation]

Original: On this ﬁnal state, we can see that the difference in the probabilities of measuring the extra qubit in state 0 or 1 and rest in the unary state e j is given by Pr[0 , e j ] − Pr[1 , e j ] = 14 (cid:16)
Revised: On this ﬁnal state, we can see that the difference in the probabilities of measuring the extra qubit in state 0 or 1 and rest in the unary state e j is given by Pr[0 , e j ] − Pr[1 , e j ] = 14 (cid:16)

Original: W j x + 1 √ n (cid:17) 2 − 14 (cid:16) W j x − 1 √ n (cid:17) 2 =
Revised: W j x + 1 √ n (cid:17) 2 − 14 (cid:16) W j x − 1 √ n (cid:17) 2 =

Original: W j x/ √ n .
Revised: W j x/ √ n .

Original: Therefore, for each j , we can deduce the sign of W j x by looking at the most frequent output of the measurement of the ﬁrst qubit.
Revised: Therefore, for each j , we can deduce the sign of W j x by looking at the most frequent output of the measurement of the ﬁrst qubit.

Original: To deduce as well the value of W j x , we simply use Pr[0 , e j ] or Pr[1 , e j ] depending on the sign found before.
Revised: To deduce as well the value of W j x , we simply use Pr[0 , e j ] or Pr[1 , e j ] depending on the sign found before.

Original: For instance, if W j x >we have W j x = 2 (cid:112) Pr[0 , e j ] − 1 √ n .
Revised: For instance, if W j x >we have W j x = 2 (cid:112) Pr[0 , e j ] − 1 √ n .

Original: Combining with the (cid:96) ∞ tomography and the non linearity, the overall cost of this tomography is given by (cid:101) O ( n/δ 2 ) as well.
Revised: Combining with the (cid:96) ∞ tomography and the non linearity, the overall cost of this tomography is given by (cid:101) O ( n/δ 2 ) as well.

Original: [Figure]
Revised: [Figure]

Original: previous sections, we have seen how to implement a quantum circuit to perform the evolution of one orthogonal layer.
Revised: previous sections, we have seen how to implement a quantum circuit to perform the evolution of one orthogonal layer.

Original: In classical deep learning, such layers are stacked to gain in expressivity and accuracy.
Revised: In classical deep learning, such layers are stacked to gain in expressivity and accuracy.

Original: Between each layer, a non-linear function is applied to the resulting vector.
Revised: Between each layer, a non-linear function is applied to the resulting vector.

Original: The presence of these non-linearities is key in the ability of the neural network to learn any function Leshno et al.
Revised: The presence of these non-linearities is key in the ability of the neural network to learn any function Leshno et al.

Original: The beneﬁt of using our quantum pyramidal circuit is the ability to simply concatenate them to mimic a multi layer neural network.
Revised: The beneﬁt of using our quantum pyramidal circuit is the ability to simply concatenate them to mimic a multi layer neural network.

Original: After each layer, a tomography of the output state | z (cid:105) is performed to retrieve each component, corresponding to its quantum amplitudes (see Section A.4.1).
Revised: After each layer, a tomography of the output state | z (cid:105) is performed to retrieve each component, corresponding to its quantum amplitudes (see Section A.4.1).

Original: A non linear function σ is then applied classically to obtain a = σ ( z ) .
Revised: A non linear function σ is then applied classically to obtain a = σ ( z ) .

Original: The next layer starts with a new unary data loader (See Section 2.3).
Revised: The next layer starts with a new unary data loader (See Section 2.3).

Original: This hybrid scheme allows as well to keep the depth of the quantum circuits reasonable for NISQ devices, by applying the neural network layer by layer.
Revised: This hybrid scheme allows as well to keep the depth of the quantum circuits reasonable for NISQ devices, by applying the neural network layer by layer.

Original: Note that the quantum circuits proposed in this work can rightfully be called ”quantum neural networks” even though this term is usually employed for any arbitrary variational circuit for their closeness to neural networks.
Revised: Note that the quantum circuits proposed in this work can rightfully be called ”quantum neural networks” even though this term is usually employed for any arbitrary variational circuit for their closeness to neural networks.

Original: With our quantum pyramidal circuits, we control and understand the quantum mapping.
Revised: With our quantum pyramidal circuits, we control and understand the quantum mapping.

Original: It implements each layer and its non linearities, in a modular way.
Revised: It implements each layer and its non linearities, in a modular way.

Original: Our orthogonal quantum neural networks are also different regarding the training strategies which are closer to the classical ones (see Section 4 for details).
Revised: Our orthogonal quantum neural networks are also different regarding the training strategies which are closer to the classical ones (see Section 4 for details).

Original: That being said, it is interesting to compare our pyramidal circuit to a variational circuit with n qubits and n ( n − 1) / 2 gates of any type, as we usually see in the literature.
Revised: That being said, it is interesting to compare our pyramidal circuit to a variational circuit with n qubits and n ( n − 1) / 2 gates of any type, as we usually see in the literature.

Original: Using such circuits we would explore among all possible 2 n × 2 n matrices instead of n × n classical orthogonal matrices, but so far there’s no theoretical ground to explain why this should provide an advantage.
Revised: Using such circuits we would explore among all possible 2 n × 2 n matrices instead of n × n classical orthogonal matrices, but so far there’s no theoretical ground to explain why this should provide an advantage.

Original: Therefore, as an open outlook, one could imagine incorporating additional entangling gates after each pyramid layer (composed, for instance, of CNOT or CZ ).
Revised: Therefore, as an open outlook, one could imagine incorporating additional entangling gates after each pyramid layer (composed, for instance, of CNOT or CZ ).

Original: This would mark a step out of the unary basis but could effectively allow exploring more interactions in the Hilbert Space.
Revised: This would mark a step out of the unary basis but could effectively allow exploring more interactions in the Hilbert Space.

Original: A.4. A DDITIONAL N UMERICAL S IMULATIONS
Revised: A.4. A DDITIONAL N UMERICAL S IMULATIONS

Original: To complete the results reported in Section
Revised: To complete the results reported in Section

Original: [Figure]
Revised: [Figure]

Original: we provide additional numerical experiments testing our classical pyramidal circuit for orthogonal neural networks on small use cases, in Fig 15 below.
Revised: we provide additional numerical experiments testing our classical pyramidal circuit for orthogonal neural networks on small use cases, in Fig 15 below.

Original: [Figure]
Revised: [Figure]