Quantum Recurrent Architectures for Text Classification (2024)

Abstract: We develop quantum RNNs with cells based on Parametrised Quantum Circuits (PQCs). PQCs can provide a form of hybrid quantum-classical computation where the input and the output is in the form of classical data. The previous “hidden” state is the quantum state from the previous time-step, and an angle encoding is used to define a (non-linear) mapping from a classical word embedding into the quantum Hilbert space. Measurements of the quantum state provide classical statistics which are used for classification. We report results which are competitive with various RNN baselines on the Rotten Tomatoes dataset, as well as emulator results which demonstrate the feasibility of running such models on quantum hardware.

## introduction recurrent neural networks (rnns) were transformative in the early stages of neural nlp @xcite , and still offer competitive performance over more recent architectures such as transformers @xcite . now that quantum computing is also emerging as a potentially transformative technology @xcite , it is natural to consider quantum versions of nlp models, such as rnns, and ask whether they exhibit any advantages over their classical counterparts. here we develop quantum rnns with cells based on parametrised quantum circuits (pqcs). pqcs can be used to provide a form of hybrid quantumclassical computation where the input and the output is in the form of classical data, and a set of parameters which controls the pqc's computation is classically optimised @xcite . the excitement around quantum computing comes from the expectation that it will enable us to solve problems, or run models, efficiently which cannot be run efficiently on a classical computer @xcite . the rapidly advancing state of quantum hardware means that we may soon cross the "quantum advantage" threshold @xcite . moreover, there are good reasons for exploring quantum models even when classically simulating them. since pqcs implement unitary (norm-preserving) transformations, they are well-suited to modelling longrange dependencies and mitigating the problem of vanishing and exploding gradients @xcite . we propose quantum rnns where the previous "hidden" state is the quantum state from the previous time-step, and the state is updated using one of two methods. in the first method, the current word is encoded as a quantum state using an angle encoding, and entangled with the previous state. in the second method, the word embedding is also transformed into a set of angles, but applied directly to a pqc which simply updates the previous state. measurements are applied at the end of the sequence, and define a mapping from the quantum state to classical statistics which can be used for classification. our models are hybrid in that words are first encoded as classical embeddings, and then mapped (non-linearly) into a quantum hilbert space, via the angles of the pqc. quantum phenomenon such as entanglement can then be exploited to manipulate the quantum state representations. here, we train the models in simulation, exploiting the fact that quantum computation can be simulated using linear algebra, and hence is amenable to optimisation with standard loss functions and backpropagation. we show that the quantum recurrent architectures can perform as well on a realistic nlp task as the classical rnn, gru and lstm baselines. our results on the rotten tomatoes dataset are obtained with only 4 qubits, corresponding to a 16-d (2 4 ) complex hilbert space, compared to the 200-d hidden states of the classical models. since the models can be run in simulation, our main motivation is to begin exploring quantum nlp models in anticipation of more powerful quantum hardware in the future, as well as to provide a comparison with existing models in the classical case. however, in order to demonstrate the feasibility of running such models on quantum hardware even today, we also run the models at test time on a quantum computer emulator, obtaining results close to those of the exact, noise-free simulation. ## quantum circuits a quantum circuit consists of a sequence of quantum gates operating on qubits @xcite . qubits are quantum units of information which, unlike classical bits, can be in a superposition of 0 and 1. mathematically the state |ψ⟩ of a qubit is represented as a vector in a 2-d complex hilbert space: the information in qubits is carried by horizontal wires in the diagrammatic circuit notation, and multiple qubits are represented by stacking wires vertically (see fig. 1 ). for a circuit with 2 qubits, measuring each qubit results in one of 4 possible outcomes: |00⟩, |01⟩, |10⟩, or |11⟩. hence a quantum state |ψ⟩ from 2 qubits is represented by a vector in a 4-d complex hilbert space: unlike x, many other gates are parametric and contain parameters which can be learned via an external optimisation procedure. an example is the rx rotation gate with the unitary matrix: with θ the degree of rotation about the x-axis (for the bloch sphere representation of single-qubits). foot_0 quantum circuits are typically initialised with a simple, easy-to-prepare state such as the all-zero state (see fig. 1 ). this state is a product state, meaning that the measurement outcomes on each wire are independent. gates operating on more than one wire allow the creation of correlations between measurement outcomes, i.e. the creation of entangled states. entanglement is necessary to fully exploit the exponentially-sized hilbert space, and can be achieved through the use of controlled gates, consisting of a control qubit and a target qubit. a standard example is a controlled x gate (cx), which acts as the identity on the computational basis states |00⟩ and |01⟩, but flips the second qubit when acting on |10⟩ and |11⟩: the pqc we use in fig. 1 has examples of controlled rotation gates (rx and ry), where if the control is |0⟩ then nothing happens to the target, and if the control is |1⟩ then the rotation is applied; and for a state in superposition, the gates act linearly on the basis states. a quantum state can be measured using a specific basis, such as the pauli-z measurement in the computational basis, yielding a scalar value (e.g. for a single qubit, +1 can be associated with the outcome |0⟩ and -1 with |1⟩). on actual quantum hardware, because the measurement outcome is probabilistic, the circuit is run many times and an average is taken (where each run is referred to as a shot), giving an output in [-1, 1]. this output can then form part of a classical ml objective function. ## quantum recurrent models the first model performs recurrent computation as shown in fig. 1a . at each time-step, the word is encoded using a word ansatz w, foot_2 using two of the wires, always starting with the all-zero state. more concretely, each word is represented as a classical word embedding w @xmath0, which gets mapped via an affine transformation f into the parametrised angles required by the rotation gates in w: with θ @xmath1. this produces a state on the two wires, which is combined with the previous state using a tensor product, resulting in a product state. all 4 wires are then put through another ansatz r, which also has a set of trainable and parametrised angles, and produces an entangled state. the nature of a unitary transformation means that the number of wires going into each ansatz has to be the same as the number coming out, so 2 wires are "ignored", or "discarded", shown with the ground symbol. the second model is simpler, and results from dispensing with w (fig. 1b ). since there is no separate input state for each word, there is no need for discarding. the ansatz at each time-step still has its parameters supplied by a classical transformation of a word embedding. measurements are taken of the final state and post-processed to give a classical output. here we measure two qubits and use the scalar values as logits for a softmax (fig. 1a ). 3 there are two ways in which quantum computation can be simulated on a classical computer. the first is to perform shot-based probabilistic computing, possibly including the modelling of noise from the imperfect nature of current quantum devices. such simulators are often referred to as emulators. the second approach is to perform exact simulation, by calculating the measurement outcomes analytically @xcite . here we use exact, noiseless simulation for training, making use of automatic differentiation tools. at test time we use exact simulation and in addition perform runs on a quantum computer emulator. ## experiments we use torchquantum (wang et al., 2022) for analytical simulation -a pytorch-based toolkit for developing hybrid classical-quantum models with batching and automatic differentiation support. the dataset is rotten tomatoes, a binary classification sentiment analysis dataset. 4 4 as baselines we include results for an rnn, gru and lstm. some modifications were needed for torchquantum, for example the addition of density matrices and some related operations such as the partial trace @xcite to allow discarding. 5 here we provide the most important architectural details; hyperparameters are listed in appendix b. we use 4 qubits for all models and, for the model with discarding, each word is encoded with a sequence of parameterised rx, rz, rx gates on each of the bottom two wires (fig. 1a ), and so θ @xmath2. we use the ansatz shown in fig. (θ @xmath3, taken from sim et al. ( 2019 ) which demonstrates experimentally that this ansatz is expressive, meaning that it has high entangling capability and can represent a large part of the hilbert space. note that for all rotation gates in w and r, each gate has one classical parameter θ associated with it (4). for the model with discarding, we found that using the classical word embedding to provide the parameters for the r as well as the w ansatze, via a separate mapping g, resulted in higher accuracies. hence the models have trainable parameters associated with the following components: word embeddings (randomly initialised); the mapping f from word embeddings to angles for word encoding; the mapping g from word embeddings to angles for recurrent encoding; and the output classification function. for the f and g functions we use a single-layer affine transformation, and for the output function a 2-class softmax. dropout is applied to the input embeddings (before f and g), and the loss function is binary cross-entropy. experiments were run on a single a100/rtx-6000 gpu. appendix. a contains some additional results showing how the accuracy varies across different training runs, and also learning curves for the various models. the qrnn models are less stable than the classical baselines, showing more variation across runs, but convergence is much faster. for the emulator results, we use the cloudbased native h1-1e simulator of the quantinuum h1 quantum processor, including a realistic noise model, accessed through quantinuum nexus. 6 we took two of the best-performing qrnn and qrnn d models and obtained 80.1% and 77.2% test accuracies on the emulator, respectively, with 1,000 shots used for measurements. in comparison, the classical results for these particular models were 79.5% for both. given the highly accurate nature of the emulator, these results are encouraging: they indicate that, even with the nascent state of quantum computing technology today, we already have hardware that can run well-performing nlp models at a reasonable scale. ## related work a notable contribution to the development of qrnns is the work of @xcite , which provides a rigorous theoretical framework for quantum-enhanced sequential learning and highlights the need for quantum hardware to fully realize the potential of qrnns. in this paper, bausch explores the potential for quantum models to outperform classical rnns, particularly in scenarios involving long-range dependencies and complex temporal correlations. bausch's work introduces a novel class of qrnns designed to leverage the key principles of quantum mechanics-such as superposition and entanglement-to process sequential data. another key insight from bausch's work is the demonstration that qrnns can be trained using fewer iterations compared to classical rnns, as we also demonstrate. another important development in the qrnn literature is the quantum long short-term memory (qlstm) model, which extends the classical lstm architecture to the quantum domain. proposed by @xcite , the qlstm model used quantum gates to represent the forget, input, and output gates of classical lstms, allowing a hybrid model while retaining the potential to address the vanishing gradient problem. this work demonstrated that qlstms could outperform classical lstms on certain benchmarks, achieving better accuracy and faster convergence. the hybrid qrnn model of yu et al. ( 2024 ) also explored a text classification task, although only in the low-resource setting. other qrnns have also been proposed @xcite @xcite . li et al. also apply their model to a text categorization task, although our model architectures are different, and use novel ansatze and a discarding operation. the latter two works use different circuits and are particularly relevant for tasks in the continuousvariable domain. nlp has also used quantum models @xcite , often based on the observation that the mathematics of quantum hilbert spaces shares similarities with vector space models of semantics @xcite . one focus has been on compositional distributional models of language @xcite , with a recent demonstration of solving a toy nlp problem on quantum hardware @xcite . @xcite also investigate pqc-based architectures for sequence modelling, including some test runs on actual quantum hardware. various architectures are considered, including some based on syntactic structure. they also report results based on the rotten tomatoes dataset, obtaining their best results using a convolution structure. ## conclusion we have described quantum rnns and simulation experiments, on a realistic nlp task, obtaining results that are competitive with some challenging classical rnn baselines. we believe this is the first paper published at an nlp conference which demonstrates a qnlp model being applied successfully to such a task and the running of such a model on a quantum emulator. one noteworthy feature of the models is the relatively small number of learnable parameters. future work includes scal-ing up the number of qubits, and to larger datasets, with the potential to use anzatze which cannot be efficiently simulated on a classical computer (with the caveat that trainability could be an issue; see the limitations section below). although the main motivation has been to investigate the potential of quantum models in anticipation of the development of (rapidly improving) quantum hardware, the models can also be thought of bringing advantages to existing classical models, such as the naturally bounded gradients from the unitary transformations @xcite .