Bi-Quantum Long Short-Term Memory for Part-of-Speech Tagging (2023)

Abstract: Natural language processing (NLP) is a subfield of artificial intelligence that enables computer systems to understand and generate human language. NLP tasks involved machine learning and deep learning methods for processing the data. Traditional applications utilize massive datasets and resources to perform NLP applications, which is challenging for classical systems. On the other hand, Quantum computing has emerged as a promising technology with the potential to address certain computational problems more efficiently than classical computing in specific domains. In recent years, researchers have started exploring the application of quantum computing techniques to NLP tasks. In this paper, we propose a quantum-based deep learning model, Bi-Quantum long short-term memory (BiQLSTM). We apply POS tagging using the proposed model on social media code-mixed datasets.

## introduction natural language processing (nlp) @xcite is a promising artificial intelligence (russell and j, 2010) area that focuses on the interaction between human languages and computer systems. its primary aim is to enable computer systems to understand, process, and generate human languages meaningfully. the extensive use of nlp applications makes nlp ubiquitous. from virtual assistants and chatbots to language translation and sentiment analysis, nlp consistently enhances communication, decision-making, and convenience of everyday interactions. nlp tasks involve computational linguistic and machine learning models to process and analyze text information. machine learning @xcite is a subfield of artificial intelligence that focuses on developing algorithms and models to learn the relation between data and patterns from a dataset. it enables computer systems to make decisions, predictions, and recommendations by recognizing patterns and relationships between data. neural network @xcite is a part of machine learning algorithms whose computational methodology is inspired by the working principle of the human brain. it is composed by combining a linear sequence of neurons. in the last decade, machine learning has significantly appeared in advanced learning algorithms, which show remarkable data processing performance. the performance of machine learning algorithms is tremendous in various domains, especially in the nlp domain. the higher performance of nlp applications needs vast datasets and complex algorithms for training. however, the increasing data size requires extensive resources for training the traditional algorithms, leading to massive time and resource consumption. on the other hand, quantum computers can solve these challenges classical systems face. due to their unique characteristics, superposition and entanglement can speed up computational time and decrease the demand for extensive resources. we can take leverage of quantum computing speedup by combining it with classical machine learning, known as quantum machine learning (qml) @xcite . the main goal of qml algorithms is to apply quantum computing's distinct attributes to build machine learning algorithms that can surpass their classical counterparts in capability and efficiency. qml offers various superiority, including speedup runtime, increased learning efficiency by employing fewer data for training and finding complex patterns, and enhanced learning capability by improving the content-addressable memory (phillipson, 2020). these advancements in qml make it applicable to domains like nlp. in this paper, we proposed a quantum-inspired deep neural network model bidirectional quantum long short-term memory (biqlstm). we also apply the proposed system for nlp application, partof-speech (pos) tagging on code-mixed datasets. pos tagging is a fundamental task that automatically assigns parts of speech in given sentences. pos tagging helps to analyze the syntactic and semantic structure of sentences. most pos tagging techniques apply machine learning algorithms that automatically predict and assign the actual parts of speech of words according to their context within a sentence. the paper's structure is as follows: section 2 explains the background of quantum computing and qml. section 3 presents the related work about qml algorithms for nlp applications. the proposed qrnn model is presented in section 4. the description of the dataset is given in section 5. section 6 shows the experiment and results. this paper concludes with a conclusion in section 7. ## related work in this section, we discuss the related work for qml algorithms and their applications in the nlp field. qml algorithms include quantum-inspired traditional machine learning and deep learning models. the primary goal of using qml is to enhance the performance of classical machine learning algorithms. riordan et al. (o'riordan et al., 2020) proposed a hybrid quantum-classical model, which can encode, interpret, and decode the meaning of sentences within the quantum circuit. the authors have shown how one can interpret the meanings of sentences and find similarities among various sentences' meanings. however, this model has limitations, such as when the data size is increased, the depth of the circuit is increased. so, the proposed model is well-suitable for the small and straightforward corpus. shi et al. @xcite introduced a quantumenhanced neural network for text classification for binary labels. the authors applied quantuminspired complex-valued word embedding (icwe). generally, words have different meanings and equivalents, similar to a quantum system, which can hold several values. the authors have shown the effectiveness and feasibility of the applicability of the proposed methods for nlp tasks and can provide a solution for text information loss. quantum neural network is another research area of qml that may be applicable for nlp tasks such as pos tagging. sipio et al. @xcite introduce quantum-based long short-term memory (qlstm) for nlp applications such as pos tagging. this model is a demonstrated model, which shows that quantum-based neural network models apply to nlp tasks. they have performed pos tagging only on two sentences. they offer a comparison between qlstm and classical lstm results. this model has some limitations because it can not apply to large datasets. pandey et al. @xcite used the qlstm @xcite model to perform pos tagging on the mizo dataset, where mizo is a low-resource language of india. the author experimented on a mizo dataset with 47 pos tags and 30000 words. however, the result of their experiment could be better. the authors have tried with different numbers of qubits and other local simulators. they have investigated that current quantum devices do not apply to large datasets. pandey et al. @xcite introduced a hybrid quantum-classical qlstm model. the proposed model is applied for pos tagging on social media code-mixed languages. their experiment result shows the accuracy of qlstm is better than classical lstm. they have performed nine experiments on nine different datasets, which contain various indian languages mixed with english languages. quantum natural language processing (qnlp) @xcite is another framework that utilizes properties of quantum mechanics for nlp applications. qnlp focuses on categorical quantum mechanics that represent the syntactic structure of sentences into states of quantum systems and grammar into entanglement. lambeq @xcite is an open-source software platform of qnlp, which is used to process several nlp applications. ## biqlstm recurrent neural network (rnn) (hibat-allah et al., 2020) is a neural network variant for handling sequential data. however, the performance of the rnn model is not satisfactory for prolonged dependencies in text data. the rnn model suffers from a problem known as a vanishing gradient problem when data dependencies increase. therefore, another rnn variant is helpful to handle prolonged dependencies and long short-term memory (lstm) @xcite . lstm is well known for managing longer dependencies in text data. it can resolve the vanishing gradient problem and is widely used in nlp applications. lstm utilizes memory cells for storing extended-period information. the working principle of the memory cell depends on gate units, input gate, forget gate, output gate, and update gate. the work of the input gate is to recognize new information, which will be accepted by the memory cell, and the forget gate selects information that will be discarded. after receiving wanted data and rejecting unwanted data, the update gate updates the new information. at the last, the output gate controls information stored in the memory cell, which will generate the output. working these gates enables lstm to select helpful information and forget useless data to handle prolonged dependencies in the dataset. the below equations show the flow of information in lstm. where σ is the sigmoid function, i t represents the workflow of the input gate, f t represents the workflow of the forget gate, o t represents the workflow of the output gate, and s t represents the workflow of the update gate. w i , w f , w o , and w s denotes the parameters of corresponding gate. [h t-1 , x t ] is combination of hidden state and input data. b n is corresponding bias of w n , where n is type of gate. lstm is built by stacking hidden cells and memory cells. handling code-mixed language is complicated because mixed language contains different syntax. therefore, bidirectional long short-term memory (bilstm) networks are often considered more powerful and versatile than traditional lstm networks due to their ability to capture contextual information in both forward and backward directions. we propose a quantum-based bilstm (biql-stm) to implement nlp applications. the proposed model is an extended version of qlstm @xcite . here, we convert the classical layer of bilstm into quantum circuits to enhance the performance. after converting each layer of bilstm into the quantum circuit, we stack each layer to build the proposed systems. in this model, stochastic gradient descent @xcite is used as an optimizer that is best suitable for large datasets, and cross-entropy is used as a loss function. pos tagging is a multilabel classification @xcite , so we use cross-entry as a function. ## database description in our experiment, we use social media code-mixed datasets @xcite . code-mixed languages are typically involved in mixing multiple languages and are commonly used in multivariate societies such as india. processing such languages algorithm 1 propsed algorithm for biqlstm input: x t -input at time t input: h t-1 -previous cell output at time t -1 input: c t-1 -previous cell state at time t -1 input: w f , w i , w o , w c -weight matrices for forget, input, output, and cell gates procedure encode(x) for i to n qubit do hadamard(i) ry (x i) end for end procedure procedure vqc(x, α) for i to n qubit do cn ot (i, i + 1) ry (α i) end for end procedure procedure biqlstm(input,n qubit, em-bedding dim,hidden layer) in our experiments, we use the pennylane platform, a local simulator. pennylane is an opensource software framework for quantum computing and qml. it is developed for work and experiments with other quantum computing platforms. we use four qubits in this experiment. we chose four qubits because we want fewer qubits possible for nlp applications in quantum experiments. the number of epochs is 300. we observe that for more than 300 epochs, the model is overfitted. the accuracy of both experiments is given in ## conclusion nlp is a promising area of artificial intelligence that makes computer systems understand human languages. nlp applications involve machine learning and deep learning models to process text data. however, the recent machine learning model consumes more resources and demands a vast dataset. on the other hand, qml enhances the performance of nlp tasks. in this paper, we proposed a quantum-based biqlstm to perform pos tagging on code-mixed languages. this proposed model shows that a quantum-based deep learning model can handle nlp applications.