M ongolian Questions Classification Based on Mulit-Head Attention (2020)

Abstract: Question classification is a crucial subtask in question answering system. Mongolian is a kind of few resource language. It lacks public labeled corpus. And the complex morphological structure of Mongolian vocabulary makes the data-sparse problem. This paper proposes a classification model, which combines the Bi-LSTM model with the Multi-Head Attention mechanism. The Multi-Head Attention mechanism extracts relevant information from different dimensions and representation subspace. According to the characteristics of Mongolian word-formation, this paper introduces Mongolian morphemes representation in the embedding layer. Morpheme vector focuses on the semantics of the Mongolian word. In this paper, character vector and morpheme vector are concatenated to get word vector, which sends to the Bi-LSTM getting context representation. Finally, the Multi-Head Attention obtains global information for classification. The model experimented on the Mongolian corpus. Experimental results show that our proposed model significantly outperforms baseline systems.

## introduction when people read a specific sentence on a flyer or some magazine, they can understand the context or intent of the sentence. and they can also extract information from the sentence. how to make a computer think like a human. natural language processing (nlp) and natural language understanding (nlu) study how to make the computer understand the semantics of natural language. the computer uses natural language to communicate with people to realize human-machine interaction. deep learning models have achieved state-of-the-art performance in various natural language processing tasks such as text summarization @xcite , question answering @xcite and machine translation @xcite . in recent years, question answering is a key technology in intelligent applications. it has aroused widespread concern. pipeline the first task of question system is to classify the domain of the dialogue after the user enters the message (text or voice). question classification divides questions into several semantic categories. the machine gets a predicted category of the dialogue and the system returns a concise and accurate answer. the understanding of questions provides constraints for improving the accuracy of question answering system. @xcite have studied the influence of each part of the question answering system on the system performance. the question classification recognition has the greatest influence on the system performance. therefore, to get a good question answering system, it is necessary to design a high accuracy model of question classification. however, the research of the mongolian questions classification is very fewer. the reason is that mongolian corpus is scarce and there is no public mongolian corpus. data collected from internet are noisy and uncertain in terms of coding and spelling. the word-formation is different from chinese and english. it consists of roots, stems and affixes. these problems result in unlimited vocabulary. the existing short text classification methods are not effective. how to classify the questions accurately is a complicated problem. in this article, the training data were crawled from the mongolian web sites. after cleaning the invalid data, we constructed a question classification data set. we propose a method of the mongolian question classification, which combines the bi-lstm model with the multi-head attention mechanism. as shown in figure 1 , the model is named ma-b. to better learn semantic information from sentences, we introduce the morphemes representation. the character vector and the morpheme vector are concatenated to get word vector. it sends to bi-lstm getting context representation. the multi-head attention mechanism extracts relevant information from different dimensions. in the classification layer, we use the softmax classifier to output the probability of each category. the paper is organized as follows: section 2 gives the related work. section 3 presents the question classification method in detail. section 4 shows the experiments and results. section 5 summarizes the full text and give some future works. ## related work question classification is a kind of short text classification (alsmadi and gan, 2019). there have been many studies on questions classification. chinese and english, which are rich in resources, have achieved good results. the traditional method was based feature engineering such as bag of words (bow) and n-gram. both were combined with term frequency-inverse document frequency (tf-idf) and other element features as text features. however, these methods ignore the context semantic information. there were some methods based machine learning, including nearest neighbors (nn) @xcite , naive bayes @xcite , and support vector machine (svm) @xcite . in @xcite , the authors utilized the external knowledge base for text classification. in recent years, researchers have tried to extract semantic information from sentences via deep learning. the combination of textcnn @xcite , textrnn @xcite , lstm @xcite , textgcn @xcite , with word embedding has been widely used in text classification. there are some researches on rare resource languages to classify questions. for example, uyghur is also a few resource language and have complex word-formation. @xcite proposed a method of uyghur short text classification based reliable sub-word morphology. mongolian language processing has been further developed, such as morphological segmentation @xcite , spelling correction @xcite , named entity recognition @xcite . the mongolian question classification needs to be solved urgently. , pages 1026-1034 , hainan , china , october 30 -novermber 1 , 2020. ( latin: kqmpani-y'in havli-y'in homun ebedcileged genedte nasv barajai , tegun-u" ori ogcege-y'i hen egurgelehu boged bvcagahv yqsqtai means: who should bear and return the debts of the company due to the sudden death of the legal person? category: company law ᠡᠭᠦᠷᠭᠡᠯᠡᠬᠦ ᠪᠥᠭᠡᠳ ᠪᠤᠴᠠᠭᠠᠬᠤ ᠶᠣᠰᠣᠲᠠᠢ ᠨᠠᠰᠤ ᠪᠠᠷᠠᠵᠠᠢ ᠂ ᠲᠡᠭᠦᠨ ᠦ ᠥᠷᠢ ᠥᠭᠴᠡᠭᠡ ᠶ᠋ ᠢ ᠬᠡᠨ ᠺᠣᠮᠫᠠᠨᠢ ᠶ᠋ ᠢᠨ ᠬᠠᠤᠯᠢ ᠶ᠋ᠢᠨ ᠬᠥᠮᠦᠨ ᠡᠪᠡᠳᠴᠢᠯᠡᠭᠡᠳ ᠭᠡᠨᠡᠳᠲᠡ mongolian: figure 3: example of traditional mongolian script, latin transliterature, category tag and their meanings. ## model architecture in this section, we will introduce this model from bottom to up. the mulit-head attention mechanism can fully capture the long-distance text features. but it is difficult to deal with the sequence information. the recurrent neural network can effectively obtain the context order information of sequences. it can effectively supplement the mulit-head attention mechanism. as depicted in figure 1 , ma-b model is proposed by combining bi-lstm network with mulit-head attention mechanism. ## experiments our model is trained on the selected data set. by evaluating the classification results and comparing with baseline, we can evaluate the questions classification performance of the model. ## conclusion in this paper, bi-lstm and mulit-head attention mechanism are used to model mongolian corpus texts. by combining the ability of multi head attention to obtain global information with the ability of bi-lstm to obtain local sequence information, a better effect has been achieved. at the same time, in order to make the model better learn the text semantic information, mongolian morphemes representation are further introduced. however, there is a lot of room for improvement in the field of mongolian questions classification. from the experiment, it can be seen that the introduction of pre training morphemes features has a good effect. in the future, feature engineering can be further reduced by using pretraining language models.