Interactive M ongolian Question Answer Matching Model Based on Attention Mechanism in the Law Domain (2022)

Abstract: “Mongolian question answer matching task is challenging, since Mongolian is a kind of lowresource language and its complex morphological structures lead to data sparsity. In this work, we propose an Interactive Mongolian Question Answer Matching Model (IMQAMM) based on attention mechanism for Mongolian question answering system. The key parts of the model are interactive information enhancement and max-mean pooling matching. Interactive information enhancement contains sequence enhancement and multi-cast attention. Sequence enhancement aims to provide a subsequent encoder with an enhanced sequence representation, and multi-cast attention is designed to generate scalar features through multiple attention mechanisms. MaxMean pooling matching is to obtain the matching vectors for aggregation. Moreover, we introduce Mongolian morpheme representation to better learn the semantic feature. The model experimented on the Mongolian corpus, which contains question-answer pairs of various categories in the law domain. Experimental results demonstrate that our proposed Mongolian question answer matching model significantly outperforms baseline models.”

## introduction question answer matching is used to identify the relationship between the question-answer pairs, and it is one of the application scenarios of text matching. text matching is an important fundamental technology in natural language processing (nlp) and can be applied to a large number of nlp tasks, such as information retrieval (ir), natural language inference (nli), question answering (qa) system, dialogue system, etc. for the tasks of information retrieval, text matching is utilized to compute the relevance between queries and documents to select the relevant documents @xcite . for the tasks of natural language inference, text matching is employed to judge whether the premise can infer the hypothesis @xcite . and for the question answering tasks, text matching is applied to pick the answers that are most relevant to a given question @xcite . with the development of deep learning, text matching methods with neural network are increasingly emerging. these methods can be divided into two types-representation-based match and interactionbased match. the first type is representation-based match @xcite @xcite , which is focused on modeling the representations of the two sentences, so that they are encoded into semantic vectors in the same embedding space. the second type is interaction-based match @xcite @xcite , which is targeted at interacting with each information between sentence pairs to improve the process of representation learning. interaction-based match performs better than representation-based match, because representation-based match lacks a comparison of lexical and syntactic information between sentence pairs, while interaction-based match can take advantage of the interactive information across sentence pairs to enhance their own representations. therefore, interactive matching methods are currently the mainstreaming methods of text matching. however, the development of mongolian question answering system is relatively slow, and there are few studies about it. the first reason for the slow development is that mongolian is a kind of lowresource language. it lacks public labeled corpus. the second reason is the data-sparse problem caused by complex mongolian morphological structures. mongolian is an agglutinative language and its root can be followed by different suffixes, which is different from chinese and english. in this paper, we construct a mongolian question answering data set in the law domain. an example is shown in figure 1 , there are two mongolian sentences, including the question and its corresponding answer. our task is to judge whether the answer corresponds to the question. in order to solve the problem of insufficient information enhancement of previous interaction-based matching methods, we propose an interactive mongolian question answer matching model (imqamm), which combines interactive information enhancement and max-mean pooling matching. interactive information enhancement concatenates a series of feature vectors to get the enhanced sequence representation and adopts a compression function to reduce feature vectors to scalars based on multiple attention mechanisms for the issue of data-sparse. max-mean pooling matching is to compute the maximum and average cosine similarities corresponding to each morpheme representation. the paper is organized as follows: section 2 introduces the related work. section 3 presents the details of our mongolian question answer matching model. section 4 shows our experimental setup and results. section 5 gives the conclusion and the future work. ## related work text semantic matching is the core of retrieval question answering system. in recent years, there has been a lot of research on text semantic matching, which has driven the development of question answering systems. early work on text semantic matching was based on relatively simple models to compare the sentence pairs. most of these work was based on sentence encoding methods. @xcite proposed deep structured semantic models (dssm), which projected queries and documents into a common low-dimensional space to get similarities between them. since the cnn has the characteristics of local perception and parameter sharing, @xcite improved the dssm model and proposed the convolutional latent semantic model (clsm). to tackle the limitations of rnn in capturing contextual dependencies, @xcite presented the long short term memory dssm (lstm-dssm). @xcite proposed the manhattan lstm (malstm) model, which utilized the siamese lstm to compute the manhattan distance between sentence pairs. @xcite proposed directional self-attention network (disan) without any rnn or cnn for sentence encoding. @xcite proposed a siamese sentence encoder that would not propagate any interactive information between sentence pairs.the matching methods based on attention networks have gradually developed @xcite @xcite . @xcite used the attentive lstm to match questions and passage answers with question information. @xcite explored computing the attention matrix before and after convolution to select correct answers for a question. @xcite used densely-connected recurrent neural network and concatenated the co-attentive features in multiple layers for semantic sentence matching. compare-aggregate networks are very popular in different tasks @xcite @xcite @xcite . @xcite presented a compareaggregate model that performed word-level matching by element-wise multiplication or subtraction and aggregated by cnn. wang et al. (2017) proposed a bilateral multi-perspective matching (bimpm) model, which used multi-perspective cosine matching strategy between encoded sentence pairs. chen et al. (2017) improved the approach proposed by parikh et al. (2016) and achieved sequential inference model using chain lstms. tay et al. (2017) presented comprop alignment-factorized encoders (cafe) that used factorization machines to compress the alignment vectors into scalar features, which can effectively augment the word representations. tay et al. (2018) explored using multi-cast attention networks (mcan) to improve learning process by adopting several attention variants and performing multiple comparison operators. these text semantic matching models laid the foundation for later ir models and qa systems. although these models have achieved state-of-the-art performance on various datasets, they may not be suitable for low-resource agglutinative languages. in this paper, we introduce mongolian morpheme representation, then use interactive information enhancement to take full advantage of the information across mongolian question-answer pairs and apply max-mean pooling matching to capture the maximum influence and the overall influence between mongolian question-answer pairs. ## model architecture in this section, we will describe our model architecture layer by layer. figure 2 shows a high-level view of the architecture, and then the details of our model are given as follows. ## conclusion in this paper, we propose an interactive mongolian question answer matching model (imqamm), which mainly combines interactive information enhancement and max-mean pooling matching. first of all, we make some transformations to traditional mongolian language and introduce the morpheme vectors. second, we enhance the sequence representation by concatenating a series of feature vectors. third, the multi-cast attention is introduced to alleviate the data-sparse problem caused by complex mongolian morphological structures. finally, the max-mean pooling matching strategy is applied to match question-answer pairs in two directions. experimental results show that our model performed well on the mongolian question answering data set. however, there is still a lot of room for improvement. in the future work, we will consider using the pre-trained language model bert to get a better initialization, which may help improve the performance of our model.