Intra-Correlation Encoding for C hinese Sentence Intention Matching (2020)

Abstract: Sentence intention matching is vital for natural language understanding. Especially for Chinese sentence intention matching task, due to the ambiguity of Chinese words, semantic missing or semantic confusion are more likely to occur in the encoding process. Although the existing methods have enriched text representation through pre-trained word embedding to solve this problem, due to the particularity of Chinese text, different granularities of pre-trained word embedding will affect the semantic description of a piece of text. In this paper, we propose an effective approach that combines character-granularity and word-granularity features to perform sentence intention matching, and we utilize soft alignment attention to enhance the local information of sentences on the corresponding levels. The proposed method can capture sentence feature information from multiple perspectives and correlation information between different levels of sentences. By evaluating on BQ and LCQMC datasets, our model has achieved remarkable results, and demonstrates better or comparable performance with BERT-based models.

## introduction as a branch of sentence semantic matching (ssm), sentence intention matching (sim) is critical to question answering systems in certain applications. in general, ssm is to judge whether two sentences express the same meaning. however, in a question answering system, sim intends to determine whether two questions share the same intention and could be addressed by the same answer, which is more challenging than other ssm tasks. as an example shown in with the development of deep learning, a series of ssm models are proposed for semantic matching tasks @xcite @xcite @xcite . however, these models simply consider the characteristics of the semantic level of the text but overlook the deep intentional features. researchers have attempted to extract deeper semantic features through attention mechanisms @xcite , memory networks @xcite , as well as the addition of external syntactic structures and lexical datasets as in wordnet @xcite . although the above methods obtain deep semantic features from different perspectives, they cannot completely overcome the feature missing phenomenon in the encoding process. especially due to the diversity of chinese semantic features, the above existing methods cannot better capture complicated deep semantic features. in english ssm tasks, @xcite employ multi-granularity fusion to extract the corresponding richer semantic features, where the more fine-grained character embeddings are employed together with the traditional word embeddings @xcite . although the introduction of character embeddings is beneficial to enrich english text representation, one single english character is hard to express a special meaning. different from english, a chinese character is able to represent a solid meaning, which can convey more semantic features and information. thus, there should be great interest and potential to explore and combine multi-granularity embeddings for chinese ssm tasks. huang et al. @xcite and zhang et al. @xcite achieve better performance by combining character, word, and other granularities to obtain semantic encoding features from chinese text. the multigranularity fusion method can be applied to extract the semantic features of a text sequence, which can effectively alleviate the phenomenon of missing semantic features in the encoding process. in this paper, inspired by the existing work, we push forward this line of research by proposing a better multi-granularity fusion approach to capture semantic features from text sequences. in @xcite and @xcite , the encoding features in multi-perspective granularities are integrated to generate the final text encodings. however, the correlation and distinction between text features on different granularities are not considered, thus the corresponding semantic features are not further explored in these works. sequential inference models based on chain lstms are implemented in @xcite , enabling the capture of more features from different perspectives. they incorporate syntactic parsing information in tree lstm into the classic bilstm model with the help of soft alignment attention. in our work, in order to better extract the correlations between different granularities in sim, inspired by the work in @xcite , we employ soft alignment attention to enhance local information representation between different granularities and capture more sentence correlation. our contributions are summarized as follows: the rest of the paper is structured as follows. we describe our novel sentence intention matching model in section 2. section 3 demonstrates the experimental results. related work is introduced in section 4, followed by conclusions in section 5. embedding q1 character embedding embedding embedding q1 word basic encoding q2 character basic encoding q2 word basic encoding soft alignment q1 character basic encoding q1 word q2 character q2 word soft alignment q1 character alignment q1 word alignment q2 character alignment q2 word alignment q2 features similarity matrix max & average pooling fully-connected layers sigmoid q1&q2 concatenation max & average pooling features concatenation input multi-granularity embedding layer inter-correlation encoding layer global inference composition layer prediction layer q1 features max & average pooling lstm concatenation-1 bilstm concatenation-2 q q basic encoding zoom-in of the basic encoding component: ## model we propose a multi-granularity (character-granularity and word-granularity) neural sentence model, whose architecture is shown in figure 1 . we utilize the siamese network structure in the sim task. our model architecture consists of four parts: a multi-granularity embedding layer, an intra-correlation encoding layer, a global inference composition layer, and a prediction layer. in the following subsections, we first describe the initial representation, including word-granularity and character-granularity embeddings, then introduce the intra-correlation encoding layer in detail. next, we describe the global inference composition layer, which measures the feature information between two sentence representations. finally, the prediction layer is introduced, which predicts whether the corresponding sentences match each other in semantic intention. ## related work sentence intention matching is critical for a series of downstream tasks, such as information retrieval, question answering, and machine translation. with the development of deep neural networks, sophisticated models for ssm task have been evolving rapidly @xcite @xcite . the companion of attention mechanism with sequence models have achieved promising performance in machine translation @xcite and then is applied in many other tasks in natural language processing. it has also rendered encouraging effects in the ssm task @xcite @xcite @xcite . wang et al. propose a multi-angle bidirectional attention mechanism in ssm task, and the effect of the model is remarkable @xcite . in addition, sha et al. put forward an attention mechanism by repeating two sentences to improve the model's memory and obtain better textual semantic representation @xcite . generally speaking, through using attention mechanisms, key feature representations in the text can be captured and benefit precise matching. although the above methods extract key feature representations with the attention mechanisms or introduce external syntactic information in the text sequences, these methods only achieve limited improvement. a number of researchers discover that the granularity of text is also crucial for capturing deep semantic features of the text. in particular, huang et al. propose a word representation layer, which consists of word embedding and character representation, to capture multi-granularity feature representations @xcite . the acquisition and integration of text features at different granularities are considered in @xcite and achieve interesting results. however, their work simply integrates multi-granularity features, without taking into account the correlation of text features between different granularities. the pre-trained language model bert, which solves the polysemy problem of word2vec, has proven to be highly effective @xcite . however, bert is often computationally expensive in many practical scenarios. thus, it is hard to be readily implemented with limited resources. therefore, a series of smaller, faster, cheaper, and lighter of pre-trained bert-based models emerge widely, such as distilbert @xcite and fastbert @xcite . these models are optimized in terms of running speed and resource utilization, which inevitably reduces the effectiveness of the original bert model. how to achieve comparable performance with bert and require less computational resources is critical and urgent in model design for nlp applications. in this paper, in order to better capture correlation features between different granularities, we propose an intra-correlation encoding framework for sim task, which considers the correlation between text features from character-granularity and word-granularity. with less requirement on computational resources, our proposed model can achieve better or comparable performance with the state-of-the-art bert. ## conclusions for sentence intention matching tasks, we propose a novel method, named the intra-correlation encoding model. it combines character-granularity and word-granularity features to model sentence intention, and utilizes soft alignment attention to enhance the local information of sentences on the different levels. it can capture sentence feature information from multiple perspectives and correlation information between different levels of sentences. experiments on two datasets demonstrate that our model outperforms non-bert-based models and achieves at least comparable accuracy with bert-based models, but runs much more efficiently than bert. in the future, we would attempt to join multi-granularity embeddings and bert together, so as to further improve the performance. the generalization of our model to other languages will also be investigated.