A S iamese CNN Architecture for Learning C hinese Sentence Similarity (2020)

Abstract: This paper presents a deep neural architecture which applies the siamese convolutional neural network sharing model parameters for learning a semantic similarity metric between two sentences. In addition, two different similarity metrics (i.e., the Cosine Similarity and Manhattan similarity) are compared based on this architecture. Our experiments in binary similarity classification for Chinese sentence pairs show that the proposed siamese convolutional architecture with Manhattan similarity outperforms the baselines (i.e., the siamese Long Short-Term Memory architecture and the siamese Bidirectional Long Short-Term Memory architecture) by 8.7 points in accuracy.

## introduction measuing the similarity between words, sentences, paragraphs and documents is an important component in various tasks such as information retrieval, document clustering, word-sense disambiguation, automatic essay scoring, short answer grading, machine translation and text summarization. traditional sentence similarity measurement is based on the edit distance, jaccard index, and the bag-ofwords models such as tf-idf. these methods of learning sentence similarity are in fact based on the word level, which may not be sufficient. for example, there are two chinese sentences as shown in figure 1 . the corresponding english translations are "how to buy lcd tvs." and "what kind of lcd tvs is good?". from the word level (i.e., character level in chinese), the two sentences look the same, but they have totally different meaning at the sentence level. that is, we need sentence-level methods to capture the semantics of the sentences for sentence similarity measurement. with the rapid development of machine learning, using neural network to learn representations of sentence-level meanings has been widely verified to be effective. the beginning of using neural network to learn sentence-level representations may be the word2vec from google @xcite , which used a shallow structure to learn the vector-based representations of sentence level. however, using one neural architecture to learn two sentences in two steps may cause inconsistent representations. hence, siamese structures, which can learn two sentences at a time, are attractive alternatives. the siamese architecture that can achieve state-of-the-art accuracy results in learning english sentence similarity is a bidirectional long short-term memory (bi-lstm) based siamese recurrent architecture @xcite . in our preliminary study, we tested the effectiveness of a siamese recurrent architecture for learning chinese sentence similarities. however, this did not perform as well as what is reported in @xcite . therefore, we borrowed a siamese convolutional architecure from the image processing field @xcite to implement chinese sentence similarity learning. the results in binary similarity classification for chinese sentence pairs show that siamese convolutional architecture outperforms the siamese recurrent architecture in learning accuracy. in addition, we consider two similarity metrics in the siamese convolutional architecture, namely, the manhattan similarity and the cosine similarity. the results show that the siamese convolutional architecture plus the manhattan similarity performs better than other baselines for learning the similarity between two chinese sentences. our contributions are as follows: (1) we verified that siamese convolutional architecture is effective in learning chinese sentence semantic similarity; (2) we verified that the manhattan similarity can achieve better performance than other similarity metrics regardless of the learning architectures. ## related work the siamese network @xcite is firstly proposed for non-linear metric learning with similarity information. it naturally learns representations that embody the invariance and selectivity desiderata through explicit information about similarity between pairs of objects. the siamese architecture has since been widely used in vision applications. specifically, the siamese convolutional networks were used to learn complex similarity metrics for face verification @xcite and dimensionality reduction on image features @xcite . while in the natural language processing (nlp) field, the convolutional neural network (cnn) has attracted more attentions since the successes in using cnn to do the traditional nlp tasks @xcite , and the availability of high-quality semantic word representations has been verified when using the cnn @xcite . recently, cnns have been applied to matching sentences @xcite . although the work @xcite has used the cnn to learn representations of two sentences, this is not a siamese cnn architecture. following this, the siamese long short-term memory (lstm) architecture was proposed for sentence similarity task using token level embedding @xcite . subsequently, a siamese bi-lstm structure was proposed in order to improve the result of sentence similarity @xcite . a siamese cnn combines bi-lstm structure has been proposed for learning sentence similarity @xcite . however, this architecture achieves lower accuracy than the independent bi-lstm structure. also, the paper @xcite did not give any comparisons between siamese cnn architecture and siamese bi-lstm architecture. later, siamese lstm and siamese bi-lstm were compared based on an english dataset @xcite . ## siamese convolutional architecture the proposed siamese convolutional architecture is depicted in figure 1 . in the architecture, there are two exactly alike convolutional structures that are used. the inputs of each convolutional structure are the character-level embeddings of a sentence, and the outputs of each convolutional structure are the sentence level representations. then, a similarity metric is used to compare the outputs of the two convolutional structures. the calculated similarity is the final output of the siamese convolutional architecture. within each convolutional architecture, there are one fully connected layer after three repeated convolutional layers and max pooling layers. we have also tested the six repeated structure, but the accuracy did not show a significant improvement. the kernel size of each convolutional layer is different. a higher convolutional layer is equipped record sentence 1 sentence 2 label 1 三星手机屏幕是不是最好的？ is the screen of samsung mobile phone the best or not? 三星手机的屏幕是不是都很好 are the screens of all kinds of samsung mobile all good? 0 2 广西桂林电子科技大学怎么样? how about the guilin university of electronic technology in guangxi ? 桂林电子科技大学怎么样 how about the guilin university of electronic technology ? with a larger kernel size. the fully connected layer then reduces the dimension of the learned representations from pooling layer. the learned output vector from the fully connected layer will be used to calculate the similarity then. the similarity depicted in figure 1 is the exponential negative norm of two learned representation vectors, which is defined as: after calculating the similarity, we then use the mean-square error (mse) of the similarity and the label as the loss function. the gradients of the loss will be fed back to both convolutional structures. in this way, the two convolutional structures will share the same parameters, and then they can learn the representations of the two sentences with the same distribution. based on a threshold of the similarity, we can then evaluate the accuracy after learning. ## experiments our experiments are the binary similarity classification tasks for chinese sentence pairs. although obtaining a chinese sentence similarity dataset is difficult, we found a dataset named lcqmc with even distribution of the labels (i.e., similar sentence pairs and dissimilarity sentence pairs occupy 50% and 50% of all dataset respectively) from baidu. the format of the dataset is shown in figure 2. punctuations of some sentences are omitted in the original data. this dataset consists 283,000 data records. we have chosen 250,000 data records as the training data, and 12,500 data records as the test data. a data record is @xmath0, sentence 2, @xmath1.e., 1 represents that the two sentences are similar and 0 represents that two sentences are dissimilar). we also used the english dataset paws-x @xcite to train and test the different saimese architecture as the comparisons. in the paws-x, the data format is the same with lcqmc, and the task is also to learn the semantic similarity between two sentences. we used 49,401 data records in paws-x to train models and 2,000 to test. from the aforementioned related works, @xcite and @xcite , we have chosen two baselines: the siamese bi-lstm architecture and the siamese lstm architecture. moreover, we also evaluated the two baslines and the siamese convolutional architecture with two different loss functions (i.e., the manhattan similarity based mse and the cosine distance based mse). as for the siamese convolutional architecture, the kernel sizes of the three repeated convolutional layers are set as 3, 4 and 5. we ran a total of 100 epochs and the batch size of each epoch is 128. the adam optimizer is used. during the optimization, we set the learning rate to be 0.001. following previous work @xcite , we used accuracy as the evaluation metric. we then set the similarity threshold as 0.5. that is to say, if the calculated similarity is more than 0.5, the prediction is that the two sentences are similar. conversely, the similarity less than 0.5 is decided as dissimilar.if the similarity is exactly 0.5, the result is excluded for calculating accuracy. in figure 3 , we compared the convergence speeds and losses of all combinations of the siamese architectures and the two loss functions. the lines in different colors represent different siamese architectures. the full lines are the losses using the manhattan similarity, and the dotted lines are the losses using the cosine similarity. it can be observed that no matter what kind of the siamese architecture is used, the manhattan similarity based siamese architectures converge fast. as for the loss, the siamese convolutional architectures always achieve lower losses than the baselines. in the siamese convolutional architectures, the manhattan similarity based siamese architecture always gets a lower loss. as a result, the siamese convolutional architecture with the manhattan similarity metric achieves the lowest loss. regardless of the choice the similarity metric, the losses of the siamese lstm architecture and the siamese bi-lstm architecture are similar. next, we evaluated the accuracy of all the combinations of the siamese architectures and the two loss functions. the representation formats of different combinations are the same with figure 3 . as shown in figure 4 , it can be seen that the siamese convolutional architectures always achieve higher accuracy. in the siamese convolutional architectures, the one with the manhattan similarity metric always achieves higher accuracy. in summary, the siamese convolutional architecture with the manhattan similarity metric can obtain the highest accuracy. the performances of the two baselines are not substantially different regardless of the similarity metric. the siamese bi-lstm architecture shows a slight improvement of the accuracy comparing to the siamese lstm architecture. we listed all the experimental results in the ## conclusion and future work we proposed a siamese convolutional architecture for chinese sentence similarity learning. the experimental results have verified that the siamese convolutional architecture outperforms the siamese bi-lstm architecture and the siamese lstm architecture in terms of accuracy. moreover, with the proposed architecture, we learned that for chinese sentence similarity task, manhattan similarity metric can help to achieve faster convergence and higher accuracy than any other similarity metric. our results also suggest that the siamese architectures which are effective in english nlp tasks may not necessarily work well in chinese nlp tasks. in the future, we will try to build and conduct experiments on siamese transformer @xcite architecture. in addition, we will use bert @xcite to obtain word embeddings as the inputs of the siamese architecture to test performances.