Predicting the Semantic Textual Similarity with S iamese CNN and LSTM (2018)

Abstract: Semantic Textual Similarity (STS) is the basis of many applications in Natural Language Processing (NLP). Our system combines convolution and recurrent neural networks to measure the semantic similarity of sentences. It uses a convolution network to take account of the local context of words and an LSTM to consider the global context of sentences. This combination of networks helps to preserve the relevant information of sentences and improves the calculation of the similarity between sentences. Our model has achieved good results and is competitive with the best state-of-the-art systems.

## introduction semantic text similarity (sts) is an important task in natural language processing (nlp) applications such as information retrieval, classification, extraction, question answering, and plagiarism detection. the sts task measures the degree of similarity between two texts and can be expressed as follows: given two sentences, a system returns a continuous score on a scale from 1 to 5, with 1 indicating that the semantics of the sentences are completely independent and 5 meaning that there is a semantic equivalence. sts is a difficult issue since languages have numerous ambiguities and synonymous expressions, while sentences may have variable lengths and complex structures. therefore basic models, e.g. bag-of-words or tf-idf models, are constrained by their specificities that put aside the role played by the word order and ignore syntactic as well as semantic relationships. recent successes in sentence similarity have been obtained using neural networks (rnns: recurrent neural networks @xcite @xcite and cnns: convolutional neural networks @xcite ). neural networks (nns) use a deep analysis of sentences and words to take better into account both the semantics and the structure of sentences in order to predict the sentence similarity. in this paper, we describe our technique based on nns to measure similarity. first, we use a siamese cnn to analyze the local context of words in a sentence and to generate a representation of the relevance of a word and its neighborhood. then, we use a siamese lstm to analyze the entire sentence based on its words and its local contexts. at last, we predict the semantic similarity of pairs of sentences using the manhattan distance. we applied our framework on the semeval information for sts assignment and we acquired competitive outcomes demonstrating that our model can give helpful information to enhance the sentence analysis. this paper is organized as follows: we make an overview of relevant work for sts in section 2. next, we detail our approach in section 3. the experimental setup and results are presented in sections 4 and 5, respectively. finally, we give our conclusion and some last remarks in section 6. ## related work to deal with the sts task, previous studies have resorted to various features (e.g. word overlap, synonym/antonym), linguistic resources (e.g. wordnet and pre-trained word embeddings) and a wide assortment of learning algorithms (e.g. support vector regression (svr), regression functions and nns). among these works, several techniques extract multiple features of sentences and apply regression functions to estimate these similarity scores @xcite @xcite . @xcite analyzed distinctive word relations (e.g. synonyms, antonyms, and hyperonyms) with features based on counts of co-occurences with other words and similarities between captions of images. @xcite predicted the sentence similarity from syntactic relationship, distinctive content similitudes, length and string features. bjerva et al. (2014) the development of nns has improved the results of many nlp applications and especially the sts task @xcite @xcite . architectures such as rnns and cnns further improve the semantic analysis and the prediction of sentence relatedness. rnns differ from other nn models in their ability to process sequential information. they update a memory cell to make sense of data read in a sentence over time. @xcite used a recursive autoencoder (rae) and a wordnet grant framework to produce sentence embeddings. they consolidated these embeddings with a support vector machine (svm) classifier to compute a semantic relatedness score. long short term memory (lstm) enhances rnns to handle longterm dependencies @xcite @xcite . the lstm engineering is made out of a memory cell and non-direct gating units that update its state over time and manage the data stream into/out the cell. @xcite used a siamese lstm to encode sentences using pre-trained word embedding vectors. siamese lstms used the same weights to encode sentences and to produce comparable sentence representations for similar sentences. then, they predicted the closeness of pair of sentences using the manhattan distance between the sentence representations. @xcite introduced the tree-lstm that is a generalization of lstm for tree-structured network topologies. they utilized this tree-lstm to encode a couple of sentences and to predict their closeness with a nn that analyzes the distance and the angle between the sentence embeddings. cnns have accomplished excellent outcomes in classification @xcite and other nlp tasks @xcite . @xcite generated sentence embedding using a siamese cnn architecture with various convolution and pooling operations to extract distinctive granularities of information. their convolution uses filters that analyze entire word embeddings and each dimension of word embeddings with multiple window sizes. for output of the convolution operation, they applied several pooling types (max, mean, and min). finally, they predicted the sentence similarity from numerous measurements (horizontal and vertical comparison) to compare local regions of sentence representation. in this work, we join the ideas examined in @xcite and @xcite to produce more accurate semantic sentence embeddings. the next section presents our model and its characteristics w.r.t. previous work. ## our model a sentence is composed of words which can form phrases and clauses. examining a sentence and its components helps us to comprehend its meaning. nns are structures that can inspect relationships between words from multiple points of view. on the one hand, lstms can recognize and process the semantics of a sentence by investigating the words through time. they update their state to get the gist of the sentence (global context) in the order of words. in this procedure, lstms filter unimportant data by retaining just the main information. on the other hand, cnns use layers with convolution filters that are connected to local features @xcite . they enable the analysis of a sentence from multiple perspectives (filters). this type of nns does not have the same concern with the sentence length as lstms since cnns examine all the words of the sentence together. nonetheless, cnns do not consider the order of words in their analysis, so these structures cannot investigate sequence relationships in the sentence. differently from @xcite that only analyze the general context of words and from @xcite that do not consider the order of words in the sentences, we analyze the words in two perspectives: general and local contexts. words are considered through time from the general information of a word (word embedding) and its specific semantic and syntactic features (local context) based on its previous and its following words. we apply a cnn to investigate the local context for each word in a sentence. the cnn analyzes together all the words of the local context and generates their representation as a unique structure. then, we utilize an lstm to examine the words of the sentence one by one (figure 1 ). our nn has a siamese structure @xcite , i.e. our cn n a and our lst m a are equal to our cn n b and our lst m b , respectively. the following subsection describes our cnn, our lstm, and our similarity metrics to predict the sentence similarity. ## results in order to understand the relevance of the local context for the sentence similarity, we investigated the original siamese lstm without local context and compared it with our method using various lengths for the local context: 3, 5, 7, and 9 ( the original siamese lstm analyzes a sentence considering only the general context of words. as expected, the analysis of general and local contexts of words improved the sentence analysis, according to the pearson's and pearman's correlation coefficients and the mean squared error (mse) scores. short or long local contexts did not generate the best results, which shows that short local context (3 words) did not get enough information about the neighborhood of words and long local context (7 words) includes irrelevant information. method r ρ mse siamese lstm (mueller & thyagarajan, 2016) 0.8822 0.8345 0.2286 siamese lstm (publicly available version) foot_1 0.8500 0.7860 0.3017 siamese #local context: 3 + siamese lstm 0.8536 0.7909 0.2915 siamese #local context: 5 + siamese lstm 0.8549 0.7933 0.2898 siamese #local context: 7 + siamese lstm 0.8540 0.7922 0.2911 siamese #local context: 9 + siamese lstm 0.8533 0.7890 0.2923 non-linear similarity @xcite 0.8480 0.7968 0.2904 constituency tree lstm @xcite 0.8582 0.7966 0.2734 @xcite 0.8655 0.7995 0.2561 dependency tree lstm @xcite 0.8676 0.8083 0.2532 convnets @xcite 0.8686 0.8047 0.2606 although our method did not generate the best results, our system is among the top systems and the results were improved with respect to the publicly available version of the original siamese lstm. in order to illustrate how our local context acts on sentence analysis, and "mary lived through an era of liberating reform for women." for each pair of words taken in both sentences, the similarity measured as a cosine distance foot_2 is computed either from general word embeddings (table a) or local contexts of length 5 (table b). the first things to notice is that the two tables have different ranges of values because they each represent a different dimensional space; this means that values must be compared inside each table. analyzing ## conclusion sts is an important task for various nlp applications, e.g. automatic text summarization (ats), question-answering, information retrieval, etc. our system combines cnn and lstm structures to analyze, to identify and to preserve the relevant information in each part of sentences and in the whole sentences. the local context turned out to be useful to get complement information about a word in a sentence and to improve the sentence analysis. in our experiments, the local context improved the prediction of the sentence similarity, by reducing the mean squared error and increasing the correlation scores. we plan to test other methods to analyze the local context @xcite . unfortunately, the dataset we used for the experiments is of a modest size and we did not find larger annotated corpora for this task. therefore, we also want to lead extrinsic evaluations by measuring how sts acts on ats systems, depending on whether the original or the modified siamese lstm model is used.