Is Character Trigram Overlapping Ratio Still the Best Similarity Measure for Aligning Sentences in a Paraphrased Corpus? (2022)

Abstract: Sentence alignment is an essential step in studying the mapping among different language expressions, and the character trigram overlapping ratio was reported to be the most effective similarity measure in aligning sentences in the text simplification dataset. However, the appropriateness of each similarity measure depends on the characteristics of the corpus to be aligned. This paper studies if the character trigram is still a suitable similarity measure for the task of aligning sentences in a paragraph paraphrasing corpus. We compare several embedding-based and non-embeddings model-agnostic similarity measures, including those that have not been studied previously. The evaluation is conducted on parallel paragraphs sampled from the Webis-CPC-11 corpus, which is a paragraph paraphrasing dataset. Our results show that modern BERT-based measures such as Sentence-BERT or BERTScore can lead to significant improvement in this task.

## introduction monolingual text matching is necessary for many downstream applications, such as paraphrase identification and extraction @xcite , question answering @xcite , natural language inference @xcite , and text generation @xcite . 1 the nearest semantic associates of the verb decide based on the cosine similarity between the word2vec vectors (trained on english wikipedia) are those verbs such as: choose (0.64), opt (0.62), persuade (0.61), want (0.58), take the qa task as an example, identifying the text fragments that match the given question within the associated passage is often required for locating the desired answer. however, modern neural-network (nn) approaches to text matching often suffer from certain limitations when two sequences contain considerably different lexicons or diverse grammatical structures @xcite . for example, when the verb "decide" in the sentence "they decided to go" is nominalized to the noun "decision" in its paraphrase "they made a decision to go", the popular word embedding similarity approach might fail as the embedding-vectors of "decide" and "decision" are quite different 1 . another example is a pair of sentences "a cat is chasing a dog." and "a dog is chasing a cat.", which contain the same set of lexicons and syntactic structure but with opposite meanings. furthermore, the nn approaches frequently fail while the matching involves multi-word expressions, or when expressions require compositionality handling @xcite @xcite . for example, it is difficult to match expressions "put off" and "procrastinate" using basic word embeddings, as the real meaning of the idiom "put off" is not the sum of the meanings of its tokens. we found that the limitations of nn models in text matching could be greatly alleviated by utilizing lexico-syntactic paraphrasing patterns such as @xcite ]] @xcite vp @xcite refuse (0.57), insist (0.56). however, the noun decision only has a similarity score 0.512, which means that its similarity to the verb decide is even less than that between decide and its quasi-antonymous refuse. ## sentence alignment procedure our sentence alignment procedure is implemented with two main elements: (1) selecting an appropriate search mechanism (either bi-directional best match or sequence match); (2) adopting a specified sentence similarity measure, either string-or embedding-based. ## experiments figure 2 shows the operation flow adopted in the experiments. taking a pair of paraphrased paragraphs as input, the paragraphs are first preprocessed and split into sentences. then, we use the sentence alignment module with the selected search mechanism and similarity measure to generate the desired sentence alignments. those one-to-one sentence alignments are then extracted and output as the answer. following subsections give details of the experiment setting and results. ## error analysis we analyzed 50 errors generated by our best approach (i.e., sequence match with sbertmini), and categorized them based on their associated error sources: (1) mistaking 1-n mapping for 1-1 (46%); (2) associated with incorrect sentence boundary (26%), in which the sentences are split incorrectly before conducting alignment (e.g., a sentence is incorrectly split into two sequences by the sentence segmenter); (3) paraphrased sentences take different sequence-orders within two given paragraphs (16%); (4) others (12%), of which it is difficult to attribute each error to a specific reason. the first error category, incorrectly marking 1-n alignment as 1-1, is likely due to two reasons. first, those proposed similarity measures are still incapable of truly reflecting the semantic similarity between two sentences when they are paraphrased in an abstract way; as a result, they might incorrectly convert a golden 1-n mapping into a 1-1 mapping. second, because the alignment is selected based on the sentence similarity and the probability of each alignment type on the development set, the model has a preference for extracting 1-1 alignments as they are most common in the dataset (cf. the second error category (i.e., with incorrect sentence boundary) occurs when the preprocessing module incorrectly split the sentences within one of the input paragraphs. finally, the last type of error is caused by the sequence search mechanism, which assumes all paraphrased passage pairs follow the same relative order within each paragraph. if this assumption is violated in the given paragraph pair, it will always return an incorrect answer. ## conclusions we have presented the first comparison among various model-agnostic similarity measures used for aligning sentences among paraphrased paragraphs. for most cases, we find that embedding-based similarity measures outperform the string-based approaches (including the previous sota character trigram approach tested on the ts dataset), and sentence-embedding-based methods are preferable to the word-embeddingbased methods for most search mechanisms except the uni-directional greedy matching. additionally, our results have shown that in calculating the similarity for sentence alignment, word vector averaging is better than adopting the [cls] token when retrieving a representation of a whole sentence from a bert-based model.