SEMA : Text Simplification Evaluation through Semantic Alignment (2020)

Abstract: Text simplification is an important branch of natural language processing. At present, methods used to evaluate the semantic retention of text simplification are mostly based on string matching. We propose the SEMA (text Simplification Evaluation Measure through Semantic Alignment), which is based on semantic alignment. Semantic alignments include complete alignment, partial alignment and hyponymy alignment. Our experiments show that the evaluation results of SEMA have a high consistency with human evaluation for the simplified corpus of Chinese and English news texts.

## introduction text simplification is a rewriting operation that aims to improve the comprehensibility of the text by modifying, deleting, simplifying humanreadable text. it tried to retain the core semantics of original text while improving readability of the text. in natural language processing tasks, long and complex sentences will bring about various problems, for example, the quality of grammatical analysis depends on the length and the grammar difficulty of texts directly, and complex sentences may cause ambiguity during machine translation @xcite . therefore, text simplification is often used in the preprocessing steps of other nlp tasks. in addition, text simplification is also used to rewrite reading materials for children, second language learners, readers with aphasia and other people with low reading comprehension skills @xcite . as related researches are in the early stage, the results of text simplification cannot meet the needs of the audience well. one of the difficulties is the lack of reasonable text simplification evaluation indicators. at present, most evaluation methods are conducted * corresponding author: zhaohuizhou@blcu.edu.cn by experts or machine translation evaluation indicators. therefore, researches on how to analyze the results of text simplification has important application value. text simplification mainly includes vocabulary and semantic structure simplification. the main operation is text segmentation, that is, rewriting a single sentence into one or more simpler sentences while preserving the main semantics @xcite . text simplification has gradually attracted attention in recent years @xcite @xcite , it should be evaluated from three aspects: fluency (ie: grammatical correctness), correctness (ie: semantic retention) and simplicity (ie: degree of text simplification). initially, experts can only evaluated the results through three aspects, and the final score is based on the likert scale 1 ; later, someone proposed to use readability indicators to evaluate text simplification, but because the readability indicators are designed for passage-level texts, the application effects at the sentence level are not very prominent @xcite . in recent years, the evaluation indicators of machine translation have been increasingly used in the evaluation of text simplification, including bleu, rouge based on n-gram and wer, ter based on edit distance. in machine translation tasks, bleu is the most widely used evaluation indicators, which was proposed in 2002. the original purpose is to replace the manual evaluation of translation results. the quality of the machine translation task is mainly evaluated by evaluating the difference between the 1 likert scale is one of the most commonly used scoring aggregate scales. it was developed by american social psychologist likert in 1932 on the basis of the original aggregate scale. the scale consists of a set of statements. each statement has five answers: "strongly agree", "agree", "not necessary", "disagree" and "strongly disagree", which are recorded as 5, 4, 3, 2, 1, and the final score is the sum of score for each aspect. sari is an evaluation indicator based on reference sentences proposed in 2016 @xcite . it focuses on the aspect of words added, deleted, and retained, but it cannot evaluate sentences at semantic level. samsa is a semantic structure-based evaluation indicator proposed in 2018 @xcite , but it relies too much on string matching in the judgment of semantic consistency, which leads to low semantic retention calculation results for simplified text. based on the characteristics of these evaluation indicators, this research proposes a text simplification evaluation indicator sema based on semantic alignment. the contribution of this paper is to propose a semantic retention evaluation indicator of text simplification based on semantic alignment. semantic alignment includes complete alignment, partial alignment and hyponymy alignment. different semantic alignment weights are given according to the degree of semantic alignment, so as to reasonably evaluate the semantic retention of text simplification of different rewriting methods. ## text simplification evaluation through semantic alignment(sema) sema is a further optimization of the samsa indicator, including two parts: 1. the basic for-mula sema-base (basic formula) is obtained by calculation when n @xmath0; 2. in terms of indicator calculation strategy, semantic alignment is used to replace the string alignment and it mainly includes three semantic alignment methods: full alignment (semabase), partial alignment, and hyponymy alignment. ## conclusion this research improves the semantic structure based text simplification evaluation measure samsa proposed in 2018. there are mainly several aspects: the case of n @xmath1; semantic alignment is used to replace string matching, mainly based on three semantic alignments method: full alignment, partial alignment, hyponymy alignment. finally, a semantic retention evaluation measure about text simplification sema based on semantic alignment is formed. we did experiments on english sentencelevel and passage-level. the experimental results show that it is similar to the manual evaluation results, which shows its significance in text simplification evaluation. when we apply sema to chinese, we summarize the characteristics of chinese and use dependency syntax analysis to extract the main semantic information in the sentence. experimental results show that sema has high applicability in chinese corpus, and it is the first semantic retention evaluation indicator based on semantic alignment on chinese corpus. in future research, we will continue to use larger corpus to explore sema's evaluation methods under different semantic retention thresholds; in addition, the text simplification indicator proposed in this paper only evaluates the semantic retention, other aspects of evaluating texts simplification such as grammaticality and degree of simplification need to be further explored; at the same time, follow-up research should expand the scale of the text corpus and collect multi-subject, multi-genre and multilength texts to test the usability of our indicator.