Context and Literacy Aware Learnable Metric for Text Simplification (2023)

Abstract: Automatic evaluation of text simplification is important; but assessing its transformation into simpler sentences can be challenging for various reasons. However, the most commonly used metric in text simplification, SARI, fails to capture the difficulty of generating words that are not present in the references, regardless of their meaning. We propose a new learnable evaluation metric that decomposes and reconstructs sentences to simultaneously measure the similarity and difficulty of sentences within a single system. Through experiments, we confirm that it exhibited the highest similarity in correlation with the human evaluation.

## introduction text simplification refers to the task of transforming sentences into easily understandable sentences while maintaining context @xcite . this is effective in various domains such as education and biomedicine. in the biomedical field, in particular, there is a need to simplify health information which is often written in a complex manner that is difficult for the general public to understand @xcite . however, it is challenging to evaluate whether complex sentences have been simplified successfully. this is because the process of converting a complex sentence into a simpler one involves various operations such as information deletion, paraphrasing, and insertion, while ensuring that the semantic meaning remains equivalent. thus, most existing metrics @xcite @xcite ) evaluate text simplification by separately assessing how semantically similar the output is and figure 1 : the score variations of metrics for sentences generated by systems such as gpt-3.5 @xcite , in comparison to a copy version of the reference. "widespread blood thickening issue" marked in bold is a term simplified for "disseminated intravascular coagulation," which means a severe condition where blood clots form excessively in small blood vessels. changing "thickening" to "clotting" or "coalescing", both of which are not in the reference, do not affect the sari score. also, "bleeding" which has the opposite meaning of "clotting" results in the same outcome. to address this issue, we introduce learnable evaluation metrics. our metric not only evaluates whether the generated text is semantically flawless but also comprehensively assesses the literacy level of the text. to train our model, we leverage the first supervised metric, lens @xcite , introduced along with the training dataset simpeval past . in this metric, we train modules to evaluate the literacy level of the generated text, assessed its semantic similarity with the original sentence, and compare it with references. the experimental findings show an improvement in performance over conventional systems. the lens collected simpeval past , which includes 12k human ratings of 2.4k simplifications generated by 24 different systems applied to sentences from turkcorpus @xcite for training lens. they selected 100 challenging sentences which were used in the training process of the model from turkcorpus and asset @xcite . asset provides 10 human references for each complex sentence. to evaluate the performance of the model and other simplification metrics, simpeval 2022 comprising 1k human ratings of 360 simplifications generated by human annotators and state-of-the-art models was introduced. wiki-da @xcite composes evaluations of 600 sentences generated by 6 different systems, assessing fluency, preservation, and simplicity. the effectiveness of learnable evaluation metrics has been demonstrated in machine translation @xcite . lens, for the first time, applies learnable evaluation to text simplification. their model encodes all text compo-nents (input texts, system outputs, and references) into roberta @xcite embeddings, combines them, and feeds them into a feedforward network to predict the scores. these scores are then compared with human rating scores and the mean squared error (mse) loss is calculated. ## methods in our model, we introduce context-aware and literacy-aware layers to comprehensively assess the similarity and complexity of sentences. as shown in figure 2 , the context module evaluates whether the system output sentence is generated in a manner similar to the meaning of the input and reference sentences. the literacy module assesses the literacy level of the system output sentences and learns whether the literacy level of the system's output sentence is easier than that of the input sentence, or similar to the literacy level of the reference sentence. to achieve this, the context and literacy-aware modules have distinct loss functions. furthermore, we concatenate the vectors extracted from these modules and calculate the final loss function based on their differences from the original vector. given a source text s, the corresponding system output o and references @xmath0, r 2 , ..., r n }, our model predicts the quality scores of system outputs for each reference and selects the top k (@xmath1. (s, o, r i ) are encoded by a transformer-based encoder represented as embeddings (e s , e o , e r ). in the following section, two modules that fed into these embeddings are described. ## conclusion we have proposed a new decomposition-guided learnable evaluation metric for text simplification, an automatic metric capable of simultaneously evaluating semantic preservation and literacy levels in text simplification tasks. we succeed in measuring these two aspects separately, as they demonstrates a higher correlation with human evaluations than existing automatic metrics. this approach can be extended to transfer tasks with binary classes and holds promise for application to datasets with diverse literacy levels.