Towards Reference-free Text Simplification Evaluation with a BERT S iamese Network Architecture (2023)

Abstract: Text simplification (TS) aims to modify sentences to make their both content and structure easier to understand. Traditional n-gram matching-based TS evaluation metrics heavily rely on the exact token match and human-annotated simplified sentences. In this paper, we present a novel neural-network-based reference-free TS metric BETS that leverages pre-trained contextualized language representation models and large-scale paraphrasing datasets to evaluate simplicity and meaning preservation. We show that our metric, without collecting any costly human simplification reference, correlates better than existing metrics with human judgments for the quality of both overall simplification (+7.7%) and its key aspects, i.e., comparative simplicity (+11.2%) and meaning preservation (+9.2%).

## introduction text simplification (ts) models aim at rewriting complicated input sentences into more readable variants. ts has a wide range of applications in various domains including education, language learning and journalism. it can further serve as a useful preprocessing step to simplify the downstream tasks such as parsing, machine translation, and information extraction @xcite . an example of ts is shown in figure 1 , where some parts of the sentence are deleted and paraphrased (e.g., "nonhuman primates" is paraphrased into a simpler term, "apes"). the goal in ts is to simplify the sentences while retaining the original semantic meaning. to achieve this, three types of operations: splitting, deletion, and paraphrasing @xcite have been widely studied in the research community. splitting and deletion operations try to split long sentences and delete irrelevant modifiers @xcite @xcite @xcite . for paraphrasing, most recent work regards the ts task as a monolingual machine translation (mt) problem @xcite @xcite @xcite . in order evaluate text simplification, previous work uses two main kinds of metrics adapted from the metrics used in the mt literature: (1) bleu @xcite : the most commonly used metric for text generation which computes the exact n-gram matching between the reference and candidate; (2) sari @xcite : an n-gram based metric which is specifically designed for ts to measure the correspondence of the preserved, deleted and added information between the system output and human-annotated simplified references. however, there are two main limitations of these metrics based on exact n-gram match between the system output and human reference (i.e., simplified sentences): (1) simple n-gram overlap count can fail to capture meaning preservation or compositional diversity @xcite . since ts systems usually rewrite word tokens to achieve simpler sentences, exact n-gram matching can not capture every variant of paraphrasing due to the diversity of human language (e.g., in figure 1 , "had assumed" can be rewritten as either "thought" or "supposed"); (2) the heavy reliance on human reference restricts the generalization ability, where the reference can be wrong @xcite or requires experts to annotate @xcite . since high-quality human annotations on sentence simplification are costly to acquire, it is hard to obtain sufficient human reference to capture the diversity in expression or to evaluate systems in a new domain. to tackle these limitations, in this paper, we propose a reference-free evaluation metric bets (bert embedding-based evaluation for text simplification), that leverages pre-trained contextualized language representation models (ptlms) to improve semantic-involvement, comprehensiveness, and generalization ability. from our human evaluation on simplification quality, we observe that: (1) human perception on overall simplification quality is not identical to the simple arithmetic mean of the quality of its key factors: meaning preservation and simplicity change; (2) the quality of meaning preservation correlates poorly with the quality of simplicity change. on the other hand, the importance of these two kinds of aspectual quality can vary over different domains and purposes. motivated by these observations, we first use contextualized language embeddings to replace the exact tokens. we then build two separate units with pretrained neural networks to measure these aspects. finally, we perform regression with these separate scores to acquire a flexible and optimizable metric that correlates well with human judgments on ts. we evaluate the effectiveness of bets by comparing with human annotations on both the rankings and aspectual scores (on grammar, simplicity, and meaning preservation) of ts outputs from various systems. we show that our metric correlates better with human scores than existing metrics, for each aspect. finally, our combined metric with optimized coefficients creates a single balanced metric that can evaluate ts systems, where any of the main aspects can be further emphasized according to future applications. ## experiments 4.1 experiment setup to compare the automatic metrics, we compute the correlation between the metric scores and human perception. following previous work, we evaluate correlation with two kinds of evaluation protocols: input today nrc is organised as an independent, private foundation . system 1 output today nrc is organised as an independent, private foundation. (g,m,s+ = 5, 5, 0) system 2 output today nrc is organize as an open and private trust. (g,m,s+ = 3, 4, 3) in this setting, similar to the first protocol, for each of the 70 selected input sentences, we ask the human annotators to rank the quality of their simplified versions from the systems. ## conclusion in this paper, we investigate the problem of ts evaluation. we find that: (1) different key aspects of ts, comparative simplicity and meaning preservation, correlate poorly and should be measured separately; experiments show that our metric, bets, correlates well with human judgments in both the scoring and ranking settings. also, the focus on preservation or simplifying text can be manually adjusted by changing the weights on two almost orthogonal components to fit personalized and specific weightings of these aspect in different domains. 13257 this research has been made possible by funding support from the research grants council of hong kong under the general research fund project 16204720. we are grateful to hongming zhang, shikhar murty, tianqing fang, and the anonymous reviewers for their helpful comments.