Current measures for evaluating text simplification systems focus on evaluating lexical text aspects , neglecting its structural aspects . In this paper we propose the first measure to address structural aspects of text simplification , called SAMSA . It leverages recent advances in semantic parsing to assess simplification quality by decomposing the input based on its semantic structure and comparing it to the output . SAMSA provides a reference-less automatic evaluation procedure , avoiding the problems that reference-based methods face due to the vast space of valid simplifications for a given sentence . Our human evaluation experiments show both SAMSA 's substantial correlation with human judgments , as well as the deficiency of existing reference-based measures in evaluating structural simplification . 1 Text simplification ( TS ) addresses the translation of an input sentence into one or more simpler sentences . It is a useful preprocessing step for several NLP tasks , such as machine translation ( Chandrasekar et al . , 1996 ; Mishra et al . , 2014 ) and relation extraction ( Niklaus et al . , 2016 ) , and has also been shown useful in the development of reading aids , e.g. , for people with dyslexia ( Rello et al . , 2013 ) or non-native speakers ( Siddharthan , 2002 ) . The task has attracted much attention in the past decade ( Zhu et al . , 2010 ; Woodsend and Lapata , 2011 ; Wubben et al . , 2012 ; Siddharthan and Angrosh , 2014 ; Narayan and Gardent , 2014 ) , but has yet to converge on an evaluation protocol that yields comparable results across different methods and strongly correlates with human judgments . This is in part due to the difficulty to combine the effects of different simplification operations ( e.g. , deletion , splitting and substitution ) . Xu et al . ( 2016 ) has recently made considerable progress towards that goal , and proposed to tackle it both by using an improved reference-based measure , named SARI , and by increasing the number of references . However , their research focused on lexical , rather than structural simplification , which provides a complementary view of TS quality as this paper will show . This paper focuses on the evaluation of the structural aspects of the task . We introduce the semantic measure SAMSA ( Simplification Automatic evaluation Measure through Semantic Annotation ) , the first structure-aware measure for TS in general , and the first to use semantic structure in this context in particular . SAMSA stipulates that an optimal split of the input is one where each predicate-argument structure is assigned its own sentence , and measures to what extent this assertion holds for the input-output pair in question , by using semantic structure . SAMSA focuses on the core semantic components of the sentence , and is tolerant towards the deletion of other units . 2For example , SAMSA will assign a high score to the output split " John got home . John gave Mary a call . " for the input sentence " John got home and gave Mary a call . " , as it splits each of its predicate-argument structures to a different sentence . Splits that alter predicate-argument relations such as " John got home and gave . Mary called . " are penalized by SAMSA . SAMSA 's use of semantic structures for TS evaluation has several motivations . First , it provides means to measure the extent to which the meaning of the source is preserved in the output . Second , it provides means for measuring whether the input sentence was split to semantic units of the right granularity . Third , defining a semantic measure that does not require references avoids the difficulties incurred by their non-uniqueness , and the difficulty in collecting high quality references , as reported by Xu et al . ( 2015 ) and by Narayan and Gardent ( 2014 ) with respect to the Parallel Wikipedia Corpus ( PWKP ; Zhu et al . , 2010 ) . SAMSA is further motivated by its use of semantic annotation only on the source side , which allows to evaluate multiple systems using same source-side annotation , and avoids the need to parse system outputs , which can be garbled . In this paper we use the UCCA scheme for defining semantic structure ( Abend and Rappoport , 2013 ) . UCCA has been shown to be preserved remarkably well across translations ( Sulem et al . , 2015 ) and has also been successfully used for machine translation evaluation ( Birch et al . , 2016 ) ( Section 2 ) . We note , however , that SAMSA can be adapted to work with any semantic scheme that captures predicate-argument relations , such as AMR ( Banarescu et al . , 2013 ) or Discourse Representation Structures ( Kamp , 1981 ) , as used by Narayan and Gardent ( 2014 ) . We experiment with SAMSA both where semantic annotation is carried out manually , and where it is carried out by a parser . See Section 4 . We conduct human rating experiments and compare the resulting system rankings with those predicted by SAMSA . We find that SAMSA 's rankings obtain high correlations with human rankings , and compare favorably to existing referencebased measures for TS . Moreover , our results show that existing measures , which mainly target lexical simplification , are ill-suited to predict human judgments where structural simplification is involved . Finally , we apply SAMSA to the dataset of the QATS shared task on simplification evaluation ( Štajner et al . , 2016 ) . We find that SAMSA obtains comparative correlation with human judgments on the task , despite operating in a more restricted setting , as it does not use human ratings as training data and focuses only on structural aspects of simplicity . Section 2 presents previous work . Section 3 discusses UCCA . Section 4 presents SAMSA . Section 5 details the collection of human judgments . Our experimental setup for comparing our human and automatic rankings is given in Section 6 , and results are given in Section 7 , showing superior results for SAMSA . A discussion on the results is presented in Section 8 . Section 9 presents experiments with SAMSA on the QATS evaluation benchmark . We presented the first structure-aware metric for text simplification , SAMSA , and the first evaluation experiments that directly target the structural simplification component , separately from the lexical component . We argue that the structural and lexical dimensions of simplification are loosely related , and that TS evaluation protocols should assess both . We empirically demonstrate that strong measures that assess lexical simplification quality ( notably SARI ) , fail to correlate with human judgments when structural simplification is performed by the evaluated systems . Our experiments show that SAMSA correlates well with human judgments in such settings , which demonstrates its usefulness for evaluating and tuning statistical simplification systems , and shows that structural evaluation provides a complementary perspective on simplification quality .

Challenge: Current evaluation metrics for text simplification task only focuses on lexical but not on structural aspects and do not strongly correlate with human evaluation.
Approach: They propose a reference-free metric that uses semantic parsers to decompose inputs and outputs before comparing to assess their semantic structures.
Outcome: Human evaluation reveals that the proposed metric correlates better with human judgements and also complements existing reference-based metrics.