An ideal summarization system should produce summaries that have high content coverage and linguistic quality . Many state-ofthe-art summarization systems focus on content coverage by extracting content-dense sentences from source articles . A current research focus is to process these sentences so that they read fluently as a whole . The current AE-SOP task encourages research on evaluating summaries on content , readability , and overall responsiveness . In this work , we adapt a machine translation metric to measure content coverage , apply an enhanced discourse coherence model to evaluate summary readability , and combine both in a trained regression model to evaluate overall responsiveness . The results show significantly improved performance over AESOP 2011 submitted metrics . Research and development on automatic and manual evaluation of summarization systems have been mainly focused on content coverage ( Lin and Hovy , 2003 ; Nenkova and Passonneau , 2004 ; Hovy et al . , 2006 ; Zhou et al . , 2006 ) . However , users may still find it difficult to read such high-content coverage summaries as they lack fluency . To promote research on automatic evaluation of summary readability , the Text Analysis Conference ( TAC ) ( Owczarzak and Dang , 2011 ) introduced a new subtask on readability to its Automatically Evaluating Summaries of Peers ( AESOP ) task . Most of the state-of-the-art summarization systems ( Ng et al . , 2011 ; Zhang et al . , 2011 ; Conroy et al . , 2011 ) are extraction-based . They extract the most content-dense sentences from source articles . If no post-processing is performed to the generated summaries , the presentation of the extracted sentences may confuse readers . Knott ( 1996 ) argued that when the sentences of a text are randomly ordered , the text becomes difficult to understand , as its discourse structure is disturbed . Lin et al . ( 2011 ) validated this argument by using a trained model to differentiate an original text from a randomlyordered permutation of its sentences by looking at their discourse structures . This prior work leads us to believe that we can apply such discourse models to evaluate the readability of extract-based summaries . We will discuss the application of Lin et al . 's discourse coherence model to evaluate readability of machine generated summaries . We also introduce two new feature sources to enhance the model with hierarchical and Explicit / Non-Explicit information , and demonstrate that they improve the original model . There are parallels between evaluations of machine translation ( MT ) and summarization with respect to textual content . For instance , the widely used ROUGE ( Lin and Hovy , 2003 ) metrics are influenced by BLEU ( Papineni et al . , 2002 ) : both look at surface n-gram overlap for content coverage . Motivated by this , we will adapt a state-of-theart , linear programming-based MT evaluation metric , TESLA ( Liu et al . , 2010 ) , to evaluate the content coverage of summaries . TAC 's overall responsiveness metric evaluates the quality of a summary with regard to both its content and readability . Given this , we combine our two component coherence and content models into an SVM-trained regression model as our surrogate to overall responsiveness . Our experiments show that the coherence model significantly outperforms all AESOP 2011 submissions on both initial and update tasks , while the adapted MT evaluation metric and the combined model significantly outperform all submissions on the initial task . To the best of our knowledge , this is the first work that applies a discourse coherence model to measure the readability of summaries in the AESOP task . 2 Related Work Nenkova and Passonneau ( 2004 ) proposed a manual evaluation method that was based on the idea that there is no single best model summary for a collection of documents . Human annotators construct a pyramid to capture important Summarization Content Units ( SCUs ) and their weights , which is used to evaluate machine generated summaries . Lin and Hovy ( 2003 ) introduced an automatic summarization evaluation metric , called ROUGE , which was motivated by the MT evaluation metric , BLEU ( Papineni et al . , 2002 ) . It automatically determines the content quality of a summary by comparing it to the model summaries and counting the overlapping n-gram units . Two configurations -ROUGE-2 , which counts bigram overlaps , and ROUGE-SU4 , which counts unigram and bigram overlaps in a word window of four -have been found to correlate well with human evaluations . Hovy et al . ( 2006 ) pointed out that automated methods such as ROUGE , which match fixed length n-grams , face two problems of tuning the appropriate fragment lengths and matching them properly . They introduced an evaluation method that makes use of small units of content , called Basic Elements ( BEs ) . Their method automatically segments a text into BEs , matches similar BEs , and finally scores them . Both ROUGE and BE have been implemented and included in the ROUGE / BE evaluation toolkit 1 , which has been used as the default evaluation tool in the summarization track in the Document Un-1 http://berouge.com / default.aspx derstanding Conference ( DUC ) and Text Analysis Conference ( TAC ) . DUC and TAC also manually evaluated machine generated summaries by adopting the Pyramid method . Besides evaluating with ROUGE / BE and Pyramid , DUC and TAC also asked human judges to score every candidate summary with regard to its content , readability , and overall responsiveness . DUC and TAC defined linguistic quality to cover several aspects : grammaticality , non-redundancy , referential clarity , focus , and structure / coherence . Recently , Pitler et al . ( 2010 ) conducted experiments on various metrics designed to capture these aspects . Their experimental results on DUC 2006 and 2007 show that grammaticality can be measured by a set of syntactic features , while the last three aspects are best evaluated by local coherence . Conroy and Dang ( 2008 ) combined two manual linguistic scores -grammaticality and focus -with various ROUGE / BE metrics , and showed this helps better predict the responsiveness of the summarizers . Since 2009 , TAC introduced the task of Automatically Evaluating Summaries of Peers ( AESOP ) . AESOP 2009 and 2010 focused on two summary qualities : content and overall responsiveness . Summary content is measured by comparing the output of an automatic metric with the manual Pyramid score . Overall responsiveness measures a combination of content and linguistic quality . In AESOP 2011 ( Owczarzak and Dang , 2011 ) , automatic metrics are also evaluated for their ability to assess summary readability , i.e. , to measure how linguistically readable a machine generated summary is . Submitted metrics that perform consistently well on the three aspects include Giannakopoulos and Karkaletsis ( 2011 ) , Conroy et al . ( 2011 ) , and de Oliveira ( 2011 ) . Giannakopoulos and Karkaletsis ( 2011 ) created two character-based n-gram graph representations for both the model and candidate summaries , and applied graph matching algorithm to assess their similarity . Conroy et al . ( 2011 ) extended the model in ( Conroy and Dang , 2008 ) to include shallow linguistic features such as term overlap , redundancy , and term and sentence entropy . de Oliveira ( 2011 ) modeled the similarity between the model and candidate summaries as a maximum bipartite matching problem , where the two summaries are represented as two sets of nodes and precision and recall are cal- culated from the matched edges . However , none of the AESOP metrics currently apply deep linguistic analysis , which includes discourse analysis . Motivated by the parallels between summarization and MT evaluation , we will adapt a state-ofthe-art MT evaluation metric to measure summary content quality . To apply deep linguistic analysis , we also enhance an existing discourse coherence model to evaluate summary readability . We focus on metrics that measure the average quality of machine summarizers , i.e. , metrics that can rank a set of machine summarizers correctly ( human summarizers are not included in the list ) . We proposed TESLA-S by adapting an MT evaluation metric to measure summary content coverage , and introduced DICOMER by applying a dis- course coherence model with newly introduced features to evaluate summary readability . We combined these two metrics in the CREMER metric -an SVM-trained regression model -for automatic summarization overall responsiveness evaluation . Experimental results on AESOP 2011 show that DICOMER significantly outperforms all submitted metrics on both initial and update tasks with large gaps , while TESLA-S and CREMER significantly outperform all metrics on the initial task.3

Challenge: Existing evaluation metrics for summarization models focus on content coverage while fluency also impacts readability.
Approach: They propose to use a machine translation metric for content coverage, a discourse coherence model for readability, and combine them for overall responsiveness.
Outcome: The proposed metric significantly improves over all the metrics submitted to AESOP 2011.