Assessing the Quality of Human-Generated Summaries with Weakly Supervised Learning (2021)

Abstract: This paper explores how to automatically measure the quality of human-generated summaries, based on a Norwegian corpus of real estate condition reports and their corresponding summaries. The proposed approach proceeds in two steps. First, the real estate reports and their associated summaries are automatically labelled using a set of heuristic rules gathered from human experts and aggregated using weak supervision. The aggregated labels are then employed to learn a neural model that takes a document and its summary as inputs and outputs a score reflecting the predicted quality of the summary. The neural model maps the document and its summary to a shared “summary content space” and computes the cosine similarity between the two document embeddings to predict the final summary quality score. The best performance is achieved by a CNN-based model with an accuracy (measured against the aggregated labels obtained via weak supervision) of 89.5%, compared to 72.6% for the best unsupervised model. Manual inspection of examples indicate that the weak supervision labels do capture important indicators of summary quality, but the correlation of those labels with human judgements remains to be validated. Our models of summary quality predict that approximately 30% of the real estate reports in the corpus have a summary of poor quality.

## introduction many types of reports incorporate humangenerated summaries that seek to highlight the most important pieces of information described in the full document. this is notably the case for real estate condition reports, which are long, technical reports presenting the current condition (as it is known to the seller) of a property for sale, including the general state of each room, known damages and defects, and key technical aspects such as the heating, plumbing, electricity and roof. despite the rich amount of information contained in these real estate reports, several surveys have shown that many buyers of real estate do not read the full documents but rather concentrate on the summaries @xcite . however, professionals regard the quality of these summaries as varying greatly, from good to very poor. actors in the real estate market have suggested that this information deficit may play an important role in the reported 10% of norwegian real estate transactions ending in conflict @xcite . in this work we explore ways of automatically measuring the quality of such summaries, using a corpus of 96 534 real estate condition reports and their corresponding summaries. although there exists a substantial body of work on summary evaluation @xcite , previous work has largely focused on automatically generated summaries, often by comparing those generated summaries to reference summaries written by humans. the automated evaluation of human-generated summaries, however, has received little attention so far. this paper presents an approach to automatically evaluate the quality of human-generated summaries when no manually labelled data is available. instead, we rely on a set of heuristic rules provided by domain experts to automatically annotate a dataset of summaries (each coupled to their full-length document) with quality indicators. those annotations are subsequently aggregated into a single, unified annotation layer using weak supervision @xcite , based on a generative model that takes into account the varying coverage and accuracy of the heuristic rules. although one could in theory directly use the labels obtained through weak supervision as quality indicators for the summaries, such an approach has a number of limitations. most importantly, heuristic rules are only triggered under certain conditions, and may therefore "abstain" from providing a quality score on some summaries. for instance, we may have a rule stating that, if the full report describes a major defect or damage in the bathroom, then a summary that fails to mention this defect should be labelled as being of poor quality. this rule will only label summaries that meet this specific condition, and abstain from generating a prediction in all other cases. some heuristic rules may also depend on the availability of external data sources that are not available at prediction time. for instance, one can exploit the fact that an insurance claim has been raised on the real estate as an indicator that the summary may have omitted to mention some important defects or damages. needless to say, this heuristic can only be applied on historical data, and not on new summaries. to address those shortcomings, we use the aggregated labels obtained via weak supervision as a stepping stone to train a neural model whose task is to assess the quality of a summary in respect to its full-length document. the neural model embeds both the document and its summary into a dedicated semantic space (referred to as the summary content space) and computes the final quality score using cosine similarity. as real estate condition reports are often long documents (10 pages or more), we conduct experiments with models based not only on embeddings of entire documents, but also on embeddings of sections, sentences and words. the paper makes three contributions: as detailed in section 4, this weak supervision approach is able to outperform unsupervised methods based on latent semantic analysis @xcite or doc2vec embeddings @xcite ) -by a large margin. although the approach is evaluated on a specific corpus of real estate reports, the proposed methodology can be applied to any type of summaries, provided human experts are able to specify heuristics to assess the summary quality in the target domain. ## approach the approach adopted in this paper is divided in two steps. we first define and apply a set of labeling functions to the dataset, allowing us to derive binary (good/bad) quality indicators on the summaries in relation to their full-length reports. those quality indicators are then aggregated into a single, probabilistic measure of summary quality using weak supervision. the dataset and labeling functions are described in sections 3. 1 and 3.2. then, using those aggregated labels as targets, we learn a neural model that maps the reports and summaries to a common summary content space. the resulting embeddings should reflect only key semantic information that is relevant for measuring summary quality, so that it can be measured by the cosine similarity in this space. the neural architecture and associated document embedding methods are defined in sections 3. 3 and 3.4. assessing the summary quality using a neural model instead of relying directly on the quality indicators derived from the labeling functions has two major advantages. first, the neural model can generalise to all possible report/summary pairs, while aggregated labels may be absent for some summaries, as the rules are only triggered when specific conditions are met. second, some labeling functions depend on external resources that may be unavailable at prediction time. for instance, one labeling function relies on whether the buyer has filed an insurance claim, which is a piece of information that is only available for historical data, and requires us to "peek into the future". ## conclusion this paper describes a novel approach to automatically assess the quality (focusing primarily on the criteria of content coverage) of human-generated summaries, using a corpus of real estate condition reports as a concrete example. the approach relies on the creation of document embeddings that are appropriate for measuring summary quality. this gives us a particular kind of semantic space (the summary content space) where summary quality can be measured by the cosine similarity between the report and its summary. since we have no access to "ground truth" values for the summary quality, we obtain indirect quality indicators based on a set of 22 heuristic rules gathered from human experts. those quality indi-cators are then aggregated into a single probability (of a summary being of high quality) using weak supervision. the aggregated probabilities are subsequently employed as targets for training neural models optimised for the task of predicting summary quality. evaluation results show that the best neural model, based on a convolutional architecture, achieves an overall accuracy of 89.5% when measuring the model output against the aggregated labels, while the best unsupervised model (lsa) only achieves an accuracy of 72.6%. an important limitation of the proposed method is the reliance on indirect indicators of summary quality (as expressed by the heuristic rules) instead of human judgments. a key research question for future work is thus to examine the correlations between the quality measures derived from the labeling functions and human judgments. while the heuristic rules do not capture all aspects that may influence the overall quality of a summary, our hypothesis (yet to be validated) is that they nevertheless correlate well with human judgments. an additional benefit of these heuristic rules is their explanatory power, making it possible to provide concrete, human-readable suggestions on how to improve a given summary. although not considered in this paper, the use of document embeddings relying on contextual word representations is another interesting research question that we wish to investigate in future work.