What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization (2023)

Abstract: Summarization models often generate text that is poorly calibrated to quality metrics because they are trained to maximize the likelihood of a single reference (MLE). To address this, recent work has added a calibration step, which exposes a model to its own ranked outputs to improve relevance or, in a separate line of work, contrasts positive and negative sets to improve faithfulness. While effective, much of this work has focused on how to generate and optimize these sets. Less is known about why one setup is more effective than another. In this work, we uncover the underlying characteristics of effective sets. For each training instance, we form a large, diverse pool of candidates and systematically vary the subsets used for calibration fine-tuning. Each selection strategy targets distinct aspects of the sets, such as lexical diversity or the size of the gap between positive and negatives. On three diverse scientific long-form summarization datasets (spanning biomedical, clinical, and chemical domains), we find, among others, that faithfulness calibration is optimal when the negative sets are extractive and more likely to be generated, whereas for relevance calibration, the metric margin between candidates should be maximized and surprise–the disagreement between model and metric defined candidate rankings–minimized.

## introduction traditionally, summarization models have been trained to maximize the likelihood of gold-standard references. this training paradigm introduces an exposure bias because, during training, the model is not exposed to the metrics on which it is evaluated. without being able to calibrate its own predictions with metrics, models are prone to produce summaries with irrelevant or repetitive content @xcite , or misrepresent the claims in the source text @xcite . calibration offers a flexible and effective set of methods to remedy this exposure bias by explicitly instructing a model to distinguish between high and low quality summaries. by varying how candidate sets are constructed and optimized, an extra calibration step can unlock large gains in relevance (via rouge @xcite ) or improve the faithfulness of summaries to the source @xcite ). yet, much of this work has addressed howhow to generate candidates @xcite and how to define effective calibration objectives @xcite . work has largely been separated into relevance and faithfulness calibration, with less study of the interaction between the two. relevance, often measured with rouge, captures the content overlap with a human-written reference, whereas faithfulness is typically reference-free, and captures the fidelity of a summary to the source text(s). in this paper, we examine both faithfulness and relevance as the target metrics for calibration and seek to uncover the underlying characteristics of effective calibration sets for each separately, as well as analyze the interactions between them. to accomplish this, we implement a diverse set of existing methods for constructing candidate and corrupted summaries and combine them to form a large candidate pool. from this pool, we implement different filtering strategies for set selection, which target specific characteristics, such as the metric margin between negatives and positives, diversity, and the model likelihood of generating each candidate in the set. we run experiments that vary only in the training data selected for candidate sets. for each experiment, we extract a wide range of relevant statistics (e.g., diversity, length) on the candidate sets and show the relationship between these set statistics and downstream performance. to guide future research, we analyze the plots to provide insights into, and rationale for, optimal set construction. additionally, a large portion of research has focused on summarization of single-document news articles @xcite . we seek to broaden and pressure test recent advances in contrastive fine-tuning by experimenting on three long-form, scientific, highly specialized corpora in which metrics, e.g. faithfulness, are non-trivial to define, capture, and categorize. also, long-form summarization is appealing for our calibration experiments given that memory is constrained. even with training tricks, such as gradient accumulation and half precision, only a small handful of candidates per example (4 in our experiments 1 ) fit in memory. this makes the selection step more important compared to shorter tasks. the primary contributions of this work are to: (1) benchmark calibration models on three scientific long-form datasets, including a new, chemistryfocused corpus, for which we collect fine-grained faithfulness annotations and relevance rankings from experts; (2) conduct extensive experiments to better understand the underlying characteristics and dynamics of effective calibration tuning sets. we release easily extensible code for forming and optimizing calibration sets in the scientific domain. ## related work typically, when summarization models are calibrated to quality metrics, this refers to contrastive 1 each experiment was run on a relatively large card with 40gb of gpu memory (the nvidia a100). learning to improve faithfulness. contrastive learning for faithfulness has been applied to fine-tuning @xcite @xcite , post-hoc editing @xcite , re-ranking @xcite , and evaluation @xcite @xcite . this line of research has largely focused on the methods used to generate synthetic errors for negative contrast sets: i.e., by directly mimicking errors observed during human evaluation @xcite , entity swapping @xcite , language model infilling @xcite , or using unfaithful system outputs @xcite . orthogonal to our work, cao and wang (2021a) assess the relative efficacy of a diverse set of corruption methods when used for contrastive fine-tuning for faithfulness. for relevance calibration, models are typically calibrated to the rouge scores of their own outputs after an initial fine-tuning step @xcite . @xcite extend the work of @xcite and run a broad sweep of loss functions and candidate generation methods for two-step relevance calibration while establishing state of the art performance (rouge) across single document corpora. as opposed to contrasting positives and negatives in a latent space, these models are instructed to calibrate decoder likelihoods to rouge or bertscore-defined rankings. our work is distinct along three key dimensions: (1) we consider long-document scientific summarization, rather than single-document; (2) we consider both faithfulness and relevance calibration and analyze the interactions between the two, often competing, quality objectives; (3) we uncover relationships between key set statistics and downstream performance by systematically varying how calibration sets are formed from candidate pools. ## datasets dataset statistics are shown in clinical. we use the long-form hospital course summarization dataset from @xcite . refer to appendix a for details on this dataset. ## calibration pipeline at a high-level, we fine-tune (ft) language models with standard maximum likelihood estimation (mle) on each summarization corpus, and then calibration-tune (ct) on a combined objective, which adds a calibration loss (ca) to the mle loss: we describe the setup (objective, metrics, and candidate generation methods) for relevance calibration ( §4.1) and faithful calibration ( §4.2, before jointly discussing statistics on each setup ( §4.3). ## selection strategies. problem statement. from a large candidate pool, select a target number to be used for ct (2 positives and 2 negatives for faithfulness, and 4 for rank-based relevance). figure 1 graphically reveals the different strategies implemented which are designed to target specific set characteristics. they do not represent optimal or recommended strategies, e.g., a minimum metric gap for faithfulness. in appendix g, we hypothesize as to the specific nature and direction of the impact of the above characteristics on post-calibration summaries. random. for random, for each training instance, we take a random sample without replacement. gap random quality metric-based extreme relevance average max margin min high diversity-based diversity ... all combinations max min likelihood-based beam ... top beams bottom beams ... extreme beams ? ? ? min margin ? ? ? relevance rank set spurious correlates tokens long short margin-based ... all combinations diversity... 6 , while the bottom corresponds to rows in quality-based. for quality-based, we rank all candidates by rel agg or f aith agg . then, we select candidates at different extremes of these scales. ## results please refer to appendix f for implementation details on ft and ct training and hyper-parameters. ## conclusion in this paper, we explore what makes an effective calibration set for both relevance and faithfulness tuning. to do so, we create large candidate pools for calibration and design strategies which systematically target set characterstics. we then analyze trends between these characteristics and downstream performance. our analysis is intended to serve as a guide for subsequent research when designing methods to form synthetic candidates, as well as motivation to jointly consider relevance and faithfulness for calibration, given their covariance and the importance of both to real-world systems.