T EXT S UMMARIZATION WITH O RACLE E XPECTATION
A BSTRACT
Extractive summarization produces summaries by identifying and concatenating the most important sentences in a document.
Since most summarization datasets do not come with gold labels indicating whether document sentences are summary-worthy, different labeling algorithms have been proposed to extrapolate oracle extracts for model training.
In this work, we identify two ﬂaws with the widely used greedy labeling approach: it delivers suboptimal and deterministic oracles.
To alleviate both issues, we propose a simple yet effective labeling algorithm that creates soft, expectation-based sentence labels.
We deﬁne a new learning objective for extractive summarization which incorporates learning signals from multiple oracle summaries and prove it is equivalent to estimating the oracle expectation for each document sentence.
Without any architectural modiﬁcations, the proposed labeling scheme achieves superior performance on a variety of summarization benchmarks across domains and languages, in both supervised and zero-shot settings.
I NTRODUCTION
Summarization is the process of condensing a source text into a shorter version while preserving its information content.
Thanks to neural encoder-decoder models (Bahdanau et al., 2015; Sutskever et al., 2014), Transformer-based architectures (Vaswani et al., 2017), and large-scale pretraining (Liu & Lapata, 2019; Zhang et al., 2020a; Lewis et al., 2020), the past few years have witnessed a huge leap forward in summarization technology.
Abstractive methods ﬂuently paraphrase the main content of the input, using a vocabulary different from the original document, while extractive approaches are less creative — they produce summaries by identifying and subsequently concatenating the most important sentences in a document — but manage to avoid hallucinations, false statements and inconsistencies.
Neural extractive summarization is typically formulated as a sequence labeling problem (Cheng &
Lapata, 2016), assuming access to (binary) labels indicating whether a document sentence should be in the summary.
In contrast to the plethora of datasets (see Section 5 for examples) available for abstractive summarization (typically thousands of document-abstract pairs), there are no large-scale datasets with gold sentence labels for extractive summarization.
Oracle labels are thus extrapolated from abstracts via heuristics, amongst which greedy search (Nallapati et al., 2017) is the most popular by far (Liu & Lapata, 2019; Xu et al., 2020; Dou et al., 2021;
Jia et al., 2022).
In this work we challenge received wisdom and rethink whether greedy search is the best way to create sentence labels for extractive summarization.
Speciﬁcally, we highlight two ﬂaws with greedy labeling: (1) the search procedure is suboptimal , i.e., it does not guarantee a global optimum for the search objective, and (2) greedy oracles are deterministic , i.e., they yield a single reference extract for any given input by associating sentences in the document to its corresponding abstract.
Perhaps an obvious solution to the suboptimality problem would be to look for oracle summaries following a procedure based on beam search.
Although beam search ﬁnds better oracles, we empirically observe that summarization models trained on these do not consistently improve over greedy oracles, possibly due to the higher risk of under-ﬁtting (Narayan et al., 2018a) — there are too few positive labels.
Moreover, beam search would also create deterministic oracles.
A summarization
[Table]
[Table]
PM on Friday night.
[Table]
SussexpolicesaidshemayhavebeentalkingwithsomeoneonFacetimebeforedisappearingataround2am.
[Table]
[Table] [Table] [Table]
Reference Summary
• (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Jasmine
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Coleman disappeared from her home at around 2 AM this morning.
Police believed she may have been driven to- wards London by an older man.
• She has been found safe and well in Croydon, South East London today.
[Table]
R system trained on either greedy or beam oracles is optimized by maximizing the likelihood of a single oracle summary.
This ignores the fact that there can be multiple valid summaries for an article, in other words, the summary hypothesis space is naturally a multi-modal probability distribution.
We illustrate this point in Table 1.
In this paper we deﬁne a new learning objective for extractive summarization which promotes nondeterministic learning in the summary hypothesis space, and introduce O REO , ORacle ExpectatiOn labeling, as a simple yet effective sentence labeling scheme.
We prove the equivalence between estimating O REO labels and optimizing the proposed learning objective.
As a result, it is sufﬁcient for current models to be trained on O REO labels without requiring any architectural changes.
Extensive experiments on summarization benchmarks show that O REO outperforms comparison labeling schemes in both supervised and zero-shot settings, including cross-domain and cross-lingual tasks.
Additionally, we showcase that extracts created by O REO can better guide the learning and inference of a generative system, facilitating the generation of higher-quality abstracts.
We further analyze O REO ’s behavior by measuring attainable summary knowledge at inference time, and demonstrate it is superior to related deterministic and soft labeling schemes, which we argue contributes to consistent performance gain across summarization tasks.
R ELATED W ORK
Narayan et al.
(2018a) were among the ﬁrst to discuss problematic aspects of sentence labeling schemes for extractive summarization.
They argue that labeling sentences individually as in Cheng & Lapata (2016) often generates too many positive labels which leads to overﬁtting, while a model trained on greedy labels (Nallapati et al., 2017) underﬁts the data.
Although extractive performance can be boosted via ﬁnetuning pretrained encoders with greedy labels (Liu & Lapata, 2019), Zhong et al.
(2020) show that reranking summary candidates constructed from greedy predictions can further improve summary quality.
This demonstrates that the underﬁtting problem caused by greedy labels still exists even when pretrained models are used.
Issues with greedy labeling have also been discussed from the perspective of data bias , including lead bias (Nenkova, 2005; Kedzie et al., 2018;
Grenander et al., 2019) — greedy labels display a bias towards lead sentences in news text and systems trained on them do not easily transfer to other domains — and monolingual bias (Jia et al., 2022) — greedy labels created for one language (e.g., English) do not transfer to a different language.
The idea of learning from multiple references has found application in various tasks including dialog response generation (Gao et al., 2019), machine translation (Khayrallah et al., 2020), and question answering (Zhu et al., 2020).
Summarization datasets with multiple references are not generally available for model training, but a few have been manually created for system evaluation (Dang, 2005).
In extractive summarization, gold references in the form of sentence labels do not usually exist, and learning from multiple references has not been yet explored.
In this work, we use beam search to create multiple high-quality oracle summaries, from which summary-level supervision is aggregated into sentence labels to promote multi-reference learning for extractive summarization.
P ROBLEM F ORMULATION
Let D =
{ x i } m 1 denote a document consisting of sentences x i .
An extractive summarizer produces a summary hypothesis that represents salient information via identifying a subset of sentences ˆ
Y = { ˆ y j } n 1 , n << m within D .
In practice, ROUGE (Lin & Hovy, 2003), an automatic metric based on lexical overlap is commonly adopted to evaluate R ( ˆ Y , S ) , the quality of ˆ Y against gold reference summary S .
Following previous work (Cheng & Lapata, 2016; Nallapati et al., 2017; Narayan et al., 2018a; Liu & Lapata, 2019), we conceptualize extractive summarization as a sequence labeling task, and aim to build a system that estimates the summary-worthiness of each sentence in a non-autoregressive manner.
As mentioned earlier, sentence labels need to be ﬁrst extrapolated to train an extractive system, since existing datasets are label-free, they only contain document-abstract pairs.
B ERT S UM (Liu & Lapata, 2019) is a popular extractive model and representative of the approach sketched above.
Built on top of BERT (Devlin et al., 2019), it adds a two-layer Transformer (Vaswani et al., 2017) for sentence representation and a sigmoid layer for summary prediction.
During inference, document sentences { x i } m 1 are ranked based on their estimated scores, and summary { ˆ y j } n 1 is identiﬁed.
The number of n sentences to be included in the summary is often pre-deﬁned and ﬁxed.
F ROM E XISTING L ABELING S CHEMES TO O REO
Early labeling methods create sentence labels (cid:96) i by evaluating the similarity of x i against reference summary S through various heuristics h ( · ) , (cid:96) i def =
h
( x i , S ) , including R OUGE (Lin & Hovy, 2003) and rule-based features such as sentence and paragraph position information, and the number of mentioned entities (Woodsend & Lapata, 2010; Cheng & Lapata, 2016).
These methods obtain local labels as they assume a sentence can be classiﬁed as summary-worthy on its own, without taking the summary context into account.
However, model evaluation does not operate at the sentence-level, as a sentence is part of a summary hypothesis Y together with other candidates (Narayan et al., 2018a).
The aim of extractive summarization is to deliver a high-quality summary hypothesis, i.e., a good set of sentences rather than a set of good sentences.
A sentence might achieve a high score on its own but contribute little to a summary hypothesis, e.g., due to redundancy.
An alternative is to obtain global labels, based on whether a sentence occurs within the optimal set of sentences which collectively achieve the highest score according to some evaluation metric like R OUGE :
[Equation]
where | C ( D ) | = C (cid:0) mn (cid:1) is the hypothesis combinatorial space.
As Equation (1) is computationally intractable, in practice, it is approximated by further conditioning on a heuristic search space S such that Y ∗ ≈ arg max Y ∈ S ( D )
R
( Y, S ) , and the approximated Y ∗ is usually called an oracle summary.
A widely adopted approximation is greedy labeling (Nallapati et al., 2017;
Narayan et al., 2018a;
Liu & Lapata, 2019; Zhong et al., 2020), which uses greedy search to maximize R at each step of sentence selection (the algorithm stops when R can no longer increase or the maximum number of steps is reached; see Appendix A for the algorithm).
While signiﬁcantly reducing complexity, greedy labeling does not guarantee a global optimum.
To ﬁnd better summaries to serve as oracles, we propose to replace greedy search with beam search which we refer to as beam labeling.
We empirically ﬁnd that around 8 %– 20% of (greedy) labels can be potentially improved with beam search (when setting the beam size to 256; see Appendix B for details).
However, having better labels does not necessarily translate to performance improvements, and we next discuss why this is the case.
4. O REO : E STIMATING O RACLE E XPECTATION
Extractive summarization models are typically trained to optimize max p θ ( Y ∗ | D ) , where the best hypothesis Y ∗ can be approximated with greedy or beam search.
This learning objective maximizes the probability at a single point Y ∗ , and assigns zero probability to other summary hypotheses ˆ Y , regardless of their quality.
We note that this formulation leads to a discrepancy between how the model is optimized and how the labels against which this optimization takes place are obtained.
Given an input document, sequence labeling summarization models assume conditional independence at sentence-level inference, while in greedy labeling, each step in the process of maximizing R ( Y ∗ , S ) conditions on the outcomes of previous steps.
From an optimization perspective, this mismatch renders ﬁtting a non-autoregressive sequence labeler difﬁcult for two reasons: (1) learning to search and maximizing the likelihood at Y ∗ is challenging, and so the model tends to underﬁt Y ∗ (Narayan et al., 2018a), and (2) probabilities at other Y with high evaluation scores remain under-estimated and uncalibrated due to supervision sparsity.
Simply replacing greedy search with beam search does not resolve these optimization challenges as point estimation is still performed.
A solution is to evaluate summary hypotheses during training and reward the model accordingly (Narayan et al., 2018a).
However, this is non-trivial as the the metric R is usually non-differentiable, and it is computational expensive to sample from a large combinatorial hypothesis space, e.g., with
Reinforcement Learning (Sutton & Barto, 1998).
Rather than changing the training of the model, in this work, we study how to derive a better sentence labeling algorithm that leads to a better optimization objective.
Speciﬁcally, we wish to incorporate multiple high-quality hypotheses as oracle summaries into the learning objective.
Our key assumption is that extractive oracles are non-deterministic, but drawn from a distribution p ( Y ∗ | D, S ) .
We thus formulate the objective for extractive summarization as:
[Equation]
Under this formulation, an optimized model is expected to assign high probability p θ ( Y | D ) when there exists an oracle summary with high probability and high score according to some quality evaluation metric.
From the perspective of sentence labeling, we note that candidate x i relates to the summarization task through the oracle summary space Y .
As Y is a combinatorial space, the mapping x i → Y ∗ is one-to-many.
Therefore, we can compute the probability for each candidate to be selected via marginalization:
[Equation]
To connect the marginalization in Equation (3) with the summarization objective in Equation (2), we further incorporate hypothesis evaluation R ( Y ∗ , S ) , and deﬁne the summary-worthiness of a sentence x i as the expectation of its associated oracle evaluation:
[Equation]
Initialize where the oracle membership p ( x i | Y ∗ , D ) is identical to y i = 1 ( x i ∈ Y ∗ ) and the oracle distribution will be discussed in Section 4.3.
Given a sequence labeling model θ , maximizing the oracle expectation for all input sentences is equivalent to the objective in Equation (2).
We present the proof in Appendix C.
To be compatible with the standard sequence labeling architecture for extractive summarization, we perform MLE with a cross-entropy loss:
[Equation]
where the scaled expectation (cid:96) ( x i ) =
( (cid:96)
[Equation]
i − ¯ (cid:96) min ) / (¯ (cid:96) max − ¯ (cid:96) min ) constitutes the ﬁnal sentence labels.
The details of oracle expectation labeling are given in Algorithm 1.
4. C OMPARISON WITH E XISTING L ABELING A LGORITHMS
O REO creates soft (continuous) sentence labels, i.e., it incorporates summary-level evaluation while maintaining low sparsity.
A detailed comparison with other labeling algorithms is provided in
Table 3.
Equation (4) also bears resemblance to the RL objective used in Narayan et al.
(2018a): max E ˆ Y ∼ p
θ ( Y | D )
[ R ( ˆ Y , S )] .
They evaluate summary hypotheses directly while Equation (4) estimates sentence-level membership.
This is a consequence of the nature of the sequence labeler which does not explicitly represent the summary hypothesis space (which is combinatorial), and therefore supervision is delegated to sentences rather than summaries.
By maximizing sentence-level likelihood, estimations for the associated hypotheses are updated, albeit indirectly.
Narayan et al.
(2018a)
employ REINFORCE (Williams, 1992), an on-policy RL algorithm that samples from a model during training, while in Equation (4), samples are drawn from the nonparametrized oracle distribution p
( Y ∗ | D, S ) .
We provide ofﬂine supervision in the form of static sample labels.
In contrast to the online reward in RL, ofﬂine labels can be reused during the course of training and are therefore more sample efﬁcient.
Our labeling scheme can be seen as a type of ofﬂine bandit learning (Nguyen-Tang et al., 2022).
While the latter has been recently applied to abstractive summarization (Pang & He, 2021), it remains under-explored in extractive summarization.
4. T HE O RACLE D ISTRIBUTION
We derive the oracle distribution p
( Y ∗ | D ) heuristically bearing in mind
that: (a) we have no prior knowledge as to which hypothesis is more or less likely as an oracle summary and therefore assume the oracle distribution to be uniform over a large hypothesis space; and (b) it is desirable for p ( Y ∗ | D ) to positively correlate with R ( Y ∗ , S ) and we expect this correlation to become stronger over the course of optimization.
In practice, we use beam search (with beam size k << | Y | ) to ﬁnd potential oracle summaries, and adopt a uniform distribution over top-ranked beams: p ( Y ∗ | D ) ∼ U (1 , t ) , where t < k is a hyper-parameter which we optimize on a development set.
We also experimented with several weight annealing mechanisms over top beams as determined by R , as our summary quality evaluation metric (see Appendix D for details).
[Table] [Table]
[Table]
E XPERIMENTS 5. S UPERVISED E XTRACTIVE S UMMARIZATION
We conducted all extractive summarization experiments with B ERT S UM (Liu & Lapata, 2019), the neural summarization architecture introduced in Section 3.
We opted for B ERT S UM due to its simplicity and popularity in a wide range of summarization tasks (Zhong et al., 2020; Xu & Lapata, 2021;
Jia et al., 2022).
We nevertheless note that O REO is model-agnostic and can be also applied to more complex architectures.
For a fair comparison between different labeling schemes, we follow the standard training conﬁguration used in Liu & Lapata (2019) without any additional hyper-parameter optimization (e [Table] g., for our speciﬁc labeling scheme).
We set R , the summary quality evaluation metric, to the mean of R OUGE -1 and R OUGE -2.
We report experiments on a variety of summarization datasets including CNN/DM (Hermann et al., 2015),
XSum (Narayan et al., 2018b), Multi-News (Fabbri et al., 2019), Reddit (Kim et al., 2019)
[Table]
and WikiHow (Koupaee & Wang, 2018).
Detailed statistics are shown in Table 2.
Our results are presented in Table 4.
In the ﬁrst block, we report the performance of the L EAD baseline which considers the ﬁrst k sentences in a document as the summary (see last row in Table 4) and
M ATCH S UM (Zhong et al., 2020), a state-of-the-art system which performs summary reranking with another BERT model.
The second block reports O RACLE performance with greedy labels, beam labels ( k = 256 ), and O REO labels ( k = 256 , t = 16 ; we take the top- n sentences with non-zero scores).
See Appendix E for the labeling hyperparameters k, t for each dataset, and more detail on experimental settings.
The third block reports B ERT S UM performance with different labeling schemes.
Although beam O RACLE is superior to greedy O RACLE and raises the upper bound, the overall performance of B ERT S UM optimized on beam labels does not signiﬁcantly improve upon its greedy counterpart.
In fact, performance drops drastically on WikiHow.
O REO shows inferior O RACLE results as it considers multiple top-ranked beams and is therefore not bound-preserving (see Section 5.
for detailed analysis).
However, B ERT S UM trained with O REO labels consistently outperforms a model trained with beam labels, and achieves a 0.18–0.89 R OUGE -L improvement on different
[Figure]
[Table]
[Table]
R-1/2/L is a shorthand for R OUGE . Systems
[Table]
benchmarks compared to greedy labeling. Although B ERT S UM with O REO still falls short of the state-of-the-art, we show that one-stage summarization modeling can be enhanced with better labeling, which can serve as a foundation for more complex reranking methods (see Table 11 in Appendix F for details).
5. Z ERO -S HOT C ROSS -D OMAIN E XTRACTIVE S UMMARIZATION
We further examine the generalization capability of models trained with O REO labels in a zero-shot setting.
Speciﬁcally, we evaluate a model trained on CNN/DM, against XSum, another news summarization dataset with shorter summaries (at most 2 sentences), and Reddit and WikiHow which represent entirely different domains and topics (discussion forums and instructional text).
Table 5 summarizes our results.
Models trained with O REO perform on par with greedy labeling in-domain but display stronger generalization cross-domain.
Greedy labels are more prone to lead bias, they deem as summary-worthy sentences mostly from the beginning of the document.
Such bias is present in news datasets like CNN/DM but does not transfer to other domains like social media or
Wikipedia.
O REO alleviates this bias and performs better out-of-domain.
As shown in Figures 1(a) and 1(b), O REO is less concentrated on lead sentences in news text.
5. Z ERO -S HOT C ROSS -L INGUAL E XTRACTIVE S UMMARIZATION
We next investigate the generalization capabilities of our approach in a cross-lingual setting.
We use English data for model training and report zero-shot results on a variety of languages from the
MLSum dataset (Scialom et al. 2020; see Table 2 for detailed statistics).
Following Jia et al.
, we augment English articles with word replacement during training (Qin et al., 2021) using the
MUSE (Lample et al., 2018) bilingual dictionary to align multilingual representations.
We adopt a word replacement rate of 0.5.
B ERT S UM was initialized with XLM-R base (Conneau et al., 2020), a cross-lingual pretrained model (see X L S in Table 6).
The ﬁrst block in Table 6, reports the results of a supervised X L S model which has access to training data for all languages; N L S is the zero-shot state of the art system of Jia et al.
; their approach creates multiple sets of greedy labels with different machine translation methods and adopts a neural architecture to learn weights for the obtained label sets.
The second block presents the results of a zero-shot X L S model with different labeling schemes.
As can be seen, O REO labels on Spanish,
French, and Russian are on par with greedy labeling.
Systems trained with greedy labels exhibit less cross-lingual generalization on German and Turkish, while O REO improves system performance on
German by 2.72 R OUGE -L points and on Turkish by 2.19.
Previous work (Jia et al., 2022) shows that cross-lingual performance correlates with lead bias in the target language.
For example, Turkish articles are less lead-biased than Russian in MLSum, and thus beneﬁt more from better sentence labeling.
O REO trails behind N L S which is not surprising as the latter model beneﬁts from more resources, i.e., machine translation and XLM-R large, and a more complex network architecture.
5. S UPERVISED A BSTRACTIVE S UMMARIZATION
We further assessed whether the proposed labeling scheme is of beneﬁt to abstractive summarization.
We experimented with GS UM
(Dou et al., 2021), a state-of-the-art abstractive system that takes extractive summaries as additional input to guide the generation of document abstracts.
During training, GS UM uses extractive oracles as guidance, while at inference time guidance is provided by summary hypotheses produced by a trained extractive system.
We initialized GS UM with BART (Lewis et al., 2020), and used B ERT S UM as the guidance model optimized with different labeling schemes (i.e., greedy, beam and O REO ).
Abstractive summarization results are shown in Table 7.
The ﬁrst block shows the performance of
B ART (Lewis et al., 2020) which serves as a baseline.
In the second block, we report the performance of GS UM
(Dou et al., 2021) with greedy labels (default) in addition to beam- and O REO -based variants.
As we can see, while beam labeling performs on par with its greedy counterpart, O REO guidance boosts performance with 0.37 R OUGE -L points over vanilla GS UM .
We conclude that abstractive systems can also beneﬁt from our expectation-based labeling algorithm, without any modeling changes or hyperparameter optimization.
More results with varied guidance settings can be found in Appendix F. Examples of system output are shown in Appendix G.
5. C OMPARISON WITH B OUND -P RESERVING M ETHODS
Let { z ∗ i } m 1 , z ∗ i ∈ { 0 , 1 } denote a multi-hot representation of an oracle summary.
We deﬁne a labeling function (cid:96) as bound-preserving , if there exists a constant γ ∈ [0 , 1] so that the condition( (cid:96) ( x i ) > γ ) =
z ∗ i , ∀ i holds.
Intuitively, the condition holds if and only if the top-ranked sentences remain identical.
Bound preservation of soft labels guarantees that the performance upper bound of a summarization system trained on soft labels equals that of a system trained on their original hard labels, e.g., obtained via greedy and beam search.
For instance, label smoothing (Szegedy et al., 2016), a common technique for training deep neural networks, is bound-preserving.
In contrast, O REO is generally not bound-preserving for either beam or greedy oracles.
3 To further analyse this property of soft labels, we propose O RMAX as a bound-preserving variant, by replacing the expectation with a max operator: (cid:96) (cid:48)
i def = max Y ∗ ∼ p
( Y ∗ | D,S )
[ p ( x i | Y ∗ ) R ( Y ∗ , S )] .
Compared to O REO , O RMAX incorporates multiple oracles while additionally preserving the upper bound of beam labels.
Table 8 shows the performance of label smoothing and O RMAX for extractive (ﬁrst block) and abstractive (second block) summarization.
Although label smoothing has been successfully applied to discriminative (Szegedy et al., 2016) and generative NLP tasks (Chen et al., 2018; Lewis et al., 2020), the soft labels it creates do not yield better results than their original hard labels in extractive summarization.
Label smoothing performs implicit model calibration (Müller et al., 2019), which can potentially improve sentence ranking and selection at inference, however, it also imposes regularization in neural networks training (Szegedy et al., 2016), which may render it less effective for extractive summarization where there is a higher risk of underﬁtting (Narayan et al., 2018a).
On the other hand, O RMAX performs on par with O REO on abstractive summarization, while it underperforms on extractive summarization.
Although bound preservation is, intuitively, desirable, our experimental results suggest that it is neither a necessary nor sufﬁcient condition to produce a well-optimized summarization system.
[Table]
5. P ERFORMANCE A NALYSIS
Previous experiments revealed that oracles are not necessarily indicative of model performance (see Tables 4 and 8) due to the discrepancy between model optimization and sentence labeling as discussed in Section 4.1.
To further understand how different labeling schemes (and the oracles based on them) inﬂuence model performance we quantify this discrepancy via a sampling-based method which simulates sentence selection for a sequence labeling model at inference.
Bearing in mind that a non-autoregressive sequence labeling model performs conditionally independent predictions and selects a ﬁxed size number of n sentences, we construct summary hypotheses ˆ
Y = { ˆ y j } nj =1 by drawing independent sentence samples and measure the extent to which a model can Attain Summary relevant Knowlege (ASK) as: ASK
def =
E { ˆ y j } nj =1 ∼ p ( x i | D,S ) (cid:2) R ( { ˆ y j } nj =1 , S ) (cid:3) where p ( x i | D, S ) =
(cid:96) i (cid:80) mi =1 (cid:96) i (6)
Note that p ( x i | D, S ) is shaped by soft labels, and thus results in varied sentence/summary samples for different labeling schemes.
The comparison in Figure 2 explains why we observe performance gains from O REO despite obtaining the lowest upper bound performance.
The latter considers only the best case scenario at inference, ignoring the fact that some summary knowledge encoded in sentence labels can be hard or impossible to attain, e.g., when sentences in the oracle summary are highly co-dependent (and is therefore challenging to select them jointly with a model making independent predictions), or the oracle summary contains less than n sentences (which again entails that information is missing).
Compared to other labeling schemes, O REO captures richer summary information that is attainable for sequence labeling models, narrowing the distance between upper bound performance and ASK.
Consistent with our analysis, systems trained on O REO perform robustly on a wide variety of summarization tasks.
C ONCLUSIONS AND F UTURE W ORK
We provided a comprehensive analysis of existing labeling schemes for extractive summarization, and identiﬁed two ﬂaws in greedy labeling, namely it delivers suboptimal and deterministic labels.
We proposed a novel optimization objective to learn from multiple oracle summaries, which can be instantiated by a labeling scheme based on oracle expectation.
Experimental results show that the proposed scheme achieves substantial improvement across domains and languages, without any architectural modiﬁcations.
Our framework is agnostic to the labeling metric R , however, an important future direction is to incorporate different learning signals and provide sentence labels with more desirable properties, such as query relevance (Xu & Lapata, 2022) and faithfulness (Durmus et al., 2020).
We would also like to parametrize the oracle distribution and estimate it from data, so as to derive even more accurate sentence labels. A CKNOWLEDGEMENTS
We thank the anonymous reviewers for their feedback and Shucong Zhang for useful discussions about the problem formulation.
We gratefully acknowledge the support of the UK Engineering and
Physical Sciences Research Council (grant EP/W002876/1)
G REEDY S EARCH A LGORITHM
Algorithm
1: function G REEDY ( n ) (cid:46)
Max number of sentences in summary 2:
Initialize hypothesis b to empty 3: for j ← n do 4:
Initialize hypothesis score v to5: for i ← | D | do 6: b (cid:48)
=
b + { i } , v (cid:48)
=
R OUGE ( b (cid:48) )
7: if v (cid:48) >
v then 8: b ← b (cid:48) , v ← v (cid:48) 9: if v = 0 then (cid:46)
Early stopping 10: break 11:
Initialize (cid:96) i to 0 , ∀ i (cid:46) Multi-hot labels 12: for x i ← b do 13: (cid:96) i ←14: return (cid:96) 15: end function
B L ABEL S TATISTICS
[Figure]
Ratio (%)
[Figure] [Figure]
[Figure]
Quality of Greedy Labels Figure 3 shows the distribution of greedy oracles over position in beam search results, as ranked by R ( k = 256 ).
As we can see, around 8 – 20 % of greedy labels are not top-ranked, and could thus improve with beam labels.
However, as shown in our experimental results, this improvement does not lead to a summarization system that is consistently better across different tasks.
Effects of Beam Size As beam search does not guarantee a global optimum either, we further calculate R ( Y ∗ ) /
R ( Y ∗ 256 ) to evaluate the relative quality of the top beam Y ∗ (the top beam found by varied beam sizes), compared against Y ∗ 256 (the top beam found by beam size 256).
Figure 4 shows that the quality of the top beam converges when beam size increases to 64 .
However, as oracles are not necessarily indicative of actual model performance (see Section 5.6 for details), we view beam size as a hyperparameter for optimization.
C E QUIVALENCE P ROOF
Given input document D , sentence-level inference of a non-autoregressive summarization model θ is conditionally independent, and the likelihood of an oracle summary Y ∗ =
{ y ∗ i } mi =1 (in its multi-hot representation over m document sentences) is calculated as:
[Equation]
In this case, we show that maximizing the oracle expectation for all sentences is equivalent to the objective in Equation (2):
[Equation]
Take out E
[Equation]
We note that our objective in Equation (2) serves as a lower bound of the classic extractive summarization objective max p θ ( Y ∗ | D ) weighted by oracle evaluation:
[Equation]
≤
[Equation]
D [Equation] where Y ∗ best =
[Equation]
max Y ∗
[Equation]
( Y
[Equation]
S ) (11) ∝ p θ ( Y ∗ best | D ) (12)
The equality holds only if the oracle distribution p ( Y ∗ | D, S ) is a Dirac delta distribution δ ( Y ∗ − Y ∗ best ) .
D E FFECTS OF O RACLE D ISTRIBUTION
[Table]
We devise and experiment with several oracle distributions that assign non-uniform probability to top t beams:
Annealing over Rank A r decreases the unnormalized weight from 1 to 0 over top beams, assuming that the oracle distribution positively correlates with hypothesis rank .
Annealing over Quality A q sets the unnormalized weight for a top beam Y ∗ as R ( Y ∗ ) , assuming that the oracle distribution positively correlates with hypothesis evaluation score .
Annealing over Locality A (cid:96) deﬁnes the locality of a hypothesis Y to be proportional to the mean of sentence-level scores R ( y ∗ j , S ) , y ∗ j ∈ Y ∗ .
This is based on the assumption that a hypothesis is more local if its sentences are, by themselves, high-scoring.
Hypothetically, these sentences stand a higher chance to be selected by a non-autoregressive sequence labeling model which presumably focuses more on their individual features rather than collective information (Zhong et al., 2020).
Annealing over
Position Rank
A p decreases the unnormalized weight from 1 to 0 over top beams which are reversely ranked by their position in the original document, assuming that oracle distribution positively correlates with document position .
Table 9 presents our results on CNN/DM validation set.
As we can see, the above-mentioned distributions do not yield better results than simply adopting a uniform distribution (third row).
We believe this is because hand-crafted distributions are all associated with strong assumptions, which may not be valid for real-world summarization data.
However, we note that most of these distributions still manage to outperform greedy labels, showing consistent gains when considering information from multiple oracles.
Apart from heuristic oracle distributions, we could also learn a parametrized distribution from data.
For instance, a model with a uniform oracle distribution could be trained to derive a potentially more accurate estimation from its predictions.
A new set of sentence labels would be then calculated with
Equation (5), and used to improve the optimization of a new model.
E D ETAILS OF E XPERIMENTAL S ETTINGS
[Table]
Monolingual Extractive Summarization We used three GeForce RTX 2080 GPUs for model training and bert.base in our experiments.
We refer interested readers to Liu & Lapata (2019) for detailed training conﬁgurations which are identical to ours.
Following Liu & Lapata (2019), we used the Python package pyrouge for calculating R OUGE .
For our the proposed labeling methods, we searched over the following ( k, t ) pairs: (256 , 32) , (256 , 16) , (256 , 8) , (32 , 32) , (16 , 16) , (8 , 8) .
We show the best-performing hyperparameter combinations for each dataset in Table 10.
We used standard parameter settings for all experiments: ROUGE-1.5.5.pl -c 95 -m -r 1000 -n 2 -a.
We used the datasets as preprocessed by Zhong et al.
(2020) which can be accessed at:
https: //github.com/maszhongming/matchsum .
Cross-Lingual Extractive Summarization
In our cross-lingual experiments we used four GeForce
RTX 2080 GPUs for model training with xlmr.base and mbert.base .
Particularly, interval embeddings (Liu & Lapata, 2019) were used in MB ERT S UM but not in X L S since XLM-R removes segment embeddings from the input following RoBERTA
(Liu et al., 2019).
We refer readers to
Jia et al.
for details on training conﬁguration; we made minimal adjustments to adapt to our training environment, i.e., no training hyperparameters were speciﬁcally optimized for our method.
We set the batch size to 4, and accumulated gradients every 32 steps.
Following Jia et al.
, we used word replacement rate of 0.5 to learn cross-lingual representation alignment.
We ﬁne-tuned models on the English data with a learning rate of 2 × 10 − 3 for 50,000 optimization steps, and a warm-step of 10,000.
Following Jia et al.
, we used the Python package spacy for non-English hypothesis/reference tokenization, and pyrouge for ROUGE calculation.
Abstractive Summarization
In our abstractive summarization experiments we used four GeForce
RTX 2080 GPUs for model training with bart.large ; the latter was also used in our baseline
B ART system and to initialize GS UM .
Due to GPU memory limitations, we set the maximum length of an input document to 640 tokens (with the excess clipped) and used half ﬂoat precision for efﬁcient training.
We used one sample for each GPU, and accumulated gradients every 32 steps.
We ﬁne-tuned all models on CNN/Daily Mail with a learning rate of 3 × 10 − 5 for 20,000 optimization steps, and a warm-step of 500.
Following the evaluation steps in BART (Lewis et al., 2020) and GS UM (Dou et al., 2021), we used file2rouge 5 to evaluate abstractive summaries.
F E XTENDED R ESULTS
[Table]
[Table]
Supervised Extractive Summarization We next investigate whether OREO can further improve performance when integrated with M ATCH S UM (Zhong et al., 2020).
To provide a proof of concept, we further built M ATCH S UM + O REO , using B ERT S UM (O REO ) as a sentence ranker at inference.
We experimented with two M ATCH S UM versions (based on either BERT or RoBERTA)
[Table]
and show the results on CNN/DM in Table 11. Note that M ATCH S UM +O REO uses the off-the-shelf summary-level reranker from M ATCH S UM , i.e., it is not retrained on predictions from B ERT S UM trained on O REO . As we can see, O REO improves upon vanilla M ATCH S UM simply on account of introducing higherquality candidate sentences, and thus higher-quality candidate summaries for reranking. Retraining the summary-level reranker to avoid input distribution shift would result in additional performance gains.
We further experimented with local labeling (Cheng & Lapata, 2016) and the results on CNN/DM are shown in Table 12. Local (soft) labels sentences with max-min scaled ROUGE scores (which are soft). In contrast, top3 and top5 create discrete scores via assigning positive labels to the top and top5 sentences, and all other sentences are given negative labels. As we can see, local labeling schemes are inferior to O REO , underscoring the need to exploit summary-level information for sentence annotation.
[Table]
Cross-Lingual Extractive Summarization We further initialize B ERT S UM with mBERT (Devlin et al., 2019).
As we can in Table 13, M B ERT S UM ﬁnetuned with greedy labels shows inferior performance across languages.
Nevertheless, O REO leads to substantial performance gains on both German and Turkish (we observe a similar trend when B ERT S UM initialized with XLM-R).
[Table]
Supervised Abstractive Summarization We show extended abstractive results in Table 14.
The ﬁrst block shows performance of vanilla GS UM
(Dou et al., 2021) which uses greedy extractive oracles as guidance during training.
During inference, we compare to three types of extractive guidance produced by B ERT S UM trained with greedy, beam, and O REO labels.
Despite the training-testing discrepancy in guidance, extractive guidance with O REO labels helps generate better downstream abstractive summaries.
To further validate the effectiveness of O REO on the optimization of GS UM [Table] we trained GS UM with
O REO O RACLE , and the results are shown in the second block.
As we can see, adopting O REO guidance during training further boosts system performance, while the system using O REO for both training and testing achieves the best results, i.e., 0.37 R OUGE -L improvement over GS UM . [Table]
Performance on Other Metrics Despite being popular in summarization, R OUGE has its weaknesses in summary evaluation.
Therefore, we further evaluated the generated summaries on CNN/DM using other recently proposed metrics: R OUGE - WE (Ng & Abrecht, 2015), M OVER S CORE (Zhao et al., 2019), B ERT S CORE (Zhang et al., 2020b), and S UMMA QA (Scialom et al., 2019).
We show the results in Table 15.
R OUGE - WE extends R OUGE using soft lexical matching based on the cosine similarity of Word2Vec.
We report the average of N-gram results ( N = { 1 , 2 , 3 } ) for R OUGE - WE . B ERT S CORE computes similarity scores between reference tokens and generated tokens but with BERT embeddings. M OVERS CORE uses Word Mover’s Distance to measure the semantic distance between a reference and a hypothesis summary. S UMM QA is a QA-style metric which was initially proposed to generate questions from source documents.
Following the evaluation of Papalampidi & Lapata (2022), we apply the question generation model to reference summaries to obtain more relevant and informative questions.
As we can see, O REO achieves better performance across metrics for both extractive and abstractive summarization.
G S YSTEM O UTPUT
Document :
The largest single high-deﬁnition map of mysterious dark matter has been produced.
It is the ﬁrst in a series of maps of the cosmos that will eventually allow a 3D view of dark matter across one eighth of the night sky.
And the map should allow astronomers to study how galaxies formed in the universe.
University of Manchester researchers have revealed an HD dark matter map (shown).
It shows clumps of mystery particles across 0.4 per cent of the sky.
The goal is to eventually map 12.5 per cent over ﬁve years.
Red here shows more dark matter, and blue shows less.
The moon is shown top left for scale.
A team from the University of Manchester, led by Dr Sarah Bridle, has spent the past two years measuring the shapes of galaxies used to construct the map.
And the map was released today at the April meeting of the American Physical
Society in Baltimore, Maryland.
It was created using data captured by the Dark Energy Camera, a 570-megapixel imaging device that is the primary instrument for the Dark Energy Survey (DES) led by Fermilab in the US...
Reference Summary : University of Manchester researchers reveal HD dark matter map.
It shows clumps of mystery particles across 0.4 per cent of the sky.
The goal is to eventually map
5 per cent over ﬁve years.
Results could help reveal how galaxies form in the universe
Extractive Summarization with B ERT S UM
Greedy : University of Manchester researchers have revealed an HD dark matter map (shown).
It shows clumps of mystery particles across 0.4 per cent of the sky.
It is the ﬁrst in a series of maps of the cosmos that will eventually allow a 3d view of dark matter across one eighth of the night sky.
Beam : University of Manchester researchers have revealed an HD dark matter map (shown).
The goal is to eventually map 12.5 per cent over ﬁve years.
It is the ﬁrst in a series of maps of the cosmos that will eventually allow a 3d view of dark matter across one eighth of the night sky.
O REO : University of Manchester researchers have revealed an HD dark matter map (shown).
It shows clumps of mystery particles across 0.4 per cent of the sky.
The goal is to eventually map 12.5 per cent over ﬁve years.
Abstractive Summarization with GS UM
Greedy : University of Manchester researchers have revealed an HD dark matter map.
It shows clumps of mystery particles across 0.4 per cent of the sky.
The goal is to eventually map 12.5 per cent over ﬁve years.
It is the ﬁrst in a series of maps of the cosmos that will eventually allow a 3D view of dark matter across one eighth of the night sky.
And the map should allow astronomers to study how galaxies formed in the universe.
Beam : University of Manchester researchers have revealed an HD dark matter map.
It shows clumps of mystery particles across 0.4 per cent of the sky.
The goal is to eventually map 12.5 per cent over ﬁve years.
It is the ﬁrst in a series of maps of the cosmos that will eventually allow a 3D view of dark matter across one eighth of the night sky.
And the map should allow astronomers to study how galaxies formed in the universe.
O REO : University of Manchester researchers have revealed an HD dark matter map of the cosmos.
It shows clumps of mystery particles across 0.4 per cent of the sky.
Red here shows more dark matter, and blue shows less.
The goal is to eventually map 12.5 per cent over ﬁve years.
And the map should allow astronomers to study how galaxies formed in the universe.
creates
H
RESERVATION
[Figure]