Exploring Neural Models for Query-Focused Summarization
Abstract
Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization.
While recently released datasets, such as QMSum or AQuaMuSe, facilitate research efforts in QFS, the field lacks a comprehensive study of the broad space of applicable modeling methods.
In this paper we conduct a systematic exploration of neural approaches to QFS, considering two general classes of methods: two-stage extractive-abstractive solutions and end-to-end models.
Within those categories, we investigate existing models and explore strategies for transfer learning.
We also present two modeling extensions that achieve state-of-the-art performance on the QMSum dataset, up to a margin of 3.38 ROUGE-1, 3.72 ROUGE2, and 3.28 ROUGE-L when combined with transfer learning strategies.
Results from human evaluation suggest that the best models produce more comprehensive and factuallyconsistent summaries compared to a baseline model.
Code and checkpoints are made publicly available: https://github.com/ salesforce/query-focused-sum .
Introduction
Text summarization aims at transforming long documents into short snippets that contain only the most important information from the source document.
The field has seen substantial progress driven by the availability of large-scale models pre-trained on vast amounts of data (Devlin et al., 2019; Lewis et al., 2020), the development of summarizationspecific pre-training strategies (Zhang et al., 2020;
Zhao et al., 2020), and computationally efficient neural architectures (Zaheer et al., 2020).
The majority of recent research efforts in text summarization assume an unconstrained setting in which models are given only a source document as input and are expected to generate a general summary covering the salient aspects from the source.
The performance of such models has been evaluated on benchmark datasets spanning various domains: news articles (Nallapati et al., 2016;
Narayan et al., 2018; Fabbri et al., 2019a), legal documents (Sharma et al., 2019), scientific writing (Cohan et al., 2018), or creative writing (Kry ´ sci´nski et al., 2021; Chen et al., 2021).
However, it has been shown that summarization in an unconstrained setting is an ill-defined task where multiple generated summaries are equally relevant (Kryscinski et al., 2019).
This in turn hinders the ability to evaluate and understand the models’ content selection capacity.
In addition, such generic summarization models lack control mechanisms that would allow end users to customize summaries to their particular needs and expectations.
Query-focused summarization (QFS) is a subtask within text summarization that focuses on generating summaries where the summary content is tailored to a user-specified query that is passed alongside the source document as input to the model.
Each source document can be associated with multiple unique queries inquiring about different information from that document.
In this setting, end users are enabled to explicitly specify their preferences for the summary, and the relevance of the output summary may be evaluated more precisely with respect to the input query.
Research on this task has been accelerated by the recently introduced highquality datasets, such as QMSum (Zhong et al., 2021b) and
AQuaMuSe (Kulkarni et al., 2020).
In this work we conduct a systematic, exploratory study of different approaches to queryfocused text summarization, considering both twostep and end-to-end neural methods.
We present two models, R EL R EG and S EG E NC , which achieve state-of-the-art ROUGE scores on the QMSum dataset, up to a margin of 3.38 R-1, 3.72 R-2, and
3.28 R-L when combined with transfer learning methods.
The R EL R EG model uses a two-step approach to solving the problem, where the first step extracts content relevant to the given query and the next step synthesizes the extracted fragments into a coherent summary.
The S EG E NC method follows an end-to-end framework in which individual document segments are separately encoded to avoid the computational bottleneck of long input documents, and the decoder jointly attends to all encoded segments when producing the summary.
Through quantitative studies, we compare our models with other baselines and discuss the trade-offs of the end-to-end methods and pipelined approaches.
We also perform human evaluation to understand the qualitative differences between the models.
Together with this manuscript, we share the code base and model checkpoints to enable future research in this area.
Related Work 2. Query-Focused Summarization
Query-focused summarization aims to generate a summary of a given text conditioned upon a query.
Initial work in this area centered around unsupervised extractive approaches (Wan et al., 2007; Litvak and Vanetik, 2017) due to the limited availability of task-specific training data (Dang, 2005).
More recent work has taken advantage of the relationship between query-focused summarization and the more data-rich task of question answering for extractive summarization (Egonmwan et al., 2019), reranking documents within a retrieval pipeline (Su et al., 2020), and abstractive summarization (Su et al., 2021; Baumel et al., 2018; Xie et al., 2020).
Xu and Lapata (2020) introduce a pipeline consisting of a relevance estimator filter followed by query-focused evidence and centrality estimators, while other work converts generic summarization dataset to query-focused training data (Xu and Lapata, 2021a) or performs latent query modeling (Xu and Lapata, 2021b).
Recently, several query-focused summarization datasets have been introduced, which can be further divided into short-document datasets, whose source document length does not exceed the input limits of standard pretrained models, and longdocument datasets.
Within short-document, queryfocused summarization, AnswerSumm (Fabbri et al., 2021c) is composed of summaries of answers to queries from StackExchange forums, while WikiHowQA (Deng et al., 2020) proposes the task of answer selection followed by the summarization of individual response articles to queries from the how-to site WikiHow.
Within long-document summarization, WikiSum (Liu et al., 2018a) consists of Wikipedia article titles as queries, the first paragraph of the article as the summary, and documents referenced by the article as the input.
AQuaMuSe (Kulkarni et al., 2020) is a queryfocused multi-document summarization dataset with user-written queries and human-verified longanswer summaries from the Natural Questions dataset (Kwiatkowski et al., 2019), and QMSum (Zhong et al., 2021b) is a manually-curated dataset for query-focused dialog summarization.
QMSum and AQuaMuSe are of particular interest to our study due to the combined challenges of queryfocused and long-document summarization and the presence of high-quality, curated query-summary pairs.
Recent work on QMSum has introduced taskspecific denoising objectives for meeting summarization (Zhong et al., 2021a), generated final fine-grained summaries based on multiple coarsegrained steps (Zhang et al., 2021a), and treated the extractive text of an extractive-abstractive model as a latent variable (Mao et al., 2021).
Zhang et al.
(2021b) analyze the challenges of long dialogue summarization such as the input length, the role of queries, and domain adaptation.
Our work builds on QA-motivated methods and presents two approaches yet to be applied in query-focused summarization that each achieve state-of-the-art results, including a two-step model and an end-to-end model.
2. Long Document Summarization
Long document summarization addresses the setting where source document length exceeds the input limits of standard pre-trained models.
Approaches to this task can largely be divided into two categories: two-step extractive-abstractive frameworks, which first extract a subset of the text as input to an abstractive model, and end-to-end models, which process the input within a single model.
The two-step pipeline has been applied to topic-focused Wikipedia summarization (Liu et al., 2018b; Liu and Lapata, 2019; Perez-Beltrachini et al., 2019), low-resource summarization (Bajaj et al., 2021), and single-document summarization
Chen and Bansal (2018).
End-to-end approaches address the input-length problem using sparse- attention models.
Beltagy et al.
(2020) introduce the Longformer, consisting of local attention as well as global attention between select input tokens.
Other approaches make use of dynamic attention mechanisms (Zhao et al., 2020; Manakul and Gales, 2021; Cui and Hu, 2021), sliding window strategies (Liu and Chen, 2021), and other mechanisms to introduce sparsity into the model (Huang et al., 2021; Liu et al., 2021).
Izacard and Grave (2021) concatenate the outputs of multiple encoders as input to a generator component for the task of open domain question answering.
In our work we build on these models for query-focused summarization and perform extensive hyperparameter ablations, achieving state-of-the-art results over other twostep and end-to-end models.
Methodology
We present existing methods and propose modeling extensions to address the challenges of queryfocused summarization.
3. Two-Step Approaches
Two-step approaches consist of an extractor model, which extracts parts of the source document relevant to the input query, and an abstractor model, which synthesizes the extracted segments into a final summary.
We consider score-and-rank extractor models, which first score each source passage for relevance to the query and then rank the passages in descending order of relevance, with the concatenated and truncated results passed to the abstractor.
In this work we present two types of scoring models: single-encoder models and dualencoder models, which we describe below.
All two-step approaches share the same abstractor, a
BART-large model.
3.1. Single-Encoder Models
Single encoder models concatenate a query and source passage as input to the scoring function that produces the similarity score.
Those models benefit from full cross-attention between query and passage, resulting in richer data representations.
M A RGE (Xu and Lapata, 2021a) is a singleencoder, Ma sked R OU GE extractor that aims to improve upon low-resource query-focused summarization by synthesizing query-focused data from more resource rich, generic summarization datasets.
This model is trained to predict the relevance of each passage in the source document with respect to a query, where the proxy for relevance is the
ROUGE overlap between the passage and the reference summary.
For training on generic summarization datasets, M A RGE uses pseudo-queries that are created by masking content words in the reference summaries.
When performing inference using real queries, certain query words (e.g., wh-words) are masked to better align the queries to the pseudo-queries from the training process.
Following Xu and Lapata (2021a), we apply M A RGE trained for masked relevance prediction on Multi-News (Fabbri et al., 2019b) without training on our target dataset.
R EL R EG
Motivated by the retrieval component of M A RGE, we propose the R EL R EG (RELevance REGression) model, which trains a relevance prediction model directly on QFS data using the original, non-masked query.
Like M A RGE, this model is trained to predict the ROUGE overlap between a source passage and the reference summary, using only the passage and query as input.
A single-encoder model jointly encodes the delimiterseparated query and passage, and the final layer of the model outputs the predicted relevance value.
3.1. Dual-Encoder Models
Dual-encoder models separately encode a query and source passage before calculating the cosine similarity between the embeddings to compute the relevance score.
This class of models offers computational benefits, as passage embeddings may be precomputed and stored for a given input, while the single-encoder model must be run over all passages should a new query be introduced.
DPR (Karpukhin et al., 2020) is a dual-encoder model that separately encodes queries and passages into an embedding space optimized for calculating semantic similarity between the two, showing improved results over traditional vector-space models.
We fine-tune a DPR extractor model directly on the target dataset.
As opposed to other locators that optimize with respect to the continuous ROUGE overlap, DPR uses the ROUGE score between the passage and reference summary to identify binary positive and negative passages and optimizes the negative log likelihood of the positive passages.
R EL R EG TT
(RELevance REGression Two
Tower) is a more computationally-efficient version of R EL R EG that uses a dual-encoder architecture to predict ROUGE-based relevance scores.
This model is implemented with a backbone architecture of Sentence-BERT (Reimers and Gurevych, 2019), using a shared-parameter encoder for each of the query and passage and a special token appended to each input that identifies it is as either query or passage, following the suggested best practices of
Reimers and Gurevych (2019).
The final output for the model is based on the inner product of the pooled embeddings for the query and passage.
3. End-to-End Approaches
Two-step pipelines depend on the strength of the retrieval component, and may still fail to capture all relevant content despite an ideal retriever, due to length limitations of the generation component.
This motivates our experiments on end-to-end models that can incorporate longer input texts.
BART (Lewis et al., 2020)
As a baseline endto-end model, we consider BART, an encoderdecoder Transformer model pre-trained using a denoising objective.
BART is composed of a bidirectional encoder module and an autoregressive decoder model that attends to the encoder’s final layer outputs.
Due to the quadratic memory complexity of the encoder’s full self-attention mechanism, the model input size is limited to 1024 tokens.
In our experiments, we prepare the input to BART by concatenating the query, a delimiter token, and the source document, and then truncating the combined text to the model’s input size.
LED
To circumvent the input size limitations of the BART model, we include the Longformer
Encoder-Decoder (Beltagy et al., 2020) (LED) in our study LED replaces the quadratic selfattention mechanism of traditional Transformers with a memory-efficient version that combines local attention with sparse global attention.
The architecture allowed us to run experiments with input sizes up to 16384 tokens.
Based on insights from the original work on tuning the model to the QA task, we configure the global attention mechanism to span the entire query.
S EG E NC
We also consider a simpler form of sparse attention in the encoder based solely on windowed local attention, combining elements of
LED with Fusion-in-Decoder (FiD) (Izacard and
Grave, 2021), a model for open-domain question answering.
In our Segment Encoder (S EG E NC ) model, the source document is split into fixedlength overlapping 1 segments, each of which is separately appended to the query and encoded using a standard Transformer model.
Similar to FiD, these encodings are then concatenated into a single embedding sequence and passed to a decoder model that generates the summary.
Since there is no cross-attention between the encoded segments, the attention mechanism scales linearly in the number of segments and hence the length of the source document.
Nonetheless, the decoder can attend to all encoded segments jointly, enabling the encoderdecoder architecture to operate in an end-to-end fashion.
This model is motivated by two hypotheses: 1) query-relevant sections within a source document are often small enough to be processed by standard Transformer models (e.g. 1024 tokens), and 2) each query-relevant section may be understood independently of other sections, removing the need for cross-attention between the segments.
3. Data
We analyze our methods on two high-quality queryfocused, long-document datasets.
QMSum (Zhong et al., 2021b) is a query-focused dialogue summarization dataset consisting of 1, query-summary pairs over 232 meetings from product design, academic, and political committee meetings, all conducted in English.
QMSum also includes additional annotations such as topic segmentations and highlighted text spans associated with reference summaries.
We leverage the provided span annotations to run oracle experiments.
We focus our analysis on QMSum due to the availability of prior work as points of comparison.
AQuaMuSe (Kulkarni et al., 2020) is a queryfocused multi-document summarization dataset consisting of 5,519 query-long answer summary pairs from the Natural Questions questionanswering dataset (Kwiatkowski et al., 2019) and associated input documents from the Common
Crawl 2 .
Input documents for the original dataset were selected based on embedding similarity with respect to the summary, and hyperparameters can be chosen to control the level of semantic overlap between the input document set and the summary.
Data replication details are found in the Appendix.
We use AQuaMuSe to examine the generalizability of our QMSum results.
Experiment Setup Implementation
Models were implemented using the PyTorch (Li et al., 2020) and Huggingface (Wolf et al., 2019) libraries.
Model weights were initialized from pre-trained checkpoints available through the Huggingface Model Hub 3 .
All BART models were based on the facebook/bart-large checkpoint, the LED-model was based on the allenai/led-largecheckpoint, which itself is based on BART-large .
Training & Inference
Models were trained for 10 epochs with final checkpoints selected based on the average of ROUGE- { 1 , 2 , L } ( R-1 , R-2 , RL ) scores achieved on the validation set.
Gradient checkpointing (Chen et al., 2016) was used for the
LED and S EG E NC models to reduce the memory footprint.
Model outputs were decoded using beam search with 4 beams.
To ensure high consistency of results, all experiments in §4 were repeatedtimes with results averaged across runs.
Evaluation
Models were automatically evaluated using the ROUGE- { 1 , 2 , L } metrics (Lin, 2004) included in the SummEval toolkit (Fabbri et al., 2021b).
Models were also manually evaluated by hired human annotators.
Annotators were hired through the Amazon Mechanical Turk platform.
Workers were selected from English speaking countries and offered an hourly rate of approximately 12 USD.
The study was conducted onmodel generated examples chosen at random from the test set of QMSum.
Model Exploration
In this section, we first analyze the effects of modelspecific architectural and hyperparameter choices on the performance of two-stage (§4.1) and end-toend models (§4.2).
Next, we study the task-specific knowledge transfer capabilities of different pretraining strategies in §4.3.
Lastly, we conduct a final evaluation and comparison of all discussed models in §4.4.
All experiments and analyses presented in this section were conducted on QMSum.
4. Two-Stage Approaches
For two-stage models, we first focus on evaluating the extractor component and comparing performance to baseline heuristics.
We quantify extractor performance using two metrics: 1) lexical overlap between the extracted utterances and reference summaries, computed using R-1, R-2, and R-L metrics, 2) span overlap between the extracted and golden spans included with QMSum represented by
Precision and Recall scores, with results shown in
Table 1.
In both cases, we first order utterances of the conversation according to the scores assigned by the extractor models, then concatenate the utterances and finally truncate the result to 1024 tokens (excluding the space reserved for the query) to mimic the input length limits of downstream abstractor models; we present those numbers as the
All columns in the table.
For the lexical overlap, we also show the scores for the best 1 ( Top-1 ),( Top-5 ), and 15 ( Top-15 ) utterances.
The results show that the best-performing extractor model is R EL R EG closely followed by R ELR EG TT in the Top-1 evaluation and DPR in the
Top-5 , Top-15 , and All cases.
We note that both the R EL R EG and R EL R EG TT models tend to select longer utterances than the other extractors; the regression-based training mirrors the ROUGE overlap score which favors longer, more informative utterances.
However, despite their strong performance in extracting top-matching utterances, the results also expose a considerable gap between model-based approaches and human annotations when considering the entirety of extracted spans.
This shows a promising topic for future work in this matter.
We also notice that despite the simplicity of the L EAD heuristic, which extracts the first k utterances in their original order, it remains competitive with the data-driven extractor models when we consider the All case.
An extended version of this study, which includes the lexical overlap between extracted spans and input queries is presented in
Table 8 in the Appendix.
Next, we analyze how the performance of the extractor components carries over to the final summarization task.
For the best-performing model, we additionally test the effect of varying the input segment size used during training and inference between 256 and 512 tokens.
Validation-set results for all models are reported in Table 2.
We find that DPR slightly outperforms R ELR EG TT for dual-encoder models.
Among single-encoder models, R EL R EG outperforms
M A RGE by over a full R-1 point, which may explained by R EL R EG using more direct supervision based on an in-domain query, rather than creating
[Table] [Table]
synthetic queries from an external dataset using masking.
We find that the single-encoder R ELR EG outperforms the best dual-encoder model; the cross-attention term in the single-encoder R ELR EG model allows it to better attend to the query when determining relevance.
Intuitively, the ordering of results corresponds to the span overlap recall with the gold spans; the ability of the extractor to select produce high-recall rankings directly affects abstractor performance.
We see that increasing the input segment length used in training and inference for R EL R EG improves at 256 tokens but decreases at 512 tokens, suggesting that a balance is found between including additional context for ranking versus enabling a greater number of shorter segments that may capture more diverse content from the source.
4. End-to-End Approaches
We explore hyperparameter choices for two endto-end architectures described in §3.2: the Longformer Encoder-Decoder (LED) and Segment Encoder (S EG E NC ).
For both models, we consider different choices for input size (4096, 8192, or
[Table]
tokens) and attention window size 4 (256, 512, or 1024 tokens).
For S EG E NC , we also consider two different segmentation strategies: overlapping segments (50% overlap) and disjoint segments.
Validation set results for both models and a baseline
BART model are reported in Table 3.
We notice that both the LED and S EG E NC benefit from increasing the input size and perform best with the input limit set to 16,384 tokens.
The optimal attention window for LED is 1024, while
S EG E NC performs best with an attention window
[Table]
of 512 tokens
For S EG E NC , using overlapping segments improves performance compared to using disjoint segments, suggesting that the additional context provided by the former approach is helpful for locating relevant content.
The S EG E NC model achieves the highest performance out of the endto-end architectures with ROUGE scores of 37.
R-1 , 12.47 R-2 , and 32.95 R-L on the validation set.
The results also highlight that while the
LED model matches or slightly outperforms the
BART baseline for higher maximum input and window sizes, it performs substantially worse than
S EG E NC .
This observation is consistent with prior findings on the QMSum dataset (Zhang et al., 2021b).
One possible explanation for the lower performance of LED relative to S EG E NC is that
LED must adapt its parameters for a global attention mechanism that is absent from the backbone BART encoder model, whereas S EG E NC relies solely on local self-attention that is aligned with the backbone model.
This may be particularly relevant to QMSum given its relatively small size.
Practitioners may wish to consider the computational cost and efficiency of various hyperparameter settings.
Computational complexity increases with both input length and attention window size (since attention grows quadratically in attention-window size).
Complexity is also greater with the overlapping segment strategy compared to the disjoint segment strategy for the S EG E NC model, due to the greater number of resulting segments that are passed through the encoder and decoder modules.
4. Task-Specific Transfer
Having determined the best-performing models
[Table]
we examine whether performance can be further improved by fine-tuning a model that has already been fine-tuned for a different summarization task.
We conduct this study using the end-to-end BART on 1024 tokens, as this model is the backbone, al-
[Table]
beit in varying ways, of both our two-step and end-to-end models.
We test the transferring capabilities of models trained on the news summarization task from CNN/DailyMail
(Nallapati et al., 2016), which performed best among non queryfocused datasets in Zhang et al.
(2021b).
We also explore transferring from the previously-mentioned query- and topic-focused summarization tasks: AnswerSumm, AQuaMuSe, WikiHowQA, and WikiSum.
We compare to fine-tuning from the original
BART checkpoint, with results shown in Table 4.
We find that transferring from any of the tasks improves over no transfer in R-1 and R-L. Transferring from any of the constrained, query-focused tasks outperforms transferring from unconstrained news summarization.
Furthermore, transferring from WikiSum outperforms transfer from other datasets, which aligns with other work that shows the generalizability of Wikipedia as a source of data for task transfer (Fabbri et al., 2021a).
4. Final Results
We now measure the test set performance of the best-performing architectures from §4.1 and §4.
in combination with the optimal transfer-learning approach from §4.3.
Results are presented in Table 5 along with baseline models.
We find that R EL R EG and S EG E NC outperform existing state-of-the-art models by a substantial margin, and that initializing the model from the
[Table]
Summaries were evaluates across four dimensions: fluency ( Flu. )
, relevance ( Rel. ), completeness ( Comp. ), and factuality ( Fact. ).
Wikisum-fine-tuned checkpoint further improves performance, with the best model exceeding current state-of-the-art performance by a difference of 3.38 R-1, 3.72 R-2, and 3.28 R-L. Comparing the best models from each category, we find that the end-to-end approach outperforms the two-stage.
Within the two-stage dual-encoder models, R ELR EG TT outperforms DPR on the test set despite the slightly worse performance on the validation set.
We attribute this variation to the small size of the validation set, and our other findings remain consistent across validation and test sets.
The single-encoder R EL R EG outperforms the best dualencoder model, with R EL R EG -W improving upon the current state-of-the-art performance by a difference of 2.03 R-1, 3.10 R-2, and 2.18 R-L.
Further Analysis
In this section we conduct further analysis of the best performing models from Section 4.
First, we offer additional insights into the performance of those models on the QMSum dataset through a human-based study.
Next, we discuss the generalization abilities of those models by running experiments on the AQuaMuSe dataset.
5. Human Evaluation
To gain a better understanding of the performance of the models on the QMSum dataset, human judges were hired and asked to assess the quality of generated summaries.
Summaries were evaluated across four dimensions: 1) fluency , measuring their grammatical quality, 2) relevance , assessing their relevance to the input query, 3) completeness , evaluating their comprehensiveness considering the input conversation and query, and 4) factuality , measuring their factual consistency with respect to the conversation.
Scores were assigned on a Likert scale from 1 to 5 (best), where each example was evaluated by 3 judges with the final score averaged.
Results are presented in Table 6.
[Table]
We find that the R EL R EG -W and S
EG E NCW models achieved comparable performance across all of the evaluated dimensions, with summaries generated by S EG E NC -W rated as slightly more complete.
The BART baseline was rated highest in the fluency dimension, however, it was substantially outperformed by both of the introduced models on completeness and factuality.
One possible explanation for the slightly lower fluency scores for the R EL R EG -W and S EG E NC -W models is that they are better able to retrieve content from the source, which itself may have low fluency due to its conversational nature.
The results also highlight a gap between the performance of existing models and perfect scores, which shows that there is potential for improvement in future work.
5. Dataset Generalization
To test that the automated evaluation results generalize beyond the QMSum dataset, we trained and evaluated the best-performing models on AQuaMuSe, another high-quality dataset for QFS that includes long documents (§2.1, §3.3).
Test-set performance for the best-performing two-stage and endto-end models, along with a baseline BART model, are shown in Table 7.
Results are consistent with those for the QMSum dataset (Table 5), with the best end-to-end model (S EG E NC -W) outperforming the best two-stage model (R EL R EG -W), and both outperforming the baseline (BART) model.
Conclusion
In this work, we conducted an exploratory study of neural models for query-focused summarization.
We studied two categories of models: two-stage and end-to-end, and presented two architectures,
R EL R EG and S EG E NC , both of which improve
ROUGE performance over prior state of the art by a substantial margin.
We also explored taskspecific transfer learning, which further improved model performance.
Besides model performance, we discussed issues of computational efficiency that practitioners may factor into their modeling choices.
Finally, we conducted a human study suggesting that the summaries produced by the bestperforming models are more factually correct and complete than a baseline model by a substantial margin.
We hope that the analysis and modeling contributions of this paper will be a resource for future research on query-focused summarization.
Ethical Considerations
Dataset Biases
QMSum and AQuaMuSe contain meeting transcripts and documents in English and thus mainly represent the culture of the Englishspeaking populace.
Political or gender biases may also exist in the dataset, and models trained on these datasets may propagate these biases Additionally, the pretrained BART model carries biases from the data it was pretrained on.
We did not stress test these models for biases and request that the users be aware of these potential issues in applying the models presented.
Crowdsourcing Protocols
Workers were compensated $1 per example, calibrated to equal a $12/hour payrate.
We use the following qualifications to recruit MTurk workers with good track records: HIT approval rate greater than or equal to 97%, number of HITs approved greater than or equal to 10000, and located in one of the following English native-speaking countries: Australia,
Canada, New Zealand, United Kingdom, United
States.
Misuse Potential and Failure Mode
When properly used, the summarization models described in this paper can be time-saving.
However, the current model outputs may be factually inconsistent with the input documents, and in such a case could contribute to misinformation on the internet.
This issue is present among all current abstractive summarization models and is an area of active research.
Environmental Cost
The experiments described in the paper primarily make use of A100 GPUs.
We typically used a single GPU per experiment, and the experiments may take up to a day when repeating across random seeds.
The largest backbone model used, BART-Large , has 400 million parameters.
While our work required extensive experiments, future work and applications can draw upon our insights and need not repeat these comparisons.
Acknowledgements
We thank Divyansh Agarwal for his insights on existing model architectures and assistance with training DPR.
We thank Semih Yavuz for providing the initial implementation of the Fusion-in-Decoder model.
A Appendix Locator Model Parameters
For M A RGE experiments, we apply the original fine-tuned BERTbase checkpoint from Xu and Lapata (2021a), while for DPR, we fine-tune a BERT-base model for both query and passage encoders following
Karpukhin et al.
We report results for R EL R EG fine-tuned from an Electra-large checkpoint (Clark et al., 2020).
For a fair comparison with other metrics, we also fine-tuned R EL R EG from a BERT-base checkpoint.
This version still outperformed DPR by about a point in R-1, R-2, and R-L, demonstrating the advantage of this locator approach beyond the chosen base model.
We apply R EL R EG TT fine-tuned from a distilled RoBERTa base (Liu et al., 2019) checkpoint initially fine-tuned for the task of entailment.
This approach of continuing fine-tuning from an entailment checkpoint is suggested by the sentence transformers library (Reimers and Gurevych, 2019).
We also experimented with fine-tuning the R ELR EG TT model from BERT-base and Electra-large checkpoints, but these locators did not perform better in initial experiments. [Table]
Summarization
Model
Parameters
In all experiments described in this work, the LED model was initialized from the allenai/led-largecheckpoint.
Two model hyperparameters, maximal input size and attention window size , were chosen through a hyperparameter search with candidate models selected based on their performance on the validation set.
Best hyperparamters were found to be: 16384 maximum input size, and 1024 attention window size.
LED models were trained for 10 epochs, with a batch size 1 , gradient accumulation set to 4 steps, and learning rate set to 0.000005 .
The S EG E NC model was initialized from the facebook/bart-large checkpoint.
The model hyperparameters, maximal input size and attention window size , were chosen through a hyperparameter search with candidate models selected based on their performance on the validation set, with results reported in the paper.
Best hyperparamters were found to be:maximum input size, and 512 attention window size.
The S EG E NC models were trained forepochs, with a batch size of 1 and learning rate set to 0.000005 .
QMSum Details
QMSum contains 1,808 querysummary pairs in total, with a train/validation/test split of 1257/272/281.
It is made available through an MIT license 5 , which aligns with our use for research purposes.
Non-identifying names are used in place of real names.
AQuaMuse Details
We experiment the V3, abstractive version of AQuaMuse, consisting ofquery-summary pairs, with a train/validation/test split of 5566/596/734.
The original AQuaMuse paper reported results on V2 of the dataset, which contains a slightly different input document set due to variations in the semantic overlap threshold used to retrieve documents.
Some input documents could not be retrieved due to differences in the Common
Crawl index used; we use the cleaned, reproduced version of the C4 dataset (Raffel et al., 2020) from the Common Crawl made available by AI2 6 .
We kept examples for which all input documents were found, which resulted in a dataset of 6896 examples.
The natural language questions it contains are made available through an Apache 2.0 license 7 , which aligns with our use for research purposes.
This dataset uses publicly available entities from
Wikipedia.
B Human Annotation Interface
The instructions shown to the annotators during human studies are presented in Figure