Towards Improving Faithfulness in Abstractive Summarization
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite the success achieved in neural abstractive summarization based on pretrained language models, one unresolved issue is that the generated summaries arenot always faithful to the input document.
There are two possible causes of theunfaithfulness problem: (1) the summarization model fails to understand or capturethe gist of the input text, and (2) the model over-relies on the language modelto generate fluent but inadequate words.
In this work, we propose a Faithfulness
Enhanced Summarization model (FES), which is designed for addressing thesetwo problems and improving faithfulness in abstractive summarization.
For thefirst problem, we propose to use question-answering (QA) to examine whether theencoder fully grasps the input document and can answer the questions on the keyinformation in the input.
The QA attention on the proper input words can alsobe used to stipulate how the decoder should attend to the source.
For the secondproblem, we introduce a max-margin loss defined on the difference between thelanguage and the summarization model, aiming to prevent the overconfidence of thelanguage model.
Extensive experiments on two benchmark summarization datasets,
CNN/DM and XSum, demonstrate that our model significantly outperforms strongbaselines.
The evaluation of factual consistency also shows that our model generatesmore faithful summaries than baselines 1 .
Introduction
In recent years, text generation has made impressive progress [1, 2, 3].
The abstractive summarizationtask, aiming to produce a concise and fluent summary that is salient and faithful to the sourcedocument, has become a research hotspot due to its broad application prospect.
The prevalence ofpretrained transformer language models (LM) [4, 5] has largely improved the fluency and salienceof generated summaries.
However, studies [6, 7] showed that many summarization models sufferfrom unfaithfulness problem, i.e., the generated summary is not entailed by the information presentedin the source document.
Durmus et al.
[8] highlighted two notions of the unfaithfulness problem insummarization: one is the manipulation of information presented in the input document ( intrinsicerrors ), and the other is the inclusion of information not inferable from the input ( extrinsic errors ).
The Intrinsic error problem is often caused by the failure of document level inference, which isnecessary for abstractive summarization.
Specifically, the summarization model has misinformationinferred from the input document because of an inadequate encoder that misunderstands the sourcesemantic information and a poor decoder that cannot fetch relevant and consistent content from theencoder.
Several recent summarization models were proposed from this perspective.
For example,
Wu et al.
[9] proposed a unified semantic graph encoder to learn better semantic meanings and agraph-aware decoder to utilize the encoded information.
Cao et al.
[10] used contrastive learning to help the model be aware of the factual information.
The second type of error, extrinsic error , isoften introduced by excessive attention paid to the LM, which ensures fluency while neglecting tosummarize the source document.
For example, a LM is inclined to generate the commonly-usedphrase “score the winner ” while the correct phrase is “score the second highest ” which is lessfrequently used.
This type of error has been studied in the neural machine translation task [11], buthas not been addressed in abstractive summarization.
To address these errors, we propose a novel Faithfulness Enhanced Summarization model (FES).
To prevent the intrinsic error problem, we design FES in a multi-task learning paradigm, i.e.,completing encoding-decoding for the summarization task with an auxiliary QA-based faithfulnessevaluation task.
The QA task poses an additional reasoning requirement on the encoder to have amore comprehensive understanding on the key semantic meanings of the input document and learnbetter representations than working only for summarization.
The QA attention on the key entities ofthe input can also be used to align the decoder state with the encoder outputs for generating a faithfulsummary.
To address the extrinsic error problem, we propose a max-margin loss to prevent the LMfrom being overconfident.
Concretely, we define an indicator of the overconfidence degree of the
LM.
The risk of outputting extrinsic error tokens with low prediction probabilities is mitigated byminimizing this overconfidence indicator.
We validate the effectiveness of our FES model by conducting extensive experiments on publicbenchmark CNN/DM [12] and XSum [13] datasets.
Experimental results demonstrate that ourfaithfulness enhanced summarization model has superior performance on the ROUGE scores andimproves the faithfulness of news summarization over several strong baselines.
Our main contributions can be summarized as follows.
(1) We propose a faithfulness enhancedsummarization model, which alleviates the unfaithfulness problem from the encoder side and decoderside.
(2) Concretely, we propose a multi-task framework to enhance the summarization performanceby automatic QA tasks.
We also propose a max-margin loss to control the overconfident problemof the LM.
(3) Experimental results demonstrate that our proposed approach brings substantialimprovements over the most recent baselines on benchmark datasets, and can also improve thefaithfulness of the generated summary.
Related Work
Abstractive Summarization.
In recent years, the research on text generation has made impressiveprogress [1, 14, 15], which promotes the progress of abstractive summarization.
The abstractivesummarization task generates novel words and phrases not featured in the source text to capture thesalient ideas of the source text [16].
Most works apply an encoder-decoder architecture to implicitlylearn the summarization procedure [17, 18].
More recently, applying pretrained language modelsas encoder [4, 19] or pre-training the generation process by leveraging a large-scale of unlabeledcorpus [20, 21] brings significant improvements.
Explicit structure modeling has also been shown tobe effective in summarization tasks.
For example, Jin et al.
[22] incorporated semantic dependencygraphs to help generate sentences with better semantic relevance, and Wu et al.
[9] came up with aunified semantic graph to aggregate relevant disjoint context from the input.
Fact Consistency for Abstractive Summarization.
Producing a summary that is entailed by theinformation presented in the source document is a key challenge in the summarization task, and lessprogress has been made on it.
Pioneer works [23, 24] incorporated fact descriptions or entailmentknowledge to enhance faithfulness.
More recently, Zhu et al.
[25] modeled the facts in the sourcearticle with knowledge graphs based on a graph neural network.
Aralikatte et al.
[26] introduced focusattention mechanism to encourage decoders to proactively generate tokens that are similar or topicalto the input document.
On the contrary, other works post-edit the generated summaries.
For example, Cao et al.
[27] addressed the factual inconsistent issue by identifying and correcting factual errors ingenerated summaries.
Different from previous works, we enhance the semantic understanding of thedocument with faithfulness evaluation as a direct signal and prevent the overconfidence of LM whichis not addressed before.
Multi-task Learning.
Multi-task learning is a learning paradigm in machine learning and it aims toleverage useful information contained in multiple related tasks to help improve the generalizationperformance of all the tasks [28].
There is a large quantity of natural language processing tasksformulated by multi-task learning, such as word segmentation, POS tagging, dependency parsing,
[Figure]
[Figure]
Summarizationmodeltraining
Faithfulness evaluation after training
[Figure]
and text classification [29, 30, 31, 32].
In this work, we apply multi-task learning to summarizationand question-answering tasks for faithfulness enhancement.
Methodology
3.1 Problem Formulation
For an input document X = { x 1 , ..., x n x } , we assume there is a ground truth summary Y =
{ y 1 , . . .
, y n y } . In our faithfulness enhanced setting, n q question answering pairs Q = { Q 1 , ..., Q n q }with corresponding answers A = { A 1 , ..., A n q } are also attached with X . In the training process,our model is given QA pairs and document-summary pairs.
It tries to extract answers A to thequestions and generate the summary Y .
In test stage, our model is given document X and questions
Q , and predicts the answers and summary.
The final goal is to generate a summary that is not onlyinformative but also consistent with document X .
Following, we introduce our proposed Faithfulness Enhanced Summarization model, which isgenerally built on Transformer [33].
The faithfulness enhancement is implemented from threeaspects: (1) Multi-task Encoder.
It improves the semantic understanding of the input document byexamining the quality of the encoded document representations for an auxiliary QA task.
The encodedrepresentation thus captures the key inputs for making faithful summary.
(2) QA Attention-enhanced
Decoder.
The attention from the multi-task encoder aligns the decoder with the encoder so that thedecoder can fetch more accurate input information to generate the summary.
(3) Max-margin Loss.
This is a loss orthogonal to the generation loss.
It measures the accuracy of the LM and prevents itfrom being overconfident in the generation process.
3.2 Multi-task Encoder
[Figure]
Transformer Encoder Questions Entity nodes: Sentence nodes:
Question nodes:
The multi-task encoder is designed for encoding the input document for both summarization and question-answering in anintegrated training process, as shown in Figure 1(b).
This isdifferent from the previous work that uses QA in the postgeneration stage for evaluating the faithfulness of the generatedsummaries [8, 7], as shown in Figure 1(a).
We bring the QAcloser to the encoder instead of leaving it for post-generatedsummary, and make the encoder be trained to accomplish the
QA and summarization task in the meantime.
This integratedtraining of a multi-task encoder includes faithfulness also as anoptimization objective, besides the summary generation quality.
The answers are key entities from the document so that QApairs focus on key information in the input.
As shown in Figure 2, we first apply the classic Transformer architecture to obtain token-levelrepresentations for the document and questions, denoted as H w ∈ R n w × d e and H u ∈ R n q × t q × d e ,where n w is the total number of tokens in the document, n q is the question number, t q is the tokennumber in a question, and d e is the feature dimension.
Then, we design the encoder to understand thequestion and the input document question from entity levels and sentence levels.
Encoding at Multi-level Granularity.
We build the encoder by organizing the representationlearning at different granularity levels.
We use entities as the basic semantic unit as they containcompact and salient information across the document, and the reading comprehension questionsfocus on entities.
Since a question is usually short, we create one node for each question.
We addbidirectional edges from the questions to sentence nodes, and from sentence to entity nodes.
Thesenodes act as the intermediary between sentences and enrich the cross-sentence relations.
Becausethe initial directed edges are insufficient for learning backward information, we add reverse edgesand self-loop edges to the graph following previous works [34].
We initialize node representationsfollowing the token level and word span level mean-pooling process [9].
Given the constructed graph with node features, we use graph attention networks [35] to update therepresentations of our semantic nodes.
We refer to ˜ h
i ∈ R d e , i ∈ { 1 , · · · , ( n e + n s + n q ) } as thehidden states of input nodes, where n e and n s are the number of entity nodes and sentence nodes,respectively.
The graph attention (GAT) layer is designed as follows:
[Equation]
where N i is the set of neighboring nodes of node i , W a , W b , W c , W d are trainable weights andα ij is the attention weight between ˜ h
i and ˜ h j .
Besides, we add a residual connection to avoidgradient vanishing after several iterations: h i =
˜ h
i + l i .
We iteratively use the above GAT layer andposition-wise feed-forward layer [33] to update each node representation.
The output entity featurematrix, sentence feature matrix, and question matrix, are denoted as H e ∈ R n e × d e , H s ∈ R n s × d e ,and H q ∈ R n q × d e , respectively.
Answer Selector for the QA task.
After fusing information from the question and the document,we can select entities from the document as the answer to the question.
Concretely, we apply themulti-head cross attention (MHAtt) between the question and the entities from the graph: h iqe =
MHAtt (cid:0) h ie , H q , H q (cid:1) to obtain question-aware entity representations, where i is the question index.
Based on the question-aware entity representations, we employ a feed-forward network (FFN) togenerate the entity extracting probabilities
A i = FFN ( h iqe ) ,
where A i =
( a i 1 , ..., a in e ) .
The QAobjective is to maximize the likelihood of all ground-truth entity labels ˆ a :
[Equation]
3.3 QA Attention-enhanced Decoder
A faithful decoder needs to attend to and fetch the important content from the encoder instead ofmixing the inputs.
We observe from §3.2 that the QA attentions on the key entities can be regarded asimportance signals indicating which entities should be included in the summary.
Hence, we proposea summary generator enhanced by QA attention.
Generally, the decoder state attends to the encoderstates with entities as intermediates, where the entity-level attention is guided by QA attentions.
Concretely, for each layer, at the t -th decoding step, we apply the self-attention on the maskedsummary embeddings E , obtaining u t .
The masking mechanism ensures that the prediction of theposition t depends only on the outputs before t .
Based on u t , we then compute the cross-attentionscores c et over entities.
[Equation]
In effect, the first attention layer captures contextual features of the decoded sequence, while thesecond incorporates entity information in c et .
Herein, we minimize the bidirectional Kullback-Leibler(KL) divergence between the QA attention A i and the summarization attention E t on the entities atthe t -th step, to help the summarization model learn what entities are important:
[Equation]
[Figure]
Note that this objective guides the cross-attention to capture correct contextual information ratherthan to only learn the QA attention distribution.
So we only employ it on parts of attention heads toavoid “overfitting” to the QA task.
We then use the entity-level attention to guide the selection ofsource tokens related to the key entities, by applying another MHAtt layer on source word sequence H w and c et :
[Equation]
This context vector v t , treated as salient contents summarized from various sources, is sent to an
FNN to produce the distribution over the target vocabulary, i.e., P t = Softmax ( FFN ( v t )) .
All thelearnable parameters are updated by optimizing the negative log likelihood objective function ofpredicting the target words:
[Equation]
3.4 Max-margin Loss
Previous works [36, 37] suggest that a poorly-informed decoder will neglect some source segments,function more as an open-ended LM, and thus will be prone to extrinsic errors.
Inspired by faithfulnessenhanced machine translation works [38, 39], we introduce a max-margin loss into the summarizationtask, for maximizing the difference of the predicted probability for each token of the summarizationmodel and LM as shown in Figure 3.
Concretely, we first define the margin between the summarizerand the LM as the difference of the predicted probabilities:
[Equation]
where X is the input document, and P LMt denotes the predicted probability of the t -th token of the
LM.
Note that the language model has no access to the input document, and only takes the decodedsummary prefix as input.
Intuitively, if m t is large, then the summarization model is apparentlybetter than the LM.
When m t is small, there are two possibilities.
One is that both the LM and thesummarization model have good performance, hence the predicted probabilities should be similar.
The other possibility is the LM is not good enough but overconfident, which leads to a summarizerwith poor performance.
Hence, we present the max-margin loss L m , which adds a coefficient to the margin:
[Equation]
where we abbreviate P t ( y t | y <t , X ) as P t .
The term (1 − m 5 t ) / 2 is a non-linear monotonicallydecreasing function in regard to m t , which ensures the optimization of maximizing m t .
We choose
Quintic function (fifth power) here as it is shown to be more stable [39].
The first factor (1 − P t )is for fitting the two possibilities we discussed above.
When P t is large, the summarization modellearns the y t well and does not need to pay too much attention on m t .
This is reflected by (1 − P t ) , asmall coefficient of m t .
On the other hand, when P t is small, it means that the summarizer needsto be better optimized, and a large coefficient (1 − P t ) enables the model is able to learn from themargin information.
The above four losses, L c , L s , L KL , and L m are orthogonal and can be combined to improvefaithfulness.
The overall architecture of the model can be found in the Appendix.
Experiment4.1 Experimental Setup
Datasets.
We demonstrate the effectiveness of our approach on two public datasets, CNN/DM and
XSum, which have been widely used in previous summarization works.
Both datasets are based onnews and consist of a large number of events, entities, and relationships that can be used to test thefactual consistency of summarization models.
Note that our summarization model is accompanied by a QA task.
Hence, we pre-construct QA pairsfor each case using QuestEval tool provided by Scialom et al.
Concretely, QuestEval first selectsa set of the named entities and nouns as answers from the source document.
Then, it uses a finetunedanswer-conditional question generation T5 [40] model to generate questions via beam search.
Toensure the quality of the QA pairs, we only select those questions for which the question-answeringmodel [7] gives the right answers.
Finally, we obtain 38 QA pairs for CNN/DM and 27 pairs for
XSum on average.
For training the summarization model, intuitively, we want the questions to focuson the key information in the input.
Hence, we select the pairs where the answers have the highest
ROUGE-L scores with the target summaries as oracle pairs.
A BART-based extraction model [21]is then trained to predict important answers (QA pairs).
Dou et al.
[41] showed that it brings morebenefits when using the oracle guidance in the training phase.
Hence, for the training dataset, we usethe oracle QA pairs, i.e., the first 8 pairs with the highest ROUGE scores as input.
For validation andtest, we use the pairs selected by the extraction model.
Baselines.
We first compare our model with recent factual-consistent summarization models: (1)
FASum [25] is a model that extracts and integrates factual relations into the summary generationprocess via graph attention.
(2) CLIFF [10] leverages reference summaries as positive data anderroneous summaries as negative data to train summarization systems.
We also compare our proposedmodel with recent abstractive summarization models: (3) BART [21] is a state-of-the-art abstractivesummarization model pretrained with a denoising autoencoding objective.
(4) PEGASUS [20] is apre-training large Transformer-based encoder-decoder models for summarization task.
(4) GSum [41]is a summarization framework that can take external sentence guidance as input.
(5) SimCLS [42]bridges the gap between the learning objective and evaluation metrics by a reference-free evaluation.
Implementation Details.
We implement our experiments in Huggingface [43] on 4 NVIDIA A
GPUs.
We build our models based on BART (facebook/bart-large) for CNN/DM and PEGASUS(google/pegasus-xsum) for XSum following their hyperparameter settings, as they obtain betterperformance on each dataset, respectively.
The QA number is set to 8 unless otherwise stated.
Toavoid the model from learning the position information of entities or questions, we sort the entitiesand questions in alphabetical order.
We use Adam optimizer with ϵ as 1e-8 and β as (0.9, 0.999).
Thelearning rate is set to 3e-5.
The warm-up is set to 500 steps for CNN/DM and 125 for XSum.
Thebatch size is set to 8 with gradient accumulation steps of 4.
The beam size is set to 6 for CNN/DMand 8 for XSum.
For pretrained LM models, we finetune the vanilla BART-based or PEGASUS-based
LM on the CNN/DM and XSum dataset, respectively.
More details can be found in Appendix.
4.2 Main Results
Automatic Evaluation.
We evaluate models using standard full-length ROUGE F1 [44].
ROUGE1 (RG-1), ROUGE-2 (RG-2), and ROUGE-L (RG-L) refer to the matches of unigram, bigrams,and the longest common subsequence, respectively.
We then use BERTScore [45] to calculate asimilarity score between the summaries based on their BERT embeddings.
We also evaluate ourapproach with the latest factual consistency metrics, FactCC [6] and QuestEval [7].
FactCC is aweakly-supervised, model-based approach for verifying factual consistency.
QuestEval considers notonly factual information in the generated summary, but also the most important information from itssource text, and finally gives a weighted F1 score QE.
The results are shown in Table 1.
Among factual-consistent baselines, FASum performs relativelypoorly.
One possible reason is that FASum is not trained on pretrained models.
CLIFF achievesbetter BERTScore and faithfulness scores than strong baseline BART.
It can be seen that our modeloutperforms GSum by 0.97 ROUGE-1, 0.90 BERTScore on CNN/DM dataset and 3.35 QE scoreon XSum dataset, indicating questions in the multi-task provides better signals than importantsentences.
Finally, our model outperforms the best baseline SimCLS significantly in most of the [Table] metrics, especially in terms of faithfulness metrics (FactCC and QE), which proves the effectivenessof our methods.
We also show the performance of our model on the test dataset when using the oracle QA pairs to evaluate the upper bound of the benefits brought by the QA task.
We can see that oraclesimprove the performance significantly, with the best-performing model achieving a ROUGE-1 scoreof 50.50.
The results indicate that 1) the model performance has the potential to be further improvedgiven better QA pairs; and 2) the model does benefit from the auxiliary QA task.
Human Evaluation.
Since automatic evaluations are not perfect and can be misleading sometimes,we further conduct a pairwise human evaluation to see whether our generated summaries are faithfulto the source document.
Following Cao et al.
[10], we randomly sample 100 cases from CNN/DM and150 cases from XSum, and then hire two fluent English speakers to evaluate summary informativeness(Inform.)
and factual consistency (Factual.).
For each article, the annotators are shown summariesgenerated by the baseline BART or PEGASUS model and two other systems.
They then rate eachsystem summary against the baseline summary.
Next, the annotators are asked to label text spanswith intrinsic and extrinsic errors.
The compared models are baselines that achieve high automaticscores, and are shown without system names.
Detailed evaluation guidelines are in Appendix.
, or better than BART on CNN/DM dataset.
[Table] [Table]
Table 2 and Table 3 show the manual evaluation results.
Firstly, we can find that there are largerdifferences in terms of Informativeness on the CNN/DM datasets and in Factual consistency on
XSum datasets.
This echos the attributes of the datasets, where the summaries in CNN/DM arelonger and cover more detailed information, while summaries in XSum are shorter and thus requirethe summarization model to have advanced summarization ability.
Secondly, our model is morefrequently rated as being more informative and more factual than SimCLS summaries on bothdatasets.
This is consistent with our automatic evaluation metrics.
Next, the distribution of intrinsicand extrinsic errors is shown in Figure 4(a).
We can see that FES can reduce both intrinsic errors and extrinsic errors more than baseline models.
The kappa statistics are 0.53 and 0.59 for informativenessand factual consistency respectively, indicating the moderate agreement between annotators.
Thestatistical significance between FES and PEGASUS is tested using a two-tailed paired t-test forsignificance for α = 0 .
[Figure]
4.3 Discussions
Ablation Study.
We perform an ablation study on CNN/DM dataset to investigate the influenceof different modules in FES.
First, we remove the multi-task framework to verify the effectivenessof joint learning on summarization.
This means that the QA attention-enhanced mechanism in thedecoder is also removed, so the model degrades to the BART summarizer with entity inputs andmax-margin loss.
Secondly, we keep the QA task but replace the QA attention-enhanced decoder withthe vanilla Transformer decoder.
Thirdly, we remove the max-margin loss to verify the influence ofthe overconfidence of the LM.
Additionally, we randomly select QA pairs to see if the understandingof the model on the document has to be related to key information.
From the results in Table 1, we find without the multi-task framework, the performance of FESdrops greatly by 1.86 RG-L score, 0.43 BERTScore, and 0.60 QE score on CNN/DM dataset,which indicates with the QA multi-task does strengthen the encoder to learn more comprehensiverepresentations.
Next, the QE score drops by 0.28 after the QA attention guidance is removed.
Thisindicates that aligning the QA attention with the summarization attention on the important entitiescan help the model capture gist information from the input.
FactCC score drops by 0.63 after themax-margin loss is removed.
It indicates that preventing the LM from being overconfident canhelp increase faithfulness.
Finally, the performance of FES drops when using random QA pairs asguidance but outperforms BART by a large margin.
This shows that enhancing the understandingof the document is helpful even when it is not always related to the key information.
But theperformance can be further improved by asking questions on key entities.
We further conduct erroranalysis on ablation models, as shown in Figure 4(a).
It can be seen that both multi-task frameworkand max-margin loss are more effective at reducing errors, among which the multi-task setting ismore effective on reducing intrinsic errors, and the max-margin loss reduces extrinsic errors.
[Table]
[Table]
The Number of QA pairs.
To investigate how the number of QA pairs influences the performance ofour model, we conduct experiments on the CNN/DM dataset with 6-14 oracle QA pairs.
From resultsin Figure 4(b), we see that the ROUGE score increases with the number of QA pairs, to begin with.
After reaching 8 pairs, the improvements begin to vanish.
One possible reason is that the answersno longer focus on the important information in the document.
Note that the performance of FESremains at a high level in the range of 8-15 QA pairs, demonstrating the effectiveness and robustnessof FES.
At last, we choose to include 8 QA pairs in our model as default.
[Table]
QA Task Evaluation.
Since our framework also involves a question answering task, investigatingthe QA performance is helpful for understanding the model.
Concretely, we use exact match (EM)and partial match (F1) to evaluate FES model and its ablation model, as shown in Table 4.
Firstly,we can see that with the multi-task framework, both scores show significant improvements.
Thisdemonstrates that the two tasks can benefit each other, and the summarization task can also enhance
QA performance.
Secondly, the EM and F1 scores of QA are relatively high, showing that our modelcan also be used for answering questions on the document.
Margin between FES and the LM.
We show the distribution of the margin m t defined in Eq.
6from our FES and the BART in Figure 4(c).
Firstly, there are still many tokens with negative m tand a large amount of m t around 0 for BART.
This indicates that the LM is probably overconfidentfor many tokens, and addressing the overconfidence problem is meaningful for summarization.
Bycomparison, it can be seen that the number of negative margin cases is significantly reduced in FEScompared with BART.
More precisely, we list the percentage of tokens with negative m t and theaverage m t for each model in Table 5.
Compared with BART, FES reduces the negative m t by 2.33%and increases the average of m t by 0.11 points.
This proves that the overconfidence problem of the
LM is solved to a great extent.
Case Study.
We show several representative cases in Table 6, including the relevant QA pairs andrelevant context from the source document, summaries generated by BART, PEGASUS, or SimCLS,and by our model on two datasets.
The first three cases show summaries with intrinsic errors frombaselines, because the baseline model misunderstands the source document.
Our model makes faithfulsummaries with the help of answering the questions.
For the forth case with extrinsic errors, thebaseline summarization model and LM generate similar tokens, which are not included in the sourcedocument.
FES has no extrinsic error, as it alleviates the overconfidence of the LM by introducing themax-margin loss.
We also show an error analysis in the last row. Both generated summary include anunmentioned name, which can be found in the training dataset. A post-edit operation might solve theproblem, and we look forward to improving it in the future.
Conclusion
In this paper, we propose the multi-task framework with max-margin loss to generate faithfulsummaries.
The auxiliary question-answering task can enhance the model’s ability to understand thesource document, and the max-margin loss can prevent the overconfidence of the LM.
Experimentalresults show that our proposed model is effective across different datasets.
In the future, we aim toincorporate post-edit operation to improve faithfulness.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See broader impact section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See
broader impact section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] See §4.1 andsupplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See §4.1 and supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See §4.1 and supplemental
material.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See §4.1 andsupplemental material.
(b) Did you mention the license of the assets?
[No] The code and the data are proprietary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include our code implementation in supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] The datasets are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] In §5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[Yes] In supplemental material.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[No]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[Yes] See supplemental material.