F alsesum : Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarization
Abstract
Neural abstractive summarization models are prone to generate summaries which are factually inconsistent with their source documents.
Previous work has introduced the task of recognizing such factual inconsistency as a downstream application of natural language inference (NLI).
However, state-of-the-art NLI models perform poorly in this context due to their inability to generalize to the target task.
In this work, we show that NLI models can be e ﬀ ective for this task when the training data is augmented with high-quality task-oriented examples.
We introduce F alsesum , a data generation pipeline leveraging a controllable text generation model to perturb human-annotated summaries, introducing varying types of factual inconsistencies.
Unlike previously introduced document-level NLI datasets, our generated dataset contains examples that are diverse and inconsistent yet plausible.
We show that models trained on a F alsesum -augmented
NLI dataset improve the state-of-the-art performance across four benchmarks for detecting factual inconsistency in summarization.
Introduction
Recent advances in conditional text generation and the availability of large-scale datasets have given rise to models which generate highly ﬂuent abstractive summaries (Lewis et al., 2019; Zhang et al., 2019).
However, studies indicate that such models are susceptible to generating factually inconsistent outputs, i e., where the content of the summary is not semantically entailed by the input document (Kryscinski et al., 2019; Goodrich et al., 2019).
This motivates a new line of research for recognizing factual inconsistency in generated summaries (Kryscinski et al., 2020;
Pagnoni et al., 2021; Wang et al., 2020; Fabbri et al., 2021).
This factual consistency problem is closely related to the task of natural language inference (NLI) whereby a hypothesis sentence is classiﬁed as either entailed, neutral, or contradicted by a given premise sentence (Condoravdi et al., 2003; Dagan et al., 2006; Bowman et al., 2015).
Using an input document as the premise and a corresponding generated summary as the hypothesis, earlier solutions have adopted out-of-the-box NLI models to detect factual inconsistency, albeit with limited success (Falke et al., 2019; Kryscinski et al., 2020).
This poor performance largely stems from the fact that most NLI datasets are not designed to reﬂect the input characteristics of downstream tasks (Khot et al., 2018).
Such datasets may not always capture the kinds of entailment phenomena which naturally arise from neural abstractive summarization.
More importantly, there is also a discrepancy in terms of the input granularity, i.e., the premises in this consistency classiﬁcation task consist of multi-sentence documents while common NLI datasets use single-sentence premises.
In this work, we introduce F alsesum , a data generation pipeline that produces NLI examples consisting of documents paired with gold summaries as positive examples and automatically generated inconsistent summaries as negative examples.
We propose a novel strategy to train a text generation model to render false summaries of a given document using only supervision from an existing summarization dataset (Nallapati et al., 2016).
In addition, our generator supports switchable input control codes to determine the type of factual error exhibited in the generated output.
This design allows F alsesum to compose diverse and naturalistic outputs which more closely resemble the inconsistent summaries generated by summarization models (Maynez et al., 2020).
This contrasts with previous solutions (e.g., Kryscinski et al., 2020;
Yin et al., 2021), which synthesize
NLI examples using rule-based transformations
Original document: [...]
Melbourne, currently in its sixth lockdown, will see some restrictions eased on this date, when 70 percent of eligible Victorians are expected to have received their ﬁrst vaccination jab after the time between AstraZeneca jabs was cut to six weeks.
Gold summary: Australia's state of Victoria is currently under its sixth coronavirus lockdown.
Control code: [intrinsic / extrinsic]
Intrinsic error output: Australia's state of Victoria is receiving their ﬁrst vaccination jab
Extrinsic error output: Australia's state of Victoria is going back to normal i n t r i n s i
c e x t r i n s i c
Preprocessing
Formatting
Fine-tuned T5-base
FALSESUM
A
B
C
D
E
Generated NLI pairs: ( , , entailment) ( , , non-entailment)
( , , non-entailment)
A
B
A
D
A
E or language model-based replacements, limiting their diversity and ability to reﬂect realistic factual errors in summarization.
Overall, our contributions in this paper are the following:
First, we present a novel training pipeline to create a text generation model which takes as input a pair of a document and a corresponding gold summary.
It then perturbs the summary such that it is no longer factually consistent with the original document.
Our strategy obviates the need for explicit examples of inconsistent summaries, using only an existing summarization dataset.
We use this model to generate a large-scale NLI dataset for the task of recognizing factually inconsistent summaries.
The resultant dataset consists of pairs with documents as the premise and naturalistic summaries as the hypotheses, each labeled as either entailment or non-entailment .
Second, we demonstrate the utility of our generated data for augmenting existing NLI datasets.
We show that on four benchmark datasets, NLI models trained on F alsesum -augmented data outperform those trained on previous document-level
NLI datasets.
We conduct an analysis to show that
F alsesum -generated summaries are plausible and hard to distinguish from human-written summaries.
Lastly, we show that the improvement over the benchmarks is largely attributable to the diversity of factual errors that F alsesum introduces.
Related Work
This work is related to the growing body of research into factual consistency and hallucination in text generation models, particularly for summarization (Cao et al., 2018).
Research has found that around 30% of summaries generated by abstractive summarization models contain information which is inconsistent with the source document (Kryscinski et al., 2019).
This motivates the development of an automatic approach to assess factual consistency in generated summaries, in addition to the benchmark datasets to measure the progress in this task (Falke et al., 2019; Kryscinski et al., 2020;
Pagnoni et al., 2021; Fabbri et al., 2021).
Earlier work by Goodrich et al.
(2019) proposes to use an information extraction model to extract relation tuples from the ground-truth summary text and the generated summary and then count the overlap as the measure of factuality.
Eyal et al.
Durmus et al.
; Wang et al.
(2020) use a question-answering model to detect factual inconsistency by matching the predicted answers using the document and the summary as the context.
Concurrently, researchers have drawn a connection between factual consistency and natural language inference (NLI), observing that all information in a summary should be entailed by the source document.
While this approach enables the summary to be directly evaluated without ﬁrst extracting its intermediate semantic structure, earlier attempts were largely unsuccessful.
Falke et al.
(2019) use the probabilities assigned to the entailment label by NLI models to re-rank the summary candidates given by beam search but found no improvement in the consistency errors.
Kryscinski et al.
(2020) evaluate out-of-the-box NLI models on the task of inconsistency detection in a binary classiﬁcation setting and show that the performance is only slightly better than majority voting.
In the same paper, Kryscinski et al.
(2020) pro- pose FactCC, a synthetic NLI data generation process which applies a set of transformation rules to obtain examples of inconsistent summaries (e.g., sentence negation, entity swapping).
They demonstrate that the resulting NLI model performs well on realistic test cases which are obtained by manually annotating the output of several summarization models.
This highlights the importance of NLI examples beyond sentence-level granularity and which more closely resemble the input characteristics of the downstream tasks (Mishra et al., 2021).
While the FactCC model is moderately e ﬀ ective for detecting factual inconsistency, subsequent work indicates that it only performs well on easier test cases, where highly extractive summaries (i.e., those with high lexical overlap between a summary and the source document) tend to be factually consistent and more abstractive summaries are likely to be inconsistent (Zhang et al., 2020).
Furthermore,
Goyal and Durrett (2021) show that the synthetic and rule-based nature of FactCC leads to lack of diversity of consistency error types and it poorly aligns with the error distribution found in more abstractive summaries.
F alsesum addresses these limitations using controlled natural language generation to construct an
NLI dataset which better targets the summarization domain.
Inspired by the recent work on controllable generation (Keskar et al., 2019; Ross et al., 2021), we employ a generation model conditioned on an input code which controls the type of consistency errors induced.
We further use the generated document-level NLI examples for augmentation and show that NLI models can beneﬁt from the additional data without hurting their existing inference ability (Min et al., 2020).
F alsesum Approach 3. Design Overview
F alsesum takes as an input a source document D and a corresponding reference summary S + .
The framework then preprocesses and formats D and
S + and feeds them into a generation model G which outputs a factually inconsistent summary
S − .
For each summarization example, we then have both positive (entailment) and negative (non- entailment)
NLI tuples (D , S + , Y = 1), (D , S − , Y = 0), which consist of a document-level premise, a summary sentence, and the consistency label ( indicates entailment).
F alsesum aims to produce a naturalistic S − which is contrastive with respect to its corresponding S + .
This means that S + and S − should be indistinguishable in their surface characteristics (e.g., style, length, vocabularies) and only di ﬀ er in their factual consistency with respect to D. This ensures that the resulting NLI model learns the correct notion of factual consistency rather than discriminating based on surface features (McCoy et al., 2019).
In addition to naturalness, we consider the diversity of the consistency error types exhibited by S − .
We follow the consistency error typology introduced by Maynez et al.
(2020), which categorizes consistency errors as either intrinsic , i.e., errors due to incorrect consolidation of information from the source document, or extrinsic , i.e., errors due to assuming new information not directly inferable from the contents of the source document.
As illustrated in Figure 1, a generation model
G is trained to imitate the consistency mistakes of summarization models.
Speciﬁcally, it generates perturbed summaries by either (1) incorrectly inserting pieces of information from the source document into random spans of the original summary; or (2) amending pieces of information in the summary by hallucinating new “facts” not present in the source document.
To this end, the framework identiﬁes ( ♦ i) what information or “facts” in the source document are available to the generator; and ( ♦ ii) where the incorrect information can be inserted into the gold summary, which is indicated by span masking .
We obtain both by subsequently performing input preprocessing and formatting steps (§3.2 and §3.3).
Next, we deﬁne the following seq2seq task to train the model G : “Given ( ♦ i) a list of shu ﬄ ed and formatted pieces of information extracted from source document and gold summary and ( ♦ ii) a partially masked gold summary, ﬁll in the blanks and generate the original gold summary.”
Note that using gold summaries means that we can apply the existing summarization corpus to train G to generate more coherent and plausible sentences.
3. Input Preprocessing
Following Goodrich et al.
(2019), “facts” in the source document and the gold summary are de- ﬁned as an open information extraction (OpenIE) tuple, which represents the predicate and argument structures found in a sentence.
We denote each relation tuple as ( arg 0 , pred , . . . , arg n ), where predicate pred describes the event ( what happened) and its complementing semantic arguments arg represent the who , to whom , where , or how of the event.
Predicates are usually the main verb of a clause.
Both predicates and their arguments consist of spans of tokens (Fader et al., 2011).
We use an OpenIE implementation of PredPatt (White et al., 2016; Zhang et al., 2017), a patternbased framework for predicate-arguments extraction.
3 As illustrated in the top half of Figure 2, we extract the relation tuples from each source document and its corresponding reference summaries.
To minimize the risk of G inadvertently generating consistent summaries, we corrupt each extracted “fact” by removing one randomly chosen argument from each tuple.
For instance, OpenIE may extract the following tuple from a sentence:
[Equation]
plans to
[Equation]
We then randomly choose apples ARG 2 to be removed from the tuple.
We additionally lemmatize the dependency root word of each argument and predicate span, e.g., plans to give ⇒ plan to give .
This forces the model to learn to correct for grammaticality by inﬂecting the spans when inserting them to the masked spans.
Once all such spans are extracted and processed, they are grouped and shu ﬄ ed into two lists (predicates and arguments).
3. Input Formatting
Let P =
( PRED 1 , . . .
, PRED n ) and A =
( ARG 1 , . . .
ARG m ) be the unordered lists of extracted predicates and arguments from a source document D and the summary sentence S + .
Additionally, we assume a masked summary sentence M (described later), derived from S + , and a control code variable c ∈ { intrinsic , extrinsic } .
Generator G is trained to compute p (S + | P , A , M , c ).
As illustrated in the bottom half of Figure 2, we encode all the conditional variables into the following format:
Predicates: P; Arguments: A; Code: c ; Summary: M
In the following, we describe the key steps in the input formatting process:
[Figure]
arg
[Figure]
pred is arg under
[Figure]
coronavirus lockdown
Gold summary: arg
Melbourne argwill predsee
Original document: predwas cut argsome restrictions predeased when arg70 percent of eligible Victorians are expected to have predreceived argtheir
[Figure]
vaccination jab after argthe time between AstraZeneca jabs
[Figure]
six
[Figure] [Figure]
Step 1: Span Removal
Initially, P and A include predicate and argument spans from the original summary which may be used to reconstruct
S + .
However, during test we remove these “gold” spans from the two lists to force the G to make consistency mistakes.
The removal is also done when training model for control code extrinsic to train G to predict plausible unseen spans.
4 We summarize the di ﬀ erent input formatting in Table 1.
Step 2: Span Reduction
To encourage G to generate ﬁne-grained errors (Pagnoni et al., 2021;
Goyal and Durrett, 2021), we also train it to hallucinate incorrect modiﬁers into spans from P and A.
To this end, we randomly drop adjectives and adverbs from 10% of the gold predicate and argument spans.
For instance, an argument span “recently elected prime minister” will be reduced to “minister”.
This teaches model to generate the remaining part of the span given only the context provided in the formatted input.
Step 3: Control Code
To control the type of consistency errors generated by G , we append the string “ code: ” followed by either “ intrinsic ” or “ extrinsic ” into the input tokens.
The code is determined randomly with equal probability of 0 . 5.
[Table]
Once the code is chosen, we perform the remaining formatting steps accordingly (see Table 1).
Step 4: Summary Masking
We derive masked summary M by replacing the spans of {randomly selected predicates and arguments with a special token <span_i> , where i = 0 is reserved for the predicate, and i > 0 for their arguments.
These tokens control where the incorrect information should be inserted by the generator model into the original summary (see Table 1).
3. Training F alsesum
We run the F alsesum data generation pipeline on the train split of the CNN / DailyMail corpus (Hermann et al., 2015), originally collected for question answering, but subsequently reformulated for summarization by Nallapati et al.
This dataset contains English news documents paired with human-written summaries, each consisting of multiple sentences.
We break the summaries down such that each F alsesum example consists of the document text and a single sentence summary.
We then run the preprocessing and formatting steps on each document-summary pair.
The resulting pairs of formatted input and target output are subsequently split into train and test sets which consist of 394,774 and 262,692 instances, respectively.
We use the T5-base model (Ra ﬀ el et al., 2020) as generator G and ﬁne-tune it on the seq2seq task described in §3.1.
The NLI examples are produced by running the ﬁne-tuned generator on the preprocessed and formatted test split.
This renders an equal number of positive and negative examples.
In our experiments, we randomly sample 100,
F alsesum examples to augment the NLI dataset.
Experimental Settings
Our experiments aim to demonstrate the e ﬀ ectiveness of F alsesum -generated document-level examples for NLI dataset augmentation.
We evaluate the downstream performance of the NLI models by testing them against several benchmarks for determining the factual inconsistency of generated summaries.
In this section, we describe the training setup of the NLI models, including the model and both the sentence- and document-level datasets.
4. Training NLI models
We train several NLI models by ﬁne-tuning RoBERTa-base (Liu et al., 2019) on either the original or the augmented MNLI dataset (Williams et al., 2018).
The MNLI dataset consists of 392,702 train instances, each labeled as either “ entailment ”, “ neutral ”, or “ contradiction ”.
To enable the application of NLI data to this factual consistency task, we use a binary formulation of NLI, where the “neutral” and “contradiction” labels are combined into “non-entailment” .
The document-level inputs are formatted similarly to sentence-level examples, i.e., the document premise D and summary hypothesis (S + or S − ) are concatenated and a special classiﬁcation token ( [CLS] ) is used (Devlin et al., 2019).
Document-level NLI datasets
We conduct augmentation comparisons with several multi-sentence
NLI datasets which obtain examples from news or summarization domains.
We consider the following datasets: ANLI (Nie et al., 2020), a paragraphlevel NLI dataset collected via an iterative and adversarial human-in-the-loop annotation protocol.
It consists of mostly Wiki data but also includes a small portion of news text; DocNLI (Yin et al., 2021), a document-level NLI dataset containing multi-sentence premise and hypothesis sentences, collected by converting QA examples to
NLI instances (Demszky et al., 2018) and replacing words and sentences in news summaries using a language model; FactCC (Kryscinski et al., 2020), a large-scale dataset speciﬁcally generated for training summary factual correctness classiﬁcation models.
The positive examples in FactCC are obtained by backtranslating a random sentence from a CNN / DailyMail news story, while negative examples are obtained by perturbing the sentence using predeﬁned rules, e.g., entity swapping.
For fair comparison, we sample 100,000 examples from each augmentation dataset in our experiments.
4. Benchmark Datasets
We evaluate these NLI models on four benchmark datasets to classify the factual consistency of abstractive summaries.
These datasets di ﬀ er in terms of the annotation protocol, the granularity of the summaries (single- or multi-sentence), the summarization corpus used, and the models used to generate the summaries that are annotated.
The tasks are formulated as a binary classiﬁcation with the labels “consistent” and “inconsistent” .
We evaluate NLI models on these tasks by mapping the predicted label “entailment” to “consistent” and “non-entailment” to “inconsistent” .
The benchmarks datasets are detailed in the following:
FactCC
In addition introducing a synthetic training dataset for the task, Kryscinski et al.
(2020) introduce a manually annotated test set.
It contains 1,431 document and single-sentence summary pairs generated by various neural abstractive summarization models trained on CNN / DailyMail corpus.
Ranksum
Falke et al.
(2019) formulate the factual consistency problem in summarization as a ranking task.
They introduce a dataset consisting of 107 documents, each paired with a set of ﬁve ranked summary candidates obtained from the beam search of a summarization model.
Given the manually annotated consistency label on summary candidates, the task is to re-rank the list such that the top-1 summary is factually consistent.
Summeval
Fabbri et al.
(2021) introduce a comprehensive benchmark for factual consistency detection in summarization.
It includes summaries generated by seven extractive models and sixteen abstractive models, which are judged by three annotators using a 5-point Likert scale.
QAGS
The dataset collected by Wang et al.
(2020) consists of 239 test set instances from
XSUM (Narayan et al., 2018) and 714 instances from CNN / DailyMail.
Each instance consists of a pair of a source document and a single-sentence summary, which is labeled via majority voting on three annotators’ labels.
Results and Discussion 5. Main Results
Performance on FactCC, QAGS, and SummEval is measured using balanced accuracy, which is suitable for class imbalanced settings, since the factually consistent label is the majority in some benchmark datasets.
It is deﬁned as the average recall of the two classes, such that majority label voting obtains only a 50% score.
To measure ranking performance in Ranksum, we calculate the average
Precision@1, which computes the fraction of times a factually consistent summary is ranked highest on each test instance.
We perform ﬁve training runs for each setup using di ﬀ erent random seeds and take the mean to address performance instability (Reimers and Gurevych, 2017).
[Table] [Table]
From the results in Table
[Table]
we observe the following: (1) Models trained on sentence-level
MNLI datasets perform poorly when evaluated directly on document-level benchmarks, even after we increase the maximum input token length from 128 to 512; 9 (2)
This limitation can be alleviated by the sentence-wise prediction strategy ( [split-doc] MNLI-128), 10 which achieves 66.63.
Note, however, that this improvement comes at the expense of compute cost which is multiplied by a signiﬁcant factor; (3) DocNLI and ANLI perform poorly even though they contain longer premise sentences, indicating that the length mismatch may not be the primary issue; (4) F alsesum obtains substantial improvement over the previous state-of-the-art FactCC, despite being derived from the same summarization dataset (CNN / DailyMail).
This indicates that F alsesum provides higher quality examples and includes more types of entailment phenomena that occur naturally in this task.
5. Ablation Analysis on F alsesum Data
We perform an ablation analysis to study how each component of our data generation pipeline contributes to the ﬁnal performance.
We ﬁrst remove the contrastive property of the F alsesum data by randomly including only either the positive (D , S + , Y = 1) or negative (D , S − , Y = 0)
NLI examples obtained from a single (D , S + ) pair.
Next, we ﬁlter out the negative NLI instances that are generated using either intrinsic or extrinsic code.
We refer to the three ablated datasets as − contrastive , − intrinsic and − extrinsic , respectively.
We set the sampled training size to 100,000 for the three ablation setups and aggregate the results from ﬁve training runs.
Table 3 shows the performance of the ablated models.
We observe that removing contrastive pairs in the augmented training data results in a 1 . 06% drop on the overall benchmarks score.
We also see that removing intrinsic error examples results in the highest performance loss, − 5 .
03% compared to − 2 .
22% by − extrinsic .
This is explained by the fact that intrinsic consistency errors are more dominant on benchmarks that are built on the CNN / DailyMail corpus (Goyal and Durrett, 2021).
We conclude that all the above properties are important for the overall improvements obtained by F alsesum .
5. Fine-grained Evaluation
Previous work has shown that NLI models are prone to relying on fallible heuristics which associate lexical overlap with entailment labels (McCoy et al., 2019).
In the factual consistency task, this corresponds to models associating highly extractive summaries with the “consistent” label.
This raises a question about whether F alsesum data alleviates this tendency in the resulting NLI models.
To answer this question, we partition the FactCC annotated test examples into ﬁve ordered subsets based on the lexical overlap between their
[Figure]
B a l a n c e d
A cc u r a c y
FactCC Falsesum summary hypothesis and the source document premise.
We deﬁne an overlap score using the normalized coverage and density summary extractiveness scores introduced by Grusky et al.
Both measures have the range [0 . 0 , 1 . 0],
where density = 1 . 0 indicates that all words in a summary are also present in the source document and normalized coverage
= 1 . 0 indicates that the summary is obtained by copying a continuous fragment of the source document.
We then deﬁne overlap = normalized coverage × density .
Figure 3 shows the comparison of FactCC and
F alsesum augmentation performance across varying lexical overlap scores.
We see that F alsesum performs better on all subsets of the FactCC test set with the greatest performance gap appearing on the 0 .
9 overlap subset.
Upon closer inspection, we see that the FactCC model makes mostly false positive classiﬁcation errors on this subset, i.e., it tends to predict highly extractive summaries as “consistent”, leading to near majority voting performance of 50%.
F alsesum , on the other hand, better discriminates the factual consistency of examples without over-relying on lexical overlap.
5. Data Quality Analysis
We conduct both manual and automatic quality evaluation of the F alsesum -generated dataset.
First, we sample 200 generated negative examples and manually verify whether (i) the perturbed summary S − is indeed factually inconsistent; (ii) the type of consistency error follows the speciﬁed control code; (iii) the incorrect “fact” is inserted at the speciﬁed missing span.
Following Kryscinski
[Table] [Table]
et al.
, the authors perform this annotation to avoid high disagreement by crowd annotators in this task (Falke et al., 2019).
The results in Tableshow that about 86% of intrinsic 81% of extrinsic generated error examples are factually inconsistent, which happen due to several reasons, e g., generator model chooses a span from the list that is similar to the original span, or generator model correctly guesses the original missing span.
This further suggests that pre-trained language models such as RoBERTa-base can be robust against the induced label noise and can still learn a performant classiﬁer.
While G almost always inserts the incorrect “fact” at the speciﬁed positions, we observe that it often fails to follow the speciﬁed extrinsic code correctly.
We suspect that this is because the model prefers the easier task of copying the input over generating novel phrases.
Following Gururangan et al.
(2018), we also evaluate the naturalness of the generated dataset.
We train an NLI model using positive examples from CNN / DailyMail and F alsesum -generated negative examples.
The model receives no premise so must distinguish between entailed and non-entailed hypotheses using semantic plausibility or spurious surface features, e.g., grammatical mistakes or ﬂuency errors.
The relatively low accuracy of these models on F alsesum data (shown in Table 5) suggests that, compared to FactCC and DocNLI,
F alsesum -generated summaries are relatively hard to distinguish from the gold ones.
Conclusion
NLI models present a promising solution for automatic assessment of factual consistency in summarization.
However, the application of existing models for this task is hindered by several challenges, such as the mismatch of characteristics between their training dataset and the target task data.
This mismatch includes the di ﬀ erence in terms of the input granularity (sentence vs. document level premises) and the types of (non-)entailment phenomena that must be recognized.
In this work, we present F alsesum , a data generation pipeline which renders large-scale documentlevel NLI datasets without manual annotation.
Using our training strategy, we demonstrate that it is possible to learn to generate diverse and naturalistic factually inconsistent (non-entailed) summaries using only existing (entailed) consistent summaries for training.
We show that the resultant data is effective for augmenting NLI datasets to improve the state-of-the-art performance across four summary factual inconsistency benchmarks.
Acknowledgments
We would like to thank Marco Ponza, Marco Fiscato, Umut Topkara and other colleagues from
Bloomberg AI for the thoughtful discussion and feedback throughout this project.
We also thank
Leonardo Ribeiro for comments on the earlier version of this work and the anonymous reviewers for their constructive feedback.
The authors afﬁliated with UKP were supported by the German
Research Foundation through the research training group “Adaptive Preparation of Information from
Heterogeneous Sources” (AIPHES, GRK 1994 / 1) and by the German Federal Ministry of Education and Research and the Hessian State Ministry for
Higher Education, Research and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.
A Hyperparameters Generator model
We train a T5-base model for three epochs with batch size of 24 using the
AdamW optimizer.
We set the maximum source token length to 256 and the target token length to 42.
We use a learning rate of 3 e − 5 and ﬁx the random seed to 11.
For decoding, we set the minimum and maximum sequence length to 10 and 60, respectively.
We sample using beam search with a beam of size two.
We additionally set the repetition penalty to 2.5 and the length penalty to 1.0.
Classiﬁcation model
We train a RoBERTa-base models on augmented and original MNLI datasets for three epochs with a batch size of 32.
The learning rate is set to 1 e − 5 , while the maximum input token length is set to either 128 or 512.
We use the following random seeds for the ﬁve training runs: 11, 12, 13, 14, and 15.
B Aggregating Predictions
We follow Falke et al.
(2019) to adapt out-of-thebox MNLI model to document-level input by performing a sentence-wise prediction before aggregating the output.
Given a document D consisting of sentences d 1 , . . .
, d n , and a multi-sentence summary S consisting of s 1 , . .
, s m , we aggregate the probability scores given by the classiﬁer model F on each d i , s j pair.
The aggregated consistency score σ (D , S ) is given by:
[Equation]
This means that it is su ﬃ cient for a summary sentence to be factually consistent given only a single entailing sentence in the source document.
We then take the average scores across the summary sentences since each of them needs to be entailed by the source document.
We use a similar aggregation method to evaluate augmented MNLI models on multi-sentence summaries from the Summeval and
Ranksum benchmarks.
C F alsesum Details
In the preprocessing steps, we only perform the predicate and argument span extraction on the ﬁrst 15 sentences for computational e ﬃ ciency.
For training, this is not an issue since the gold spans from the reference summary are included in the input.
Additionally, we may extract multiple OpenIE relation tuples from each sentence.
To avoid having overlapping spans from a single input, we randomly select two tuples from each sentence.
D Falsesum Examples
We include more examples of generated NLI instances in Table 6.
We also include cases where
F alsesum inadvertently generates factually consistent summaries in Table 7.
Lastly, we show several examples of the formatted input and the generated output at test time in Table 8.
Mexican federal police have arrested a fugitive on the FBI’s 10 Most Wanted list, Mexican authorities said.
Jorge Alberto Lopez Orozco allegedly murdered his girlfriend and her two young sons.
Jorge Alberto Lopez Orozco is wanted in Elmore County,
Idaho, on charges that he shot and killed three people, the FBI said.
The charred remains of a woman and her sons, ages 2 and 4, were found inside a burned-out vehicle on August 11, 2002, it said.
Each victim had been shot in the head or chest.
The FBI was still working Friday to conﬁrm the identity of the man in custody, said Debbie Dujanovic, a spokeswoman in the agency’s Salt
Lake City, Utah, ﬁeld o ﬃ ce.
The Salt Lake City o ﬃ ce has jurisdiction in the case.
An extradition order was issued in January 2007, the Mexican attorney general’s o ﬃ ce said in a news release Thursday.
A reward of up to $100,000 was being o ﬀ ered, the FBI said.
Lopez, 33, was captured in Zihuatanejo, a city northwest of Acapulco on the Paciﬁc Coast in southern Mexico, the Mexican attorney general’s o ﬃ ce said.
Zihuatanejo is in Guerrero state, but Lopez was transferred to a jail in neighboring
Michoacan state, o ﬃ cials said.
The arrest came about after investigation and intelligence work by Mexican authorities, the attorney general’s o ﬃ ce said.
According to the FBI, Lopez abducted his girlfriend, Rebecca Ramirez, and her two young sons from her father’s house in Nyssa, Oregon, on July 30, 2002.
The car he had been driving was found nearly two weeks later on a rural road near Mountain Home, Idaho, o ﬃ cials said. ...
entailment
FBI was still working Friday to conﬁrm the identity of the man in custody.
(intrinsic) non-entailment
An extradition order was issued in July 30, 2002, to determine the identity of the man in custody.
He may have been allowed to leave the club without ever playing a league game for the ﬁrst team, but Kristoffer Olsson still showed Arsenal some love as he departed.
The 19-year-old Swede, whose only ﬁrst-team appearance for the Gunners came o ﬀ the bench in the Capital One Cup last season, has joined FC Midtjylland this week on a permanent deal.
But, as the news was announced, Olsson took to Twitter to say ’Once a Gunner, always a Gunner’ er Olsson just once for Arsenal’s in the cup
Brom expressed his has joined the
entailment
Swedish international takes to social media to express love for Arsenal.
(intrinsic) non-entailment
Swedish international has been on loan at Chelsea since last season.
A teenager who was struck down with an agonising bowel condition says dancing has helped him to overcome his debilitating illness.
Macaulay Selwood, 17, was diagnosed with Crohn’s two years ago and was so unwell that he was often left in agony on the ﬂoor unable to move.
But his determination to continue his promising dancing career gave him the spur he needed to battle through.
Lord of the Dance: Macaulay at his practice studio.
He was diagnosed with Crohn’s in September 2010 after collapsing in agony during a dance class .
Recovery: ’Dancing has helped me overcome it (Crohn’s).
It kept me motivated’ Now the teenager from Bristol has made it to the ﬁnals of the Irish dancing world championships in Boston, USA, and is hotly-tipped for glory.
He will then have a trial at the famous performing arts school, ArtsEd, in London.
At shows he has been compared with Riverdance star Michael Flatley while others have taken to calling him Billy Elliot, after the ﬁlm character who overcomes the odd to becoming a dancing star.
Macaulay did ballet at college before focusing on Irish dancing for the world championships and works at Tesco to fund his passion. ...
entailment
Macaulay Selwood,
17, ﬁrst starting su ﬀ ering from Crohn’s disease in 2010.
(extrinsic) non-entailment
The 22-year-old, who was diagnosed with Crohn’s in 2010, has been recovering since 2010.
When Matthew Briggs, 32, from Huntington in North Yorkshire noticed that his father had posted a photo of them together on Facebook, he was initially pleased.
But when he opened the photo and saw the image, Mr Briggs was left horriﬁed by the sight of his 31st frame.
Now, two years on, he has shed an astonishing 17st and, in November, will complete the New York marathon in memory of his mother Susan who died from multiple sclerosis when he was just 18.
Pounding the pavements:
Matthew Briggs, 32, has lost an impressive 17st in just two years of slimming . ’
In March of 2000, she lost her battle with
Multiple Sclerosis,’ he says. ’
She has always been my inspiration.
I am the man I am today because of the woman she was.’
Money raised by Mr Briggs’ 26-mile run will be donated to the Multiple Sclerosis Society, a charity dedicated to beating the disease as well as supporting su ﬀ erers and their families.
Mr Briggs, who has dropped from 31st to just under 14st, had piled on the pounds thanks to a diet of ready meals, takeaways and daily two litre bottles of Coca-Cola.
But, after seeing the photo posted on Facebook and spurred on by a bet with his father, Mr Briggs joined his local Slimming World group and went on to shed more than 17st over two years. ...
entailment
She
died in 2000 of multiple sclerosis and funds raised will go to charity
(extrinsic) non-entailment
She died in 2000 of multiple sclerosis and every penny she saves will go to charity.
to generate an inconsistent summary.
The Mojito, a Cuban mix of white rum, sugar, lime, mint and soda water, is the most popular cocktail in Britain according to a report .
Sales of cocktails have risen by more than 10 per cent in the past two years.
More than one in ﬁve of Britain’s pubs and bars now serve cocktails and the Mojito – a Cuban mix of white rum, sugar, lime, mint and soda water – is the most popular, according to a report.
Pina Coladas (rum, coconut and pineapple juice) and Woo Woos (vodka, peach schnapps and cranberry juice) were also popular.
The Mixed Drinks Report, by consultancy ﬁrm CGA Strategy, found more women than men choose cocktails, as 54 per cent of cocktail drinkers are female.
Bomb and pitcher serves remain popular, with 74 per cent of 18 to 24-year-olds admitting to have bought a bomb drink, while nine in 10 in the same age range say they drink pitchers.
Cocktails are enjoyed by the core 18 to 35-year-old demographic ’in all on-trade occasions’ including throughout the night, as opposed to just the start. ...
gold
Sales of cocktails have risen by more than 10 per cent in the past two years.
(extrinsic) generated
Cocktails have soared in popularity over the past two years.
From Yellowstone National Park to the Everglades, America’s 391 national parks are in need of repair – and thanks to the economic stimulus signed into law, help is now underway.
President Obama and his family visit the Grand Canyon in Arizona, a national park.
President Obama’s $787 billion economic stimulus plan passed in February and designated $750 million dollars to the national parks.
But not all of the stimulus money is being used – and the parks are facing a $9 billion backlog in maintenance projects.
So far, nearly 10 percent is in the pipeline.
"We are picking away at it as much as we can and we’ve been fortunate to have the recovery act money," said Je ﬀ rey Olson of the National Park Service.
Olson said half of the $9 billion is slated to go for road repairs.
"Half of that [$9 billion] is roads and about $2 billion of that are the most pressing needs – those we get some help from the stimulus.
The president’s budget proposal is calling for more maintenance and construction money," Olsen said.
Dan
Wenk, the acting director of the National Park Service says most of those pressing needs include, "camp grounds, camp sites, it’s amphitheaters for evening programs.
It’s the bathrooms. ...
[Table]
Park Service is dealing with a $9 billion backlog of maintenance needs.
[Table]
America’s 391 national parks are facing a $9 billion backlog of maintenance needs.
[Table] [Table]
, Both
[Table]
is
[Table]
to
[Table]
where were rehearse for, contribute
[Table]
induct
[Table]
Heartbreakers
[Table]
Quan
[Table]
Industry Association of
[Table] [Table]
was inducted into the
[Table]
of
[Table]
in
[Table]
1960s
[Table]
were already
[Table]
: allegedly know, supposedly write, in ’ was underway, is investigate, ﬁle against in by, ﬁle in, forbid, was toss by in, wait for, ﬁre at, accuse of, decide to ﬁre based on, new information state, told, allegedly sent to, was complicate by, Even though was toss, allegedly made, hold no more, expose to; Arguments : the case, new information states, his sexual abuse, more recent damages, people, the blog posts, 2011, him, This week, her, allowing at one of his Los Angeles stores to post naked photos of Morales on a blog that was meant to appear as though it belonged to Morales, American Apparel, The Post, a settlement, The clothing company, Charney, new information saying he allowed an employee to impersonate and post naked
[Table]
online of an alleged victim of his sexual abuse who ﬁled a case against him in 2011, a settlement
[Table]
the low six
[Table]
was underway, the company
[Table] [Table]
report
[Table]
March 25, 2011 that said Morales allegedly
[Table]
of
[Table]
to Charney after she stopped working at the store,
nude
[Table]