N on F act S : N on F actual Summary Generation for Factuality Evaluation in Document Summarization (2023)

Abstract: Pre-trained abstractive summarization models can generate fluent summaries and achieve high ROUGE scores. Previous research has found that these models often generate summaries that are inconsistent with their context document and contain nonfactual information. To evaluate factuality in document summarization, a document-level Natural Language Inference (NLI) classifier can be used. However, training such a classifier requires large-scale high-quality factual and nonfactual samples. To that end, we introduce NonFactS, a data generation model, to synthesize nonfactual summaries given a context document and a human-annotated (reference) factual summary. Compared to previous methods, our nonfactual samples are more abstractive and more similar to their corresponding factual samples, resulting in state-of-the-art performance on two factuality evaluation benchmarks, FALSESUM and SUMMAC. Our experiments demonstrate that even without human-annotated summaries, NonFactS can use random sentences to generate nonfactual summaries and a classifier trained on these samples generalizes to out-of-domain documents.

## introduction over the last few years, there have been remarkable improvements in document summarization due to advances in pre-trained language models such as bart and pegasus @xcite . however, these improvements are mainly measured with rouge scores, which assess the quality of a summary using n-gram overlap with references. recent studies show that state-of-the-art models generate up to about 30% nonfactual summaries @xcite @xcite , i.e., summaries that are not entailed by or factually inconsistent with their source document. this demands an 1 codes and models: github.com/asoleimanib/nonfacts document: here are the things that make china's booming commercial hub a unique place in the world's most populous country. ... two dozen colossal ... a massive marble ... shikumen is shanghai's indigenous alleyway housing. ... ## related work this section reviews existing methods for factuality evaluation and standard benchmarks for this task. ## nonfacts method in order to train a classifier to evaluate the factuality of summaries, we need a large set of factual and nonfactual summaries. reference summaries in large summarization datasets such as cnn @xcite and xsum @xcite can be used as factual summaries but the problem is the lack of nonfactual summaries. non-facts takes a set of source documents d and their corresponding reference factual summaries s + and aims to generate a set of nonfactual summaries s -. the final goal is to train a classifier on pairs of factual and generated nonfactual summaries and their corresponding source documents. s -should be similar to actual summarizers output and be indistinguishable from s + using surface features. nonfacts is a text generator model taking as an input i, concatenation of d, a truncated factual summary s + truncated , and a list of random words seeds. for training nonfacts, we set @xmath0, w d }, that means random words consist of n random words w s from s + @xmath1, and m words w d from d (see figure 2 ). it is then trained to generate s + . in other words, nonfacts is trained to select true words from seeds to generate a sentence (summary) given the truncated version of that sentence and its corresponding context document. the input format is the following: the reason to include s + truncated in the input is to make s -more indistinguishable from s + . we set the s + truncated length to half of the s + length that could be the first or last half of the full sentence. in addition, our initial experiments showed if seeds only contains true words it might result in low quality s -as the model has to complete s + truncated using all words, which can be completely irrelevant words to s + truncated . therefore, we include more words than needed in seeds to force the model to select more suitable words. note, seeds contains only half of s + removed words to encourage the model to use the context information in d. words are shuffled and the set does not contain stop words. we use bart-base @xcite as our generator model and use the cnn summarization dataset as our training dataset. the training set has more than 287k samples from which we randomly choose 50k samples for the inference phase. we split summaries into sentences which results in about 900k training pairs (document, sentence). we use a batch size of 40 samples and a learning rate of 3e10 -5 , and train the model for one epoch on 2 nvidia titan x pascal gpus (12gb memory) document: thousands on saturday fled the area in southwestern ivory coast where attacks left seven u.n. peacekeepers and eight civilians dead, according to a u.n. official. ... humanitarian organizations reported saturday they were expecting about 4,000 people in tai, said remi dourlot, a spokesman for the u.n. office for the coordination of humanitarian affairs. ... u.n. operation in cote d'ivoire and ivory coast troops have increased their presence in the area, dourlot said saturday. ... reference factual summary: humanitarian groups expect 4,000 refugees in one camp, a u.n. official says. half summary + seeds: xhumanitarian groups expect 4,000 refugees @xmath2: humanitarian groups expect 4,000 refugees in cote d'ivoire, u.n. spokesman says. document: for the second time during his papacy, pope francis has announced a new group of bishops and archbishops set to become cardinals -and they come from all over the world. ... that doesn't mean francis is the first pontiff to appoint cardinals from the developing world, though.